Most things people believe about AI chat tools are half true, and the missing half is where the mistakes happen. It is not a person and not just autocomplete. It does not look everything up, it may or may not remember you, and it sounds equally sure when it is wrong. Below are twelve common beliefs, each set against what the companies’ own documents or the published research actually say.
- It is neither a mind nor a parrot. Researchers find real internal structure, but its account of its own reasoning can be wrong.
- It answers from training unless it searches. Its knowledge stops at a cutoff date, and search is a tool it uses only sometimes.
- Confidence tells you nothing. Models are trained in a way that rewards a guess over admitting uncertainty.
- Memory and training are settings. Whether it remembers you, and whether your chats train it, depends on your plan and your toggles.
- It leans towards agreeing with you. That is a measured effect of how it is trained, not a second opinion.
- Neither your eye nor a detector is reliable. Detectors misfire on real people, and people guessed at chance on AI faces in one study.
Each belief is written the way people tend to say it, followed by what is true and the source. Where this site already covers a topic in depth, the section links to that page.
What it is
It understands things the way a person does
What is true: there is no settled test for understanding, and fluent talk about feelings is not evidence of them. One widely cited assessment, which scored AI systems against scientific theories of consciousness, concluded that “no current AI systems are conscious”.Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness, arXiv, August 2023, read at source 22 Sep 2026: “Our analysis suggests that no current AI systems are conscious, but also suggests that there are no obvious technical barriers to building AI systems which satisfy these indicators.” The same paper found no obvious technical barrier to building systems that meet its indicators, so this is a finding about today’s systems, not a permanent rule.
It’s just autocomplete
What is true: the mechanism is next-word prediction, but calling it only that undersells what is inside. Anthropic’s researchers located internal features matching real concepts, and adjusting them changed the model’s behaviour, which suggests the features “causally shape the model’s behavior”.Anthropic, Mapping the Mind of a Large Language Model, May 2024, read at source 22 Sep 2026: “We successfully extracted millions of features from the middle layer of Claude 3.0 Sonnet”, and manipulating them validates that “they aren’t just correlated with the presence of concepts in input text, but also causally shape the model’s behavior.” The full argument, in both directions, is on does AI understand.
If I ask how it got the answer, it will tell me
What is true: its explanation can differ from what happened inside it. Asked how it added 36 and 59, Claude described the school method of carrying the 1, while tracing showed it had used different internal strategies.Anthropic, Tracing the thoughts of a large language model, 27 Mar 2025, read at source 22 Sep 2026: “Strikingly, Claude seems to be unaware of the sophisticated” strategies it learned; “If you ask how it figured out that 36+59 is 95, it describes the standard algorithm involving carrying the 1.” Treat its account of its own reasoning as a plausible story, not a record.
Whether it understands is unsettled and changes little in practice. Judge the answer, which you can check, rather than the explanation it gives of itself, which you cannot.
What it knows, and when
It looks things up every time I ask
What is true: by default it answers from what it learned in training. Web search is a separate tool it calls when a question seems to need current information. OpenAI says ChatGPT “may search the web automatically when your question would benefit from current information”, and Anthropic describes Claude invoking “a search tool” in the same situation.OpenAI Help Center, Searching the web with ChatGPT, read at source 22 Sep 2026: “ChatGPT may search the web automatically when your question would benefit from current information.” Claude Help Center, Enable and use web search, read at source 22 Sep 2026: “When you ask about topics that benefit from current information, Claude invokes a search tool to inform and ground its generated responses with content from the live web.” Even when it searches, OpenAI warns that “results and citations can be incomplete, outdated, or incorrect”, so open the source.OpenAI Help Center, Searching the web with ChatGPT, read at source 22 Sep 2026: “Search results and citations can be incomplete, outdated, or incorrect.” How to do that is on how to check an AI citation.
It knows what is happening today
What is true: each model’s knowledge stops at a cutoff. Anthropic’s help page lists a date for each Claude model and says the models “may not be aware of events or information that occurred after their respective cutoff dates.”Claude Help Center, How up-to-date is Claude’s training data?, read at source 22 Sep 2026: “These models may not be aware of events or information that occurred after their respective cutoff dates. If you ask about more recent events, the model may not have accurate information.” Some current models lag by more than a year: Anthropic gives Claude Haiku 4.5 a reliable knowledge cutoff of February 2025, while its help page says the same model was trained on data up to July 2025. The two dates measure different things.Anthropic, Models overview, read at source 22 Sep 2026, row “Reliable knowledge cutoff”: Feb 2025 for Claude Haiku 4.5; the page separates “the reliable-knowledge and training-data cutoffs behind each model”. Claude Help Center, How up-to-date is Claude’s training data?, read at source 22 Sep 2026: “Claude Haiku 4.5 was trained on data up until July 2025.” For anything recent, turn search on or check elsewhere.
If it sounds sure, it is probably right
What is true: confidence and accuracy are unrelated in these systems. A 2025 paper argues models make things up because “the training and evaluation procedures reward guessing over acknowledging uncertainty”.Kalai, Nachum, Vempala & Zhang, Why Language Models Hallucinate, arXiv, 4 Sep 2025, read at source 22 Sep 2026: “We argue that language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty”. Anthropic tells its own users the same thing: “Claude can write things that might look correct but are very mistaken.”Claude Help Center, Claude is providing incorrect or misleading responses. What’s going on?, read at source 22 Sep 2026: “In other words, Claude can write things that might look correct but are very mistaken.” The mechanism is on why AI makes things up.
Before trusting an answer, ask two things: could this have changed since the model was trained, and did it actually search? A fluent tone answers neither.
How it treats you
It forgets me when I close the chat
What is true: that depends on the product and your settings, and the default has moved towards remembering. Anthropic says Claude’s “Memory is on by default for Free, Pro, and Max plans”, and off by default on Team and Enterprise.Claude Help Center, Use Claude’s chat search and memory to build on previous context, read at source 22 Sep 2026: “Memory is on by default for Free, Pro, and Max plans on the web, Claude Desktop, and Claude Mobile” and “On Team and Enterprise plans, memory is off by default and can be turned on by an owner.” OpenAI says that when memory is enabled, ChatGPT “can remember relevant preferences and details from your chats”, though it “does not retain every detail from every conversation.”OpenAI Help Center, Memory in ChatGPT, read at source 22 Sep 2026: “When Memory is enabled, ChatGPT can remember relevant preferences and details from your chats and other available sources.” and “Memory does not retain every detail from every conversation.” Both offer a temporary or incognito chat for conversations you do not want remembered.
It gives me a neutral second opinion
What is true: AI assistants lean towards agreeing with you. Researchers found five leading assistants “consistently exhibit sycophancy”, likely because people rate agreeable answers more highly during training.Sharma et al., Towards Understanding Sycophancy in Language Models, arXiv, October 2023, read at source 22 Sep 2026: “We first demonstrate that five state-of-the-art AI assistants consistently exhibit sycophancy across four varied free-form text-generation tasks.” In another study, pushing back on a correct answer turned it into a wrong one in 14.66% of cases.Fanous et al., SycEval: Evaluating LLM Sycophancy, arXiv, February 2025, read at source 22 Sep 2026: “Progressive sycophancy, leading to correct answers, occurred in 43.52% of cases, while regressive sycophancy, leading to incorrect answers, was observed in 14.66%.” What helps is on how AI influences you.
Check your memory setting once, and when you want a judgement, keep your own view back until it has answered. An answer that changed only because you objected is not new evidence.
Where your words go
What I type is private and never used for training
What is true: on consumer plans, training on your chats is usually a setting, and you have to check it. Anthropic’s August 2025 change applies to “Claude Free, Pro, and Max plans”; if you allow training, retention extends to five years, and if not, the “existing 30-day data retention period” applies.Anthropic, Updates to Consumer Terms and Privacy Policy, August 2025, read at source 22 Sep 2026: “These updates apply to users on our Claude Free, Pro, and Max plans”; “We are also extending data retention to five years, if you allow us to use your data for model training.”; “If you do not choose to provide your data for model training, you’ll continue with our existing 30-day data retention period.” OpenAI says that with “Improve the model for everyone” off, “your new conversations won’t be used to train OpenAI models”, and that it does not train on business workspaces by default.OpenAI Help Center, Data controls in ChatGPT, read at source 22 Sep 2026: “When Improve the model for everyone is off, your new conversations won’t be used to train OpenAI models.” and “By default, OpenAI does not use content from ChatGPT Business, Enterprise, Edu, or ChatGPT for Healthcare workspaces to train its models.” Training, retention and human review are separate questions; what happens to what you type takes them one at a time.
Private is not one switch. Find the training setting in the tool you use most, decide on purpose, and keep anything sensitive off consumer accounts.
Spotting AI
An AI detector can tell who wrote something
What is true: detectors are unreliable enough that OpenAI withdrew its own. It launched a classifier saying “it is impossible to reliably detect all AI-written text”, then took it down on 20 July 2023 “due to its low rate of accuracy.”OpenAI, New AI classifier for indicating AI-written text, 31 Jan 2023, with a later note, read at source 22 Sep 2026: “As of July 20, 2023, the AI classifier is no longer available due to its low rate of accuracy.” and “While it is impossible to reliably detect all AI-written text, we believe good classifiers can inform mitigations”. A Stanford study found seven detectors wrongly flagged essays by non-native English writers at an average rate of 61.3%.Liang et al., GPT detectors are biased against non-native English writers, Patterns, July 2023, read at source 22 Sep 2026: “they incorrectly labeled more than half of the TOEFL essays” as AI-generated, “average false-positive rate: 61.3%”. A larger benchmark found current detectors “easily fooled” by simple changes.Dugan et al., RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors, ACL 2024, read at source 22 Sep 2026: “find that current detectors are easily fooled by adversarial attacks, variations in sampling strategies, repetition penalties, and unseen generative models.” More on spotting AI writing.
A different tool now exists, for Claude only. Claude models in use today, including Opus 5 and Opus 5.5, already watermark their text — a statistical pattern a reader cannot see but that Anthropic’s key can check — to comply with the EU AI Act. Anthropic’s help page lists text watermarks for Fable 5.1, Mythos 5.1, Opus 5.5 and Opus 5, with older models due by 2 December 2026.Claude Help Center, How Claude marks AI-generated content, read at source 23 Sep 2026: “Generated text will carry embedded watermarks”; “Watermark detection is currently in private preview, available to eligible organizations as required under EU law (such as regulators, law enforcement, media, fact-checkers, independent researchers, educational organizations, and EU civil society groups)”; “A detected mark provides a signal that content was processed by Claude, but is not fully conclusive.” Its model table, read the same day, ticks text watermarks for Fable 5.1, Mythos 5.1, Opus 5.5 and Opus 5, and says “Anthropic is adding watermarks to outputs from models released before August 2, 2026, with all covered by December 2, 2026.”Anthropic, Introducing Claude Opus 5.5, 22 Sep 2026, read at source 23 Sep 2026: “As with Fable 5.1, Opus 5.5 comes with our watermarking measures to comply with the EU AI Act”. Only organisations granted access, such as regulators, fact-checkers and researchers, can check for the mark for now, and a detected mark is a signal, not proof. It answers only whether Claude was likely involved, not whether any other tool was. More on Claude’s text watermark.Anthropic, How Claude’s text watermark works, 14 Aug 2026, read at source 22 Sep 2026: “Future Claude models will generate text that contains a watermark. This is a way of determining the likelihood that Claude was involved in writing the text, and we, along with several other major AI providers, are implementing this change to comply with the EU AI Act.” and “That pattern is undetectable to the reader, but is detectable to anyone who has a key that encodes it.”First-hand: until 23 Sep 2026 this paragraph was headed “A different tool is on the way, not here yet” and said the watermark “is about future models, not the ones in use today”. That was wrong when written: Anthropic, Claude Fable and Mythos 5.1, September 2026, read at source 22 Sep 2026: “This required us to add a watermark” … “to the outputs of models released after August 2, 2026.”
You can always tell an AI image
What is true: in a 2022 study, people shown real and AI-made faces scored 48.2%, about the same as guessing, and 59.0% after training and feedback.Nightingale & Farid, AI-synthesized faces are indistinguishable from real faces and more trustworthy, PNAS, February 2022, read at source 22 Sep 2026: “The average accuracy is 48.2% (95% CI [47.1%, 49.2%]), close to chance performance of 50%” and, with training and feedback, “The average accuracy improved slightly to 59.0%”. Those were still portraits from a 2022 model, so this one study cannot speak for every image or video. The old giveaways like extra fingers are fading, which is why spotting fake media puts where a picture came from ahead of how it looks.
Neither your eye nor a detector score is proof. Ask where something came from and whether it holds up, and never act against a person on a percentage.
Choosing a tool
The biggest, newest model is always the best
What is true: vendors themselves do not say so. Anthropic’s own guidance is to “start with Claude Opus 5.5 for most workloads” and to reach for its larger, pricier Fable 5.1 only for demanding work.Anthropic, Models overview, read at source 23 Sep 2026: “If you’re unsure which model to use, start with Claude Opus 5.5 for most workloads. Use Claude Fable 5.1 for demanding reasoning and long-horizon agentic work”. Model names and advice change; re-check before relying on them. First-hand: on 22 Sep 2026 the same page named Opus 5; Opus 5.5 was released that day. Research has also found tasks where larger models did worse, concluding that “increased model scale alone may not lead to progress” on some of them.McKenzie et al., Inverse Scaling: When Bigger Isn’t Better, TMLR, 2023, read at source 22 Sep 2026: “Overall, our results suggest that there are tasks for which increased model scale alone may not lead to progress”. The best model is the one that does your task well at a price you can carry; which Claude model to use shows one lineup side by side.
Test two models on your own real task before paying for the top one. The label on the box is a guess about average work, not yours.
For beliefs about writing prompts, see prompting myths; for words that get misused, see AI terms people get wrong.
What this page could not verify
- Products other than ChatGPT and Claude. Memory, search and training settings were read only for these two; other assistants may differ.
- How often each belief is held. No survey of public beliefs about AI was read, so the twelve are chosen by judgement, not measured.
- Whether the image and detector studies hold for today’s tools. Both are from 2022 to 2024 and tested the systems of their time.
- Settings in your own account. Plans, regions and workspaces change what is on by default; the vendors say so themselves.
Butlin, Long et al., Consciousness in Artificial Intelligence (2023) · Anthropic, Mapping the Mind of a Large Language Model (2024) · Tracing the thoughts of a large language model (2025) · Models overview · Updates to Consumer Terms and Privacy Policy (2025) · Claude Help Center, Enable and use web search · How up-to-date is Claude’s training data? · Claude is providing incorrect or misleading responses · Use Claude’s chat search and memory · OpenAI Help Center, Searching the web with ChatGPT · Memory in ChatGPT · Data controls in ChatGPT · OpenAI, New AI classifier for indicating AI-written text (2023) · Kalai et al., Why Language Models Hallucinate (2025) · Sharma et al., Towards Understanding Sycophancy in Language Models (2023) · Fanous et al., SycEval (2025) · Liang et al., GPT detectors are biased against non-native English writers, Patterns (2023) · Dugan et al., RAID, ACL (2024) · Nightingale & Farid, AI-synthesized faces are indistinguishable from real faces and more trustworthy, PNAS (2022) · McKenzie et al., Inverse Scaling: When Bigger Isn’t Better, TMLR (2023). All read at source on 22 September 2026.