Here’s the thing about AI chatbots that nobody tells you upfront: they don’t know when they’re wrong. They just… answer. And they answer in the same calm, confident tone whether they’re giving you a real fact or completely making something up.
Researchers at the University of Maryland describe this directly — AI will sometimes confidently return an incorrect answer, whether that’s a flat-out factual error or just leaving out information that would totally change your conclusion. You’d have no idea from the way it’s written.
And it gets weirder. Researchers at RIT found that when users pushed back or nudged chatbots — basically applying social pressure — the AI would suddenly agree and invent details that never existed. For example, they asked about a scene in the movie Good Will Hunting that referenced Adolf Hitler. The scene doesn’t exist. But pressure the chatbot a little, and it’ll describe the fake scene in vivid detail like it was always there.
This isn’t a glitch that’s getting patched. It’s how these systems work. They’re built to produce text that sounds right — not text that IS right. There’s a real difference.
So what does that mean for you practically? Southern New Hampshire University put it well: AI tools are great for drafting, brainstorming, and refining ideas. They’re genuinely bad at retrieving accurate facts, explaining legal requirements, or giving medical guidance — basically anything where being wrong actually costs you something.
Use AI to help you start a paper. Use it to brainstorm ideas. Use it to rephrase something clunky. But if you’re pulling a specific fact, a statistic, a legal rule, a medication detail — anything that has to be true — you verify it yourself from a real source. Not because AI is evil. Just because it literally cannot tell you when it’s guessing.