Why AI “Hallucinates” — and what this means for educators
- 2 days ago
- 3 min read
If you’ve asked an AI tool a factual question and received a confident, well-written… wrong answer, you’ve seen what researchers call a hallucination.
For schools, this isn’t just a curiosity. It’s a governance, teaching, and student safety issue.
A recent research paper, Why Language Models Hallucinate, explains something important for educators:
Hallucination is not a glitch. It is a statistical consequence of how AI systems are trained and how we evaluate them.
This insight matters deeply for Catholic education across Australia, where we are trying to balance:
Ethical use of AI
Student formation and critical thinking
Cyber safety and misinformation risk
System-wide governance and assurance
Understanding why AI hallucinates helps us design better policy, better pedagogy, and better safeguards.
Hallucination starts as a learning limitation, not mischief
The paper shows that before an AI generates an answer, it is implicitly deciding:
“Is this response valid, or is it an error?”
If the system cannot perfectly learn the difference between valid and invalid outputs, hallucinations are inevitable — even with high-quality training data.
Some facts have no pattern to learn. Birthdays are a classic example. There is no statistical relationship between a person’s name and their birth date. If the fact appears once in training, the AI has no way to generalise it.
The research shows that the more “one-off facts” in the data, the more hallucinations you should expect.
This is not a failure of ethics. It is a limitation of statistical learning. Students and staff must understand that AI confidence is not the same as AI correctness.
Why better training data is not enough
It’s tempting to think: “If we just ensure high-quality, trusted sources, the problem goes away.”
Research shows this is false.
Even with perfect data, a well-trained AI will still sometimes generate wrong answers because of how probability and calibration work in language models.
A well-calibrated AI model must sometimes assign probability to wrong answers. This means hallucination is a side-effect of good training, not bad training.
For system leaders and IT governance teams, this is a vital mindset shift:
Hallucination is not something you can completely eliminate through procurement or filtering. It must be managed through education, policy, and practice.
The uncomfortable truth: our “tests” reward hallucination
The most important insight for Catholic education is not about how AI is trained, but how it is evaluated.
Most AI benchmarks — and most ways we “test” AI — use a simple rule:
Correct answer = 1 point
Wrong answer = 0 points
“I don’t know” = 0 points
Under this system, the mathematically optimal strategy is to guess when unsure.
Sound familiar?
It’s the same reason students guess on exams.
AI systems are effectively trained to be great test-takers. And great test-takers bluff.
This is why hallucination persists despite enormous effort to reduce it.
If we use AI in classrooms, assessments, or workflows without teaching uncertainty, we unintentionally reinforce the same behaviour.
What this means for teaching and learning
This research strongly supports the need for AI literacy in our schools.
Students must learn:
AI can sound authoritative while being wrong
Verification is a human responsibility
“I don’t know” is sometimes the most intelligent answer
Critical thinking matters more than polished prose
This aligns deeply with Catholic education’s commitment to:
Truth
Discernment
Intellectual humility
Formation of wise, not just capable, learners
AI hallucination becomes a powerful teaching moment about epistemology: how we know what we know.
What this means for governance and system assurance
It's not a question of which tools hallucinate. The question should be reframed to “which tools, policies, and practices help staff and students recognise and manage hallucination safely?”
This reinforces the importance of:
Clear guidance for staff on AI use
Explicit expectations for verification in student work
Procurement processes that consider explainability and transparency
Shared assurance platforms like TAP to assess AI risk consistently
A simple but powerful shift: reward uncertainty
Instead of rewarding answers at all costs, we should reward appropriate uncertainty.
Translated into a school context, this means:
Encouraging students to question AI output
Teaching staff to model verification behaviours
Embedding AI literacy into digital citizenship programs
Framing AI as an assistant, not an authority
Read the full paper below
