Explainable AI (often shortened to XAI) means AI that can give a reason for its answer that a person can understand: why it turned down a loan, flagged a scan or picked one word over another. It exists because many modern AI systems are a “black box”: even the people who built them cannot fully see how they reached a particular result.
Picture a brilliant cook who never follows a recipe and cannot tell you what went into the dish. The food may be excellent, but if a guest falls ill, there is no way to find out why. Explainable AI asks the cook to keep a recipe card, so someone can check what went in and why.
In a sentence: “The bank switched to explainable AI, so when a loan is refused, the letter now says which factors counted against it.”
What does explainable AI mean?
The plain point: it is the effort to make AI’s decisions understandable, so people can check and trust them.
IBM, the computer company, defines it this way: “Explainable artificial intelligence (XAI) is a set of processes and methods that allows human users to comprehend and trust the results and output created by machine learning algorithms.”1
The idea matters most where AI helps make decisions about people: health, money, jobs and policing. Without a reason for a decision, it is hard to tell whether it was fair or simply wrong.
Where the word comes from
“XAI” was the name of a US defence research programme. DARPA, a US defence research agency, ran a four-year Explainable Artificial Intelligence (XAI) programme. A 2019 article about it in AI Magazine by David Gunning and David Aha says the programme “endeavors to create AI systems whose learned models and decisions can be understood and appropriately trusted by end users”, and that “The XAI teams completed the first of this 4-year program in May 2018.”2 The sources checked for this page do not say who first used the phrase “explainable AI”.
Why is AI called a black box?
The plain point: you can see what goes in and what comes out, but not the reasoning in between.
Most of today’s AI is built by machine learning: the system learns its own patterns from examples instead of following rules a person wrote. The result works, but the patterns are stored as a vast set of numbers (its parameters), not as sentences anyone can read. IBM says that for such models, not even their creators “can understand or explain what exactly is happening inside them”.1
Researchers who try to look inside call that work interpretability. Anthropic, the company behind the Claude chatbot, reported in 2024 that it had found millions of recognisable “features” inside one of its models, such as patterns linked to particular ideas. It also said those were only a small part of everything the model had learned.3
What makes a good explanation?
The plain point: a reason only helps if it is true, understandable, and honest about what the system does not know.
NIST, the US government’s standards agency, proposed four principles in 2020.4
| Principle | What it means | Everyday comparison |
|---|---|---|
| Explanation | Means: every answer comes with a reason. | Like: a teacher who marks your work and writes why. |
| Meaningful | Means: the reason makes sense to the person reading it. | Like: a doctor explaining in plain words, not Latin. |
| Accurate | Means: the reason is how it really decided, not a nice story. | Like: a receipt that matches what you bought. |
| Knows its limits | Means: it holds back when it is unsure or out of its depth. | Like: a mechanic who says “that is not my area”. |
Why it matters to you
The plain point: when a chatbot explains itself, that explanation is not guaranteed to be the real reason.
Some chatbots show their “thinking” before they answer (see chain of thought meaning). It looks like an explanation, but tests suggest it often leaves things out. In one Anthropic study, models were slipped a hint and then asked to show their reasoning; on average one model mentioned the hint 25% of the time and another 39% of the time.5 How AI is tested covers what testing can and cannot catch.
Treat an AI’s explanation like its answer: useful, but worth checking. If an AI system makes a decision about you, you can ask the organisation using it what the decision was based on.
Often confused with
- Interpretability: looking inside the model to see how it works. Explainability is giving people a reason they can use. They overlap, and NIST lists them together.
- Transparency: being open that AI is used and what data it learned from. An explanation of one decision is a narrower thing. See AI ethics meaning.
- Hallucination: a made-up answer. An explanation can be made up too, which is why the “accurate” principle exists. See hallucination meaning.
Questions people ask
Can ChatGPT or Claude explain how it got an answer?
It can write an explanation, but that text is produced the same way as the answer, so it may not match what really happened inside. The Anthropic study above found reasoning models often left out a hint they had used.
Why can’t the makers just look inside?
They can see the numbers, but not what they mean. Anthropic describes a model’s inner state as a long list of numbers “without a clear meaning”, which is why turning it into plain reasons is a research field of its own.3
Learn more on this site
- How the inside works: neural network meaning.
- Checking AI: how AI is tested.
- Fairness: AI bias meaning.
- The words: the AI glossary.
- Related: What Is AI Governance? AI Governance Meaning in Plain English.
QUICK CHECK · 3 QUESTIONS
Pick an answer to see if you got it. Nothing is recorded.
1. What is explainable AI?
- AI that comes with a printed user manual
- AI that can give a reason for its answer that a person can understand
- AI that only ever answers yes or no
Show the answer
Answer: AI that can give a reason for its answer that a person can understand. Explainable AI means AI that can say why it gave an answer, such as why it turned down a loan, in a way a person can understand.
2. Why is much modern AI called a ‘black box’?
- You can see what goes in and what comes out, but not the reasoning in between
- It is always sold in a black case
- It only works when the screen is dark
Show the answer
Answer: You can see what goes in and what comes out, but not the reasoning in between. Machine learning stores its patterns as a vast set of numbers, not readable sentences, so even its makers cannot fully see how it reached a result.
3. When a chatbot explains how it got an answer, what does the page say?
- The explanation is always the real reason
- Chatbots cannot write explanations at all
- The explanation may not match what really happened inside, so check it
Show the answer
Answer: The explanation may not match what really happened inside, so check it. The explanation is produced the same way as the answer, so it may leave things out; treat it like the answer: useful, but worth checking.
Related terms: AI ethics · AI governance · Deepfake · Guardrails · Jailbreak · Prompt injection
- 1IBM, What is explainable AI?, updated 26 Feb 2026, read at source 5 Oct 2026“Explainable artificial intelligence (XAI) is a set of processes and methods that allows human users to comprehend and trust the results and output created by machine learning algorithms.” ↑a“These black box models are created directly from the data. And, not even the engineers or data scientists who create the algorithm can understand or explain what exactly is happening inside them or how the AI algorithm arrived at a specific result.” ↑b
- 2Gunning and Aha, DARPA’s Explainable Artificial Intelligence (XAI) Program, AI Magazine 40(2), 24 Jun 2019, read at source 13 Oct 2026“DARPA’s explainable artificial intelligence (XAI) program endeavors to create AI systems whose learned models and decisions can be understood and appropriately trusted by end users.” “The XAI teams completed the first of this 4-year program in May 2018.” ↑
- 3Anthropic, Mapping the Mind of a Large Language Model, 21 May 2024, read at source 5 Oct 2026“We mostly treat AI models as a black box: something goes in and a response comes out, and it’s not clear why the model gave that particular response instead of another.” … “The features we found represent a small subset of all the concepts learned by the model during training” ↑a“The internal state of the model…consists of a long list of numbers (‘neuron activations’) without a clear meaning.” ↑b
- 4NIST, NIST Asks A.I. to Explain Itself, 18 Aug 2020, read at source 5 Oct 2026“AI systems should deliver accompanying evidence or reasons for all their outputs.” … “Systems should provide explanations that are meaningful or understandable to individual users.” … “The explanation correctly reflects the system’s process for generating the output.” … “The system only operates under conditions for which it was designed or when the system reaches a sufficient confidence in its output.” ↑
- 5Anthropic, Reasoning models don’t always say what they think, 3 Apr 2025, read at source 5 Oct 2026“On average across all the different hint types, Claude 3.7 Sonnet mentioned the hint 25% of the time, and DeepSeek R1 mentioned it 39% of the time.” ↑
The comparisons on this page (a cook without a recipe, a teacher, a doctor, a receipt, a mechanic) and the difference drawn between interpretability and explainability are this site’s own ways of explaining, not claims from these sources.
Free printable AI cheat sheets, and AI news every week.
Sixteen one-page AI guides and checklists in one free PDF, no sign-up. And if you want to keep up, the newsletter covers what changed in AI each week, in plain English, with the sources.
SUBSCRIBE FREE ↗ GET THE CHEAT SHEET PACK