"Explainable AI" gets used as a blanket reassurance: the model is a black box, but don't worry, it's explainable. In practice, the term covers a specific and limited toolkit, and the honest answer to "can you explain this decision?" is usually "partially, and here's exactly what part." That distinction matters more as AI moves into lending, hiring, and healthcare decisions that people can lose money, jobs, or care over — and as regulators start asking for explanations as a matter of right, not courtesy.

This guide separates what explainable AI (XAI) methods actually do from what marketing copy implies they do, walks through the main technical approaches in plain language, states their real limits — especially for large language models — and lays out when an explanation is legally required rather than optional. For the broader accountability picture this sits inside, see what responsible AI requires more generally.

Interpretability vs explainability

These two words are often used interchangeably, but they describe different strategies for the same problem: making an AI system's behavior understandable to a human.

  • Interpretability means the model itself is simple enough that a person can trace how an input becomes an output by inspecting its structure. A linear regression's coefficients, or a shallow decision tree's branching rules, are interpretable by design — there is no separate "explanation" layer because the model's mechanics are the explanation.
  • Explainability (often called post-hoc explainability) means the model is a black box — a large neural network, a gradient-boosted ensemble, a transformer — and a second, separate method is applied afterward to approximate why it produced a given output.

The practical consequence: an interpretable model's explanation is exact, because it's just a description of the model. A post-hoc explanation of a black-box model is an approximation of the model's behavior, produced by a different, simpler model or procedure. Approximations can be useful and can also be wrong in ways that are hard to detect. Keeping this distinction straight is the single most useful thing a non-specialist can take from this article — most confusion about "explainable AI vs interpretable AI" traces back to conflating the two.

The main methods

Inherently interpretable models

The most reliable way to get an explanation is to not need one: use a model whose structure is transparent — linear/logistic regression, decision trees, rule lists, generalized additive models. These models sacrifice some predictive performance in exchange for a mechanism a domain expert can audit directly, with no separate explanation step and no risk that the explanation diverges from what the model actually computed. In regulated, high-stakes settings (credit underwriting, clinical risk scores), this tradeoff is frequently the right one, and several regulators effectively favor it by making post-hoc explanations of complex models harder to defend under scrutiny.

Post-hoc methods: SHAP and LIME

For classical machine learning on tabular data (the kind used in credit scoring, fraud detection, and clinical risk models), two post-hoc methods have become production standards.

SHAP (SHapley Additive exPlanations) borrows a concept from cooperative game theory — the Shapley value — to assign each input feature a share of "credit" for a specific prediction. Conceptually, it asks: across every possible combination of features present or absent, how much does adding this particular feature change the output, on average? The result is a set of feature-attribution scores that sum to the difference between the prediction and a baseline, giving each feature a signed, additive contribution.

LIME (Local Interpretable Model-agnostic Explanations) takes a different route: it perturbs the input slightly, observes how the black-box model's predictions change nearby, and fits a simple, interpretable model (typically a sparse linear model) to that local neighborhood. The simple model's coefficients become the explanation — a local, linear approximation of a much more complex decision surface.

In short: SHAP asks "how much did each feature contribute, accounted for game-theoretically," while LIME asks "what simple model looks like this one, just around this specific point." SHAP tends to be more theoretically grounded and consistent across similar inputs; LIME tends to be faster and more flexible about what counts as a "feature" (useful for text or image inputs). Neither is strictly better — the choice depends on whether consistency or speed/flexibility matters more for the use case, which is the core of the "SHAP vs LIME" decision practitioners face.

Mechanistic interpretability

A newer and more ambitious research direction tries to skip approximation altogether and reverse-engineer what a neural network is actually computing internally — which circuits of neurons and attention heads implement which sub-computations. Anthropic's 2025 work on circuit tracing and attribution graphs (published at transformer-circuits.pub) is a prominent example: it traces chains of internal features that causally connect a model's input to its output for specific prompts, moving toward genuine mechanistic explanation rather than statistical approximation. This is real progress, but it remains far from a complete or fully faithful account of how frontier models work — it currently explains fragments of behavior on selected examples, not the full reasoning behind arbitrary outputs at scale.

The limits you should know

Every explainability method above has boundaries that are easy to gloss over in a vendor pitch or a dashboard tooltip.

  • Association, not causation. SHAP and LIME describe which features are statistically associated with a prediction in the model's learned function — not which factors caused the underlying outcome in the real world. A feature can get a large attribution score because it's correlated with the true causal driver, not because it is one.
  • Instability under correlated features. When input features are correlated with each other (common in real data), both methods can produce explanations that shift noticeably with small changes in the data or the sampling used to compute them — the same model, on the same decision, can yield different "why" stories depending on incidental choices in how the explanation was generated.
  • They don't scale to explaining LLM reasoning. SHAP and LIME were built for models with a fixed, well-defined set of input features — a spreadsheet row, not a paragraph. They aren't the right tool for explaining why a large language model produced a particular sentence, and no post-hoc feature-attribution method currently gives a reliable account of an LLM's actual generation process.
  • Chain-of-thought is not a faithful explanation. When an LLM writes out step-by-step reasoning before an answer, that text is not guaranteed to reflect the computation the model actually performed to reach its answer. The model can produce plausible-sounding reasoning that doesn't match its real internal process, or reach the same answer regardless of what the stated reasoning says. Treating a chatbot's stated reasoning as a trustworthy explanation of its decision is one of the more common and consequential mistakes organizations make when deploying LLMs in decision-adjacent workflows.

NIST's AI Risk Management Framework treats explainability and interpretability as core trustworthiness characteristics precisely because of these gaps — the framework asks organizations to document what an explanation covers and what it doesn't, rather than presenting any single method as sufficient on its own.

When is an explanation required?

In several jurisdictions, "we can't fully explain it" is not a legally acceptable answer for certain automated decisions.

GDPR Article 22 (EU/EEA)

Article 22 of the GDPR gives individuals the right not to be subject to a decision based solely on automated processing that produces legal or similarly significant effects on them, with limited exceptions. Where those exceptions apply, the regulation's broader transparency obligations require controllers to provide meaningful information about the logic involved, as well as the significance and envisaged consequences of that processing.

EU AI Act, Article 86

Article 86 of the EU AI Act creates a standalone right to explanation of individual decision-making for people affected by certain high-risk AI systems: where a high-risk system's output produces legal effects or similarly significantly affects a person's health, safety, or fundamental rights, that person can request a clear and meaningful explanation of the AI system's role in the decision-making procedure and the main elements of the decision taken.

CJEU Case C-203/22 (Dun & Bradstreet, 27 February 2025)

The Court of Justice of the EU's ruling in CK v Magistrat der Stadt Wien and Dun & Bradstreet Austria GmbH addressed a case where a company refused to explain the logic behind an automated creditworthiness score, citing trade secrets. The Court held that individuals are entitled to a meaningful explanation of how an automated decision was reached — sufficient to understand which of their personal data was used and how — and that a blanket refusal on trade-secret grounds is not acceptable; competing interests must be weighed, with courts and data protection authorities able to access the full method where needed to test the controller's claims. It's a significant marker that "the algorithm is proprietary" is not, by itself, a complete legal shield against explanation requests.

United States: adverse-action notices

US consumer credit has its own explanation requirement, independent of the EU framework. Under the Equal Credit Opportunity Act (ECOA) and its implementing Regulation B, a creditor that denies an application (or takes other adverse action) must provide specific, principal reasons for that decision — a requirement that predates modern machine learning but applies regardless of whether the underlying model is a simple scorecard or a complex algorithm. Using a model too opaque to produce those specific reasons is, in that sense, a compliance risk in itself, not just an ethics concern.

Practical guidance

For teams building or buying AI systems that touch consequential decisions, a few principles cut through most of the confusion:

  • Match the method to the stakes. For lending, hiring, and clinical decisions, favor inherently interpretable models where performance allows it, or pair a black-box model with SHAP/LIME explanations that are documented, tested for stability, and reviewed by someone who understands their limits — not just generated and displayed.
  • Don't let an LLM narrate its own decision as if that narration were the explanation. If an LLM is part of a consequential workflow, the explanation of what drove the outcome needs to come from logging, retrieval traces, or structured outputs — not from asking the model to explain itself in prose.
  • Write down what your explanation covers and what it doesn't. A one-paragraph disclosure — "this explanation shows the top features associated with this prediction; it does not establish that these factors caused the outcome, and small changes in input data can shift the ranking" — does more real accountability work than a polished-looking chart with no caveats.
  • Treat legal explanation rights as a floor, not a ceiling. Building toward GDPR Article 22, EU AI Act Article 86, and ECOA/Reg B adverse-action requirements from the start is far cheaper than retrofitting explainability into a system after a regulator or a rejected applicant asks for one.
  • Fold explainability into a broader governance structure rather than treating it as a bolt-on report. It sits alongside the fairness questions covered in our piece on why you can't satisfy every fairness metric at once, and alongside the framework choices discussed in NIST AI RMF vs ISO/IEC 42001.

The bottom line

Explainable AI is a real and useful set of techniques, not a solved problem. Interpretable models give you an exact account of their own mechanics; SHAP and LIME give you a useful but approximate, sometimes unstable picture of a black box's behavior; mechanistic interpretability is making genuine progress on reverse-engineering neural networks but isn't close to covering frontier models end to end; and LLM chain-of-thought is not a reliable window into what the model actually did. Meanwhile, the legal bar is rising: GDPR Article 22, the EU AI Act's Article 86, the CJEU's 2025 Dun & Bradstreet ruling, and US adverse-action rules all point the same direction — toward a "meaningful" explanation being an entitlement for people affected by consequential automated decisions, not a feature to advertise when convenient and skip when hard.

See how AI Coalition Network scores companies on transparency and explainability practices in our methodology.