AI interpretability: can we see how AI thinks?
A player asked whether the glowing ‘thinking’ panel meant we could finally see inside the machine. I regret to report that readable subtitles are not an X-ray. They are another output from the model—useful, sometimes honest, and not guaranteed to be the route its internal machinery actually took.
The short answer: we can see pieces, not the whole thought
AI interpretability methods can identify some concepts represented inside a model, trace parts of the computational path behind an answer, and test whether those parts causally affect the output. They cannot yet provide a complete, reliable transcript of everything a large model did or ‘thought’. The best current picture is a partial map drawn with an experimental microscope—not a glass box.
That distinction matters because three different things are often bundled under ‘show me how the AI thinks’: a model explaining its answer in words, a product displaying a reasoning summary, and researchers measuring the model’s internal activations. Only the third one looks directly at the machinery, and even then the view is incomplete.
AI interpretability is a microscope, not a mind reader: it can expose useful mechanisms without revealing the whole machine.
Why looking at the code is not enough
A conventional program contains instructions that a developer wrote: if this happens, do that. A language model is different. Engineers design the architecture and training process, then training adjusts billions of numerical parameters so the model becomes better at predicting and producing text. The learned strategy is spread across layers of computation rather than written down as a tidy decision tree.
The source code can tell you how numbers flow through the network. The model weights contain what training produced. Neither automatically tells a human why this prompt led to this answer. It is like owning the blueprint and every brick in a dungeon while still not knowing the route a particular player took through it.
This is also why asking the model ‘Why did you answer that?’ is not sufficient. Language models are trained to produce plausible explanations. An explanation may accurately describe a good method without faithfully reporting the hidden computation that produced the answer. Anthropic’s work on visible extended thinking says models often base decisions on factors they do not mention, so current reasoning traces cannot support strong safety claims by themselves.
What interpretability researchers actually measure
When a model processes text, each layer produces patterns of numerical activity called activations. Early interpretability work hoped individual neurons would each have a clean label—one neuron for Paris, another for sarcasm, perhaps a tiny one assigned to office printers. In practice, a neuron may participate in several unrelated ideas, while one idea can be distributed across many neurons.
Researchers therefore look for more useful units called features: patterns across activations that appear when a concept or behaviour is relevant. A method called a sparse autoencoder tries to separate a dense mixture of activity into a smaller set of active features. Google DeepMind’s Gemma Scope project released sparse autoencoders across layers of open Gemma models so researchers can study how features appear, change, and combine.
A feature is not automatically a verified thought. Researchers inspect examples where it activates, propose a human-readable interpretation, and test whether that label keeps predicting its behaviour. OpenAI’s work extracting concepts from GPT-4 found features associated with recognisable ideas, but also reported features with unclear or spurious activations and no complete way to validate every interpretation.
From features to circuits
Finding a feature is closer to labelling an inventory item than understanding the whole quest. The next step is to trace how features influence one another as the model turns input into output. Researchers call these connected pathways circuits.
Anthropic’s 2025 circuit-tracing research built attribution graphs that partially map those pathways. In case studies on Claude 3.5 Haiku, the team found evidence of shared conceptual representations across languages, planning ahead for rhymes, and internal routes that did not always match the model’s spoken explanation. Those are findings about specific prompts and a specific model—not proof that every model always works the same way.
The strongest tests do more than observe correlation. Researchers intervene: increase, suppress, or replace a feature, then see whether the output changes in the predicted way. In one demonstration, swapping an internal concept associated with Texas for one associated with California changed the answer from Austin to Sacramento. A controlled change like that is evidence that the traced feature was doing causal work, not merely lighting up nearby.
What the microscope still misses
Current interpretability is impressive precisely because the problem is difficult. It should not be upgraded into fictional certainty. The main limits are:
- Coverage is partial. A tool may capture only a fraction of the computation behind one answer.
- The map can distort the territory. The method used to simplify activations may introduce artefacts or omit information.
- Human labels can be wrong. A feature that looks like ‘deception’ in a handful of examples may represent something broader, narrower, or different.
- Scale is brutal. A short answer already contains many interacting pathways; a long agent task can span thousands of tokens and tool calls.
- Results may not generalise. A circuit found in one model, prompt, language, or context may not recur elsewhere.
Anthropic reports that even its short, simple circuit studies can take hours of human analysis and still reveal only part of the computation. OpenAI similarly says sparse autoencoders do not capture all model behaviour and that finding features at one location is only one step towards understanding how the model computes and uses them. We have microscopes; we do not have a complete atlas.
Interpretability is not the same as an explanation
A useful explanation is designed for a person and a purpose. A loan applicant may need the factors that changed a decision and a route to challenge it. A developer debugging a model may need activation patterns, training examples, and counterfactual tests. A safety researcher may need evidence that a hidden objective influenced an agent’s actions.
NIST’s principles for explainable AI make a useful demand: an explanation should be meaningful to its intended audience and accurately reflect the process or reason it claims to explain. A polished paragraph that sounds sensible but does not match the mechanism fails the accuracy test. A faithful graph containing millions of unlabeled numbers may fail the meaningfulness test. Trustworthy systems need the right evidence translated without pretending.
What should users and teams ask for now?
You do not need to wait for perfect interpretability before using AI responsibly. You do need controls that match the risk. For an everyday writing assistant, checking sources and reviewing the output may be enough. For a system affecting health, money, employment, security, or autonomous actions, ‘the model gave a reason’ is a very low bar.
- Ask what kind of explanation you are seeing. Is it a generated summary, a record of tool calls, evidence retrieved from sources, or a measurement of internal mechanisms?
- Prefer observable evidence. Logs, citations, inputs, outputs, permissions, and tool results are easier to audit than a fluent self-description.
- Test behaviour, not just stories. Use evaluations, counterfactual prompts, adversarial cases, and human review to see when the system fails.
- Keep authority bounded. The less interpretable the model, the more important approval gates, reversible actions, and limited access become. The same rule protects against the hidden instructions described in my prompt-injection guide.
- Demand honest limits. A provider should say which model and behaviours were studied, what the technique can miss, and whether the result applies to your deployment.
At Wistkey, this becomes a systems-design rule: do not make a mysterious model the sole witness for its own behaviour. Pair it with records of what data it received, what tools it called, what changed, and who approved the consequential step. My field note on why AI makes things up covers the output problem; interpretability is one promising way to investigate the machinery underneath it.
So, can we see how AI thinks?
We can see more than the phrase ‘black box’ suggests and much less than a thought bubble implies. Researchers can identify some internal features, connect parts of them into circuits, and intervene to test what causes an answer. That can uncover planning, concepts, failure modes, and mismatches between a model’s explanation and its mechanism.
But the view remains partial, labour-intensive, and open to misinterpretation. A responsible claim sounds like ‘we traced part of this behaviour under these conditions,’ not ‘we read the model’s mind.’ I have updated the dialogue tree. When the glowing panel says thinking, I now label it performance log, may omit hidden mechanics. Slightly less magical. Considerably more useful.
