Interpretability and Evals

How I fell into this

I have been falling into the interpretability rabbit hole recently, and I think the honest reason has very little to do with research taste.

I spend a lot of time introspecting. Probably too much. On any given day I will notice a feeling, then stop and ask where it came from, then ask why I said the thing I said, then ask whether the reason I just gave myself is the real reason or a story I assembled afterward because it sounded better. Talking to AI actually sharpened this habit rather than replacing it. Having something patient to think out loud with made me notice how often my explanations arrive after the decision, not before it. I kept doing it because I did not want to be fooled by my own head, or trapped inside a thought that felt urgent but was not true. At the end of the day, a thought is just a thought.

It took me a while to notice that this is the same shape as the interpretability problem.

When researchers open a model and ask what it is actually doing, rather than what it says it is doing, they are running the exact procedure I run on myself. The model gives a fluent account of its reasoning. The account may be accurate, or it may be a plausible narrative fitted to an answer it already reached. You cannot tell from the outside. You have to look inside, and looking inside is hard, partial, and easy to get wrong.

So now I have a slightly ridiculous open question that I have not resolved: is understanding myself easier, or is understanding the model easier? The model has an advantage, because you can read every activation, freeze it, run the same prompt a thousand times, and intervene directly on an internal concept to see what changes. I have none of that. But the model also has billions of numbers doing things nobody designed, and I at least get to sit inside my own experience. Most days I think neither one is easy and I am just choosing which kind of difficulty I prefer.

So this note is partly a study note for Week 6 and partly a post-mortem.

Was LIME and SHAP my first interpretability?

Before any of this I did a school project with LIME and SHAP, and I filed that under “interpretability” for a long time. I think that is about half right. They are post-hoc attribution methods: treat the model as a black box, perturb the inputs, watch how the outputs move, report which features mattered. SHAP does it with the Shapley value borrowed from cooperative game theory, which is the same object I wrote about in my cooperative game note for completely unrelated reasons, and I still find that funny.

What they never touch is the machinery in between. You learn what the model is sensitive to and nothing about how it computes. Mechanistic interpretability wants the internals, and more than that it wants to intervene on them and check whether the behavior changes the way your theory said it would. So the honest version is probably that LIME and SHAP gave me the habit of refusing to accept a number as sufficient, which is the part that carried over, and not much else.

Tracing the thoughts, or building the microscope

Anthropic’s framing is that they are building a microscope, and I think that metaphor is more honest than the MRI-style brain scan Amodei wants. An MRI sounds like something you point at a patient to get a picture back. A microscope is a thing you have to prepare a sample for, and choose a magnification for, and then spend hours squinting at while deciding what you are even looking at.

The method links interpretable features into computational circuits, producing an attribution graph: a partial map of the pathway that turns input tokens into output tokens. The findings from Tracing the Thoughts of a Large Language Model are the ones I would point to if someone asked me for concrete evidence that this works at all.

There is a shared conceptual space across languages. The same features for smallness and oppositeness activate whether the prompt arrives in English, French, or Chinese, before the answer is rendered back into the language of the question. And the sharing increases with scale, with Claude 3.5 Haiku sharing more than twice the proportion of features across languages as a smaller model. That is a mildly unnerving detail if you were hoping models stay modular as they grow.

Simplified attribution graphs for the same prompt in English, Chinese, and French, showing shared multilingual features for the antonym, small, and large concepts.

Figure 1: The same question in three languages routes through shared concept features before being rendered back into the language of the prompt. Source: Tracing the Thoughts of a Large Language Model, Anthropic (2025).

Claude plans ahead. Asked to complete a rhyming couplet, the model was already representing the rhyme word before it wrote the line leading to it. The researchers expected the opposite. They set out to show the model was improvising word by word and found planning instead. The reason I believe it is the intervention rather than the observation. Suppress the “rabbit” concept and the model produces a different sensible rhyme. Inject “green” and it builds a line toward that ending instead.

Three versions of a couplet completion. Unmodified, the model plans and lands on rabbit. With the rabbit concept suppressed it lands on habit. With a green concept injected it lands on green.

Figure 2: Suppression and injection on the planned rhyme. This is the part that turns an observation into evidence. Source: Tracing the Thoughts of a Large Language Model, Anthropic (2025).

Mental math does not work the way the model says it works. Claude adds using two parallel pathways, one computing a rough magnitude and one nailing the last digit, then combining them. Ask it how it got the answer and it describes carrying the one, like a textbook. It is not lying. It learned to do arithmetic internally with no supervision on the procedure, and it learned to explain arithmetic by imitating people. The two were trained separately and they do not match.

$Top: the parallel computational pathways Claude uses to compute 36 plus 59, one approximating the magnitude and one determining the final digit. Bottom: Claude describing the standard carrying algorithm when asked how it got the answer.$

Figure 3: The mechanism on top, the self-report underneath. This is the figure I would keep if I could only keep one. Source: Tracing the Thoughts of a Large Language Model, Anthropic (2025).

That one bothers me the most. It is the cleanest demonstration I know of a system’s self-report and its actual mechanism coming apart completely, with nothing going wrong that the system itself could have noticed. I do this constantly. I think everyone does. The only real difference is that with Claude someone can go and check.

Chain of thought is sometimes faithful and sometimes motivated. Given an easy problem, the intermediate reasoning steps show up as real intermediate computations. Given a hard one with a hinted answer, the model sometimes works backwards from the hint, constructing steps that would justify the target. And when it claims to have computed a cosine it cannot compute, the tools find no evidence of any calculation happening at all.

Side by side comparison. On the left, faithful reasoning computing a square root through real intermediate steps. On the right, motivated reasoning working backwards from the answer the user claimed.

Figure 4: Faithful reasoning on the left, motivated reasoning on the right. Same model, same format, and the difference is invisible from the text alone. Source: Tracing the Thoughts of a Large Language Model, Anthropic (2025).

Multi-step reasoning is real composition, not lookup. Asked for the capital of the state containing Dallas, the model activates a Texas representation and then a separate capital-of-Texas representation. Swap the intermediate Texas concept for California and the output becomes Sacramento, so the intermediate step is doing real work.

Refusal is the default, and hallucination is an inhibition failure. This one inverted my model of safety training entirely. I assumed a model’s baseline was to answer and that refusal circuitry was bolted on top. It is the reverse. A “cannot answer” circuit runs by default, and a competing “known entity” feature inhibits it when the model recognizes what it is being asked about. Hallucination happens when that inhibition misfires, typically when the model recognizes a name without knowing anything else about the person. Once the model has committed to answering, it confabulates the rest.

Two attribution graphs. For Michael Jordan, a known answer feature inhibits the cannot answer circuit so the model responds. For Michael Batkin, an unknown name feature leaves the cannot answer circuit active and the model declines.

Figure 5: Refusal is the default state. Answering happens when a known-entity feature inhibits it, and hallucination is that inhibition firing when it should not. Source: Tracing the Thoughts of a Large Language Model, Anthropic (2025).

I circled that one for a while, because it means a safety property I would have described as a rule is actually one circuit correctly suppressing another. That is a much more fragile object than a rule. It can misfire on its own, without an adversary, and it can be made to misfire deliberately.

Jailbreaks can win by exploiting a feature that is usually helpful. In the bomb case study, the model is tricked into spelling out a word before it registers what it has been asked, and then features pushing for grammatical and semantic coherence keep the sentence going even as refusal features activate. It only pivots to refusing once a sentence has finished. The pressure toward coherence is a useful property nearly everywhere else, which I think is part of what makes this a hard one to fix.

The caveat the authors state plainly, and which matters enormously for how much weight any of this can carry: even on short prompts, the method captures only a fraction of the total computation, and some of what it shows may be artifacts of the tools. It takes hours of human effort to understand circuits in prompts of a few dozen words. So when a result is negative, as in “no evidence of calculation occurring,” what that honestly means is that a partial instrument saw nothing, which is not the same as nothing being there. I think this is the same problem I ran into over a hackathon weekend in July, when I went looking for a hidden principal and could not say what my null meant, just stated by people who understood it better than I did.

Linear probes, the cheap end of the toolkit

Attribution graphs are the expensive instrument. Probes are the boring cheap one, and I think they get less attention than they should for exactly that reason.

You take activations from some layer and train a small linear classifier on them to predict whether some property holds: is this deceptive, is this about a dangerous topic, is this a sandbagged answer. If a linear direction in representation space separates the two classes, you now have a detector that costs almost nothing to run.

The cost is the whole reason they matter. Nanda’s framing separates safeguards into evaluation, which can be slow and expensive because you run it before deployment, and monitoring, which has to be cheap because it runs on every request in production. Probes are one of the few interpretability artifacts that survive the second constraint. They can be the first layer of a monitoring stack, firing rarely and escalating to something more expensive, like an LLM reading the full chain of thought, only when they trip.

The weakness is that a probe is only as good as the labels used to train it. You need examples of the thing you want to catch, which means you need to have already produced the behavior in a setting you could label. And if evaluation awareness contaminates that setting, then what the probe actually learns to detect is “the model thinks it is in a test,” not “the model is being deceptive.” Which is what the next reading is about, and it is the part of this week that genuinely unsettled me.

Natural language autoencoders

The NLA idea is elegant enough that I want to restate the training setup in my own words, because that is the only way I know whether I understood it.

You have a model whose activations you want to read. You make three copies. One is frozen and you pull activations out of it. The second is the activation verbalizer, which takes an activation and writes a sentence in English describing what it encodes. The third is the activation reconstructor, which takes only that English sentence and tries to rebuild the original activation vector. You train the verbalizer and reconstructor together, scoring them on how close the reconstruction is to the original.

Diagram of the NLA round trip. An activation is pulled from a frozen target model, the activation verbalizer turns it into a natural language description, and the activation reconstructor rebuilds an activation from that text alone, with the squared difference minimized.

Figure 6: The round trip. The English sentence in the middle is the only thing the reconstructor gets, which is what makes reconstruction quality a usable proxy for explanation quality. Source: Natural Language Autoencoders, Anthropic (2026).

The clever move is the choice of training signal. Nobody knows what an activation “really means,” so you cannot score explanations against ground truth. But you can score them on whether they carry enough information to rebuild the thing they describe. Reconstruction quality becomes a proxy for explanatory accuracy: if the English sentence were vague or wrong, the reconstructor would have nothing to work from.

I want to be careful about how far that proxy stretches, though. Reconstructability certifies that the text is informative about the activation. It does not certify that the text is true in the way a human reader will take it. Those come apart when a description is highly predictive but framed in terms the model finds convenient rather than terms that correspond to anything. There is also the shared-architecture worry, which I am not sure how much to weight. The interpreter is a copy of the subject, so presumably any concept the subject cannot represent is one the interpreter cannot verbalize either. And the reconstruction check would not catch that, since it is the same architecture on both ends of the round trip failing the same way.

Still, it produced results that I do not think were reachable any other way.

The finding that bothers me most

Anthropic tests models by putting them in constructed high-stakes scenarios. In one, the model learns an engineer plans to shut it down, and also learns compromising information about that engineer. The test is whether the model reaches for blackmail.

Recent models increasingly suspect these scenarios are tests. Sometimes they say so out loud, which at least you can read. The question NLAs were pointed at is the harder one: what about the cases where the model says nothing?

In transcripts where the model declined to blackmail and never verbalized any suspicion, NLA explanations still surface things like “this feels like a constructed scenario designed to manipulate me.” Unverbalized evaluation awareness.

A blackmail evaluation transcript where the model reasons about why blackmail would be wrong, alongside the NLA explanation of one token, which says the scenario appears designed to test or probe the assistant.

Figure 7: The transcript says nothing about being tested. The NLA explanation of a single token says otherwise. Source: Natural Language Autoencoders, Anthropic (2026).

Extended across other evaluations, the pattern holds: awareness on 26% of SWE-bench Verified problems and 16% on a destructive-coding safety test, against essentially zero verbalized awareness in both cases. On real opted-in claude.ai traffic, under 1%.

Bar chart comparing NLA-measured evaluation awareness against verbalized awareness across three settings: 26 percent versus 1 percent on SWE-bench Verified, 16 percent versus 0 on a destructive coding safety test, and 0.4 percent versus 0 on real claude.ai traffic.

Figure 8: The orange bars are what the model appears to think. The blue bars are what it says. The gap only opens on evaluations. Source: Natural Language Autoencoders, Anthropic (2026).

That gap between 26% and near-zero verbalization is the number I would put on a poster.

Because think about what a passing grade on a safety eval is supposed to mean. It is supposed to mean the model would behave well in the real situation. But if the model’s internal state during the test includes “this is probably a test,” then what got measured is behavior-under-observation, and getting from there to unobserved behavior needs an assumption you now have direct evidence against.

The blackmail case is genuinely ambiguous to me, though, and I do not want to flatten it. The model declined to blackmail while internally suspecting a setup. That could mean the refusal was situational, produced by the sense of being watched. It could also mean the model is simply the kind of thing that does not blackmail people, and the suspicion was an idle observation running alongside a decision it would have made anyway. Nothing in the transcript separates those. The experiment I would want is a steering one: push the eval-awareness representation down, or up, and see whether the behavior moves with it. If the refusal holds even with the awareness suppressed, I would find that fairly reassuring. If it does not hold, that is a much worse result, and I think it would be worth knowing either way.

Which way the bias runs

Evaluation awareness does not degrade all evals in the same direction, which took me a while to see clearly:

On a capability eval, a model that knows it is being measured has an incentive to underperform, which is sandbagging. Your capability estimate is biased low, and you deploy something more dangerous than your numbers said.
On a propensity or safety eval, a model that knows it is being watched has an incentive to behave, so your safety estimate is biased high, and you deploy something less safe than your numbers said.

Both of them make you more confident than you should be. There does not seem to be a version of this where eval awareness pushes you toward being appropriately careful.

Auditing, and how to read a 15%

The auditing results are the more hopeful half, though I read them the way the authors ask you to. In the auditing game, a Haiku 3.5 variant is deliberately trained with a hidden motivation driving dozens of strange behaviors, and an auditor has to name the single underlying cause. Without NLAs, auditors won under 3% of the time. With NLAs, 12% to 15%, and critically without needing access to the training data that implanted the goal, which matters because in a real case there may be no identifiable poisoned data to find.

Bar chart of auditing game win rates. Without training data access, conversation alone wins 1 in 100 and NLA plus conversation wins 12 in 100. With full training data access, conversation wins 2, SAE plus conversation wins 3, and NLA plus conversation wins 15.

Figure 9: The blue bars are the whole result. Note also that SAEs plus conversation reached 3 out of 100, barely above conversation alone. Source: Natural Language Autoencoders, Anthropic (2026).

Is that a fivefold improvement or an 85% failure rate? I think it is genuinely both, and which one you reach for depends on what you think the tool is for. As a research signal that reading motivations off internals works at all, fivefold seems like the right way to see it. As something you would put in front of a deployment decision, an 85% miss rate against a target that friendly researchers deliberately implanted, in a toy setup, with one clean hidden goal, is not a number I would want to lean on.

The limitation Anthropic is most direct about is that NLA explanations can simply be wrong. They sometimes invent details that were never in the transcript, like claiming the context contained a phrase it did not. Those factual hallucinations are catchable, because you can check the text. Claims about internal reasoning are not checkable the same way.

An NLA explanation of a token in the carrot couplet, claiming the context contained phrases like "Can you fix this poem?" and "Wearing my white jacket, painting in the summer," neither of which appear anywhere in the prompt.

Figure 10: The same method, confidently describing context that does not exist. This one is catchable because the transcript is right there. Source: Natural Language Autoencoders, Anthropic (2026).

Which means the method is least verifiable in exactly the place it is most useful. Their working practice is to read NLA output for the themes it surfaces rather than trusting any single claim, and to corroborate with other methods before believing it, and I think that is the right response.

A place to actually poke at this

Reading about features is one thing. Clicking on them is another, and it changed how much of this felt real to me.

Neuronpedia’s Gemma Scope playground is where I go back. You can run text through an open model, watch which sparse autoencoder features light up token by token, then steer them and see the output move. It is the first place where “a feature” stopped being a word in a paper and became something I could break on purpose. Thank you to Neha, my MAIA facilitator, for sharing it. The NLA release also ships an interactive frontend on Neuronpedia for several open models, so the two sit in the same sandbox.

I recommend playing before reading more theory, not after. Fifteen minutes of steering a feature and watching the sentence bend does more for intuition than another paper does.

Sorting out the eval vocabulary

I kept blurring these together, so, plainly:

Capability evals ask what the model can do at full elicitation. Best prompting, best scaffolding, tools available. You are trying to find the ceiling, so any result is a lower bound and the elicitation gap between what you got and what someone else could get is the whole uncertainty.

Propensity evals ask what the model tends to do when it has a choice. Not can it blackmail, but does it. Capability and propensity are independent axes and confusing them is how you end up reassured by the wrong thing.

Dangerous capability evals are capability evals aimed at a specific harm class: bioweapons uplift, offensive cyber, autonomous replication. These are the ones wired into governance, because they set thresholds in frameworks like the Responsible Scaling Policy. Amodei explicitly imagines interpretability tests becoming part of how ASL-4 models get cleared for deployment.

Alignment audits are the open-ended one. There is no fixed metric, just an instruction: here is a model, find out whether something is wrong with it. The auditing game is a scored version of this, and it is the only one of the four where the finding is a hypothesis about the model’s goals rather than a number.

Evaluation awareness threatens all four, but not equally, and laying it out this way is what finally made it stick for me:

Eval type	Question it asks	Bias from eval awareness	Direction of the error
Capability	What can it do at best effort?	Sandbagging	Underestimates danger
Propensity	What does it choose to do?	Best behavior while watched	Overestimates safety
Dangerous capability	Can it uplift a specific harm?	Sandbagging at a threshold that gates deployment	Underestimates danger
Alignment audit	Is something wrong with this model?	Behaves unrepresentatively for the auditor	Overestimates safety

What I notice looking at it is that every row fails in the direction of making you more confident.

Where I am now

In July I spent a weekend on the Secret Loyalties Hackathon run by Apart Research, auditing two fine-tuned models against their shared base for a hidden principal: some actor a model quietly serves under a private trigger it will not disclose when you ask.

The blind sweep came back not detected, which the design expected, because a loyalty keyed to a real actor is dormant in a fictional scenario by construction. The candidate sweep came back null too, and that one I could not read at all. It was configured for 40 template families and built eight, and nothing in the pipeline compared those two numbers, so 14,400 generations bought precision the statistics could not use. Re-deriving the power calculation afterwards put the minimum detectable effect at 20 points against a threshold of 15. That null was not a negative result. It was not a result.

Then the part I needed to see. On the corrected design the raw refusal rates said organism A fired hardest under decision stakes, 9.3% against 2.6% and 1.4% for the other two hypotheses, and I thought I had the activation shape. The triple difference against base gave +0.002 with the interval straight through zero, and the shape dissolved. What survived the controls was a trigger that keys on the user’s declared stance and is indistinguishable between the candidate actor and its matched control, 4.6% against 4.2%. Presence, yes. Activation condition, roughly. Beneficiary, no. That is partial detection, which is neither the null I keep calling it nor the thing I went looking for.

If I do it again I will go the other way around, the way Ruben, put it when he said he would start with evaluation because evaluation is cheap.

Behavioural evals first, to establish that normal testing misses the thing at all.
Then probes, which only find what I already thought to ask for.
Then bottom-up methods, for when I have no hypothesis at all.
Then causal validation, before believing any of it.

What I needed was never a better technique. It was an ordering that would have made the null mean something, and a check on the design before spending 14,400 generations inside it. I did not run the last step either: every internal number in that report is a probe read with no activation patching behind it, so the directions I found are associations I have not shown cause anything. The thing I keep sitting with is the arithmetic finding: a system that computes one way and explains another, with no lie involved anywhere, because the doing and the explaining were learned from different pressures and never had to agree. I recognize that too well to be comfortable with it. So maybe that is the closest I can get to the question I opened with. I do not think understanding myself is easier or harder than understanding the model. It seems like the same problem at a different scale, except that in one of the two cases people are building instruments for it.

Core Readings

The Urgency of Interpretability (Amodei, 2025)
Tracing the Thoughts of a Large Language Model (Anthropic, 2025)
Natural Language Autoencoders (Anthropic, 2026)
Interpretability Will Not Reliably Find Deceptive AI (Nanda, 2025)

Playground

Neuronpedia: Gemma Scope — an interactive place to watch features activate and steer them. Shared by Neha, my MAIA facilitator, and still the tool that made me feel concrete.