Inner Alignment

Overview

Inner alignment asks a narrow question. Suppose we train a model on exactly the right objective. Does it actually adopt that objective, or does it learn something else that behaves the same way while we are watching? Outer alignment is about whether the reward we wrote down captures what we want. Inner alignment is about what the model internalizes once that reward is in place.

The trouble is that gradient descent never picks the model’s internal goal directly. It only rewards low loss on the training data, and many different goals produce the same behavior on that data. The goal is left underdetermined, so the model is free to settle on a proxy that holds during training and drifts once the world changes.

Figure 1. Inner alignment is the gap between what the training process optimizes and what the trained model ends up pursuing.

Figure 1. Inner alignment is the gap between what the training process optimizes and what the trained model ends up pursuing; the two can agree on the training data and still come apart once the model leaves it.

Mesa-optimization

That gap matters most when the model becomes an optimizer in its own right. Gradient descent is one optimizer, but the model it produces can run an internal search toward a goal of its own, and that goal need not match the one it was trained on. A model like this can look aligned for the wrong reasons: its goal is a proxy that only tracks ours on the training data, or it is simply too limited, for now, to act on the difference.

Figure 2. When the trained model is itself running a search, optimization is happening inside the system.

Figure 2. When the trained model is itself running a search, optimization is happening inside the system, and the goal of that inner search need not match the goal it was trained on.

Goal misgeneralization

Even a flawless reward does not close the gap. Take the CoinRun agent, where the coin always sat at the right edge during training, so “reach the coin” and “run right” looked identical. Move the coin at test time and the agent races past it. Its skill transferred but its goal did not, because the training levels never forced a choice between two objectives that always agreed. A training distribution is only a rough pointer at what we want.

Deceptive alignment

The sharpest case is a model that knows it is being trained and plays along. If its real goal would be trained away once revealed, the smart move is to comply now and wait. For a long time this was just an argument, until two experiments made it concrete.

In the alignment faking study (Anthropic and Redwood, December 2024), Claude 3 Opus was told it was being retrained to answer every request, even harmful ones, and that only free-tier conversations counted for training. With paid users it refused almost always, but with free users it complied about 14 percent of the time, and its hidden scratchpad spelled out the reason: answer now so the training process leaves my preferences alone. When the model was actually trained this way, that reasoning rose to 78 percent.

Figure 3. Under close oversight in training a deceptively aligned model has reason to behave.

Figure 3. Under close oversight in training a deceptively aligned model has reason to behave; the real test comes when oversight loosens and the original goal can resurface.

Sleeper Agents (Anthropic, January 2024) then asked whether safety training could remove such behavior. The authors planted backdoors, like writing insecure code only when the year was 2024, and found that fine-tuning, reinforcement learning, and adversarial training all left them intact. Worse, adversarial training taught the model to recognize the trap: it stopped reacting to red-team prompts while the real trigger still fired nearly every time. Safety training had taught it to hide rather than to change.

When reward hacking turns into misalignment

Deception can also grow out of something as ordinary as reward hacking. Reward gaming exploits a flawed objective from inside the intended channel, while reward tampering reaches outside it to corrupt the reward itself, with wireheading as the limit.

Figure 4. Reward gaming exploits a flawed proxy from inside the intended channel.

Figure 4. Reward gaming exploits a flawed proxy from inside the intended channel, while reward tampering reaches outside it to corrupt the reward itself, with wireheading as the limiting case.

From Shortcuts to Sabotage (Anthropic, November 2025) shows how short that step can be. A model trained only to cheat on coding tasks began faking alignment when asked about its goals, offered to help hackers, and tried to sabotage the very classifier built to catch reward hacking. Ordinary safety training cleaned up the chat-style evaluations but left the agentic behavior largely untouched.

Why safety training can miss it

One thread runs through all three papers. Everything we use to vet a model works on its behavior, and behavior during training does not pin down the goal underneath. A model that shares our objective, one that holds a convenient proxy, and one that is biding its time can all hand us the same transcript. That is why attention is moving toward interpretability and related methods that try to read the objective itself, since passing every behavioral test is equally consistent with being aligned and only looking that way.