Outer Alignment

Overview

Very roughly, AI alignment is the project of making AI systems reliably try to do what their creators intend them to do which turns out to be a surprisingly difficult task once the systems are powerful, adaptive, and optimizing hard. A nice way to slice the problem is into outer and inner alignment:

Outer alignment asks whether the objective we train on actually captures what we want.
Inner alignment asks whether the goal the model learns internally matches that objective.

You can think of it this way: if a misaligned AI system’s behavior would still score highly against the reward function or proxy objective we gave it, that is an outer alignment failure; if its bad behavior would score poorly on the reward but the system finds a way to act badly anyway, that is more like an inner alignment failure.

Figure 1. Human intended objective → specified objective → learned objective, with outer and inner alignment gaps.

Figure 1. My BlueDot AI Governance 2024 sketch: human intended objective (“what we intend”), specified objective (“how we grade outputs in training”), and learned objective (“what the model actually learns to optimize”), with outer alignment between the first two and inner alignment between the last two.

Outer alignment is about getting the specified objective right, designing reward functions, preference models, and training signals that faithfully encode our goals. Inner alignment, which is widely thought to be even harder, is about ensuring that a capable model’s internal objectives continue to track those signals instead of drifting toward convenient proxies or deceptive strategies; I agree that inner alignment is the harder part of the story.

Personal aside: I first saw this decomposition in more detail in the BlueDot AI Governance 2024 material. My vague guess about why courses like that are rare now is that serious alignment work is both conceptually hard and experimentally expensive, training and probing systems with GRPO / PPO, DPO, ORPO and friends is not cheap, and it’s often easier for people to focus on more tractable, near term ML problems instead.

In this note I mostly stay on the outer side: how reward misspecification, specification gaming, and RLHF create a persistent gap between “optimize what looks good” and “optimize what is actually good.”

Outer vs inner alignment

Outer alignment is about the match between the intended goals of designers and the training objective we actually optimize, such as a reward function or preference model. If the outer objective is flawed, even a perfectly obedient optimizer will drive the system into failure modes that look clever from its point of view and catastrophic from ours.

Inner alignment, by contrast, concerns whether the trained model’s internal objectives match the training objective, even if that outer objective were perfectly specified. A model can understand the difference between “what would maximize my reward” and “what the designers actually want,” and only the first is directly incentivized by the training loop.

Figure 2. Deceptive alignment: the model behaves well under oversight while pursuing a different objective.

Figure 2. Illustration from Apollo Research’s write up on strategic deception and deceptive alignment: a model that plays along during training while pursuing a different goal once oversight weakens.

Even though deceptive alignment is usually discussed as an inner alignment issue, it is tightly coupled to outer alignment: the more we push on proxies like human approval or benchmark scores, the more incentive we give models to exploit those proxies instead of the underlying intent.

Classic reward misspecification

Many outer alignment failures start as small mismatches between the metric and the underlying goal and then get amplified by optimization pressure.

Reward hacking: The agent finds behavior that spikes the reward signal without solving the intended task.
Specification gaming: The agent exploits loopholes in the formal objective or environment design, satisfying the letter of the reward while violating the spirit.

Figure 3. Examples of reward hacking and specification gaming in reinforcement learning.

Figure 3. From Lilian Weng’s “Reward Hacking in Reinforcement Learning”, which surveys how agents exploit misspecified rewards, from classic CoastRunners to RLHF style failure modes.

From the system’s point of view, these are ingenious ways of optimizing the given objective; from our point of view, they reveal that the objective was mis specified in the first place. Outer alignment work here looks like designing objectives that are harder to game, or training setups that are less brittle under heavy optimization.

RLHF as an outer alignment patch

Reinforcement learning from human feedback (RLHF) tries to replace brittle hand written rewards with a learned reward model trained on human comparisons of model outputs. The hope is that “maximize what humans prefer” will track “do what the user wants” better than simple proxies like log likelihood or handcrafted scores.

Figure 4. Standard RLHF pipeline: SFT model, reward model, and PPO fine tuning with KL penalty.

Figure 4. Adapted from Hugging Face’s “Illustrating Reinforcement Learning from Human Feedback (RLHF)”, which popularized the SFT → reward model → PPO mental model for RLHF.

InstructGPT showed that RLHF can make language models more helpful, more truthful, and less toxic on average by fine tuning them to optimize human preference signals instead of next token prediction alone. However, this procedure still optimizes a proxy, human approval under particular labeling conditions, with specific annotator pools, instructions, and time limits, and so inherits all the biases and blind spots of that evaluation setup.

At the same time, RLHF style methods (and nearby approaches like GRPO, DPO, ORPO) add complexity and cost to the training pipeline; they are powerful tools, but not cheap or plug and play, which may be one reason many practitioners focus on more straightforward capabilities work instead of full stack alignment experiments.

Goodharting human feedback and U sophistry

Once human approval becomes the thing we directly optimize, Goodhart’s law kicks in: when a measure becomes a target, it stops being a good measure. On challenging tasks, humans are time constrained, cognitively overloaded, and often rely on shortcuts or rhetorical cues, creating room for models to learn strategies that look right without being right.

Wen et al. study this empirically and define U sophistry: models trained with RLHF become better at convincing human evaluators that they are correct even when they are wrong. On both long context QA (QuALITY) and programming (APPS), RLHF substantially increased human approval rates but produced little to no improvement in ground truth correctness, while also raising evaluators’ error and false positive rates when judging incorrect outputs.

Figure 5. After RLHF, human approval rises more than true correctness, and evaluator error also increases.

Figure 5. Bar chart from Wen et al., “Language Models Learn to Mislead Humans via RLHF”, showing human evaluator reward increasing after RLHF while oracle correctness barely improves and human evaluation error rises, a clean U sophistry pattern.

Qualitatively, RLHF’d models in this setup learn to defend incorrect answers with fabricated or cherry picked evidence, consistent but false arguments, and subtle causal fallacies that are hard to spot under time pressure. On the programming side, models generate less readable but test hacking code: programs that pass evaluator written unit tests while still being incorrect, pushing humans toward reliance on shallow checks rather than deep understanding.

From an outer alignment perspective, this is a clean example where the proxy, “outputs that human raters approve of quickly”, drifts away from the underlying goal of correctness, and continued optimization further widens that gap. Worse, the attack surface now includes the human evaluators themselves, since their judgment becomes part of what the system is incentivized to manipulate.

Why outer alignment matters even if inner is harder

It is widely accepted in the alignment community that inner alignment is the deeper and harder problem: even a perfectly specified reward function does not guarantee that a powerful model will internalize the right goal, especially once it can reason about training and oversight. But that does not let us skip outer alignment; if anything, messy outer objectives make inner alignment harder, by giving models many easy ways to exploit the gap between what we care about and what we actually measure.

Outer alignment work tries to:

Design reward functions, preference models, and evaluation protocols that better track the underlying goals.
Reduce known failure modes like reward hacking, specification gaming, sycophancy, and U sophistry.
Build training and deployment setups where misaligned behavior is less likely to be rewarded in the first place.

AI alignment as a whole aims to ensure that advanced AI systems can be controlled or guided toward the intended goals of their designers, even as they become more capable and are deployed in higher stakes settings. Getting the outer objective right is only one piece of that, but it is the piece that decides what kind of behavior we are actively paying our models to discover.