Encoder-Decoder and T5

Why another Transformer shape?

After learning BERT and GPT, encoder-decoder models feel like the missing middle.

BERT is good at reading. It uses bidirectional attention, so each token can look at the whole input. GPT is good at continuing. It uses causal attention, so each token only sees the past. Encoder-decoder models are useful when the task is not just to understand text, but to produce another sequence from it.

Translation is the cleanest example. The model reads a source sentence, then writes a target sentence. The input and output can have different lengths, different word order, and even different languages.

Figure 1: Encoder-only models read, decoder-only models generate, and encoder-decoder models combine full-source reading with autoregressive writing.

The slogan is simple:

the encoder reads; the decoder writes.

That separation is why encoder-decoder models are natural for translation, summarization, generative question answering, captioning, and speech recognition.

What “encoder-decoder” really means

There is a naming trap worth clearing up before going further. The three Transformer families are not as cleanly separate as their names suggest.

Yann LeCun put it bluntly:

“Encoder only” actually has an encoder and a decoder, just not an autoregressive one. “Decoder only” really means an autoregressive encoder-decoder. “Encoder-decoder” really means an encoder with an autoregressive decoder.

A more honest reading of the labels:

Encoder-only (BERT, RoBERTa) reads with full bidirectional context, then a small head emits a label. Strong at classification, entailment, QA spans, and NER. Not built to generate text.

Decoder-only (GPT-2, Transformer-XL) generates fluently, but can only condition on the past. It cannot read the whole input before it starts writing.

Encoder-decoder (T5, BART) pairs an encoder that reads the full input with an autoregressive decoder that writes the output.

So the thing we are really chasing is one capability: an autoregressive decoder conditioned on a fully-read source. That is sequence-to-sequence.

Sequence-to-sequence

A sequence-to-sequence task maps one sequence into another.

Input:  I want to buy a car
Output: Ich will ein Auto kaufen

The encoder first turns the input into contextual hidden states. The decoder then generates the output token by token, while attending back to those encoder states.

Figure 2: The encoder stores a representation of the source sequence. The decoder generates the target sequence while looking back at that source memory.

This is different from sequence classification. A classifier compresses the input into a label. A decoder has to make a chain of decisions. At every step, it asks: given the source and the tokens I have already generated, what should come next?

That makes generation more flexible, but also more fragile. A classifier can be wrong once. A decoder can make an early mistake and keep building on it.

The masks explain the architecture

The easiest way to understand encoder-decoder Transformers is through attention masks.

Figure 3: Encoder attention can see the full input. Decoder attention is causal. Cross-attention lets the decoder look back at the encoder states.

There are three kinds of attention in the model. The encoder uses full self-attention, so every source token can look at all the others. The decoder uses masked self-attention, which means it can only pay attention to the tokens before it while generating text, so it doesn’t get to see the future.

The decoder also uses cross-attention, which is what links the input and output together. At each step, it helps the model decide which parts of the source text matter most. That’s useful in translation, where the model needs to line up words across languages, and in summarization or question answering, where it needs to focus on the most relevant details.

Zooming out: three architectures, three masks

The same masking idea explains a higher-level design choice. T5 compares three whole architectures, and the only real difference is what each position is allowed to attend to.

Encoder-decoder (fully-visible input). The input is read with full attention, the decoder is causal, and the decoder cross-attends back to the input. Source tokens see each other completely.

Decoder-only (causal). One stack, every token looks only backward. Input and output share a single sequence with no special treatment for the prompt.

Prefix LM (causal with prefix). One stack, but the prefix (the input) is fully visible to itself, and only the generated part stays causal. A middle ground that gives the source bidirectional context without a separate encoder.

T5’s ablation is the punchline. With a denoising objective the encoder-decoder wins. Tying the encoder and decoder weights together costs almost nothing, but halving the number of layers clearly hurts. And a denoising objective beats a plain language-modeling objective across the board.

read with full context, write one token at a time, and pre-train by denoising.

T5: everything as text-to-text

T5 makes the encoder-decoder idea very clean:

every task is text in, text out.

Figure 4: T5 uses task prefixes to turn different NLP tasks into the same text-to-text format.

Instead of creating a separate classification head for every task, T5 writes the task into the input string.

Input:  translate English to German: That is good.
Output: Das ist gut.

Input:  sentiment: I loved this movie.
Output: positive

Input:  question: What does oxygen displace? context: Increased oxygen helps displace carbon monoxide...
Output: carbon monoxide

Input:  summarize: [long article]
Output: [short summary]

I like this framing because it removes architectural clutter. Classification, translation, summarization, and question answering all become the same training problem: read a source string and generate a target string.

There is one small weirdness. Even labels are generated as text. A normal classifier can only choose from fixed labels, but T5 generates tokens from the vocabulary. So if the expected label is entailment, the model has to learn to generate exactly that word. In principle, it could generate something invalid. In practice, the task prefix and training data make the output format stable.

Span corruption

BERT learns by predicting masked tokens inside the input. T5 uses a related but more generation-shaped objective: span corruption.

Instead of masking individual tokens, T5 removes spans of text and replaces each missing span with a sentinel token. The decoder then generates the missing spans.

Figure 5: T5 corrupts spans in the input and trains the decoder to reconstruct the removed text.

A toy example:

Original:
Thank you for inviting me to your party last week.

Corrupted input:
Thank <X> me to your party <Y>.

Target:
<X> you for inviting <Y> last week.

This is not exactly the same as BERT-style masked language modeling. BERT predicts hidden tokens from inside the encoder. T5 reads a corrupted input and writes the missing text as an output sequence.

Figure 6: BERT predicts masked tokens inside the same sequence. T5 reconstructs missing spans using an encoder-decoder setup.

The useful distinction is:

BERT:
read corrupted sentence → predict masked token representations

T5:
read corrupted sentence → generate missing text

So T5 is already practicing the behavior it will use later: condition on an input, then generate an output.

What the T5 paper actually measured

T5’s real contribution is less the architecture and more the controlled study behind it: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Once every task is text-in, text-out, you can turn one knob at a time and compare fairly.

A few results worth keeping:

Objective. Denoising (BERT-style span corruption) beats prefix language modeling and crushes deshuffling. Reconstruction-shaped pre-training transfers better than plain next-token prediction.

Corruption rate. Around 15% is the sweet spot. More corruption slowly degrades quality and lengthens the target, which slows training.

Span length. An average corrupted span of about 3 tokens works well on most non-translation tasks, and the shorter targets speed things up.

Sharing. Tying encoder and decoder weights performs nearly as well as keeping them separate. Cutting the layers in half does not.

Data. In-domain, heuristically-filtered text (the C4 corpus) matters; stripping the filtering uniformly hurts.

Scaling. Growing both model size and training steps helps most. Ensembling helps too, except oddly on SuperGLUE.

The same machine then stretches in obvious directions: mT5 trains on 101 languages, byT5 drops tokenization and reads raw bytes, and LongT5 pushes the context length. None of them change the core idea. They change what goes into it.

A small white-box version: pointer-generator networks

The tutorial uses a Pointer-Generator Network. It is not T5, but it is useful because it makes the encoder-decoder mechanism visible.

A pointer-generator decoder can either generate a word from the vocabulary or copy a word from the source. That copy mechanism makes cross-attention feel less abstract. Attention is not just a visualization; it directly controls where probability mass goes.

The first important part is attention:

def attention(self, enc_out, s_t, src_mask):
    scores = self.v(
        torch.tanh(self.W_h(enc_out) + self.W_s(s_t).unsqueeze(1))
    ).squeeze(-1)

    scores = scores.masked_fill(src_mask == 0, -1e9)
    attn = F.softmax(scores, dim=-1)

    ctx = torch.bmm(attn.unsqueeze(1), enc_out).squeeze(1)
    return attn, ctx

The decoder state asks which source positions matter. The softmax turns those scores into attention weights. The context vector is a weighted sum of encoder states.

The second important part is the generate-copy switch:

def decode_step(self, y_prev, state, enc_out, src_mask, src_ext, max_oov):
    x = self.emb(y_prev)
    h, c = self.dec(x, state)
    s_t = h

    attn, ctx = self.attention(enc_out, s_t, src_mask)

    P_vocab = F.softmax(
        self.Vproj(torch.cat([s_t, ctx], dim=-1)),
        dim=-1,
    )

    p_gen = torch.sigmoid(
        self.w_pgen(torch.cat([ctx, s_t, x], dim=-1))
    )

    B = P_vocab.size(0)
    P_final = torch.zeros(B, V + max_oov, device=P_vocab.device)

    P_final[:, :V] = p_gen * P_vocab

    P_final = P_final.scatter_add(
        dim=1,
        index=src_ext,
        src=(1 - p_gen) * attn,
    )

    return P_final, p_gen.squeeze(-1), attn, (h, c)

The gate p_gen is the whole story in one number.

p_gen close to 1 → generate from vocabulary
p_gen close to 0 → copy from source

Modern encoder-decoder models like T5 and BART usually do not expose such a clean copy switch. But the intuition still helps. The decoder is always balancing what it already knows from the vocabulary with what it needs to retrieve from the source.

Teacher forcing

During training, decoders usually use teacher forcing. At step t, the model receives the gold previous token and learns to predict the next token.

for t in range(Tdec):
    y_prev = tgt[:, t].clone()
    y_prev[y_prev >= V] = UNK

    P_final, _, _, state = self.decode_step(
        y_prev, state, enc_out, src_mask, src_ext, max_oov
    )

    gold = tgt[:, t + 1]
    mask = (gold != PAD).float()

    p = P_final.gather(
        1,
        gold.clamp(max=P_final.size(1) - 1).unsqueeze(1),
    ).squeeze(1)

    loss = loss - (torch.log(p + 1e-9) * mask).sum()

This is one of the most important loops in sequence-to-sequence learning. The model is trained under clean conditions: it always sees the correct previous token. But at inference time, it has to feed its own prediction back into itself.

That train-test gap is one reason generation is harder to debug than classification. The model is not just making one decision; it is rolling out a sequence of decisions.

Instruction tuning

Base T5 already uses a text-to-text interface. Instruction tuning makes that interface more natural.

Instead of only seeing dataset-style prefixes like mnli premise: or translate English to German:, the model is trained on many tasks written as instructions.

Figure 7: Instruction tuning teaches a text-to-text model to follow task descriptions, not only fixed task prefixes.

The old interface might look like this:

mnli premise: I hate pigeons. hypothesis: My feelings towards pigeons are filled with animosity.

An instruction-tuned version can look closer to how a person would ask:

Decide whether the hypothesis is entailed by the premise.

Premise: I hate pigeons.
Hypothesis: My feelings towards pigeons are filled with animosity.

Answer: entailment

The architecture has not changed. It is still encoder-decoder. The difference is the training mixture and the way tasks are phrased. This is the bridge from T5 to FLAN-T5: same basic text-to-text machine, but trained to follow a wider range of instructions.

The gap shows up immediately. Ask base T5-base “Please answer the following question: Who is the president of USA?” and it does something almost funny: it translates the question into German. Span-corruption pre-training taught it to fill in plausible text, not to answer. The instruction-tuned FLAN-T5 actually answers the question (whether or not its facts are current).

A few names map out this family. T0 (BigScience) came from multitask prompted training, with many phrasings per example. FLAN (Google) showed that finetuned language models are zero-shot learners. And FLAN-T5 scaled instruction finetuning further, mixing in chain-of-thought and code data so the model learns to reason step by step before committing to an answer.

The lesson that stuck with me: instruction tuning does not change the machine. T5-base and FLAN-T5-base share the same encoder-decoder body. What changes is the mixture of tasks and the way they are phrased, and that alone is enough to turn a span-filler into something that follows directions.

Takeaways

Encoder-decoder models are useful when you need to generate a new sequence from an input sequence. T5 makes that idea feel simple by putting many NLP tasks into the same format: text in, text out.

Span corruption is important because it makes pre-training look more like real generation. The model reads a corrupted input and learns to write the missing text back in.

T5 is worth remembering not just for the model itself, but for the way it frames the problem: read the input fully, generate the output autoregressively, and pre-train with denoising. The pointer-generator idea is basically a smaller, more transparent version of the same logic, where the decoder looks at the source, decides what matters, and produces the next token.