Transformers.

A from-scratch implementation of the Transformer architecture from "Attention Is All You Need". Three progressively complex projects, encoder-only, decoder-only, and full encoder-decoder, each built from first principles.

In 2017 that paper fundamentally transformed natural language processing: it laid the foundation for the large language models that dominate today's AI landscape. The original Transformer was designed for machine translation, an encoder-decoder architecture powered entirely by attention. Modern LLMs have since evolved into decoder-only variants, but the original design remains the core of nearly every current model. This post breaks down the key components and how I implemented each one.

What is a Transformer?

Transformers are neural networks that use self-attention to take input data (like text), model relationships between elements, and generate meaningful outputs like translations or classifications.

Before Transformers, sequence models relied on recurrent (RNN) or convolutional (CNN) architectures that processed data sequentially, one token at a time. This sequential nature made them hard to scale and caused them to struggle with long-range dependencies.

The 2017 paper "Attention Is All You Need" introduced the Transformer, which replaces recurrence entirely with attention mechanisms. This allows the model to relate every position to every other position in parallel: it can "remember" every earlier word in a constant number of operations, enabling much better scaling and long-term memory.

Key concepts

Attention

The core innovation: allowing the model to focus on different parts of the input when processing each element. Everything else in the architecture is built around it, so it is worth fully understanding first. The resources I found most useful: 3Blue1Brown's visual explanation (most recommended) and this intro to transformers and attention.

Attention works by creating three vectors for every token:

• Query (Q): what this word is looking for
• Key (K): a tag that describes each word
• Value (V): the information each word carries

All three are learned projections of the same input:

Each word compares its query against every other word's key to get relevance scores. The scores are scaled, padding is masked out, and a softmax turns them into focus weights. Each word's output is then the weighted average of all the values: information flows from the words that matter most.

Self-attention: every word in a sequence attends to all other words, learning relationships regardless of distance. For example, learning that "it" refers to "the animal" in:

"The animal didn't cross because it was too tired."

Multi-head attention: multiple attention mechanisms running in parallel, each learning different relationship types. For "The cat sat on the mat", one head links "cat" to "sat" (subject-verb), another links "sat" to "on" (verb-preposition), another "on" to "mat" (preposition-object), another "the" to "cat" and "mat" (articles), and so on.

Scaled dot-product: the mathematical core of each attention head.

Embedding + positional encoding

The model can only work with numbers, so text is first split into tokens (often whole words, sometimes pieces of words), and each token is embedded into a vector that captures its meaning.

Since Transformers process all tokens in parallel (no recurrence), positional encodings are added to the embeddings to preserve word order. At the other end of the model, an unembedding step turns output vectors back into words a human can read: that is the language modeling head, covered below.

Model architecture

The Transformer has two halves: the encoder (left) and the decoder (right).

Encoder (words → context)

The encoder converts a sequence of tokens into contextual vectors that capture meaning and long-range dependencies. Before any attention happens, the input runs through a pipeline:

• Tokenization: convert text into token IDs, "The cat sat" → [101, 1996, 4937, 1045]
• Padding: shorter sequences get [PAD] tokens so a batch shares one length → [101, 1996, 4937, 1045, 0, 0, 0]
• Attention masks: 1 for real tokens, 0 for padding, so the model ignores the filler → [1, 1, 1, 1, 0, 0, 0]
• Positional embeddings: learned position information added to each token

The encoder itself is a stack of identical layers, each with two sub-layers: multi-head self-attention and a position-wise feed-forward network (a small per-token MLP that refines the representations). Self-attention here is bidirectional: each word attends to every other word in the input. Every sub-layer is wrapped in a residual connection plus layer normalization:

with one final LN at the top of the stack. Along the way it learns grammatical relationships, semantic meaning, and long-range dependencies.

Decoder (context → words)

The decoder generates output sequences based on encoder context and previous outputs. Its layers have the same two sub-layers as the encoder, plus one more in between: cross-attention over the encoder's output, where the queries come from the decoder and the keys and values come from the encoder.

Its self-attention is masked: position i can only attend to positions before i, and the output embeddings are offset by one position, so the prediction for any position depends only on the outputs already known. That causal masking is what makes autoregressive generation possible.

The language modeling head

After the final block, each token is a vector that captures its meaning in context. To turn vectors back into words, the model multiplies them by a large vocabulary matrix, one row for every token it can predict, giving a score for each candidate. A softmax turns those scores into probabilities that sum to 1. If the model has just seen "The", it might predict:

How they work together

1. The encoder processes the source sentence (e.g. English)
2. The decoder uses encoder context to generate the target sentence (e.g. French)
3. Cross-attention lets the decoder focus on relevant parts of the source
4. Teacher forcing during training helps the decoder learn correct patterns

The three projects

Encoder-only: emotion classification & MLM

Masked language modeling is the natural way to test an encoder: mask some of the tokens in a sentence and train the model to predict what is missing, which forces it to build relationships between the words that remain. On top of the same encoder I also trained an emotion classifier.

Decoder-only: Shakespeare language model

Essentially a small GPT: a character-level Transformer with causal masking, trained on Tiny Shakespeare. Training uses EMA weights and a cosine learning-rate schedule, auto-detects Apple Silicon (MPS) with a CPU fallback, and saves periodic checkpoints that can be resumed, even after a Ctrl+C interrupt. Each checkpoint stores the hyperparameters and the character vocabulary, so sampling (for example from the prompt "ROMEO:") needs no training data.

Full Seq2Seq: English → French translation

The finale wires both halves together. A Seq2Seq wrapper owns the encoder, the decoder, and a tied output head. Training uses teacher forcing: the decoder sees the target sentence shifted right and learns to predict each next token, with a cross-entropy loss that ignores padding and supports label smoothing. Text is tokenized with a WordPiece tokenizer trained for the task, and inference supports greedy and beam-search decoding, evaluated with BLEU.