How Transformers Actually Work
How Transformers Actually Work
Every chatbot you use — ChatGPT, Gemini, Claude — runs on one idea from a 2017 Google paper. It's called a transformer, and it's simpler than the diagrams make it look.
Read this sentence: "The animal didn't cross the street because it was too tired."
You know instantly that "it" means the animal, not the street. You didn't think about it. You just used the rest of the sentence to figure out one word.
That — a word borrowing meaning from the words around it — is the whole trick behind the transformer. Everything else is engineering around that one idea. It was introduced in a 2017 paper by eight Google researchers called Attention Is All You Need, and it quietly became the foundation of modern AI.
The old way: reading one word at a time
Before transformers, language models read like you do: one word at a time, left to right. These were called recurrent neural networks, or RNNs.
That caused two problems. First, it was slow — you can't process word five until you've finished word four, so a GPU's thousands of cores mostly sat idle. Second, memory faded. By the time an RNN reached the end of a long sentence, what it had learned at the start had been squeezed through dozens of steps and mostly washed out.
The transformer team's question was blunt: what if we throw away the one-word-at-a-time reading entirely, and let every word look at every other word directly?
Attention, in plain English
Here's how it works, using our sentence. First, every word gets turned into a list of numbers — a vector — that captures something about its meaning. Think of it as coordinates: words with similar meanings sit near each other.
Then, for each word, the model creates three smaller vectors: a query, a key, and a value. The names come from search. The query is "what am I looking for?" The key is "what do I contain?" The value is "what do I actually offer if you pick me?"
To figure out how much the word "it" should care about each other word, the model takes the dot product — basically a similarity score — of "it's" query with every word's key. High score means "these two are related."
Those scores get scaled down and passed through a softmax function, which turns them into weights that add up to 1 — like percentages. Then each word's value vector gets multiplied by its weight, and everything gets added together.
So when the model encodes "it", the result is mostly "it" itself, plus a strong helping of "animal" and a dash of "tired". As Jay Alammar puts it in The Illustrated Transformer, self-attention is how the model bakes "the understanding of other relevant words into the one we're currently processing."
The payoff is huge: because every word can reach every other word in a single step, the whole sentence can be processed in parallel. That's why transformers train so much faster than RNNs — and why they could be scaled up to the giant models we have today.
Why eight heads instead of one
One round of attention isn't enough, because words relate to each other in more than one way. In our sentence, "it" connects to "animal" (that's who "it" is) and also to "tired" (that's why it didn't cross).
So the transformer runs attention several times in parallel, with different learned weights each time. These are called attention heads. The original paper used eight of them per layer. One head might track who-did-what, another might track cause and effect, and so on.
The eight results get glued back together into a single vector, and the whole thing passes through a simple feed-forward network — the same network applied to each word separately. Stack that block a few times, and you have an encoder: a machine that turns a sentence into rich, context-aware representations of each word.
One problem: transformers can't see word order
There's a catch. If every word looks at every other word at once, the model has no idea which word came first. "Dog bites man" and "Man bites dog" would look identical to it.
The fix is positional encoding: a small pattern of numbers, added to each word's vector, that marks its position in the sentence. The original paper used a sine-and-cosine pattern, chosen because it lets the model judge distances between words — and even handle sentences longer than any it saw in training.
It's a small detail, but without it the whole architecture falls apart. Order is meaning.
From reading to writing
So far we've covered the encoder — the half that understands. The original transformer also had a decoder, the half that writes.
The decoder works almost the same way, with two twists. First, its self-attention is masked: when it's writing word five, it can only look at words one through four, never ahead. Otherwise it would be cheating by peeking at the answer. Second, it has an extra attention layer that looks back at the encoder's output, so the translation stays faithful to the original sentence.
At the end, a final layer turns the decoder's output into scores for every word in the vocabulary, and softmax turns those scores into probabilities. The word with the highest probability wins, gets fed back in as input, and the cycle repeats — one word at a time — until the model produces an end-of-sentence symbol.
That last part surprises people: transformers read everything at once, but they still write one word at a time. Every chatbot you've used is doing exactly this, word by word, in real time.
What trips people up
First, attention is not understanding. An attention map showing "it" linking to "animal" looks like comprehension, but the model is really just following statistical patterns it learned from mountains of text. Impressive patterns — but patterns, not thoughts.
Second, attention is expensive. Every word comparing itself to every other word means the cost grows with the square of the length: double the input, quadruple the work. That's a big reason long-context models are so costly to run, and why researchers keep inventing cheaper attention variants.
Third, the transformer is not one model — it's an architecture, a blueprint. ChatGPT, Gemini, Claude and the open-weight models on your laptop are all different buildings on the same foundation. When someone says "transformer", they mean the design, not any particular chatbot.
One thing to remember: attention lets each word borrow meaning from the words around it, in a single parallel step. That's it. Everything else — the heads, the positional codes, the decoder — exists to make that one idea work at scale. If you get that, the diagrams stop looking scary.
Want to go deeper? Read The Illustrated Transformer with its diagrams, then try Harvard's Annotated Transformer, which walks through a working implementation line by line. That's the fastest path from "I sort of get it" to "I could build a small one."
Questions people actually ask
How big was the original transformer?
Small by today's standards: a stack of six encoders and six decoders, with 512-dimensional word vectors and eight attention heads per layer. On English-to-French translation it scored 41.8 BLEU — a new state of the art at the time — after training for just 3.5 days on eight GPUs, according to the paper.
Why is it called a transformer?
Because it transforms one sequence into another: an English sentence in, a French sentence out. The name describes the job, not the mechanism — the mechanism is attention.
Can I build a tiny transformer myself?
Yes. Harvard's Annotated Transformer is the classic starting point: it rebuilds the paper's model in PyTorch with commentary, and it runs on a laptop. You won't match ChatGPT, but you'll understand every piece of the machine.
