Type anything
No button to click — attention runs on every keystroke, live, on this page's own default example or on whatever you type instead.
Here's the whole journey your sentence just took, start to finish — this whole left-to-right trip through the model, sentence in, prediction out, is what's called a forward pass. The rest of this page walks through each arrow, one at a time:
Turning words into numbers
Three things have to happen to your sentence before any model can do anything with it. Here's each one, in order — the idea first, then the actual thing happening to your words.
Step A: break the sentence into tokens
A model can't read letters. So the first move is chopping your sentence
into tokens — usually one per word. A long word occasionally gets cut into two tokens
instead of one (you'll see the second piece marked with a leading ##) — real
models do exactly this, just with a much bigger rulebook than this toy version uses.
Step B: turn each token into a fingerprint of numbers
A word means nothing to a math equation, so every token gets converted into an embedding — a fingerprint made of 16 numbers instead of letters. Below, each token's fingerprint is drawn as a strip of 16 colored cells: green = a positive number, red = negative, brighter = bigger. Click any token to see what tokenization did to it and its actual numbers.
Step C: but a fingerprint alone can't say where a word sits
Here's the problem: "the dog chased the cat" and "the cat chased the dog" would give "dog" and "cat" the exact same fingerprints either way — a fingerprint only captures which word it is, never where it falls in the sentence, even though that's often the entire difference in meaning. The fix is to add a second, position-based set of numbers directly on top of each fingerprint. Try it:
Flip it and watch the strips above actually change — that's a real position vector landing on top of the fingerprints you just looked at. ("Vector" is just the math name for an ordered list of numbers — every fingerprint you've already seen is one, 16 numbers long. You'll see the word a lot from here on; it never means anything fancier than that.)
The step almost every explainer skips
Right now, every word has exactly one fingerprint — but figuring out which words matter to each other means each word actually needs to play three different roles at once: something that goes looking for other words, something that other words can find it by, and something it actually hands over once it's been found. One fingerprint can't do all three jobs well at the same time, so instead of comparing the plain fingerprints directly, each one gets reshaped into three purpose-built copies.
A library makes the roles concrete: your Query is what you type into the search bar, every book's Key is its catalog entry (the thing your search actually gets matched against), and its Value is what's actually printed on the pages once you pick it up. Three different jobs, three different vectors — none of them would work as well doing double duty as one of the others.
The reshaping itself is done by multiplying each embedding by a matrix — here, that just means a fixed grid of numbers that acts like a recipe: feed in the embedding's 16 numbers, and it blends them together the same specific way every single time, handing back a new list of 8 numbers.
Concretely, and this is genuinely all "matrix multiplication" means: each of those 8 output numbers is its own weighted sum — multiply each of the 16 input numbers by its own fixed number, then add up all 16 products to get one output number. Do that 8 times, each time with a different set of 16 fixed numbers, and you've got all 8 outputs. That whole grid — 8 rows of 16 fixed numbers each — is the matrix; nothing about it is more mysterious than repeated multiply-and-add. There are three separate matrices here, one for Query, one for Key, one for Value, so the same starting fingerprint comes out three different ways depending on which job it's being reshaped for. (Want to sit with matrix multiplication on its own, worked by hand from a 2×2 up to two matrices of totally different shapes? See Matrix Matrix.)
📚 Query — what this token is looking for. Key — what this token offers, to anything comparing itself against it. Value — what it actually contributes once a match is found. Real models compute several independent Query/Key/Value triples side by side, called heads — see "Go deeper" below.
Who's looking at whom
For every pair of tokens, take the dot product of one token's Query vector with the other's Key vector — multiply each pair of matching numbers together and add up all the results, which takes two 8-number vectors and turns them into one plain number. The bigger that number, the more the two vectors point in the same direction, which is exactly what we want: a stand-in for "how relevant is this other word to me." Divide that number by √8 (8 because that's how long a Query/Key vector is here — dividing by the square root of that length is just a fixed correction that keeps the numbers from swinging wildly as vectors get longer) to keep the scale steady.
Collect one token's numbers against every token in the sentence (including itself) into a list, then turn that list into a genuine probability distribution with softmax, in two steps. First, raise e to the power of each number in the list — e is just a fixed, special number (about 2.718) that mathematicians reach for whenever they need a quick way to turn any number, negative or positive, into a positive one without changing which numbers were bigger than which. Then divide every result by their total, so the whole list becomes non-negative and sums to exactly 1.
Finally, take a weighted average of every token's Value vector using those probabilities as the weights — that weighted average is this token's new, context-dependent vector. Click a word below to see its own weights.
⚠ What's real here, and what isn't: the dot products, the softmax normalization, and the weighted averaging are exactly the real computation. What's not real is which words end up mattering to which — the Query/Key matrices are fixed, untrained numbers, not weights learned from billions of real sentences. In an actual trained model, learning those matrices is what makes "bank" attend differently near "river" than near "loan." Here, the mechanism is authentic; the judgment behind it isn't. See "How this actually works" below for why that trade was made on purpose.
Do you redo the math, or reuse it?
Everything above treated your sentence as fully typed out already. A real model instead writes one word at a time, re-running this whole computation after every new word. Here's a fact worth noticing first: token i's Key and Value vectors depend only on token i itself and the ones before it — never on anything that comes later. So once they're computed, they never change again as more tokens get appended; only a new Query vector is ever needed for the newest token.
Reusing numbers you've already computed, instead of recomputing them from scratch, is what "caching" means — here, and in software generally. The table below holds every token's Key and Value vector so far. The toggle controls which of two equivalent strategies fills it in as new tokens get generated (the button that actually generates one lives in the next section, since that's also where you'll see what gets predicted):
"Reuse old rows" only computes the Key/Value vectors for the newest token, and leaves every earlier row exactly as it was. "Recompute every row" throws all of it away and redoes every token's Key/Value vectors from zero, even the ones that didn't need it — watch the whole table flash on the next generation step instead of just the new row. Both strategies produce the exact same numbers — this toggle only changes how much repeated work happens to get there, never the result.
| token | Key | Value |
|---|
What comes next?
Take the last token's context-dependent vector from Stage 4 and run it through one more fixed matrix — the same reshaping recipe from Stage 3, just this time turning the 8 numbers back into 16, the same length the embeddings started at. Then take its dot product against every candidate word's own embedding — the same kind of comparison Stage 4 already used, just against a list of candidate next words instead of other tokens in your sentence — and run the results through that same softmax normalization to get a real probability for each candidate. Pick a word yourself below, or click "Generate next token" and let the choice be made for you (highest probability, or a probability-weighted random pick — your call), and the whole computation — tokens, embeddings, Query/Key/Value, attention, the table above — visibly reruns for one more step.
Your sentence portrait
A single image built from everything above: your sentence, its attention pattern, the embedding fingerprints, and a slice of the KV cache — rendered small and dense, like circuitry. Download it, or scan the QR code to reopen this exact sentence live.
Whatever pattern your sentence made is worth putting on something — download the portrait above, then upload it here to make it real.
Go deeper — four more things worth playing with
Each of these reuses a mechanic already built above, just isolated so you can sit with one idea at a time. Everything below stays in sync with whatever sentence you typed above.
🧵 Multi-head comparison
Real models don't run one Query/Key/Value lens — they run several in parallel, called heads, each with its own fixed random projections here. Different heads learn to track different kinds of relationships in a trained model — one might track grammar, another topic. Here, since nothing is trained, they just look like independent, differently-shaped guesses — which is itself the honest point.
📍 Positional encoding, isolated
The exact same comparison from Stage 2's toggle, replayed side by side with nothing else changing — the same embedding, position off vs. on.
🔒 Causal masking
A model generating text one token at a time never has later tokens to compare against — they don't exist yet. This simulates that on a fully-typed sentence: for every pair where token j comes after token i, that dot-product score is set to −∞ before softmax runs, so its probability becomes exactly 0. Compare the same sentence with and without it:
🌡️ Temperature & sampling
The same next-word probabilities from the prediction step, now with a dial. Before softmax runs, every score gets divided by this number first: a small value exaggerates the gap between the best score and the rest, producing a sharply peaked distribution; a large value shrinks that gap toward a flatter, closer-to-uniform one. This one number is what chat products expose as "creativity."
How this actually works — the precise version, if you want every detail
Every real computation on this page runs on your own words, live, entirely in your browser — nothing you type is sent anywhere. The one thing that's not real is the starting material: normally these numbers come from months of training on real text. Here, that training never happened.
Why the embeddings and projections are fixed, not trained
Every token's embedding here is a deterministic fingerprint generated from the word's own spelling (a seeded hash, not a lookup into any real vocabulary) — "bank" always gets the exact same 16 numbers, but those numbers carry no relationship to "river," "loan," or anything else. The Query, Key, and Value projections are small, fixed random matrices too. That was a deliberate choice, not a shortcut taken lightly: a real pretrained embedding table would make Stage 1's fingerprints "mean" something, but the attention pattern you actually watch in Stage 4 is driven by the Query/Key projections, not the embeddings themselves — and those projections would still be untrained, fixed, random numbers either way. Shipping a real embedding table would have added real asset weight for a benefit that doesn't survive to the part you actually watch. So: the mechanism (every dot product, softmax, and weighted blend) is 100% real and exactly what a trained model does. The judgment behind which words end up mattering to which is not — that's what training on real language actually buys you, and it's honestly out of scope for a page that runs entirely in a browser tab with no server and no training step.
How training actually changes these numbers
Every matrix on this page — the embedding table, all three Query/Key/Value projections, the final prediction matrix in Stage 6 — is made of exactly the kind of numbers usually called a model's weights (as in "a 7-billion-weight model," or a "pretrained weights" file you'd download). That's a different thing from the attention weights Stage 4 computes: those are a fresh, temporary result, recalculated from scratch for every sentence you type, using whatever the fixed weights happen to be. The weights themselves are what training actually produces — once, in advance — and then ships as "the model."
In a real model, those weights start out exactly the way they do here: random numbers with no meaning yet. Training nudges every one of them, a tiny bit at a time, until predictions get better. Concretely: feed the model a real sentence with the next word hidden, let it guess using its current weights, and compare that guess to the real next word with a loss function — one number measuring how wrong the guess was. Backpropagation then runs backward through the exact same forward pass named at the top of this page — starting from the prediction and working back through every step to the embeddings — to work out exactly how much each individual weight contributed to that wrongness. Gradient descent nudges every weight a small step in whichever direction would have made the guess less wrong. Repeat that cycle billions of times over a huge pile of real text, and the weights stop being random — a Key projection's weights start actually encoding "this is a noun," a Query projection's weights start actually encoding "I'm an adjective looking for the noun I modify." Nothing about the mechanism changes from what this page already computes — same forward pass, same dot products, same softmax, same weighted average, every time. Training only ever changes the weights sitting inside the matrices, never what's done with them.
Tokenization
Real tokenizers (like BPE, used by GPT-style models) learn their word-splitting rules
from millions of real documents. This page uses a much simpler stand-in: split on
whitespace, peel off trailing punctuation as its own token, and split a long word before a
common suffix (like -ing or -tion), marking the split piece with
a leading ## the same way real subword tokenizers do. It's a simplified
approximation built to make the "words sometimes split into pieces" idea visible and
genuinely happen sometimes — not a reimplementation of BPE itself.
Positional encoding
This one is the real formula from the original paper — alternating sine and cosine waves at different frequencies per position, added directly on top of the embedding. No shortcuts here: this is exactly how real transformers inject word order.
Attention
Real scaled dot-product attention: for every pair of tokens, the Query of one is dotted with the Key of the other, divided by the square root of the projection size (keeps the numbers from blowing up as dimensions grow), and every token's full row of scores is run through softmax so it sums to exactly 1. Those weights are the literal thickness/brightness of the threads you saw in Stage 4. The final output for a token is a weighted sum of every token's Value, using those exact weights — also real, also exactly what a trained model does with its own (trained) numbers.
Why this can happen to a whole sentence at once
Stage 4 computes every pair of tokens' scores at the same time, as one matrix multiplication over the whole sentence, not one pair at a time in sequence. That's not an implementation shortcut — it's the actual reason transformers train the way they do. Older architectures (recurrent networks, RNNs) read a sentence one word at a time, and each step genuinely couldn't start until the previous step finished, because each word's output fed directly into the next word's computation. Attention has no such dependency when the whole sentence is already known (which is exactly the case during training, and while reading a prompt): every token's Query needs comparing against every token's Key regardless of where either one sits, so a GPU can compute the entire grid of comparisons in one shot instead of working through the sentence step by step. That's the whole reason transformers could be trained at the scale they have been — not a cleverer computation per pair, but removing the "wait for the previous step" bottleneck entirely.
Generating new text is the one place that parallelism doesn't apply, and it's exactly why the KV cache below exists. When generating, the next word doesn't exist yet — you can't compute a Query against a Key that hasn't been produced yet — so token-by-token generation is unavoidably sequential no matter how the rest of the math works. The KV cache isn't needed during training (the full sentence already exists, so everything above applies) — it's a generation-time fix for the one part of this pipeline that genuinely can't be run all at once.
Reusing Key/Value vectors (the "cache")
Keys and Values only ever depend on a token and the ones before it, which is why they never need to be recomputed once they exist — a fact that's true in any real model, trained or not, since it's a property of the causal generation setup, not of what the numbers happen to be. The "reused instead of recomputed" counter is a real count of how many rows a from-scratch recomputation would have redone at each step that the "reuse old rows" path skipped — it's not measuring actual wall-clock time (everything here is small enough to be instant either way), just the real structural redundancy caching exists to eliminate.
Predicting the next token
A real language model's final layer projects its last hidden state back into vocabulary space and computes a probability for literally every word it knows. This page does the same kind of projection — one more fixed matrix maps the attention output back into embedding space, then compares that against a candidate word list using the exact same dot-product similarity attention itself uses — but the candidate list is a few hundred common English words, not a real vocabulary, and the matrix is untrained. That's why generated continuations read as plausible-looking word soup rather than fluent language: the mechanism (project, compare, normalize with softmax, then either take the top probability or draw a probability-weighted random word) is real; the fluency a trained model has learned on top of it is not.
The sentence portrait and share link
The portrait canvas redraws the same attention geometry, embedding strips, and a slice
of the KV cache at print resolution, entirely from your sentence — no separate export
pipeline that could drift out of sync with what you actually saw. The share link and QR
code encode nothing but your sentence text itself (URL-encoded, packed after ?s=)
— every embedding, projection, and attention weight is fully deterministic from that one
string, so there's nothing else to transmit. Generated via the same vendored
qrcode.min.js library already sitting in Tonus's, Echo State's, and Inkling's
own folders.
Learn more
Vaswani et al., Attention Is All You Need (2017) — the paper this page's name and mechanism both come from. Everything on this page is the real computation at small, untrained scale, not an illustration of it — the same claim every other tool in The Concept Lab makes about its own math.