Interactive visual guide explaining LLMs.

> WITHOUT RAG > "I don't have reliable information about a colony called Ares Base. As of my > training cutoff, no such Mars colony has been established..."Oh we must have lived in a parallel universe then if this is a "without rag" textbook example.

Highly recommend instead reading the human created "The Illustrated GPT-2" by Jay Alammar - https://jalammar.github.io/illustrated-gpt2/And his similar work.He also has a free course on "how llms work"

I disagree with some comments saying it's not worth reading since it's generated by LLM. Even though I made it clear that I have download the transcript. LLMs are exceptionally good at generating accurate information if information is directly loaded into context window.

I think that BPE visualization is slightly misleading, because it seems to imply that the "old" (smaller) tokens are thrown away and replaced with longer tokens, which is not the case.In fact, it is purely additive process: we iteratively add the most frequent pairs to the set, until we reach the desired total number of tokens. But we never remove tokens, we keep everything, including the initial 256 tokens, representing bytes.This ensures that the model is capable of producing every possible unicode sequence (in fact, I think that it is capable of producing every possible byte sequence, but bytes that are not valid unicode are filtered during sampling).Edit : also, this page entirely skips the attention mechanism, which is, in my opinion, both the most interesting part and the part that is hardest to understand (I can't say that I fully understand it, to me it is just some linear algebra matrix multiplication magic).

The page does very poor job tokenizing phrase "Noinceolik fiyulnabmed fyvaproldge" into "Noinceolik fiyulnabm ed fyvaproldge", factoring only "ed" suffix. As if made up words such as "noinceolik" are so common they are part of 100K token vocabulary.The actual application of GPT-5 tokenizer at [1] to my made up phrase results in 14 tokens, only two of them are four characters long and there are tokens containing spaces.[1] https://gpt-tokenizer.dev/I will read along, though.

Update: The "single hard drive" claim was wrong and I've corrected it to "roughly 10 consumer hard drives" (44TB ÷ ~4TB = ~11). Attribution to Karpathy is now a direct link. Added a caveat under the stats noting these are representative 2024-era figures — the exact numbers shift with every release and that's somewhat the point. Also did a few iterations on visual redesign (linked in the header as v2) with a proper top navigation bar after a few people found the dot nav hard to use and UI was jumping.Also I have not fact checked everything but I have read it and it seems to be aligned with what is described in the lecture.

This is completely AI generated..don't bother reading.

Have you reread what was produced by Claude Code before publishing ? This thing in one of the first paragraph jumps out:> you end up with about 44 terabytes — roughly what fits on a single hard driveNo normal person would think that 44 TB is a usual hard drive size (I don't think it even exists ? 32TB seems the max in my retailer of choice). I don't think it's wrong per se to use LLM to produce cool visualization, but this lack of proof reading doesn't inspire confidence (especially since the 44TB is displayed proheminently with a different color).

I haven't found an explanation yet that answers a couple of seemingly basic questions about LLMs:What does the input side of the neutral network look like? Is it enough bits to represent N tokens where N is the context size? How does it handle inputs that are shorter than the context size?I think embedding is one of the more interesting concepts behind LLMs but most pages treat it as a side note. How does embedding treat tokens that can have vastly different meanings in different contexts - if the word "bank" were a single token, for example, how does embedding account for the fact that it can mean river bank or money bank? Do the elements of the vector point in both directions? And how exactly does embedding interact with the training and inference processes - does inference generate updated embeddings at any point or are they fixed at training time?(Training vs inference time is another thing explanations are usually frustrating vague on)

Page keeps annoyingly scroll-jumping a few pixels on iOS safari