What LLMs Actually Do

You are about to put a component in your stack that is non-deterministic, occasionally wrong with total confidence, and billed by the byte. You would never do that without a mental model of how it behaves under load and where it fails. This lesson builds that model: no ML background required, because you already know the important abstractions from backend work.

The core loop: next-token prediction

An LLM does exactly one thing: given a sequence of tokens, it outputs a probability distribution over what the next token should be. A token is a chunk of text, roughly 3–4 characters of English (more on this in the next lesson). That's the whole primitive. Everything else (chat, code generation, tool calling, "reasoning") is built by running that primitive in a loop. (The machinery underneath is the transformer, Vaswani et al., 2017, but you won't need the math for anything in this track.)

The loop looks like this:

Two things in that loop matter operationally.

First, the model re-reads the entire transcript to produce each token. Generating a 500-token response means 500 forward passes, each over an ever-growing input. This is why output tokens cost 3–5x more than input tokens and why long responses are slow: generation is sequential, one token at a time, like a cursor that can't batch.

Second, the model's output is a probability distribution, and the runtime samples from it. That's the source of non-determinism. Same input, different output, by design. You can turn sampling down (temperature 0 gets you close to greedy decoding) but you should treat determinism the way you treat clock skew in distributed systems: something you engineer around, not something you assume.

Why such a dumb mechanism works

"Predict the next token" sounds like it should produce autocomplete, not a system that writes working SQL. The reason it doesn't stop at autocomplete: to predict the next token well across trillions of tokens of training text, the model is forced to internalize the structure that generated that text: grammar, code semantics, the fact that a Postgres EXPLAIN output follows certain shapes. Compression forces modeling.

The most useful emergent behavior is in-context learning: put examples or instructions in the input, and the model conditions its predictions on them, without any retraining. Show it three examples of your log format and ask it to parse a fourth; it will. From your seat, this means the model is programmable at request time via its input. The prompt is not a query. It's closer to configuration injected per-request.

Weights vs context: compiled code vs request state

There are exactly two places information can live in an LLM system, and confusing them causes most production bugs.

Weights are the model's parameters, fixed at training time. Think of them as the compiled binary: they encode everything the model "knows": language, APIs as of its training cutoff, general facts. You cannot write to them at request time. If the model was trained before your internal service existed, the weights know nothing about it, ever.

The context window is the input sequence for the current call. Think of it as request state: headers, body, session data you loaded and stuffed into the request. It's the only writable memory you have. It's finite (128k to ~1M tokens depending on model, as of mid-2026), it's billed per token, and it evaporates when the call returns.

Almost all AI engineering (prompting, RAG, tool use, agent design) is context window management. It's the working set you control.

Chat is a stateless API over a growing transcript

Here's the part that surprises backend engineers most: the model remembers nothing between calls. There is no session on the model side. No connection state, no memory of the previous request. Every API call is a cold start.

"Conversation" is a client-side illusion. Your application keeps the transcript, appends the new user message, and sends the entire history back with every request (this is literally the shape of the Messages API):

POST /v1/messages
{ "messages": [
    {"role": "user",      "content": "turn 1"},
    {"role": "assistant", "content": "reply 1"},
    {"role": "user",      "content": "turn 2"}     // model sees ALL of it, fresh
]}

This is exactly the stateless-service pattern you already know: the server holds nothing, all state travels in the request. The implications are the same ones you'd derive for any stateless service:

• You own state management. Storing, truncating, and summarizing conversation history is your application's job: the provider gives you a pure function, not a session store.
• Cost and latency grow with conversation length, because turn 20 resends turns 1–19 as input tokens. (Providers offer prompt caching to discount re-sent prefixes. More in the tokens lesson.)
• You can rewrite history. Since the model has no memory, you're free to edit, compress, or reorder the transcript between turns. This is a feature: it's how summarization-based memory and context pruning work.

What LLMs are structurally bad at

These aren't bugs awaiting a patch. They fall out of the mechanism, so you design around them the way you design around network partitions.

Exact arithmetic and precise symbol manipulation. The model predicts digit tokens from patterns, it doesn't execute an ALU. It will nail 2 + 2 and confidently fumble 7-digit multiplication. Fix: give it a calculator or code-execution tool. Never let an LLM do math you could do in code.

Guaranteed factuality. The model produces the most plausible continuation, and plausible is not the same as true. When the training data is thin around a question, the model interpolates, and the interpolation comes out fluent and confident. That's a hallucination: not a malfunction, but the same generative process that produces correct answers, operating past the edge of its data. A plausible-but-fake API method looks exactly like a real one, syntax and all.

Knowing what it doesn't know. There's no built-in "cache miss" signal. A database returns zero rows when it has no data; an LLM returns a well-formed answer anyway. Calibration ("I'm not sure") can be prompted for and has improved, but it is not a guarantee you can build on. If correctness matters, verify downstream: check citations against retrieved sources, run generated code, validate output against a schema.

"Fancy autocomplete": true, and misleading

Mechanically the phrase is accurate: the model is a next-token predictor, full stop. It's misleading in the same way "Postgres is fancy grep" is misleading: the primitive undersells what the engineering on top of it yields. Prediction pressure at sufficient scale produces a component that follows novel instructions, transforms unstructured text to structured output, and writes code that compiles.

Use the phrase as an operational reminder, not a dismissal: the thing is generating plausible continuations, not consulting a source of truth. Plausibility is the product. Truth, when you need it, is your job to bolt on.

When you're ready to see where this component actually sits in a real service, the integration patterns guide maps six ways to wire an LLM into an existing backend without rewriting it.

Key takeaways

• An LLM is a next-token predictor run in a loop; output tokens feed back in as input, one at a time, which is why generation is slow and output tokens cost more.
• Weights are the compiled binary (fixed, training-time knowledge); the context window is request state (writable, per-call, finite). You program the system by managing context.
• Chat APIs are stateless. The model remembers nothing between calls; your app resends the full transcript every turn and owns all state management.
• Hallucination is confident interpolation past the edge of the training data: a property of the mechanism, engineered around with tools, retrieval, and downstream validation, never patched away.
• Structurally weak spots: exact arithmetic (use a tool), guaranteed factuality (verify), and self-knowledge of uncertainty (don't trust confidence).
• "Fancy autocomplete" is mechanically true and operationally useful: it reminds you that plausibility, not truth, is the native output.