LLM Latency, Cost, and Non-Determinism in Production: Where Backend Instincts Break

You've spent years building systems on three assumptions: the same input produces the same output, latency is measured in milliseconds and roughly constant per request, and cost scales with request volume. An LLM call violates all three at once. This lesson is the map of exactly where your instincts will steer you wrong.

Same input, different output

Call GET /users/42 twice and you get the same JSON. Call an LLM twice with an identical prompt and you can get two different (sometimes materially different) responses. This isn't a bug. Generation is a sampling process: at each step the model produces a probability distribution over possible next tokens, and the sampler picks one. The temperature parameter controls how adventurous that pick is. High temperature flattens the distribution (more variety); temperature approaching zero concentrates it (usually pick the top token).

So set temperature=0 and you're deterministic, right? In practice, no. Four things break the guarantee:

• Floating-point non-determinism. GPU kernels don't guarantee a stable reduction order. Two logits that differ by 1e-6 can flip which token is "top," and one flipped token early in a response changes everything after it.
• Batching. Providers batch your request with strangers' requests for throughput. Batch composition affects the numerics, which affects your output.
• Provider-side changes. Inference stacks get re-optimized constantly: new kernels, new quantization, new hardware. Same model name, subtly different numbers.
• Silent model updates. A pinned model version helps, but aliases like -latest move under you, and even pinned snapshots get deprecated on a schedule you don't control (the versioning lesson is the survival guide).

Treat temperature=0 as "low variance," not "deterministic." It's closer to reading from an eventually-consistent replica than from a primary.

What this does to your testing playbook

Exact-match assertions are dead. assert response == expected will pass in CI on Monday and fail on Wednesday with zero code changes. What replaces it looks more like property-based testing than unit testing:

• Structural assertions. The output parses as JSON, the status field is one of three allowed values, the summary is under 100 words.
• Semantic assertions. The response mentions the refund policy; it does not invent an order ID. Often checked by another model: "LLM-as-judge."
• Statistical evaluation. Run 50 cases, require a 94%+ pass rate, and alert on drift over time, the way you treat a flaky-tolerance integration suite, but by design.

This discipline has a name (evals), and it's important enough that module 6 of this track is entirely about it. For now, internalize the shift: you're not testing a function, you're monitoring a distribution.

The latency profile is alien

Your Postgres p99 might be 40ms. Your Redis p99, 2ms. An LLM call routinely takes 2–30 seconds, and, more importantly, it doesn't behave like one request. It behaves like a stream with two distinct phases, and each phase gets its own metric:

• TTFT (time to first token): how long before the first byte of the answer arrives. As of mid-2026, typically 200–800ms on hosted APIs for warm requests, but p99 can hit several seconds under load or with long prompts.
• Tokens per second: how fast the rest streams out. Roughly 100–300 tok/s for small/fast models, 30–80 tok/s for frontier-tier models, as of mid-2026. (Artificial Analysis publishes live TTFT and throughput numbers per provider.)

The consequence: output length dominates total latency. A 500-token answer at 50 tok/s takes 10 seconds of generation regardless of how snappy TTFT was. If you want a faster response, the highest-leverage fix is often "make the model say less," not "get a faster model."

The p50/p99 spread is also wider than anything in your current fleet. A 3–10x spread between median and tail is normal, driven by provider load, batch scheduling, and variable output length. Your intuition that p99 sits within 2x of p50 does not survive contact.

Implications for how you build

• Stream by default. A response that starts rendering in 500ms feels fast even if it takes 12 seconds to finish. TTFT is your perceived-latency metric; total duration is your capacity metric. (What the stream actually sends is a module 2 lesson.)
• Stage-aware timeouts. A single 5-second timeout is wrong twice: it kills healthy long generations and waits too long on a dead connection. Set a tight timeout on TTFT (for example, 10s), a generous one on total duration (60–120s), and consider an inter-token stall timeout in between. What your API serves when a stage trips is failure design, its own lesson.
• Rethink SLOs. "p99 under 500ms" is not a meaningful target here. SLO on TTFT, on tokens/sec, and on completion rate instead. And expect to renegotiate them per model tier.

Stage-aware timeouts in practice (one rolling timer, two phases):

// Tight on first token, generous on total, stall-guarded in between.
async function streamWithTimeouts(url: string, init: RequestInit) {
  const ctl = new AbortController();
  let timer = setTimeout(() => ctl.abort("ttft-timeout"), 10_000);
  const deadline = Date.now() + 120_000;             // total-duration cap
 
  const res = await fetch(url, { ...init, signal: ctl.signal });
  const reader = res.body!.getReader();
  const chunks: Uint8Array[] = [];
 
  for (let r = await reader.read(); !r.done; r = await reader.read()) {
    clearTimeout(timer);                             // a token arrived
    if (Date.now() > deadline) ctl.abort("total-timeout");
    timer = setTimeout(() => ctl.abort("stall-timeout"), 15_000);
    chunks.push(r.value);
  }
  clearTimeout(timer);
  return chunks;
}

The first timer guards time-to-first-token; once tokens flow, it becomes a rolling inter-token stall detector, and the deadline caps the whole stream. Three failure modes, three distinct abort reasons in your logs.

Cost scales with tokens, not requests

Backend pricing intuition says requests are the unit of cost, so you optimize request count. LLM pricing inverts this: requests are nearly free; verbosity is expensive. You pay per token, in and out, with output typically 4–5x the price of input. As of mid-2026, frontier-tier models run roughly $2–5 per million input tokens and $10–25 per million output tokens; the fast/small tier is around $0.10–0.50 in and $0.40–2 out.

The dangerous part is what silently rides along on every call: your system prompt is re-sent every request. Conversation history is re-sent every turn. RAG context gets stuffed in wholesale. None of this shows up in your request count. All of it shows up on the invoice.

Worked example: same feature, 5x the bill

A support-ticket summarizer on a frontier-tier model at $3 per million input tokens and $15 per million output tokens:

Same feature. Same model. One team pays 5x more, and nothing in their dashboards flags it: request count, error rate, and latency all look identical. A "small" prompt edit that adds 2,000 tokens of examples is a five-figure monthly line item at scale, shipped without review.

Concretely, that log line:

const PRICE = { in: 3 / 1e6, out: 15 / 1e6 };  // USD/token; lives with the model id in config
 
log.info("llm_call", {
  feature: "ticket-summary",
  tenant: req.tenantId,
  model: res.model,
  input_tokens: res.usage.input_tokens,
  output_tokens: res.usage.output_tokens,
  cost_usd:
    res.usage.input_tokens * PRICE.in + res.usage.output_tokens * PRICE.out,
  ttft_ms: metrics.ttftMs,
  duration_ms: metrics.totalMs,
});

Group that by feature and you have per-feature cost dashboards; group by tenant and you know which customer is expensive before pricing renewals. Neither is retrofittable onto history you didn't log.

There are real mitigations: prompt caching (providers discount repeated prefixes by 50–90%), history truncation and summarization, routing easy requests to cheap models. Module 7 covers cost engineering properly. The prerequisite is measurement.

Why the rest of this track exists

Look at what these three properties demand. Non-determinism kills exact-match testing, so you need evals (module 6). Alien latency and per-token cost can't be managed blind, so you need tracing and observability on every call (module 7). Token-based pricing turns prompt design into spend management, so you need cost engineering (also module 7). These aren't optional extras bolted onto "real" AI engineering: they are the job, the same way tests, monitoring, and capacity planning are the job in backend work. The rest of this track builds that toolkit.

Key takeaways

• LLM output is sampled, not computed. Even temperature=0 is "low variance," not deterministic: floating-point math, batching, and provider updates all move the output.
• Exact-match assertions die; structural, semantic, and statistical assertions (evals) replace them.
• Latency has two numbers: TTFT (perceived speed, sub-second when healthy) and tokens/sec (30–300 depending on tier, as of mid-2026). Output length dominates total time.
• Stream by default, use stage-aware timeouts (tight on TTFT, generous on total), and expect a p50/p99 spread of 3–10x.
• Cost scales with tokens, not requests. A chatty system prompt or unbounded history can 10x your bill with zero change in request volume.
• Log tokens and cost per call from day one, and treat spend as an engineering metric, the same discipline you already apply to cloud costs.