How to Choose an LLM in 2026: A Backend Engineer's Decision Framework

"Which model should we use?" is the "which database should we use?" of AI engineering: the question sounds like it has one answer, the internet is full of benchmark charts claiming to settle it, and the real answer is "different tools for different jobs, and you'll probably run more than one." You wouldn't put session tokens in ClickHouse or analytics in Redis. Same discipline applies here.

The landscape, in tiers

As of mid-2026 the market has settled into three tiers, and the mapping to the database world is almost embarrassingly clean.

Frontier closed models: your Postgres

OpenAI's GPT-5.x line, Anthropic's Claude Opus 4.8, Google's Gemini Pro. Maximum capability: complex reasoning, long multi-step agent workflows, gnarly code generation, tasks where a wrong answer is expensive. Like Postgres, they're the safe default when you don't yet know your workload: general-purpose, deep, and rarely the wrong first choice. Also like Postgres under a heavy analytical query, they're the slowest and priciest option per request.

Fast/cheap tiers: your Redis

Claude Haiku, GPT-5 mini and nano, Gemini Flash, and Anthropic's Sonnet sitting one notch up as the mid-tier workhorse. These handle the high-volume, latency-sensitive, well-scoped work: classification, extraction, summarization, routing, autocomplete. Sub-second-ish first tokens, 10–100x cheaper than frontier. Nobody brags about them at conferences; they quietly serve most production traffic: the Redis career path.

Open-weight models: your self-hosted anything

Meta's Llama 4 family, Alibaba's Qwen 3.x, DeepSeek's V3/R1 line, Mistral. You download the weights and run them yourself (vLLM on your own GPUs) or rent them from inference hosts (Together, Fireworks, Groq) at commodity prices. The top open-weight models now land within striking distance of last year's frontier: genuinely good, and improving fast. The catch is the same as running your own Postgres on EC2 instead of RDS: total control, and total operational ownership.

The map, in one table

Prices are rough blended ranges as of mid-2026, in USD per million tokens (input / output). Treat them as order-of-magnitude (they move quarterly, always downward) and check the live pages before you commit: OpenAI, Anthropic, Gemini.

Note the shape of that table: roughly three orders of magnitude between the cheapest and most expensive cell. Model choice is the biggest cost lever you have: bigger than caching, bigger than prompt trimming. (The tokens lesson shows how to turn these rates into monthly cost projections.)

And just as most real systems run Postgres and Redis and something columnar, most mature LLM products run 2–3 models: a cheap one to route or classify, a mid-tier one for the main workload, a frontier one for the hard 5% of requests.

Why public benchmarks mislead, exactly like database benchmarks

You already know not to pick a database off a TPC-H chart or a vendor's "10x faster than Postgres" blog post. LLM leaderboards (MMLU, SWE-bench, LMArena rankings) fail the same way, plus one failure mode databases don't have:

• Train-on-test contamination. Benchmark questions leak into training data: the model has effectively seen the test. Imagine a database that recognized TPC-H queries and returned precomputed answers. Scores go up; capability doesn't. Labs don't even have to cheat deliberately; the benchmarks are all over the public internet.
• Benchmark workload is not your workload. A model that tops a competition-math leaderboard can still mangle your invoice-extraction task, the same way a database that wins bulk-insert benchmarks can fall over on your point-lookup traffic. Aggregate scores across models within a tier are separated by low single digits, noise relative to how differently they behave on your prompts.
• Teaching to the test. Once a benchmark matters commercially, labs optimize for it, and it stops measuring what it used to. Goodhart's law, GPU edition.

The only benchmark that matters is an eval you run on your own data: a few hundred real examples from your actual task, scored automatically. That's Module 6 of this track, and it's the single highest-leverage practice in this field. Until you have one, treat every leaderboard delta under about 10 points as unknowable.

The decision framework: start strong, walk down

Here's the framework, and the direction matters more than anything else in this lesson:

1. Prove the feature with the strongest model available. Prototype on GPT-5.x or Opus-class. If the frontier model can't do the task acceptably, no cheaper model will, and you've learned that for a few dollars instead of a few weeks.
2. Capture your prototype traffic as an eval set. Real inputs, graded outputs. Even 50–200 examples beats vibes.
3. Walk down the cost/latency curve. Rerun the eval on the mid tier, then the cheap tier, then open-weight. Stop one step above where quality breaks. Often the mid tier matches frontier on your task at a fifth of the price. You find out in an afternoon.
4. Split the workload if quality breaks unevenly. Cheap model for the easy 90%, escalate the hard 10%. This is read-replica-vs-primary routing, applied to inference.

The walk itself is a dozen lines (the same eval set, re-run down the cost curve):

// Stop one tier above where quality breaks.
const TIERS = ["frontier", "mid", "small"] as const;
 
let winner = TIERS[0];
for (const tier of TIERS) {
  let passed = 0;
  for (const ex of evalSet) {              // 50–200 real captured cases
    const out = await complete({ tier, prompt: ex.prompt });
    if (ex.check(out)) passed++;           // schema check, exact match, or judge
  }
  const rate = passed / evalSet.length;
  console.log(`${tier}: ${(rate * 100).toFixed(1)}% pass`);
  if (rate < 0.94) break;                  // quality broke, keep the previous tier
  winner = tier;
}

The failure mode is running this in reverse: start with the cheapest model "to keep costs down," get mediocre output, and burn weeks of prompt tuning without knowing whether the task is hard or the model is just too small. That's premature optimization with a debugging tax, like starting on SQLite "to keep it simple" and hand-rolling the features you'd have gotten free from Postgres. Prototype-phase token costs are trivial; engineer-weeks are not.

When open-weight/self-hosting actually makes sense

Self-hosting an LLM is running your own database cluster: real benefits, real 24/7 costs. It earns its keep in three situations:

• Data residency and compliance. Prompts legally cannot leave your VPC or your jurisdiction: healthcare, defense, some finance and EU workloads. Often the only option, so the calculus is short.
• Sustained high volume on a narrow task. Millions of similar requests a day can make dedicated GPUs cheaper than per-token API pricing, but only at high, steady utilization. Idle H100-class GPUs at $2–4/hour each burn money exactly like an overprovisioned RDS instance, and API prices keep falling underneath your amortization math.
• Fine-tuning and control. You need to train on proprietary data, pin exact weights for reproducibility, or squeeze latency below what APIs offer.

If none of those apply, it's premature optimization. You're signing up for GPU capacity planning, serving-stack upgrades, and model-quality ops (a database migration's worth of work) to save money you probably weren't going to spend. Managed inference hosts (Together, Fireworks, Groq) are the RDS middle ground: open-weight models, API-shaped pricing, no pager.

Model choice is config, not architecture

Here's the mindset shift that makes all of the above cheap to act on: the model is a config value, not an architecture decision. Swapping Sonnet for GPT-5-mini is editing an env var and rerunning your eval, nothing like a database migration, where storage engines and query dialects ossify into your codebase.

But only if you build for it. Route every LLM call through one internal module that owns model names, provider SDKs, prompt assembly, and usage logging: model id in config, never hard-coded at call sites. In its smallest form:

// llm/models.ts: the only file that knows model ids.
export const MODELS = {
  router:     { provider: "anthropic", id: "claude-haiku-4-5" },   // classify, route
  workhorse:  { provider: "anthropic", id: "claude-sonnet-4-5" },  // main workload
  escalation: { provider: "openai",    id: "gpt-5.2" },            // the hard 10%
} as const;
 
// Call sites ask for a role, never a model:
//   await complete({ role: "workhorse", prompt });
// Swapping a vendor or tier is a one-line diff plus an eval run.

The best model this quarter will not be the best model next quarter; prices drop 2–10x per year at constant quality; providers have incidents and deprecate model versions on a schedule. Teams that scatter model: "gpt-5" across forty files relearn this the way teams that scattered raw SQL across forty files learned to want a data-access layer. Module 2 builds this abstraction properly.

Key takeaways

• The model landscape mirrors the database landscape: frontier (Postgres), fast/cheap (Redis), open-weight (self-hosted). Mature products run 2–3 models, not one.
• Pricing spans about three orders of magnitude (roughly $0.05 to $75 per million tokens as of mid-2026), making model choice your biggest cost lever.
• Public leaderboards fail like database benchmarks (contamination, wrong workload, Goodhart). The only benchmark that matters is your eval on your data.
• Always start with the strongest model to prove the feature, then walk down the cost curve until quality breaks. Never start cheap and tune upward.
• Self-host only for data residency, sustained high volume, or fine-tuning. Otherwise it's running your own database cluster to avoid an RDS bill.
• Treat the model as a swappable config value behind one internal module. The best choice today won't be the best choice in six months.