How to Add an LLM to Your Existing Backend Without Rewriting It: 6 Integration Patterns

You don't need a new service, a vector database, or an agent framework to add an LLM to an existing backend. In production systems, an LLM feature almost always lands as one of six integration patterns: an inline endpoint on the request path, an async queue worker, an internal LLM gateway, read-path retrieval with pgvector, an event-driven enrichment consumer, or a feature-flagged parallel path. Which one you pick comes down to three constraints you already reason about daily: is a user waiting, what happens when the call fails, and how many calls per minute.

This guide walks through all six with runnable code, then gives you the decision table and the three failure modes that bite every pattern. It assumes you know your way around a backend and nothing about LLMs. If you want the raw API mechanics first, start with Your First LLM API Call.

One setup note: every snippet uses a single callLLM() helper so the patterns stay provider-neutral. Here it is once, using Anthropic's Messages API (the OpenAI equivalent is the same shape with different field names):

// lib/llm.ts: the only file that knows which provider you use
export async function callLLM(
  prompt: string,
  opts: { maxTokens?: number; signal?: AbortSignal } = {}
): Promise<string> {
  const res = await fetch("https://api.anthropic.com/v1/messages", {
    method: "POST",
    signal: opts.signal,
    headers: {
      "x-api-key": process.env.ANTHROPIC_API_KEY!,
      "anthropic-version": "2023-06-01",
      "content-type": "application/json",
    },
    body: JSON.stringify({
      model: "claude-haiku-4-5-20251001",
      max_tokens: opts.maxTokens ?? 512, // cost ceiling, per request
      messages: [{ role: "user", content: prompt }],
    }),
  });
  if (!res.ok) throw new LLMError(res.status, await res.text());
  const data = await res.json();
  return data.content[0].text;
}

Pattern 1: The inline endpoint

The simplest thing that works: one new route handler that calls the LLM synchronously, because a human is sitting there waiting for the answer.

app.post("/api/support/suggest-reply", async (req, res) => {
  // LLM tail latency is alien; a hard timeout is not optional
  const signal = AbortSignal.timeout(10_000);
  try {
    const suggestion = await callLLM(
      buildPrompt(req.body.ticketText),
      { maxTokens: 400, signal }
    );
    res.json({ suggestion });
  } catch (err) {
    // Degrade, don't 500: the feature is optional, the page is not
    res.json({ suggestion: null, degraded: true });
  }
});

Use it when the feature is interactive and the response informs what the user does next: draft suggestions, summarize-this-thread, natural-language search.

Wrong when the call volume is high and nobody is waiting (you're paying interactive-grade latency tolerance for batch work) or when your endpoint's p95 budget is tighter than the model's p95, which for non-trivial prompts is measured in seconds, not milliseconds. For anything chat-shaped, add streaming so perceived latency drops to first-token time.

Pattern 2: The queue worker

If nobody is staring at a spinner, get the LLM off the request path entirely. You already have the infrastructure: whatever queue runs your emails and exports.

// On the write path: enqueue and return immediately
await queue.add("summarize-ticket", { ticketId }, {
  attempts: 4,
  backoff: { type: "exponential", delay: 2_000 },
});
 
// Worker: the LLM call with retries handled by the queue
new Worker("summarize-ticket", async (job) => {
  const ticket = await db.tickets.find(job.data.ticketId);
  const summary = await callLLM(summaryPrompt(ticket.body), { maxTokens: 300 });
  await db.tickets.update(job.data.ticketId, {
    summary,
    summaryStatus: "done",
  });
});

Retries, backoff, dead-letter handling, concurrency limits: the queue gives you all of it for free, which matters because LLM providers throw 429s and 529s as a matter of routine. Make the job idempotent (write to a status column, not append) and you can retry fearlessly.

Use it when you're enriching data at volume: summaries, classification, tagging, extraction.

Wrong when the user needs the answer in this request. And be honest about "needs."

Pattern 3: The internal LLM gateway

The first two patterns scatter callLLM() through your codebase. That's correct at first. The moment a second feature (or second team) starts calling models, centralize into one internal module or service that owns four things: provider keys, model routing, caching, and spend accounting. (Designing this seam properly, dialect normalization and build-vs-buy included, is the track's module 2 capstone.)

interface LLMGateway {
  complete(req: {
    feature: string;          // who's asking, for cost attribution
    prompt: string;
    maxTokens: number;
    cacheKey?: string;        // exact-match cache opt-in
  }): Promise<{ text: string; tokensIn: number; tokensOut: number }>;
}

Behind that interface you can swap providers, route cheap requests to a cheap model, add a rate-limit semaphore so one feature can't starve the others, and answer "which feature spent $400 yesterday?" without grepping logs. This is the seam where almost all production LLM engineering eventually happens. The production stack lesson maps every component that hangs off it.

Use it when ≥2 call sites exist, or the monthly bill crosses "someone asks about it."

Wrong when you have one feature and one call site: an interface with one implementation and one consumer is ceremony.

Pattern 4: Read-path retrieval with pgvector

The moment the LLM must answer from your data (docs, tickets, product catalog), you need retrieval. The reflex is "add a vector database." If you run Postgres, resist it: the pgvector extension puts embeddings in the database you already back up, monitor, and know how to index.

ALTER TABLE docs ADD COLUMN embedding vector(1536);
CREATE INDEX ON docs USING hnsw (embedding vector_cosine_ops);

// At query time: embed the question, pull the 5 nearest chunks
const { rows } = await db.query(
  `SELECT title, body FROM docs
   ORDER BY embedding <=> $1 LIMIT 5`,
  [await embed(question)]
);
const answer = await callLLM(ragPrompt(question, rows), { maxTokens: 600 });

That's retrieval-augmented generation in its smallest production-honest form: embed on write (via Pattern 2's worker), search on read, stuff the results into the prompt. Chunking strategy, hybrid search, and reranking all matter at scale (that's Module 4 territory), but this version ships this week and answers real questions.

Use it when answers must be grounded in data the model has never seen.

Wrong when your corpus fits in the context window: below roughly a few hundred KB of text, skip retrieval and put everything in the prompt. Simpler, and often better.

Pattern 5: Event-driven enrichment

Your system already emits domain events: ticket.created, order.refunded, document.uploaded. Subscribe an LLM consumer to them and write derived data back. It looks like Pattern 2 but with a different ownership story: no caller anywhere is waiting or even knows the LLM exists. The consumer is a peer of your other event handlers.

events.on("ticket.created", async (evt) => {
  const triage = await callLLM(triagePrompt(evt.body), { maxTokens: 120 });
  await db.tickets.update(evt.ticketId, {
    aiPriority: parsePriority(triage), // validate; never trust raw output
  });
});

The one rule: treat model output as untrusted input. Parse it, validate it against an enum or schema, and have a defined behavior for "the model said something unparseable," because across enough events, it will.

Use it when you want ambient intelligence on existing flows without touching the code paths that create them.

Wrong when the enrichment must be transactional with the triggering write.

Pattern 6: The flagged parallel path

When the LLM replaces existing logic (a regex extractor, a rules-based ranker, a template email), don't swap it in. Run both.

const heuristic = extractWithRegex(input);          // the incumbent
if (flags.enabled("llm-extraction-shadow", user)) {
  queue.add("shadow-extract", { input, heuristic }); // async, invisible
}
return heuristic;                                    // users still get the old path

The shadow worker calls the LLM, stores both outputs, and disagreement becomes a dataset: sample it, decide which side was right, and you have an offline eval before any user sees LLM output. Then cut over gradually behind the same flag. This is the same canary discipline you'd use for any risky dependency. LLMs just make it non-negotiable, because their failures are plausible-looking rather than loud.

Use it when replacing deterministic logic, or when "the AI was wrong" has real cost.

Wrong when the feature is net-new with no incumbent: there's nothing to shadow. Ship Pattern 1 or 2 behind a plain feature flag instead.

The decision table

Start with 1 or 2. Add 4 when grounding matters, 6 when replacing something, 3 when it spreads. Pattern 5 is 2 wearing your event bus.

The three failure modes that bite every pattern

Alien tail latency. LLM p50 might be 800ms while p99 is 20s+. Set explicit timeouts everywhere (AbortSignal.timeout), and set them from your caller's budget, not the provider's defaults, which are effectively "forever."

Unbounded cost. Every request has a price that scales with tokens in and out. Cap max_tokens per call, meter spend per feature (Pattern 3 makes this trivial), and alarm on daily budget: tokens are your new resource limit, and they behave like one.

Retry storms. A 429 means back off, not try harder. Respect retry-after, add jitter, and never let a synchronous endpoint retry more than once; that's what Pattern 2 is for. Combine a provider outage with naive retries and your queue will happily convert one incident into a token bill. The upstream fix is client-side throttling against your published limits, so the 429s mostly never happen.

If you remember one thing: the LLM is a flaky, expensive, occasionally brilliant upstream dependency. Every pattern above is just the standard backend playbook for such a dependency, applied without sentimentality. The AI Engineering track builds each of these patterns out to production depth, one module at a time.

Sources & further reading: Anthropic Messages API · OpenAI API reference · pgvector · BullMQ