Chunking Is Schema Design

You have made this decision before under a different name. Splitting line_items out of orders instead of stuffing a JSON array into a column was a choice of grain: the smallest thing a query can return, filter, and index. Chunking is that decision for retrieval.

Embeddings gave you an index over text you own. The retriever can only hand the model units you chose to store, so the grain is a ceiling nothing downstream can raise.

The chunk is the grain of your index

A chunk is a row, and retrieval can only ever return rows. The model never sees your document, only the k units the retriever selected. That makes the chunk the unit of four separate jobs: the text you embed, the row you filter by permission, the passage you cite back, and the thing you re-embed when the source changes.

Grain fails in the same two directions it fails in a database. Too coarse, and every hit drags in irrelevant text you pay for and the model must ignore. Too fine, and the top-k is a pile of fragments that each mention the subject while none states the fact.

Size, overlap, and the seams already in the document

Size trades interpretability against precision, and no single number wins both. A 40-token chunk reading "set the interval to 90 days" has lost its subject: the vector is about intervals, not API keys, so a query naming API keys ranks it below a weaker passage that says the words. A 2,000-token chunk covering five topics averages five directions into one vector.

Provider limits are not the constraint. OpenAI's embeddings endpoint accepts 8,192 tokens per input and Voyage's voyage-4 models take 32,000, so a 2,000-token chunk is legal everywhere and still a bad row.

For prose documentation, start at 250 to 500 tokens with roughly 15% overlap and treat that as a hypothesis: change one variable, rerun recall@k against a labeled question set, keep what wins. Retrieval quality covers that machinery. Overlap exists for one reason: a sentence straddling a boundary would otherwise appear in neither chunk with its subject attached.

Split on the boundaries the document already has before you count anything. Headings, list items, table rows and function definitions are authored groupings, and a character counter discards a decision a human already made. Use a token budget only inside an oversized section.

type Chunk = { text: string; sectionPath: string[] };
 
// Structure first: the document's own headings, then a budget for leftovers.
function chunkMarkdown(
  doc: string,
  countTokens: (s: string) => number,
  max = 500,
): Chunk[] {
  const out: Chunk[] = [];
  for (const section of splitOnHeadings(doc)) {   // -> { path, body }
    if (countTokens(section.body) <= max) {
      out.push({ text: section.body, sectionPath: section.path });
      continue;
    }
    for (const window of packParagraphs(section.body, countTokens, max)) {
      out.push({ text: window, sectionPath: section.path });  // path survives
    }
  }
  return out;
}

Carry that heading path as metadata and prepend it to the text you embed, so Billing > Invoices > Failed payments travels with the chunk. Anthropic's contextual retrieval goes further: an LLM writes a chunk-specific blurb, prepended before embedding and before lexical indexing, which cut top-20 retrieval failures from 5.7% to 2.9% on their internal set. Two variants skip the per-chunk LLM call, Voyage's contextualized chunk embeddings and late chunking.

Metadata are columns, and one of them is a security boundary

A chunk that does not carry who may see it cannot be filtered, and an unfilterable chunk is a disclosure waiting for a plausible query. Vector search returns nearest neighbors, not authorized ones. Similarity has no opinion about tenancy, so tenant_id (plus whatever else scopes access) belongs on the chunk row, filterable in the same query that scans it.

The other columns earn their place the same way:

• document_id: which document this came from, so retracting it is one DELETE.
• ordinal: position in the document, which is how you fetch neighbors.
• section_path: the heading path, which doubles as the citation you show.
• updated_at: staleness, so you can date a confident answer.
• content_hash: a digest of the chunk text, so unchanged text is never re-embedded, which is what makes the ingestion pipeline idempotent.

CREATE TABLE doc_chunks (
  id               bigserial PRIMARY KEY,
  tenant_id        uuid         NOT NULL,
  document_id      uuid         NOT NULL,
  ordinal          int          NOT NULL,
  section_path     text[]       NOT NULL,
  content_hash     bytea        NOT NULL,
  embedding_model  text         NOT NULL,
  updated_at       timestamptz  NOT NULL DEFAULT now(),
  content          text         NOT NULL,
  embedding        vector(1536) NOT NULL,
  UNIQUE (document_id, ordinal, embedding_model)
);

The vector(1536) width is text-embedding-3-small's output size, so the column type is itself a model commitment, and embedding_model records which commitment wrote the row. pgvector also indexes a vector only to 2,000 dimensions, so a 3072-dimension model needs halfvec, not a wider column.

Embed the small chunk, return the big one

Matching and answering want opposite sizes, so serve both instead of compromising. Matching wants a narrow vector about one idea; answering wants enough surrounding text to stand alone. Parent-document retrieval, also called small-to-big, gets both: start at 150 to 300 token chunks, and when one is a hit, return the section containing it.

Two details decide whether this works. Deduplicate after expansion, because three hits in one section should send it once, not three times. And cap the parent: a whole 12,000-token document reintroduces the dilution you avoided, so expand to the section or to n neighbors by ordinal.

Tables and code break naive splitters

Two document types turn character-count splitting into a machine for writing useless rows, and both are common in the corpora backend teams own: runbooks full of tables, and source repositories.

Code has a second problem no chunk size fixes: people search a repository for exact identifiers, and identifier matching is lexical, not semantic, exactly where embeddings are weakest. Hybrid search is the first fix.

Chunking is a migration, not a setting

Changing the chunk rule invalidates every row you have already written. A new size, split boundary, or prepended context changes the text that was embedded, so the corpus gets re-chunked and re-embedded, not patched. Treat it as a backfill with a cutover, not an edit to a config value. Keeping embeddings in sync has the mechanics.

Decide it the way you decide a schema: write down the grain, the split rule, the columns, and why. The pipeline that writes these rows is the next lesson.

Key takeaways

• A chunk is a row and retrieval only returns rows, so the grain is a ceiling on quality; changing it later means re-embedding the corpus.
• Start prose at 300 to 500 tokens with about 15% overlap, then settle it with recall@k over labeled questions.
• Provider limits are not the constraint: a 2,000-token chunk is legal everywhere and still dilutes its vector.
• Split on authored boundaries first, use a token budget only inside an oversized section, and carry the heading path with the chunk.
• Metadata are columns: tenant_id makes permission filtering part of the search, content_hash makes reingest idempotent, section_path is the citation.
• Embed the small chunk and return its parent, deduplicated and capped. Two grains cost one join.