2026-07-28

Hybrid retrieval for codebase RAG — why vector search alone wasn't enough

I've been building KB, a knowledge base that indexes multiple Git repositories and answers questions with citations down to specific files and line ranges. It's open source on GitHub, and the architecture decisions are worth writing down — especially the retrieval layer, where my first approach failed in an instructive way.

The problem with vector-only retrieval on code

The naive RAG recipe — chunk everything, embed it, cosine-similarity search — works fine for prose. On codebases it falls over for one big reason: developers query with exact identifiers. When someone asks "where is ProcessShiftAcceptanceJob dispatched?", embedding similarity is the wrong tool. The embedding of an identifier is close to the embeddings of similar-looking identifiers, and the chunk that merely mentions your term can rank below chunks that are vaguely "about" the same topic. Semantic search excels at "how does auth work here?" and embarrasses itself on "find AuthenticationMiddleware".

Keyword search has the mirror-image problem: great at exact identifiers, useless when the question doesn't share vocabulary with the code.

So: use both.

The retrieval pipeline

Every query in KB runs through two searches in parallel, both in PostgreSQL:

  1. Vector search — chunks are embedded (OpenAI text-embedding-3-small) and stored in a pgvector column with an HNSW index; the query is embedded and ranked by cosine similarity.
  2. Full-text search — the same chunks have a tsvector column with a GIN index, queried via plainto_tsquery.

Each search returns a ranked list. They're merged with Reciprocal Rank Fusion: each chunk's score is the sum of 1 / (k + rank) across the lists it appears in. RRF's appeal is that it only uses ranks, never raw scores — so there's no need to normalise cosine similarity against FTS relevance, two numbers that live on incomparable scales. A chunk that ranks decently in both lists beats a chunk that tops one and misses the other, which matches intuition about what a "good" retrieval hit is.

After fusion, metadata filters (repo, path prefix, language, chunk type) narrow the result, and the top chunks go to the LLM.

Keeping both indexes in Postgres — rather than adding a dedicated vector store — was a deliberate boring-technology choice: one database, one backup story, transactional writes between chunks and their embeddings, and EXPLAIN works on everything.

Chunking is language-aware, lightly

Code, markdown and "everything else" get different chunkers. The goal isn't perfect AST-level splits — it's chunks that keep a function or section intact often enough that a citation to them is useful. (Tree-sitter-based symbol extraction is on the roadmap; so far the simple chunkers haven't been the bottleneck.)

Trust: citations and staleness

Two design rules make the answers verifiable rather than plausible:

A RAG system that quietly answers from last month's code is worse than no RAG system, because it's wrong with confidence.

Incremental indexing

Re-indexing a repo hashes every file (SHA-256) and skips unchanged ones — only new or modified files are re-chunked and re-embedded. Embedding calls are the dominant cost of indexing, so on a typical day-to-day re-index this turns "the whole repo" into "the handful of files that changed".

What I'd tell someone building the same thing

← all posts