RAG & Memory

A fan asks the assistant: "Can I bring my camera to the show at Arena Paulista, and is there step-free access to sector C?"

The answer exists — somewhere in 4,000 documents: venue rules (PDFs uploaded by each venue, in Portuguese, revised every season), the accessibility guide, the FAQ, the event's own description. The model has never seen them: they are private, and many changed after its training data was collected. Asked directly, it will produce a fluent, plausible and possibly wrong answer.

Three tempting fixes, and why they fall short:

RAG¹ turns a knowledge problem into a search problem — and search is a system design problem you already know how to measure.

The two pipelines

A RAG system is two pipelines that meet at an index: an ingestion pipeline (asynchronous, throughput-oriented) and a query pipeline (synchronous, latency-oriented).

flowchart LR
    subgraph Ingestion["Ingestion (async)"]
        src["venue PDFs, FAQ,<br/>event descriptions"] --> parse["parse + clean"]
        parse --> chunk["chunk + add context"]
        chunk --> embed["embed (batch)"]
    end
    embed --> index[("Vector DB +<br/>keyword index")]
    subgraph Query["Query (online)"]
        q["fan question"] --> rewrite["rewrite to a<br/>standalone query"]
        rewrite --> hybrid["hybrid search:<br/>vector + keyword"]
        hybrid --> rerank["rerank top 20 → 5"]
        rerank --> prompt["prompt with sources"]
        prompt --> llm["LLM answer<br/>with citations"]
    end
    index --> hybrid

Ingestion

Sources, parsing and freshness

Ingestion starts as an ordinary data pipeline: read documents from where they live (object store, CMS, the catalog database), extract clean text (PDF parsing is often the hardest part: tables, columns, headers repeated on every page), and attach metadata — doc_id, venue_id, language, version, effective date, access level. Metadata is what lets you filter (only this venue), cite (Arena Paulista rules, §4) and delete (the 2026 rules are obsolete).

The index must follow the sources. Rather than re-embedding everything nightly, propagate changes: the catalog service publishes an event when a description changes (outbox or change data capture, as in Messaging); an embedding worker consumes it and updates only the affected chunks. Deletions must propagate too — a removed rule that is still retrievable is a wrong answer waiting to happen.

Chunking

Documents are split into chunks, the unit of retrieval. Chunk size is a precision/context trade-off: small chunks match questions precisely but lose surrounding context; large chunks keep context but dilute the match and cost more tokens in the prompt.

Embeddings

An embedding model maps a text to a vector of a few hundred to a few thousand numbers, trained so that texts with similar meaning have vectors that point in similar directions. Similarity is usually measured by cosine similarity (or the inner product of normalized vectors):

Practical rules:

• Same model for documents and queries — vectors from different models live in different spaces and cannot be compared.
• Multilingual model for BoxOffice: fans ask in Portuguese, English and Spanish; some venue documents are only in Portuguese.
• Store the model name with each vector (embed_model). Changing the embedding model means re-embedding the whole corpus — run it like a blue/green deployment: build the new index alongside, switch reads, then drop the old one.
• Embedding is cheap compared to generation (one forward pass, no decoding), and the ingestion side can use batch APIs; the query side embeds one short question per request (a few milliseconds to tens of milliseconds).

How much memory does the index need? Raw vectors are \(N \times d \times\) bytes per value; an HNSW graph adds neighbor links per vector:

# Memory needed by a vector index: raw vectors + HNSW graph links.
# HNSW keeps ~2*m neighbor ids per vector on the base layer (m = 16 here);
# real overhead varies by implementation: treat as an estimate.
M, ID_BYTES = 16, 8


def size_gb(n, dims, bytes_per_value):
    vectors = n * dims * bytes_per_value
    graph = n * 2 * M * ID_BYTES
    return (vectors + graph) / 1e9


head = ["Chunks", "384 dims, float32", "1024 dims, float32", "1024 dims, float16",
        "3072 dims, float32"]
print("<table><thead><tr>" + "".join(f"<th>{h}</th>" for h in head) + "</tr></thead><tbody>")
for n in (100_000, 1_000_000, 10_000_000, 100_000_000):
    cells = [f"{n:,}"] + [f"{size_gb(n, d, b):,.1f} GB"
                          for d, b in ((384, 4), (1024, 4), (1024, 2), (3072, 4))]
    print("<tr>" + "".join(f'<td style="text-align:right">{c}</td>' for c in cells) + "</tr>")
print("</tbody></table>")

BoxOffice's 4,000 documents become ~60,000 chunks: well under 1 GB, comfortably inside the PostgreSQL the team already runs. Ten million chunks of 1,024 dimensions (~44 GB) is where dedicated vector databases, quantization (half-precision or binary vectors) and sharding start to pay off.

Vector search

Finding the exact nearest neighbors means comparing the query with every vector — fine for 60,000 chunks, too slow for 100 million. Approximate nearest neighbor (ANN) indexes trade a little recall for orders of magnitude of speed (see Databases — specialized):

The knob is the recall/latency trade-off: HNSW's ef_search (candidates explored per query) and IVF's probes raise recall at the cost of latency. Measure recall@k against exact search on a sample of real queries, as you would measure a cache's hit ratio.

Filters complicate ANN. "Rules for venue X" is a metadata filter applied to an approximate search: if the index returns the 40 nearest chunks and only 2 belong to venue X, the result has 2 rows instead of 40. Remedies: filter-aware indexes, partitioning by the filter key, or iterative scans that keep searching until enough rows pass (pgvector supports this from version 0.8 with hnsw.iterative_scan).

The query pipeline

1. Rewrite the question

"And what about sector C?" means nothing alone. Before searching, a cheap model call (or the agent itself) rewrites the turn into a standalone query using the conversation — "step-free access to sector C at Arena Paulista" — and extracts filters (venue_id). Some systems also generate several query variants or a hypothetical answer to search with.

2. Hybrid search

Dense vectors capture meaning, but they are weak with exact tokens: sector "C", gate numbers, event codes, artist names, acronyms such as PCD (pessoa com deficiência, person with disability). Keyword search with BM25⁴ is the opposite: exact, but blind to paraphrase ("step-free" vs "wheelchair accessible"). Hybrid search runs both and fuses the rankings. The standard fusion is Reciprocal Rank Fusion (RRF)⁵, which needs only ranks, not comparable scores:

A chunk ranked 1st by keywords and 3rd by vectors scores \(1/61 + 1/63 ≈ 0.0323\); one ranked 1st by vectors only scores \(1/61 ≈ 0.0164\). Agreement between the two retrievers wins.

3. Rerank

The first stage is fast but rough: each chunk was embedded without seeing the question. A reranker (a cross-encoder) reads the question and each candidate together and outputs a relevance score⁷ — much more accurate, much slower (one model pass per pair). So it runs only on the top 20–50 candidates and keeps the best 3–8. Typical cost: tens to a couple hundred milliseconds.

4. Assemble the context and cite

Put the selected chunks in the prompt with their IDs and titles, best first (models use the beginning and the end of a long context better than the middle⁶), within a token budget. Instruct the model to answer only from the sources, to cite them as [doc_id], and to say "I don't know" when the sources do not answer. Then verify: every cited ID must be one of the chunks provided (a cheap, deterministic check in the Safety Mesh).

The schema and the query in PostgreSQL

CREATE EXTENSION IF NOT EXISTS vector;

-- Knowledge base: venue rules, FAQ, accessibility guides (one row per chunk)
CREATE TABLE kb_chunk (
    id          bigserial PRIMARY KEY,
    doc_id      text         NOT NULL,     -- source document, for citations and deletes
    venue_id    text,                      -- NULL = applies to every venue
    title       text         NOT NULL,     -- "Arena Paulista — Visitor rules 2027 — §4"
    content     text         NOT NULL,
    embedding   vector(1024) NOT NULL,     -- dimension of the embedding model
    embed_model text         NOT NULL,     -- which model produced the vector
    tsv         tsvector GENERATED ALWAYS AS (to_tsvector('portuguese', content)) STORED,
    updated_at  timestamptz  NOT NULL DEFAULT now()
);

CREATE INDEX kb_chunk_embedding ON kb_chunk USING hnsw (embedding vector_cosine_ops);
CREATE INDEX kb_chunk_tsv       ON kb_chunk USING gin (tsv);
CREATE INDEX kb_chunk_venue     ON kb_chunk (venue_id);

-- $1 = query embedding, $2 = query text, $3 = venue id
WITH q AS (  -- the words of the question, joined with OR
    SELECT to_tsquery('portuguese', array_to_string(
               tsvector_to_array(to_tsvector('portuguese', $2)), ' | ')) AS query
),
semantic AS (
    SELECT id, row_number() OVER (ORDER BY dist) AS rank
    FROM (SELECT id, embedding <=> $1 AS dist          -- cosine distance (HNSW index)
          FROM kb_chunk
          WHERE venue_id = $3 OR venue_id IS NULL
          ORDER BY dist
          LIMIT 40) AS nearest
),
keyword AS (
    SELECT id, row_number() OVER (ORDER BY score DESC) AS rank
    FROM (SELECT id, ts_rank_cd(tsv, q.query) AS score -- full-text match (GIN index)
          FROM kb_chunk, q
          WHERE tsv @@ q.query AND (venue_id = $3 OR venue_id IS NULL)
          ORDER BY score DESC
          LIMIT 40) AS matches
)
SELECT c.id, c.doc_id, c.title, c.content,
       coalesce(1.0 / (60 + s.rank), 0) + coalesce(1.0 / (60 + k.rank), 0) AS rrf
FROM semantic s
FULL OUTER JOIN keyword k USING (id)
JOIN kb_chunk c USING (id)
ORDER BY rrf DESC
LIMIT 20;  -- candidates for the reranker, which keeps the best 5

PostgreSQL's ts_rank_cd is not exactly BM25, but for fusion only the rank order matters; engines such as OpenSearch provide BM25 natively.

CREATE TABLE IF NOT EXISTS assistant.documents (
    id          TEXT PRIMARY KEY,                       -- e.g. "event:123"
    kind        TEXT NOT NULL,                          -- "event" (add "faq", "policy" ... later)
    title       TEXT NOT NULL,
    content     TEXT NOT NULL,                          -- the text that was embedded
    embedding   vector({settings.embed_dim}) NOT NULL,
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- approximate nearest neighbours (HNSW) on cosine distance, the metric of the embedding model
CREATE INDEX IF NOT EXISTS documents_embedding_hnsw
    ON assistant.documents USING hnsw (embedding vector_cosine_ops);

async def search(query: str, k: int) -> list[Hit]:
    [vector] = await embed([QUERY_PREFIX + query])
    async with pool.connection() as conn:
        rows = await (await conn.execute(
            """
            SELECT id, title, content, 1 - (embedding <=> %s) AS score
            FROM assistant.documents
            ORDER BY embedding <=> %s
            LIMIT %s
            """,
            (np.array(vector, dtype=np.float32), np.array(vector, dtype=np.float32), k),
        )).fetchall()
    return [Hit(id=r[0], title=r[1], content=r[2], score=float(r[3])) for r in rows]

Latency budget of one retrieval

Retrieval itself is cheap; the extra tokens it adds to every model call are what you pay for. Retrieve fewer, better chunks.

Measuring retrieval

Evaluate the two halves separately (more in Safety, Evals & Observability):

• Retrieval: for a set of questions with known relevant chunks, recall@k (did the right chunk make the top k?), precision@k, and MRR (mean reciprocal rank: how high the first relevant chunk appears).
• Generation: faithfulness/groundedness (is every claim supported by the retrieved sources?) and answer relevance; frameworks such as RAGAS⁸ automate these with model-based judges.

If the answer is wrong, these metrics tell you where: retrieval missed the document (fix chunking, hybrid, filters) or the model ignored it (fix the prompt, the order, the model).

def labeled(limit: int) -> list[tuple[str, set[str]]]:
    events = [event(g) for g in range(1, limit + 1)]
    cases = []
    for g in [7, 42, 118, 199, 256]:  # exact lookups: one right answer
        if g <= limit:
            e = event(g)
            cases.append((f"When is {e['kind']} #{g}?", {f"event:{g}"}))
    for kind, city in [("Festival", "Recife"), ("Play", "Curitiba"), ("Match", "Salvador"),
                       ("Concert", "Manaus"), ("Festival", "São Paulo")]:  # open questions: many right answers
        right = {f"event:{e['id']}" for e in events if e["kind"] == kind and e["city"] == city}
        if right:
            cases.append((f"Is there any {WORDS[kind]} in {city}?", right))
    cases.append(("the big festival night at the stadium in São Paulo", {"event:1"}))
    return cases
    hits, rr = 0, 0.0
    print("| question | relevant | first relevant rank |\n|---|---|---|")
    for question, right in labeled(args.limit):
        found = httpx.post(f"{args.url}/assistant/search", json={"query": question, "k": args.k}, timeout=60).json()
        ranks = [i + 1 for i, h in enumerate(found) if h["id"] in right]
        rank = ranks[0] if ranks else None
        hits += rank is not None
        rr += 1 / rank if rank else 0
        print(f"| {question} | {len(right)} | {rank or '-'} |")
    n = len(labeled(args.limit))
    print(f"\nhit@{args.k} = {hits / n:.2f}   MRR = {rr / n:.2f}   ({n} questions)")

Variants you will meet

Memory

RAG answers "what does the world know?". Memory answers "what does the assistant know about this conversation and this fan?". Both end up as tokens in the prompt, but they have different lifetimes, owners and risks. A useful vocabulary comes from cognitive architectures for language agents⁹:

flowchart LR
    orch["Orchestrator"] -->|"1. load session"| redis[("Cache<br/>session, TTL 2 h")]
    orch -->|"2. load profile"| sql[("SQL<br/>preferences")]
    orch -->|"3. recall episodes"| epi[("Vector DB<br/>past conversations")]
    orch -->|"4. retrieve knowledge"| kb[("Vector DB<br/>knowledge base")]
    orch -->|"5. after the turn: append"| redis
    orch -.->|"6. async: extract memories"| worker["memory worker"]
    worker -.-> sql
    worker -.-> epi

Writing memories

Reading memory is retrieval; writing it is where the risks are:

• Explicit memories ("remember that I use a wheelchair") are reliable but rare. Implicit extraction — a worker asks a model to extract facts from finished conversations — is convenient and error-prone: a joke becomes a "fact", a wrong inference persists forever.
• Memory poisoning: content injected into a conversation (or a document) can be saved as a fact and influence every future session. Extract only from the user's own messages, never from tool results or retrieved documents, and keep the source of each memory.
• Short-term memory overflows the context in long sessions: keep the last turns verbatim and compact older ones into a running summary.
• Consistency: memories written asynchronously are not visible in the next turn — read-your-writes again (Distributed consistency). Write the session synchronously; extract long-term memories asynchronously.

Privacy is a design constraint

Memory makes the assistant a store of personal data. Under the LGPD, health information such as a disability is sensitive personal data, requiring specific consent. The design implications are concrete:

• Show the fan what is remembered, and let them edit and delete it.
• Deletion must reach every copy: the profile row, the episodic vectors, the session cache, the semantic cache, the logs and traces that captured prompts. Design the doc_id/user_id keys so a delete is one query per store, not an archaeology project.
• Apply retention (TTL) by default; keep memories per tenant and per user, and filter retrieval by them — never search one user's episodes for another.

How it connects

flowchart LR
    SRC["venue PDFs, FAQ,<br/>event descriptions"] --> ING["ingestion: parse,<br/>chunk, embed (batch)"]
    CDC["change events<br/>→ Messaging outbox / CDC"] --> ING
    ING --> IDX["vector + keyword index<br/>→ Databases specialized"]
    Q["fan question"] --> HY["rewrite, hybrid search,<br/>RRF, rerank"]
    IDX --> HY
    HY --> P["prompt with citations"]
    MEM["memory types: session,<br/>profile, episodic"] --> P
    MEM --> RYW["read-your-writes<br/>→ Distributed consistency"]
    MEM --> DEL["deletion reaches<br/>every copy (LGPD)"]
    P --> GUARD["retrieved text is untrusted<br/>→ Safety mesh"]

Palette components: memory (the whole fabric) · vectordb (knowledge base and episodic memories) · cache (session memory with a TTL) · sql (profile and preferences; pgvector lives here too) · objstore (the source documents) · kafka (change events that keep the index fresh) · worker (ingestion and memory extraction off the request path). Scenarios: ai-assistant. Continues in: Databases — specialized (vector search), Messaging — patterns (outbox, CDC), Distributed — consistency, Agents & Tools (live data goes through tools, not RAG), Safety, Evals & Observability.