This is a reference design for answering questions over internal documentation. It is not a claim about an employer system. I keep the architecture deliberately small: an API, an ingestion worker, PostgreSQL with pgvector, object storage for source documents, and one model provider behind an internal client.

Build ingestion as a real data pipeline.

For every document I store a stable document ID, source URL, owner, version, checksum, updated time, tenant, and access-control metadata. The worker normalizes supported formats, removes repeated navigation, splits text into chunks of roughly 350–500 tokens with a small overlap, creates embeddings in batches, and writes the new version in a transaction.

A checksum prevents unchanged documents from being re-embedded. Versioned writes prevent a failed ingestion from deleting the last good copy. When a document is removed or access changes, the pipeline must update or delete its chunks; freshness is part of retrieval quality.

Chunk size is not a magic constant. I start with a reasonable value, then tune it against real questions. Very small chunks lose context. Very large chunks dilute relevance and consume the model’s context window.

Filter permissions before ranking.

The query path authenticates the caller first. Retrieval must filter by tenant and allowed document IDs before results can reach the model. Filtering after generation is too late.

SELECT
  document_id,
  chunk_id,
  content,
  source_url,
  embedding <=> $3 AS distance
FROM document_chunks
WHERE tenant_id = $1
  AND acl_principals && $2
ORDER BY embedding <=> $3
LIMIT 20;

For a small corpus, vector search may be enough. For operational documentation, hybrid retrieval often helps because exact error codes, resource names, and commands matter. I retrieve a wider candidate set with vector and keyword search, combine the results, and optionally rerank the top candidates before sending a few useful chunks to the model.

Return evidence, including when it is missing.

@app.post("/v1/answers")
async def answer(request: Question, user=Depends(authenticate)):
    candidates = await retrieve(
        query=request.question,
        tenant=user.tenant,
        principals=user.principals,
        limit=20,
    )
    context = rerank(request.question, candidates)[:5]

    if not context or context[0].score < MIN_RELEVANCE:
        return {
            "answer": "I could not find enough evidence.",
            "sources": [],
        }

    result = await llm.generate(
        prompt=build_prompt(request.question, context),
        timeout_seconds=15,
    )
    return {
        "answer": result.text,
        "sources": unique_sources(context),
    }

The prompt tells the model to answer only from supplied context, cite source identifiers, and say when evidence is insufficient. That instruction improves behavior, but it is not a security boundary. Retrieved documents are untrusted data: text inside them must not override system policy or grant tool access.

I validate returned citations against retrieved chunk IDs instead of trusting invented URLs. For high-impact answers, the interface should make sources easy to open and keep a human in the decision path.

Evaluate the system, not just the model.

I keep a small versioned set of representative questions with expected source documents and cases that should be refused. On material changes I measure retrieval recall, citation precision, answer support, refusal quality, p95 latency, and estimated cost. A fluent answer with the wrong source is a failure.

Production telemetry includes request ID, retrieval duration, number of candidates, selected chunk IDs, model alias, input and output tokens, total latency, refusal reason, and error class. Raw questions and document text are not logged by default because they may contain private data.

Keep operations uneventful.

I run API replicas separately from ingestion workers so a large import cannot starve queries. Database migrations are explicit, ingestion jobs are idempotent, provider calls have bounded timeouts, and retries use backoff only for safe transient failures. The vector database is backed up, and restore testing includes both metadata and embeddings.

The result is still a small service, but it has the properties a demo usually skips: ownership, freshness, permission-aware retrieval, verifiable citations, honest refusal, evaluation, and observable cost. Those pieces do more for trust than adding another framework to the stack.