Problem
“Chat with your PDFs” is the most-built demo in the field, and most versions share the same failure: they answer confidently from the wrong chunk, and the user has no way to check. The goal here was the version that survives contact with a skeptical reader. Every claim points at a page, retrieval quality is visible while it happens, and the system says “the documents don’t support that” when they don’t.
Constraints
- Multi-user from the start, with per-user data isolation, because a single shared index is a demo, not a product.
- Answers must stream, with sources attached, without the UI freezing while retrieval runs.
- Evaluation without labeled ground truth, because real users don’t come with an answer key.
- Two LLM providers (Groq and Gemini), with the model list maintained in one place, because hosted providers retire model IDs every few months.
System
Ingestion parses PDFs and offers four chunking strategies, including parent-child: retrieve precise child chunks, feed the LLM the fuller parent context. The chat pipeline runs rewrite query → multi-query retrieval in parallel → dedupe by content hash → hybrid BM25 + dense ensemble → cross-encoder rerank → grounding check → stream. Each stage emits an NDJSON status event, which is what drives the live “rewriting → retrieving → reranking → grounding → generating” indicator in the UI.
The React frontend renders the PDF beside the chat; clicking a citation scrolls to and highlights the cited page. Conversations persist with automatic summarization of older turns. Each session gets its own ChromaDB collection; metadata lives in SQLModel/SQLite with JWT-scoped access.
Decisions
Hybrid retrieval over dense-only. BM25 catches the exact-term queries (part numbers, clause references, names) that embeddings blur; dense search catches paraphrase. The ensemble covers both failure modes, then the cross-encoder reranks the union so the LLM sees the best of each.
A grounding check as a hard step, not a prompt suggestion. After reranking, a YES/NO check asks whether the context can support an answer at all. A NO short-circuits into an explicit “can’t answer from these documents” event rather than a fluent guess.
Reference-free evaluation. Faithfulness, answer relevancy, and context relevance are scored by an LLM judge that needs no labels, so it works on live conversations at a cost of a few extra calls. The tradeoff is real (a judge can be wrong), and the roadmap names a labeled benchmark as the complement.
Streaming over NDJSON rather than SSE. Mixed event types (status, tokens, sources, grounding failure, summarization notices) fit a line-delimited JSON stream more naturally than a text event stream.
Outcome
Shipped as a Docker Compose stack (FastAPI + nginx-served React), with rate-limited auth endpoints, structured logging, health and metrics endpoints, and a one-click demo mode that loads sample research, financial, and legal PDFs.
Honest roadmap: persisted per-message citations so reloaded conversations keep their highlights, Alembic-managed migrations, and hosted vector storage plus Postgres for horizontal scaling.