Problem
Ask a general LLM about the PSX and you get confident answers assembled from whatever it half-remembers. In finance a wrong price or a misread policy rate isn’t a cosmetic error. The goal was an analyst that only says things it can point to: documents, time-series, or today’s exchange data, with the confidence stated honestly.
Constraints
- The document corpus is historical, parts of it pre-2010, so “what is the policy rate?” answered from documents alone would be years stale with no signal that it is.
- Yahoo Finance was the wrong source for PSX: no KSE-100 feed under any ticker, no
ENGRO.KA, and a lag of over a week. - Free-tier infrastructure everywhere, which means providers fail quietly (a search API returning 402 when a quota runs out) and vector stores get reclaimed if idle.
- Tickers like
NPL,CASHandNEXTare also ordinary finance words.
System
A query router categorizes intent with signals applied strongest-first: off-topic guard, definitional phrasing, ticker or company name, prediction intent, concept keywords, and only then a vote over retrieved chunks’ sector metadata. Compound questions go through a tree-of-thought split into 2–4 sub-questions searched concurrently; atomic questions skip that call entirely.
Three retrieval agents run in parallel: news and documents (about 37k embedded chunks in Qdrant), macro time-series (World Bank and FX data in Supabase), and PSX stocks (650k+ rows of end-of-day prices, synced daily from the exchange’s own portal by a cron endpoint). Live web grounding prefetches current results through a three-provider ladder (Tavily, then Jina, then Gemini’s search tool as a last resort) and injects them as citable [W1] context. A synthesis agent combines everything through a decision matrix that scores confidence by source agreement, and the React frontend renders the answer, the trace, a price chart, and clickable source badges over a token stream.
Every real query is rule-scored and persisted, so an evaluation page shows pass rate, latency and per-category quality in real time.
Decisions
Evidence in the question outranks evidence about the corpus. The sector vote used to run third and decided most routes, filing questions under whatever sector their nearest documents happened to carry. Moving it last took wrong-route from 32% to 0%.
Three grounding providers, because one was a single point of failure. When Jina’s free tier ran out, every search returned 402 and answers silently reverted to the stale corpus. Now /api/health reports which provider last served, so a silent outage can’t recur unnoticed, and a slow fetch yields an ungrounded answer rather than a late one.
Store NULL, not the close. Older backfilled history has no open/high/low. Filling those with the closing price would assert a session that never moved. Period highs and lows fall back to closing extremes and are labeled “closing basis” so the two are never confused.
A benchmark that can tell “stale expectations” from “regression.” Category H checks that specific figures were retrieved; those figures move. They’re regenerated from the live stores, and a test fails with the exact diff when they drift.
A benchmark that discriminates. The previous suite showed ablations scoring higher with an agent removed. Two changes fixed it: asking for figures held in exactly one source, and matching relevance numerically so “21” isn’t satisfied by “2021” appearing in the text. The deltas can now be quoted; the old ones couldn’t.
Outcome
On the 60-query comparative benchmark: the full pipeline beats raw Gemini by 36 points and 28 percentage points of pass rate, which is the comparison that justifies the architecture. Every ablation costs 3.4 to 4.0 points; removing the documents agent makes 14 of 60 queries worse, with losses up to 40 points on questions like which cement companies export most. Median latency 18.9 s (the report is explicit that means are polluted by a laptop suspending mid-run, and says so).
Not covered, and the endpoints say so: intraday prices and PSX announcements.