Problem
Used-car pricing is opaque: a number appears and nobody can say why. The model itself (a RandomForest on about 6,900 listings, 12 features) is the easy part. The project is everything a real ML service needs and demos skip: authentication, caching, batch jobs, monitoring, drift detection, and a retraining loop that can’t quietly ship a worse model.
Constraints
- Explanations must be in the user’s terms. The pipeline one-hot-encodes five categoricals into about 50 columns; raw SHAP explains
cat__company_Maruti, which is meaningless to someone who sent 12 fields. - The cache must never affect availability. The inherited prototype crashed at import when Redis was absent and
eval()’d cached strings, which is an outage waiting to happen with an RCE-shaped hole in it. - Automated retraining is only safe if promotion requires real, human-reviewable improvement.
System
A FastAPI service with 11 JSON endpoints, two demo pages and /metrics. Auth is either bcrypt accounts with JWT access/refresh rotation, or an X-API-Key path for service-to-service calls, with per-credential rate limiting. Predictions are cached in Redis under pred:v{model}:{sha256(sorted inputs)}, so deploying a new model version implicitly invalidates stale entries. Batch scoring (JSON or CSV, 10k-row cap) runs on an RQ worker with per-row validation errors and pollable job status.
Input drift is computed as per-feature Population Stability Index over the last 500 served predictions against training-time baselines, exported as Prometheus gauges into a provisioned six-panel Grafana dashboard. Five containers in Compose: API, worker, Redis, Prometheus, Grafana.
Decisions
Aggregate SHAP back to the 12 inputs, and prove it’s lossless. Contributions are summed per source feature using longest-prefix matching against the model card’s recorded column lists, so seller_type_Trustmark Dealer can’t mis-attribute. A CI test asserts baseline plus all 12 contributions equals the prediction to under ₹1.
Fail-open everything except the queue. Every cache operation is wrapped so a Redis outage degrades to slower-but-correct responses, the rate limiter falls back to memory, and /health reports cache state honestly. Batch queueing 503s by design, because a queue can’t fail open.
A promotion gate that rejects ties. A GitHub Actions workflow trains a challenger monthly; it must strictly beat the champion’s R² with MAE regression capped at 2%, and winners arrive as a pull request with the artifact and metrics. The first live trigger retrained on the same data with a fixed seed, produced identical metrics, and was rejected. Ties are churn, not improvement, and the workflow exited green with the champion intact.
Test the deployment artifact, not just the code. All 33 tests were green and the image built clean, and the first full docker compose up still crashed: the non-root user couldn’t write a root-owned volume mount. A one-commit fix, and a habit that made the cloud deploy work first try.
Outcome
R² 0.9395, MAE ₹70,128, RMSE ₹115,221 on a 1,386-row held-out set (33% RMSE reduction over the original notebook). Load test at 20 concurrent users: 866 requests, 0 failures, cache hits cutting median latency from 730 ms to 380 ms and p95 from 1,200 to 760 ms, measured on a single uvicorn process on a laptop, which is context, not a capacity claim. CI (lint, tests, Docker build) in about 54 seconds. Deployed to the cloud with cache hits verified against a hosted Redis over TLS.
Dataset is a public Indian-market listing set, prices in INR. No real traffic beyond demo verification, and the site doesn’t claim any.