Problem
Every team that ships an LLM feature eventually wants the same four things: lower cost, some protection against injection and data leakage, validated outputs, and survival when a provider has a bad day. Bolting each one into application code means four changes across every call site. A gateway means zero.
System
The proxy speaks the OpenAI wire format exactly: /v1/chat/completions and /v1/models, OpenAI’s error envelope, and request fields the gateway has never heard of are forwarded rather than rejected, so an SDK upgrade doesn’t break you. Everything it adds is either a response field an SDK ignores (x_gateway), an optional gateway request object, or an X-Gateway-* header.
Inside: a two-tier cache (exact and semantic), a complexity classifier that routes simple requests to cheaper models, an injection screen and PII redactor on the way in, schema validation on the way out, and failover across providers.
Worth knowing
The honesty principle. It’s trivial to “save money” by quietly serving worse answers. Every cost number the project reports is paired with the thing that makes it defensible: the quality measurement on the same requests. Routing to a cheaper model without measuring what you lost isn’t optimization, it’s hoping.
The drop-in property is the entire point of the design. If adopting the gateway required touching call sites, it would compete with every wrapper library instead of sitting underneath all of them.