LLM Token Gateway + AI FinOps

Context

Companies adopted LLMs across dozens of use cases, and lost control of the spend. Without observability, nobody knows which team consumes the most tokens, which model has the best cost/quality trade-off, or when a bug started silently burning budget. The invoice arrives at the end of the month and the postmortem starts there, which is the worst possible place to start one.

This project implements the answer: a gateway that intercepts every call, and a lakehouse that turns those calls into cost intelligence.

It is also the flagship of this portfolio, for a reason worth stating plainly: most data portfolios use AI, and this one measures and governs it. It is a self-contained, synthetic-data build: no real prior infrastructure, no real Anthropic/OpenAI API keys, no real AWS account. Every number in the Results section below is reproducible from a clean clone with zero credentials, and every part that is easy to get wrong — modeling cost as of the time it was incurred, versioning price history, choosing a threshold that alerts without crying wolf — was solved for real inside this build, not just described.

Business Problem

How do we get visibility and control over LLM API consumption, cost per team, per model, per use case, with anomaly detection before the invoice arrives?

"Before the invoice arrives" is the requirement that shapes everything else. A monthly bill tells you what happened; it does not tell you which team, which model, or which deploy caused it. Attribution has to be captured at the moment of the call or it cannot be reconstructed later.

Architecture

Request path

A transparent proxy rather than a custom SDK: applications change exactly one thing, the base URL. The gateway exposes an OpenAI-chat-completions-shaped endpoint, forwards the request to the configured LLM backend, and returns that backend's response untouched, capturing the metadata it needs on the way through: model, tokens in and out, latency measured at the gateway, status, and the calling team resolved from the gateway's own API key. By default the backend is a deterministic, seeded mock built into the repo — zero network calls, zero API keys; a real Anthropic/OpenAI adapter is an optional, documented extension point (GATEWAY_LLM_ADAPTER), never required to reproduce this project.

Data path: medallion lakehouse

• Bronze: raw, append-only log of every request, one JSON line per request, partitioned by date (dt=YYYY-MM-DD/events.jsonl): timestamp, model, tokens in/out, latency, team, use case, status. Local filesystem in this build; AWS S3 Parquet is the documented, not-executed, prod target.
• Silver: bronze parsed and typed, then priced using the price in effect at request time via a versioned per-model dbt seed — an ASOF join, not today's price.
• Gold (dbt marts): fct_llm_requests, cost_per_team_daily, model_efficiency, spend_anomalies.
• Consumption: a FinOps dashboard built directly from the gold marts, plus spend anomaly alerts.

A local NDJSON lake instead of a plain relational database: the raw log is cheap, append-only, human-auditable, and DuckDB's read_json_auto glob reads every partition without a compaction step. The trustworthy marts sit on top of it rather than replacing it — if the cost model turns out to be wrong, bronze is still the truth and silver can be rebuilt.

Data Source

There is no public dataset here. The system produces its own data. Every call that passes through the gateway becomes one usage event, which is precisely why attribution works: the team, the model and the token counts are recorded at the moment the cost is incurred, not inferred afterward from an aggregate bill. A synthetic-traffic generator exercises the same in-process capture pipeline at volume: 5 teams × 5 models over 42 days, 12,934 events, deterministic given a seed, with one deliberately injected spend anomaly.

• Events: timestamp, model, tokens in/out, latency, team, use case, status; one row per request.
• LLM backend: a deterministic, seeded mock built into the repo by default — zero real Anthropic/OpenAI calls or API keys needed to reproduce this project. A real provider adapter is an optional, documented extension point (GATEWAY_LLM_ADAPTER).
• Caller identity: resolved from the gateway's own API key, mapped to a team/project, not the provider's.
• Price reference: a per-model price table maintained as a versioned dbt seed, joined via an ASOF join to the price in effect at request time.
• Storage: local filesystem, NDJSON, append-only, partitioned by date (dev, executed). AWS S3 Parquet is the documented prod target, never executed here (no AWS credentials in this environment).

Methodology

Medallion architecture (bronze → silver → gold) applied to a FinOps use case:

1. Gateway (FastAPI): an OpenAI-chat-completions-shaped endpoint that forwards the request to the configured backend (a deterministic mock by default) and captures response metadata: usage, latency, model.
2. Caller identity via the gateway's own API key, mapped to team/project.
3. Bronze ingestion: each event appended to the local NDJSON lake, whether the call succeeds, errors, or times out (S3 is the documented, not-executed, prod target).
4. Silver/Gold with dbt: point-in-time cost calculation against the versioned price seed, aggregations, p95 latency per model.
5. Consumption & alerting: dashboard plus spend anomaly detection: daily spend beyond 3 standard deviations from each team's own trailing 7-day average, once at least 7 days of history exist.

FinOps marts

• fct_llm_requests: grain = 1 request (tokens, cost, latency, team, model, use case).
• cost_per_team_daily: budget tracking.
• model_efficiency: average cost per 1k tokens vs. p95 latency; the evidence you need to justify switching models.
• spend_anomalies: days where a team's spend clears 3 standard deviations above its own trailing 7-day average, once at least 7 days of history exist to judge against.

Latency is measured at the gateway, deliberately. That captures the real experience of the calling application, network included, rather than only what the provider reports about its own processing.

AI and responsible use

AI is the subject of this project, used with governance rather than hype. The gateway makes AI usage observable and accountable: who spends what, on which model, for what use case. The use_case tags in this build come from the synthetic generator's own weighted assignment, not an LLM classifier — there is no "validated precision" claim to make here, because no classification happened. If a real deployment added an LLM-based use-case tagger, this portfolio's rule still applies: its precision would need validating against a manually labelled sample before being trusted, exactly like any other AI-assisted classification. The design goal is to demonstrate good use of AI (cost governance, point-in-time pricing, anomaly detection with an honestly reported false-positive rate), not dependency on it.

Challenges

1. Logging without taxing the request path

The gateway sits in the hot path of every LLM call, which makes any latency it adds a tax paid by every application on every request. Writing to the lake synchronously would do exactly that: storage-write latency on top of an already slow call, to serve an analytics need the caller does not care about. The event is therefore captured in-process and written outside the response path, so the client waits for the backend and nothing else. The trade-off is explicit rather than hidden: a crash can lose a small window of events. That is acceptable for cost analytics and would not be acceptable for billing, and the difference is worth being honest about.

2. Never leaking API keys

A gateway is a natural place to concentrate secrets: in a real deployment it would hold provider keys (Anthropic, OpenAI), the gateway keys that identify each team, and AWS credentials for the lake. This build keeps that discipline even though it needs none of those secrets to run: .gitignore from commit #1 covers *.duckdb, the generated lake and .env; no secret exists in the git history; the demo gateway keys (sk-demo-*) are internal routing tokens for a mock backend, not secrets, because there is nothing behind them to protect. The subtler risk is the logging itself: a proxy that logs "the request" for debugging convenience is one careless line away from persisting an Authorization header into an append-only lake that is designed never to be mutated. Bronze stores usage metadata only (model, tokens, latency, team, status); credentials would be scrubbed before anything is written, not after, the moment a real adapter is plugged in.

3. Pricing history: the numbers move under you

Model prices change, and they mostly go down. A cost model that joins to a single current price table silently rewrites history every time a provider cuts prices: last quarter's spend report stops matching last quarter's invoice, and the dashboard quietly becomes a liar. Prices live in a versioned dbt seed, and silver joins on the price that was in effect at the request timestamp. Historical cost has to reflect the price of the time, or it is not a cost, just a recalculation.

4. Anomaly detection needs a baseline first

"Spend anomaly" is meaningless on day one: a spike is only a spike relative to something. The mart flags days beyond N deviations from a moving average, which means the detector is unreliable until enough history accumulates, and it is sensitive to the choice of N and window: too tight and it cries wolf until people mute the channel, too loose and the runaway retry loop goes unnoticed until the invoice. It also cannot distinguish a legitimate launch from a bug. It flags for review; it does not judge, and it should not be sold as if it does.

Results

From a clean clone, zero cloud credentials and zero LLM API keys required: uv sync --dev → .venv/bin/pytest tests/ -q → .venv/bin/python scripts/run_pipeline.py → .venv/bin/python dashboards/build_dashboard.py.

Every number above is read from a committed run artifact under docs/evidence/ in the repo, not from a console summary line — see docs/evidence/ for the full files.

Next steps: execute the documented prod path for real (an actual S3 bucket + IAM user, dbt build --target prod against it), plug in a real provider adapter behind GATEWAY_LLM_ADAPTER, rate limiting and per-team budgets enforced in the gateway (moving it from observer to controller), response caching for repeated prompts, and streaming (SSE) support with real-time token counting.

Tech Stack

View the code on GitHub ↗