AIPlat — AI Gateway + FinOps

One endpoint in front of every LLM provider, with the cost ledger built in. Point your application at AIPlat instead of at OpenAI, Anthropic or Amazon Bedrock, and you get routing, caching, budget caps and guardrails in one place — plus a per-request record of what everything cost and who spent it.

100% serverless: Go on AWS Lambda (ARM64), API Gateway, DynamoDB, SQS, EventBridge, Cognito and Amazon Bedrock. Deployed per domain with Terraform, each domain holding its own state.

Current release: v1.3.0 — see the CHANGELOG. It closes a credential-exposure issue (a member with no write permission could make the platform hand over the org's stored provider API key — see the Security entry) and enforces max_tokens/temperature, which were previously parsed but silently dropped before the provider call: read the Upgrade notes before deploying, since a caller relying on that being ignored will now see it honoured.

Warning

Not for production use. This repository is a reference implementation of an AI gateway and cost-governance control plane. It is published for experimentation, evaluation and learning. Do not deploy it to production without an independent security review, infrastructure hardening, and testing appropriate to your workload and compliance requirements. The IAM policies, Terraform defaults and envs/poc deployment configuration here are minimal working examples — not production-ready baselines. See Disclaimer.

Disclaimer

This is a sample, published for reference and experimentation. It is not an AWS service and it is not supported by AWS. Using it is at your own risk. Read the code and the Terraform before you put anything you care about behind it, and treat the defaults as a starting point rather than a hardened configuration.

Specifically, before any production use:

• domains/*/envs/poc is a proof of concept. It is a single environment with a local Terraform backend, sized for iteration rather than operation. Production needs remote state with locking, separated environments, and your own review of every IAM policy in the modules.
• You own the cost. Lambda, API Gateway, DynamoDB, CloudFront and — usually the largest line — the model spend on Amazon Bedrock or whichever provider you configure are billed to your account. Set the cap under Limits & Budget before you hand the gateway to callers.
• You own the security review. Security posture lists what the Terraform configures. That is a description, not a compliance claim, and it has not been assessed against any framework.
• No warranty of any kind. See LICENSE.

Known limitations

• A single request is capped at 5 minutes, and that cap is a deliberate choice rather than a platform limit. The gateway is fronted by a Regional API Gateway REST API with response streaming enabled, which is not bound by the Maximum integration timeout quota (L-E5AE38E3, 29000 ms by default) at all and reaches 900000 ms. integration_timeout_ms ships at 300000 because the same value sizes the router Lambda's timeout: a provider that accepts a connection and then stalls is billed, and holds a concurrency slot, until it expires. Raise it if you have a workload that needs more; 5 minutes is roughly 30k output tokens at the rate measured here. Independently of it, API Gateway cuts a streaming response after 5 minutes of idle time — token-by-token generation never triggers that, a long silent computation does. Turning streaming off (response_streaming = false) puts you back under the quota and requires integration_timeout_ms = 29000 unless you have raised L-E5AE38E3; the provider rejects the mismatched pair at plan time. HTTP APIs, by contrast, are fixed at 30 s and not adjustable.
• Streaming works, but it needs a Runtime API loop this repo owns, because the Go library does not have one. "stream": true returns text/event-stream frame by frame. aws-lambda-go does not implement the Runtime API side of response streaming — its response POST never sets Lambda-Runtime-Function-Response-Mode: streaming, checked by reading the source through v1.55.0 — and events.APIGatewayProxyStreamingResponse is a format helper (metadata JSON + the 8-null-byte delimiter), not a transport. So cmd/router runs its own invocation loop (internal/awslambda/runtimeapi.go, ~150 lines, one handler shape) when streaming is enabled, and drops back to lambda.Start when it is not. If the header lands upstream, that file gets deleted and AdaptStream becomes the wiring. All four provider adapters stream natively, so first-token time is genuinely lower than last-token time (measured: 2.9 s of a 7.4 s response); see the provider table in domains/core/README.md for the per-dialect details.
• First sign-in with MFA required. The console does not implement the Cognito MFA_SETUP challenge. With the user pool at the default mfa_configuration = "ON" and a user who has not yet enrolled a TOTP factor, sign-in fails instead of guiding enrolment. Deploy with mfa_configuration = "OPTIONAL" for the first sign-in, enrol from Settings → 2FA, then move the pool back to "ON".
• Cost attribution is per request, not a billing reconciliation. Figures come from the prices you declare per model, so they track your provider invoice only as closely as those prices do. Until you enter a negotiated price the console uses the provider list price and says so.
• A discovered Bedrock model arrives without a price, and stays out of automatic routing until you give it one. ListFoundationModels and ListInferenceProfiles return which models exist, never what they cost, so the console ships a small curated price/capability table and merges the live list into it. Anything outside that table is listed and usable, but flagged no price and excluded from auto-cheapest — a model priced at 0 would otherwise win every comparison. Enter the price in the route list to bring it in. Note also that Bedrock has two id spaces: most current models are reachable only through a cross-region inference profile (us.…, global.…), so those ids are what the list returns and what a route must hold; the bare foundation-model id returns 404 from Converse for those models.
• Savings are reported in two categories on purpose. Only the verified ones are free of assumption — see How savings are counted before quoting a number.

Why

Once more than one team is calling models, three problems show up together and none of them are model problems.

Nobody knows where the money went. The provider invoice is one number for the whole account. Attributing it to a team, an app or a feature after the fact means reconstructing it from logs that were never designed for accounting.

Every service re-implements the same plumbing. Retry, fallback to a second provider, timeout, caching, PII masking. Written once per service, differently each time, and tested nowhere.

Provider choice ossifies. Swapping a model means a code change and a deploy in every caller, so the swap never happens — even when a model that is 20× cheaper would answer the request just as well.

AIPlat puts those three concerns in the request path, so they are configuration instead of code:

• 🔀 Gateway, not a wrapper — an OpenAI-compatible endpoint. Your SDK keeps working; you change base_url and nothing else.
• 💵 Cost as a first-class record — every request writes cost, tokens, latency, cache outcome and the team/app/feature it belongs to. The breakdown exists before the invoice does.
• 🧾 A savings ledger that separates fact from estimate — savings are split into verified and counterfactual, and never blended into one flattering number. See how savings are counted.
• 🎚️ Config without redeploy — routing order, prices, budget caps, rate limits, guardrails and the model allowlist all live in DynamoDB and take effect in roughly 15 seconds.
• 🛡️ One policy point — budget caps, per-scope rate limits and content guardrails apply to every caller, including the service someone shipped on Friday afternoon.
• 🏢 Scopes that match an org chart — org → team → app, resolved from the API key. A team can restrict what the org allows, never widen it.
• 🧬 Isolated domains — six independent Terraform stacks. One domain being unavailable degrades a feature; it does not take the gateway down.

Feature tour

Cost & Usage — spend by model, feature, team and app, with the cumulative curve and a per-provider table
ROI & Savings — savings per mechanism over time, split into verified and counterfactual
Models & Routing — every model is an alias pointing at a provider; the Bedrock list is discovered live per region, drag to set the fallback order, declare identity, enter your contract price
Logs — per-request inspection: which model was asked for, which one served, the swap class, cost, latency and the failure reason
Audit trail — control-plane history with field-level before/after, including actions taken by a platform operator
Limits & Budget — rate limits, a monthly cap and the model allowlist, set at org, team or app scope
Guardrails — PII masking, secret detection and prompt-injection blocking, applied before the request leaves the gateway
Teams & Apps — the hierarchy that splits the cost, with membership and per-scope key counts
API Keys — issue and revoke gateway keys per team and app; only the hash is stored
Alerts — cost, latency, cache, error-rate, provider-capacity, SLO burn-rate and anomaly rules, delivered to a webhook
Playground — send a prompt through the gateway and see the model actually served, the swap, the cost and the cache outcome
Settings — organization profile, provider credit balance, 2FA enrolment and the gateway key for this browser

The console is a single static file on S3 behind CloudFront, with English, Portuguese and Spanish built in. Every screenshot above was generated from synthetic data — see Run the console locally.

Calling the gateway

The gateway speaks the OpenAI dialect, so existing clients need one line changed.

from openai import OpenAI

client = OpenAI(
    base_url="https://<gateway-id>.execute-api.<region>.amazonaws.com/prod/v1",
    api_key="<your-aiplat-key>",           # issued in the API Keys tab
)

resp = client.chat.completions.create(
    model="claude-sonnet",                 # your alias, not the provider's model id
    messages=[{"role": "user", "content": "Summarize this ticket."}],
    extra_body={"feature": "support-copilot"},   # optional: the cost dimension
)

model is your alias. What it resolves to — provider, region, account, exact model id — is configuration, so moving claude-sonnet from the platform's Bedrock account to your own, or to a different provider entirely, is a change in the console rather than in this code.

The response carries an aiplat block alongside the standard payload, so a caller can see what actually happened:

{
  "choices": [{ "message": { "role": "assistant", "content": "..." } }],
  "aiplat": {
    "model": "claude-sonnet-byo",
    "requested_model": "claude-sonnet",
    "provider": "bedrock",
    "estimated_cost_usd": 0.004128,
    "cache_hit": false,
    "latency_ms": 1180
  }
}

Streaming works the same way (stream: true, server-sent events). Streamed responses end with a final frame carrying usage and the same aiplat block, so a streaming caller does not have to give up the cost figures to get progressive output.

Tools and images on every provider

Function calling and multimodal input work on all four adapters, each in its own dialect: the OpenAI-dialect tools you send become Bedrock toolConfig, Anthropic input_schema declarations or Gemini functionDeclarations, and an image_url data URL becomes Bedrock image bytes, an Anthropic base64 image block or a Gemini inlineData part. Tool results travel back the same way, including the two shapes that differ structurally: Anthropic and Bedrock require every tool_result of a turn grouped in one message, and Gemini identifies a function response by name rather than by call id.

Two provider quirks are absorbed rather than passed to you. Gemini rejects JSON Schema keywords the others accept ($schema, additionalProperties, $ref), so those are stripped from tool schemas on that route — otherwise a schema your tooling generated comes back as a 400 naming a field you never wrote. And Gemini returns no id for a function call, so one is synthesized from the tool name and position, because the OpenAI dialect requires it and your client echoes it back.

An image in a format a provider does not accept is dropped rather than sent: these APIs reject the whole request over one bad part, and losing an image beats losing the answer.

Sampling parameters

max_tokens, temperature, top_p and stop are forwarded to every provider, translated into each one's dialect (stop becomes stop_sequences on Anthropic and stopSequences inside generationConfig on Gemini). A parameter you do not send is not sent: an absent field leaves the model's own default in place, because a zero we invented would not be neutral — temperature: 0 is a request for determinism, not an unset value.

All four also take part in the cache key. A parameter that changes the answer without changing the key would let a top_p: 0.1 request be served the answer generated at top_p: 1.0.

Reasoning models

Ask for thinking with reasoning_effort: none, low, medium or high (the OpenAI dialect). The gateway resolves the word into the token budget each provider needs — Bedrock and Anthropic take thinking.budget_tokens, Gemini takes thinkingConfig, and an OpenAI-compatible upstream gets the word passed straight through. That translation lives in one place on purpose: four adapters each deciding what "high" means would be four different products under one name.

A request that asks to think is routed to a model that can think. Reasoning is a declared capability per model (capabilities.reasoning, a toggle on the Models tab), and an undeclared model is treated as incapable — the same rule tool use follows, because asking a model that cannot think to think is a provider rejection rather than a soft failure. If the model you named cannot reason and another one in your catalog can, the request is served there and the substitution is recorded as a swap like any other.

The chain of thought comes back as reasoning_content — a separate field from content, on the streaming delta and on the buffered message. It is separate on purpose: every OpenAI SDK concatenates delta.content, so merging the two would print the model's private reasoning inside the answer an end user reads. A client that does not know the field ignores it and behaves exactly as before.

Thinking costs output tokens, so reasoning_effort is a client-side decision with a provider-side price. max_thinking_tokens is the operator's ceiling on what one request may buy (default 8k, hard maximum 32k). A request above it is served with the budget reduced and the reduction recorded as reasoning_clamped — never silently, because a customer who asked for high effort and quietly got less has no way to find out. The same field records the two other reconciliations: a budget cut to leave room for the answer under an explicit max_tokens, and thinking dropped entirely when max_tokens is too small for any valid budget.

Two things follow from thinking taking time and producing tokens:

• The silence is covered. Idle timeouts count bytes, not progress — a proxy in front of your client commonly closes an idle connection at 60 s, and API Gateway cuts an idle streaming response at 5 minutes. So an SSE comment frame is sent during silence (every 15 s by default, configurable per scope with sse_keepalive_seconds; a negative value turns it off). The SSE spec tells clients to ignore comment frames, so this is inert for every consumer, and it is excluded from the frame counters — a keepalive is not a token and never appears as cost.
• The thinking is in the ledger, without being double-counted. Bedrock reports input/output/total tokens only, so reasoning tokens are already inside tokens_out — already billed, already attributed. What the usage record adds is the split: reasoning_chars, present only when the model actually reasoned. Characters, not tokens, because the provider does not report a reasoning token count and deriving one would be a guess printed as a measurement. reasoning_redacted: true marks reasoning the provider encrypted, so "did not think" stays distinguishable from "not shown to us".

Per scope, request_timeout_ms stops generation early enough to still send the closing frames and write the usage record. It only ever narrows the deployment's own limit — it cannot raise it — and without it a request that runs to the platform ceiling is cut mid-answer, losing the accounting for tokens the provider already charged for.

Splitting spend across several projects with one key

One key can attribute each request to a different app, which is what lets a developer working across several local projects — or one agent driving several repositories — see the spend split per project without issuing a key per project.

Issue the key with the apps it may charge:

curl -X POST "$KEYADMIN/admin/keys" -H "authorization: Bearer $TOKEN" \
  -d '{"team":"platform","app":"web","apps":["api","batch","docs-bot"]}'

Then name one per request, by header or in the body:

resp = client.chat.completions.create(
    model="claude-sonnet",
    messages=[...],
    extra_headers={"x-aiplat-app": "docs-bot"},   # or extra_body={"app": "docs-bot"}
)

Everything downstream follows: the usage record, the by_app breakdown, the ?app= filter on /usage/records, and the App column in the CSV export.

Two things are deliberate here. The app must be one the key carries — naming any other returns 403 app_not_allowed and lists the valid values. That is not bureaucracy: app is the last link of the configuration scope chain, so it selects the budget, the rate limit, the allowed-model list and the guardrails. An unchecked app on the request would let a caller charge its spend to another project's budget and inherit its limits. Comparison is exact, with no case folding, because Api and api merging on a report while staying two separate budget scopes is worse than a 403.

If you want a free-form dimension with no governance attached, that is what feature is for (extra_body={"feature": "..."} or x-aiplat-feature). It never selects a scope, so it accepts any label — and it is the right choice for slicing one project's traffic by task.

How savings are counted

Most gateways report one savings number. That number usually mixes two very different claims, and the weaker one inflates it. AIPlat keeps them apart, in the API and on the screen.

Verified — no assumption required, because the request either was not sent or went to the identical model:

Mechanism	What happened
cache	An identical response was reused. The provider was never called.
provider_prompt_cache	The provider charged less for a repeated prefix.
provider_arbitrage	The same model served from a cheaper path — your own account, another region.

Counterfactual — a different model served the request, so the saving is measured against what the requested model would have cost. That comparison assumes the requested model was the real intent:

Mechanism	What happened
semantic_cache	A semantically close answer was reused instead of an identical one.
auto_cheapest	An equivalent cheaper model was chosen.
fallback	The requested provider failed and the next one served.
budget_degrade	The budget was exceeded, so routing dropped to a cheaper model.

The Logs tab carries the same distinction per request as a swap class: same_model (provider changed, no quality risk), equivalent (same tier or higher) and downgrade (lower tier, quality loss expected). That is the answer to "what did this saving cost me in quality?", which a savings total alone cannot give.

Costs are computed from the price you declare per model. Until you enter your negotiated price the console uses the provider list price and says so, because list-price maths overstates both cost and savings for anyone with a discount.

Architecture

Six domains, one Terraform state each. They communicate through events and shared data stores, not through synchronous calls to each other — which is why the cost pipeline being slow never adds latency to a model request.

flowchart TB
    App["📱 your application<br/>(OpenAI-compatible SDK)"]

    subgraph Core["core — the gateway"]
        RT["router λ<br/>auth · cache · routing · guardrails"]
        KA["keyadmin λ<br/>issue / revoke keys"]
    end

    subgraph Gov["governance — control plane"]
        CFG["config-api λ"]
        DDB[("config<br/>DynamoDB")]
        SEC[("provider credentials<br/>Secrets Manager")]
        COG["Cognito<br/>console sign-in · MFA"]
    end

    subgraph Obs["observability — FinOps"]
        Q(["usage queue<br/>SQS"])
        UW["usage-writer λ"]
        CS[("cost store<br/>DynamoDB")]
        UA["usage-api λ"]
        AN["alert-notifier λ<br/>EventBridge schedule"]
    end

    subgraph Aud["audit"]
        AQ(["audit queue<br/>SQS"])
        AW["audit-writer λ"]
        AT[("trail<br/>DynamoDB")]
        AA["audit-api λ"]
    end

    subgraph Front["frontend + help"]
        UI["console.html<br/>S3 + CloudFront/OAC"]
        HA["help-api λ"]
    end

    Providers["🤖 Bedrock · Anthropic · OpenAI-compatible · Gemini"]

    App -->|"POST /v1/chat/completions"| RT
    RT --> Providers
    RT -->|"async, off the hot path"| Q
    Q --> UW --> CS
    UA --> CS
    AN --> CS
    AN -->|webhook| Hook["🔔 your endpoint"]

    RT -->|"reads config ~15s"| DDB
    CFG --> DDB
    CFG --> SEC
    RT --> SEC
    CFG --> AQ
    KA --> AQ
    AQ --> AW --> AT
    AA --> AT

    UI --> COG
    UI --> CFG & UA & AA & KA & HA

Loading

Domain	Owns	Key components
core	The request path: tenant auth, model resolution, cache, routing with fallback and auto-cheapest, guardrails, OpenAI-dialect responses with SSE	router, keyadmin
governance	Dynamic config and provider credentials, org/team/app registry, members, Cognito sign-in	config-api, pretoken
observability	Usage capture off the hot path, cost store, savings maths, SLI/SLO, alert evaluation and delivery	usage-writer, usage-api, alert-notifier, hints-publisher
audit	Append-only control-plane trail with field-level diffs, CSV export, archival	audit-writer, audit-api, audit-archiver
help	In-console FAQ and internal deep-dives, in English, Portuguese and Spanish	help-api
frontend	The console as static content on private S3 behind CloudFront/OAC	console.html, WAF, response-header policy

Every domain follows the same hexagonal layout. internal/ports declares the interfaces the domain needs, internal/adapters implements them against DynamoDB, SQS, Secrets Manager, Bedrock and the provider APIs, and the decision logic sits in packages that import none of it — gateway and routing in core, govcore in governance, savings and repository in observability, auditcore in audit, help in help. That separation is what lets the routing, savings and policy logic be tested without mocking AWS.

Quick start

Prerequisites: Terraform >= 1.9, Go >= 1.26, the AWS CLI configured, and Amazon Bedrock model access enabled in the region you deploy to.

seed_user_password has no default on purpose — a committed default becomes the real credential of an account that can change budgets and read cost data:

export AWS_REGION=us-west-2
export TF_VAR_seed_user_password="$(openssl rand -base64 18)"   # >= 16 chars

Deploy core first: it owns the account-level API Gateway CloudWatch role that the other domains' access logging depends on. Then the rest in any order.

for d in core governance observability audit help frontend; do
  (cd "domains/$d" && bash build.sh)          # compile the Go Lambdas (arm64)
  (cd "domains/$d/envs/poc" && terraform init && terraform apply)
done

build.sh matters: Terraform only republishes a Lambda when source_code_hash changes, so a Go change that was not rebuilt deploys nothing at all.

Then open the CloudFront URL from the frontend output, sign in with the seeded admin user, and issue your first API key in the API Keys tab.

Run the console locally (no AWS)

The console can run against a local fixture server — no AWS account, no credentials, no deployment. This is also how every screenshot in this README is produced, which is why none of them contain real data.

node demo/server.mjs
# open http://127.0.0.1:8787/console.html

demo/fixtures.mjs generates a deterministic 30-day dataset from a fixed seed: usage series, per-request logs with every result and swap class, savings by mechanism, teams, members, keys, credit balances and an audit trail. Same input, same numbers, so refreshing the screenshots produces a real diff instead of noise.

To regenerate the images:

cd demo && npm install          # playwright
node capture.mjs                # writes ../assets/screenshots/*.png
# or reuse an installed Chrome instead of downloading Chromium:
BROWSER_CHANNEL=chrome node capture.mjs

The capture fails the run on any console error, so a broken view cannot quietly ship as a screenshot.

Security posture

What the deployed stack does by default. None of this is a compliance claim — it is the list of controls the Terraform sets up, so you can check them against your own requirements.

• The gateway data plane authenticates with a hashed API key; only the prefix is ever stored or displayed. The control-plane APIs sit behind an API Gateway Cognito authorizer, with TOTP MFA required on the console user pool by default.
• Role-based access (owner, admin, billing, dev, platform_admin) is enforced server-side from the token claims; the console hides what a role cannot use, but the backend is the gate.
• CORS is deny-by-default: an origin has to be in the allowlist to get a response header.
• The console is private S3 behind CloudFront with OAC, a CSP and HSTS response-header policy, WAF in front, and its Tailwind build self-hosted rather than pulled from a CDN.
• All four SQS queues and the SNS topic are encrypted at rest, and all eight DynamoDB tables have point-in-time recovery enabled.
• Access logging is on for CloudFront and every API Gateway stage; control-plane mutations land in an append-only audit trail with field-level diffs, including anything a platform operator does.
• Provider credentials live in Secrets Manager and are never returned by an API or written to the audit trail. Bedrock can run cross-account against a role in your account, so the platform never holds your model credentials at all. That role is created by you, must be named AIPlatGatewayAccess*, is assumed with an External ID derived from your org, and needs only bedrock:InvokeModel, bedrock:ListFoundationModels and bedrock:ListInferenceProfiles.
• Guardrails (PII masking, secret detection, prompt-injection blocking) run before the request leaves the gateway.

Repository layout

domains/            six independent Terraform stacks, each with src/ tf/ envs/
  <domain>/src/     Go: cmd/<lambda>/ + internal/{ports,adapters,<domain logic>}
  <domain>/tf/       the reusable module
  <domain>/envs/poc/ the environment wrapper that instantiates it
demo/               local fixture server + screenshot capture (not deployed)
scripts/            operational tooling, load tests, i18n checks
testdata/           contract fixtures required by the Go tests
assets/screenshots/ the images used in this README

Contributing

Issues and pull requests are welcome. Before opening a PR, build and test the domains you touched and keep the Terraform formatted:

cd domains/<domain>/src && go build ./... && go test ./...
terraform fmt -check -recursive domains

See CONTRIBUTING for the full guidelines.

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.