diff --git a/README.md b/README.md index 4ffeea5..81df48d 100644 --- a/README.md +++ b/README.md @@ -179,7 +179,13 @@ To change the host signing port or bind on all interfaces, use a Compose overrid ## OpenMeter/Konnect bootstrap Provision meters, features, and the default pay-per-use plan before starting the collector. -Use the Go `clearinghouse-bootstrap` CLI or your existing Konnect setup. +Use the [`kongctl` bootstrap scripts](openmeter-collector/provision/README.md) or your existing Konnect setup. + +```bash +cd openmeter-collector/provision +./bootstrap.sh catalog +./bootstrap.sh customer demo-client demo-user "Demo User" +``` Creates: diff --git a/openmeter-collector/provision/README.md b/openmeter-collector/provision/README.md new file mode 100644 index 0000000..3fb0483 --- /dev/null +++ b/openmeter-collector/provision/README.md @@ -0,0 +1,107 @@ +# OpenMeter / Konnect metering bootstrap + +Idempotent provisioning of the clearinghouse metering catalog (meters, features, +plan) and per-tenant customers, driven by [`kongctl`](https://developer.konghq.com/kongctl/) +against the Konnect Metering & Billing (OpenMeter) API. + +`kongctl` has no native meter resource yet ([Kong/kongctl#1334](https://github.com/Kong/kongctl/issues/1334)), +so these scripts use `kongctl api` — its authenticated passthrough to the Konnect REST API — +to drive `/v3/openmeter/*`. The catalog is data, defined in [`catalog.json`](catalog.json); +the scripts are thin and idempotent. + +## Prerequisites + +- `kongctl` and `jq` on `PATH` (the PowerShell script needs only `kongctl`). +- A Konnect Personal Access Token (`kpat_…`) with Metering & Billing access. + +## Configuration (env) + +Both scripts auto-load the repo-root [`.env`](../../.env) when present (override with +`ENV_FILE`). Put your Konnect PAT there as `OPENMETER_API_KEY` — no need to `source` the +file first (`source .env` does **not** export vars to child processes). + +| Variable | Purpose | Default | +| --- | --- | --- | +| `KONGCTL_DEFAULT_KONNECT_PAT` | Konnect PAT (preferred). Falls back to `OPENMETER_API_KEY`. | from `.env` | +| `OPENMETER_API_KEY` | Same PAT as the collector service. | from `.env` | +| `OPENMETER_URL` | Metering API base. **Must match your Konnect org region** (US vs EU). | `https://us.api.konghq.com/v3/openmeter` | + +If `OPENMETER_URL` is unset, the scripts derive it from `OPENMETER_INGEST_URL` (strip `/events`). + +One-time setup: + +```bash +cp .env.example .env # at the repo root +# edit OPENMETER_API_KEY=kpat_… +# EU orgs: OPENMETER_URL=https://eu.api.konghq.com/v3/openmeter +# OPENMETER_INGEST_URL=https://eu.api.konghq.com/v3/openmeter/events +``` + +## Usage + +```bash +cd openmeter-collector/provision + +# Catalog only — ensure meters, features, and the active plan. +./bootstrap.sh catalog + +# Provision one tenant customer (key = client_id:external_user_id). +./bootstrap.sh customer demo-client demo-user "Demo User" + +# Catalog + customer in one run; --subscribe also ensures a plan subscription. +./bootstrap.sh all demo-client demo-user "Demo User" --subscribe +``` + +Windows (PowerShell): + +```powershell +.\bootstrap.ps1 catalog +.\bootstrap.ps1 customer demo-client demo-user "Demo User" +.\bootstrap.ps1 all demo-client demo-user "Demo User" -Subscribe +``` + +Both scripts are safe to re-run: existing meters are left untouched; features missing +a meter link are recreated; plans are created and published when no active version exists. + +## What it provisions + +From [`catalog.json`](catalog.json): + +| Kind | Key | Notes | +| --- | --- | --- | +| Meter | `network_fee_usd_micros` | SUM of `$.network_fee_usd_micros` | +| Meter | `billable_usd_micros` | SUM of `$.billable_usd_micros` (not emitted by collector until phase-2; meter stays empty until then) | +| Meter | `signed_ticket_count` | COUNT | +| Feature | `network_spend` | linked to `network_fee_usd_micros` meter | +| Feature | `billable_spend` | linked to `billable_usd_micros` meter | +| Plan | `clearinghouse_default_ppu` | usage-based rate card on `billable_spend` at $0.000001/unit (1 USD micro); created as draft then published | + +## Identity contract (important) + +The CloudEvent **`subject` is the compound `client_id:external_user_id`** (e.g. +`demo-client:demo-user`), which is also the customer key and its single `subject_key`. +OpenMeter attributes usage by exact subject match, and **forbids changing a customer's +`subject_keys` once it has an active subscription** — so the subject must be compound and +correct from creation. Break usage down per-tenant/user with the `client_id` / `external_user_id` +meter dimensions, not by changing the subject. The scripts therefore never mutate +`subject_keys` on existing customers; they warn if an existing customer is missing the +expected compound key. + +## Limitations + +- Customer lookup lists customers and exact-matches the key locally (the API `key` + filter is a partial match). For very large customer bases, add pagination. +- Features are immutable except for `unit_cost`; if an existing feature lacks a meter + link (e.g. created with an older bootstrap), the script deletes and recreates it. +- Subscriptions are best-effort with `--subscribe`; plan pricing changes require a new + plan version in Konnect (out of scope for this script). + +## Konnect first-time setup + +Walkthrough for provisioning against a fresh Konnect org: + +1. **Create org** — from the org picker, click **+ Create** to provision a clean Metering & Billing workspace. +2. **Name the org** — e.g. *Clearinghouse Example Org*. +3. **Create a Personal Access Token** — Profile menu → **Personal access tokens** → **Generate**. Copy the `kpat_…` token immediately (shown once). +4. **Configure `.env`** — copy `.env.example` to `.env` at the repo root, set `OPENMETER_API_KEY`, and set `OPENMETER_URL` / `OPENMETER_INGEST_URL` to your org's region (`us` or `eu`). +5. **Run bootstrap** — `cd openmeter-collector/provision && ./bootstrap.sh catalog`. Re-run to confirm idempotency (`exists` / `active` lines, no errors). diff --git a/openmeter-collector/provision/bootstrap.ps1 b/openmeter-collector/provision/bootstrap.ps1 new file mode 100644 index 0000000..9ed4639 --- /dev/null +++ b/openmeter-collector/provision/bootstrap.ps1 @@ -0,0 +1,309 @@ +#requires -version 5.1 +<# +.SYNOPSIS + Bootstrap the OpenMeter/Konnect metering catalog for the clearinghouse collector. + +.DESCRIPTION + Idempotent: creates only what is missing. Meters are immutable in OpenMeter, so this + script never updates or deletes them. Uses `kongctl api` against /v3/openmeter/*. + + Auth: $env:KONGCTL_DEFAULT_KONNECT_PAT (preferred) or $env:OPENMETER_API_KEY (Konnect kpat_…). + Endpoint: $env:OPENMETER_URL (default https://us.api.konghq.com/v3/openmeter). + +.EXAMPLE + .\bootstrap.ps1 catalog +.EXAMPLE + .\bootstrap.ps1 customer demo-client demo-user "Demo User" +.EXAMPLE + .\bootstrap.ps1 all demo-client demo-user "Demo User" -Subscribe +#> +[CmdletBinding()] +param( + [ValidateSet('catalog', 'customer', 'all')] + [string]$Command = 'catalog', + [string]$ClientId, + [string]$ExternalUserId, + [string]$DisplayName, + [switch]$Subscribe +) + +$ErrorActionPreference = 'Stop' +$ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path +$Catalog = if ($env:CATALOG) { $env:CATALOG } else { Join-Path $ScriptDir 'catalog.json' } + +function Die($m) { Write-Error $m; exit 1 } +function Info($m) { Write-Host $m } +function Warn($m) { Write-Warning $m } + +if (-not (Get-Command kongctl -ErrorAction SilentlyContinue)) { Die 'kongctl not found (https://developer.konghq.com/kongctl/)' } +if (-not (Test-Path $Catalog)) { Die "catalog not found: $Catalog" } + +# --- env file (repo-root .env) --------------------------------------------- +$EnvFile = if ($env:ENV_FILE) { $env:ENV_FILE } else { Join-Path (Split-Path (Split-Path $ScriptDir -Parent) -Parent) '.env' } +if (Test-Path $EnvFile) { + Get-Content $EnvFile | ForEach-Object { + if ($_ -match '^\s*#' -or $_ -match '^\s*$') { return } + if ($_ -match '^([^=]+)=(.*)$') { + $k = $Matches[1].Trim() + $v = $Matches[2].Trim() + if (($v.StartsWith('"') -and $v.EndsWith('"')) -or ($v.StartsWith("'") -and $v.EndsWith("'"))) { + $v = $v.Substring(1, $v.Length - 2) + } + if (-not (Test-Path "Env:$k")) { Set-Item -Path "Env:$k" -Value $v } + } + } +} + +# --- auth + endpoint ------------------------------------------------------- +$pat = if ($env:KONGCTL_DEFAULT_KONNECT_PAT) { $env:KONGCTL_DEFAULT_KONNECT_PAT } else { $env:OPENMETER_API_KEY } +if (-not $pat) { Die "no Konnect PAT — set KONGCTL_DEFAULT_KONNECT_PAT or OPENMETER_API_KEY in the environment or in $EnvFile" } +$env:KONGCTL_DEFAULT_KONNECT_PAT = $pat + +if (-not $env:OPENMETER_URL -and $env:OPENMETER_INGEST_URL) { + $env:OPENMETER_URL = ($env:OPENMETER_INGEST_URL.TrimEnd('/') -replace '/events/?$', '') +} +$omUrl = if ($env:OPENMETER_URL) { $env:OPENMETER_URL } else { 'https://us.api.konghq.com/v3/openmeter' } +$omUrl = $omUrl.TrimEnd('/') +$BASE = [regex]::Replace($omUrl, '(https?://[^/]+).*', '$1') +$PREFIX = [regex]::Replace($omUrl, 'https?://[^/]+', '') +if (-not $PREFIX) { $PREFIX = '/v3/openmeter' } + +$catalogObj = Get-Content -Raw $Catalog | ConvertFrom-Json + +# --- kongctl api helpers --------------------------------------------------- +function Kapi-Err($method, $path, $detail) { + Die "kongctl api $method $path failed — check OPENMETER_URL ($omUrl) and your PAT: $detail" +} +function Kapi-Warn($method, $path, $detail) { + Warn "kongctl api $method $path failed — check OPENMETER_URL ($omUrl) and your PAT: $detail" +} +function Kapi-BodyError($obj) { + if ($obj.message) { return $obj.message } + if ($obj.detail) { return $obj.detail } + return $obj.title +} +function Kapi-BodyIsError($obj) { + if ($null -eq $obj) { return $true } + $msg = Kapi-BodyError $obj + if (-not $msg) { return $false } + $hasData = $obj.PSObject.Properties.Name -contains 'data' -or $obj.PSObject.Properties.Name -contains 'items' + return -not $hasData +} +function Kapi-Run($soft, $method, $path, $bodyJson) { + $errFile = [System.IO.Path]::GetTempFileName() + try { + $out = if ($method -eq 'delete') { + & kongctl api delete "$PREFIX$path" --base-url $BASE -o json 2>$errFile + } + elseif ($method -eq 'get') { + & kongctl api get "$PREFIX$path" --base-url $BASE -o json 2>$errFile + } + else { + $bodyJson | & kongctl api $method "$PREFIX$path" --base-url $BASE -o json -f - 2>$errFile + } + if ($LASTEXITCODE -ne 0) { + if ($soft) { Kapi-Warn $method $path (Get-Content -Raw $errFile); return $null } + Kapi-Err $method $path (Get-Content -Raw $errFile) + } + if (-not $out) { + if ($soft) { Kapi-Warn $method $path 'empty response'; return $null } + Kapi-Err $method $path 'empty response' + } + $obj = $out | ConvertFrom-Json + if (Kapi-BodyIsError $obj) { + if ($soft) { Kapi-Warn $method $path (Kapi-BodyError $obj); return $null } + Kapi-Err $method $path (Kapi-BodyError $obj) + } + return $obj + } + finally { Remove-Item -Force $errFile -ErrorAction SilentlyContinue } +} +function Kapi-Get($path) { return Kapi-Run $false get $path $null } +function Kapi-Send($method, $path, $bodyJson) { return Kapi-Run $false $method $path $bodyJson } +function Kapi-Delete($path) { Kapi-Run $false delete $path $null | Out-Null } +function Kapi-Get-Soft($path) { return Kapi-Run $true get $path $null } +function Kapi-Send-Soft($method, $path, $bodyJson) { return Kapi-Run $true $method $path $bodyJson } +function Kapi-Delete-Soft($path) { Kapi-Run $true delete $path $null | Out-Null } +function Plan-ConfigKey { + if ($catalogObj.plan -and $catalogObj.plan.key) { return $catalogObj.plan.key } + return $catalogObj.plan_key +} +function Meter-IdFor($meterKey) { + $m = Items (Kapi-Get '/meters') | Where-Object { $_.key -eq $meterKey } | Select-Object -First 1 + if (-not $m) { return $null } + return $m.id +} +function Feature-For($featureKey) { + return Items (Kapi-Get '/features') | Where-Object { $_.key -eq $featureKey } | Select-Object -First 1 +} +function Find-PlanByStatus($planKey, $status) { + return Items (Kapi-Get "/plans?filter[key]=$planKey&filter[status]=$status") | + Where-Object { $_.key -eq $planKey } | Select-Object -First 1 +} +function Feature-MeterKey($feature) { + if ($feature.meter_key) { return $feature.meter_key } + return $feature.meter_slug +} +function New-FeatureBody($key, $name, $meterId) { + return (@{ key = $key; name = $name; meter = @{ id = $meterId } } | ConvertTo-Json -Compress) +} +function Items($resp) { + if ($null -eq $resp) { return @() } + if ($resp.PSObject.Properties.Name -contains 'data') { return @($resp.data) } + return @($resp) +} + +# --- catalog --------------------------------------------------------------- +function Ensure-Meters { + $existing = (Items (Kapi-Get '/meters')).key + foreach ($m in $catalogObj.meters) { + if ($existing -contains $m.key) { Info "meter $($m.key) - exists"; continue } + $body = [ordered]@{ name = $m.name; key = $m.key; description = $m.description; event_type = $m.event_type; aggregation = $m.aggregation } + if ($m.value_property) { $body.value_property = $m.value_property } + if ($m.dimensions) { $body.dimensions = $m.dimensions } + Kapi-Send 'post' '/meters' ($body | ConvertTo-Json -Depth 6 -Compress) | Out-Null + Info "meter $($m.key) - created" + } +} +function Ensure-Features { + foreach ($f in $catalogObj.features) { + $meterKey = Feature-MeterKey $f + if (-not $meterKey) { Die "feature $($f.key) requires meter_key in catalog.json" } + $meterId = Meter-IdFor $meterKey + if (-not $meterId) { Die "meter $meterKey not found for feature $($f.key)" } + + $feat = Feature-For $f.key + if (-not $feat) { + Kapi-Send 'post' '/features' (New-FeatureBody $f.key $f.name $meterId) | Out-Null + Info "feature $($f.key) - created" + continue + } + + $linked = $feat.meter.id + if ($linked -eq $meterId) { Info "feature $($f.key) - exists"; continue } + + Warn "feature $($f.key) - exists without meter link; recreating" + Kapi-Delete-Soft "/features/$($feat.id)" + Kapi-Send 'post' '/features' (New-FeatureBody $f.key $f.name $meterId) | Out-Null + Info "feature $($f.key) - recreated (meter: $meterKey)" + } +} +function Build-PlanBody { + $featMap = @{} + foreach ($feat in Items (Kapi-Get '/features')) { $featMap[$feat.key] = $feat.id } + $p = $catalogObj.plan + $phases = @() + foreach ($phase in $p.phases) { + $rateCards = @() + foreach ($rc in $phase.rate_cards) { + $rateCards += [ordered]@{ + key = $rc.key + name = $rc.name + feature = @{ id = $featMap[$rc.feature_key] } + billing_cadence = $rc.billing_cadence + price = $rc.price + } + } + $phases += [ordered]@{ key = $phase.key; name = $phase.name; rate_cards = $rateCards } + } + $body = [ordered]@{ + key = $p.key + name = $p.name + currency = $p.currency + billing_cadence = $p.billing_cadence + phases = $phases + } + if ($p.description) { $body.description = $p.description } + return ($body | ConvertTo-Json -Depth 8 -Compress) +} +function Publish-Plan($planId, $planKey) { + $published = Kapi-Send-Soft 'post' "/plans/$planId/publish" '{}' + if ($published -and $published.status -eq 'active') { + Info "plan $planKey - published" + return $true + } + Warn "plan $planKey - could not publish (ensure features have meter links)" + return $false +} +function Ensure-Plan { + $planKey = Plan-ConfigKey + if (-not $planKey) { Info 'plan - none configured'; return } + if (-not $catalogObj.plan) { Die 'catalog plan block missing - add .plan or remove plan_key' } + + if (Find-PlanByStatus $planKey 'active') { Info "plan $planKey - active"; return } + + $draft = Find-PlanByStatus $planKey 'draft' + if ($draft) { + Info "plan $planKey - draft exists, publishing" + Publish-Plan $draft.id $planKey | Out-Null + return + } + + $bodyJson = Build-PlanBody + $bodyObj = $bodyJson | ConvertFrom-Json + foreach ($phase in $bodyObj.phases) { + foreach ($rc in $phase.rate_cards) { + if (-not $rc.feature.id) { Die 'plan rate cards reference unknown features - run ensure_features first' } + } + } + $created = Kapi-Send 'post' '/plans' $bodyJson + Info "plan $planKey - created (draft)" + Publish-Plan $created.id $planKey | Out-Null +} +function Invoke-Catalog { + Info "== catalog ($BASE$PREFIX) ==" + Ensure-Meters; Ensure-Features; Ensure-Plan +} + +# --- customer -------------------------------------------------------------- +function Ensure-Customer($clientId, $externalUserId, $display, $subscribe) { + if (-not $clientId -or -not $externalUserId) { Die 'customer requires ' } + # CloudEvent subject = compound client_id:external_user_id = customer key = its only + # subject_key. OpenMeter forbids changing subject_keys on subscribed customers, so we + # set it at creation and never mutate it. + $compound = "$clientId`:$externalUserId" + if (-not $display) { $display = $compound } + + $cust = Items (Kapi-Get '/customers') | Where-Object { $_.key -eq $compound } | Select-Object -First 1 + if (-not $cust) { + $body = [ordered]@{ key = $compound; name = $display; usage_attribution = @{ subject_keys = @($compound) } } + $created = Kapi-Send 'post' '/customers' ($body | ConvertTo-Json -Depth 5 -Compress) + $id = $created.id + Info "customer $compound - created (subject: $compound)" + } + else { + $id = $cust.id + if ($cust.usage_attribution.subject_keys -contains $compound) { Info "customer $compound - up to date" } + else { + Warn "customer $compound exists but its subject_keys do not include '$compound'" + Warn " (OpenMeter blocks subject_key changes on subscribed customers - reconcile manually)" + } + } + if ($subscribe) { Ensure-Subscription $id $compound } +} + +function Ensure-Subscription($customerId, $label) { + $planKey = Plan-ConfigKey + if (-not $planKey) { Warn 'no plan_key in catalog; skipping subscription'; return } + $existing = Items (Kapi-Get-Soft "/subscriptions?customer_id=$customerId") | Where-Object { $_.customer_id -eq $customerId } + if ($existing) { Info "sub $label - exists"; return } + $body = [ordered]@{ + customer = [ordered]@{ key = $label } + plan = [ordered]@{ key = $planKey } + } + $created = Kapi-Send-Soft 'post' '/subscriptions' ($body | ConvertTo-Json -Compress) + if ($created) { + Info "sub $label - created on $planKey" + } + else { + Warn "sub $label - could not create subscription on $planKey (create manually if needed)" + } +} + +# --- dispatch -------------------------------------------------------------- +switch ($Command) { + 'catalog' { Invoke-Catalog } + 'customer' { Ensure-Customer $ClientId $ExternalUserId $DisplayName $Subscribe.IsPresent } + 'all' { Invoke-Catalog; Ensure-Customer $ClientId $ExternalUserId $DisplayName $Subscribe.IsPresent } +} +Info 'done.' diff --git a/openmeter-collector/provision/bootstrap.sh b/openmeter-collector/provision/bootstrap.sh new file mode 100755 index 0000000..6bcae6a --- /dev/null +++ b/openmeter-collector/provision/bootstrap.sh @@ -0,0 +1,388 @@ +#!/usr/bin/env bash +# +# Bootstrap the OpenMeter/Konnect metering catalog for the clearinghouse collector. +# +# Idempotent: creates only what is missing. Meters are immutable in OpenMeter, so +# this script never updates or deletes them — it only adds missing meters/features. +# For customers it never mutates subject_keys on existing records (warns only). +# +# Requires: kongctl (https://developer.konghq.com/kongctl/) and jq. +# Auth: KONGCTL_DEFAULT_KONNECT_PAT (preferred) or OPENMETER_API_KEY — a Konnect PAT (kpat_…). +# Endpoint: OPENMETER_URL (default https://us.api.konghq.com/v3/openmeter). +# +# Usage: +# ./bootstrap.sh catalog +# Ensure meters + features exist and the configured plan is present. +# +# ./bootstrap.sh customer [display_name] [--subscribe] +# Ensure an OpenMeter customer keyed : exists with +# subject_keys = [:] (matches the CloudEvent subject). +# --subscribe also ensures a subscription on the catalog plan (best-effort). +# +# ./bootstrap.sh all [display_name] [--subscribe] +# catalog + customer in one run. +# +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CATALOG="${CATALOG:-$SCRIPT_DIR/catalog.json}" + +die() { printf 'error: %s\n' "$*" >&2; exit 1; } +info() { printf '%s\n' "$*" >&2; } +warn() { printf 'warning: %s\n' "$*" >&2; } + +command -v kongctl >/dev/null 2>&1 || die "kongctl not found (https://developer.konghq.com/kongctl/)" +command -v jq >/dev/null 2>&1 || die "jq not found" +[ -f "$CATALOG" ] || die "catalog not found: $CATALOG" + +# --- env file (repo-root .env) --------------------------------------------- +# Plain `source .env` does not export vars to child processes; load here so +# `./bootstrap.sh catalog` works without `set -a; source …`. +ENV_FILE="${ENV_FILE:-$SCRIPT_DIR/../../.env}" +if [ -f "$ENV_FILE" ]; then + set -a + # shellcheck disable=SC1090 + source "$ENV_FILE" + set +a +fi + +# --- auth + endpoint ------------------------------------------------------- +PAT="${KONGCTL_DEFAULT_KONNECT_PAT:-${OPENMETER_API_KEY:-}}" +if [ -z "$PAT" ]; then + die "no Konnect PAT — set KONGCTL_DEFAULT_KONNECT_PAT or OPENMETER_API_KEY in the environment or in $ENV_FILE" +fi +export KONGCTL_DEFAULT_KONNECT_PAT="$PAT" + +if [ -z "${OPENMETER_URL:-}" ] && [ -n "${OPENMETER_INGEST_URL:-}" ]; then + OPENMETER_URL="${OPENMETER_INGEST_URL%/events}" +fi +OPENMETER_URL="${OPENMETER_URL:-https://us.api.konghq.com/v3/openmeter}" +OPENMETER_URL="${OPENMETER_URL%/}" +BASE="$(printf '%s' "$OPENMETER_URL" | sed -E 's#(https?://[^/]+).*#\1#')" +PREFIX="$(printf '%s' "$OPENMETER_URL" | sed -E 's#https?://[^/]+##')" +[ -n "$PREFIX" ] || PREFIX="/v3/openmeter" + +# --- kongctl api helpers (return response body as JSON on stdout) ---------- +kapi_err() { + local method="$1" path="$2" err="$3" + die "kongctl api $method $path failed — check OPENMETER_URL ($OPENMETER_URL) and your PAT: $err" +} + +kapi_warn() { + local method="$1" path="$2" err="$3" + warn "kongctl api $method $path failed — check OPENMETER_URL ($OPENMETER_URL) and your PAT: $err" +} + +kapi_body_error() { + printf '%s' "$1" | jq -r '.message // .detail // .title // empty' +} + +kapi_body_is_error() { + printf '%s' "$1" | jq -e 'type == "object" and (.message // .detail // .title // empty) != "" and (.data // .items // null) == null' >/dev/null 2>&1 +} + +kapi_run() { + local mode="$1" method="$2" path="$3" + shift 3 + local body err + err="$(mktemp)" + case "$method" in + get) + if ! body="$(kongctl api get "$PREFIX$path" --base-url "$BASE" -o json 2>"$err")"; then + if [ "$mode" = soft ]; then kapi_warn get "$path" "$(cat "$err")"; rm -f "$err"; return 1; fi + kapi_err get "$path" "$(cat "$err")" + fi + ;; + post) + if ! body="$(kongctl api post "$PREFIX$path" --base-url "$BASE" -o json -f - 2>"$err")"; then + if [ "$mode" = soft ]; then kapi_warn post "$path" "$(cat "$err")"; rm -f "$err"; return 1; fi + kapi_err post "$path" "$(cat "$err")" + fi + ;; + put) + if ! body="$(kongctl api put "$PREFIX$path" --base-url "$BASE" -o json -f - 2>"$err")"; then + if [ "$mode" = soft ]; then kapi_warn put "$path" "$(cat "$err")"; rm -f "$err"; return 1; fi + kapi_err put "$path" "$(cat "$err")" + fi + ;; + delete) + if ! body="$(kongctl api delete "$PREFIX$path" --base-url "$BASE" -o json 2>"$err")"; then + if [ "$mode" = soft ]; then kapi_warn delete "$path" "$(cat "$err")"; rm -f "$err"; return 1; fi + kapi_err delete "$path" "$(cat "$err")" + fi + ;; + *) die "unknown kapi method: $method" ;; + esac + rm -f "$err" + if [ -z "$body" ]; then + if [ "$mode" = soft ]; then kapi_warn "$method" "$path" "empty response"; return 1; fi + kapi_err "$method" "$path" "empty response" + fi + if kapi_body_is_error "$body"; then + if [ "$mode" = soft ]; then kapi_warn "$method" "$path" "$(kapi_body_error "$body")"; return 1; fi + kapi_err "$method" "$path" "$(kapi_body_error "$body")" + fi + printf '%s' "$body" +} + +kapi_get() { kapi_run hard get "$1"; } +kapi_post() { kapi_run hard post "$1"; } +kapi_put() { kapi_run hard put "$1"; } +kapi_delete() { kapi_run hard delete "$1"; } +kapi_get_soft() { kapi_run soft get "$1"; } +kapi_post_soft() { kapi_run soft post "$1"; } +kapi_delete_soft() { kapi_run soft delete "$1"; } + +plan_config_key() { jq -r '.plan.key // .plan_key // empty' "$CATALOG"; } + +meter_id_for() { + kapi_get /meters | jq -r --arg k "$1" '(.data // .)[] | select(.key == $k) | .id' +} + +feature_for() { + kapi_get /features | jq -c --arg k "$1" '(.data // .)[] | select(.key == $k)' +} + +find_plan_by_status() { + local plan_key="$1" status="$2" + kapi_get "/plans?filter[key]=${plan_key}&filter[status]=${status}" \ + | jq -c --arg k "$plan_key" '(.data // .)[] | select(.key == $k)' | head -n 1 +} + +# --- catalog --------------------------------------------------------------- +ensure_meters() { + local existing + existing="$(kapi_get /meters | jq -r '(.data // .)[].key')" + while IFS= read -r m; do + [ -n "$m" ] || continue + local key; key="$(jq -r '.key' <<<"$m")" + if printf '%s\n' "$existing" | grep -qxF "$key"; then + info "meter $key — exists" + continue + fi + local body + body="$(jq '{name, key, description, event_type, aggregation} + + (if .value_property then {value_property} else {} end) + + (if .dimensions then {dimensions} else {} end)' <<<"$m")" + printf '%s' "$body" | kapi_post /meters >/dev/null + info "meter $key — created" + done < <(jq -c '.meters[]' "$CATALOG") +} + +feature_meter_key() { jq -r '.meter_key // .meter_slug // empty' <<<"$1"; } + +create_feature() { + local key="$1" name="$2" meter_id="$3" + local body + body="$(jq -n --arg key "$key" --arg name "$name" --arg mid "$meter_id" \ + '{key:$key, name:$name, meter:{id:$mid}}')" + printf '%s' "$body" | kapi_post /features >/dev/null +} + +ensure_features() { + while IFS= read -r f; do + [ -n "$f" ] || continue + local key meter_key meter_id feat_rec feat_id linked_meter body + key="$(jq -r '.key' <<<"$f")" + meter_key="$(feature_meter_key "$f")" + [ -n "$meter_key" ] || die "feature $key requires meter_key in catalog.json" + meter_id="$(meter_id_for "$meter_key")" + [ -n "$meter_id" ] || die "meter $meter_key not found for feature $key" + + feat_rec="$(feature_for "$key")" + if [ -z "$feat_rec" ]; then + create_feature "$key" "$(jq -r '.name' <<<"$f")" "$meter_id" + info "feature $key — created" + continue + fi + + feat_id="$(jq -r '.id' <<<"$feat_rec")" + linked_meter="$(jq -r '.meter.id // empty' <<<"$feat_rec")" + if [ "$linked_meter" = "$meter_id" ]; then + info "feature $key — exists" + continue + fi + + warn "feature $key — exists without meter link; recreating" + kapi_delete_soft "/features/$feat_id" >/dev/null 2>&1 || true + create_feature "$key" "$(jq -r '.name' <<<"$f")" "$meter_id" + info "feature $key — recreated (meter: $meter_key)" + done < <(jq -c '.features[]' "$CATALOG") +} + +build_plan_body() { + local feat_map + feat_map="$(kapi_get /features | jq '[(.data // .)[] | {(.key): .id}] | add')" + jq --argjson feats "$feat_map" ' + .plan as $p + | { + key: $p.key, + name: $p.name, + description: ($p.description // empty), + currency: $p.currency, + billing_cadence: $p.billing_cadence, + phases: [ + $p.phases[] + | { + key, + name, + rate_cards: [ + .rate_cards[] + | { + key, + name, + feature: { id: $feats[.feature_key] }, + billing_cadence, + price + } + ] + } + ] + } + | if .description == "" then del(.description) else . end + ' "$CATALOG" +} + +publish_plan() { + local plan_id="$1" plan_key="$2" resp + if resp="$(printf '{}' | kapi_post_soft "/plans/$plan_id/publish")" \ + && printf '%s' "$resp" | jq -e '.status == "active"' >/dev/null; then + info "plan $plan_key — published" + return 0 + fi + warn "plan $plan_key — could not publish (ensure features have meter links)" + return 1 +} + +ensure_plan() { + local plan_key + plan_key="$(plan_config_key)" + [ -n "$plan_key" ] || { info "plan — none configured"; return 0; } + jq -e '.plan' "$CATALOG" >/dev/null 2>&1 \ + || die "catalog plan block missing — add .plan or remove plan_key" + + if [ -n "$(find_plan_by_status "$plan_key" active)" ]; then + info "plan $plan_key — active" + return 0 + fi + + local draft draft_id body plan_id + draft="$(find_plan_by_status "$plan_key" draft)" + if [ -n "$draft" ]; then + draft_id="$(jq -r '.id' <<<"$draft")" + info "plan $plan_key — draft exists, publishing" + publish_plan "$draft_id" "$plan_key" || true + return 0 + fi + + body="$(build_plan_body)" + if printf '%s' "$body" | jq -e '[ + .phases[].rate_cards[].feature.id + | select(. == null or . == "") + ] | length == 0' >/dev/null; then + : + else + die "plan rate cards reference unknown features — run ensure_features first" + fi + + plan_id="$(printf '%s' "$body" | kapi_post /plans | jq -r '.id')" + info "plan $plan_key — created (draft)" + publish_plan "$plan_id" "$plan_key" || true +} + +cmd_catalog() { + info "== catalog ($BASE$PREFIX) ==" + ensure_meters + ensure_features + ensure_plan +} + +# --- customer -------------------------------------------------------------- +# Find a customer by exact key. NOTE: the list filter is a partial match, so we +# fetch and exact-match locally. For very large customer bases add pagination. +find_customer() { + kapi_get "/customers" | jq -c --arg k "$1" '(.data // .)[] | select(.key == $k)' | head -n 1 +} + +ensure_customer() { + local client_id="$1" external_user_id="$2" display="$3" subscribe="$4" + [ -n "$client_id" ] && [ -n "$external_user_id" ] || die "customer requires " + # The CloudEvent subject is the compound client_id:external_user_id (globally + # unique, tenant-scoped). It is also the customer key and its only subject_key. + # OpenMeter forbids changing subject_keys once a customer has an active + # subscription, so we set it correctly at creation and never mutate it. + local compound="$client_id:$external_user_id" + [ -n "$display" ] || display="$compound" + + local cust; cust="$(find_customer "$compound")" + local id + if [ -z "$cust" ]; then + local body + body="$(jq -n --arg key "$compound" --arg name "$display" \ + '{key:$key, name:$name, usage_attribution:{subject_keys:[$key]}}')" + id="$(printf '%s' "$body" | kapi_post /customers | jq -r '.id')" + info "customer $compound — created (subject: $compound)" + else + id="$(jq -r '.id' <<<"$cust")" + if jq -e --arg c "$compound" '(.usage_attribution.subject_keys // []) | index($c)' <<<"$cust" >/dev/null; then + info "customer $compound — up to date" + else + warn "customer $compound exists but its subject_keys do not include '$compound'" + warn " (OpenMeter blocks subject_key changes on subscribed customers — reconcile manually)" + fi + fi + + [ "$subscribe" = "1" ] && ensure_subscription "$id" "$compound" || true +} + +# Best-effort subscription on the catalog plan. Skips if the customer already has one. +ensure_subscription() { + local customer_id="$1" label="$2" + local plan_key; plan_key="$(plan_config_key)" + [ -n "$plan_key" ] || { warn "no plan_key in catalog; skipping subscription"; return 0; } + + local existing resp + existing="$(kapi_get_soft "/subscriptions?customer_id=$customer_id" 2>/dev/null \ + | jq -r --arg c "$customer_id" '(.data // .)[]? | select(.customer_id == $c) | .id' 2>/dev/null || true)" + if [ -n "$existing" ]; then + info "sub $label — exists" + return 0 + fi + local body + body="$(jq -n --arg ck "$label" --arg pk "$plan_key" \ + '{customer:{key:$ck}, plan:{key:$pk}}')" + if resp="$(printf '%s' "$body" | kapi_post_soft /subscriptions)" && [ -n "$resp" ]; then + info "sub $label — created on $plan_key" + else + warn "sub $label — could not create subscription on $plan_key (create manually if needed)" + fi +} + +# --- arg parsing ----------------------------------------------------------- +SUBSCRIBE=0 +ARGS=() +for a in "$@"; do + case "$a" in + --subscribe) SUBSCRIBE=1 ;; + *) ARGS+=("$a") ;; + esac +done +if [ "${#ARGS[@]}" -eq 0 ]; then set -- catalog; else set -- "${ARGS[@]}"; fi + +cmd="${1:-catalog}"; shift || true +case "$cmd" in + catalog) + cmd_catalog + ;; + customer) + ensure_customer "${1:-}" "${2:-}" "${3:-}" "$SUBSCRIBE" + ;; + all) + cmd_catalog + ensure_customer "${1:-}" "${2:-}" "${3:-}" "$SUBSCRIBE" + ;; + *) + die "unknown command '$cmd' (expected: catalog | customer | all)" + ;; +esac +info "done." diff --git a/openmeter-collector/provision/catalog.json b/openmeter-collector/provision/catalog.json new file mode 100644 index 0000000..1b7bccc --- /dev/null +++ b/openmeter-collector/provision/catalog.json @@ -0,0 +1,83 @@ +{ + "$comment": "OpenMeter/Konnect metering catalog for the clearinghouse collector. Provisioned idempotently by bootstrap.sh / bootstrap.ps1. Meters are immutable in OpenMeter — the scripts only create missing entries, they never update or delete. Dimensions/value_property must match the CloudEvent fields emitted by openmeter-collector/collector.yaml.", + "meters": [ + { + "key": "network_fee_usd_micros", + "name": "Network fee (USD micros)", + "description": "Raw network cost in USD micros from the signer (observability).", + "event_type": "create_signed_ticket", + "aggregation": "sum", + "value_property": "$.network_fee_usd_micros", + "dimensions": { + "client_id": "$.client_id", + "external_user_id": "$.external_user_id", + "model_id": "$.model_id", + "pipeline": "$.pipeline" + } + }, + { + "key": "billable_usd_micros", + "name": "Billable usage (USD micros)", + "description": "Post-markup billable amount in USD micros for pay-per-use billing. Not emitted by the collector until phase-2 markup; meter stays empty until then.", + "event_type": "create_signed_ticket", + "aggregation": "sum", + "value_property": "$.billable_usd_micros", + "dimensions": { + "client_id": "$.client_id", + "external_user_id": "$.external_user_id", + "model_id": "$.model_id", + "pipeline": "$.pipeline" + } + }, + { + "key": "signed_ticket_count", + "name": "Signed ticket count", + "description": "Count of create_signed_ticket events (request volume).", + "event_type": "create_signed_ticket", + "aggregation": "count", + "dimensions": { + "client_id": "$.client_id", + "external_user_id": "$.external_user_id", + "model_id": "$.model_id", + "pipeline": "$.pipeline" + } + } + ], + "features": [ + { + "key": "network_spend", + "name": "Network spend", + "meter_key": "network_fee_usd_micros" + }, + { + "key": "billable_spend", + "name": "Billable spend", + "meter_key": "billable_usd_micros" + } + ], + "plan": { + "key": "clearinghouse_default_ppu", + "name": "Clearinghouse default pay-per-use", + "description": "Default pay-per-use plan billed on billable_usd_micros usage (1 USD micro = $0.000001).", + "currency": "USD", + "billing_cadence": "P1M", + "phases": [ + { + "key": "default", + "name": "Default", + "rate_cards": [ + { + "key": "billable_spend", + "name": "Billable spend", + "feature_key": "billable_spend", + "billing_cadence": "P1M", + "price": { + "type": "unit", + "amount": "0.000001" + } + } + ] + } + ] + } +}