feat: self-contained identity webhook + collector price oracle and metering fixes

[eliteprox]

Summary

Four related changes to the clearinghouse stack:

1. Identity webhook — new in-compose Node.js service implementing go-livepeer's /authorize wire protocol in-repo, with API-key and OIDC (JWT/JWKS via jose) verifiers. No external SDK dependency.
2. Live ETH/USD price oracle in the OpenMeter collector — replaces static ETH_USD_PRICE with a fetched-and-cached oracle price.
3. Collector CloudEvents fixes — compound subject for Konnect customer attribution, billable_usd_micros passthrough, and a Benthos pipeline bugfix that silently dropped all events.
4. Unified configuration — single root .env (from .env.example) for all Compose services; per-service .env.example files removed.

1. Identity webhook

• identity-webhook/ — Node service at repo root. Implements the go-livepeer remote-signer webhook protocol (protocol.mjs) and pluggable end-user verifiers (verifiers.mjs): API keys (sk_* from DEMO_API_KEY / DEMO_API_KEYS) and optional OIDC JWTs (JWKS via jose). IDENTITY_AUTH_MODE (api_key | oidc) selects exactly one verifier — no fallback. Stateless — no local identity DB. 32 unit tests.
• Compose wiring — remote-signer depends_on kafka (healthcheck-gated) only; identity-webhook starts independently. This supports signer-only alive checks (docker compose up kafka remote-signer with REMOTE_SIGNER_WEBHOOK_URL unset). When the webhook is used, set REMOTE_SIGNER_WEBHOOK_URL=http://identity-webhook:8090/authorize (default in .env.example).
• End users present Authorization: Bearer sk_… (API key mode) or Authorization: Bearer <jwt> (OIDC mode) to the signer; the webhook resolves credentials to auth_id = "{client_id}:{usage_subject}".
• Signing port published as 127.0.0.1:8081 by default; use a Compose override to bind 0.0.0.0 only when the webhook is configured.

Note: leave REMOTE_SIGNER_WEBHOOK_URL empty for local alive checks only (inherited from main). Production deployments must set REMOTE_SIGNER_WEBHOOK_URL and WEBHOOK_SECRET.

2. ETH/USD price oracle (collector)

• PRICE_ORACLE_URL (default Coinbase spot) and PRICE_ORACLE_REFRESH (default 5m) replace static ETH_USD_PRICE.
• Price is warmed at startup via a Benthos sequence input that blocks Kafka consumption until the cache is populated, then refreshed on the configured interval. Refresh errors keep the last cached value; runtime events read from cache only (cache miss throws).
• Oracle JSON accepted in three shapes: {"price":…}, {"ethereum":{"usd":…}} (CoinGecko), {"data":{"amount":"…"}} (Coinbase).

3. Collector CloudEvents fixes

• Compound subject: OpenMeter attributes usage by exact subject match; customers are keyed by client_id:usage_subject. The collector now emits subject = auth_id (compound) instead of bare usage_subject, with data.client_id / data.usage_subject for per-tenant/user meter dimensions. Verified against live Konnect.
• billable_usd_micros: emitted as interim passthrough equal to network_fee_usd_micros so the billable meter validates and accumulates; phase-2 markup rules come later.
• Bugfix: the eth_usd price branch used root = @, overwriting the entire message body with the metadata map. Downstream mapping then read an empty this.data, parsed an empty auth_id, and silently dropped every create_signed_ticket — so no events reached OpenMeter/Konnect. Dropping root = @ leaves the original event intact.
• Guard: events missing auth_id are dropped (protects against pre-0.8.11 go-livepeer builds that omit the field).

4. Unified configuration

• Single .env.example at repo root; all Compose services read from .env via env_file or volume mount.
• Removed per-service .env.example files (kafka/, remote-signer/, openmeter-collector/, identity-webhook/).

Stack

identity-webhook → remote-signer → Kafka → openmeter-collector → OpenMeter/Konnect

Quick start

cp .env.example .env
$EDITOR .env
# WEBHOOK_SECRET is shared between identity-webhook and remote-signer

docker compose up -d --build kafka identity-webhook remote-signer

Signer-only alive check (no identity webhook):

# In .env: REMOTE_SIGNER_WEBHOOK_URL=
docker compose up -d --build kafka remote-signer

Validate webhook authentication response:

docker compose exec identity-webhook \
  curl -sS -X POST http://localhost:8090/authorize \
    -H "Authorization: Bearer dev-webhook-secret-change-me" \
    -H "Content-Type: application/json" \
    -d '{"headers":{"Authorization":["Bearer sk_demo_local_key"]}}'
# expected: "status":200, "auth_id":"demo-client:demo-user"

Test plan

• Fresh clone: cp .env.example .env → docker compose up succeeds
• kafka healthy; identity-webhook healthy (when started); remote-signer Up
• /authorize curl returns auth_id: demo-client:demo-user
• remote-signer logs show Using remote signer webhook URL http://identity-webhook:8090/authorize
• Signer port 127.0.0.1:8081 published; CLI port 4935 not published
• cd identity-webhook && npm test passes (32 tests)
• Collector warms ETH/USD price at startup before consuming Kafka
• Collector emits normalized CloudEvents with compound subject (output_sent > 0, output_error = 0)
• billable_usd_micros and network_fee_usd_micros meters accumulate on live Konnect

Closes #2
Closes #4

[linear-code]

ENG-115

ENG-116

ENG-114

[eliteprox]

@rickstaa This PR is now ready for review

[eliteprox]

Need to bump to 0.5.0

[Copilot]

Pull request overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated 3 comments.

Files not reviewed (1)

• identity-webhook/package-lock.json: Generated file

[Copilot]

Pull request overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated 4 comments.

Files not reviewed (1)

• identity-webhook/package-lock.json: Generated file

[Copilot]

Pull request overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated 4 comments.

Files not reviewed (1)

• identity-webhook/package-lock.json: Generated file

[Copilot]

Pull request overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated 4 comments.

Files not reviewed (1)

• identity-webhook/package-lock.json: Generated file

[Copilot]

Pull request overview

Copilot reviewed 17 out of 19 changed files in this pull request and generated 4 comments.

Files not reviewed (1)

• identity-webhook/package-lock.json: Generated file

[Copilot]

Pull request overview

Copilot reviewed 17 out of 19 changed files in this pull request and generated 2 comments.

Files not reviewed (1)

• identity-webhook/package-lock.json: Generated file