feat(docker): add Docker Compose stack for runtime services

deploy/.env.example

@@ -0,0 +1,26 @@
+# Runtime stack env for deploy/docker-compose.yml
+#
+#   cp deploy/.env.example deploy/.env
+#   $EDITOR deploy/.env
+
+# --- Identity webhook (required) ---
+REMOTE_SIGNER_WEBHOOK_URL=https://your-platform.vercel.app/api/signer/authorize
+WEBHOOK_SECRET=
+
+# --- Signer ---
+SIGNER_NETWORK=arbitrum-one-mainnet
+ETH_RPC_URL=https://arb1.arbitrum.io/rpc
+SIGNER_ETH_ADDR=
+SIGNER_HOST_PORT=8081
+SIGNER_REMOTE_DISCOVERY=0
+# ORCH_WEBHOOK_URL=
+# LIVE_AI_CAP_REPORT_INTERVAL=
+
+# --- Kafka ---
+KAFKA_GATEWAY_TOPIC=livepeer-gateway-events
+
+# --- OpenMeter / Konnect (required for collector) ---
+OPENMETER_URL=https://us.api.konghq.com/v3/openmeter
+OPENMETER_INGEST_URL=https://us.api.konghq.com/v3/openmeter/events
+OPENMETER_API_KEY=
+ETH_USD_PRICE=3500

deploy/README.md

@@ -0,0 +1,99 @@
+# clearinghouse deploy/
+
+Docker Compose stack for the clearinghouse runtime:
+**Kafka/Redpanda → go-livepeer remote signer → OpenMeter/Benthos collector → Konnect metering**.
+
+## Design decisions
+
+**No Apache DMZ.** The remote signer container runs `go-livepeer` directly —
+there is no Apache reverse proxy or `mod_authnz_jwt` layer in front of it.
+Identity validation is handled by the `-remoteSignerWebhookUrl` hook (your
+`/authorize` endpoint) and the shared `WEBHOOK_SECRET`.
+
+**CLI port not exposed.** go-livepeer's `-cliAddr` (admin/RPC) is bound to
+`127.0.0.1:4935` inside the container and is never published or mapped to the
+host — not in `docker-compose.yml`. Only the signing HTTP port
+(`8081`) is exposed.
+
+## Local stack
+
+```bash
+cp deploy/.env.example deploy/.env
+$EDITOR deploy/.env
+
+# If using clearinghouse-bootstrap (feat/go-bootstrap-cli), merge WEBHOOK_SECRET
+# and Konnect vars from .env.livepeer into deploy/.env.
+
+docker compose -f deploy/docker-compose.yml --env-file deploy/.env up -d --build
+docker compose -f deploy/docker-compose.yml --env-file deploy/.env logs -f
+docker compose -f deploy/docker-compose.yml --env-file deploy/.env down
+```
+
+Kafka + signer only (no metering):
+
+```bash
+docker compose -f deploy/docker-compose.yml --env-file deploy/.env up -d --build kafka remote-signer
+```
+
+Verify CLI port is not published:
+
+```bash
+docker compose -f deploy/docker-compose.yml --env-file deploy/.env port remote-signer 4935
+# expected: no output / error (port is not mapped)
+docker compose -f deploy/docker-compose.yml --env-file deploy/.env port remote-signer 8081
+# expected: 0.0.0.0:8081
+```
+
+## Environment variables
+
+| Variable | Required | Default | Description |
+|---|---|---|---|
+| `REMOTE_SIGNER_WEBHOOK_URL` | yes | — | Identity webhook URL (`/authorize` endpoint) |
+| `WEBHOOK_SECRET` | yes | — | Shared secret passed as `Authorization:Bearer <secret>` to the webhook (from bootstrap) |
+| `SIGNER_NETWORK` | no | `arbitrum-one-mainnet` | go-livepeer `-network` |
+| `ETH_RPC_URL` | no | public arb1 endpoint | Arbitrum RPC |
+| `SIGNER_ETH_ADDR` | no | — | Funded signer Ethereum address |
+| `SIGNER_HOST_PORT` | no | `8081` | Host port for the signing HTTP endpoint |
+| `KAFKA_GATEWAY_TOPIC` | no | `livepeer-gateway-events` | Kafka topic |
+| `OPENMETER_URL` | yes | — | OpenMeter / Konnect base URL (from bootstrap) |
+| `OPENMETER_INGEST_URL` | yes | — | Ingest endpoint (`${OPENMETER_URL}/events` for Konnect) |
+| `OPENMETER_API_KEY` | yes | — | Konnect PAT (`kpat_…`) (from bootstrap) |
+| `ETH_USD_PRICE` | no | `3500` | ETH/USD rate for Wei→USD micros conversion |
+
+## OpenMeter/Konnect bootstrap
+
+Provision meters, features, and the default pay-per-use plan before starting the collector.
+Use the Go `clearinghouse-bootstrap` CLI (`feat/go-bootstrap-cli`) or your existing Konnect setup.
+
+Creates (additive — existing pymthouse objects are untouched):
+
+| Object | Key | Purpose |
+|--------|-----|---------|
+| Meter | `network_fee_usd_micros` | Raw network cost from signer |
+| Meter | `billable_usd_micros` | Post-markup billable amount (collector phase 2) |
+| Meter | `signed_ticket_count` | Request counts |
+| Feature | `network_spend` | Trial/network spend feature |
+| Feature | `billable_spend` | Billable usage feature |
+| Plan | `clearinghouse_default_ppu` | Pay-per-use rate card |
+
+Idempotent — safe to re-run.
+
+### Two-meter billing model
+
+```text
+Signer computed_fee (wei)
+  → collector: network_fee_usd_micros   (raw network cost — observability)
+  → collector: billable_usd_micros      (network × pipeline/model markup — billing)
+       → billable_spend feature
+            → clearinghouse_default_ppu subscription per customer
+```
+
+Markup rules are defined in the bootstrap CLI catalog. Collector
+pipeline config: [`deploy/openmeter-collector/collector.yaml`](openmeter-collector/collector.yaml).
+The collector does not yet emit `billable_usd_micros` (phase 2); until then the billable meter
+stays empty while the catalog is ready.
+
+### Identity contract (collector)
+
+The collector expects Kafka `auth_id` as `client_id:external_user_id` (first-colon split).
+Konnect customer key matches that compound id (e.g. `demo-client:demo-user`).

deploy/docker-compose.yml

@@ -0,0 +1,72 @@
+# Clearinghouse deploy stack: Kafka + go-livepeer remote signer + OpenMeter collector.
+#
+# No Apache DMZ — the signer container runs go-livepeer directly. Only its signing
+# HTTP port (8081) is published to the host. The CLI/admin port (-cliAddr) is bound
+# to loopback inside the container and is never exposed or published.
+#
+# Full stack:
+#   docker compose -f deploy/docker-compose.yml --env-file .env up -d --build
+#
+# Kafka + signer only (no metering):
+#   docker compose -f deploy/docker-compose.yml --env-file .env up -d --build kafka remote-signer
+
+services:
+  kafka:
+    build:
+      context: ..
+      dockerfile: deploy/kafka/Dockerfile
+    restart: unless-stopped
+    environment:
+      KAFKA_ADVERTISED_ADDR: kafka:9092
+    healthcheck:
+      test: ["CMD-SHELL", "rpk cluster health | grep -q 'Healthy'"]
+      interval: 10s
+      timeout: 5s
+      retries: 12
+      start_period: 15s
+
+  remote-signer:
+    build:
+      context: ..
+      dockerfile: deploy/remote-signer/Dockerfile
+    restart: unless-stopped
+    depends_on:
+      kafka:
+        condition: service_healthy
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    ports:
+      - "${SIGNER_HOST_PORT:-8081}:8081"
+    volumes:
+      - ${SIGNER_DATA_DIR:-./data}:/data
+    environment:
+      SIGNER_NETWORK: ${SIGNER_NETWORK:-arbitrum-one-mainnet}
+      SIGNER_PORT: "8081"
+      ETH_RPC_URL: ${ETH_RPC_URL:-https://arb1.arbitrum.io/rpc}
+      SIGNER_ETH_ADDR: ${SIGNER_ETH_ADDR:-}
+      SIGNER_REMOTE_DISCOVERY: ${SIGNER_REMOTE_DISCOVERY:-0}
+      ORCH_WEBHOOK_URL: ${ORCH_WEBHOOK_URL:-}
+      LIVE_AI_CAP_REPORT_INTERVAL: ${LIVE_AI_CAP_REPORT_INTERVAL:-}
+      KAFKA_BROKERS: kafka:9092
+      KAFKA_GATEWAY_TOPIC: ${KAFKA_GATEWAY_TOPIC:-livepeer-gateway-events}
+      REMOTE_SIGNER_WEBHOOK_URL: ${REMOTE_SIGNER_WEBHOOK_URL}
+      WEBHOOK_SECRET: ${WEBHOOK_SECRET}
+
+  openmeter-collector:
+    build:
+      context: ..
+      dockerfile: deploy/openmeter-collector/Dockerfile
+    restart: unless-stopped
+    depends_on:
+      kafka:
+        condition: service_healthy
+    environment:
+      KAFKA_BROKERS: kafka:9092
+      KAFKA_GATEWAY_TOPIC: ${KAFKA_GATEWAY_TOPIC:-livepeer-gateway-events}
+      OPENMETER_URL: ${OPENMETER_URL}
+      OPENMETER_INGEST_URL: ${OPENMETER_INGEST_URL}
+      OPENMETER_API_KEY: ${OPENMETER_API_KEY}
+      ETH_USD_PRICE: ${ETH_USD_PRICE:-3500}
+
+# Signer datadir is bind-mounted from deploy/data (see SIGNER_DATA_DIR).
+# Copy keystore + .eth-password from pymthouse/data before first run.

deploy/kafka/Dockerfile

@@ -0,0 +1,13 @@
+# Clearinghouse Kafka bus (Redpanda dev-container mode).
+# KAFKA_ADVERTISED_ADDR defaults to kafka:9092 for local docker-compose.
+FROM docker.redpanda.com/redpandadata/redpanda:v24.2.4
+
+ENV CLUSTER_ID=6ac74f21-e6d3-453b-b5d6-de4fb811e8bb
+ENV KAFKA_ADVERTISED_ADDR=kafka:9092
+
+USER root
+COPY deploy/kafka/entrypoint.sh /clearinghouse-entrypoint.sh
+RUN chmod +x /clearinghouse-entrypoint.sh
+USER redpanda
+
+ENTRYPOINT ["/clearinghouse-entrypoint.sh"]

deploy/kafka/entrypoint.sh

@@ -0,0 +1,15 @@
+#!/bin/sh
+set -eu
+
+ADVERTISED="${KAFKA_ADVERTISED_ADDR:-kafka:9092}"
+
+# Base image ENTRYPOINT is /entrypoint.sh -> exec /usr/bin/rpk "$@".
+# Shell-form CMD becomes /bin/sh -c ..., which rpk rejects (-c flag).
+# Exec-form CMD works; entrypoint allows runtime KAFKA_ADVERTISED_ADDR override.
+exec /usr/bin/rpk redpanda start \
+  --kafka-addr "internal://0.0.0.0:9092" \
+  --advertise-kafka-addr "internal://${ADVERTISED}" \
+  --mode dev-container \
+  --smp 1 \
+  --memory 512M \
+  --overprovisioned

deploy/openmeter-collector/Dockerfile

@@ -0,0 +1,6 @@
+# OpenMeter Benthos collector: Kafka create_signed_ticket events -> Konnect/OpenMeter ingest.
+FROM ghcr.io/openmeterio/benthos-collector:latest
+
+COPY deploy/openmeter-collector/collector.yaml /config.yaml
+
+CMD ["-c", "/config.yaml"]

deploy/openmeter-collector/collector.yaml

@@ -0,0 +1,74 @@
+# OpenMeter Collector: go-livepeer remote signer Kafka -> hosted OpenMeter/Konnect.
+#
+# Notes:
+# - Topic carries multiple event types; filter to create_signed_ticket only.
+# - auth_id is expected as "client_id:external_user_id" from the signer webhook response.
+# - Uses first-colon split semantics to preserve compatibility with builder-sdk.
+# - Konnect ingest: POST {OPENMETER_INGEST_URL} (set to ${OPENMETER_URL}/events).
+#   For self-hosted OpenMeter set OPENMETER_INGEST_URL to ${OPENMETER_URL}/api/v1/events.
+
+input:
+  kafka:
+    addresses:
+      - ${KAFKA_BROKERS}
+    topics:
+      - ${KAFKA_GATEWAY_TOPIC}
+    consumer_group: openmeter-collector
+    start_from_oldest: true
+
+pipeline:
+  processors:
+    - mapping: |
+        root = if this.type != "create_signed_ticket" { deleted() }
+
+    - mapping: |
+        let data = this.data.or({})
+        let auth_id = $data.auth_id.or("").string().trim()
+        root = if $auth_id == "" { deleted() }
+
+        let colon = $auth_id.index_of(":")
+        let is_compound = $colon > 0 && $colon < $auth_id.length() - 1
+        let client_id = if $is_compound { $auth_id.slice(0, $colon) } else { $auth_id }
+        let external_user_id = if $is_compound { $auth_id.slice($colon + 1) } else { $auth_id }
+
+        let eth_usd = env("ETH_USD_PRICE").number().or(3500)
+        let fee_wei = $data.computed_fee.number().or(0)
+        let fee_usd_micros = ($fee_wei * $eth_usd / 1000000000000).round()
+
+        root = {
+          "specversion": "1.0",
+          "type": "create_signed_ticket",
+          "id": $data.request_id.or(uuid_v4()),
+          "source": "go-livepeer-remote-signer",
+          "subject": $auth_id,
+          "time": $data.current_time.or(now()),
+          "data": {
+            "client_id": $client_id,
+            "external_user_id": $external_user_id,
+            "network_fee_usd_micros": $fee_usd_micros,
+            "pipeline": if $data.pipeline != "" && $data.pipeline != null { $data.pipeline } else { "unknown" },
+            "model_id": if $data.model_id != "" && $data.model_id != null { $data.model_id } else { "unknown" },
+            "pixels": $data.pixels.string().or("0"),
+            "fee_wei": $data.computed_fee.or("0"),
+            "gateway_request_id": $data.request_id,
+            "auth_id": $auth_id,
+          }
+        }
+
+    - catch:
+        - log:
+            level: ERROR
+            message: "signed_ticket mapping failed: ${! error() }"
+        - mapping: root = deleted()
+
+output:
+  http_client:
+    url: ${OPENMETER_INGEST_URL}
+    verb: POST
+    headers:
+      Authorization: "Bearer ${OPENMETER_API_KEY}"
+      Content-Type: application/cloudevents+json
+    successful_on:
+      - 200
+      - 202
+      - 204

deploy/remote-signer/Dockerfile

@@ -0,0 +1,25 @@
+# go-livepeer remote signer — extends livepeer/go-livepeer:latest (CUDA libs required by binary).
+# Do not copy livepeer into debian:slim; the binary links libnppig.so and other CUDA NPP libs.
+#
+# No Apache DMZ: go-livepeer runs directly in this container. Only the signing HTTP port
+# is EXPOSE'd and published. The CLI/admin port (-cliAddr) is bound to loopback inside
+# the container and is never exposed or published — not here, not in docker-compose.
+FROM livepeer/go-livepeer:latest
+
+RUN mkdir -p /data
+
+COPY deploy/remote-signer/entrypoint.sh /entrypoint.sh
+RUN chmod +x /entrypoint.sh
+
+ENV SIGNER_NETWORK=arbitrum-one-mainnet
+ENV SIGNER_PORT=8081
+ENV ETH_RPC_URL=https://arb1.arbitrum.io/rpc
+ENV SIGNER_ETH_ADDR=""
+ENV SIGNER_REMOTE_DISCOVERY=0
+ENV KAFKA_BROKERS=kafka:9092
+ENV KAFKA_GATEWAY_TOPIC=livepeer-gateway-events
+
+EXPOSE 8081
+VOLUME ["/data"]
+
+ENTRYPOINT ["/entrypoint.sh"]

deploy/remote-signer/entrypoint.sh

@@ -0,0 +1,53 @@
+#!/bin/sh
+set -eu
+
+SIGNER_NETWORK="${SIGNER_NETWORK:-arbitrum-one-mainnet}"
+SIGNER_PORT="${SIGNER_PORT:-8081}"
+ETH_RPC_URL="${ETH_RPC_URL:-https://arb1.arbitrum.io/rpc}"
+KAFKA_BROKERS="${KAFKA_BROKERS:-kafka:9092}"
+KAFKA_GATEWAY_TOPIC="${KAFKA_GATEWAY_TOPIC:-livepeer-gateway-events}"
+
+if [ -z "${REMOTE_SIGNER_WEBHOOK_URL:-}" ]; then
+  echo "entrypoint: REMOTE_SIGNER_WEBHOOK_URL is required (identity webhook URL)" >&2
+  exit 1
+fi
+if [ -z "${WEBHOOK_SECRET:-}" ]; then
+  echo "entrypoint: WEBHOOK_SECRET is required" >&2
+  exit 1
+fi
+
+if [ ! -f /data/.eth-password ]; then
+  echo "" >/data/.eth-password
+fi
+
+set -- \
+  -remoteSigner \
+  "-network=${SIGNER_NETWORK}" \
+  "-httpAddr=0.0.0.0:${SIGNER_PORT}" \
+  "-cliAddr=127.0.0.1:4935" \
+  "-ethUrl=${ETH_RPC_URL}" \
+  "-ethPassword=/data/.eth-password" \
+  "-datadir=/data" \
+  -v=99 \
+  "-remoteSignerWebhookUrl=${REMOTE_SIGNER_WEBHOOK_URL}" \
+  "-remoteSignerWebhookHeaders=Authorization:Bearer ${WEBHOOK_SECRET}" \
+  -monitor \
+  "-kafkaBootstrapServers=${KAFKA_BROKERS}" \
+  "-kafkaGatewayTopic=${KAFKA_GATEWAY_TOPIC}"
+
+if [ -n "${SIGNER_ETH_ADDR:-}" ]; then
+  set -- "$@" "-ethAcctAddr=${SIGNER_ETH_ADDR}"
+fi
+
+if [ "${SIGNER_REMOTE_DISCOVERY:-0}" = "1" ] || [ "${SIGNER_REMOTE_DISCOVERY:-0}" = "true" ]; then
+  set -- "$@" -remoteDiscovery=true
+  if [ -n "${ORCH_WEBHOOK_URL:-}" ]; then
+    set -- "$@" "-orchWebhookUrl=${ORCH_WEBHOOK_URL}"
+  fi
+  if [ -n "${LIVE_AI_CAP_REPORT_INTERVAL:-}" ]; then
+    set -- "$@" "-liveAICapReportInterval=${LIVE_AI_CAP_REPORT_INTERVAL}"
+  fi
+fi
+
+echo "entrypoint: starting livepeer remote-signer on 0.0.0.0:${SIGNER_PORT}" >&2
+exec /usr/local/bin/livepeer "$@"