feat(docker): add Docker Compose stack for runtime services

.gitignore

@@ -0,0 +1,4 @@
+kafka/.env
+remote-signer/.env
+openmeter-collector/.env
+remote-signer/data/

README.md

@@ -0,0 +1,138 @@
+# clearinghouse
+
+Docker Compose stack for the clearinghouse runtime:
+**Redpanda → go-livepeer remote signer → OpenMeter/Benthos collector → Konnect metering**.
+
+## Components
+
+| Service | Role | Docs |
+| --- | --- | --- |
+| **Redpanda** (`kafka`) | Kafka-compatible event bus. The signer publishes gateway events; the collector consumes them. | [Redpanda docs](https://docs.redpanda.com/) |
+| **go-livepeer remote signer** (`remote-signer`) | Signs Livepeer payment tickets and emits `create_signed_ticket` events to Kafka. | [go-livepeer](https://github.com/livepeer/go-livepeer) |
+| **OpenMeter collector** (`openmeter-collector`) | Benthos pipeline: filters Kafka events, converts fees to USD micros, POSTs CloudEvents to OpenMeter ingest. | [OpenMeter collector](https://openmeter.io/docs/collectors) |
+| **Konnect / OpenMeter** (external) | Hosted metering and billing API. Set `OPENMETER_INGEST_URL` to your ingest endpoint. | [Konnect OpenMeter](https://docs.konghq.com/konnect/openmeter/), [self-hosted OpenMeter](https://openmeter.io/docs/deploy/kubernetes) |
+
+Data flow:
+
+```text
+Signer HTTP request
+  → identity webhook (/authorize)
+  → signed ticket + Kafka create_signed_ticket event
+  → collector transforms event
+  → OpenMeter ingest API
+```
+
+## Design decisions
+
+**Redpanda over Apache Kafka.** The stack uses Redpanda as the Kafka-compatible broker. Redpanda runs as a single-binary dev container with no ZooKeeper dependency and faster local startup.
+
+**Identity & auth.** The signer container runs `go-livepeer` directly. In the normal path, every signing request is authorized by go-livepeer's `-remoteSignerWebhookUrl` hook, which calls your `/authorize` endpoint with `Authorization: Bearer <WEBHOOK_SECRET>` — no reverse proxy or gateway in front of the signer. For local alive checks only, leave `REMOTE_SIGNER_WEBHOOK_URL` empty to omit the webhook hook.
+
+**CLI port not exposed.** go-livepeer's `-cliAddr` (admin/RPC) is bound to `127.0.0.1:4935` inside the container and is never published or mapped to the host. Only the signing HTTP port (`8081`) is exposed.
+
+**Per-service configuration.** Each service reads a local `.env` file mounted at `/service/.env` and sourced by its entrypoint. Copy the `.env.example` in each service directory before starting the stack.
+
+## Local stack
+
+### 1. Quick check — Kafka + signer
+
+Start here before wiring identity or metering. This runs only the Kafka broker and remote signer so you can confirm the core path is alive.
+
+```bash
+cp kafka/.env.example kafka/.env
+cp remote-signer/.env.example remote-signer/.env
+$EDITOR remote-signer/.env
+# For a local alive check without an identity webhook:
+#   REMOTE_SIGNER_WEBHOOK_URL=
+#   WEBHOOK_SECRET=
+
+docker compose up -d --build kafka remote-signer
+docker compose logs -f remote-signer
+```
+
+Expected result: `remote-signer` starts cleanly, connects to Kafka, and serves the signing HTTP port.
+
+Verify CLI port is not published:
+
+```bash
+docker compose port remote-signer 4935
+# expected: no output / error (port is not mapped)
+docker compose port remote-signer 8081
+# expected: 0.0.0.0:8081
+```
+
+### 2. Full stack — add metering
+
+After the quick check passes, add the OpenMeter collector and hosted metering configuration. Provision OpenMeter meters/features (see [OpenMeter/Konnect bootstrap](#openmeterkonnect-bootstrap)), then configure the collector:
+
+```bash
+cp openmeter-collector/.env.example openmeter-collector/.env
+$EDITOR openmeter-collector/.env
+
+docker compose up -d --build
+docker compose logs -f
+docker compose down
+```
+
+## Environment variables
+
+Each service documents its variables in its own `.env.example`:
+
+| Service | Config file | Key variables |
+| --- | --- | --- |
+| `kafka` | [`kafka/.env.example`](kafka/.env.example) | `KAFKA_ADVERTISED_ADDR` |
+| `remote-signer` | [`remote-signer/.env.example`](remote-signer/.env.example) | `REMOTE_SIGNER_WEBHOOK_URL`, `WEBHOOK_SECRET`, `SIGNER_*`, `KAFKA_BROKERS`, `KAFKA_GATEWAY_TOPIC` |
+| `openmeter-collector` | [`openmeter-collector/.env.example`](openmeter-collector/.env.example) | `KAFKA_BROKERS`, `KAFKA_GATEWAY_TOPIC`, `OPENMETER_INGEST_URL`, `OPENMETER_API_KEY`, `ETH_USD_PRICE` |
+
+Signer state (keystore, `.eth-password`, chain DB) is stored under [`remote-signer/data/`](remote-signer/data/), bind-mounted to `/data` in the container.
+
+```bash
+mkdir -p remote-signer/data/keystore
+cp /path/to/your/keystore/* remote-signer/data/keystore/
+cp /path/to/your/.eth-password remote-signer/data/.eth-password
+
+cp remote-signer/.env.example remote-signer/.env
+$EDITOR remote-signer/.env
+```
+
+Set `SIGNER_ETH_KEYSTORE_PATH=/data/keystore` (container path) and `SIGNER_ETH_ADDR` to your funded signer address. If `SIGNER_ETH_KEYSTORE_PATH` is unset, the entrypoint uses `/data/keystore` when that directory exists.
+
+To change the host signing port from `8081`, use a Compose override file.
+
+## OpenMeter/Konnect bootstrap
+
+Provision meters, features, and the default pay-per-use plan before starting the collector.
+Use the Go `clearinghouse-bootstrap` CLI or your existing Konnect setup.
+
+Creates:
+
+| Object | Key | Purpose |
+| --- | --- | --- |
+| Meter | `network_fee_usd_micros` | Raw network cost from signer |
+| Meter | `billable_usd_micros` | Post-markup billable amount (collector phase 2) |
+| Meter | `signed_ticket_count` | Request counts |
+| Feature | `network_spend` | Trial/network spend feature |
+| Feature | `billable_spend` | Billable usage feature |
+| Plan | `clearinghouse_default_ppu` | Pay-per-use rate card |
+
+Idempotent — safe to re-run.
+
+### Two-meter billing model
+
+```text
+Signer computed_fee (wei)
+  → collector: network_fee_usd_micros   (raw network cost — observability)
+  → collector: billable_usd_micros      (network × pipeline/model markup — billing)
+       → billable_spend feature
+            → clearinghouse_default_ppu subscription per customer
+```
+
+Markup rules are defined in the bootstrap CLI catalog. Collector
+pipeline config: [`openmeter-collector/collector.yaml`](openmeter-collector/collector.yaml).
+The collector does not yet emit `billable_usd_micros` (phase 2); until then the billable meter
+stays empty while the catalog is ready.
+
+### Identity contract (collector)
+
+The collector expects Kafka `auth_id` as `client_id:external_user_id` (first-colon split).
+Konnect customer key matches that compound id (e.g. `demo-client:demo-user`).

docker-compose.yml

@@ -0,0 +1,58 @@
+# Clearinghouse stack: Redpanda + go-livepeer remote signer + OpenMeter collector.
+#
+# Redpanda is the Kafka-compatible broker. The signer container runs go-livepeer directly —
+# only its signing HTTP port (8081) is published to the host. The CLI/admin port (-cliAddr)
+# is bound to loopback inside the container and is never exposed or published.
+#
+# Per-service config: copy each service's .env.example to .env before starting.
+#
+# Full stack:
+#   docker compose up -d --build
+#
+# Kafka + signer only (no metering):
+#   docker compose up -d --build kafka remote-signer
+
+services:
+  kafka:
+    build:
+      context: .
+      dockerfile: kafka/Dockerfile
+    restart: unless-stopped
+    volumes:
+      - ./kafka/.env:/service/.env:ro
+    healthcheck:
+      test: ["CMD-SHELL", "rpk cluster health | grep -q 'Healthy'"]
+      interval: 10s
+      timeout: 5s
+      retries: 12
+      start_period: 15s
+
+  remote-signer:
+    build:
+      context: .
+      dockerfile: remote-signer/Dockerfile
+    restart: unless-stopped
+    depends_on:
+      kafka:
+        condition: service_healthy
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    ports:
+      - "8081:8081"
+    volumes:
+      - ./remote-signer/.env:/service/.env:ro
+      - ./remote-signer/data:/data
+
+  openmeter-collector:
+    build:
+      context: .
+      dockerfile: openmeter-collector/Dockerfile
+    restart: unless-stopped
+    depends_on:
+      kafka:
+        condition: service_healthy
+    volumes:
+      - ./openmeter-collector/.env:/service/.env:ro
+
+# Signer datadir is bind-mounted from remote-signer/data.
+# Copy keystore + .eth-password into remote-signer/data before first run.

kafka/.env.example

@@ -0,0 +1,5 @@
+# Kafka (Redpanda) service env — copied to kafka/.env at runtime.
+#
+#   cp kafka/.env.example kafka/.env
+
+KAFKA_ADVERTISED_ADDR=kafka:9092

kafka/Dockerfile

@@ -0,0 +1,9 @@
+# Clearinghouse Kafka bus (Redpanda dev-container mode).
+FROM docker.redpanda.com/redpandadata/redpanda:v24.2.4
+
+USER root
+COPY kafka/entrypoint.sh /clearinghouse-entrypoint.sh
+RUN chmod +x /clearinghouse-entrypoint.sh
+USER redpanda
+
+ENTRYPOINT ["/clearinghouse-entrypoint.sh"]

kafka/entrypoint.sh

@@ -0,0 +1,29 @@
+#!/bin/sh
+set -eu
+
+if [ -f /service/.env ]; then
+  set -a
+  # shellcheck disable=SC1091
+  . /service/.env
+  set +a
+fi
+
+ADVERTISED="${KAFKA_ADVERTISED_ADDR:-kafka:9092}"
+
+# NOTE: Deployment: See https://github.com/livepeer/clearinghouse/issues/43 for tracking.
+# The --mode dev-container flag below launches Redpanda in "development container" mode:
+# - No security/encryption (PLAINTEXT)
+# - Not for production! This enables fast local startup, disables most persistence/durability,
+#   and relaxes networking for development and Docker Compose stacks.
+# Update this script if production requirements change.
+
+# Base image ENTRYPOINT is /entrypoint.sh -> exec /usr/bin/rpk "$@".
+# Shell-form CMD becomes /bin/sh -c ..., which rpk rejects (-c flag).
+# Exec-form CMD works; entrypoint allows runtime KAFKA_ADVERTISED_ADDR override.
+exec /usr/bin/rpk redpanda start \
+  --kafka-addr "internal://0.0.0.0:9092" \
+  --advertise-kafka-addr "internal://${ADVERTISED}" \
+  --mode dev-container \
+  --smp 1 \
+  --memory 512M \
+  --overprovisioned

openmeter-collector/.env.example

@@ -0,0 +1,15 @@
+# OpenMeter collector service env — copied to openmeter-collector/.env at runtime.
+#
+#   cp openmeter-collector/.env.example openmeter-collector/.env
+#   $EDITOR openmeter-collector/.env
+
+# --- Kafka (event consume) ---
+KAFKA_BROKERS=kafka:9092
+KAFKA_GATEWAY_TOPIC=livepeer-gateway-events
+
+# --- OpenMeter / Konnect (required) ---
+# Konnect: https://<region>.api.konghq.com/v3/openmeter/events
+# Self-hosted: https://<host>/api/v1/events
+OPENMETER_INGEST_URL=https://us.api.konghq.com/v3/openmeter/events
+OPENMETER_API_KEY=
+ETH_USD_PRICE=3500

openmeter-collector/Dockerfile

@@ -0,0 +1,8 @@
+# OpenMeter Benthos collector: Kafka create_signed_ticket events -> Konnect/OpenMeter ingest.
+FROM ghcr.io/openmeterio/benthos-collector:main-6b60ab6-1782310960
+
+COPY openmeter-collector/collector.yaml /config.yaml
+COPY openmeter-collector/entrypoint.sh /entrypoint.sh
+RUN chmod +x /entrypoint.sh
+
+ENTRYPOINT ["/entrypoint.sh"]

openmeter-collector/collector.yaml

@@ -0,0 +1,77 @@
+# OpenMeter Collector: go-livepeer remote signer Kafka -> hosted OpenMeter/Konnect.
+#
+# Notes:
+# - Topic carries multiple event types; filter to create_signed_ticket only.
+# - auth_id is expected as "client_id:external_user_id" from the signer webhook response.
+# - Uses first-colon split semantics to preserve compatibility with builder-sdk.
+# - Konnect ingest: POST OPENMETER_INGEST_URL (e.g. https://<region>.api.konghq.com/v3/openmeter/events).
+#   For self-hosted OpenMeter use https://<host>/api/v1/events.
+
+input:
+  kafka:
+    addresses:
+      - ${KAFKA_BROKERS}
+    topics:
+      - ${KAFKA_GATEWAY_TOPIC}
+    consumer_group: openmeter-collector
+    start_from_oldest: true
+
+pipeline:
+  processors:
+    - mapping: |
+        root = if this.type != "create_signed_ticket" { deleted() }
+
+    - mapping: |
+        let data = this.data.or({})
+        let auth_id = $data.auth_id.or("").string().trim()
+        root = if $auth_id == "" { deleted() }
+
+        let colon = $auth_id.index_of(":")
+        let is_compound = $colon > 0 && $colon < $auth_id.length() - 1
+        let client_id = if $is_compound { $auth_id.slice(0, $colon) } else { $auth_id }
+        let external_user_id = if $is_compound { $auth_id.slice($colon + 1) } else { $auth_id }
+
+        let eth_usd_str = env("ETH_USD_PRICE").or("").string().trim()
+        let _ = if $eth_usd_str == "" { throw("ETH_USD_PRICE is required") }
+        let eth_usd = $eth_usd_str.number()
+        let _ = if $eth_usd <= 0 { throw("ETH_USD_PRICE must be a positive number") }
+        let fee_wei = $data.computed_fee.number().or(0)
+        let fee_usd_micros = ($fee_wei * $eth_usd / 1000000000000).round()
+
+        root = {
+          "specversion": "1.0",
+          "type": "create_signed_ticket",
+          "id": $data.request_id.or(uuid_v4()),
+          "source": "go-livepeer-remote-signer",
+          "subject": $auth_id,
+          "time": $data.current_time.or(now()),
+          "data": {
+            "client_id": $client_id,
+            "external_user_id": $external_user_id,
+            "network_fee_usd_micros": $fee_usd_micros,
+            "pipeline": if $data.pipeline != "" && $data.pipeline != null { $data.pipeline } else { "unknown" },
+            "model_id": if $data.model_id != "" && $data.model_id != null { $data.model_id } else { "unknown" },
+            "pixels": $data.pixels.string().or("0"),
+            "fee_wei": $data.computed_fee.or("0"),
+            "gateway_request_id": $data.request_id,
+            "auth_id": $auth_id,
+          }
+        }
+
+    - catch:
+        - log:
+            level: ERROR
+            message: "signed_ticket mapping failed: ${! error() }"
+        - mapping: root = deleted()
+
+output:
+  http_client:
+    url: ${OPENMETER_INGEST_URL}
+    verb: POST
+    headers:
+      Authorization: "Bearer ${OPENMETER_API_KEY}"
+      Content-Type: application/cloudevents+json
+    successful_on:
+      - 200
+      - 202
+      - 204

openmeter-collector/entrypoint.sh

@@ -0,0 +1,11 @@
+#!/bin/sh
+set -eu
+
+if [ -f /service/.env ]; then
+  set -a
+  # shellcheck disable=SC1091
+  . /service/.env
+  set +a
+fi
+
+exec /usr/local/bin/benthos -c /config.yaml

remote-signer/.env.example

@@ -0,0 +1,29 @@
+# Remote signer service env — copied to remote-signer/.env at runtime.
+#
+#   cp remote-signer/.env.example remote-signer/.env
+#   $EDITOR remote-signer/.env
+
+# --- Identity webhook ---
+REMOTE_SIGNER_WEBHOOK_URL=https://your-platform.vercel.app/api/signer/authorize
+WEBHOOK_SECRET=
+# Leave REMOTE_SIGNER_WEBHOOK_URL empty to start signer without webhook authorization.
+# REMOTE_SIGNER_WEBHOOK_URL should always be set in production.
+
+# --- Signer ---
+# Paths below are inside the container. Host files live under remote-signer/data/
+# (bind-mounted to /data). Copy your keystore directory and .eth-password there
+# before first run.
+SIGNER_NETWORK=arbitrum-one-mainnet
+SIGNER_PORT=8081
+ETH_RPC_URL=https://arb1.arbitrum.io/rpc
+SIGNER_ETH_ADDR=
+# Directory or keyfile under /data — e.g. /data/keystore or /data/keystore/UTC--...json
+# Defaults to /data/keystore when that directory exists and this is unset.
+SIGNER_ETH_KEYSTORE_PATH=/data/keystore
+SIGNER_REMOTE_DISCOVERY=0
+# ORCH_WEBHOOK_URL=
+# LIVE_AI_CAP_REPORT_INTERVAL=
+
+# --- Kafka (event publish) ---
+KAFKA_BROKERS=kafka:9092
+KAFKA_GATEWAY_TOPIC=livepeer-gateway-events