Fix Codex OTel metrics export (Grafana rejects delta temporality) and enable Claude Code telemetry #4067

Summary

Research on Codex/Claude Code OTel support, why the current Codex OTel export "isn't working", and free options for hosting an OTLP backend.

TL;DR: the Codex config itself is correct and telemetry is partially working. Logs and traces export to Grafana Cloud successfully. Metrics fail: the Grafana Cloud OTLP gateway rejects them with HTTP 400 because Codex's Rust OTel SDK exports delta temporality metrics and Grafana Cloud (Mimir) only accepts cumulative. Codex has no temporality knob, so it needs either an OTel Collector in between or a Grafana-side fix. Additionally, exported logs may be silently dropped/misplaced by backends due to a known Codex bug (timeUnixNano=0).

Diagnosis (verified locally, codex-cli 0.150.1, 2026-08-28)

~/.codex/config.toml has [otel.exporter.otlp-http] (logs), [otel.trace_exporter.otlp-http], and [otel.metrics_exporter.otlp-http] pointed at https://otlp-gateway-prod-au-southeast-1.grafana.net/otlp/v1/{logs,traces,metrics} with Basic auth — the exact shape from Grafana's official OpenAI Codex integration. Endpoints correctly include the full /v1/<signal> path (Codex passes them verbatim, unlike Claude Code).

1. Auth/endpoints are fine — empty OTLP POSTs to all three signal paths return 200/204 with the configured credentials.
2. Live run with RUST_LOG="codex_otel=trace,opentelemetry=trace,opentelemetry_otlp=trace,opentelemetry_sdk=debug" codex exec ... shows:
   • HttpLogsClient.ExportSucceeded ✅
   • HttpTracesClient.ExportSucceeded ✅
   • HttpMetricsClient.ExportFailed error="reqwest::Error { kind: Status(400, ...), url: \".../otlp/v1/metrics\" }" ❌
3. Root cause confirmed by replay: a synthetic delta-temporality histogram posted to the gateway returns exactly 400 otlp parse error: invalid temporality and type combination for metric "..."; the identical cumulative payload returns 200. Codex emits delta histograms (codex.turn.ttfm.duration_ms, codex.responses_api_*.duration_ms, ...), so every metrics batch is rejected wholesale.
4. No client-side fix exists: Codex's otlp-http exporter config only has endpoint/protocol/headers/tls — no temporality option, and the binary has no OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE support (only OTEL_EXPORTER_OTLP_*_TIMEOUT env vars are honored).

So if "not working" was judged from metrics, that matches. Logs (codex.sse_event, codex.tool_result, codex.turn_ttft, token counts, ...) and traces are being delivered — check the Loki/Tempo datasources. Caveat: openai/codex#30936 — Codex logs are exported with timeUnixNano=0 (only observedTimeUnixNano set), which some backends silently drop or mis-timestamp.

Other Codex OTel gotchas found (upstream)

• #26271: analytics.enabled = false silently disables a configured metrics_exporter (metrics default to a statsig exporter shipping to ab.chatgpt.com otherwise).
• #14465: ${VAR} env interpolation in headers does not work — tokens must be hardcoded in config.toml (secret-management concern, relates to Manage mutable Codex config safely in dotfiles #3833).
• #38655: OTLP metrics with Grafana stack showing no codex.* series — likely the same delta-temporality rejection.
• Malformed header values are silently dropped; gRPC exporter has known bugs (#6153, discussion #17687) — stick with otlp-http + binary.
• Debugging: RUST_LOG=codex_otel=debug,opentelemetry_otlp=debug, or codex -c log_dir=./.codex-log.

Fix options for Codex metrics

• OTel Collector (local on WSL, or on a free box) with the deltatocumulative processor between Codex and Grafana. Logs/traces can keep going direct (or also via the collector, which would centralize the token and let the collector fix log timestamps too).
• Grafana-side: check whether the stack can enable OTLP delta ingestion (Mimir has experimental delta support; may require a support request).
• Upstream: file a codex issue requesting configurable metrics temporality.
• Punt: drop [otel.metrics_exporter] and build usage dashboards from Loki log events (they carry token counts and durations).

Claude Code OTel support (not configured yet)

Native support, docs: https://code.claude.com/docs/en/monitoring-usage. Enable via the env block in user ~/.claude/settings.json:

{
  "env": {
    "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
    "OTEL_METRICS_EXPORTER": "otlp",
    "OTEL_LOGS_EXPORTER": "otlp",
    "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
    "OTEL_EXPORTER_OTLP_ENDPOINT": "https://otlp-gateway-prod-au-southeast-1.grafana.net/otlp",
    "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Basic <base64(instanceID:token)>",
    "OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE": "cumulative"
  }
}

• Unlike Codex, temporality is configurable — cumulative is required for Grafana Cloud (the default delta would hit the same 400).
• Endpoint is the /otlp root (the exporter appends /v1/<signal> — opposite of Codex); gateway is HTTP-only, so http/protobuf.
• Signals: metrics (claude_code.token.usage, claude_code.cost.usage, claude_code.session.count, claude_code.lines_of_code.count, claude_code.commit.count, claude_code.pull_request.count, claude_code.active_time.total, ...) and log events (claude_code.user_prompt, claude_code.api_request, claude_code.tool_result, ...). Traces are beta (CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 + OTEL_TRACES_EXPORTER).
• Useful toggles: OTEL_METRIC_EXPORT_INTERVAL (default 60000 ms), OTEL_LOG_USER_PROMPTS=1 (off by default), OTEL_METRICS_INCLUDE_SESSION_ID (default true).
• Grafana Cloud has an official Claude Code integration with a prebuilt dashboard; community dashboards: 25052, 25255.
• Same plain-text token concern as Codex (header lives in settings.json; there is an otelHeadersHelper script hook for dynamic headers, which could fetch the token from Bitwarden instead — relates to Manage mutable Codex config safely in dotfiles #3833).

Free remote OTLP hosting (researched 2026-08)

Recommendation: stay on Grafana Cloud Free; fix Codex metrics with a collector (deltatocumulative) or Grafana delta ingestion.

Tasks

• Verify Codex logs/traces are visible in Grafana Cloud (Loki/Tempo); check whether #30936 (timeUnixNano=0) affects Loki queries
• Decide Codex metrics fix: OTel Collector with deltatocumulative vs Grafana delta ingestion vs logs-only; consider filing upstream codex issue for configurable temporality
• Enable Claude Code telemetry (env block with cumulative temporality) and install the Grafana Cloud Claude Code integration dashboard
• Track OTel config in dotfiles without leaking the Grafana token (Codex can't interpolate env vars in headers — Not able to use env var for otlp headers in config openai/codex#14465; Claude Code could use otelHeadersHelper) — relates to Manage mutable Codex config safely in dotfiles #3833