Open Parliament TV - Conductor

FastAPI + HTMX web app that orchestrates the per-parliament tooling (currently DE) — queues and runs download/parse/merge/NEL/alignment/NER jobs, schedules recurring runs, streams logs, and publishes results to the OpenParliamentTV-Data-* repos.

The legacy optv bash script lives alongside and is unchanged.

Components

• Conductor (this repo) — FastAPI app at src/, runs in Docker.
• Tools — cloned into data/OpenParliamentTV-Tools. Each job spawns python -m optv.parliaments.<id>.workflow ... as a fresh subprocess with PYTHONPATH=/data/OpenParliamentTV-Tools, so Tools changes pulled before the job take effect immediately. Per-parliament metadata (name, language, periods, supported stages, official entity-dump URL) lives in optv/parliaments/<id>/manifest.yaml inside the Tools repo.
• Data repos — cloned into data/OpenParliamentTV-Data-<ID> per the parliaments.yaml config. Push uses a dedicated SSH key generated by setup.sh at config/ssh/id_ed25519, installed as a deploy key (write access) on each Data repo.

First-time setup

./scripts/setup.sh

setup.sh is idempotent. It:

1. Clones the Tools repo into data/ (skipped if present).
2. Copies config/*.sample → config/*.yaml (skipped per-file if present).
3. Generates JWT_SECRET if empty.
4. Generates a dedicated SSH key at config/ssh/id_ed25519 if missing (no passphrase — required for non-interactive container use).
5. Iterates over enabled parliaments in config/parliaments.yaml, clones each git_remote into data/OpenParliamentTV-Data-<ID>, and downloads the per-parliament entity dump (URL from the Tools manifest) if applicable.
6. Prints the public key + GitHub deploy-key URLs at the end.

Then:

• Add the printed public key as a deploy key with write access on each Data repo (URLs printed by setup.sh). Re-run setup.sh if any clones failed the first time.
• Edit config/secrets.env: GITHUB_CLIENT_ID, GITHUB_CLIENT_SECRET, BASE_URL, GIT_USER_NAME, GIT_USER_EMAIL, optionally SLACK_WEBHOOK_URL. JWT_SECRET is already filled by step 3.
• Edit config/users.yaml: add at least one GitHub username with role admin. The manager refuses to start otherwise.

Adding a new parliament after install: ship a manifest.yaml + workflow code in the Tools repo, add a git_remote + stages block to config/parliaments.yaml, re-run setup.sh (clones the new Data repo + downloads its entity dump), then docker compose restart.

Running

docker compose up -d              # dev: plain HTTP on :8000
docker compose --profile production up -d   # + nginx TLS on :443
docker compose --profile ner up -d          # + entity-fishing on :8090

Health check: curl http://localhost:8000/health

Development (no Docker)

python3.13 -m venv .venv
.venv/bin/pip install -r requirements.txt
PYTHONPATH=data/OpenParliamentTV-Tools .venv/bin/uvicorn src.main:app --reload
.venv/bin/pytest -q

Layout

src/
  api/          # JSON API routes (/api/*)
  auth/         # GitHub OAuth + JWT + RBAC
  services/     # job manager, scheduler, status tracker, log streamer, notifier
  workflow/     # adapter that runs Tools stages with progress + cancellation
  web/pages.py  # HTMX page + fragment routes
  templates/    # Jinja2 + Tailwind CDN + HTMX + Alpine
  static/
config/         # *.yaml + secrets.env (gitignored, see *.sample)
status/         # file-backed queue, current job, history, log files (gitignored)
data/           # cloned Tools + Data-* repos (gitignored)
scripts/        # setup.sh, backup.sh, compat_check.sh
nginx/          # reverse proxy config + ssl/ (mount real certs here for prod)
                # config/*.conf — drop-in server-block snippets (gitignored)
tests/          # pytest
optv            # legacy bash script — untouched

Operations

• Backup: ./scripts/backup.sh tars config/ (without secrets) and status/ into backups/optv-backup-<utc>.tar.gz.
• Compatibility check: ./scripts/compat_check.sh <session-id> runs the legacy path + a manager job over the same session and diffs processed/. Expects the manager to be reachable at $BASE_URL with $COMPAT_COOKIE set to a valid optv_token.
• Logs: live tail at /jobs/<id> (WebSocket), persisted under status/logs/jobs/<id>.log.
• Schedules: edit config/schedules.yaml — changes hot-reload within seconds via watchfiles. The UI at /schedules also toggles/triggers.
• Custom nginx config (production profile only): drop *.conf files into nginx/config/ and they're included into the nginx server block (the dir is bind-mounted read-only into the nginx container). Use this for extra location blocks, headers, redirects, etc. without touching the tracked nginx/nginx.conf. The whole directory is gitignored (only .gitkeep is tracked), so your overrides stay local. Reload with docker compose --profile production restart nginx after editing.

Troubleshooting

• Startup fails with "No parliaments configured" — copy the samples in config/ and fill them in. parliaments.yaml + users.yaml are required.
• "User '…' is not in users.yaml allow-list" — add the GitHub login (the URL login, not email) to config/users.yaml with a role.
• GitHub OAuth callback mismatch — set the GitHub OAuth app's callback URL to <BASE_URL>/auth/callback; BASE_URL is configured in secrets.env.
• Jobs stuck in queue — check docker logs openparliamenttv-conductor; the worker runs inside the app container and polls every 1s.
• Slack not firing — flip slack.enabled: true in config/notifications.yaml and confirm SLACK_WEBHOOK_URL is set.
• git push / git pull fails from container — by default the container mounts config/ssh/ (created by setup.sh). The public key (config/ssh/id_ed25519.pub) must be installed as a deploy key with write access on each Data repo. Override with SSH_KEY_DIR=~/.ssh in config/secrets.env if you want the legacy behavior.
• git commit fails with "Please tell me who you are" — set GIT_USER_NAME and GIT_USER_EMAIL in config/secrets.env. They're threaded into git -c user.name=... -c user.email=... per commit.

Security checklist

• All secrets live in config/secrets.env, which is gitignored.
• JWT_SECRET ≥ 32 chars; rotated by restarting the container.
• Auth cookie is httpOnly, SameSite=Lax, Secure when BASE_URL is HTTPS.
• RBAC enforced in every /api/* handler via require_role(...).
• Page routes redirect unauthenticated requests to /login instead of leaking 401 pages.
• Nginx profile terminates TLS; app binds to :8000 inside the container only.
• config/ is mounted read-only inside the container.