From f3cffdc56a64fd01e68c3fcc69c1a86924b7bbd1 Mon Sep 17 00:00:00 2001 From: jsn Date: Thu, 16 Jul 2026 17:20:23 -0500 Subject: [PATCH 1/3] docs: Add CLAUDE.md for Claude Code guidance Orients Claude Code on the Django layout, the domain model, the current Docker Swarm deployment, and the in-flight move to the CloudVPS Compose Deploy toolkit under T402055. Every concrete claim was validated against the code, the project wiki, and Phabricator before writing; the deployment section points at the toolkit contract so changes favor the migration rather than the swarm-era setup. Assisted-by: Claude Opus 4.8 --- CLAUDE.md | 163 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 163 insertions(+) create mode 100644 CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..91b85cc --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,163 @@ +# CLAUDE.md + +Guidance for Claude Code working in this repository. + +> **Sandboxing**: when launched via the `claude` alias from +> [wmf-claude](https://gitlab.wikimedia.org/repos/product-safety-and-integrity/wmf-claude), +> Claude Code runs inside a [nono](https://github.com/always-further/nono) sandbox. +> A denied path or domain is nono enforcing the wmf-engineer profile, not a bug to +> work around. + +## What this is + +Wikilink (repo `externallinks`) tracks link additions to Wikimedia projects made +through Wikimedia partnerships, primarily The Wikipedia Library. It is a Django +app **hosted on GitHub** (https://github.com/WikipediaLibrary/externallinks), +**not Gerrit**, deployed on a Cloud VPS instance and served at wikilink.wmflabs.org. +Issues: the [Wikilink-Tool Phabricator tag](https://phabricator.wikimedia.org/tag/wikilink-tool/). + +Layout: + +- `./extlinks/` is the Django project. Apps: `links` (LinkEvent, URLPattern, + LinkSearchTotal), `organisations` (Organisation, Collection, User), `programs` + (Program), `aggregates` (rollups), `healthcheck`, `common`, plus `templates/` + and `logs/`. Settings split under `extlinks/settings/`: `base`, `local` (dev + default), `production`, `helpers`, `logging`. +- `./requirements/`: `django.txt` (production) and `local.txt` (dev overlay: + debug toolbar, pudb). No separate test file. +- `./Dockerfile`: multi-stage, targets `eventstream` (base: Python 3.11 + deps), + `externallinks` (adds gunicorn), `cron`. No virtualenv. +- `./docker-compose.yml`: single file, no overlays yet. Services `externallinks` + (gunicorn), `eventstream` (stream consumer), `crons`, `db` (MariaDB 10), + `cache` (memcached), `nginx`. +- `./bin/`: operational scripts (`gunicorn.sh`, `cron.sh`, `django_wait_for_db.sh`, + `restore.sh`, `swarm_update.sh`, `example_data.sh`, ...). No venv wrappers. +- `./backup.py` + `./backup/`: gzipped SQL-dump backups, 14-day retention, + filelock-guarded. `./crontab` schedules them alongside the aggregate fills, + `linksearchtotal_collect`, `linkevents_archive`, and hourly `users_update_lists`. +- `./db.cnf`, `./nginx.conf`, `./template.env`, `./django_wait_for_migrations.py` + (30s migration wait, wraps the CI test run), `./wiki-list.csv`, `./static/`. + +## Domain and data flow + +**Program** groups **Organisation**s; each organisation owns **Collection**s; +each collection tracks one or more **URLPattern**s. Two independent sources feed +the stats: + +- **Link events** from the Wikimedia page-links-change EventStream, consumed by + `extlinks/links/management/commands/linkevents_collect.py` (SSE; `--historical` + resumes from the last stored event; 5-minute retry on drops). Matched adds and + removes become `LinkEvent` rows. `users_update_lists` refreshes each org's + authorized-user list hourly (from the Library Card Platform's AuthorizedUsers + view) so events can be flagged `on_user_list`. +- **Total link counts** from the MediaWiki externallinks table, pulled per URL + pattern per day into `LinkSearchTotal` by `linksearchtotal_collect`. + +`aggregates/` precomputes the rollups the org/collection/program pages render; +`healthcheck/` exposes per-cron health endpoints so a stalled collector or +aggregate job is visible. Fuller writeups: the wiki's +[App structure](https://github.com/WikipediaLibrary/externallinks/wiki/App-structure) +and [Data tracking](https://github.com/WikipediaLibrary/externallinks/wiki/Data-tracking) +pages. + +## Deployment, and the migration in flight + +Production today runs on **Docker Swarm** (`bin/swarm_update.sh` does +`docker stack deploy`; docker-compose is pinned to 1.25.5). The current build is +documented on the wiki's +[Debian Server Setup](https://github.com/WikipediaLibrary/externallinks/wiki/Debian-Server-Setup) +page. + +That VM is a deprecated Bullseye instance being replaced under **T402055** +(deadline 2026-07-31), and the replacement moves deployment to the +[CloudVPS Compose Deploy toolkit](https://gitlab.wikimedia.org/repos/modtools/cloudvps-compose-deploy) +(rootless docker compose), the same toolkit TWLight already runs. Its +contract sets the target layout: + +- One VM runs one environment, and that environment is a git branch (`staging` or + `production`) named by the VM's `env` instance metadata. +- `COMPOSE_FILE` pins `docker-compose.yml` (base) plus `docker-compose.deploy.yml` + (deploy overlay); `docker-compose.override.yml` is left free as the local-dev + overlay so a bare `docker compose up` stays a dev command. +- `template.env` is copied to `.env` on first setup; scheduled jobs live in + `conf/crontab`. + +**Guidance:** favor the migration direction. This repo is already clean (no +virtualenv-in-container, one compose file, a modest `bin/`); keep it that way, and +don't add swarm-specific coupling, new overlays, or wrapper-script sprawl. Both +the swarm path and the new compose path must work during the rollout, so prefer +additive changes (a new `docker-compose.deploy.yml` alongside the existing file) +over in-place rewrites of anything the live swarm deployment still depends on. + +## Conventions + +- **Python 3.11, Django 4.2, MariaDB 10** (mysqlclient), memcached (pymemcache), + gunicorn (7 gthread workers, see `bin/gunicorn.sh`). Event ingestion via + sseclient; retries via tenacity. Server-rendered templates, no node toolchain. +- **PEP 8 / Django style**, four-space indent. No formatter or linter is enforced + (no black, flake8, pyproject.toml, setup.cfg, .flake8, or pre-commit config). + Match surrounding code; the dominant local pattern wins over a strict reading of + any one style guide. If you run black, run it only on files you already touched. +- Imports: stdlib, third-party, first-party (`extlinks.*`), local. +- **Keep the diff small.** Reuse existing helpers and management commands rather + than inventing parallel ones. Split needed refactors into their own commit ahead + of the feature commit. +- This runs on Cloud VPS, not core production, so absolute scale differs from + MediaWiki proper, but the principles hold: no synchronous HTTP in the request + path, batch DB reads, defer slow work to cron. The stream consumer and the + aggregate fills are the hot spots. +- Logs go to container stdout; `docker compose logs externallinks` (or the swarm + equivalent for now) is the first place to look. Slow queries show up in the `db` + container logs. + +## Testing + +Tests are per-app `tests.py` files (no `tests/` dir). Run inside the container: + +```bash +docker exec -ti externallinks-externallinks-1 python manage.py test +``` + +CI runs `docker compose exec -T externallinks /app/bin/django_wait_for_db.sh +python django_wait_for_migrations.py test`. Coverage config is `.coveragerc` +(dynamic `test_function` context, migrations omitted); the README covers the +`htmlcov/` report. + +## CI + +`.github/workflows/dockerpublish.yml` builds the images and runs the suite on +every PR and push. On `master` / `staging`, after tests pass, it pushes +`externallinks`, `eventstream`, and `externallinks_cron` to +`quay.io/wikipedialibrary/`, tagged by branch and commit sha. Dependencies are +tracked by Dependabot (`.github/dependabot.yml`). + +## Commit messages + +Wikimedia format, GitHub variant: no `Change-Id:` trailer (that is Gerrit-only). +`/wmf-claude:write-commit-msg` drafts one. Subject `component: Subject`, body wraps +at 72; trailers in order `Assisted-by:` (kernel style, model name only) then +`Bug: TXXXXX` (Phabricator, still applies here). No `Co-Authored-By:`. + +## wmf-claude skills and tooling + +`/wmf-claude:write-commit-msg`, `write-phab-task`, `vuln-audit`, and `codesearch` +apply here. The MediaWiki-specific skills (`run-tests`, `test-coverage`, `lint`, +`manual-test`, `compare-rebase`, `perf-audit`) and the Gerrit-only +`review-patch` / `gerrit-reviewer` do NOT: this is a GitHub-hosted Django app, so +use normal PR review flows. + +- **Code search**: [codesearch.wmcloud.org](https://codesearch.wmcloud.org/search/) + spans the WMF ecosystem; the Hound backend API is + `https://codesearch-backend.wmcloud.org/search/api/v1/search?q={query}&repos=*`. +- **Web fetch**: prefer `curl` (piped to `jq`) over `WebFetch` for Wikimedia sites + (mediawiki.org, wikitech, phabricator, gerrit, gitlab.wikimedia.org); WebFetch + often gets 403'd. Try WebFetch first for non-Wikimedia sites. + +## Documentation + +- Project wiki (the real docs): https://github.com/WikipediaLibrary/externallinks/wiki +- Phabricator: https://phabricator.wikimedia.org/tag/wikilink-tool/ +- Migration toolkit: https://gitlab.wikimedia.org/repos/modtools/cloudvps-compose-deploy +- Cloud VPS practice: https://www.mediawiki.org/wiki/Moderator_Tools/Development/Cloud_VPS + (rootless-compose model: the + [Hashtags playbook](https://www.mediawiki.org/wiki/Moderator_Tools/Development/Cloud_VPS/Hashtags)) From bf7bc2ec0c09e0d486e5c5503db9d15bbd0ed5f2 Mon Sep 17 00:00:00 2001 From: jsn Date: Thu, 16 Jul 2026 18:53:02 -0500 Subject: [PATCH 2/3] deploy: Add CloudVPS compose-deploy config Prepares the repo for the CloudVPS Compose Deploy toolkit (rootless docker compose) that replaces the Bullseye swarm VM; master is the deploy branch. Compose files: - Publish only nginx to the host everywhere; drop the dev-only host publishes on externallinks, db, and cache. The app is reached through nginx, and ad-hoc service access goes through docker compose exec - Add docker-compose.deploy.yml carrying the production-runtime concerns the base omits for dev, currently a restart policy on every service. The toolkit selects it via COMPOSE_FILE and pulls prebuilt images, since the base already declares image: on every service Toolkit contract: - Add conf/crontab with the redeploy tick and the toolkit's envsubst placeholders; the app's own jobs stay in-container off ./crontab - Repoint the HOST_BACKUP_DIR guidance in template.env at the toolkit's backup mount CI: - Validate the base + deploy-overlay merge the toolkit runs, which the existing build-and-test job never exercises Assisted-by: Claude Opus 4.8 Bug: T402055 --- .github/workflows/dockerpublish.yml | 6 ++++++ conf/crontab | 14 ++++++++++++++ docker-compose.deploy.yml | 22 ++++++++++++++++++++++ docker-compose.yml | 6 ------ template.env | 3 ++- 5 files changed, 44 insertions(+), 7 deletions(-) create mode 100644 conf/crontab create mode 100644 docker-compose.deploy.yml diff --git a/.github/workflows/dockerpublish.yml b/.github/workflows/dockerpublish.yml index 1f0534e..9999187 100644 --- a/.github/workflows/dockerpublish.yml +++ b/.github/workflows/dockerpublish.yml @@ -21,6 +21,12 @@ jobs: run: | cp template.env .env docker compose up -d --build + - name: Validate deploy overlay merge + # CI builds and starts the base above; the toolkit additionally layers + # docker-compose.deploy.yml. Validate that merge too so an overlay typo + # fails here instead of stalling the toolkit's redeploy on the VM. + run: | + docker compose -f docker-compose.yml -f docker-compose.deploy.yml config -q - name: Run tests run: | docker compose exec -T externallinks /app/bin/django_wait_for_db.sh python django_wait_for_migrations.py test diff --git a/conf/crontab b/conf/crontab new file mode 100644 index 0000000..1f757fc --- /dev/null +++ b/conf/crontab @@ -0,0 +1,14 @@ +# Host crontab for the CloudVPS Compose Deploy toolkit, installed for the +# service account. The toolkit substitutes the ${...} placeholders via envsubst +# at provision time (the checkout path and branch are only known then). +# +# This carries the redeploy tick only. The app's own scheduled jobs (aggregate +# fills, linksearchtotal_collect, backups, ...) run inside the crons service off +# ./crontab, independent of how the stack is deployed. +SHELL=/bin/bash +PATH=${TOOLKIT_BIN}:${DEPLOY_ROOT}/bin:/usr/bin:/bin +DEPLOY_ROOT=${DEPLOY_ROOT} +COMPOSE_FILE=${DEPLOY_ROOT}/docker-compose.yml:${DEPLOY_ROOT}/docker-compose.deploy.yml + +# git sync, pull, restart the stack on real change +*/5 * * * * deploy.sh ${ENV} diff --git a/docker-compose.deploy.yml b/docker-compose.deploy.yml new file mode 100644 index 0000000..9c5638e --- /dev/null +++ b/docker-compose.deploy.yml @@ -0,0 +1,22 @@ +--- +# Deployment overlay for the CloudVPS Compose Deploy toolkit (rootless docker +# compose). The toolkit selects this alongside the base via COMPOSE_FILE. The +# base already declares image: on every service, so the toolkit pulls prebuilt +# images rather than building on the instance. +# +# Only production-runtime concerns belong here. Restart every service so the +# stack survives a host reboot or a crashed worker; the base leaves restart +# unset for local dev. +services: + externallinks: + restart: unless-stopped + crons: + restart: unless-stopped + db: + restart: unless-stopped + nginx: + restart: unless-stopped + eventstream: + restart: unless-stopped + cache: + restart: unless-stopped diff --git a/docker-compose.yml b/docker-compose.yml index badda26..dd2a5bf 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -13,8 +13,6 @@ services: - ".env" depends_on: - db - ports: - - "8000:8000" command: ["/app/bin/gunicorn.sh"] volumes: - type: bind @@ -54,8 +52,6 @@ services: image: quay.io/wikipedialibrary/mariadb:10-updated env_file: - ".env" - ports: - - "3306:3306" volumes: - type: volume source: mysql @@ -127,8 +123,6 @@ services: memory: "48M" cache: image: quay.io/wikipedialibrary/memcached:latest - ports: - - "11211:11211" entrypoint: - memcached depends_on: diff --git a/template.env b/template.env index 4f66959..83043be 100644 --- a/template.env +++ b/template.env @@ -13,7 +13,8 @@ TWL_API_TOKEN=token # When pulling, these define which images are pulled EVENTSTREAM_TAG=latest EXTERNALLINKS_TAG=latest -# Change to something like /data/project/prod for real servers. +# Change to the host backup path on real servers (/usr/local/backup on a +# compose-deploy toolkit instance). HOST_BACKUP_DIR=./backup # Swift Object Store (for storing archives), `local` will just ignore Swift usage SWIFT_APPLICATION_CREDENTIAL_ID=local From 9d144fb896a59b99abc5499013dff4910ce7274a Mon Sep 17 00:00:00 2001 From: jsn Date: Fri, 17 Jul 2026 07:01:53 -0500 Subject: [PATCH 3/3] docs: Describe compose-deploy as the deployment The compose-deploy config is now in, so the swarm-is-current framing in the deployment section is stale. Assisted-by: Claude Opus 4.8 Bug: T402055 --- CLAUDE.md | 41 +++++++++++++++++------------------------ 1 file changed, 17 insertions(+), 24 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 91b85cc..b6e2834 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -60,34 +60,28 @@ aggregate job is visible. Fuller writeups: the wiki's and [Data tracking](https://github.com/WikipediaLibrary/externallinks/wiki/Data-tracking) pages. -## Deployment, and the migration in flight +## Deployment -Production today runs on **Docker Swarm** (`bin/swarm_update.sh` does -`docker stack deploy`; docker-compose is pinned to 1.25.5). The current build is -documented on the wiki's -[Debian Server Setup](https://github.com/WikipediaLibrary/externallinks/wiki/Debian-Server-Setup) -page. - -That VM is a deprecated Bullseye instance being replaced under **T402055** -(deadline 2026-07-31), and the replacement moves deployment to the +Production is deployed with the [CloudVPS Compose Deploy toolkit](https://gitlab.wikimedia.org/repos/modtools/cloudvps-compose-deploy) -(rootless docker compose), the same toolkit TWLight already runs. Its -contract sets the target layout: +(rootless docker compose), the same toolkit TWLight runs. Its contract sets the +layout: -- One VM runs one environment, and that environment is a git branch (`staging` or - `production`) named by the VM's `env` instance metadata. +- One VM runs one environment, and that environment is the git branch named by + the VM's `env` instance metadata; for this repo the deploy branch is `master`. - `COMPOSE_FILE` pins `docker-compose.yml` (base) plus `docker-compose.deploy.yml` (deploy overlay); `docker-compose.override.yml` is left free as the local-dev overlay so a bare `docker compose up` stays a dev command. -- `template.env` is copied to `.env` on first setup; scheduled jobs live in - `conf/crontab`. +- `template.env` is copied to `.env` on first setup; `conf/crontab` carries the + redeploy tick, while the app's own scheduled jobs run in the crons container + off `./crontab`. -**Guidance:** favor the migration direction. This repo is already clean (no -virtualenv-in-container, one compose file, a modest `bin/`); keep it that way, and -don't add swarm-specific coupling, new overlays, or wrapper-script sprawl. Both -the swarm path and the new compose path must work during the rollout, so prefer -additive changes (a new `docker-compose.deploy.yml` alongside the existing file) -over in-place rewrites of anything the live swarm deployment still depends on. +The old Bullseye VM ran on **Docker Swarm** (`bin/swarm_update.sh` does +`docker stack deploy`). It is being retired under **T402055** (deadline +2026-07-31) and kept live for rollback through the cutover soak, so +`bin/swarm_update.sh` and the `deploy:` resource blocks in `docker-compose.yml` +stay until the decommission step removes them. Don't add new swarm-specific +coupling; the repo is on the compose-deploy path now. ## Conventions @@ -106,9 +100,8 @@ over in-place rewrites of anything the live swarm deployment still depends on. MediaWiki proper, but the principles hold: no synchronous HTTP in the request path, batch DB reads, defer slow work to cron. The stream consumer and the aggregate fills are the hot spots. -- Logs go to container stdout; `docker compose logs externallinks` (or the swarm - equivalent for now) is the first place to look. Slow queries show up in the `db` - container logs. +- Logs go to container stdout; `docker compose logs externallinks` is the first + place to look. Slow queries show up in the `db` container logs. ## Testing