Skip to content

fix: the bundled PostgreSQL shuts down cleanly when the container stops - #541

Merged
kikootwo merged 2 commits into
mainfrom
reland/512-postgres-clean-stop
Nov 2, 2026
Merged

kikootwo merged 2 commits into
mainfrom
reland/512-postgres-clean-stop

Conversation

@kikootwo

@kikootwo kikootwo commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Summary

Every stop, restart or update of the unified container killed the bundled PostgreSQL, so the next start logged database system was not properly shut down; automatic recovery in progress and ran crash recovery. supervisord stopped PostgreSQL with SIGTERM, which is PostgreSQL's smart shutdown: it waits for every client to disconnect. node keeps its database connections open during a stop, so PostgreSQL never finished and was SIGKILLed at supervisord's 10 s default (or by Docker at its own 10 s).

PostgreSQL now stops with SIGINT (fast shutdown: it ends open sessions, writes a checkpoint and exits) and gets up to 60 s to do it. A normal stop takes about 1-2 s, well inside Docker's default 10 s, so existing installs get the fix from the image alone. The compose files and examples also set stop_grace_period: 90s for headroom on slow storage.

Based on #512 by @drwho725.

Changes

  • docker/unified/supervisord.conf: [program:postgresql] gets stopsignal=INT and stopwaitsecs=60, with a short comment. Nothing else changes: supervisord already stops programs in reverse priority (app, then Redis, then PostgreSQL) and postgres-start.sh already execs postgres, so the signal reaches PostgreSQL directly.
  • stop_grace_period: 90s in docker-compose.yml, docker-compose.local.yml and the README compose example; documentation/deployment/unified.md compose examples and docker run --stop-timeout 90 example. 90 s is above the 80 s worst case (10 + 10 + 60).
  • Docs: new Shutdown Sequence section in documentation/deployment/unified.md (order, settings, grace period, Unraid, what a clean stop logs, the one recovery after updating), Fixed Issue 22, TABLEOFCONTENTS line.
  • Unraid template not changed: every stop Unraid starts (Stop button, update, array stop) passes its global Docker Stop Timeout (Settings > Docker, default 10 s), which overrides a per-container value. A normal stop fits in 10 s; the docs say to raise it for headroom.

Evidence

Same container, stopped and started twice per image, --network none, a health request right before each stop so node holds a DB connection. Shutdown counts are from the lines written during the stop; recovery counts from the next start.

image cycle stop s exit fast shutdown smart shutdown is shut down supervisord SIGKILL recovery on next start
main docker stop --time 10 (Docker's default) 11.5 137 0 1 0 0 (Docker killed first) 1
main docker stop --time 90 11.7 0 0 1 0 1 1
this PR docker stop --time 10 0.6 0 1 0 1 0 0
this PR docker stop --time 90 2.1 0 1 0 1 0 0

Upgrade path (named volumes for the database and /app/config): a volume last stopped by the old image recovers once on the first start of the new image, as expected, since that was the old image's stop. The new image's own stop/start after that: fast shutdown in 1.8 s, exit 0, no recovery.

The PostgreSQL and supervisord lines from every cycle: https://gist.github.com/kikootwo/5f68c04e4bf8acd0370f7fbd7fdc5d38

Tests

  • New tests/docker/supervisord-config.test.ts: PostgreSQL stopsignal=INT, stopwaitsecs >= 30, postgres-start.sh execs postgres, and the grace period in both compose files and the README is above the summed stopwaitsecs. Against main's files 5 of its 6 tests fail (the exec check passes on both).
  • The compose rows skip with a reason in the containerized unit run (its build context leaves out docker-compose*.yml) and never skip on GitHub Actions.

Checks

  • npx vitest run tests/docker/supervisord-config.test.ts tests/docker/node-version-pins.test.ts: 12/12
  • npx tsc --noEmit: clean
  • npx eslint tests/docker/supervisord-config.test.ts: 0 errors, 0 warnings
  • Image build (dockerfile.unified): OK
  • e2e smoke on the built image: 1/1 passed

Risk and rollback

  • A stop is only slower when PostgreSQL needs it (a long checkpoint on slow storage), bounded at 60 s for PostgreSQL and 80 s overall. Fast shutdown does not wait on clients, so it cannot hang on node.
  • Under Docker's default 10 s (installs that don't add the grace period) the worst case is today's: Docker SIGKILLs at 10 s either way.
  • A stop during the first seconds of a start, before the entrypoint hands over to supervisord, is ignored by the entrypoint shell (no change here), so with stop_grace_period: 90s Docker now waits up to 90 s instead of 10 s before killing it. Same outcome as before, only slower.
  • During the stop PostgreSQL now logs FATAL: terminating connection due to administrator command for each open session, and node may log Prisma connection errors before the container exits. Cosmetic. node still does not receive SIGTERM through app-start.sh and is killed when the container exits, as before; that is a separate issue: Unified image: node never receives SIGTERM when the container stops #540.
  • USE_EXTERNAL_POSTGRES=true: the program is exec sleep infinity, which SIGINT ends exactly like SIGTERM did.
  • Rollback: revert this commit. No data or schema change.

🤖 Generated with Claude Code

kikootwo and others added 2 commits November 2, 2026 12:30
supervisord stopped PostgreSQL with SIGTERM (smart shutdown), which waits for
every client. node keeps its connections open, so PostgreSQL was SIGKILLed at
supervisord's 10 s default and every start ran crash recovery.

- [program:postgresql]: stopsignal=INT (fast shutdown) and stopwaitsecs=60
- stop_grace_period: 90s in docker-compose.yml, docker-compose.local.yml, the
  README compose example and the unified.md examples (docker run --stop-timeout 90)
- docs: Shutdown Sequence in deployment/unified.md, Fixed Issue 22, TOC line
- tests/docker/supervisord-config.test.ts

Based on #512 by @drwho725.

Co-authored-by: drwho725 <887768+drwho725@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d in the standard compose example

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kikootwo
kikootwo marked this pull request as ready for review November 2, 2026 18:12
@kikootwo
kikootwo merged commit 5e404f3 into main Nov 2, 2026
1 check passed
@kikootwo
kikootwo deleted the reland/512-postgres-clean-stop branch November 2, 2026 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant