[3.0][Testing] Check that an upgraded database matches a fresh install

.docker/README.md

@@ -53,6 +53,94 @@ forum.
 
 ## Installing the forum
 
+```sh
+.docker/install-forum.sh --engine mysql
+.docker/install-forum.sh --engine postgresql
+.docker/install-forum.sh --engine both
+```
+
+That resets the engine's database and installs a forum into it, with no browser
+involved. It takes about a minute. Log in at http://localhost:8080 as
+`admin` / `password`.
+
+SMF 3.0's installer is CLI-native: `Maintenance::parseCliArguments()` turns
+`--name=value` into `$_POST`, and `Maintenance::execute()` then runs every step
+in one process, stopping at the first that still needs input. The script makes
+two passes, because `databasePopulation()` always stops the first time even
+though it succeeded — it pauses so a human can read its "N duplicate tables
+ignored" report, and the form's `pop_done` field is the short-circuit past it.
+Passing `pop_done` on the first pass would skip building the schema entirely.
+
+It then deletes `install.php`, which the installer asks for but cannot do
+itself — its `?delete` link is a GET, and command line arguments only ever reach
+`$_POST`. That matters more than it sounds: while the file is there
+`Settings.php` redirects every request back into the installer, and SMF puts a
+"MAJOR SECURITY RISK" box on every page it shows an administrator. Reinstalling
+still works, because `reset.sh` runs first and does not return until the
+entrypoint has staged a fresh copy.
+
+Two flags worth knowing:
+
+- `--force` reinstalls even when a forum is already there. Without it the
+  script leaves an existing install alone.
+- `--pin-secrets` fixes `auth_secret` and `image_proxy_secret` to known values
+  instead of the random ones `ForumSettings()` generates. Both installs then
+  differ only in their database, so a login cookie survives `use-engine.sh`.
+  Dev-only values for a throwaway forum: never reuse them.
+
+### Two forums at once
+
+`--engine both` installs MySQL first and PostgreSQL second, one after the other.
+It has to be sequential: `Settings.php` pins a single `$db_type`, and
+`Db::load()` hands back the connection it already made, so only one engine can
+ever be live in a process.
+
+Both installs are kept. Switch between them with:
+
+```sh
+.docker/use-engine.sh postgresql
+```
+
+That puts the saved `Settings.php` back and clears `cache/`. No restart is
+needed — the entrypoint only writes `Settings.php` when there is not one, so it
+leaves whatever is in place alone. The copies live in `.docker/settings/` and
+are gitignored.
+
+`reset.sh` is the other half: it empties one engine's database and restages the
+installer, discarding that forum. `use-engine.sh` switches between forums,
+`reset.sh` throws one away.
+
+## Accounts and passwords
+
+Two forums, each with its own administrator, and a password chosen months ago is
+a recipe for an afternoon of hand written SQL. `user.sh` is there so it is not:
+
+```sh
+.docker/user.sh list
+.docker/user.sh check admin 'password'
+.docker/user.sh reset admin 'a new password'
+```
+
+`check` exits 0 when SMF would accept the password and 1 when it would not, so
+it works in a conditional as well as by eye. It also points out an account that
+is not activated, which fails to log in with a correct password and looks
+exactly like a wrong one.
+
+`--engine mysql|postgresql` reads the settings `use-engine.sh` saved for that
+engine, so the *other* forum can be inspected without switching to it:
+
+```sh
+.docker/user.sh check admin 'password' --engine mysql
+```
+
+The hashing goes through SMF's own `Security` class rather than being written
+here, so what `reset` puts in the table is by construction what `Login2` expects
+to find. It clears `passwd_flood` at the same time: SMF locks an account out for
+a while after enough wrong guesses, and a fresh password behind a lockout looks
+exactly like a password that did not take.
+
+### Installing in a browser instead
+
 On first boot the entrypoint writes a `Settings.php` pre-filled for the chosen
 engine and copies `other/install.php` to the web root, so
 http://localhost:8080 redirects into the installer.
@@ -102,8 +190,69 @@ The repository is bind-mounted at `/var/www/html`, so edits on the host are
 live on the next request. Opcache is on but revalidates every request, so you
 never need to restart for a PHP change.
 
-To reinstall from scratch: `docker compose down -v`, delete `Settings.php` and
-`Settings_bak.php`, then `docker compose up -d`.
+To reinstall from scratch: `.docker/install-forum.sh --engine mysql --force`.
+To wipe everything including the volumes: `docker compose down -v`.
+
+## Comparing an upgrade against a fresh install
+
+The installer builds the schema from `Sources/Db/Schema/v3_0/` in one go. The
+upgrader arrives at the same place through a hundred-odd migrations applied to
+whatever 2.1 left behind. They are meant to converge, and nothing checks that
+they do:
+
+```bash
+.docker/compare-upgrade.sh --engine mysql --baseline path/to/a-2.1-dump.sql
+```
+
+That empties the database, loads the dump, upgrades it, reads the schema,
+reinstalls from scratch, reads that too, and reports every place the two
+disagree — a column of the wrong type, an index that was never created, a
+primary key quietly dropped. It ends with the fresh install in place, and takes
+five to ten minutes.
+
+`--baseline` takes any SQL dump of a 2.1 database. A dump of a real forum is
+the better test; the [2.1 development environment][baseline] builds a synthetic
+one designed to hold something in every table an upgrade touches, which is
+useful when you have no real forum to hand.
+
+Two kinds of difference are reported but do not fail the run, because a real
+forum always has some: the contents of `settings`, and the order columns sit in
+within a table. Everything else is a schema difference and sets the exit code.
+
+If the upgrade does not reach the end, the script stops there and says so
+rather than comparing anyway. A half-upgraded database differs from a fresh
+install in hundreds of places, all of them the honest consequence of the
+migrations that never ran, and none of them worth reading.
+
+On PostgreSQL the reading also covers sequences and the compatibility functions
+SMF installs — `find_in_set()`, `instr()`, the `group_concat` aggregate and the
+rest. A query naming one of those fails outright when it is not there, so a
+missing function counts as a schema difference like a missing column does.
+
+Both readings were checked name by name against the engine's own schema dump —
+`pg_dump --schema-only` and `mysqldump --no-data --routines --triggers
+--events` — and agree with them: 72 tables, 538 columns, 179 keys, and on
+PostgreSQL 41 sequences and 19 functions besides. The only things either dump
+reports that this does not are the `public` schema and the comment on it, and
+the `AUTO_INCREMENT` counter, which measures how much a database has been used
+rather than what shape it is.
+
+The tool underneath is usable on its own, against any two SMF databases on the
+same engine — two forums you already have, or the same forum before and after
+something you are testing:
+
+```bash
+docker compose exec web php .docker/schema-tool.php dump --engine mysql --db smf > before.json
+# ... do the thing ...
+docker compose exec web php .docker/schema-tool.php dump --engine mysql --db smf > after.json
+docker compose exec web php .docker/schema-tool.php diff before.json after.json
+```
+
+It talks to the database directly rather than through SMF, so it works on a
+database SMF would refuse to run on — which is usually the one you want to look
+at.
+
+[baseline]: https://github.com/SimpleMachines/SMF/pull/9330
 
 ## Debugging SQL with the PostgreSQL log
 
@@ -170,4 +319,12 @@ compose.yaml                     the stack
 .docker/mysql/init/10-smf.sh     runs once on first mysql database creation
 .docker/postgres/init/10-smf.sh  runs once on first postgres database creation
 .docker/env.example              optional overrides
+
+.docker/lib.sh                   paths, credentials and engine names, shared
+.docker/install-forum.sh         install a forum with no browser involved
+.docker/reset.sh                 empty one engine and restage the installer
+.docker/use-engine.sh            switch which installed forum is live
+.docker/user.sh                  inspect accounts, check and reset passwords
+.docker/compare-upgrade.sh       upgrade a 2.1 dump, install 3.0, diff the two
+.docker/schema-tool.php          read a database's shape, and compare readings
 ```

.docker/compare-upgrade.sh

@@ -0,0 +1,197 @@
+#!/usr/bin/env bash
+# Upgrades a 2.1 database to 3.0, installs 3.0 from scratch, and reports where
+# the two schemas disagree.
+#
+#   .docker/compare-upgrade.sh --engine mysql      --baseline ../SMF-2.1/.docker/baseline/artifacts/2.1.7-1/small/mysql.sql
+#   .docker/compare-upgrade.sh --engine postgresql --baseline ../SMF-2.1/.docker/baseline/artifacts/2.1.7-1/small/postgres.sql
+#
+# The installer builds the schema from Sources/Db/Schema/v3_0/ in one go. The
+# upgrader arrives at the same place through a hundred-odd migrations applied
+# to whatever 2.1 left behind. Nothing checks that those two agree, and where
+# they do not, the forum that upgraded is running on a schema that has never
+# been tested against -- a column of the wrong type, an index that was never
+# created, a primary key quietly dropped.
+#
+# --baseline takes any SQL dump of a 2.1 database. The one this was written
+# against is the committed baseline from the 2.1 development environment, which
+# is built to hold something in every table an upgrade touches, but a dump of a
+# real forum works and is a better test.
+#
+# Both databases for the chosen engine are rebuilt, twice. Anything already
+# installed on that engine is destroyed, and what remains at the end is the
+# fresh install. The other engine is not touched.
+#
+# Runs on the host. Expect five to ten minutes per engine.
+set -euo pipefail
+
+. "$(dirname -- "${BASH_SOURCE[0]}")/lib.sh"
+
+ENGINE=''
+BASELINE=''
+OUT="$DOCKER_DIR/compare"
+
+while [ $# -gt 0 ]; do
+	case "$1" in
+		--engine) ENGINE="$2"; shift 2 ;;
+		--engine=*) ENGINE="${1#*=}"; shift ;;
+		--baseline) BASELINE="$2"; shift 2 ;;
+		--baseline=*) BASELINE="${1#*=}"; shift ;;
+		--out) OUT="$2"; shift 2 ;;
+		--out=*) OUT="${1#*=}"; shift ;;
+		-h|--help) sed -n '2,25p' "${BASH_SOURCE[0]}"; exit 0 ;;
+		*) die "unknown argument: $1" ;;
+	esac
+done
+
+[ -n "$ENGINE" ] || die 'need --engine mysql|postgresql|both'
+ENGINES=$(engine_list "$ENGINE") || die "unknown engine: $ENGINE"
+[ -n "$BASELINE" ] || die 'need --baseline <a 2.1 SQL dump>'
+[ -f "$BASELINE" ] || die "no such file: $BASELINE"
+
+cd "$BOARD_DIR"
+mkdir -p "$OUT"
+OUT=$(cd -- "$OUT" && pwd)
+
+# The tool that reads the two files runs in the container, which sees the
+# repository and nothing else, so the output has to live somewhere inside it.
+case "$OUT" in
+	"$BOARD_DIR"/*) OUT_REL="${OUT#"$BOARD_DIR"/}" ;;
+	*) die "--out has to be somewhere inside the repository, since the container cannot see anywhere else" ;;
+esac
+
+# Reads the shape of the database the given engine is pointed at. The tool runs
+# in the container, so it uses the container-internal host and port rather than
+# the ones published in compose.yaml.
+snapshot() {
+	local smf_type="$1" label="$2" file="$3"
+
+	docker compose exec -T web php .docker/schema-tool.php dump \
+		--engine "$smf_type" \
+		--db "$DB_NAME" \
+		--prefix "$DB_PREFIX" \
+		--host "$(engine_server "$smf_type")" \
+		--port "$(engine_port "$smf_type")" \
+		--user "$DB_USER" \
+		--pass "$DB_PASSWORD" \
+		--label "$label" > "$file"
+
+	[ -s "$file" ] || die "${smf_type}: the ${label} reading came back empty"
+}
+
+# The version this checkout is, which is what a finished upgrade has to leave
+# in the database.
+smf_version() {
+	sed -n "s/.*define('SMF_VERSION', '\([^']*\)').*/\1/p" index.php | head -1
+}
+
+load_baseline() {
+	local smf_type="$1" service
+	service=$(engine_service "$smf_type")
+
+	if [ "$smf_type" = 'mysql' ]; then
+		docker compose exec -T -e MYSQL_PWD="$DB_PASSWORD" "$service" \
+			mysql -u"$DB_USER" -D "$DB_NAME" < "$BASELINE"
+	else
+		# ON_ERROR_STOP so that a dump taken from a database that is not empty,
+		# or one taken with a different owner, stops here rather than producing
+		# a half-loaded forum that then fails somewhere in the upgrader and
+		# looks like a migration bug.
+		docker compose exec -T "$service" \
+			psql -v ON_ERROR_STOP=1 -q -U "$DB_USER" -d "$DB_NAME" < "$BASELINE" >/dev/null
+	fi
+}
+
+compare_one() {
+	local smf_type="$1" version
+
+	# ---------------------------------------------------------- the upgrade
+	log "${smf_type}: emptying the database"
+	"$DOCKER_DIR/reset.sh" --engine "$smf_type" >/dev/null
+
+	log "${smf_type}: loading ${BASELINE##*/}"
+	load_baseline "$smf_type"
+
+	version=$(installed_version "$smf_type" || true)
+
+	if [ -z "$version" ]; then
+		warn "${smf_type}: the dump loaded but there is no forum in it -- wrong prefix, or not an SMF dump"
+
+		return 1
+	fi
+
+	log "${smf_type}: the dump is SMF ${version}"
+
+	# The upgrader ships in other/ and expects to be run from the board
+	# directory, the same way install.php does. Nobody upgrading a real forum
+	# has install.php sitting there, so neither does this.
+	rm -f install.php
+	cp other/upgrade.php upgrade.php
+
+	log "${smf_type}: upgrading"
+
+	local upgraded=0
+	docker compose exec -T web php upgrade.php > "$OUT/upgrade-${smf_type}.log" 2>&1 || upgraded=$?
+
+	rm -f upgrade.php
+
+	if [ "$upgraded" -ne 0 ]; then
+		warn "${smf_type}: the upgrader exited ${upgraded} -- ${OUT}/upgrade-${smf_type}.log"
+
+		return 1
+	fi
+
+	# The upgrader reports a failed migration and stops, but still exits 0, so
+	# the version in the database is the only thing that says whether it got to
+	# the end. Checking it matters more than it looks: a half-upgraded database
+	# differs from a fresh install in hundreds of places, all of them the honest
+	# consequence of the migrations that never ran, and none of them the kind of
+	# difference this script exists to find.
+	version=$(installed_version "$smf_type" || true)
+
+	if [ "$version" != "$(smf_version)" ]; then
+		warn "${smf_type}: the upgrade stopped at SMF ${version:-nothing}, expected $(smf_version)"
+		warn "${smf_type}: the last thing it said is at the end of ${OUT}/upgrade-${smf_type}.log"
+
+		return 1
+	fi
+
+	log "${smf_type}: upgraded to SMF ${version}"
+
+	snapshot "$smf_type" upgraded "$OUT/upgraded-${smf_type}.json"
+
+	# ----------------------------------------------------- the fresh install
+	# --force because there is an installed forum now, and install-forum.sh
+	# leaves one alone unless told otherwise.
+	log "${smf_type}: installing from scratch"
+	"$DOCKER_DIR/install-forum.sh" --engine "$smf_type" --force >/dev/null
+
+	snapshot "$smf_type" fresh "$OUT/fresh-${smf_type}.json"
+
+	# ------------------------------------------------------------ the report
+	local status=0
+
+	docker compose exec -T web php .docker/schema-tool.php diff \
+		"${OUT_REL}/fresh-${smf_type}.json" \
+		"${OUT_REL}/upgraded-${smf_type}.json" \
+		> "$OUT/report-${smf_type}.txt" || status=$?
+
+	echo
+	cat "$OUT/report-${smf_type}.txt"
+	echo
+
+	if [ "$status" -eq 0 ]; then
+		log "${smf_type}: the upgraded schema matches a fresh install"
+	else
+		log "${smf_type}: the schemas differ -- ${OUT}/report-${smf_type}.txt"
+	fi
+
+	return "$status"
+}
+
+failed=0
+
+for smf_type in $ENGINES; do
+	compare_one "$smf_type" || failed=1
+done
+
+exit "$failed"