Resolve Subject ids through a MediaWiki table

composer.json

@@ -53,7 +53,10 @@
 			"php ../../maintenance/generateSchemaSql.php --json sql/neowiki_rebuild_runs.json --sql sql/postgres/neowiki_rebuild_runs.sql --type postgres",
 			"php ../../maintenance/generateSchemaChangeSql.php --json sql/abstractSchemaChanges/patch-neowiki_rebuild_runs-nwrr_phase.json --sql sql/mysql/patch-neowiki_rebuild_runs-nwrr_phase.sql --type mysql",
 			"php ../../maintenance/generateSchemaChangeSql.php --json sql/abstractSchemaChanges/patch-neowiki_rebuild_runs-nwrr_phase.json --sql sql/sqlite/patch-neowiki_rebuild_runs-nwrr_phase.sql --type sqlite",
-			"php ../../maintenance/generateSchemaChangeSql.php --json sql/abstractSchemaChanges/patch-neowiki_rebuild_runs-nwrr_phase.json --sql sql/postgres/patch-neowiki_rebuild_runs-nwrr_phase.sql --type postgres"
+			"php ../../maintenance/generateSchemaChangeSql.php --json sql/abstractSchemaChanges/patch-neowiki_rebuild_runs-nwrr_phase.json --sql sql/postgres/patch-neowiki_rebuild_runs-nwrr_phase.sql --type postgres",
+			"php ../../maintenance/generateSchemaSql.php --json sql/neowiki_subject_page.json --sql sql/mysql/neowiki_subject_page.sql --type mysql",
+			"php ../../maintenance/generateSchemaSql.php --json sql/neowiki_subject_page.json --sql sql/sqlite/neowiki_subject_page.sql --type sqlite",
+			"php ../../maintenance/generateSchemaSql.php --json sql/neowiki_subject_page.json --sql sql/postgres/neowiki_subject_page.sql --type postgres"
 		]
 	},
 	"scripts-descriptions": {

docs/adr/032-subject-page-index.md

@@ -0,0 +1,91 @@
+# Subject-to-Page Index
+
+Date: 2026-08-20
+
+Status: Accepted
+
+## Context
+
+Subject ids are not derived from the page title ([ADR 5](005-subject-guids.md)), so finding the page that holds a
+Subject takes a query. The only index answering it was the Neo4j projection: `HasSubject` edges walked backwards.
+Subject CRUD, `{{#view}}`, `{{#neowiki_value}}` and the `mw.neowiki.*` getters all resolved ids that way, so a wiki
+with no graph backend, or with only a SPARQL store, could not store or read Subjects at all.
+
+Identity resolution has stronger requirements than the projection it was living in. A graph store is a rebuildable
+query projection ([ADR 19](019-graph-database-architecture.md)) and its failures during edit, delete and undelete are
+swallowed by design so the user's operation still commits. A Subject written while a backend was unreachable would
+then be unfindable, and so uneditable, until an administrator rebuilt that store. With more than one backend
+configured, per-backend failure isolation makes it worse: two stores can hold different answers to "which page holds
+this Subject", and which one replies becomes an accident of wiring order.
+
+## Decision
+
+The subject-to-page mapping is authoritative in a MediaWiki table, `neowiki_subject_page`, keyed on the Subject id
+and the page id. It is not a fallback and not a cache: it is the implementation of `PageIdentifiersLookup`.
+
+**Written with the revision it derives from.** `RevisionFromEditComplete` fires inside `PageUpdater`'s atomic section,
+so indexing an edit on the wiki's primary connection joins the transaction that writes the revision: it commits with
+that revision or not at all. The index can therefore never be staler than the subject slot it reads, and replica reads
+inherit the revision's own session-consistency guarantees. It sits outside the projection's failure isolation, so an
+index write that fails aborts the edit rather than being logged and passed over.
+
+Delete, undelete and import are indexed from hooks outside that atomic section. A web request still has its
+transaction open, so a failure there takes the operation down with it, as on the edit path. A maintenance script has
+no such transaction, so there the index write lands after the operation is already done: a failure leaves the index
+wrong until a rebuild. Only a delete can strand rows that way, and those resolve to nothing anyway.
+
+**Ids come from the slot's raw JSON**, never from the deserializer. An invalid Subject is a persisted, supported state
+([ADR 21](021-add-backend-validation.md), [ADR 26](026-validation-severity-levels.md)), and the lookup is what gets an
+editor to the page holding it, so a Subject too broken to deserialize must still be findable. Ids that are not
+well-formed are left out, since no caller could ask about them.
+
+**Reads join `page`.** The title and namespace are the page's current ones, so a move needs no index maintenance, and
+the rows a deleted page leaves behind resolve to nothing.
+
+**Duplicate ids resolve to the lowest page id.** Cross-wiki transfer may bring one id onto two pages, which must not
+fail either page's save ([ADR 5](005-subject-guids.md)), so the key is not unique. Every reader getting the same
+answer matters more than which page wins. Ids are keyed bare, as stored ([ADR 22](022-multi-wiki-node-identity.md),
+[ADR 23](023-subject-sources.md)): `null` means "no local page holds this", not "does not exist".
+
+Graph backends are unchanged: still rebuildable query projections, each registering its query surfaces only when it is
+configured. What changes is that Subjects no longer need one. One editing feature still does: relation-target
+suggestions are read from Neo4j, and without it the field offers none.
+
+Because identity is now resolved authoritatively, an unresolvable Subject is refused rather than measured against the
+wiki-global `edit` right: every right a Subject write needs is a right on the page holding it, so with no page there
+is nothing to allow.
+
+## Consequences
+
+* NeoWiki works with no graph backend configured. That is a supported mode, not a misconfiguration.
+* A NeoWiki table is now on the correctness path, whereas `neowiki_rebuild_runs` only carries bookkeeping. Every save
+  writes the index, so a wiki that has not run `update.php` cannot be edited at all, rather than degrading to
+  Subjects being unfindable.
+* `RebuildSubjectPageIndex.php` backfills an existing wiki, registered to run from `update.php`. MediaWiki's web
+  updater runs no post-update scripts, so a wiki upgraded that way gets an empty table, leaving every Subject that
+  predates the upgrade unresolvable until the script is run by hand.
+* Resolving an id is a primary-key read against a local table rather than a network round-trip, on a path taken once
+  per Subject per page render.
+* The index shares the projection's blind spot: a history merge that leaves the source page as a redirect writes that
+  revision without firing a hook either covers. The source page keeps its rows and still exists, so its Subjects go on
+  resolving to a page that no longer holds them, and a write to one lands there. The rebuild script is the repair
+  path. The impact is worse than for a projection, which only misreports a query.
+* A graph rebuild does not write the index. It walks the wiki from a replica, so the revision it projects may already
+  have been superseded — eventual consistency the index must not inherit.
+* The index rebuild is a plain walk, with none of the graph rebuild's per-store machinery: no resume cursor, no run
+  record, no way to start it from the wiki. It runs once at upgrade and rarely after, so a run that dies starts over.
+
+## Alternatives Considered
+
+* **`page_props`**: only indexed by `pp_propname`, so the Subject id would have to be encoded into the property name,
+  one name per Subject. `Special:PagesWithProp` and `list=pagepropnames` enumerate distinct property names wiki-wide,
+  so that breaks core surfaces NeoWiki cannot patch. `page_props` is also written POSTSEND, deliberately outside
+  ChronologyProtector: an eventually-consistent index under a strongly-consistent write path.
+* **The search index**: optional infrastructure and eventually consistent, so it cannot carry a correctness
+  dependency.
+* **Scanning content at query time**: revision content offers no indexed access and may be compressed or external, so
+  every lookup would load every Subject page's blob.
+* **Making the id the page title**, as Wikidata does: dissolves the problem, but reverses
+  [ADR 5](005-subject-guids.md) and the [several-Subjects-per-page model](007-multiple-subjects-per-page.md).
+* **A cache in front of the graph**: cannot answer a miss authoritatively, so it optimizes an implementation rather
+  than being one.

docs/operations/installation.md

@@ -64,7 +64,7 @@ Use this to add NeoWiki to a MediaWiki you already run. You provide the surround
 | MediaWiki 1.43.0 or later | |
 | PHP 8.3 with `ext-json`   | |
 | Composer                  | Installs NeoWiki's runtime dependencies. No `vendor/` is shipped. |
-| Neo4j 5.x over Bolt       | The graph backend. Required to use NeoWiki's structured-data features; the wiki boots without it. |
+| Neo4j 5.x over Bolt       | Optional. The graph backend behind Cypher queries and relation-target suggestions. |
 | Node.js 24 or later       | Needed only to build the frontend bundle in step 2. |
 
 These extensions are recommended. NeoWiki runs without them, but you lose the matching functionality:
@@ -122,8 +122,8 @@ wfLoadExtension( 'CodeEditor' );
 wfLoadExtension( 'ParserFunctions' );
 ```
 
-Without both Neo4j URLs set, the wiki still loads and ordinary pages render, but NeoWiki's structured-data features
-and the query surfaces (`{{#cypher_raw}}`, `nw.query`, `POST /neowiki/v0/query/cypher`) stay disabled.
+Without both Neo4j URLs set, NeoWiki's structured-data features still work. You lose the Cypher query surfaces
+(`{{#cypher_raw}}`, `nw.query`, `POST /neowiki/v0/query/cypher`) and relation-target suggestions.
 
 ### 4. Run the updater
 
@@ -151,14 +151,16 @@ php maintenance/run.php NeoWiki:RebuildGraphDatabases
    {{#view:}}
    ```
 
-4. **Query the graph.** Only this step checks the Neo4j projection. On any page, add a Cypher query that lists the
-   stored pages, independent of your data model:
-   ```
-   {{#cypher_raw: MATCH (p:Page) RETURN p.name }}
-   ```
-   The result renders as JSON. If Neo4j is unreachable, it renders an error instead.
+Your install is complete once those three steps work.
 
-If all four steps work, your install is complete.
+With Neo4j configured, one more step checks its projection. On any page, add a Cypher query that lists the stored
+pages, independent of your data model:
+
+```
+{{#cypher_raw: MATCH (p:Page) RETURN p.name }}
+```
+
+The result renders as JSON. If Neo4j is unreachable, it renders an error instead.
 
 ### Optional: Pretty URLs for the Data tab
 
@@ -175,8 +177,8 @@ These are the settings you are most likely to change. For the full list with des
 
 | Setting | Purpose | Default | Required |
 |---|---|---|---|
-| `$wgNeoWikiNeo4jInternalWriteUrl` | Bolt URL for writing the graph projection | _none_ | For features |
-| `$wgNeoWikiNeo4jInternalReadUrl` | Bolt URL for read and query traffic | _none_ | For features |
+| `$wgNeoWikiNeo4jInternalWriteUrl` | Bolt URL for writing the graph projection | _none_ | For Neo4j features |
+| `$wgNeoWikiNeo4jInternalReadUrl` | Bolt URL for read and query traffic | _none_ | For Neo4j features |
 | `$wgNeoWikiEnableDevelopmentUI` | Enables development-only UIs | `false` | No |
 | `$wgNeoWikiEnforceValidation` | Rejects writes that introduce new `error`-severity violations | `false` | No |
 | `$wgNeoWikiAutoRenderMainSubject` | Automatically renders a page's Main Subject as an infobox | `true` | No |
@@ -218,13 +220,10 @@ content model, validated, or read.
 
 ## Optional: SPARQL graph stores
 
-Alongside Neo4j, NeoWiki can keep one or more SPARQL 1.1 graph stores in sync with page changes. This works with
+With or without Neo4j, NeoWiki can keep one or more SPARQL 1.1 graph stores in sync with page changes. This works with
 QLever, Oxigraph, Fuseki and any other SPARQL 1.1 store. Each configured store receives the NeoWiki data as RDF:
 every page becomes a named graph, replaced on each edit and dropped on deletion.
 
-A SPARQL store does not yet replace Neo4j: NeoWiki's interactive features (the Subject editing UIs, views, and value
-accessors) still require a configured Neo4j backend.
-
 Configure the stores with `$wgNeoWikiSparqlStores`, a list of objects:
 
 ```php

docs/operations/maintenance.md

@@ -102,11 +102,27 @@ store holding that projection in the background. It is off by default: such a re
 wiki, and it lets anyone who may edit Mapping pages set that going — work `neowiki-admin` otherwise gates. A
 rebuild somebody started by hand is left to finish rather than restarted; the store shows up stale once it ends.
 
+## Rebuilding the subject index
+
+NeoWiki keeps an index of which page holds which Subject. Editing, deletion, undeletion and import all keep it
+current and `update.php` fills it, so this is rarely needed. Run it when Subjects are not found on the pages holding
+them — after merging page histories, or after upgrading through MediaWiki's web updater, which does not fill the
+index:
+
+```sh
+php maintenance/run.php NeoWiki:RebuildSubjectPageIndex --force
+```
+
+`--force` re-runs it after `update.php` has recorded it as done.
+
+Neither [rebuilding the graph](#rebuilding-the-graph) nor a null edit repairs this index.
+
 ## What happens during a Neo4j outage
 
 - **Editing pages works.** Edits, deletions and undeletions all commit. NeoWiki logs the projection failure on the
   `NeoWiki` channel.
-- **Editing and displaying Subjects fails**, along with queries and anything else that reads the graph.
+- **Editing and displaying Subjects works.**
+- **Queries fail**, along with relation-target suggestions and anything else that reads the graph.
 
 Once Neo4j is back, [rebuild the graph](#rebuilding-the-graph): it repairs both a failed save and a failed delete.
 

extension.json

@@ -30,6 +30,10 @@
 		"ProfessionalWiki\\NeoWiki\\": "src/"
 	},
 
+	"AutoloadClasses": {
+		"ProfessionalWiki\\NeoWiki\\Maintenance\\RebuildSubjectPageIndex": "maintenance/RebuildSubjectPageIndex.php"
+	},
+
 	"TestAutoloadNamespaces": {
 		"ProfessionalWiki\\NeoWiki\\Tests\\": "tests/phpunit",
 		"ProfessionalWiki\\RedHerb\\": "tests/RedHerb/src"

maintenance/RebuildSubjectPageIndex.php

@@ -0,0 +1,70 @@
+<?php
+
+declare( strict_types = 1 );
+
+namespace ProfessionalWiki\NeoWiki\Maintenance;
+
+use MediaWiki\Maintenance\LoggedUpdateMaintenance;
+use ProfessionalWiki\NeoWiki\Persistence\MediaWiki\DatabaseSubjectPageIndexRebuilder;
+
+$basePath = getenv( 'MW_INSTALL_PATH' ) !== false ? getenv( 'MW_INSTALL_PATH' ) : __DIR__ . '/../../..';
+
+require_once $basePath . '/maintenance/Maintenance.php';
+
+/**
+ * Rebuilds the subject -> page index from the subject slots it derives from.
+ *
+ * update.php runs this once, which is what gets a wiki that predates the index a filled one. It is also
+ * the repair path afterwards, for a history merge that leaves the source page as a redirect: that writes
+ * a revision without firing a hook the index is built on. Such a run needs --force, since the update is
+ * logged as done.
+ */
+class RebuildSubjectPageIndex extends LoggedUpdateMaintenance {
+
+	public function __construct() {
+		parent::__construct();
+
+		$this->setBatchSize( DatabaseSubjectPageIndexRebuilder::DEFAULT_BATCH_SIZE );
+		$this->requireExtension( 'NeoWiki' );
+		$this->addDescription(
+			'Rebuilds the subject -> page index from the Subjects every page currently holds. ' .
+			'Run this after merging page histories, which can write a revision the index is not told about.'
+		);
+	}
+
+	protected function getUpdateKey(): string {
+		return 'neowiki-rebuild-subject-page-index';
+	}
+
+	protected function doDBUpdates(): bool {
+		$this->output( "Rebuilding the NeoWiki subject -> page index...\n" );
+
+		// Left at 0 when there is nothing to index, so the closing line is right either way.
+		$indexed = 0;
+
+		foreach ( $this->newRebuilder()->rebuild() as $indexed ) {
+			$this->output( "...$indexed pages indexed\n" );
+			$this->waitForReplication();
+		}
+
+		$this->output( "Done. Indexed $indexed pages holding Subjects.\n" );
+
+		return true;
+	}
+
+	/**
+	 * Built from MediaWiki's services alone: update.php runs this on wikis whose NeoWiki configuration
+	 * is not readable yet, and nothing here depends on that configuration.
+	 */
+	private function newRebuilder(): DatabaseSubjectPageIndexRebuilder {
+		return new DatabaseSubjectPageIndexRebuilder(
+			$this->getPrimaryDB(),
+			$this->getServiceContainer()->getRevisionLookup(),
+			$this->getBatchSize()
+		);
+	}
+
+}
+
+$maintClass = RebuildSubjectPageIndex::class;
+require_once RUN_MAINTENANCE_IF_MAIN;

sql/mysql/neowiki_subject_page.sql

@@ -0,0 +1,10 @@
+-- This file is automatically generated using maintenance/generateSchemaSql.php.
+-- Source: sql/neowiki_subject_page.json
+-- Do not modify this file directly.
+-- See https://www.mediawiki.org/wiki/Manual:Schema_changes
+CREATE TABLE /*_*/neowiki_subject_page (
+  nwsp_subject_id VARBINARY(32) NOT NULL,
+  nwsp_page_id INT UNSIGNED NOT NULL,
+  INDEX nwsp_page_id (nwsp_page_id),
+  PRIMARY KEY(nwsp_subject_id, nwsp_page_id)
+) /*$wgDBTableOptions*/;

sql/neowiki_subject_page.json

@@ -0,0 +1,40 @@
+[
+	{
+		"name": "neowiki_subject_page",
+		"comment": "Which page holds which Subject. Written with the revision that changes a page's Subjects, so it is never staler than the slot it derives from. Reads join page, so page moves need no maintenance here and rows left by a deleted page resolve to nothing.",
+		"columns": [
+			{
+				"name": "nwsp_subject_id",
+				"comment": "Subject id as stored in the subject slot",
+				"type": "binary",
+				"options": {
+					"notnull": true,
+					"length": 32
+				}
+			},
+			{
+				"name": "nwsp_page_id",
+				"comment": "Page holding the Subject. Not a foreign key: a row for a page that no longer exists is inert, since reads join page.",
+				"type": "integer",
+				"options": {
+					"notnull": true,
+					"unsigned": true
+				}
+			}
+		],
+		"indexes": [
+			{
+				"name": "nwsp_page_id",
+				"comment": "Index for reading and replacing everything a page holds",
+				"columns": [
+					"nwsp_page_id"
+				],
+				"unique": false
+			}
+		],
+		"pk": [
+			"nwsp_subject_id",
+			"nwsp_page_id"
+		]
+	}
+]

sql/postgres/neowiki_subject_page.sql

@@ -0,0 +1,11 @@
+-- This file is automatically generated using maintenance/generateSchemaSql.php.
+-- Source: sql/neowiki_subject_page.json
+-- Do not modify this file directly.
+-- See https://www.mediawiki.org/wiki/Manual:Schema_changes
+CREATE TABLE neowiki_subject_page (
+  nwsp_subject_id TEXT NOT NULL,
+  nwsp_page_id INT NOT NULL,
+  PRIMARY KEY(nwsp_subject_id, nwsp_page_id)
+);
+
+CREATE INDEX nwsp_page_id ON neowiki_subject_page (nwsp_page_id);

sql/sqlite/neowiki_subject_page.sql

@@ -0,0 +1,11 @@
+-- This file is automatically generated using maintenance/generateSchemaSql.php.
+-- Source: sql/neowiki_subject_page.json
+-- Do not modify this file directly.
+-- See https://www.mediawiki.org/wiki/Manual:Schema_changes
+CREATE TABLE /*_*/neowiki_subject_page (
+  nwsp_subject_id BLOB NOT NULL,
+  nwsp_page_id INTEGER UNSIGNED NOT NULL,
+  PRIMARY KEY(nwsp_subject_id, nwsp_page_id)
+);
+
+CREATE INDEX nwsp_page_id ON /*_*/neowiki_subject_page (nwsp_page_id);

src/Application/Actions/CreateSubject/CreateSubjectAction.php

@@ -103,8 +103,7 @@ public function createSubject( CreateSubjectRequest $request ): void {
 		}
 
 		// The page identifiers come from the page id the request named, not from the subject -> page
-		// index: that index is the graph projection, which a read replica may not carry yet for the
-		// revision just written.
+		// index, which is read from a replica that may not carry the revision just written.
 		$this->presenter->presentCreated(
 			GetSubjectResponseItem::fromSubject(
 				$subject,
@@ -151,8 +150,9 @@ private function buildSubject( CreateSubjectRequest $request, ?Schema $schema ):
 	}
 
 	/**
-	 * Best-effort global uniqueness check: the subject -> page index lags slot writes, so this can
-	 * miss a very recently created Subject; ID entropy carries the rest (same posture as relation IDs).
+	 * Best-effort global uniqueness check: the subject -> page index is read from a replica, so this
+	 * can miss a Subject another request just created; ID entropy carries the rest (same posture as
+	 * relation IDs).
 	 */
 	private function subjectIdIsInUse( SubjectId $id ): bool {
 		return $this->pageIdentifiersLookup->getPageIdOfSubject( $id ) !== null;