From b3d39d3a061e800d9dc27bd08e6a21f9d7aa2cc1 Mon Sep 17 00:00:00 2001 From: Jordyn Adrales Date: Thu, 6 Aug 2026 14:10:22 -0500 Subject: [PATCH 1/3] Improve MySQL-to-PostgreSQL migration troubleshooting docs Addresses gaps surfaced by support ticket 51867 (pgloader column-mismatch failure caused by migration-version skew): - Clarify that --run-migrations builds the Postgres schema from the specified version's migrations, not from MySQL; add guidance on confirming the actual running Mattermost version before migrating - Warn against starting the Mattermost server between schema creation and pgloader, which causes duplicate-key constraint errors - Expand the "Failed to find column or table" troubleshooting entry with root cause, real error text, and resolution steps - Add new "Duplicate key errors on a fresh target" troubleshooting section covering the non-pristine target scenario - Add new "Reloading a single failed table" section with step-by-step recovery using INCLUDING ONLY TABLE NAMES MATCHING and row-count verification Co-Authored-By: Claude Sonnet 4.6 --- .../postgres-migration-assist-tool.rst | 14 ++- .../deployment-guide/postgres-migration.rst | 95 ++++++++++++++++++- 2 files changed, 105 insertions(+), 4 deletions(-) diff --git a/source/deployment-guide/postgres-migration-assist-tool.rst b/source/deployment-guide/postgres-migration-assist-tool.rst index 7d8369def38..359f2881cbd 100644 --- a/source/deployment-guide/postgres-migration-assist-tool.rst +++ b/source/deployment-guide/postgres-migration-assist-tool.rst @@ -25,6 +25,7 @@ Usage - If you encounter heap exhaustion errors in ``pgloader``, edit your generated ``migration.load`` and under the ``WITH`` block set: ``prefetch rows = 1000`` and consider reducing it if the issue persists. - Please make sure you have the necessary environment to perform the migration. Ensure that the MySQL and PostgreSQL databases are running and accessible. To set up a PostgreSQL instance, see the :doc:`prepare your Mattermost database ` documentation for details. - If you were previously utilizing a database for handling the :doc:`Mattermost configuration `, those tables will not be migrated from your MySQL database with the migration `script <#migrate-the-data>`__. You will need to manually migrate those configuration settings to your PostgreSQL database after completing the migration process. See the :ref:`configuration in database ` documentation for details. + - Stop the Mattermost server before beginning these steps and do **not** start it against the target PostgreSQL database until after pgloader has completed. Starting the app between schema creation and data transfer writes rows to the target (for example, plugin key-value store entries) that will conflict with pgloader's inserts and produce ``duplicate key value violates unique constraint`` errors. Step 1 - Check the MySQL database schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -63,6 +64,17 @@ Then run: - ```` example: ``postgres://user:password@address:5432/db_name`` - ```` example: ``10.5.4`` +.. important:: + + ``--run-migrations`` builds the PostgreSQL schema by applying that version's migration scripts — it does **not** copy the schema from MySQL. Set ``--mattermost-version`` to the version of Mattermost currently running against your source MySQL database (or a later patch release on the same line). Specifying an older version omits columns that were added in later patch releases, causing pgloader to fail with ``pgloader failed to find column … in target table``. + + To confirm your actual version before migrating: + + - **Mattermost**: click the grid menu in the top left and select **About Mattermost**. + - **CLI**: run ``mattermost version`` on the server. + + If you are migrating multiple environments, use the same ``--mattermost-version`` value across all of them. Version skew between nodes produces inconsistent target schemas and is a common source of pgloader failures. + By default, two pre-checks run before migration: - ``--check-schema-owner=true`` @@ -104,7 +116,7 @@ Step 4 - Run pgloader pgloader migration.load > migration.log -Carefully review `migration.log` for errors (e.g., duplicate-key or missing-table warnings). Use the ``mattermost/pgloader:latest`` Docker image to avoid build/auth issues. +Carefully review ``migration.log`` for errors. In particular, check the summary table printed at the end of the run: any table whose ``errors`` column is non-zero did not load successfully and its rows were not transferred. For those tables, compare row counts between source and target to confirm what is missing, then see :ref:`Reloading a single failed table ` for recovery steps. Use the ``mattermost/pgloader:latest`` Docker image to avoid build/auth issues. Step 5 - Restore full-text indexes & create all indexes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/deployment-guide/postgres-migration.rst b/source/deployment-guide/postgres-migration.rst index f73f1dccac3..cdaaef2e7a4 100644 --- a/source/deployment-guide/postgres-migration.rst +++ b/source/deployment-guide/postgres-migration.rst @@ -89,7 +89,7 @@ If you encounter errors during the execution of the ``pgloader`` command, ensure .. note:: - For experienced users, it is recoverable to run the ``pgloader`` without requiring a restart of the migration from scratch. In this case, you will need to manually fix the issues with the table, and run the ``pgloader`` command with a tailored configuration specifically for those tables. Also ensure that the schema name is reverted back to ``public``, and the ``search_path`` is restored (or remove necessary clauses from the configuration). + For experienced users, it is possible to reload only the failed table without restarting the entire migration. See :ref:`Reloading a single failed table ` for step-by-step instructions. The following sections detail how to resolve some common errors you may encounter during the execution of the ``pgloader`` command: @@ -117,9 +117,29 @@ If you receive an error message similar to the following: .. code-block:: text - pgloader failed to find column + KABOOM! SIMPLE-ERROR: pgloader failed to find column + "mattermost"."Roles"."schemeid" in target table "\"mattermost\".\"roles\"" -The column or table is missing in the PostgreSQL database. You can fix this issue by checking whether you have created the correct version of Postgres schema. After re-creating the schema, you can run the ``pgloader`` command again. +The column exists in the source MySQL database but is absent from the target PostgreSQL schema. This is almost always caused by passing an older ``--mattermost-version`` to ``migration-assist postgres --run-migrations`` than the version actually running on your source server. + +``--run-migrations`` builds the PostgreSQL schema by replaying that version's migration scripts — it does not copy the schema from MySQL. If the version you specify predates a migration that adds a column your source database already has (for example, a column added in a recent patch release), pgloader finds a source column with no target counterpart and aborts. + +**To resolve:** + +1. Confirm your actual Mattermost version (the version running against the source MySQL database): + + - **Mattermost**: click the grid menu in the top left and select **About Mattermost**. + - **CLI**: run ``mattermost version`` on the server. + +2. Drop and recreate the target PostgreSQL database, then rebuild the schema using the correct version: + + .. code-block:: sh + + migration-assist postgres "" \ + --run-migrations \ + --mattermost-version="" + +3. Re-run pgloader against the freshly built schema. Fell through ECASE expression ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -136,6 +156,75 @@ It is a `known issue `__ with p Also, there may be cases where pgloader continues to migrate remaining tables and skip one or more tables during migration. In such cases, we recommend identifying issues with the table and fixing them before running the ``pgloader`` command again with a clean database. It is possible to run the ``pgloader`` command with the ``--debug`` flag to get more information about the errors. +Duplicate key errors on a fresh target +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you receive an error message similar to the following: + +.. code-block:: text + + ERROR Database error 23505: duplicate key value violates unique constraint + "pluginkeyvaluestore_pkey" + mattermost.pluginkeyvaluestore errors=1 rows=0 + +The target table already contains rows before pgloader attempts to insert. On a freshly built PostgreSQL database (one created with ``--run-migrations`` and nothing else), this is caused by one of: + +- The **Mattermost server was started against the target database** before pgloader ran. Starting the app causes plugins to write key-value entries and other initial data into target tables. +- The ``--run-migrations`` step was **run more than once**, which seeds duplicate rows on some table types. + +**Prevention:** Stop the Mattermost server before the migration and do not start it against the target PostgreSQL database until after pgloader completes successfully. + +**Resolution:** If only one or a few tables have errors and the rest of the migration succeeded, truncate the affected table(s) and reload them — see :ref:`Reloading a single failed table ` below. If multiple tables are affected, the cleanest path is to rebuild the target from scratch: drop and recreate the PostgreSQL database, re-run ``--run-migrations`` without starting the Mattermost server, then run pgloader again. + +Reloading a single failed table +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If pgloader failed on one table but succeeded on the rest, you can reload only that table without restarting the full migration: + +1. Generate the pgloader configuration file (if you don't have one already): + + .. code-block:: sh + + migration-assist pgloader \ + --mysql="" \ + --postgres="" \ + --remove-null-chars \ + --output=migration.load + +2. Open ``migration.load`` and replace the ``EXCLUDING TABLE NAMES MATCHING`` clause with an ``INCLUDING ONLY TABLE NAMES MATCHING`` clause that names only the failed table: + + .. code-block:: text + + -- Change from: + EXCLUDING TABLE NAMES MATCHING 'schema_migrations', ... + + -- To (use your actual table name): + INCLUDING ONLY TABLE NAMES MATCHING 'pluginkeyvaluestore' + +3. If the table already has partial data on the target, truncate it first: + + .. code-block:: sql + + TRUNCATE ; + +4. Re-run pgloader with the modified configuration: + + .. code-block:: sh + + pgloader migration.load + +5. Verify that the load succeeded by comparing row counts between source and target: + + .. code-block:: sql + + -- MySQL (source) + SELECT COUNT(*) FROM ; + + -- PostgreSQL (target) + SELECT COUNT(*) FROM ; + + Also check the pgloader summary at the end of the run — the ``errors`` column for the reloaded table should be ``0``. + Mattermost can't connect to the PostgreSQL database ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 6b22e93b547e9a0e061184c47e01023c3fac6e8e Mon Sep 17 00:00:00 2001 From: Jordyn Adrales Date: Mon, 17 Aug 2026 13:31:32 -0500 Subject: [PATCH 2/3] Address PR feedback on MySQL-to-PostgreSQL migration docs - Condense maintenance window warnings into a single planning bullet in the top-level important block, with detailed reminders added inline at Step 2 (don't start Mattermost against the new Postgres DB) and Step 4 (confirm Mattermost is offline on MySQL before pgloader runs) - Note that --mattermost-version fetches scripts from GitHub.com and link to the air-gapped environments section for offline deployments Co-Authored-By: Claude Sonnet 4.6 --- .../postgres-migration-assist-tool.rst | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/source/deployment-guide/postgres-migration-assist-tool.rst b/source/deployment-guide/postgres-migration-assist-tool.rst index 359f2881cbd..0a2d5d26e7a 100644 --- a/source/deployment-guide/postgres-migration-assist-tool.rst +++ b/source/deployment-guide/postgres-migration-assist-tool.rst @@ -25,7 +25,7 @@ Usage - If you encounter heap exhaustion errors in ``pgloader``, edit your generated ``migration.load`` and under the ``WITH`` block set: ``prefetch rows = 1000`` and consider reducing it if the issue persists. - Please make sure you have the necessary environment to perform the migration. Ensure that the MySQL and PostgreSQL databases are running and accessible. To set up a PostgreSQL instance, see the :doc:`prepare your Mattermost database ` documentation for details. - If you were previously utilizing a database for handling the :doc:`Mattermost configuration `, those tables will not be migrated from your MySQL database with the migration `script <#migrate-the-data>`__. You will need to manually migrate those configuration settings to your PostgreSQL database after completing the migration process. See the :ref:`configuration in database ` documentation for details. - - Stop the Mattermost server before beginning these steps and do **not** start it against the target PostgreSQL database until after pgloader has completed. Starting the app between schema creation and data transfer writes rows to the target (for example, plugin key-value store entries) that will conflict with pgloader's inserts and produce ``duplicate key value violates unique constraint`` errors. + - This migration requires a maintenance window. Mattermost must be taken offline on the MySQL host before pgloader runs, and must not be started against the target PostgreSQL database until the migration is complete. Step 1 - Check the MySQL database schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -75,6 +75,12 @@ Then run: If you are migrating multiple environments, use the same ``--mattermost-version`` value across all of them. Version skew between nodes produces inconsistent target schemas and is a common source of pgloader failures. + When using ``--mattermost-version``, the tool fetches migration scripts directly from GitHub.com. If your environment is air-gapped or does not have internet access, see :ref:`Air-gapped environments ` for an alternative approach. + +.. note:: + + Once this step completes, do not start Mattermost against the new PostgreSQL database until pgloader has finished. Starting the app writes initial data to the target (for example, plugin key-value store entries) that will conflict with pgloader's inserts and produce ``duplicate key value violates unique constraint`` errors. + By default, two pre-checks run before migration: - ``--check-schema-owner=true`` @@ -110,6 +116,8 @@ Run the following command to emit a pgloader configuration file: Step 4 - Run pgloader ~~~~~~~~~~~~~~~~~~~~~ +Before running pgloader, confirm that Mattermost is offline on the MySQL host so that no new data is written to the source database during the transfer. + :ref:`Run pgloader ` with the generated configuration file: .. code-block:: sh From b10a27f9a3aa80b6276b0b4d12f6462d525d6ffe Mon Sep 17 00:00:00 2001 From: Jordyn Adrales Date: Fri, 21 Aug 2026 09:14:49 -0500 Subject: [PATCH 3/3] Address PR review comments on migration docs - Fix multiple-environment guidance: use each environment's own --mattermost-version rather than one shared value when versions differ - Fix partial load description: non-zero errors means incomplete load, not zero rows transferred - Acknowledge --applied-migrations path in --run-migrations note; correct pre-packaged migrations claim (ships through v10.12/141, no internet access required for MySQL-supported versions) - Add schema-owner setup step to rebuild-from-scratch recovery paths; dropping the database resets public schema ownership - Add TRUNCATE prerequisite note (Mattermost stopped, dedicated target) Co-Authored-By: Claude Sonnet 4.6 --- .../postgres-migration-assist-tool.rst | 8 ++++---- source/deployment-guide/postgres-migration.rst | 12 +++++++++--- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/source/deployment-guide/postgres-migration-assist-tool.rst b/source/deployment-guide/postgres-migration-assist-tool.rst index 0a2d5d26e7a..b579531e9bd 100644 --- a/source/deployment-guide/postgres-migration-assist-tool.rst +++ b/source/deployment-guide/postgres-migration-assist-tool.rst @@ -66,16 +66,16 @@ Then run: .. important:: - ``--run-migrations`` builds the PostgreSQL schema by applying that version's migration scripts — it does **not** copy the schema from MySQL. Set ``--mattermost-version`` to the version of Mattermost currently running against your source MySQL database (or a later patch release on the same line). Specifying an older version omits columns that were added in later patch releases, causing pgloader to fail with ``pgloader failed to find column … in target table``. + ``--run-migrations`` builds the PostgreSQL schema by applying Mattermost's migration scripts — it does **not** directly copy the schema structure from MySQL. Set ``--mattermost-version`` to the version of Mattermost currently running against your source MySQL database (or a later patch release on the same line). Specifying an older version omits columns that were added in later patch releases, causing pgloader to fail with ``pgloader failed to find column … in target table``. Alternatively, if your database schema has diverged from an official release, ``--applied-migrations`` can be supplied with the ``mysql.output`` file generated by ``migration-assist mysql`` to apply only the migrations matching your source database's state — see :ref:`Air-gapped environments ` for details. To confirm your actual version before migrating: - **Mattermost**: click the grid menu in the top left and select **About Mattermost**. - **CLI**: run ``mattermost version`` on the server. - If you are migrating multiple environments, use the same ``--mattermost-version`` value across all of them. Version skew between nodes produces inconsistent target schemas and is a common source of pgloader failures. + If multiple environments run the same Mattermost version, use the same ``--mattermost-version`` value. Otherwise, use the version running against each environment's source MySQL database. Version skew between environments produces inconsistent target schemas and is a common source of pgloader failures. - When using ``--mattermost-version``, the tool fetches migration scripts directly from GitHub.com. If your environment is air-gapped or does not have internet access, see :ref:`Air-gapped environments ` for an alternative approach. + The tool ships with pre-packaged migration scripts through v10.12 (migration 141), covering all MySQL-supported Mattermost versions — no internet access is required for these versions. If your environment is air-gapped or your database schema has diverged from an official release, see :ref:`Air-gapped environments ` for an alternative approach using ``--applied-migrations``. .. note:: @@ -124,7 +124,7 @@ Before running pgloader, confirm that Mattermost is offline on the MySQL host so pgloader migration.load > migration.log -Carefully review ``migration.log`` for errors. In particular, check the summary table printed at the end of the run: any table whose ``errors`` column is non-zero did not load successfully and its rows were not transferred. For those tables, compare row counts between source and target to confirm what is missing, then see :ref:`Reloading a single failed table ` for recovery steps. Use the ``mattermost/pgloader:latest`` Docker image to avoid build/auth issues. +Carefully review ``migration.log`` for errors. In particular, check the summary table printed at the end of the run: any table whose ``errors`` column is non-zero did not load completely; some rows may have transferred. For those tables, compare row counts between source and target to confirm what is missing, then see :ref:`Reloading a single failed table ` for recovery steps. Use the ``mattermost/pgloader:latest`` Docker image to avoid build/auth issues. Step 5 - Restore full-text indexes & create all indexes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/deployment-guide/postgres-migration.rst b/source/deployment-guide/postgres-migration.rst index cdaaef2e7a4..db9de9db585 100644 --- a/source/deployment-guide/postgres-migration.rst +++ b/source/deployment-guide/postgres-migration.rst @@ -131,7 +131,13 @@ The column exists in the source MySQL database but is absent from the target Pos - **Mattermost**: click the grid menu in the top left and select **About Mattermost**. - **CLI**: run ``mattermost version`` on the server. -2. Drop and recreate the target PostgreSQL database, then rebuild the schema using the correct version: +2. Drop and recreate the target PostgreSQL database. Before running ``--run-migrations``, restore the schema-owner setup — dropping the database resets public schema ownership and the pre-check will fail without it: + + .. code-block:: sh + + sudo -u postgres psql -d mattermost -c "ALTER SCHEMA public OWNER TO mmuser; GRANT ALL ON SCHEMA public TO mmuser;" + + Then rebuild the schema using the correct version: .. code-block:: sh @@ -174,7 +180,7 @@ The target table already contains rows before pgloader attempts to insert. On a **Prevention:** Stop the Mattermost server before the migration and do not start it against the target PostgreSQL database until after pgloader completes successfully. -**Resolution:** If only one or a few tables have errors and the rest of the migration succeeded, truncate the affected table(s) and reload them — see :ref:`Reloading a single failed table ` below. If multiple tables are affected, the cleanest path is to rebuild the target from scratch: drop and recreate the PostgreSQL database, re-run ``--run-migrations`` without starting the Mattermost server, then run pgloader again. +**Resolution:** If only one or a few tables have errors and the rest of the migration succeeded, truncate the affected table(s) and reload them — see :ref:`Reloading a single failed table ` below. If multiple tables are affected, the cleanest path is to rebuild the target from scratch: drop and recreate the PostgreSQL database, re-run the schema-owner setup commands from Step 2 of the migration guide, then re-run ``--run-migrations`` without starting the Mattermost server, and run pgloader again. Reloading a single failed table ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -201,7 +207,7 @@ If pgloader failed on one table but succeeded on the rest, you can reload only t -- To (use your actual table name): INCLUDING ONLY TABLE NAMES MATCHING 'pluginkeyvaluestore' -3. If the table already has partial data on the target, truncate it first: +3. If the table already has partial data on the target, truncate it first. Ensure Mattermost is stopped and confirm this is your dedicated migration target database before proceeding: .. code-block:: sql