Improve MySQL-to-PostgreSQL migration troubleshooting docs #9131
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -25,6 +25,7 @@ Usage | |||||
| - If you encounter heap exhaustion errors in ``pgloader``, edit your generated ``migration.load`` and under the ``WITH`` block set: ``prefetch rows = 1000`` and consider reducing it if the issue persists. | ||||||
| - Please make sure you have the necessary environment to perform the migration. Ensure that the MySQL and PostgreSQL databases are running and accessible. To set up a PostgreSQL instance, see the :doc:`prepare your Mattermost database </deployment-guide/server/preparations>` documentation for details. | ||||||
| - If you were previously utilizing a database for handling the :doc:`Mattermost configuration </administration-guide/configure/configuration-in-your-database>`, those tables will not be migrated from your MySQL database with the migration `script <#migrate-the-data>`__. You will need to manually migrate those configuration settings to your PostgreSQL database after completing the migration process. See the :ref:`configuration in database <deployment-guide/manual-postgres-migration:configuration in database>` documentation for details. | ||||||
| - Stop the Mattermost server before beginning these steps and do **not** start it against the target PostgreSQL database until after pgloader has completed. Starting the app between schema creation and data transfer writes rows to the target (for example, plugin key-value store entries) that will conflict with pgloader's inserts and produce ``duplicate key value violates unique constraint`` errors. | ||||||
|
|
||||||
| Step 1 - Check the MySQL database schema | ||||||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||||||
|
|
@@ -63,6 +64,17 @@ Then run: | |||||
| - ``<POSTGRES_DSN>`` example: ``postgres://user:password@address:5432/db_name`` | ||||||
| - ``<MATTERMOST_VERSION>`` example: ``10.5.4`` | ||||||
|
|
||||||
| .. important:: | ||||||
|
|
||||||
| ``--run-migrations`` builds the PostgreSQL schema by applying that version's migration scripts — it does **not** copy the schema from MySQL. Set ``--mattermost-version`` to the version of Mattermost currently running against your source MySQL database (or a later patch release on the same line). Specifying an older version omits columns that were added in later patch releases, causing pgloader to fail with ``pgloader failed to find column … in target table``. | ||||||
|
|
||||||
| To confirm your actual version before migrating: | ||||||
|
|
||||||
| - **Mattermost**: click the grid menu in the top left and select **About Mattermost**. | ||||||
| - **CLI**: run ``mattermost version`` on the server. | ||||||
|
|
||||||
| If you are migrating multiple environments, use the same ``--mattermost-version`` value across all of them. Version skew between nodes produces inconsistent target schemas and is a common source of pgloader failures. | ||||||
|
|
||||||
|
coderabbitai[bot] marked this conversation as resolved.
|
||||||
| By default, two pre-checks run before migration: | ||||||
|
|
||||||
| - ``--check-schema-owner=true`` | ||||||
|
|
@@ -104,7 +116,7 @@ Step 4 - Run pgloader | |||||
|
|
||||||
| pgloader migration.load > migration.log | ||||||
|
|
||||||
| Carefully review ``migration.log`` for errors. In particular, check the summary table printed at the end of the run: any table whose ``errors`` column is non-zero did not load successfully and its rows were not transferred. For those tables, compare row counts between source and target to confirm what is missing, then see :ref:`Reloading a single failed table <deployment-guide/postgres-migration:reloading a single failed table>` for recovery steps. Use the ``mattermost/pgloader:latest`` Docker image to avoid build/auth issues. | ||||||
|
Contributor
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
# Locate and inspect the referenced file and relevant documentation around the reported lines.
if [ -f source/deployment-guide/postgres-migration-assist-tool.rst ]; then
echo "== file exists =="
wc -l source/deployment-guide/postgres-migration-assist-tool.rst
echo "== lines 1-40 =="
sed -n '1,40p' source/deployment-guide/postgres-migration-assist-tool.rst | cat -n
echo "== lines 105-130 =="
sed -n '105,130p' source/deployment-guide/postgres-migration-assist-tool.rst | cat -n
echo "== pgloader image mentions =="
rg -n "pgloader|authentication|caching_sha2_password|Docker image|source-build|custom-image" source/deployment-guide/postgres-migration-assist-tool.rst
else
echo "file not found"
fiRepository: mattermost/docs Length of output: 9235 Qualify the Docker image recommendation. Line 18 says 🤖 Prompt for AI Agents🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg '(^|/)(postgres-migration-assist-tool\.rst|postgres-migration|pgloader|migration\.log|RELEAS|CHANGELOG|README|.*migration.*)' || true
echo "== target file around lines 90-135 =="
sed -n '90,140p' source/deployment-guide/postgres-migration-assist-tool.rst 2>/dev/null || true
echo "== search migration log summary/errors context =="
rg -n "migration\.log|summary|errors column|columns errors|did not load|rows were not transferred|row count" source docs deployment-guide 2>/dev/null || true
echo "== search pgloader migration assist references =="
rg -n "pgloader|migration assist|pg_migration|migration log|errors" . --glob '!**/.git/**' --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' 2>/dev/null | head -n 200 || true
echo "== git diff stat/name =="
git diff --stat || true
git diff --name-only || trueRepository: mattermost/docs Length of output: 50371 🌐 Web query:
💡 Result: When running a load, pgloader displays a terminal summary table that tracks the progress and outcome of the operation [1][2]. This table includes columns for the table name, the number of rows read, the number of rows successfully imported, the number of errors encountered, and the time taken for each stage [1][2]. Error Handling and Batching Behavior pgloader uses the PostgreSQL COPY protocol for high-performance data loading [3][4][5]. Because the COPY protocol rejects an entire data set if any single part fails, pgloader manages this by splitting input data into batches (typically 25,000 rows each) [3][4][5]. When a batch is rejected: 1. pgloader parses the PostgreSQL CONTEXT error message to identify the specific row and column causing the failure [3][4][5]. 2. It then isolates the erroneous row(s) and retries the remaining rows in smaller batches to ensure maximum data recovery [3][4][5]. 3. Any rows that cannot be loaded are written to reject files [3][4][5]. Error Reporting When errors occur, pgloader generates two files in a directory named after the target database within your specified root directory [3][4][5]: -.dat file: Contains the raw data of the rejected rows [3][4][5]. -.log file: Contains the detailed PostgreSQL client-side logs related to the rejection [3][4][5]. Summary Output You can generate a machine-readable summary of the entire operation using the --summary command-line option [6][4]. By providing a filename with extensions such as.csv,.copy, or.json, you can export this summary for further automated processing [6][4]. If no extension or a standard filename is provided, the output defaults to a human-readable format [6][4]. Citations:
🌐 Web query:
💡 Result: In the pgloader output summary table, the columns provide a breakdown of the migration process for each table or task [1][2][3]. The meaning of the columns is as follows: - table name: The name of the table, schema object, or migration phase (e.g., "fetch meta data", "Create Indexes") being processed [2][3]. - read: The number of rows (or items) read from the source database or file [2][3]. - imported: The number of rows successfully inserted into the target PostgreSQL database [2][3]. - errors: The number of rows that failed to import, typically due to data issues such as constraint violations or format mismatches [4][2][5]. - total time: The duration taken to process that specific table or task [2][3]. When the "errors" column shows a value other than zero, it indicates that some rows were rejected during the migration [4][5]. Depending on your configuration, pgloader may be set to "on error resume next" (the default for file-based loads), allowing it to continue migrating remaining rows while logging the rejected ones to separate files, typically named reject.dat and reject.log [5]. If you see "some rows transferred" alongside reported errors, it confirms that while the migration completed, it was only a partial success due to those specific rejected records [4][5]. For more detailed diagnostic information, you should check the reject files to identify the specific rows and the nature of the errors (e.g., invalid data types, null constraint violations) [4][5]. Citations:
Describe partial loads accurately. A non-zero Suggested wording- any table whose ``errors`` column is non-zero did not load successfully and its rows were not transferred.
+ any table whose ``errors`` column is non-zero did not load completely; some rows may have transferred.📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents |
||||||
|
|
||||||
| Step 5 - Restore full-text indexes & create all indexes | ||||||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||||||
|
|
||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -89,7 +89,7 @@ If you encounter errors during the execution of the ``pgloader`` command, ensure | |
|
|
||
| .. note:: | ||
|
|
||
| For experienced users, it is possible to reload only the failed table without restarting the entire migration. See :ref:`Reloading a single failed table <deployment-guide/postgres-migration:reloading a single failed table>` for step-by-step instructions. | ||
|
|
||
| The following sections detail how to resolve some common errors you may encounter during the execution of the ``pgloader`` command: | ||
|
|
||
|
|
@@ -117,9 +117,29 @@ If you receive an error message similar to the following: | |
|
|
||
| .. code-block:: text | ||
|
|
||
| KABOOM! SIMPLE-ERROR: pgloader failed to find column | ||
| "mattermost"."Roles"."schemeid" in target table "\"mattermost\".\"roles\"" | ||
|
|
||
| The column exists in the source MySQL database but is absent from the target PostgreSQL schema. This is almost always caused by passing an older ``--mattermost-version`` to ``migration-assist postgres --run-migrations`` than the version actually running on your source server. | ||
|
|
||
| ``--run-migrations`` builds the PostgreSQL schema by replaying that version's migration scripts — it does not copy the schema from MySQL. If the version you specify predates a migration that adds a column your source database already has (for example, a column added in a recent patch release), pgloader finds a source column with no target counterpart and aborts. | ||
|
|
||
| **To resolve:** | ||
|
|
||
| 1. Confirm your actual Mattermost version (the version running against the source MySQL database): | ||
|
|
||
| - **Mattermost**: click the grid menu in the top left and select **About Mattermost**. | ||
| - **CLI**: run ``mattermost version`` on the server. | ||
|
|
||
| 2. Drop and recreate the target PostgreSQL database, then rebuild the schema using the correct version: | ||
|
|
||
| .. code-block:: sh | ||
|
|
||
| migration-assist postgres "<POSTGRES_DSN>" \ | ||
| --run-migrations \ | ||
| --mattermost-version="<ACTUAL_VERSION>" | ||
|
|
||
| 3. Re-run pgloader against the freshly built schema. | ||
|
Contributor
There was a problem hiding this comment. 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win Add a warning before destructive recovery. Both recovery paths can delete the target database. The instructions do not require the administrator to confirm that the database is a dedicated migration target with no required data. Add a backup or new-database requirement, and require Mattermost to remain stopped. As per coding guidelines, use a Also applies to: 175-177 🤖 Prompt for AI AgentsSource: Coding guidelines 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
printf 'Files:\n'
git ls-files | rg 'source/deployment-guide/(postgres-migration|postgres-migration-assist-tool)\.rst$' || true
printf '\nReference snippet postgres-migration.rst:\n'
sed -n '110,155p' source/deployment-guide/postgres-migration.rst
printf '\nRelevant snippet postgres-migration-assist-tool.rst:\n'
sed -n '35,70p' source/deployment-guide/postgres-migration-assist-tool.rst
printf '\nSearch schema owner/grants mentions:\n'
rg -n "schema-owner|owner to|OWNER TO|check-schema-owner|schema owner|pgloader|recreate|DROP DATABASE|CREATE DATABASE|run-migrations|ACTUAL_VERSION|POSTGRES_DSN|migration-assist" source/deployment-guide/postgres-migration*.rstRepository: mattermost/docs Length of output: 21918 🌐 Web query:
💡 Result: In PostgreSQL, managing databases, schemas, and ownership involves specific rules that have evolved significantly in recent versions. DATABASE OWNERSHIP AND CREATION To create a database, a user must have the CREATEDB privilege or be a superuser [1][2]. When creating a database, you can specify an owner using the OWNER clause [1][2]. If no owner is specified, it defaults to the user executing the command [1][2]. The DROP DATABASE command removes the entire database, including all its catalog entries and data files, and can only be executed by the database owner or a superuser [3]. PUBLIC SCHEMA AND PG_DATABASE_OWNER Starting with PostgreSQL 15, there was a major security change regarding the default public schema [4][5]. 1. Ownership: The public schema is now owned by a special predefined role called pg_database_owner [6][7][4]. 2. Implicit Membership: The pg_database_owner role has exactly one implicit member: the current database owner [6][7]. This ensures that the owner of a database automatically becomes the owner of that database's public schema [6][7][4]. 3. Security Upgrade: Previously, all users had CREATE and USAGE privileges on the public schema by default [4][5]. PostgreSQL 15+ revoked the default CREATE privilege from PUBLIC, meaning only the database owner can create objects in the public schema by default [4][5]. Users still retain the USAGE privilege [4]. PRIVILEGES AND ROLE MANAGEMENT When dropping a database, it is not necessary to manually drop internal schemas or tables; the DROP DATABASE command handles the cleanup of all objects contained within [3]. However, if you are attempting to drop a role (user) who owns objects within a database, you must first reassign or drop those objects [8]. - DROP OWNED: This command removes all objects within the current database owned by a specific role and revokes privileges granted to that role [9]. - REASSIGN OWNED: This command reassigns ownership of all objects owned by a role to another role [9][8]. Note that these commands must be run inside each database where the role owns objects, as they do not affect shared objects like the database itself or tablespaces [9][8]. Summary of Key Commands: - CREATE DATABASE name OWNER user_name;: Create a database with a specific owner [1][2]. - DROP DATABASE name;: Drop an entire database (requires being the owner or a superuser) [3]. - REVOKE/GRANT CREATE ON SCHEMA public FROM/TO role;: Adjust privileges for the public schema if you need to deviate from the default security model [10][11]. Citations:
🌐 Web query:
💡 Result: The --check-schema-owner flag is a configuration option within the Mattermost migration-assist tool, used during the migration from MySQL to PostgreSQL [1][2]. Functionality: By default, the migration-assist tool performs a pre-check to verify that the database user (e.g., mmuser) owns the public schema in the target PostgreSQL database [1]. If the migration-assist tool encounters a mismatch—where the connecting user is not the owner of the public schema—it will fail with an error such as: "could not check schema owner: the user 'mmuser' is not owner of the 'public' schema" [3][4]. Configuration: - To enable or explicitly set this check: --check-schema-owner=true [1] - To disable this check: --check-schema-owner=false [1] Resolution: If you encounter this error, it indicates that the PostgreSQL user specified in your connection string lacks the necessary ownership privileges [1][5]. You can resolve this by connecting to your PostgreSQL instance as a superuser (e.g., postgres) and executing the following commands to grant the required permissions to your database user (replace mmuser and mattermost with your actual username and database name) [3][6]: ALTER SCHEMA public OWNER TO mmuser; GRANT ALL ON SCHEMA public TO mmuser; Alternatively, you may need to grant usage and create permissions: GRANT USAGE, CREATE ON SCHEMA PUBLIC TO mmuser; Note: Some versions or workflows might also require ensuring the user is the owner of the database itself [3][6]: ALTER DATABASE mattermost OWNER TO mmuser; Always ensure your database configuration aligns with the privileges required by the migration-assist tool to successfully apply migrations [1][5]. Citations:
Repeat the schema-owner setup after rebuilding the target database. Rebuilding the PostgreSQL target with 🤖 Prompt for AI Agents |
||
|
|
||
| Fell through ECASE expression | ||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
@@ -136,6 +156,75 @@ It is a `known issue <https://github.com/dimitri/pgloader/issues/1183>`__ with p | |
|
|
||
| Also, there may be cases where pgloader continues to migrate remaining tables and skip one or more tables during migration. In such cases, we recommend identifying issues with the table and fixing them before running the ``pgloader`` command again with a clean database. It is possible to run the ``pgloader`` command with the ``--debug`` flag to get more information about the errors. | ||
|
|
||
| Duplicate key errors on a fresh target | ||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| If you receive an error message similar to the following: | ||
|
|
||
| .. code-block:: text | ||
|
|
||
| ERROR Database error 23505: duplicate key value violates unique constraint | ||
| "pluginkeyvaluestore_pkey" | ||
| mattermost.pluginkeyvaluestore errors=1 rows=0 | ||
|
|
||
| The target table already contains rows before pgloader attempts to insert. On a freshly built PostgreSQL database (one created with ``--run-migrations`` and nothing else), this is caused by one of: | ||
|
|
||
| - The **Mattermost server was started against the target database** before pgloader ran. Starting the app causes plugins to write key-value entries and other initial data into target tables. | ||
| - The ``--run-migrations`` step was **run more than once**, which seeds duplicate rows on some table types. | ||
|
|
||
| **Prevention:** Stop the Mattermost server before the migration and do not start it against the target PostgreSQL database until after pgloader completes successfully. | ||
|
jadrales marked this conversation as resolved.
|
||
|
|
||
| **Resolution:** If only one or a few tables have errors and the rest of the migration succeeded, truncate the affected table(s) and reload them — see :ref:`Reloading a single failed table <deployment-guide/postgres-migration:reloading a single failed table>` below. If multiple tables are affected, the cleanest path is to rebuild the target from scratch: drop and recreate the PostgreSQL database, re-run ``--run-migrations`` without starting the Mattermost server, then run pgloader again. | ||
|
|
||
| Reloading a single failed table | ||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| If pgloader failed on one table but succeeded on the rest, you can reload only that table without restarting the full migration: | ||
|
|
||
| 1. Generate the pgloader configuration file (if you don't have one already): | ||
|
|
||
| .. code-block:: sh | ||
|
|
||
| migration-assist pgloader \ | ||
| --mysql="<MYSQL_DSN>" \ | ||
| --postgres="<POSTGRES_DSN>" \ | ||
| --remove-null-chars \ | ||
| --output=migration.load | ||
|
|
||
| 2. Open ``migration.load`` and replace the ``EXCLUDING TABLE NAMES MATCHING`` clause with an ``INCLUDING ONLY TABLE NAMES MATCHING`` clause that names only the failed table: | ||
|
|
||
| .. code-block:: text | ||
|
|
||
| -- Change from: | ||
| EXCLUDING TABLE NAMES MATCHING 'schema_migrations', ... | ||
|
|
||
| -- To (use your actual table name): | ||
| INCLUDING ONLY TABLE NAMES MATCHING 'pluginkeyvaluestore' | ||
|
|
||
| 3. If the table already has partial data on the target, truncate it first: | ||
|
|
||
| .. code-block:: sql | ||
|
|
||
| TRUNCATE <table_name>; | ||
|
Contributor
There was a problem hiding this comment. 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win Protect the target before
As per coding guidelines, use a 🤖 Prompt for AI AgentsSource: Coding guidelines |
||
|
|
||
| 4. Re-run pgloader with the modified configuration: | ||
|
|
||
| .. code-block:: sh | ||
|
|
||
| pgloader migration.load | ||
|
|
||
| 5. Verify that the load succeeded by comparing row counts between source and target: | ||
|
|
||
| .. code-block:: sql | ||
|
|
||
| -- MySQL (source) | ||
| SELECT COUNT(*) FROM <TableName>; | ||
|
|
||
| -- PostgreSQL (target) | ||
| SELECT COUNT(*) FROM <table_name>; | ||
|
|
||
| Also check the pgloader summary at the end of the run — the ``errors`` column for the reloaded table should be ``0``. | ||
|
|
||
|
|
||
| Mattermost can't connect to the PostgreSQL database | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It does somehow copies the migrations if the
mysql.outputfile is provided. Which is automatically generated whenmigration-assist mysqlcommand runs.However, it doesn't automatically does the so called copying, instead it has to be supplied with
--applied-migrationsinput.This is a handy way of dealing with some irregular database schemas if they are diverged from the official releases somehow.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the review, really appreciate it! I've pushed changes addressing this, please let me know if any further changes are needed.