[19.0] fs_attachment: single-transaction force_storage() races fs.file.gc autovacuum — uploads deleted mid-migration or swept entirely on failure

[panayiotiska]

Module

fs_attachment 19.0.1.1.2 (fs_storage 19.0.1.1.2)

Describe the bug

_force_storage_to_object_storage() migrates the entire filestore in one long-lived uncommitted transaction (per-attachment savepoints, single commit at the very end), while fs.file.gc._mark_for_gc() deliberately commits its marks in a separate cursor so they survive rollbacks. The combination makes a large migration structurally unsafe:

1. GC race: _gc_files() (via @api.autovacuum, i.e. the daily "Base: Auto-vacuum internal data" cron) deletes every marked file that has no committed ir_attachment referencing it. While a long migration is still running, all of its uploads are uncommitted, so an autovacuum pass that fires mid-migration deletes objects the migration will then commit store_fname pointers to — silent dangling pointers on success.
2. Total upload loss on failure: if the migration transaction dies mid-run (in our case it stalled inside an S3 write for hours — no botocore timeouts configured — and the transaction eventually aborted), the rollback plus the already-committed GC marks mean every uploaded object is swept on the next autovacuum. We reconstructed this precisely from S3 bucket versioning on a 14k-file production migration: continuous per-file delete markers during the run (version churn), an 884-object burst at transaction death, and a final sweep when the daily autovacuum ran two hours later.

Related: #600 (the shared-file dedup problem also bites here — the classic filestore dedups identical content into one file, and clean_fs() after commit removes files that other, not-yet-migrated attachments with the same checksum still point to).

To Reproduce

1. Configure an S3 (or any fs) storage as default for attachments on a database with enough attachments that the migration takes longer than the gap to the next autovacuum run (or trigger env["fs.file.gc"]._gc_files() manually mid-migration from a second session).
2. Run env["ir.attachment"].force_storage().
3. Observe objects being deleted from the bucket while the migration is still running (bucket versioning makes this easy to see); on completion, migrated store_fnames point at deleted keys.

Expected behavior

Migration should be resilient to concurrent GC and to mid-run failure. What worked for us in production (14,254 attachments, zero errors, zero losses, verified under an adversarial _gc_files() loop every 5 seconds):

• Commit in batches (~200 attachments): uploads become committed-referenced within seconds, so _gc_files()'s NOT EXISTS check protects them; a failure costs one batch; reruns resume naturally from the same domain.
• Defer clean_fs() (or skip it and let the core filestore GC reap unreferenced files): deleting shared dedup'd files between batches breaks same-checksum attachments migrated later (fs_attachment: migrating a large number of attachments to S3 #600 within a single run).
• Recommend documenting botocore timeouts (config_kwargs: {"connect_timeout": ..., "read_timeout": ..., "retries": ...}) in fs_attachment_s3 — an S3 write with no timeout can hang a migration transaction indefinitely.

Happy to turn the batched approach into a PR if maintainers agree with the direction.

Additional context

Odoo 19.0, s3fs/fsspec 2026.7.0, EC2 → same-region S3. Sibling issue about configuration: #657.