chore: auto-start pending file downloads with stall timeout#41335
chore: auto-start pending file downloads with stall timeout#41335ricardogarim wants to merge 4 commits into
Conversation
|
Looks like this PR is not ready to merge, because of the following issues:
Please fix the issues and try again If you have any trouble, please check the PR guidelines |
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🦋 Changeset detectedLatest commit: 7635fba The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #41335 +/- ##
===========================================
- Coverage 69.06% 68.96% -0.10%
===========================================
Files 3655 3757 +102
Lines 143366 147736 +4370
Branches 25725 26419 +694
===========================================
+ Hits 99018 101893 +2875
- Misses 40041 41349 +1308
- Partials 4307 4494 +187
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
coderabbitai
Bot
added
the
type: feature
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/meteor/app/file-upload/server/lib/FileUpload.ts (1)
900-917: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winCatch block cleans up the upload record but not the on-disk temp file.
When
pipeline()throws,tmpFilemay already contain partial data written to disk. The catch only callsthis.store.removeById(...), which removes the DB upload record, not the file attmpFile.ufsComplete's own temp-file cleanup never runs in this failure path since it's never reached. A stray file cron job likely reclaims this eventually, but explicit cleanup here avoids transient disk buildup and matches the PR's stated intent of cleaning up "temporary files when insertion fails."🧹 Suggested fix
} catch (e) { - // The record is created before the content is written; drop it so a failed - // stream doesn't leave an orphaned upload record and tmp file behind. await this.store.removeById(fileId, { session: options?.session }).catch(() => undefined); + await fs.promises.unlink(tmpFile).catch(() => undefined); throw e; }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/meteor/app/file-upload/server/lib/FileUpload.ts` around lines 900 - 917, Update the catch block around the content write and ufsComplete flow to explicitly remove tmpFile in addition to deleting the upload record. Ensure file cleanup is attempted even when pipeline fails, and preserve the original error while tolerating cleanup failures.
🧹 Nitpick comments (3)
packages/models/src/models/Messages.ts (2)
1738-1744: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRemove newly-added implementation comments. Both sites add explanatory comments inside implementation code, which conflicts with the path instruction to avoid code comments in the implementation for
**/*.{ts,tsx,js}files.
packages/models/src/models/Messages.ts#L1738-L1744: drop the// The downloaded guard keeps a renewal racing...comment at line 1739 (or move the rationale into the PR description/commit message).apps/meteor/app/file-upload/server/lib/FileUpload.ts#L912-L916: drop the two-line comment at lines 913-914 above the cleanup call.As per path instructions: "Avoid code comments in the implementation."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/models/src/models/Messages.ts` around lines 1738 - 1744, Remove the explanatory implementation comment above renewPendingFileImportLease in packages/models/src/models/Messages.ts at lines 1738-1744, preserving the method logic unchanged. Also remove the two-line explanatory comment above the cleanup call in apps/meteor/app/file-upload/server/lib/FileUpload.ts at lines 912-916; no other code changes are needed.Source: Path instructions
1721-1753: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winAdd a supporting index for the pending-file eligibility query; consider extracting the shared predicate.
findOneAndClaimPendingFileImportandcountAllImportedMessagesWithFilesToDownloadboth filter on_importFile.downloadUrl/rocketChatUrl/downloaded/externalwith no matching entry inmodelIndexes()(lines 41-79). With a 10-worker pool polling this collection concurrently, this becomes a hot, unindexed query on what is typically the largest collection in a deployment. The eligibility predicate is also duplicated verbatim between the two methods.💡 Suggested index and shared predicate
protected override modelIndexes(): IndexDescription[] { return [ ... + { key: { '_importFile.downloadUrl': 1 }, sparse: true }, ]; }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/models/src/models/Messages.ts` around lines 1721 - 1753, Define a shared pending-file eligibility predicate for the repeated _importFile filters and reuse it in findOneAndClaimPendingFileImport and countAllImportedMessagesWithFilesToDownload, preserving each method’s additional conditions. Add a supporting index for these fields in modelIndexes(), including the eligibility fields and an ordering suitable for the lockedUntil lease query.apps/meteor/app/importer-pending-files/server/PendingFileImporter.ts (1)
78-78: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRemove the new inline implementation comments.
Use descriptive helper names for the bypass and response-draining behavior.
As per coding guidelines, “Avoid code comments in the implementation.”
Also applies to: 132-134
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/meteor/app/importer-pending-files/server/PendingFileImporter.ts` at line 78, Remove the inline comments near the fileStore filter bypass and response-draining logic, including the additional comments around the referenced later block. Replace any unclear inline behavior with descriptive helper names that communicate the bypass and response-draining responsibilities.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/meteor/app/importer-pending-files/server/PendingFileImporter.ts`:
- Around line 53-56: Update the URL validation in PendingFileImporter’s import
flow to parse downloadUrl and reject URLs that do not satisfy the approved
source-specific protocols and hosts or configured egress policy before any
request is made. Use the parsed protocol to select the corresponding HTTP
client, and preserve the existing skipMessageIds and addCountError handling for
rejected URLs.
- Around line 79-94: Update the pending-file import flow around
FileUpload.getStore('Uploads')._doInsert so attachment generation and
setImportFileRocketChatAttachment are treated as finalization, and delete the
newly inserted upload if that finalization fails. Keep progress updates such as
addCountCompleted separate and only run them after attachment commit succeeds,
preserving the existing error bookkeeping and retry behavior.
- Line 73: Update the pending-file processing flow around the renewal interval
and the broadcast calls in PendingFileImporter so rejected lease-renewal
promises propagate to the worker instead of remaining detached, and ensure
broadcast promises are awaited or explicitly caught before proceeding to DONE.
Preserve the existing lease-renewal cadence and cleanup behavior while making
failures observable to the processing flow.
In `@apps/meteor/app/importer-slack/server/SlackImporter.ts`:
- Around line 130-136: Update the pending-file flow around PendingFileImporter
and Import.newOperation so operation creation is single-flight: atomically find
or create one active pending-files operation for the relevant import backlog,
and reuse that operation when another Slack import is already processing it.
Ensure concurrent callers cannot both create operations, while preserving the
existing zero-backlog early return and prepareFileCount behavior.
In `@apps/meteor/app/importer/server/classes/Importer.ts`:
- Around line 229-234: Update the post-import handling in the Importer progress
flow around onImportComplete so a hook failure is persisted as a failed
follow-up operation or scheduled for retry instead of leaving the source import
at ProgressStep.DONE. Preserve the existing error logging while ensuring pending
files remain eligible for later processing.
---
Outside diff comments:
In `@apps/meteor/app/file-upload/server/lib/FileUpload.ts`:
- Around line 900-917: Update the catch block around the content write and
ufsComplete flow to explicitly remove tmpFile in addition to deleting the upload
record. Ensure file cleanup is attempted even when pipeline fails, and preserve
the original error while tolerating cleanup failures.
---
Nitpick comments:
In `@apps/meteor/app/importer-pending-files/server/PendingFileImporter.ts`:
- Line 78: Remove the inline comments near the fileStore filter bypass and
response-draining logic, including the additional comments around the referenced
later block. Replace any unclear inline behavior with descriptive helper names
that communicate the bypass and response-draining responsibilities.
In `@packages/models/src/models/Messages.ts`:
- Around line 1738-1744: Remove the explanatory implementation comment above
renewPendingFileImportLease in packages/models/src/models/Messages.ts at lines
1738-1744, preserving the method logic unchanged. Also remove the two-line
explanatory comment above the cleanup call in
apps/meteor/app/file-upload/server/lib/FileUpload.ts at lines 912-916; no other
code changes are needed.
- Around line 1721-1753: Define a shared pending-file eligibility predicate for
the repeated _importFile filters and reuse it in
findOneAndClaimPendingFileImport and
countAllImportedMessagesWithFilesToDownload, preserving each method’s additional
conditions. Add a supporting index for these fields in modelIndexes(), including
the eligibility fields and an ordering suitable for the lockedUntil lease query.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 41cf1910-7c53-40ea-90a4-ee944eb28e61
📒 Files selected for processing (10)
.changeset/deep-ways-poke.mdapps/meteor/app/file-upload/server/lib/FileUpload.tsapps/meteor/app/importer-pending-files/server/PendingFileImporter.tsapps/meteor/app/importer-slack/server/SlackImporter.tsapps/meteor/app/importer/server/classes/Importer.tsapps/meteor/app/importer/server/classes/converters/MessageConverter.tspackages/core-typings/src/IMessage/IMessage.tspackages/core-typings/src/import/IImportMessage.tspackages/model-typings/src/models/IMessagesModel.tspackages/models/src/models/Messages.ts
📜 Review details
⏰ Context from checks skipped due to timeout. (12)
- GitHub Check: 🔨 Test UI (EE) / MongoDB 8.0 coverage (4/5)
- GitHub Check: 🔨 Test UI (EE) / MongoDB 8.0 coverage (3/5)
- GitHub Check: 🔨 Test UI (EE) / MongoDB 8.0 coverage (5/5)
- GitHub Check: 🔨 Test UI (EE) / MongoDB 8.0 coverage (2/5)
- GitHub Check: 🔨 Test UI (EE) / MongoDB 8.0 coverage (1/5)
- GitHub Check: 🔨 Test UI (CE) / MongoDB 8.0 (1/4)
- GitHub Check: 🔨 Test UI (CE) / MongoDB 8.0 (4/4)
- GitHub Check: 🔨 Test API (EE) / MongoDB 8.0 coverage (1/1)
- GitHub Check: 🔨 Test UI (CE) / MongoDB 8.0 (2/4)
- GitHub Check: 🔨 Test UI (CE) / MongoDB 8.0 (3/4)
- GitHub Check: 🔨 Test API Livechat (EE) / MongoDB 8.0 coverage (1/1)
- GitHub Check: 🔨 Test Federation Matrix
🧰 Additional context used
📓 Path-based instructions (1)
**/*.{ts,tsx,js}
📄 CodeRabbit inference engine (.cursor/rules/playwright.mdc)
**/*.{ts,tsx,js}: Write concise, technical TypeScript/JavaScript with accurate typing in Playwright tests
Avoid code comments in the implementation
Files:
packages/core-typings/src/import/IImportMessage.tsapps/meteor/app/importer/server/classes/Importer.tspackages/core-typings/src/IMessage/IMessage.tspackages/model-typings/src/models/IMessagesModel.tsapps/meteor/app/importer/server/classes/converters/MessageConverter.tsapps/meteor/app/file-upload/server/lib/FileUpload.tsapps/meteor/app/importer-slack/server/SlackImporter.tsapps/meteor/app/importer-pending-files/server/PendingFileImporter.tspackages/models/src/models/Messages.ts
🧠 Learnings (4)
📚 Learning: 2026-03-16T21:50:37.589Z
Learnt from: amitb0ra
Repo: RocketChat/Rocket.Chat PR: 39676
File: .changeset/migrate-users-register-openapi.md:3-3
Timestamp: 2026-03-16T21:50:37.589Z
Learning: For changes related to OpenAPI migrations in Rocket.Chat/OpenAPI, when removing endpoint types and validators from rocket.chat/rest-typings (e.g., UserRegisterParamsPOST, /v1/users.register) document this as a minor changeset (not breaking) per RocketChat/Rocket.Chat-Open-API#150 Rule 7. Note that the endpoint type is re-exposed via a module augmentation .d.ts in the consuming package (e.g., packages/web-ui-registration/src/users-register.d.ts). In reviews, ensure the changeset clearly states: this is a non-breaking change, the major version should not be bumped, and the changeset reflects a minor version bump. Do not treat this as a breaking change during OpenAPI migrations.
Applied to files:
.changeset/deep-ways-poke.md
📚 Learning: 2026-02-26T19:25:44.063Z
Learnt from: gabriellsh
Repo: RocketChat/Rocket.Chat PR: 38778
File: packages/ui-voip/src/providers/useMediaSession.ts:192-192
Timestamp: 2026-02-26T19:25:44.063Z
Learning: In the Rocket.Chat repository, do not reference Biome lint rules in code review feedback. Biome is not used even if biome.json exists; only reference Biome rules if there is explicit, project-wide usage documented. For TypeScript files, review lint implications without Biome guidance unless the project enables Biome rules.
Applied to files:
packages/core-typings/src/import/IImportMessage.tsapps/meteor/app/importer/server/classes/Importer.tspackages/core-typings/src/IMessage/IMessage.tspackages/model-typings/src/models/IMessagesModel.tsapps/meteor/app/importer/server/classes/converters/MessageConverter.tsapps/meteor/app/file-upload/server/lib/FileUpload.tsapps/meteor/app/importer-slack/server/SlackImporter.tsapps/meteor/app/importer-pending-files/server/PendingFileImporter.tspackages/models/src/models/Messages.ts
📚 Learning: 2026-02-26T19:25:44.063Z
Learnt from: gabriellsh
Repo: RocketChat/Rocket.Chat PR: 38778
File: packages/ui-voip/src/providers/useMediaSession.ts:192-192
Timestamp: 2026-02-26T19:25:44.063Z
Learning: In this repository (RocketChat/Rocket.Chat), Biome lint rules are not used even if a biome.json exists. When reviewing TypeScript files (e.g., packages/ui-voip/src/providers/useMediaSession.ts), ensure lint suggestions do not reference Biome-specific rules. Rely on general ESLint/TypeScript lint rules and project conventions instead.
Applied to files:
packages/core-typings/src/import/IImportMessage.tsapps/meteor/app/importer/server/classes/Importer.tspackages/core-typings/src/IMessage/IMessage.tspackages/model-typings/src/models/IMessagesModel.tsapps/meteor/app/importer/server/classes/converters/MessageConverter.tsapps/meteor/app/file-upload/server/lib/FileUpload.tsapps/meteor/app/importer-slack/server/SlackImporter.tsapps/meteor/app/importer-pending-files/server/PendingFileImporter.tspackages/models/src/models/Messages.ts
📚 Learning: 2026-05-06T12:21:44.083Z
Learnt from: juliajforesti
Repo: RocketChat/Rocket.Chat PR: 40256
File: apps/meteor/client/components/CreateDiscussion/CreateDiscussion.tsx:121-149
Timestamp: 2026-05-06T12:21:44.083Z
Learning: Field wrappers in rocket.chat/fuselage-forms (Field, FieldLabel, FieldRow, FieldError, FieldHint) auto-create htmlFor/id associations, aria-describedby, and role="alert" for errors. Do not manually set htmlFor, id, aria-describedby, or role attributes when using these wrappers. This automatic wiring does not apply to plain rocket.chat/fuselage components, which require explicit ID wiring per the accessibility docs. In code reviews, prefer using fuselage-forms wrappers for form fields and verify there is no unnecessary manual ID/aria wiring in files that use these wrappers. If a component uses plain fuselage components, ensure proper id wiring as per docs.
Applied to files:
packages/core-typings/src/import/IImportMessage.tsapps/meteor/app/importer/server/classes/Importer.tspackages/core-typings/src/IMessage/IMessage.tspackages/model-typings/src/models/IMessagesModel.tsapps/meteor/app/importer/server/classes/converters/MessageConverter.tsapps/meteor/app/file-upload/server/lib/FileUpload.tsapps/meteor/app/importer-slack/server/SlackImporter.tsapps/meteor/app/importer-pending-files/server/PendingFileImporter.tspackages/models/src/models/Messages.ts
🔇 Additional comments (9)
packages/core-typings/src/IMessage/IMessage.ts (1)
395-408: LGTM!.changeset/deep-ways-poke.md (1)
1-8: LGTM!packages/model-typings/src/models/IMessagesModel.ts (1)
284-285: 🎯 Functional CorrectnessRemove this comment — there are no remaining callers of
findAllImportedMessagesWithFilesToDownload.> Likely an incorrect or invalid review comment.packages/models/src/models/Messages.ts (1)
1721-1736: 🩺 Stability & AvailabilityClaim is already deduped by file id
PendingFileImporter.tskeeps aninFlightFileIdsset and skips sibling messages with the same_importFile.id, so this_id-scoped claim does not create duplicate downloads or orphaned uploads in this flow.> Likely an incorrect or invalid review comment.packages/core-typings/src/import/IImportMessage.ts (1)
51-51: LGTM!apps/meteor/app/importer-slack/server/SlackImporter.ts (1)
1-8: LGTM!Also applies to: 516-518, 589-591
apps/meteor/app/importer/server/classes/converters/MessageConverter.ts (1)
107-107: LGTM!apps/meteor/app/importer/server/classes/Importer.ts (1)
279-280: LGTM!apps/meteor/app/importer-pending-files/server/PendingFileImporter.ts (1)
3-15: LGTM!Also applies to: 37-42, 100-118, 139-150
1 participant
Proposed changes (including videos or screenshots)
Follow-up to #41285. That PR fixed Slack files being imported as raw URLs instead of attachments; this one makes the download step itself automatic and resilient.
_importFile.lockedUntil), following the same claim pattern as LivechatInquiry. Concurrent runs (multiple instances, or a re-run after a crash) can't download the same file twice, and files stuck in a crashed run become claimable again as soon as their lock expires.FileUpload._doInsertnow usesstream.pipeline, so a broken input stream rejects the insert instead of hanging it forever — and it removes the upload record and temp file it created when the insert fails, instead of leaving orphans behind.Issue(s)
CORE-2044 — follow-up to #41285
Steps to test or reproduce
db.rocketchat_message.countDocuments({ '_importFile.lockedUntil': { $exists: true } })Further comments
The lock is the only new persisted state (a single field). There is deliberately no per-file retry bookkeeping: Slack export links stay valid until the token is revoked on Slack's admin page, so failed files are simply left pending and any later run resumes them.
Summary by CodeRabbit
New Features
Bug Fixes