fix(python): make job schedulers advance and support the postgres backend - #4498
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes Python job scheduler advancement (repeatable job factories) and makes schedulers backend-agnostic so they work on both Redis and PostgreSQL, addressing #4483.
Changes:
- Advance job schedulers when workers pick up scheduler-produced jobs (
Worker.nextJobFromJobData), ensuring iterations continue beyond the first. - Route scheduler datastore operations through the
Backendabstraction and implement them in both Redis and Postgres backends. - Add regression tests for worker advancement and lazy
jobSchedulerinitialization; adjust CI postgres job to skip when Node/Lua are unchanged.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| python/tests/job_scheduler_test.py | Adds regression tests for worker-driven scheduler advancement and lazy jobScheduler creation. |
| python/bullmq/worker.py | Adds lazy jobScheduler on Worker and advances schedulers when consuming scheduled jobs. |
| python/bullmq/queue.py | Removes Redis-only guard so schedulers can be used with non-Redis backends. |
| python/bullmq/job_scheduler.py | Makes JobScheduler backend-agnostic by calling backend scheduler APIs instead of Redis scripts/client. |
| python/bullmq/backends/redis_backend.py | Implements scheduler operations on the Redis backend via Lua scripts + Redis data structures. |
| python/bullmq/backends/postgres_backend.py | Implements scheduler operations on the Postgres backend via SQL commands and row mapping. |
| python/bullmq/backend.py | Extends the backend interface with scheduler-related abstract methods. |
| .github/workflows/test.yml | Skips node/postgres test steps when Node/Lua inputs are unchanged. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
manast
force-pushed
the
fix/broken-python-job-schedulers
branch
from
August 4, 2026 21:19
5d1a90b to
780deb0
Compare
roggervalf
force-pushed
the
fix/broken-python-job-schedulers
branch
2 times, most recently
from
August 5, 2026 03:14
09c5a8b to
05af323
Compare
roggervalf
force-pushed
the
fix/broken-python-job-schedulers
branch
from
August 5, 2026 03:40
05af323 to
7301f10
Compare
roggervalf
approved these changes
Aug 5, 2026
Contributor
|
🎉 This PR is included in version 6.0.8 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Contributor
|
🎉 This PR is included in version 3.0.4 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Job schedulers were effectively broken in the Python package: the worker
never advanced a scheduler to its next iteration, so upsertJobScheduler
only ever materialized the first job and the schedule stopped. They were
also unavailable on the PostgreSQL backend.
worker): when a processed job carries a repeatJobKey, upsert its
scheduler for the next iteration.
abstraction (addJobScheduler, updateJobSchedulerNextMillis,
removeJobScheduler, isJobScheduler, getJobScheduler, getJobSchedulers,
getJobSchedulersCount) and implement them in both the Redis and
PostgreSQL adapters, so JobScheduler is backend-agnostic.
fixes #4483