Skip to content

fix(python): make job schedulers advance and support the postgres backend - #4498

Merged
roggervalf merged 1 commit into
masterfrom
fix/broken-python-job-schedulers
Aug 5, 2026
Merged

fix(python): make job schedulers advance and support the postgres backend#4498
roggervalf merged 1 commit into
masterfrom
fix/broken-python-job-schedulers

Conversation

@manast

@manast manast commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Job schedulers were effectively broken in the Python package: the worker
never advanced a scheduler to its next iteration, so upsertJobScheduler
only ever materialized the first job and the schedule stopped. They were
also unavailable on the PostgreSQL backend.

  • Advance the scheduler in Worker.nextJobFromJobData (mirrors the Node
    worker): when a processed job carries a repeatJobKey, upsert its
    scheduler for the next iteration.
  • Route all scheduler datastore operations through the Backend
    abstraction (addJobScheduler, updateJobSchedulerNextMillis,
    removeJobScheduler, isJobScheduler, getJobScheduler, getJobSchedulers,
    getJobSchedulersCount) and implement them in both the Redis and
    PostgreSQL adapters, so JobScheduler is backend-agnostic.
  • Drop the Redis-only NotImplementedError gate in Queue/Worker.jobScheduler.

fixes #4483

@manast
manast requested review from Copilot and roggervalf August 4, 2026 13:26
@manast manast changed the title fix(python): make job schedulers advance and support the postgres bac… fix(python): make job schedulers advance and support the postgres backend Aug 4, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes Python job scheduler advancement (repeatable job factories) and makes schedulers backend-agnostic so they work on both Redis and PostgreSQL, addressing #4483.

Changes:

  • Advance job schedulers when workers pick up scheduler-produced jobs (Worker.nextJobFromJobData), ensuring iterations continue beyond the first.
  • Route scheduler datastore operations through the Backend abstraction and implement them in both Redis and Postgres backends.
  • Add regression tests for worker advancement and lazy jobScheduler initialization; adjust CI postgres job to skip when Node/Lua are unchanged.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
python/tests/job_scheduler_test.py Adds regression tests for worker-driven scheduler advancement and lazy jobScheduler creation.
python/bullmq/worker.py Adds lazy jobScheduler on Worker and advances schedulers when consuming scheduled jobs.
python/bullmq/queue.py Removes Redis-only guard so schedulers can be used with non-Redis backends.
python/bullmq/job_scheduler.py Makes JobScheduler backend-agnostic by calling backend scheduler APIs instead of Redis scripts/client.
python/bullmq/backends/redis_backend.py Implements scheduler operations on the Redis backend via Lua scripts + Redis data structures.
python/bullmq/backends/postgres_backend.py Implements scheduler operations on the Postgres backend via SQL commands and row mapping.
python/bullmq/backend.py Extends the backend interface with scheduler-related abstract methods.
.github/workflows/test.yml Skips node/postgres test steps when Node/Lua inputs are unchanged.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread python/bullmq/worker.py
@manast
manast force-pushed the fix/broken-python-job-schedulers branch from 5d1a90b to 780deb0 Compare August 4, 2026 21:19
@roggervalf
roggervalf force-pushed the fix/broken-python-job-schedulers branch 2 times, most recently from 09c5a8b to 05af323 Compare August 5, 2026 03:14
@roggervalf
roggervalf force-pushed the fix/broken-python-job-schedulers branch from 05af323 to 7301f10 Compare August 5, 2026 03:40
@roggervalf
roggervalf merged commit 5eb9dca into master Aug 5, 2026
44 of 46 checks passed
@roggervalf
roggervalf deleted the fix/broken-python-job-schedulers branch August 5, 2026 04:12
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 6.0.8 🎉

The release is available on:

Your semantic-release bot 📦🚀

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 3.0.4 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Python: Job Schedulers are broken and unavailable

3 participants