Skip to content

FEAT: Improve interactions of user pools - #1009

Open
ColmTalbot wants to merge 1 commit into
bilby-dev:mainfrom
ColmTalbot:user-pools
Open

FEAT: Improve interactions of user pools#1009
ColmTalbot wants to merge 1 commit into
bilby-dev:mainfrom
ColmTalbot:user-pools

Conversation

@ColmTalbot

@ColmTalbot ColmTalbot commented Oct 30, 2025

Copy link
Copy Markdown
Collaborator

I noticed that it was difficult to pass a user-specified pool through run_sampler, and it isn't used at all in the post processing. This PR:

  • unifies how to create a pool for Bilby with a context (a la with multiprocessing.pool()...)
  • explicitly passes the user pool around including for post processing
  • moves logic out of Sampler._setup_pool and Sampler._close_pool
  • support MPI pools using schwimmbad. This change will effectively make parallel_bilby moot.

@ColmTalbot
ColmTalbot marked this pull request as draft October 30, 2025 14:22
@ColmTalbot
ColmTalbot marked this pull request as ready for review November 3, 2025 16:03
@ColmTalbot ColmTalbot added >100 lines refactoring sampling Issues about sampling algorithms, their efficiency and robustness labels Nov 4, 2025
mj-will
mj-will reviewed Dec 11, 2025
View reviewed changes

@mj-will mj-will left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, I look forward to trying it! I've added some initial comments but might need to have a second look.

Comment thread bilby/core/sampler/__init__.py
Comment thread bilby/core/sampler/__init__.py
Comment thread bilby/core/sampler/ptemcee.py Outdated
Comment thread bilby/core/utils/parallel.py
Comment thread bilby/core/utils/parallel.py
Comment thread bilby/core/result.py Outdated
Comment thread bilby/gw/conversion.py Outdated
Comment thread bilby/gw/conversion.py Outdated
@mj-will mj-will mentioned this pull request Jan 8, 2026
Closed
ColmTalbot
ColmTalbot commented Jan 23, 2026
View reviewed changes

@ColmTalbot ColmTalbot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't remember exactly what the behaviour for some of these questions is, so I'll go back and check, and ideally write docstrings about them.

Comment thread bilby/core/sampler/ptemcee.py Outdated
Comment thread bilby/core/sampler/__init__.py
Comment thread bilby/core/sampler/__init__.py
@ColmTalbot
ColmTalbot requested a review from mj-will January 26, 2026 16:30
@mj-will mj-will mentioned this pull request Jan 29, 2026
Draft
@ColmTalbot ColmTalbot added this to the 3.0.0 milestone Jan 29, 2026
@mj-will
mj-will requested a review from a team April 30, 2026 14:43
mj-will
mj-will requested changes Apr 30, 2026
View reviewed changes

@mj-will mj-will left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot to submit my comments as a review, but see them above.

@mj-will
mj-will requested a review from a team April 30, 2026 14:48
mj-will
mj-will reviewed Apr 30, 2026
View reviewed changes
Comment thread bilby/core/utils/parallel.py Outdated
@mj-will
mj-will requested a review from a team April 30, 2026 16:03
@mj-will mj-will added the to discuss To be discussed on an upcoming call label Apr 30, 2026
adivijaykumar
adivijaykumar reviewed May 14, 2026
View reviewed changes

@adivijaykumar adivijaykumar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a test for schwimmbad/MPI support?

@mj-will

mj-will commented May 14, 2026

Copy link
Copy Markdown
Collaborator

We should documentation for user pools and note that the pool will only work correctly in the initializer has been called.

ColmTalbot
ColmTalbot commented Aug 10, 2026
View reviewed changes
Comment thread bilby/core/utils/parallel.py Outdated
Comment thread bilby/core/sampler/__init__.py
mj-will
mj-will requested changes Aug 25, 2026
View reviewed changes

@mj-will mj-will left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coming back to this again, I think the only pending item is documentation for the new pool support.

@ColmTalbot

ColmTalbot commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator Author

Can we add a test for schwimmbad/MPI support?

I haven't added an MPI test, but we have a schwimmbad test using the SerialPool, which should have the same API.

@ColmTalbot
FEAT: improve user pool passing
d9fe332 
FEAT: improve reweighting parallelisation

FEAT: add parameters as argument to new pool

BUG: test that pool exists at cleanup

BUG: test pool exists at closing

REFACTOR: refactor run_sampler to simplify pool logic

DEP: discourage setting up pool in sampler

REFACTOR: remove top level multiprocessing import

BUG: make sure prior is passed to pool creation

BUG: fix test failures

TEST: fix reproducibility test

BUG: fix a typo in conversion function test

MAINT: don't create pool of size 1

BUG: only include chunksize in multiprocessing map

DOC: add docstrings for pool functions

DOC: update pool docstrings

Address review comments

TYPO: Fix typo in parameter description comments

BUG: move definition of chunk size in reweighting

MAINT: remove old function

Fix wrong function name in docstring

REFACTOR: refactor pool initialization and add documentation

FMT: precommit fixes

TST: fix failing tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adivijaykumar adivijaykumar Awaiting requested review from adivijaykumar
@mj-will mj-will Awaiting requested review from mj-will

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

refactoring sampling Issues about sampling algorithms, their efficiency and robustness to discuss To be discussed on an upcoming call waiting on review >100 lines

Projects

None yet

Milestone

3.0.0

Development

Successfully merging this pull request may close these issues.

3 participants

@ColmTalbot @mj-will @adivijaykumar