Skip to content

release: orchestrate signed Python driver releases - #1175

Merged
dkropachev merged 3 commits into
masterfrom
release/manual-orchestration
Nov 5, 2026
Merged

dkropachev merged 3 commits into
masterfrom
release/manual-orchestration

Conversation

@dkropachev

@dkropachev dkropachev commented Nov 5, 2026 •

Copy link
Copy Markdown

Issue

Release tags currently upload to PyPI directly from the build workflow. This leaves metadata preparation, signing, integration validation and recovery outside a single approved release flow.

Fix

Add a manual release.yml flow modeled on the Java driver release process. It prepares metadata from a pinned master commit, exports the candidate as a Git bundle, and validates the existing wheel coverage plus a pinned python-driver-matrix runner across four Scylla aliases and protocols 3/4.

One approval-protected publish job signs and publishes the real release. It creates a release commit with exactly the tested tree, a signed annotated tag, distribution signatures and a checksum manifest. An immutable artifact checkpoint retains the original packages and signed history before the top-level PyPI OIDC job uploads missing files. Publication verifies PyPI hashes, atomically pushes the retained history as scylladb-publisher, completes an immutable GitHub release, and then calls docs deployment. Existing build workflows remain available with their PyPI upload removed.

Decisions

  • Release stable X.Y.Z versions from master, retaining scylla-driver and X.Y.Z-scylla tags. Keep the package at its released version afterward.
  • dry-run builds, tests, signs and verifies without publication or environment approval. Its master-only release-dry-run environment has signing secrets only; real releases retain approval. Full test coverage is preserved. skip-tests affects project tests only; helper tests always run and the external matrix has a separate skip input.
  • Require approval only for the real signing/publication job that uses release credentials. Preparation, builds, validation and downstream docs have no approval gate; dry-run signing has neither the publisher token nor OIDC permission.
  • Reruns reuse the original signed checkpoint, reject conflicting hashes/tags and stop on incompatible branch advancement. Recovery never rebases or re-signs published history. Maintenance branches, prereleases and historical republishing remain deferred.

Validation

  • Eleven primary-suite release tests passed with skips disabled and on CPython 3.10.
  • 69 release helper tests passed, including real disposable GPG keys, cleanup during cancellation, metadata and bundle identities, partial uploads, hash conflicts, publication ordering, checkpoint restoration when only the failed publication job is rerun, and dry-run/skip/approval dependency gates.
  • Prepared source built a source distribution and Linux CPython 3.13 wheel; both passed strict Twine checks and name/version validation.
  • New/changed workflows passed actionlint. The existing reusable builder also passed with its pre-existing shellcheck warnings excluded; Python static checks and git diff --check passed.
  • Full platform builds and the live Scylla matrix require the post-merge full dry run. No package, release tag or GitHub release was published during implementation.

Cutover

Already configured: release environment with scylladb/drivers approval, self-review allowed, administrator bypass disabled and master only; signing secrets and publisher credential loaded through stdin; public certificate registered on scylladb-publisher; repository release immutability enabled; release-dry-run configured without reviewers, restricted to master, containing only the two signing secrets.

Still required before the first real release:

  • An organization administrator must create the publisher-only bypass team and add its ruleset bypass. The available token lacks admin:org; publication checks this before any PyPI upload.
  • A PyPI project owner must register the publisher for scylladb/python-driver, release.yml, environment release, and retire the old upload-workflow registrations at cutover. No authenticated PyPI management connection was available.
  • After merge, run a full dry run on master with both test skips disabled. The first real release exercises PyPI OIDC.

The release runbook documents these steps and retained-state recovery. This changes release automation without changing the driver runtime or protocol behavior.

Pre-review checklist

  • I have split my patch into logically separate commits.
  • All commit messages clearly explain what they change and why.
  • I added relevant tests for new features and bug fixes.
  • All commits compile, pass static checks and pass test.
  • PR description sums up the changes and reasons why they should be introduced.
  • I have provided docstrings for the public items that I want to introduce.
  • I have adjusted the documentation in ./docs/source/.
  • I added appropriate Fixes: annotations to PR description.

@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Explain Change →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment thread scripts/release/signing.py Fixed
Comment thread scripts/release/signing.py Fixed
Comment thread scripts/release/test_signing.py Fixed
Comment thread scripts/release/test_signing.py Fixed
Comment thread scripts/release/signing.py Fixed
Pin stable release metadata and retain the tested source tree before
publication. Preserve signed history and distribution bytes so retries
can verify existing uploads and continue without moving tags or signing
published files again.

Add public-only signature verification, isolated signing-key cleanup and
helper coverage for failure, cancellation and partial-upload recovery.
Run portable metadata guards in the primary unit suite as well.
@dkropachev
dkropachev force-pushed the release/manual-orchestration branch from d6994c1 to fc52a4b Compare November 5, 2026 19:14
Replace tag-triggered uploads with a manual master-only release flow.
Validate the bundled candidate across the existing wheel targets and the
pinned integration matrix before signing and publication.

Run complete dry runs with signing secrets and no approval gate. Keep
real releases behind approval, publish through PyPI OIDC, and deploy docs
after the retained history and immutable GitHub release are verified.
@dkropachev
dkropachev force-pushed the release/manual-orchestration branch from fc52a4b to bc64542 Compare November 5, 2026 19:17
Run preparation, builds and validation before the protected environment.
Keep dry-run signing approval-free without publisher or OIDC credentials.

On failed-job reruns, recheck the current run signed checkpoint before
signing so saved history and signatures remain unchanged.
@codecov

codecov Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.39%. Comparing base (7b7feba) to head (2513665).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1175      +/-   ##
==========================================
+ Coverage   87.36%   87.39%   +0.02%     
==========================================
  Files          44       44              
  Lines       19272    19272              
  Branches     3015     3015              
==========================================
+ Hits        16837    16842       +5     
+ Misses       1783     1780       -3     
+ Partials      652      650       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dkropachev
dkropachev marked this pull request as ready for review November 5, 2026 20:50
@dkropachev
dkropachev merged commit 31e75b5 into master Nov 5, 2026
32 checks passed
@dkropachev
dkropachev deleted the release/manual-orchestration branch November 5, 2026 20:50
@qodo-scylladb

qodo-scylladb Bot commented Nov 5, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (3) 📘 Rule violations (1)



🔴 High

1. Release stops for write-only publishers 🐞 Bug
Description
prepublish() reads the repository's immutable-release setting using PUBLISHER_TOKEN, but that
endpoint requires administration read access. If the publisher receives only the documented
repository write access, preflight fails before the PyPI upload even after the bypass team is
configured.
Code

scripts/release/release.py[R793-795]

+        api("immutable-releases")["enabled"] is True,
+        "Enable repository release immutability before publication.",
+    )
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The immutable-release settings endpoint requires administration read access, while the documented publisher setup grants repository write access only. That configuration blocks real releases at preflight.
## Fix Focus Areas
- scripts/release/release.py[788-796]
- scripts/release/README.md[97-106]
- .github/workflows/release.yml[244-248]
## Recommended Fix
Ensure the credential used for this preflight has the required administration read access without broadening the publisher bypass team's membership; document and verify the required credential permissions during cutover.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗



🟠 Medium

2. Internal signing helper is exposed under a public name 📘 Rule violation
Description
protected_subkey is defined without an underscore prefix in scripts/release/signing.py, yet it
is called only from two places inside that same module (execute-path helpers at the two call sites
around lines 368 and 416) and is not referenced by release.py, the tests, or the release README.
Nothing in the PR documents it as public API, so a later reader may treat it as a supported entry
point and call it from outside the signing flow, where it skips the surrounding guards.
Code

scripts/release/signing.py[139]

+def protected_subkey(env):
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

Issue description
The module-level helper `protected_subkey` in `scripts/release/signing.py` is public by name but used only inside that module and is not documented as public API.

Fix Focus Areas
- scripts/release/signing.py[139-139]
- scripts/release/signing.py[361-361]
- scripts/release/signing.py[409-409]

Recommended Fix
Rename `protected_subkey` to `_protected_subkey` in its definition and both internal call sites. Confirm no test or external script imports it before renaming.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


View medium (2)
3. Extra branch rules escape release checks 🐞 Bug
Description
publisher_ready() checks only the first page returned by the branch-rules endpoint, whose default
page size is 30. If more than 30 rules apply to master, an unchecked pull-request rule can lack
the publisher bypass, allowing PyPI upload before the subsequent Git push fails.
Code

scripts/release/release.py[R768-769]

+    rules = api("rules/branches/master")
+    pr_rules = [rule for rule in rules if rule["type"] == "pull_request"]
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Publisher preflight inspects only the default first page of effective branch rules, so additional pull-request rules can go unchecked before PyPI publication.
## Fix Focus Areas
- scripts/release/release.py[768-785]
## Recommended Fix
Paginate the effective branch-rules endpoint through its final page, then require the publisher bypass on every returned pull-request rule before proceeding.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


4. Failed releases hide their cause 🐞 Bug
Description
main() catches unexpected exceptions and prints one fixed message without an exception type,
traceback, or failed operation. Network errors and malformed responses across preparation, signing,
and publication therefore leave the same output when an operator must diagnose or recover a run.
Code

scripts/release/release.py[R1015-1019]

+    except Exception:
+        print(
+            "Release operation failed; output withheld. "
+            "Inspect retained artifacts before "
+            "recovery.",
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Unexpected release failures are reduced to a generic message, leaving operators unable to distinguish network, parsing, and filesystem failures during recovery.
## Fix Focus Areas
- scripts/release/release.py[997-1022]
## Recommended Fix
Report the operation and a safe exception category or sanitized diagnostic for unexpected errors. Preserve the existing protection against logging credentials, command arguments, or untrusted response bodies.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗



Tip of the day
💡 Did you know, you can turn these tips off under Display > Comment parts

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗


Powered by Qodo

Comment on lines +793 to +795
api("immutable-releases")["enabled"] is True,
"Enable repository release immutability before publication.",
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 High

1. Release stops for write-only publishers 🐞 Bug ≡ Correctness

prepublish() reads the repository's immutable-release setting using PUBLISHER_TOKEN, but that
endpoint requires administration read access. If the publisher receives only the documented
repository write access, preflight fails before the PyPI upload even after the bypass team is
configured.
Agent Prompt
## Issue description
The immutable-release settings endpoint requires administration read access, while the documented publisher setup grants repository write access only. That configuration blocks real releases at preflight.
## Fix Focus Areas
- scripts/release/release.py[788-796]
- scripts/release/README.md[97-106]
- .github/workflows/release.yml[244-248]
## Recommended Fix
Ensure the credential used for this preflight has the required administration read access without broadening the publisher bypass team's membership; document and verify the required credential permissions during cutover.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant