Skip to content

Drop Python 3.9, use 3.10 for dist, bump Ledger - #819

Open
Sjors wants to merge 11 commits into
bitcoin-core:masterfrom
Sjors:2026/01/python-bump
Open

Drop Python 3.9, use 3.10 for dist, bump Ledger#819
Sjors wants to merge 11 commits into
bitcoin-core:masterfrom
Sjors:2026/01/python-bump

Conversation

@Sjors

@Sjors Sjors commented Jan 21, 2026

Copy link
Copy Markdown
Member

This PR first switches the deterministic builds to Python 3.10. It then drops the end-of-life 3.9.

Then it bumps Speculos and Ledger and enables the tests that were previously disabled.

There's also a few minor cleanup commits, plus changes to make the Docker containers Podman friendly so I can easily run them locally.

@Sjors
Sjors force-pushed the 2026/01/python-bump branch 2 times, most recently from 8489c74 to 57f6690 Compare January 23, 2026 09:21
@Sjors
Sjors force-pushed the 2026/01/python-bump branch from 57f6690 to a608ea3 Compare January 30, 2026 10:00
@Sjors
Sjors marked this pull request as ready for review January 30, 2026 10:01
@Sjors

Sjors commented Jan 30, 2026

Copy link
Copy Markdown
Member Author

Rebased after #818 landed, ready for review.

I lightly tested the dist builds on Ubuntu (x86_64, including qt) and macOS (only command line).

I ran the GUI in Wine, but that doesn't have USB access so not very informative. I also ran it on a native Windows machine. Windows Defender wasn't happy about it, but I didn't do any code signing. There it detected a connected device, so that's good.

I wonder if it makes sense for the deterministic binary releases to jump straight ahead to a more modern Python?

Also, would it make sense to do one final HWI release with Python 3.9 before merging this?

@Sjors

Sjors commented Jan 30, 2026

Copy link
Copy Markdown
Member Author

Added commit to re-enable the Ledger tests.

@Sjors
Sjors force-pushed the 2026/01/python-bump branch from a4e3a84 to ae9e49d Compare January 31, 2026 09:04
@Sjors

Sjors commented Jan 31, 2026

Copy link
Copy Markdown
Member Author

Rebased just in case after recent merges.

@achow101

achow101 commented Feb 3, 2026

Copy link
Copy Markdown
Member

Does the dist build work on in an arm docker container, as described in the release process?

@Sjors
Sjors force-pushed the 2026/01/python-bump branch from ae9e49d to f473a2e Compare February 3, 2026 11:20
@Sjors

Sjors commented Feb 3, 2026

Copy link
Copy Markdown
Member Author

Does the dist build work on in an arm docker container

I built inside an arm64 docker container on the x86 linux machine. I also made it do an hwi --help.

Rebased again, this time using generate_setup.py in the relevant commits, so e.g. the description field is updated when I touch the README.

The first time I ran that command, it moved hwilib.ui from packages to ui/* in package_data, see 35aa66a. No idea if that's correct.

I also built and tested the x86 macOS build on a native machine. I was able to run the QT application and it connected to a test deviced and fetch the descriptors.

Checksums:

19ee9fa6181421de1d6b404145438a1d00c61b3f0e55cc19a344f54fb5e76289  hwi-3.2.0-rc.1-linux-aarch64.tar.gz
3987142f1a441f893c28372b7ccd6a143b06c324cc9ee50f86b9cdb4d21f3204  hwi-3.2.0-rc.1-linux-aarch64.tar.gz/hwi
9909786e92a9cf0079d4aba986a6f8b856abba49c52a6df9e57dcae4c2e3c3a7  hwi-3.2.0-rc.1-linux-x86_64.tar.gz
64b5568167acd0d619f34c1739c430491b214bbf326bd8a4a2e0528c6b58d040  hwi-3.2.0-rc.1-linux-x86_64.tar.gz/hwi
97554105f474ba60b46d3b26534d254318114c05f5193c53e79bdf7167836239  hwi-3.2.0-rc.1-linux-x86_64.tar.gz/hwi-qt
83ac0f79ec5d9ecfb6aca98d325aaf05e47b3ba31b5c1e4fa69a2bfb088719d4  hwi-3.2.0-rc.1-mac-arm64.tar.gz
f9e884b02c2cd37e43f0668d32afef43cda649262d8ec891302a6d63f95026ed  hwi-3.2.0-rc.1-mac-arm64.tar.gz/hwi
6c070537a9f547c5b1e63a0c6d42aff9658a4e9305fa501c9e4ee752b04cdf3f  hwi-3.2.0-rc.1-mac-x86_64.tar.gz
241dea62a11ae2e18a547d8dd496e30cff7e005256ef460b6fde77aa303c6f40  hwi-3.2.0-rc.1-mac-x86_64.tar.gz/hwi
eea44149abb0a800c18f3895ea7b4f92937d01378bbfaa68ae74f89b4ef37aa3  hwi-3.2.0-rc.1-mac-x86_64.tar.gz/hwi-qt
be2d6ddb0fc18be6d9525c5004bc899e487e20ca51bd20aaa3e8e3ff43b28d0a  hwi-3.2.0-rc.1-windows-x86_64.zip
bf9e8a41e84537e1e3fa2310e8453a13cce50ee5befd8d6767e546bc29c17e2b  hwi-3.2.0-rc.1-windows-x86_64.zip/hwi-qt.exe
04b63d2a0b1e73253fe167aa7d9fe10e4da05f6c416319846b927deb00b000b1  hwi-3.2.0-rc.1-windows-x86_64.zip/hwi.exe
902438e1e1e988ab99099895f515ef701fc813fa42ad1620a733a6745f25933c  hwi-3.2.0rc1-py3-none-any.whl
f01ac678aa1c4da6e84eb83d42044a559acd0177313dd77b49a9494f9db9b7d6  hwi-3.2.0rc1.tar.gz

@Sjors

Sjors commented Feb 3, 2026

Copy link
Copy Markdown
Member Author

The spurious Ledger failure appears to be the result of the test suite occasionally hitting the Cancel / Reject button.

2026-02-03T12:43:44.9074661Z seproxyhal: received (tag: 0x60, size: 0x02): b'\x00\x00'
2026-02-03T12:43:44.9074759Z automation: getting actions for "" (0, 0)
2026-02-03T12:43:44.9074882Z automation: getting actions for "Reject" (46, 36)
2026-02-03T12:43:44.9075003Z seproxyhal: applying automation ['button', 1, True]
2026-02-03T12:43:44.9075128Z seproxyhal: applying automation ['button', 1, False]
2026-02-03T12:43:44.9075220Z seproxyhal.packet: send 05000100
...
2026-02-03T12:43:44.9077945Z apdu: < 6985

Where 0x6985: DenyError according to device_exception.py

Added a commit to drop that rule, since it's unused anyway.

@Sjors

Sjors commented Jul 3, 2026

Copy link
Copy Markdown
Member Author

Rebased just in case.

  • bumped the Python patch version to 3.10.20.
  • bumped Speculos to v0.26.9
  • bumped ledger-app-builder to 5.3.2
  • bumped Ledger app to 2.4.6
    • plus two fixes from the develop branch
    • some test automation changes
    • the new version restricts xpub derivations, which test: make device signing cases more granular compensates for. This only impacts getkeypool.

I briefly tested the x86 linux gui.

@Sjors

Sjors commented Jul 3, 2026

Copy link
Copy Markdown
Member Author

Added a commit here to fix Trezor CI. Unrelated so can be its own PR if it works.

@Sjors
Sjors force-pushed the 2026/01/python-bump branch from c85d39c to a8bd342 Compare July 3, 2026 15:57
@Sjors

Sjors commented Jul 3, 2026

Copy link
Copy Markdown
Member Author

The one bitbox01 and one ledger failures seem spurious. But the jade test failures were real, so I'm pushing a new fix shortly (basically #832).

@Sjors
Sjors force-pushed the 2026/01/python-bump branch from d2093d9 to d21b545 Compare July 3, 2026 17:03
@Sjors Sjors mentioned this pull request Jul 3, 2026
@Sjors

Sjors commented Jul 3, 2026

Copy link
Copy Markdown
Member Author

Actually #832 did not do the trick. This time I ran the simulator locally to reproduce the issue. It was the PSBTv2 change on Bitcoin Core's master (again, this change bit me multiple times). I dropped the cbor commits, because they're not needed here, but based on local testing they should not be a problem.

I opened a fresh PR for just the CI fixes: #836

@Sjors
Sjors force-pushed the 2026/01/python-bump branch from d21b545 to 712823a Compare July 3, 2026 19:48
This was referenced Jul 3, 2026

@andreasgriffin andreasgriffin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and I let codex also review with no issues found.

@Sjors

Sjors commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

The one failing ledger just is spurious.

@andreasgriffin much of this was written with the help of Claude and Codex, so I wouldn't expect it to find anything. Unless you give it a more specific prompt based on your own review, of something that might be incorrect.

In general for this Python bump it would be great if someone can test the binaries on every supported platform. I did that before the rebase, so I don't expect anything new to have broken, but still.

@Sjors
Sjors force-pushed the 2026/01/python-bump branch from 712823a to 887843c Compare July 4, 2026 10:33
@Sjors

Sjors commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

Fable discovered that the one flaky failure here was due to a base58 address ending in To, which ended up at the start of the screen when the address was split. This collides with the automation rule "To".

It's a pre-existing bug, but I kept the fix be87c6e here, because it only impacts tests that were disabled before this PR.

@andreasgriffin

andreasgriffin commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

In general for this Python bump it would be great if someone can test the binaries on every supported platform. I did that before the rebase, so I don't expect anything new to have broken, but still.

Tested (and they work) https://github.com/bitcoin-core/HWI/actions/runs/28703459171/artifacts/8080924437 (from tests) wheel file on

  • Linux x86_64 Python 3.12.1
  • Mac arm64 Python 3.12.3
  • Win x86_64 Python 3.12.3

achow101 added a commit that referenced this pull request Jul 31, 2026
d3e4ce7 test: only match Ledger "To" screen on the title row (Sjors Provoost)
ca2caf0 psbt: don't overwrite PSBTv2 tx version and fallback locktime (Sjors Provoost)
f15d6d9 Fix repo renamings for Ledger apps (Salvatore Ingala)
7ac09cf ci: drop unused Python 3.7 Docker file (Sjors Provoost)
9ded73e ci: drop cirrus leftovers (Sjors Provoost)
eb36bd6 test: pin Trezor T Rust nightly to 2025-04-15 (Sjors Provoost)

Pull request description:

  Commits are taken from #819 (which is now based on this).

  - Trezor builds started to fail when Rust nightly introduced some new rule, so we pin the nightly
  - bitcoin/bitcoin#21283 broke the Jade tests (and probably more than that)

  Two minor commits drop unused CI code.

  Ledger moved repos around, so also include:

  - #837

ACKs for top commit:
  achow101:
    ACK d3e4ce7

Tree-SHA512: d9b9bc47a9f1d72bb13166edc382f545eefcac19472883e929d23dc879c7dd497d85b2719329095f0a121337912248f3956ec280b3b49edcf0499e809b09b0f5
@Sjors
Sjors force-pushed the 2026/01/python-bump branch from 887843c to b43b016 Compare July 31, 2026 07:56
@Sjors

Sjors commented Jul 31, 2026

Copy link
Copy Markdown
Member Author

Rebased after #836.

@achow101

Copy link
Copy Markdown
Member

c0990a1 is incorrect and should be dropped. It excludes the ui files necessary for hwi-qt in the python distributions to work.

@achow101

achow101 commented Jul 31, 2026

Copy link
Copy Markdown
Member

Edit: This was a local docker issue apparently.

Building the arm container fails:

$ docker buildx build --no-cache --platform linux/arm64 -t hwi-builder-arm64 -f contrib/build.Dockerfile .
[+] Building 284.8s (10/10) FINISHED                                                                                                                           docker:default
 => [internal] load build definition from build.Dockerfile                                                                                                               0.0s
 => => transferring dockerfile: 1.03kB                                                                                                                                   0.0s
 => [internal] load metadata for docker.io/library/debian:bookworm-slim                                                                                                  0.6s
 => [internal] load .dockerignore                                                                                                                                        0.0s
 => => transferring context: 259B                                                                                                                                        0.0s
 => CACHED [1/6] FROM docker.io/library/debian:bookworm-slim@sha256:7b140f374b289a7c2befc338f42ebe6441b7ea838a042bbd5acbfca6ec875818                                     0.0s
 => [internal] load build context                                                                                                                                        0.0s
 => => transferring context: 81B                                                                                                                                         0.0s
 => [2/6] RUN apt-get update                                                                                                                                             7.9s
 => [3/6] RUN apt-get install -y     apt-transport-https     git     make     build-essential     libssl-dev     zlib1g-dev     libbz2-dev     libreadline-dev     li  161.5s
 => [4/6] RUN curl https://pyenv.run | bash                                                                                                                              7.8s 
 => [5/6] COPY contrib/reproducible-python.diff /opt/reproducible-python.diff                                                                                            0.0s 
 => ERROR [6/6] RUN /bin/bash -c 'eval "$(pyenv init --path)" && eval "$(pyenv virtualenv-init -)" && cat /opt/reproducible-python.diff | pyenv install -kp 3.10.20'   107.0s 
------                                                                                                                                                                        
 > [6/6] RUN /bin/bash -c 'eval "$(pyenv init --path)" && eval "$(pyenv virtualenv-init -)" && cat /opt/reproducible-python.diff | pyenv install -kp 3.10.20':                
4.715 Downloading Python-3.10.20.tar.xz...                                                                                                                                    
4.716 -> https://www.python.org/ftp/python/3.10.20/Python-3.10.20.tar.xz                                                                                                      
10.71 Installing Python-3.10.20...                                                                                                                                            
10.82 patching file Makefile.pre.in
10.82 Hunk #1 succeeded at 787 (offset 46 lines).
106.7 
106.8 BUILD FAILED (Debian GNU/Linux 12 using python-build 2.8.1)
106.8 
106.9 Inspect or clean up the working tree at /root/.pyenv/sources/3.10.20
106.9 Results logged to /tmp/python-build.20260731194350.430.log
106.9 
106.9 Last 10 log lines:
106.9 checking size of double... 0
106.9 checking size of fpos_t... 0
106.9 checking size of size_t... 0
106.9 checking alignment of size_t... 0
106.9 checking size of pid_t... 0
106.9 checking size of uintptr_t... 0
106.9 checking for long double... yes
106.9 checking size of long double... configure: error: in `/root/.pyenv/sources/3.10.20/Python-3.10.20':
106.9 configure: error: cannot compute sizeof (long double)
106.9 See `config.log' for more details
------
build.Dockerfile:39
--------------------
  37 |     ENV BUILD_DATE="Jan  1 2026"
  38 |     ENV BUILD_TIME="00:00:00"
  39 | >>> RUN /bin/bash -c 'eval "$(pyenv init --path)" && eval "$(pyenv virtualenv-init -)" && cat /opt/reproducible-python.diff | pyenv install -kp 3.10.20'
  40 |     
  41 |     ENV LC_ALL=C.UTF-8
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c /bin/bash -c 'eval \"$(pyenv init --path)\" && eval \"$(pyenv virtualenv-init -)\" && cat /opt/reproducible-python.diff | pyenv install -kp 3.10.20'" did not complete successfully: exit code: 1

@Sjors

Sjors commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

@achow101 Codex says it's your fault :-)

I can reproduce a successful no-cache ARM64 build both natively [on macOS] and under QEMU on x86_64 [Ubuntu]. The resulting image contains runnable aarch64 Python 3.10.20. The zero-valued configure probes suggest a local QEMU/binfmt issue; can you check docker buildx inspect --bootstrap and that /proc/sys/fs/binfmt_misc/qemu-aarch64 includes the F flag?

@Sjors
Sjors force-pushed the 2026/01/python-bump branch from 831193d to 76bd097 Compare August 1, 2026 19:18
@Sjors

Sjors commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

c0990a1 is incorrect and should be dropped.

Done, but added a184337 build: generate UI before building distributions, because it seems the original commit stemmed from confusion around the sequence of build scripts.

@achow101

achow101 commented Aug 1, 2026

Copy link
Copy Markdown
Member

Codex says it's your fault :-)

Well, since I do the releases, if it doesn't work on my machine, it ain't gonna be merged.

@Sjors

Sjors commented Aug 2, 2026

Copy link
Copy Markdown
Member Author

@achow101 did the suggestion work? I can't reproduce the issue, so will need more details.

@achow101

achow101 commented Aug 4, 2026

Copy link
Copy Markdown
Member

The current error is

 > [6/6] RUN /bin/bash -c 'eval "$(pyenv init --path)" && eval "$(pyenv virtualenv-init -)" && cat /opt/reproducible-python.diff | pyenv install -kp 3.10.20':                
4.843 Downloading Python-3.10.20.tar.xz...                                                                                                                                    
4.844 -> https://www.python.org/ftp/python/3.10.20/Python-3.10.20.tar.xz                                                                                                      
11.22 Installing Python-3.10.20...                                                                                                                                            
11.33 patching file Makefile.pre.in
11.33 Hunk #1 succeeded at 787 (offset 46 lines).
438.4 
438.4 BUILD FAILED (Debian GNU/Linux 12 using python-build 2.8.1)
438.5 
438.5 Inspect or clean up the working tree at /root/.pyenv/sources/3.10.20
438.5 Results logged to /tmp/python-build.20260804021345.430.log
438.5 
438.5 Last 10 log lines:
438.6 gcc -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include -I/root/.pyenv/versions/3.10.20/include -I/root/.pyenv/versions/3.10.20/include -fPIC -DPy_BUILD_CORE -o Python/ceval.o Python/ceval.c
438.6 gcc -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include -I/root/.pyenv/versions/3.10.20/include -I/root/.pyenv/versions/3.10.20/include -fPIC -DPy_BUILD_CORE -o Python/codecs.o Python/codecs.c
438.6 gcc -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include -I/root/.pyenv/versions/3.10.20/include -I/root/.pyenv/versions/3.10.20/include -fPIC -DPy_BUILD_CORE -o Python/compile.o Python/compile.c
438.6 gcc -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include -I/root/.pyenv/versions/3.10.20/include -I/root/.pyenv/versions/3.10.20/include -fPIC -DPy_BUILD_CORE -o Python/context.o Python/context.c
438.6 gcc -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include -I/root/.pyenv/versions/3.10.20/include -I/root/.pyenv/versions/3.10.20/include -fPIC -DPy_BUILD_CORE -o Python/dynamic_annotations.o Python/dynamic_annotations.c
438.6 make: *** [Makefile:1858: Objects/setobject.o] Segmentation fault (core dumped)
438.6 make: *** Waiting for unfinished jobs....
438.6 Objects/obmalloc.c:1413:1: warning: 'always_inline' function might not be inlinable [-Wattributes]
438.6  1413 | arena_map_get(block *p, int create)
438.6       | ^~~~~~~~~~~~~
------
build.Dockerfile:39
--------------------
  37 |     ENV BUILD_DATE="Jan  1 2026"
  38 |     ENV BUILD_TIME="00:00:00"
  39 | >>> RUN /bin/bash -c 'eval "$(pyenv init --path)" && eval "$(pyenv virtualenv-init -)" && cat /opt/reproducible-python.diff | pyenv install -kp 3.10.20'
  40 |     
  41 |     ENV LC_ALL=C.UTF-8
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c /bin/bash -c 'eval \"$(pyenv init --path)\" && eval \"$(pyenv virtualenv-init -)\" && cat /opt/reproducible-python.diff | pyenv install -kp 3.10.20'" did not complete successfully: exit code: 1

@achow101

achow101 commented Aug 4, 2026

Copy link
Copy Markdown
Member

Apparently I have to disable ASLR...

git clone --branch 2.4.6 --depth 1 https://github.com/LedgerHQ/app-bitcoin.git
cd app-bitcoin
make DEBUG=1 BOLOS_SDK=$NANOX_SDK
# Work around register-wallet policy parsing/display bugs fixed after v2.4.6.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In 50da3f1 "Bump Speculos and Ledger Bitcoin app"

Can we instead pin a commit that has these included instead of cherry picking them? We're already building from source anyways.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I bumped to 2.5.0, which has been released since and contains these fixes.

Comment thread test/setup_environment.sh Outdated
# Work around -Werror build failures in Speculos' bundled deps.
# GCC < 15 errors out on unknown "-Wno-error=..." options, so only add the
# unterminated-string-initialization suppression when the compiler supports it.
CFLAGS="-O -fno-builtin -fPIC -Wall -Wextra -Werror -Wno-error=maybe-uninitialized -Wno-error=array-parameter -Wno-error=array-bounds -Wno-error=discarded-qualifiers"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In 50da3f1 "Bump Speculos and Ledger Bitcoin app"

Where did these flags come from? I don't see them in the speculos codebase at all.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped. Not sure if this was useful in earlier versions (it's been there since at least January), but it's definitely not needed now.

Sjors added 11 commits August 4, 2026 10:36
Most device simulators accept arbitrary keypool paths, so keep testing
that behavior by default.

Some devices enforce their own derivation path policies. Let those
emulators opt out of the arbitrary-path portion while still running the
remaining keypool checks.
It's unused and occasionally trips up a test.
Limit the warning rule to the centered title so it does not also click the repeated header on the details screen. This prevents the queued extra click from selecting Back to safety before Continue anyway is confirmed.
@Sjors
Sjors force-pushed the 2026/01/python-bump branch from 8ae811e to ac68456 Compare August 4, 2026 08:49
@Sjors

Sjors commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

Rebased after #838 and addressed inline feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants