Skip to content

Make lz4 optional on Python 3.14+ - #1139

Merged
joe-clickhouse merged 4 commits into
ClickHouse:mainfrom
29antonioac:optional-lz4-py314
Nov 2, 2026
Merged

joe-clickhouse merged 4 commits into
ClickHouse:mainfrom
29antonioac:optional-lz4-py314

Conversation

@29antonioac

@29antonioac 29antonioac commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Summary

On Python 3.14+, lz4 is optional. Install clickhouse-connect[lz4] to use it. Default installs below 3.14 are unchanged.

python-lz4 lacks wheels for Python 3.14+ free-threaded builds, CPython 3.15, and musllinux. Its latest release was November 2025. These platforms must build it from source, which requires a C compiler. Python 3.14 includes zstd, so default compression no longer needs a third-party package.

Without lz4:

  • compress=True still gets zstd responses because the server prefers zstd.
  • Inserts use zstd. Our benchmark found no measurable change in insert time and about 45% fewer request bytes.
  • compress="lz4" raises ProgrammingError with instructions to install the extra.

With lz4 installed, compression behaves as before.

Changes

  • Make lz4 imports optional and check availability before use.
  • Add the lz4 extra and remove the default dependency on Python 3.14+.
  • Update errors, docs, docstrings, and the changelog.
  • Add CI coverage for the full suite on Python 3.14 without lz4.

Import lz4 lazily via compression.py so the package works without it.
available_compression only lists lz4 when installed, so compress=True
falls through to zstd/gzip. Add a clickhouse-connect[lz4] extra.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@joe-clickhouse joe-clickhouse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks @29antonioac. I added clearer install instructions when lz4 is missing, updated the docs and changelog, simplified the extra, and added CI coverage without lz4. These changes make the optional dependency easier to use and keep that path tested. I also updated the PR description to accurately represent the final state of the PR.

@29antonioac

Copy link
Copy Markdown
Contributor Author

Thank you so much @joe-clickhouse! Really appreciate the support ❤️ It'll come handy for 3.14+ to avoid installing extra deps in Lambda 😊

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The 1.x bare-install compatibility change requiring an extra for Python 3.14+ clients requesting lz4 has no established explicit maintainer approval, as required by repository guidance.

No open issues 🎉

What changed in this PR

Makes lz4 optional on Python 3.14+, allowing default installations to use standard-library zstd without building lz4 from source. Default dependencies below Python 3.14 remain unchanged.

Changes:

  • Add the lz4 extra and guard compression imports with installation guidance when lz4 is requested but unavailable.
  • Update documentation and docstrings to explain the zstd insert fallback.
  • Add missing-lz4 tests and a Python 3.14 CI job, skipping lz4-specific tests when unavailable.

No findings remain. Tests, linting, and type checking were not run during this review.

File Description
tests/​unit_tests/​test_streaming_source.py Skip lz4 streaming tests when unavailable.
tests/​unit_tests/​test_driver/​test_httputil.py Guard the lz4 response test.
tests/​unit_tests/​test_driver/​test_compression.py Test missing-lz4 imports and errors.
tests/​integration_tests/​test_streaming.py Skip unavailable lz4 parameter cases.
setup.py Make lz4 optional on Python 3.14+ and add an extra.
docs/​index.mdx Document installation and wheel implications.
docs/​additional-options.mdx Explain compression fallback and optional lz4 support.
clickhouse_connect/​driver/​streaming.py Use the guarded lz4 decompressor helper.
clickhouse_connect/​driver/​httputil.py Remove mandatory lz4 imports from response handling.
clickhouse_connect/​driver/​compression.py Detect lz4 availability and guard codec creation.
clickhouse_connect/​driver/​_backend/​httpcommon.py Provide missing-lz4 guidance and guarded decompression.
clickhouse_connect/​driver/​_backend/​chdb_backend.py Guard lz4 decompression in chDB.
clickhouse_connect/​driver/​__init__.py Update sync and async compression docstrings.
CHANGELOG.md Record the dependency and fallback changes.
.github/​workflows/​on_push.yml Add Python 3.14 coverage without lz4.

Review effort: Balanced

@joe-clickhouse

Copy link
Copy Markdown
Contributor

The 1.x bare-install compatibility change requiring an extra for Python 3.14+ clients requesting lz4 has no established explicit maintainer approval, as required by repository guidance.

Approved as a deliberate 1.x change: lz4 becomes an extra on 3.14+ only, nothing changes below 3.14.

@joe-clickhouse
joe-clickhouse merged commit 5520216 into ClickHouse:main Nov 2, 2026
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants