[sebastianbreguel]

Summary

Reading an Array column with the Python codec ends in a Python loop that cuts the flat element list into one list per row:

for x in offset_range:
    data.append(column[last:x])
    last = x

On 1M rows of Array(UInt64) this loop was the top entry in cProfile for query, query_np and query_df, about 0.26 to 0.30 s of a 0.47 s query. #1008 reports the same cost from the query_np side.

This change moves the loop into a build_array_rows(column, offsets) helper in dataconv, with a Cython version in driverc/dataconv.pyx:

• The pure Python version is the same loop, moved. With CLICKHOUSE_CONNECT_USE_C=0 nothing changes beyond one function call per Array nesting level.
• The Cython version runs the same slices and appends without Python bytecode. Most of the gain on Python 3.12+ comes from the garbage collector. Since 3.12, CPython runs the cyclic GC from the eval breaker. While the Python loop creates a million new lists, the GC runs hundreds of times during the loop. Inside one C call it runs after the loop. On 3.10 the GC still runs at allocation time, so the gain there is smaller.
• Result rows contain the same element objects as before: Python ints, numpy scalars when the element column came back as a numpy array, strings, None, nested lists. Offsets past the end of the element list are clamped the same way Python slicing clamps them, so malformed offsets give the same rows with C on or off.
• Array.read_column_data now passes the elements to the helper only when they are exactly a list (type(...) is list instead of isinstance). The Cython signature takes list, so a list subclass is copied to a plain list first. No built-in reader returns a list subclass. The check only keeps a custom type from failing in the Cython path.

I first tried a pure Python version, list(map(column.__getitem__, map(slice, starts, ends))). It made rows about 26% faster on macOS with 3.12, but about 2% slower with 3.10, so I dropped it. I also tried preallocating the result list in Cython. It was as fast as appending, but on Linux with Python 3.10 its peak RSS moved by -9 to +4 MB depending on the query, while the append version stays within 2 MB of main. I kept the append version.

Performance

Table: perf.arr (u64 UInt64, arr Array(UInt64)), 1M rows, arr = range(number % 8) (0 to 7 elements, 3.5M elements in total). Queries:

• rows: query("SELECT u64, arr FROM perf.arr").result_rows
• np: query_np of the same query
• df: query_df of the same query
• nested: query(...).result_rows of SELECT u64, arrayMap(x -> range(x % 4), arr) FROM perf.arr, an Array(Array(UInt64))

Setup: ClickHouse 26.3 in Docker from the repo's docker-compose.yml, compress=False, native codec python. Each run is a fresh process that connects, runs SELECT 1, then times one query. main and this PR alternate, 11 runs each, and the order flips every pair. Wall and CPU (time.process_time) cover the query call. Peak RSS is ru_maxrss. The Python heap peak is one extra tracemalloc run per side. OPENBLAS_NUM_THREADS=1. Host: Apple Silicon. Linux runs use python:<ver>-bookworm containers (aarch64) on the ClickHouse container's Docker network, with the extensions built inside each container. The host load average was 4 to 7 during the runs because other work was running, so I compare medians of interleaved runs and show the range.

C extensions on, wall p50 in seconds [min-max], main -> this PR:

Environment	rows	np	df	nested
macOS, Python 3.10.20	0.428 -> 0.405 (-5%)	0.398 -> 0.375 (-6%)	0.552 -> 0.538 (-3%)	0.981 -> 0.860 (-12%)
macOS, Python 3.12.13	0.382 -> 0.262 (-31%)	0.411 -> 0.334 (-19%)	0.578 -> 0.434 (-25%)	0.902 -> 0.463 (-49%)
macOS, Python 3.13.13	0.347 -> 0.283 (-18%)	0.373 -> 0.306 (-18%)	0.471 -> 0.388 (-18%)	0.866 -> 0.499 (-42%)
Linux, Python 3.10.22	0.410 -> 0.378 (-8%)	0.390 -> 0.359 (-8%)	0.507 -> 0.477 (-6%)	1.108 -> 0.969 (-13%)
Linux, Python 3.13.18	0.337 -> 0.245 (-27%)	0.405 -> 0.321 (-21%)	0.467 -> 0.340 (-27%)	1.006 -> 0.544 (-46%)

Peak RSS p50 in MB, main / this PR, same runs:

Environment	rows	np	df	nested
macOS 3.10	262.8 / 262.6	309.6 / 309.6	352.2 / 351.9	533.4 / 533.2
macOS 3.12	265.0 / 264.9	318.7 / 319.9	388.9 / 390.2	535.1 / 536.2
macOS 3.13	267.8 / 267.1	313.9 / 315.1	364.2 / 362.8	536.9 / 536.2
Linux 3.10	230.5 / 230.6	292.7 / 292.7	337.0 / 337.0	500.3 / 498.5
Linux 3.13	236.1 / 236.2	300.8 / 300.4	341.8 / 342.4	498.9 / 499.2

The macOS RSS spread within one side is 5 to 15 MB, so the differences of about 1 MB in both directions are noise. On Linux the spread is tighter and the largest increase is 0.6 MB on df with 3.13, with the same min-max range on both sides. The Python heap peak from tracemalloc is the same on both sides in every cell, for example 178.8 MB for rows and 406.6 MB for nested.

C extensions off (CLICKHOUSE_CONNECT_USE_C=0), wall p50, main / this PR: every cell is within 1.3%, in both directions. For example rows on macOS 3.12 is 0.372 / 0.377 s, and on Linux 3.13 it is 0.353 / 0.352 s. The pure Python loop is unchanged.

Tests

• tests/unit_tests/test_driver/test_c_parity.py: test_build_array_rows_parity runs both helpers on offsets given as array.array("Q") and as a list, the two types the buffers return. It covers empty rows, an empty block, mixed element types, and malformed offsets [2**63, 0, 1]. test_build_array_rows_parity_keeps_element_objects checks that the rows hold the same numpy scalar objects as the input.
• tests/unit_tests/test_driver/test_native_read.py: test_array_native_round_trip_and_stream_alignment writes and reads Array(UInt8), Array(Nullable(String)), Array(LowCardinality(String)), Array(Array(UInt8)) and Array(Array(Array(UInt8))) with both buffer classes, then reads a following column to check that the stream stays aligned. test_array_numpy_read_keeps_numpy_scalars checks that a numpy-context read still returns numpy.uint64 elements. These pass on main too and pin the existing behavior.

Suites, Python 3.12 on macOS, ClickHouse 26.3:

• tests/unit_tests: 4933 passed, 957 skipped, with CLICKHOUSE_CONNECT_USE_C=1 and =0.
• tests/integration_tests: 2132 passed, 35 skipped, with CLICKHOUSE_CONNECT_USE_C=1 and =0.
• ruff format --check, ruff check and mypy: clean.

To reproduce, create the table with INSERT INTO perf.arr SELECT number, range(number % 8) FROM numbers(1000000) and time client.query("SELECT u64, arr FROM perf.arr").result_rows on main and on this branch with the extensions rebuilt (python setup.py build_ext --inplace).

Risk

The new code runs only with the C extensions, on every Array read with the Python codec, including arrays inside Tuple, Map values, Nested, Variant and Dynamic. The Rust codec is not affected. The main risk is a difference between the Cython and Python helpers. The parity tests cover the input types that the buffers produce. Reverting this commit restores the old loop.

Related to #1008. This does not add the flat values plus offsets API discussed there. It only makes the existing list-of-lists result cheaper to build.

Checklist

• Unit and integration tests covering the common scenarios were added
• A human-readable description of the changes was provided to include in CHANGELOG