Describe the bug, including details regarding any error messages, version, and platform.
pyarrow.compute.indices_nonzero() segfaults when given a ChunkedArray that has zero chunks.
This is not the same thing as an empty array. A zero-length Array works, and a ChunkedArray
holding one empty chunk works. It is specifically the chunkless ChunkedArray that crashes —
which is exactly the shape you get from ordinary operations like filtering a table down to no rows,
or slicing zero rows out of it. So this is easy to hit from normal code and hard to reproduce if you
construct "an empty array" the obvious way.
The process dies with SIGSEGV; there is no Python-level exception to catch.
Minimal reproducer
import pyarrow as pa
import pyarrow.compute as pc
t = pa.table({"x": pa.array([1, 2, 3])})
empty = t.filter(pc.equal(t["x"], 99)) # matches nothing
mask = pc.equal(empty.column("x"), 1) # ChunkedArray, num_chunks == 0, len == 0
pc.indices_nonzero(mask) # <- segfault
What works vs. what crashes
| input |
num_chunks |
result |
pa.array([], type=pa.bool_()) |
n/a (Array) |
✅ returns [] |
pa.chunked_array([pa.array([], type=pa.bool_())]) |
1 |
✅ returns [] |
pa.chunked_array([], type=pa.bool_()) |
0 |
🔴 SIGSEGV |
Every one of these routes to a zero-chunk boolean ChunkedArray crashes:
pa.chunked_array([], type=pa.bool_()) # constructed directly
pc.equal(pa.chunked_array([[1]]).filter(pa.array([False])), 1)
pc.equal(t.filter(pc.equal(t["x"], 99)).column("x"), 1) # table filtered to no rows
pc.equal(t.slice(0, 0).column("x"), 1) # zero-row slice
pc.equal(pc.list_flatten(empty_list_column), 1) # flatten of an empty list column
Expected behaviour
Return an empty index array, consistent with the zero-length and one-empty-chunk cases.
Affected versions
Reproduced on macOS (arm64), CPython 3.12, on every version I tried:
| pyarrow |
result |
| 18.1.0 |
segfault |
| 21.0.0 |
segfault |
| 24.0.0 |
segfault |
| 25.0.0 |
segfault |
| 25.0.1 (latest release) |
segfault |
So it is long-standing rather than a recent regression, and there is no released version to upgrade to.
Scope
It appears specific to this kernel rather than general to zero-chunk handling. On the identical
input, unique, value_counts, sort_indices, dictionary_encode, fill_null_forward and rank
all return cleanly; list_flatten, cumulative_sum and index_in raise ordinary Python exceptions.
indices_nonzero was the only one that crashed the process.
Possibly related
Zero-chunk ChunkedArray has been a crash source before and was fixed for Slice:
This looks like the same class of defect in a different kernel.
Component(s)
C++, Python
Describe the bug, including details regarding any error messages, version, and platform.
pyarrow.compute.indices_nonzero()segfaults when given aChunkedArraythat has zero chunks.This is not the same thing as an empty array. A zero-length
Arrayworks, and aChunkedArrayholding one empty chunk works. It is specifically the chunkless
ChunkedArraythat crashes —which is exactly the shape you get from ordinary operations like filtering a table down to no rows,
or slicing zero rows out of it. So this is easy to hit from normal code and hard to reproduce if you
construct "an empty array" the obvious way.
The process dies with SIGSEGV; there is no Python-level exception to catch.
Minimal reproducer
What works vs. what crashes
num_chunkspa.array([], type=pa.bool_())Array)[]pa.chunked_array([pa.array([], type=pa.bool_())])[]pa.chunked_array([], type=pa.bool_())Every one of these routes to a zero-chunk boolean
ChunkedArraycrashes:Expected behaviour
Return an empty index array, consistent with the zero-length and one-empty-chunk cases.
Affected versions
Reproduced on macOS (arm64), CPython 3.12, on every version I tried:
So it is long-standing rather than a recent regression, and there is no released version to upgrade to.
Scope
It appears specific to this kernel rather than general to zero-chunk handling. On the identical
input,
unique,value_counts,sort_indices,dictionary_encode,fill_null_forwardandrankall return cleanly;
list_flatten,cumulative_sumandindex_inraise ordinary Python exceptions.indices_nonzerowas the only one that crashed the process.Possibly related
Zero-chunk
ChunkedArrayhas been a crash source before and was fixed forSlice:[C++] Slicing a ChunkedArray with zero chunks segfaults(ARROW-8911)This looks like the same class of defect in a different kernel.
Component(s)
C++, Python