Skip to content

Commit 97513b3

Browse files
authored
Merge pull request #420 from CovertLab/pq-nd-array
Document limitation with N-D array emits
2 parents 5156b08 + 98f37fe commit 97513b3

4 files changed

Lines changed: 26 additions & 3 deletions

File tree

‎doc/output.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -180,6 +180,12 @@ of integers where some entries are empty lists).
180180
The Parquet emitter is poorly suited for storing large listeners that have more
181181
than a single dimension per time step. We recommend splitting these listeners up
182182
if possible, especially if you plan to read specific indices along those dimensions.
183+
If you choose to emit multi-dimensional listeners, they must have a default value
184+
defined through a ports schema that is the same shape as the emitted values at
185+
all time steps. For example, if emitted values are 2-D arrays of shape (3, 4),
186+
the default value must also be a 2-D array of shape (3, 4). This is because
187+
the Parquet emitter pre-allocates its NumPy buffer from the default's shape
188+
and any later mismatch falls to the Polars path, which doesn't handle N-D arrays.
183189

184190
The Parquet emitter saves the serialized tabular data to two Hive-partitioned
185191
directories in the output folder (``out_dir`` or ``out_uri`` option under

‎ecoli/library/parquet_emitter.py‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1123,7 +1123,22 @@ def emit(self, data: dict[str, Any]):
11231123
:emit_idx
11241124
].tolist() + [None] * (self.batch_size - emit_idx)
11251125
# Fall back Polars serialization
1126-
v = pl.Series([v])
1126+
try:
1127+
v = pl.Series([v])
1128+
except ValueError as e:
1129+
if isinstance(v, np.ndarray):
1130+
if v.ndim > 1:
1131+
raise ValueError(
1132+
f"Field {k} is a NumPy array with "
1133+
"more than 1 dimension, which cannot be "
1134+
"serialized using the fallback Polars logic. "
1135+
"If this field has a consistent shape, update "
1136+
"the ports schema to define a default value "
1137+
"with that expected shape. Otherwise, convert "
1138+
"the value to Python lists using tolist() or "
1139+
"flatten it to 1-D and emit the shape separately."
1140+
) from e
1141+
raise
11271142
# Ensure type consistency
11281143
curr_type = self.pl_types.setdefault(k, pl.Null)
11291144
if v.dtype != curr_type:

‎ecoli/library/test_parquet_emitter.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1391,7 +1391,7 @@ def test_expected_failures(self, temp_dir):
13911391
}
13921392
with pytest.raises(
13931393
ValueError,
1394-
match=re.escape("cannot parse numpy data type dtype('O')"),
1394+
match=re.escape("more than 1 dimension"),
13951395
):
13961396
emitter.emit(sim_data7)
13971397

‎ecoli/processes/metabolism_redux.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -395,7 +395,9 @@ def ports_schema(self):
395395
"estimated_exchange_dmdt": {},
396396
"estimated_intermediate_dmdt": [],
397397
"target_kinetic_fluxes": [],
398-
"target_kinetic_bounds": [],
398+
"target_kinetic_bounds": np.zeros(
399+
(len(self.kinetic_constraint_reactions), 2), dtype=int
400+
),
399401
"reaction_catalyst_counts": [],
400402
"maintenance_target": 0,
401403
}

0 commit comments

Comments
 (0)