Skip to content

[SPARK-58766][PYTHON][TESTS] Represent out-of-range temporal PyArrow scalars in golden files #58000

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Spenserrrr wants to merge 3 commits into
base: master
Choose a base branch
from
+196 −175
Open

[SPARK-58766][PYTHON][TESTS] Represent out-of-range temporal PyArrow scalars in golden files #58000

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2a226c5
[SPARK-58766][PYTHON][TESTS] Represent out-of-range temporal PyArrow …
Spenserrrr Aug 14, 2026
b86f25c
[SPARK-58766][PYTHON][TESTS] Use temporal[raw](value) for out-of-rang…
Spenserrrr Aug 14, 2026
0a718f6
[SPARK-58766][PYTHON][TESTS] Render out-of-range temporal scalars as …
Spenserrrr Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 24 additions & 3 deletions python/pyspark/testing/goldenutils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,12 +311,33 @@ def repr_type(t: Any) -> str:
# "halffloat" -> "float16", "float" -> "float32", "double" -> "float64"
return _ARROW_FLOAT_ALIASES.get(s, s)

@staticmethod
def _scalar_str(scalar: Any) -> str:
"""
Render one PyArrow scalar for a golden cell via PyArrow's own ``str(scalar)``.

A temporal value can be valid in Arrow yet outside Python's ``datetime`` range
(e.g. a date32 past year 9999), where ``str`` builds a Python datetime and
raises ``OverflowError``. For those, record the raw stored count as
``raw=<value>`` (its unit is in the type suffix) instead of failing. A
non-temporal ``OverflowError`` is unexpected, so it propagates.
"""
try:
return str(scalar).replace("\x00", "\\0")
except OverflowError:
# No Python datetime exists for this value; record the raw stored count as
# ``raw=<value>``. A non-temporal overflow is unexpected, so re-raise.
if pa.types.is_temporal(scalar.type):
return f"raw={scalar.value}"

@Yicong-Huang Yicong-Huang Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two questions:

  1. for upstreaming mornitoring purpose, we want to write this into a golden file, so we can choose a format to represent it. however currently raw does not read as a timestamp. maybe a format like temporal[raw](xxxx) or raw-temporal=xxxx would be better?
  2. eventually the time value has to be processed in python, using python datatime, pandas to_datetime etc. Also values more than 9999 year are not really meaningful anyway IMO. can we check on the scala side what is the range for datetime? if year 9999 is a valid upper bound for spark, we might not need to care about arrow's loose bound?

@Spenserrrr Spenserrrr Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Yicong-Huang!

(1) format: yeah raw= is not clear. I updated it to temporal[raw](<value>) shape, so the cell now reads [temporal[raw](2147483647), temporal[raw](-2147483648), None]@date32[day]. The unit stays in the @date32[day] suffix.

(2) Spark's date range: I checked the Scala side. In sql/api/src/main/scala/org/apache/spark/sql/types/DateType.scala, the [0001-01-01, 9999-12-31] is a documented "valid range" in line 23-24, but it isn't actually enforced:

  • In that file, DateType is defined as an int32 day-count in line 35, so its real range is the same as Arrow's date32
  • In sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala, stringToDate parses up to 7-digit years, with the comment "An integer is able to represent a date within [+-]5 million years" in line 578

The one place year 9999 matters is in toPandas. When Spark passes date_as_object=True, each date becomes a Python datetime.date, and this is hard-capped at year 9999. With date_as_object=False it wouldn't raise.

So I agree that a year that large isn't a meaningful Spark date, but we need a rendering for this cell anyway because int32:max/min is a valid Arrow date32 and Python's str() can't render it correctly. Another way to make this explicit is that we can just drop the value and render it as when the date overflows. Please let me know if you have any other suggestions.

@Yicong-Huang Yicong-Huang Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we are monitoring arrow's behavior for pyspark/spark, so it is valid to say that we don't support year > 9999. maybe we can just render overflow?

@Spenserrrr Spenserrrr Aug 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that makes sense. I went with temporal overflow, and the cell now renders [temporal overflow, temporal overflow, None]@date32[day].

raise

@classmethod
def repr_arrow_value(cls, value: Any, max_len: int = 32) -> str:
"""
Format a PyArrow Array/ChunkedArray for golden file.

Each element uses str(scalar) from PyArrow's own scalar formatting.
Each element is rendered by ``_scalar_str`` (PyArrow's scalar formatting, with
an out-of-range temporal fallback).

Parameters
----------
Expand All @@ -331,7 +352,7 @@ def repr_arrow_value(cls, value: Any, max_len: int = 32) -> str:
"[val1, val2, None]@arrow_type"
"""
# Escape NULL bytes so the value can be safely stored in CSV files.
elements = [str(scalar).replace("\x00", "\\0") for scalar in value]
elements = [cls._scalar_str(scalar) for scalar in value]
v_str = "[" + ", ".join(elements) + "]"
if max_len > 0:
v_str = v_str[:max_len]
Expand All @@ -353,7 +374,7 @@ def repr_arrow_table_value(cls, value: Any, max_len: int = 32) -> str:
columns = []
for name, column in zip(value.column_names, value.columns):
# Escape NULL bytes so the value can be safely stored in CSV files.
elements = [str(scalar).replace("\x00", "\\0") for scalar in column]
elements = [cls._scalar_str(scalar) for scalar in column]
columns.append(f"{name}: [" + ", ".join(elements) + "]")
v_str = "{" + ", ".join(columns) + "}"
if max_len > 0:
Expand Down
2 changes: 1 addition & 1 deletion python/pyspark/tests/upstream/pyarrow/golden_pyarrow_scalar_cast_safe.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ int16:negative [-1, None]@int8 [-1, None]@int16 [-1, None]@int32 [-1, None]@int6
int16:max_min ERR@ArrowInvalid [32767, -32768, None]@int16 [32767, -32768, None]@int32 [32767, -32768, None]@int64 ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid [32768.0, -32768.0, None]@float16 [32767.0, -32768.0, None]@float32 [32767.0, -32768.0, None]@float64 [True, True, None]@bool [32767, -32768, None]@string [32767, -32768, None]@large_string ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [32767.0000000000, -32768.0000000000, None]@decimal128(38, 10) [32767.0000000000, -32768.0000000000, None]@decimal256(76, 10) ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
int32:standard [0, 1, None]@int8 [0, 1, None]@int16 [0, 1, None]@int32 [0, 1, None]@int64 [0, 1, None]@uint8 [0, 1, None]@uint16 [0, 1, None]@uint32 [0, 1, None]@uint64 [0.0, 1.0, None]@float16 [0.0, 1.0, None]@float32 [0.0, 1.0, None]@float64 [False, True, None]@bool [0, 1, None]@string [0, 1, None]@large_string ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [0E-10, 1.0000000000, None]@decimal128(38, 10) [0E-10, 1.0000000000, None]@decimal256(76, 10) [1970-01-01, 1970-01-02, None]@date32[day] ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [00:00:00, 00:00:01, None]@time32[s] [00:00:00, 00:00:00.001000, None]@time32[ms] ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
int32:negative [-1, None]@int8 [-1, None]@int16 [-1, None]@int32 [-1, None]@int64 ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid [-1.0, None]@float16 [-1.0, None]@float32 [-1.0, None]@float64 [True, None]@bool [-1, None]@string [-1, None]@large_string ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [-1.0000000000, None]@decimal128(38, 10) [-1.0000000000, None]@decimal256(76, 10) [1969-12-31, None]@date32[day] ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [23:59:59, None]@time32[s] [23:59:59.999000, None]@time32[ms] ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
int32:max_min ERR@ArrowInvalid ERR@ArrowInvalid [2147483647, -2147483648, None]@int32 [2147483647, -2147483648, None]@int64 ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid [2147483647.0, -2147483648.0, None]@float64 [True, True, None]@bool [2147483647, -2147483648, None]@string [2147483647, -2147483648, None]@large_string ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [2147483647.0000000000, -2147483648.0000000000, None]@decimal128(38, 10) [2147483647.0000000000, -2147483648.0000000000, None]@decimal256(76, 10) ERR@OverflowError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [03:14:07, 20:45:52, None]@time32[s] [20:31:23.647000, 03:28:36.352000, None]@time32[ms] ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
int32:max_min ERR@ArrowInvalid ERR@ArrowInvalid [2147483647, -2147483648, None]@int32 [2147483647, -2147483648, None]@int64 ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid [2147483647.0, -2147483648.0, None]@float64 [True, True, None]@bool [2147483647, -2147483648, None]@string [2147483647, -2147483648, None]@large_string ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [2147483647.0000000000, -2147483648.0000000000, None]@decimal128(38, 10) [2147483647.0000000000, -2147483648.0000000000, None]@decimal256(76, 10) [raw=2147483647, raw=-2147483648, None]@date32[day] ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [03:14:07, 20:45:52, None]@time32[s] [20:31:23.647000, 03:28:36.352000, None]@time32[ms] ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
int64:standard [0, 1, None]@int8 [0, 1, None]@int16 [0, 1, None]@int32 [0, 1, None]@int64 [0, 1, None]@uint8 [0, 1, None]@uint16 [0, 1, None]@uint32 [0, 1, None]@uint64 [0.0, 1.0, None]@float16 [0.0, 1.0, None]@float32 [0.0, 1.0, None]@float64 [False, True, None]@bool [0, 1, None]@string [0, 1, None]@large_string ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [0E-10, 1.0000000000, None]@decimal128(38, 10) [0E-10, 1.0000000000, None]@decimal256(76, 10) ERR@ArrowNotImplementedError [1970-01-01, 1970-01-01, None]@date64[ms] [1970-01-01 00:00:00, 1970-01-01 00:00:01, None]@timestamp[s] [1970-01-01 00:00:00, 1970-01-01 00:00:00.001000, None]@timestamp[ms] [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, None]@timestamp[us] [1970-01-01 00:00:00, 1970-01-01 00:00:00.000000001, None]@timestamp[ns] [1970-01-01 00:00:00+00:00, 1970-01-01 00:00:01+00:00, None]@timestamp[s, tz=UTC] [1970-01-01 00:00:00+00:00, 1970-01-01 00:00:00.001000+00:00, None]@timestamp[ms, tz=UTC] [1970-01-01 00:00:00+00:00, 1970-01-01 00:00:00.000001+00:00, None]@timestamp[us, tz=UTC] [1970-01-01 00:00:00+00:00, 1970-01-01 00:00:00.000000001+00:00, None]@timestamp[ns, tz=UTC] [1969-12-31 19:00:00-05:00, 1969-12-31 19:00:01-05:00, None]@timestamp[s, tz=America/New_York] [1970-01-01 08:00:00+08:00, 1970-01-01 08:00:01+08:00, None]@timestamp[s, tz=Asia/Shanghai] [0:00:00, 0:00:01, None]@duration[s] [0:00:00, 0:00:00.001000, None]@duration[ms] [0:00:00, 0:00:00.000001, None]@duration[us] [0 days 00:00:00, 0 days 00:00:00.000000001, None]@duration[ns] ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [00:00:00, 00:00:00.000001, None]@time64[us] [00:00:00, 00:00:00, None]@time64[ns]
int64:negative [-1, None]@int8 [-1, None]@int16 [-1, None]@int32 [-1, None]@int64 ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid [-1.0, None]@float16 [-1.0, None]@float32 [-1.0, None]@float64 [True, None]@bool [-1, None]@string [-1, None]@large_string ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [-1.0000000000, None]@decimal128(38, 10) [-1.0000000000, None]@decimal256(76, 10) ERR@ArrowNotImplementedError [1969-12-31, None]@date64[ms] [1969-12-31 23:59:59, None]@timestamp[s] [1969-12-31 23:59:59.999000, None]@timestamp[ms] [1969-12-31 23:59:59.999999, None]@timestamp[us] [1969-12-31 23:59:59.999999999, None]@timestamp[ns] [1969-12-31 23:59:59+00:00, None]@timestamp[s, tz=UTC] [1969-12-31 23:59:59.999000+00:00, None]@timestamp[ms, tz=UTC] [1969-12-31 23:59:59.999999+00:00, None]@timestamp[us, tz=UTC] [1969-12-31 23:59:59.999999999+00:00, None]@timestamp[ns, tz=UTC] [1969-12-31 18:59:59-05:00, None]@timestamp[s, tz=America/New_York] [1970-01-01 07:59:59+08:00, None]@timestamp[s, tz=Asia/Shanghai] [-1 day, 23:59:59, None]@duration[s] [-1 day, 23:59:59.999000, None]@duration[ms] [-1 day, 23:59:59.999999, None]@duration[us] [-1 days +23:59:59.999999999, None]@duration[ns] ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [23:59:59.999999, None]@time64[us] [23:59:59.999999, None]@time64[ns]
int64:max_min ERR@ArrowInvalid ERR@ArrowInvalid [2147483647, -2147483648, None]@int32 [2147483647, -2147483648, None]@int64 ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid [inf, -inf, None]@float16 ERR@ArrowInvalid [2147483647.0, -2147483648.0, None]@float64 [True, True, None]@bool [2147483647, -2147483648, None]@string [2147483647, -2147483648, None]@large_string ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [2147483647.0000000000, -2147483648.0000000000, None]@decimal128(38, 10) [2147483647.0000000000, -2147483648.0000000000, None]@decimal256(76, 10) ERR@ArrowNotImplementedError [1970-01-25, 1969-12-07, None]@date64[ms] [2038-01-19 03:14:07, 1901-12-13 20:45:52, None]@timestamp[s] [1970-01-25 20:31:23.647000, 1969-12-07 03:28:36.352000, None]@timestamp[ms] [1970-01-01 00:35:47.483647, 1969-12-31 23:24:12.516352, None]@timestamp[us] [1970-01-01 00:00:02.147483647, 1969-12-31 23:59:57.852516352, None]@timestamp[ns] [2038-01-19 03:14:07+00:00, 1901-12-13 20:45:52+00:00, None]@timestamp[s, tz=UTC] [1970-01-25 20:31:23.647000+00:00, 1969-12-07 03:28:36.352000+00:00, None]@timestamp[ms, tz=UTC] [1970-01-01 00:35:47.483647+00:00, 1969-12-31 23:24:12.516352+00:00, None]@timestamp[us, tz=UTC] [1970-01-01 00:00:02.147483647+00:00, 1969-12-31 23:59:57.852516352+00:00, None]@timestamp[ns, tz=UTC] [2038-01-18 22:14:07-05:00, 1901-12-13 15:45:52-05:00, None]@timestamp[s, tz=America/New_York] [2038-01-19 11:14:07+08:00, 1901-12-14 04:45:52+08:00, None]@timestamp[s, tz=Asia/Shanghai] [24855 days, 3:14:07, -24856 days, 20:45:52, None]@duration[s] [24 days, 20:31:23.647000, -25 days, 3:28:36.352000, None]@duration[ms] [0:35:47.483647, -1 day, 23:24:12.516352, None]@duration[us] [0 days 00:00:02.147483647, -1 days +23:59:57.852516352, None]@duration[ns] ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [00:35:47.483647, 23:24:12.516352, None]@time64[us] [00:00:02.147483, 23:59:57.852516, None]@time64[ns]
Expand Down
Loading