[SPARK-58766][PYTHON][TESTS] Represent out-of-range temporal PyArrow scalars in golden files

[Spenserrrr]

What changes were proposed in this pull request?

The golden-file cell formatter GoldenFileTestMixin.repr_arrow_value (in python/pyspark/testing/goldenutils.py) renders each element with str(scalar). This raises for a PyArrow temporal value that is valid in Arrow but outside Python's datetime range: for example, int32 max cast to date32 is a valid date32 whose day count lands around year 5.88M, so building a Python datetime.date raises OverflowError. The cast succeeds; only the string rendering fails.

This PR extracts a _scalar_str helper (used by both repr_arrow_value and repr_arrow_table_value) that, on OverflowError, records the raw stored value as raw=<value> when the scalar is temporal (pa.types.is_temporal), and re-raises otherwise. The two affected cast golden cells (int32:max_min x date32, in the safe and unsafe goldens) change from ERR@OverflowError to [raw=2147483647, raw=-2147483648, None]@date32[day].

This was surfaced while following up on the review of #57939 (SPARK-58720), which suggested narrowing the golden cell try/except to guard only the conversion call: with the formatter no longer raising on a valid-but-unrenderable value, that narrowing becomes safe. The analogous non-UTF-8 binary -> string case (which raises UnicodeDecodeError), together with the narrowing itself, will follow as a separate follow-up.

Why are the changes needed?

ERR@OverflowError reads like a failed conversion, but the conversion actually succeeded and produced a valid date32; only Python's str() cannot render it. Recording the raw stored value keeps the golden honest and reserves ERR@ for genuine conversion failures. It also lets the cell try/except be narrowed to the conversion (a follow-up) without a successful-but-unrenderable value crashing the test.

Does this PR introduce any user-facing change?

No. This changes test infrastructure only (golden-file test helpers and their golden files).

How was this patch tested?

Existing golden-file tests, with the two affected cells regenerated. Verified across PyArrow 18-25 x pandas 2/3 (16 combinations), all passing; the affected scalar cast matrix runs wherever numpy >= 2.0.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Isaac

---

This pull request and its description were written by Isaac.

[Spenserrrr]

Hi @Yicong-Huang! This is a PR to fix overflow date representation in golden files. Could you take a look when you have time? Thanks!

[Yicong-Huang]

Two questions:

1. for upstreaming mornitoring purpose, we want to write this into a golden file, so we can choose a format to represent it. however currently raw does not read as a timestamp. maybe a format like temporal[raw](xxxx) or raw-temporal=xxxx would be better?
2. eventually the time value has to be processed in python, using python datatime, pandas to_datetime etc. Also values more than 9999 year are not really meaningful anyway IMO. can we check on the scala side what is the range for datetime? if year 9999 is a valid upper bound for spark, we might not need to care about arrow's loose bound?

[Spenserrrr]

Thanks @Yicong-Huang!

(1) format: yeah raw= is not clear. I updated it to temporal[raw](<value>) shape, so the cell now reads [temporal[raw](2147483647), temporal[raw](-2147483648), None]@date32[day]. The unit stays in the @date32[day] suffix.

(2) Spark's date range: I checked the Scala side. In sql/api/src/main/scala/org/apache/spark/sql/types/DateType.scala, the [0001-01-01, 9999-12-31] is a documented "valid range" in line 23-24, but it isn't actually enforced:

• In that file, DateType is defined as an int32 day-count in line 35, so its real range is the same as Arrow's date32
• In sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala, stringToDate parses up to 7-digit years, with the comment "An integer is able to represent a date within [+-]5 million years" in line 578

The one place year 9999 matters is in toPandas. When Spark passes date_as_object=True, each date becomes a Python datetime.date, and this is hard-capped at year 9999. With date_as_object=False it wouldn't raise.

So I agree that a year that large isn't a meaningful Spark date, but we need a rendering for this cell anyway because int32:max/min is a valid Arrow date32 and Python's str() can't render it correctly. Another way to make this explicit is that we can just drop the value and render it as when the date overflows. Please let me know if you have any other suggestions.

[Yicong-Huang]

I think we are monitoring arrow's behavior for pyspark/spark, so it is valid to say that we don't support year > 9999. maybe we can just render overflow?

[Spenserrrr]

Yeah, that makes sense. I went with temporal overflow, and the cell now renders [temporal overflow, temporal overflow, None]@date32[day].

[Yicong-Huang]

LGTM, thanks @Spenserrrr