[sebastianbreguel]

Summary

This PR fixes two ways DateTime64 values lose sub-second precision on the way to the server. The second part was opened separately as #1130 and is folded in here, since both touch binding.py and test_params.py.

1. pandas.Timestamp nanoseconds in binds and row inserts

pandas.Timestamp values lost their nanoseconds on two paths that handle DateTime64(9).

ts = pd.Timestamp("2024-01-02 03:04:05.123456789")
client.query("SELECT toString({d:DateTime64(9)})", parameters={"d": ts})  # 2024-01-02 03:04:05.123456000
client.insert("t", [[ts]])  # DateTime64(9) column stores 2024-01-02 03:04:05.123456000

insert_df already keeps the nanoseconds, and so does a pd.Timedelta bound to Time64(9).

• Binding: DT64Param.format used strftime("%f"), which only has microseconds. It now appends Timestamp.nanosecond when it is non-zero. Plain datetime output is unchanged.
• Row inserts (Python codec): DateTime64._datetime64_ticks used floor(value.timestamp()) plus microsecond. For values that are not plain datetime, it now takes whole seconds from the timestamp and adds the exact sub-second part (microsecond * 1000 + nanosecond).

This also fixes a second problem on the same path. Timestamp.timestamp() rounds to microseconds, so 03:04:05.999999999 became ...06.0. The floor then landed on the next second, and the row was stored one second late in any DateTime64 column. The same happened for pre-epoch values. The new code subtracts the sub-second part before rounding, so this cannot happen.

The plain datetime path is untouched, because it is the hot loop for row inserts. The only added cost is a type(value) is not datetime check per value.

Related to #429. That issue asks for nanosecond-precise binding of custom types. This PR does not add custom formatters, but it covers the built-in case behind it: binding and inserting nanosecond pandas.Timestamp values.

Out of scope, left unchanged: the Rust codec's DateTime64 insert, which has the same microsecond floor; non-64 DateTime inserts; and client-side %s binding of a bare datetime.

2. DateTime64 elements inside named Tuple, Nested, Variant and Map binds

When a datetime is bound server-side to a DateTime64 element, _promote_datetime64 wraps it in DT64Param to keep sub-second precision. This only worked at the top level, in Array, and in unnamed Tuple. Other container types sent whole seconds (ClickHouse 26.3, D = datetime(2024, 1, 2, 3, 4, 5, 123456), U = the same time in UTC):

{p:Tuple(DateTime64(6), Int32)}            ('2024-01-02 03:04:05.123456',1)   <- works
{p:Tuple(a DateTime64(6), b Int32)}        ('2024-01-02 03:04:05.000000',1)
{p:Tuple(a DateTime64(6, 'Asia/Tokyo'))}   ('2024-01-02 03:04:05.000000')    <- precision and timezone lost (U)
{p:Variant(DateTime64(6), String)}         2024-01-02 03:04:05.000000
{p:Nested(a DateTime64(6))}                [('2024-01-02 03:04:05.000000')]
{p:Map(String, DateTime64(6))}             {'k':'2024-01-02 03:04:05.000000'}  (dict_parameter_format='map')

Named elements failed because "a DateTime64(6)" was read as a type called a.

What changed, all in clickhouse_connect/driver/binding.py:

• _strip_element_name removes the name from a named element before promotion. It handles quoted names, including backtick or double quotes with doubled or backslash-escaped quotes, and names containing parentheses. DateTime64 (6) is not mistaken for a name.
• _promote_datetime64 now also handles Nested (as an array of tuples), Variant, and Map.
  • Map is handled only with dict_parameter_format='map'. JSON mode serializes the dict itself and is unchanged.
  • Variant is promoted only when it has no plain DateTime alternative. I checked on the server that a value with a fraction makes it pick DateTime64 instead of DateTime. Promoting would therefore change the variant that Variant(DateTime, DateTime64(6)) gets today, so that case stays as it is.
• DT64Param takes an optional tz. A promoted element carries the timezone named in its own DateTime64 type, so Tuple(a DateTime64(6, 'Asia/Tokyo'), b DateTime64(6, 'UTC')) renders each element in its own zone. Before, the whole parameter used one timezone. Top-level binds and Array binds produce the same strings as before. A DT64Param with no timezone placed inside one of these containers gets the element's timezone too. The wrapper is copied, so the caller's object is not changed. The SQLAlchemy subclass _DateTime64QueryParam honors that timezone. A DT64Param that already has a timezone, and the _64 suffix, are unchanged.
• _unwrap_outer and the element-timezone lookup are cached (lru_cache(maxsize=256)). Before this, every array element parsed the same type string again. Both caches hold only immutable results.

Still unchanged: plain DateTime (non-64) elements in a multi-timezone Tuple still share the first timezone found. That case has no wrapper, and fixing it is a separate change.

Verification

Both parts together, merged with main (f77899c): ruff check and mypy are clean. Unit test_params.py and test_parser.py: 301 passed. Integration test_params.py and test_pandas.py against ClickHouse 26.3: 104 passed.

Part 1

• Unit test test_dt64param_keeps_pandas_timestamp_nanoseconds. On main it fails: '2024-01-02 03:04:05.123456' == '2024-01-02 03:04:05.123456789'.

• Integration tests (sync and async): test_pandas_timestamp_datetime64_bind_nanoseconds and test_pandas_timestamp_datetime64_insert_nanoseconds. The insert test covers DateTime64(9), DateTime64(6), Nullable(DateTime64(9, 'UTC')) with a tz-aware value, a .999999999 carry case, and a 1960 value. On main all 4 fail, for example .123456000 vs .123456789, and 04:06.999999 vs 04:05.999999 for the carry. With the fix all 4 pass.

• I compared the new tick formula with the old one on 200,000 random plain datetime values from 1900 to 2300: 0 mismatches.

• ruff format --check, ruff check, and mypy are clean. Unit tests: 4338 passed. Integration (-n 4, ClickHouse 26.3): 1796 passed, 29 skipped. 2 failures are test_wrong_port_error_message, which needs local port 9000 (environment).

• Performance: I serialized 3 x 300k plain datetime values with DateTime64._write_column_binary. There were 20 interleaved runs per side, each in a fresh process, measured with /usr/bin/time -l. The machine was shared, with load average around 18.

  type	side	wall p50/p80/p95 (s)	CPU p50/p80/p95 (s)	peak RSS p50 (MiB)
  DateTime64(9)	main	0.61 / 0.62 / 0.66	0.58 / 0.59 / 0.61	80.8
  DateTime64(9)	PR	0.62 / 0.63 / 0.65	0.59 / 0.61 / 0.61	81.1
  DateTime64(3)	main	0.70 / 1.11 / 1.34	0.60 / 0.64 / 0.67	81.1
  DateTime64(3)	PR	0.64 / 1.05 / 1.31	0.61 / 0.68 / 0.69	81.0

  The differences are within noise. /usr/bin/time CPU resolution is 10 ms.

Part 2

• New unit tests in TestBindQueryDateTime64Precision: named tuple, named tuple tz hint, per-element timezones, quoted names (4 variants), existing DT64Param / SQLAlchemy wrappers taking the element timezone (directly and through Variant), nested, variant, map. Plus a guard showing that Variant(DateTime, DateTime64(6)) is unchanged.

• Integration tests test_datetime64_params_in_containers (6 cases, including a wrapped DT64Param) and test_datetime64_params_in_map, sync and async. They compare against the server's toString. On main, 25 of the selected unit and integration cases fail. With the fix all pass.

• ruff format --check, ruff check, and mypy are clean. Unit tests: 4352 passed. Integration (-n 4, ClickHouse 26.3, including the SQLAlchemy suite): 1806 passed, 29 skipped. 2 failures are test_wrong_port_error_message, which needs local port 9000 (environment). 18 unit errors are test_httpclient.py tests that hardcode localhost:8123 (environment).

• Performance: bind_query on origin/main vs this PR, timed with /usr/bin/time -l. There were 16 interleaved runs per side, each in a fresh process, and the order alternated each round. Python 3.12, Apple Silicon. Wall and CPU cover the bind loop. I started each case once the 1-minute load average was under 3; it stayed between 2.5 and 3.3 during the runs. Cases:

  • plain: {x:Int32}, {s:String}, {d:DateTime}, 20,000 binds.
  • arr1k: Array(DateTime64(6)) with 1,000 values, 200 binds.
  • arr100k: 100,000 values, 2 binds.
  • _tz: the same type with 'Asia/Tokyo'.

  case	side	wall p50/p80/p95 (s)	CPU p50/p80/p95 (s)	peak RSS p50 (MiB)
  plain	main	0.114 / 0.115 / 0.117	0.113 / 0.115 / 0.117	33.5
  plain	PR	0.101 / 0.102 / 0.103	0.101 / 0.102 / 0.103	33.5
  arr1k	main	0.506 / 0.514 / 0.517	0.505 / 0.513 / 0.517	33.7
  arr1k	PR	0.406 / 0.412 / 0.414	0.405 / 0.411 / 0.413	33.8
  arr1k_tz	main	0.756 / 0.757 / 0.761	0.754 / 0.756 / 0.760	33.8
  arr1k_tz	PR	0.428 / 0.433 / 0.437	0.427 / 0.432 / 0.437	33.8
  arr100k	main	0.513 / 0.523 / 0.530	0.511 / 0.522 / 0.529	61.6
  arr100k	PR	0.421 / 0.425 / 0.427	0.420 / 0.424 / 0.426	61.7
  arr100k_tz	main	0.796 / 0.803 / 0.809	0.795 / 0.799 / 0.805	61.7
  arr100k_tz	PR	0.452 / 0.462 / 0.464	0.451 / 0.460 / 0.463	61.6

  Nothing got slower. DateTime64 array binds are about 18 to 43% faster because of the parse cache, and plain about 11%. Peak RSS is within noise: the medians differ by at most 0.1 MiB.

Checklist

• Unit and integration tests covering the common scenarios were added
• A human-readable description of the changes was provided to include in CHANGELOG

[CLAassistant]

All committers have signed the CLA.

[joe-clickhouse]

Approved.

On top of your work, I:

• Restricted Variant promotion to a sole scalar DateTime64 datetime alternative, preserving existing ambiguous behavior and avoiding timezone regressions.
• Reused and cached parse_columns for Tuple/Nested hints, fixing whitespace handling without duplicating parsing or adding per-row overhead.
• Shared nanosecond rendering with SQLAlchemy and fixed compatibility with DT64Param subclasses that skip the base initializer.
• Added regression tests, scoped Python-codec insert coverage correctly, and updated the docs and changelog.

[Copilot]

🟡 Changes recommended

The substring check misclassifies enum labels and tuple field names as datetime alternatives, causing fractional-second loss during binding and requiring parsed-type classification with regression coverage.

1 open issue

• Classify datetime alternatives by type names, not labels or fields · New

What changed in this PR

Preserves DateTime64 precision across query binding and Python-codec row inserts, including pandas nanoseconds and datetime values inside containers.

Changes:

• Retains pandas nanoseconds in driver and SQLAlchemy binds, and fixes timestamp rounding during Python-codec inserts.
• Extends container binding support with per-element timezones and cached type parsing.
• Adds regression coverage and documents supported behavior and limitations.

File	Description
tests/​unit_tests/​test_driver/​test_params.py	Tests nanoseconds, container promotion, timezones, and parsing caches.
tests/​integration_tests/​test_sqlalchemy/​test_datetime64.py	Tests pandas nanoseconds in SQLAlchemy parameters and literals.
tests/​integration_tests/​test_params.py	Tests container and Variant binding through both clients.
tests/​integration_tests/​test_pandas.py	Tests nanosecond binds and Python-codec row inserts.
docs/​sqlalchemy.mdx	Documents nanosecond preservation in typed binds.
docs/​driver-api.mdx	Documents container support and Variant restrictions.
clickhouse_connect/​driver/​binding.py	Preserves nanoseconds and promotes container elements with timezone hints.
clickhouse_connect/​datatypes/​temporal.py	Preserves timestamp nanoseconds and corrects second-boundary rounding.
clickhouse_connect/​cc_sqlalchemy/​datatypes/​base.py	Reuses nanosecond rendering and honors wrapper timezones.
CHANGELOG.md	Records precision fixes and container-binding behavior.

_{Review effort: Balanced}