Skip to content

Make tutorial helpers work with pandas 1.5 and pandas 3 - #28

Merged
JortGroen merged 2 commits into
devfrom
fix/pandas3-helpers
Nov 6, 2026
Merged

JortGroen merged 2 commits into
devfrom
fix/pandas3-helpers

Conversation

@JortGroen

@JortGroen JortGroen commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

What

Four lines in the tutorial helpers, so they work with both pandas 1.5 and pandas 3, plus one line in the CI environment:

  • tutorials/power_balance/functions_T1.py (2×) and tutorials/physical_congestion/functions_T3.py (1×):
    result_pd_df.index = pd.to_datetime(result_pd_df.index) → result_pd_df.index = result_pd_df.index.map(pd.Timestamp)
  • tutorials/power_balance/functions_T1.py: resample('H') → resample('h')
  • environment.yml: add nest-asyncio. Separate commit, unrelated to pandas: ipykernel 7 now depends on nest-asyncio2 instead of nest_asyncio, so in a fresh environment the notebooks' import nest_asyncio failed and every test errored. dev has the same problem; its last green run was in June, before this change.

Why

  • Illuminator's Collector writes the first output row as a bare date (2012-06-01) and later rows with a time (2012-06-01 00:15:00). pandas ≥ 2 infers one format from the first value, so pd.to_datetime fails with ValueError: unconverted data remains ... " 00:15:00". index.map(pd.Timestamp) parses each value on its own.
  • pandas 3 removed the 'H' frequency alias.
  • Illuminator's check-example-compatibility.yaml runs this repo's dev tests against Illuminator PR branches. The Illuminator Python 3.14 + pandas 3 upgrade (issue 259, branch 259) can't pass that check until this is in dev.

pandas 1.5 / 3 compatibility

  • format="mixed" (the fix used in Update Python version from 3.11 to 3.14 #25) doesn't exist in pandas 1.5.3, which the PyPI release of illuminator (3.1b0) pins and this repo's CI installs. index.map(pd.Timestamp) works in both and returns a DatetimeIndex.
  • 'h' is accepted by pandas 1.5 as well.
  • Tested locally: full pytest -v suite passes on Python 3.12 + pandas 3.1.1 + Illuminator branch 259. This PR's CI covers pandas 1.5 (PyPI illuminator).

Relation to #25

Supersedes only the helper part of the closed #25. That PR also moved the env to Python 3.14 + pandas 3 + illuminator @ git…@259, which is left out here (see follow-ups).

Follow-ups (not in this PR)

  • Remove nest_asyncio (import nest_asyncio / nest_asyncio.apply()) from the notebooks, together with the Python 3.14 / pandas 3 env move from Update Python version from 3.11 to 3.14 #25, after an Illuminator release that includes the new Simulation.run() fix (runs mosaik in a worker thread when Jupyter already has an event loop). The published illuminator still needs nest_asyncio, so it stays for now. On Python 3.14, nest_asyncio gives silently wrong simulation results. Note that the in-progress branches tutorial-5, tutorial-5-ide-integration and ide-case-final add more notebooks that use it.
  • tutorials/residential/Illuminator_BSc_Tutorial.ipynb (~line 443): pd.to_datetime(df_latest5['date'], errors='coerce') has the same mixed-format problem. Under pandas 3 every row after the first becomes NaT (checked: ['2012-06-01', '2012-06-01 00:15:00'] → [Timestamp('2012-06-01'), NaT]). Not covered by the test suite.

🤖 Generated with Claude Code

JortGroen and others added 2 commits November 6, 2026 11:43
Parse the Collector output index with index.map(pd.Timestamp) so the
first bare-date row ("2012-06-01") doesn't fix the inferred format for
later rows with times, and use the 'h' resample alias ('H' was removed
in pandas 3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ipykernel 7 now depends on nest-asyncio2 instead of nest_asyncio, so
the notebooks' `import nest_asyncio` fails in a fresh environment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JortGroen
JortGroen requested a review from Dialectiek November 6, 2026 11:04
@JortGroen
JortGroen merged commit c26c66e into dev Nov 6, 2026
1 check passed
@JortGroen
JortGroen deleted the fix/pandas3-helpers branch November 6, 2026 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant