Optimize Dask EWA persist and prune fornav tasks

[djhoese]

This is an evolution of #707 that @Zaczero started. I rebased it and then had Claude overhaul it with ideas that it had and that I had suggested in that PR. I didn't feel comfortable forcing pushing all of this on to your PR @Zaczero. You're still the author of your commit so no worries there. This PR now does the same fixes as #707 and pruning improvements but:

• Does more strict filtering of the input ll2cr blocks by not only checking for finite values, but values inside the target grid. ll2cr, by design, reports cols/rows even if they are outside the target grid. Fornav uses some of these values so Claude made sure to include a margin.
• Newer dask blockwise dependency objects which now provide overlap information to fornav tasks per-block
• Small optimizations. For example, using cols.min(where=...) rather than getting the filtered result and then doing min and max on it.
• Type annotations

Next I'm going to have Claude run a ton of benchmarks on this and main and see what it gets.

• Closes #xxxx
• Tests added
• Tests passed
• Fully documented

[codecov]

Codecov Report

❌ Patch coverage is 99.60938% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 93.87%. Comparing base (8aca19c) to head (82b485c).
⚠️ Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
pyresample/ewa/dask_ewa.py	99.28%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #750      +/-   ##
==========================================
+ Coverage   93.68%   93.87%   +0.19%     
==========================================
  Files          89       89              
  Lines       13721    13888     +167     
==========================================
+ Hits        12854    13037     +183     
+ Misses        867      851      -16     

Flag	Coverage Δ
unittests	93.87% <99.60%> (+0.19%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[djhoese]

Ok so this is basically a win in every way. The array-only objects in the task graph are also faster than what #707 had. Here is Claude's full benchmark results:

SPEC-019 EWA benchmark results

main = origin/main @ 2140cedc; final = ewa-persist-prune @ d440be35.
Suffix /F = persist=False, /T = persist=True (the persist argument of
DaskEWAResampler.resample()). main/T was not run: main's persist path is
broken (SPEC-018).

• cold = first resample() + compute() on a fresh resampler (includes precompute); warm = a second resample() + compute() on the same instance.
• vs main/F = main_time / this_time; > 1 is faster.
• ll2cr calls c/w, fornav calls c/w = calls into the Cython functions in the cold / warm phase.
• stack blocks = input row blocks kept in the fornav stack; real/placeholder = fornav tasks that do work vs. tasks skipped by the extent check.
• Every row is bit-identical to its main/F counterpart.
• Areas: a = 4400×3000 East Coast 1 km grid (all 7 granules overlap); b = 1000×1000 Chesapeake 500 m box (2 granules); c = 1000×1000 box straddling the swath's western edge (3 granules). granule = satpy's native 7 input blocks; 10scan = rechunked to 34 blocks.
• Datasets: M15 5376×3200 (750 m), I05 10752×6400 (375 m); inputs persisted in memory before timing. threads scheduler, num_workers=8, 1 warm-up + 5 timed reps, medians shown.

Core matrix — out_chunks=1024 (areas b/c are one 1000×1000 output chunk)

label	dataset	area	input_chunks	cold (s)	warm (s)	vs main/F cold	vs main/F warm	ll2cr calls c/w	fornav calls c/w	stack blocks	real/placeholder	graph_tasks	peak RSS (MB)
main/F	M15	a	granule	0.828	0.825	1.00	1.00	7/7	105/105	7	105/0	283	485
main/F	M15	a	10scan	1.633	1.656	1.00	1.00	34/34	510/510	34	510/0	1351	305
main/F	M15	b	granule	0.403	0.437	1.00	1.00	7/7	2/2	7	7/0	45	426
main/F	M15	b	10scan	0.383	0.399	1.00	1.00	34/34	6/6	34	34/0	217	27
main/F	M15	c	granule	0.425	0.432	1.00	1.00	7/7	3/3	7	7/0	45	262
main/F	M15	c	10scan	0.406	0.408	1.00	1.00	34/34	11/11	34	34/0	217	15
main/F	I05	a	granule	2.818	2.856	1.00	1.00	7/7	105/105	7	105/0	283	2222
main/F	I05	a	10scan	3.246	3.201	1.00	1.00	34/34	510/510	34	510/0	1351	440
main/F	I05	b	granule	1.652	1.626	1.00	1.00	7/7	2/2	7	7/0	45	2112
main/F	I05	b	10scan	1.381	1.416	1.00	1.00	34/34	6/6	34	34/0	217	156
main/F	I05	c	granule	1.643	1.638	1.00	1.00	7/7	3/3	7	7/0	45	2116
main/F	I05	c	10scan	1.351	1.356	1.00	1.00	34/34	11/11	34	34/0	217	187
final/F	M15	a	granule	0.823	0.823	1.01	1.00	7/7	105/105	7	105/0	283	414
final/F	M15	a	10scan	1.253	1.255	1.30	1.32	34/34	510/510	34	510/0	1351	188
final/F	M15	b	granule	0.423	0.419	0.95	1.04	7/7	2/2	7	7/0	45	447
final/F	M15	b	10scan	0.373	0.381	1.03	1.05	34/34	6/6	34	34/0	217	31
final/F	M15	c	granule	0.413	0.418	1.03	1.03	7/7	3/3	7	7/0	45	225
final/F	M15	c	10scan	0.395	0.373	1.03	1.09	34/34	11/11	34	34/0	217	23
final/F	I05	a	granule	2.599	2.591	1.08	1.10	7/7	105/105	7	105/0	283	2264
final/F	I05	a	10scan	3.056	2.961	1.06	1.08	34/34	510/510	34	510/0	1351	475
final/F	I05	b	granule	1.484	1.679	1.11	0.97	7/7	2/2	7	7/0	45	2119
final/F	I05	b	10scan	1.387	1.407	1.00	1.01	34/34	6/6	34	34/0	217	156
final/F	I05	c	granule	1.889	1.803	0.87	0.91	7/7	3/3	7	7/0	45	2112
final/F	I05	c	10scan	1.487	1.471	0.91	0.92	34/34	11/11	34	34/0	217	172
final/T	M15	a	granule	0.780	0.271	1.06	3.05	7/0	42/42	7	42/63	269	730
final/T	M15	a	10scan	1.024	0.590	1.59	2.81	34/0	165/165	34	165/345	1283	336
final/T	M15	b	granule	0.471	0.035	0.85	12.34	7/0	2/2	2	2/0	23	489
final/T	M15	b	10scan	0.418	0.032	0.92	12.64	34/0	6/6	6	6/0	95	59
final/T	M15	c	granule	0.481	0.032	0.88	13.29	7/0	3/3	3	3/0	27	488
final/T	M15	c	10scan	0.406	0.036	1.00	11.21	34/0	11/11	11	11/0	116	90
final/T	I05	a	granule	2.675	0.745	1.05	3.83	7/0	42/42	7	42/63	269	2166
final/T	I05	a	10scan	2.210	0.780	1.47	4.11	34/0	165/165	34	165/345	1283	1260
final/T	I05	b	granule	1.669	0.066	0.99	24.63	7/0	2/2	2	2/0	23	2100
final/T	I05	b	10scan	1.358	0.039	1.02	36.22	34/0	6/6	6	6/0	95	391
final/T	I05	c	granule	1.665	0.065	0.99	25.28	7/0	3/3	3	3/0	27	2100
final/T	I05	c	10scan	1.407	0.051	0.96	26.55	34/0	11/11	11	11/0	116	625

Extended — out_chunks=256 on areas b/c (4×4 output chunks)

label	dataset	area	input_chunks	cold (s)	warm (s)	vs main/F cold	vs main/F warm	ll2cr calls c/w	fornav calls c/w	stack blocks	real/placeholder	graph_tasks	peak RSS (MB)
main/F	M15	b	granule	0.550	0.530	1.00	1.00	7/7	32/32	7	112/0	300	371
main/F	M15	b	10scan	0.999	0.945	1.00	1.00	34/34	96/96	34	544/0	1432	12
main/F	M15	c	granule	0.545	0.536	1.00	1.00	7/7	48/48	7	112/0	300	279
main/F	M15	c	10scan	0.966	0.997	1.00	1.00	34/34	176/176	34	544/0	1432	12
main/F	I05	b	granule	1.669	1.688	1.00	1.00	7/7	32/32	7	112/0	300	2252
main/F	I05	b	10scan	1.805	1.818	1.00	1.00	34/34	96/96	34	544/0	1432	191
main/F	I05	c	granule	1.890	1.928	1.00	1.00	7/7	48/48	7	112/0	300	2254
main/F	I05	c	10scan	2.109	2.078	1.00	1.00	34/34	176/176	34	544/0	1432	254
final/F	M15	b	granule	0.441	0.444	1.25	1.19	7/7	32/32	7	112/0	300	443
final/F	M15	b	10scan	0.668	0.634	1.50	1.49	34/34	96/96	34	544/0	1432	9
final/F	M15	c	granule	0.489	0.478	1.11	1.12	7/7	48/48	7	112/0	300	258
final/F	M15	c	10scan	0.681	0.624	1.42	1.60	34/34	176/176	34	544/0	1432	8
final/F	I05	b	granule	1.706	1.772	0.98	0.95	7/7	32/32	7	112/0	300	2250
final/F	I05	b	10scan	1.624	1.575	1.11	1.15	34/34	96/96	34	544/0	1432	159
final/F	I05	c	granule	1.967	1.888	0.96	1.02	7/7	48/48	7	112/0	300	2212
final/F	I05	c	10scan	1.791	1.771	1.18	1.17	34/34	176/176	34	544/0	1432	235
final/T	M15	b	granule	0.488	0.073	1.13	7.28	7/0	20/20	2	20/12	98	409
final/T	M15	b	10scan	0.423	0.071	2.36	13.27	34/0	37/37	6	37/59	320	66
final/T	M15	c	granule	0.482	0.061	1.13	8.81	7/0	15/15	3	15/33	132	393
final/T	M15	c	10scan	0.460	0.101	2.10	9.91	34/0	37/37	11	37/139	506	113
final/T	I05	b	granule	1.806	0.208	0.92	8.13	7/0	20/20	2	20/12	98	2100
final/T	I05	b	10scan	1.400	0.103	1.29	17.71	34/0	37/37	6	37/59	320	406
final/T	I05	c	granule	1.796	0.192	1.05	10.04	7/0	15/15	3	15/33	132	2100
final/T	I05	c	10scan	1.486	0.131	1.42	15.82	34/0	37/37	11	37/139	506	672

And the human-readable summary:

Headline results (medians, threads/8)

• H0 pass: all 60 rows are bit-identical to main/F for the same key; the empty-area smoke agrees too.
• H1/H2 confirmed: main/F re-runs ll2cr on every compute (warm ≈ cold); final/T warm makes 0 ll2cr calls — warm speedup 2.8–4.1× on the full-overlap area a, 11–36× on the small areas at one output chunk, 7–18× at 4×4 chunks.
• H3 refuted in part: persist=True cold is not slower than non-persist on area a (0.78 vs 0.82 s M15, 2.68 vs 2.60 s I05) — 60 % of block/chunk pairs are pruned even there; vs main/F it is ≈ equal (0.85–1.02×) where the run is ll2cr-bound and up to 2.4× faster where there are many pairs.
• Clipping on the swath-edge area c works: at 256-px chunks fornav calls drop 48 → 15 (granule) and 176 → 37 (10scan) vs main.
• final/F vs main/F: ≈ 1.0× on ll2cr-bound rows, 1.3–1.5× (M15) / 1.1–1.2× (I05) with many pairs. Two I05 area-c rows are 0.87×/0.91×, with their ll2cr-only time (identical Cython) 9–22 % higher — likely host contention (load rose 2 → 5–6 during the runs), but I left them as-is and said so in the spec.
• Graph size is not smaller: graph_tasks identical for final/F vs main/F; the placeholder tasks SPEC-018 §5.5 accepted cost ~0.1 ms each.
• Cost of persist is memory: the persisted ll2cr result adds ~0.5 GB (M15) / 1.1–1.3 GB (I05) RSS on the full-overlap area for the resampler's lifetime.

[djhoese]

Merging because Claude also identified some memory leaks and some optimizations that I want to implement in other PRs. @Zaczero thanks for getting this started and pointing me to the big issues.