Finish hklpy2 migration, add 4-ID-G fly scanning, fix startup/logging/SPEC bugs

[gfabbris]

Summary

Beamline-driven changes accumulated over the 2026-2 run at 4-ID-G, plus the
fixes they forced. Three main threads:

1. Complete the hklpy → hklpy2 migration for the POLAR diffractometers and
   the HKL user utilities.
2. Add 4-ID-G piezo fly-scanning (SoftGlue Zynq waveform driving + Eiger /
   position-stream integration).
3. Fix startup, logging, and SPEC-output bugs hit while running.

Closes #70. Closes #71.

---

1. hklpy2 migration finished

polar_diffractometer_hklpy2.py is now the real polar_diffractometer.py; the
old gi/hkl-based implementation is kept as
polar_diffractometer_obsolete.py for reference. hkl_utils_hklpy2.py is
likewise renamed to hkl_utils.py. devices.yml, _common_startup.py,
id4_raman/startup.py, utilities.py, and the docs pages follow the rename.

Behind the rename:

• The four geometries are built with hklpy2.diffractometer_class_factory
  (hkl_soleil / APS POLAR) plus a shared DiffractometerMixin carrying the
  table, area-detector, filter, slit, and analyzer components.
• DeferredEpicsMonochromatorRO — hklpy2's DiffractometerBase.__init__
  reads beam.wavelength/beam.energy while seeding the solver, which with a
  plain EpicsSignalRO blocks for 60 s and breaks make_devices(connect=False)
  when the VDCM IOC is off. The deferred signals return a sane fallback until
  EPICS connects, so the two-phase startup still works with a dead IOC.
• UBMatrixSignal (_ub_sync) makes sample.UB a subscribable ophyd signal,
  so Sync_UB_Matrix can mirror UB (and the h2/k2/l2 azimuthal extras and
  simulated motor positions) from the main geometry to its _psi companion.
• Real/pseudo positioners are set to Kind.config | Kind.normal, and
  auxiliary_axis_names drops nested PseudoPositioner sub-devices — hklpy2's
  wh(full=True)/pa() call round(component.position, ...), and .ana
  returns a namedtuple with no __round__.

hkl_utils.py user-facing changes

• read_diffractometer_config_scan() replaces restore_huber_from_scan(),
  which was still on the hklpy run_orientation_info / restore_sample /
  restore_reflections path. It now uses hklpy2.run_utils.get_run_orientation
  and diffractometer.restore(..., restore_samples=True, restore_extras=True, restore_constraints=True), prompts when the scan holds a different
  diffractometer, and recomputes UB. Closes Restoring hklpy2 setup from scan not working #71.
• write_config / read_config renamed to
  write_diffractometer_config_file / read_diffractometer_config_file;
  both file and scan restore share a new _prompt_clear_mode() overwrite-vs-append prompt.
• reset_constraints() fixed — it called _geom_.reset_constraints() /
  _geom_.show_constraints(), neither of which exists on an hklpy2
  diffractometer.
• New pa_new(): POLAR-specific full printout (orienting reflections with
  their wavelengths, real and reciprocal lattice, U and UB, azimuthal
  reference, constraints, mono energy/wavelength/d-spacing).
• New analyzer_get(); ca() takes an optional energy.

Analyzer calibration

AnalyzerDevice.set_energy() now calibrates the tth_trans offset alongside
th, calc() prints to 4 decimals and restores both offsets on release, and
the new convert_energy_to_tth_pseudo() / ath_reset_offset() support that.
The analyzer moved to pseudo-motor (th_motor, tth_trans) addressing.

2. Fly scanning at 4-ID-G

• plans/flyscans.py — new flyscan() plan: 2D snake over the two Jena
  piezo axes, fast X and slow Y driven from SoftGlue RAM waveforms through
  DAC1/DAC2. The module docstring documents the DAC-swing/modulation-summing
  geometry and the position-unit convention.
• devices/softgluezynq_g_new.py — new SoftGlueZynq device for the ISN
  setup: UpDown counter, pulse-train generator, per-interferometer trackers,
  dual-RAM components, DAC1 mux, threshold trigger, plus the snake-waveform
  builders. Registered as gsgz_isn; coexists with the existing
  softgluezynq_g and reuses the softgluezynq_parts block library.
• Eiger — external trigger switched from External Gate to External
  Enable (gate is for pump-probe); new setup_flyscan_mode(); stage_sigs
  pop of enable made conditional so re-staging does not raise.
• pos_stream (PositionStreamDevice) registered in devices.yml and
  given CountersMixin with a no-op plot API so it slots into the counters
  machinery without offering fake plot channels.
• piezo_jena_setup() in shorts.py to check/toggle the per-axis modulation
  input that gates the FPGA analog drive.

3. Bug fixes

• Cannot start from /gdata (Cannot start bluesky from gdata #70). apsbits resolves its log directory to
  <cwd>/.logs at import time; at the beamline the cwd is the read-only DM
  experiment directory, so os.makedirs() raised PermissionError and aborted
  the import before we could redirect it. The apsbits import now runs from a
  private temp dir, and the fallback path checks os.access(cwd, os.W_OK)
  before choosing <cwd>/.logs. Closes Cannot start bluesky from gdata #70.
• Stale-handle log spam. Concurrent sessions shared one
  RotatingFileHandler on NFS, so one process rotating the file others held
  open produced [Errno 116] Stale file handle. Logs are now per-session
  (logging.<host>.<pid>.log, ipython_logs.<host>.<pid>.log), and pymongo —
  whose monitor thread emits a heartbeat DEBUG record every few seconds — is
  pinned to WARNING.
• SPEC files unreadable by pymca. Non-numeric columns (string HDF5 paths,
  arrays) were substituted inline and reported on a #U line written between
  every data row, breaking specfile column detection. They are now dropped from
  #L entirely so the data block stays a contiguous numeric matrix. Covered by
  the new tests/test_spec_writer.py (labels, #N/column-count consistency,
  block contiguity).
• te() wrote to the controller device instead of its .setpoint.
• Dichro/lockin with a scaler — stage_dichro_wrapper resolved hinted
  fields via oregistry.find(name.replace("_", ".")), which does not work for
  scaler channels; it now goes through det.channels_name_map.
• experiment_setup / change_sample default reset_scan_id to None
  (prompt) instead of silently keeping the old scan id.
• _setup_file_io warns when a detector intends to save images but
  predict_save_path() returns None, instead of silently dropping it from
  the NeXus externals link.
• ringlight removed from the baseline; MAX_IMAGES for the Xspress3 lowered
  to 12000; fetch_shift_events default 3 → 5.

4. Docs / repo housekeeping

CLAUDE.md split from 326 lines into 136 lines of core guidance plus five
on-demand detail docs under .claude/docs/ (configuration, startup flow,
device loading, data output & logging, documentation site), which .gitignore
now allows through. A separate commit corrects stale claims that audit turned up.

Testing

• pytest — 62 passed (needs a real apstools; the shared conftest stub makes
  the new SPEC-writer test error at collection otherwise).
• ruff check / ruff format --check clean on every file this branch touches.
  Two pre-existing issues on main are deliberately untouched: import ordering
  in id4_raman/startup.py (startup import order is load-bearing) and
  formatting in peak_position_legacy.py.
• Fly scanning, the analyzer calibration, and the diffractometer migration were
  exercised on the real hardware at 4-ID-G; they have no unit coverage.

Known gaps

• VortexXspress34.setup_flyscan_mode() is a stub — the docstring describes
  num_images / acq_time / hdf_images, but the body only calls
  setup_external_trigger() and ignores all three arguments. The Eiger
  equivalent is implemented.
• Related open issues this moves toward without closing: Add function to reset the HKL settings #73 (HKL reset) and
  Printing analyzer setup in pa #47 (analyzer info in pa) — commented on each with the specifics.

🤖 Generated with Claude Code