Skip to content

Add LeRobot support (convert, train, infer, review) + SmolVLA, Train tab, config builder and settings page - #5

Merged
pascd merged 8 commits into
INESCTEC:mainfrom
pascd:adding-lerobot
Nov 3, 2026
Merged

pascd merged 8 commits into
INESCTEC:mainfrom
pascd:adding-lerobot

Conversation

@pascd

@pascd pascd commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds LeRobot as a second end-to-end format next to robomimic: convert recordings to a LeRobot dataset, train ACT / Diffusion / SmolVLA from the new Train tab (or relay-d-train), run the checkpoint with lfd-inference-node, and browse datasets in Review. Verified against LeRobot 0.6.1.

Added

  • LeRobot inference backend (dispatch/lerobot_runner.py); --backend auto picks robomimic or LeRobot from the checkpoint. The custom backend (model_runner.py) is removed.
  • SmolVLA policy (fine-tuned from lerobot/smolvla_base) with a task sentence at inference (--task, default from the dataset).
  • Train tab and relay-d-train: start/list/stop robomimic and LeRobot runs; per-format environments under ~/.relayd/envs/ built by a "Set up environment" button.
  • Config page "Build from ROS": build acquisition_config.yaml from the live ROS graph, with a shared config check.
  • Settings page generated from one table (utils/user_settings.py), also read by the CLI tools.
  • Review tab opens LeRobot datasets (view-only).
  • Headless converter relay-d-convert.
  • Safe stop: stop: [...] in action_config.yaml, sent once when inference stops.
  • Recorded-rate check: demos >10% off recording_frequency fail validation.
  • CI: ROS-dependent tests run in a ros:jazzy container.

Changed

  • Absolute training actions are now next-step targets (action[t] = value[t+1]).
  • LeRobot policies run at their training fps by default.
  • Output folder names: _Recording, _Robomimic, _Lerobot.
  • Record page Stop saves in a background thread.
  • Exported inference_script.py is a thin wrapper around lfd-inference-node.
  • App logo shrunk (3.9 MB → 11 KB); resources_rc.py loses ~250k lines.

Fixed (highlights)

  • Converter no longer guesses missing paths or stretches mis-sized datasets.
  • AppAPI.shutdown() no longer loses an unsaved recording.
  • Dispatcher: bad mappings / in_min == in_max scales stop startup instead of failing at runtime.
  • Joint states with missing joints are ignored instead of shifting values.
  • TF_OLD_DATA warning spam filtered.
  • Recording: no demo overwrites; point clouds count towards the buffer limit.
  • Robomimic StopIteration on small datasets.
  • pyarrow<21 so LeRobot conversion works with ROS Jazzy's NumPy 1.26.

Full list in CHANGELOG.md → [Unreleased].

Upgrade notes

  • Re-convert datasets and retrain: old datasets taught the policy to hold still.
  • Press Set up lerobot environment again (adds lerobot[smolvla]).
  • relay_d.dispatch.DispatcherHelpers removed → use save_dispatch_templates(output_dir).
  • lfd-inference-node --hz defaults to the LeRobot dataset fps; for robomimic pass --hz <recording_frequency>.
  • Action configs copied from the template: robotiq_gripper must map position: 0, not 7.
  • Removed: scipy dependency, dispatch extra, --model-type, --obs-keys, --action-horizon, infer_backend: custom.

Test plan

  • pytest (ROS-free) passes
  • pytest with ROS sourced passes (CI ros:jazzy job)
  • Record → Post-process (LeRobot) → Train (ACT) → lfd-inference-node on the robot
  • Same with robomimic
  • SmolVLA fine-tune + inference with --task

pascd added 8 commits October 8, 2026 09:01
…ve container management

- Updated replay_parity.py to allow dataset and checkpoint paths to be set via environment variables.
- Added error handling for required arguments in replay_parity.py.
- Modified pyproject.toml to relax numpy version constraints.
- Improved Qt application initialization in app.py by sanitizing the Qt environment.
- Refactored topic_containers.py to clean up comments and improve readability.
- Replaced CvBridge usage with custom image conversion functions in container_visualizer.py and data_recorder.py.
- Introduced container_manager.py to encapsulate container lifecycle management in the record page.
- Enhanced demo_playback_reconstructor.py to use custom image message conversion.
- Updated obs_builder.py to remove CvBridge dependency and utilize custom image decoding.
- Added qt_env.py to manage OpenCV and PyQt5 environment conflicts.
…ties

- Implement tests for output-format buttons to ensure single-choice toggle behavior.
- Create end-to-end tests for converting recorded HDF5 data to LeRobotDataset format.
- Add tests for the LeRobotDataset writer to validate data integrity and structure.
- Introduce tests for the shared logger to verify output format and level control.
- Develop tests for offline configuration loading to handle various scenarios and errors.
- Implement tests for post-process options to ensure correct handling of configuration files.
- Add tests for the RecordingController to validate button states and recording logic.
- Introduced tests for settings application, ensuring correct handling of ROS domain IDs and startup page defaults.
- Added tests for settings consumers to validate that user settings are correctly applied and overridden by YAML configurations.
- Implemented tests for the settings page, verifying UI interactions, saving behavior, and reset functionality.
- Created tests for the training CLI, covering argument parsing, dry runs, and error handling.
- Developed tests for the train page, focusing on dataset loading, parameter editing, and run initiation.
- Added extensive tests for the training launcher, validating dataset detection, command generation, and run management.
- Included tests for user settings to ensure defaults, validation, and migration are functioning as expected.
- Implemented `run_in_background` function for executing tasks in a separate thread while updating the GUI.
- Created `reexec.py` to handle re-execution of the inference node in the correct Python environment.
- Developed `envsetup.py` for managing dedicated Python environments for LeRobot and robomimic, including installation scripts and status checks.
- Introduced `tf_filter.py` to filter out stale TF transforms and log warnings for ignored transforms.
- Added unit tests for the configuration builder page, including saving and loading configurations, handling errors, and validating configurations.
- Implemented tests for environment setup scripts, ensuring they handle existing environments and validate bash syntax.
- Created tests for the first run folder selection and ROS catalog functionality, ensuring proper topic classification and message field extraction.
- Added tests for user settings persistence and validation of configuration drafts, ensuring correct behavior of the settings page.
- Included tests for URDF limits parsing and validation, ensuring correct extraction of joint limits and error handling.
…g and validation

- Added _check_mapping method in ActionDispatcher to validate message mappings during initialization.
- Enhanced dispatch logic to prevent sending non-finite values after transformations.
- Updated ObsBuilder to utilize new utility functions for input synchronization and trigonometric operation stripping.
- Introduced message encoding module to handle various ROS message types for DataRecorder.
- Improved environment setup scripts to manage process states and prevent concurrent builds.
- Added tests for new functionalities in ActionDispatcher, ObsBuilder, and message encoding.
- Added validation checks for normalized actions in test_config_draft.py.
- Updated joint series saving logic to include empty rows in test_data_recorder_save.py.
- Improved action template mappings and inference script execution in test_dispatch_templates.py.
- Adjusted package list assertions in test_envsetup.py to include new dependencies.
- Enhanced lerobot conversion tests to handle broken files and missing action streams in test_lerobot_conversion.py.
- Added checks for task sentence handling in batch building in test_lerobot_runner.py.
- Introduced new tests for action generation and mask splitting in test_numerics.py.
- Updated postprocess options tests to ensure output folder naming aligns with dataset format in test_postprocess_options.py.
- Improved recording controller tests to handle save failures and retries in test_recording_controller.py.
- Enhanced session folder naming tests to include recording suffix in test_settings_consumers.py.
- Added tests for fine-tuning SmolVLA from pretrained models in test_training_launcher.py.
- Introduced new tests for AppAPI session handling and demo validation in test_app_api.py and test_demo_validator.py.
- Added tests for inference session handling of stale inputs and rate warnings in test_inference_session.py.
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


pascd seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@pascd
pascd merged commit 7ed6a5e into INESCTEC:main Nov 3, 2026
4 of 5 checks passed
@pascd
pascd deleted the adding-lerobot branch November 3, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants