Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
cae8e47
Introducing yaml spec for rose api
AymenFJA Jun 12, 2026
b7bd9ac
support config learner/task
AymenFJA Jun 12, 2026
ed0ad86
docs, tests and schema expansion
AymenFJA Jun 16, 2026
846d33f
major docs refine
AymenFJA Jun 18, 2026
fae7af7
fix Spec API and docs inconsistency
AymenFJA Jun 18, 2026
59990d3
improve docs
AymenFJA Jun 18, 2026
44512ff
This commit:
AymenFJA Jun 19, 2026
fa09051
fix pre-commit
AymenFJA Jun 19, 2026
8c30b1e
adding missing init for tests
AymenFJA Jun 20, 2026
7cce508
update rose with new orbit
AymenFJA Jul 6, 2026
85f2f17
leftovers
AymenFJA Jul 13, 2026
a4b3f0a
rename rose/al,rl,uq to descriptive package names; add rose run CLI
AymenFJA Jul 13, 2026
2556dbd
pre-commit
AymenFJA Jul 13, 2026
3c60d65
implement `rose run --remote` (spec-driven remote execution via ORBIT)
AymenFJA Aug 18, 2026
7e07df8
pre-commit
AymenFJA Aug 18, 2026
3a47c55
ruff
AymenFJA Aug 18, 2026
0a374f8
adding pre-commit to tox
AymenFJA Aug 19, 2026
1ae6633
another attempt to fix pre-commit
AymenFJA Aug 19, 2026
d6ff2fe
adding orbit as dep
AymenFJA Aug 19, 2026
8bb05e0
reorganize remote execution, make HPC wait configurable, rework docs
AymenFJA Aug 31, 2026
4c22af7
cleanup docs
AymenFJA Aug 31, 2026
9811068
adding examples and renaming package
AymenFJA Sep 1, 2026
c6af76d
cleanup
AymenFJA Sep 1, 2026
2cc81ca
ruff examples
AymenFJA Sep 1, 2026
22c4e77
Merge pull request #97 from radical-cybertools/feature/yaml-spec
AymenFJA Sep 1, 2026
3ce6e2d
adding release changelog
AymenFJA Sep 1, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 42 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

---

## [0.3.0] - 2026-03-05
## [0.3.0] - 2026-09-01

### Added
- **`IterationState.learner_id`**: New field (`int | str | None`, default `None`) on `IterationState`
identifying which parallel learner produced a given state. Integer index for
`ParallelActiveLearner` and `ParallelReinforcementLearner`; learner name string for
`ParallelUQLearner`.
- **`rose run --remote`**: spec-driven remote execution against an ORBIT broker, driven
entirely by a `remote:` block in the workflow YAML (`remote.target`, `remote.broker_url`) —
no separate manual endpoint-bootstrap step required.
- **`remote.embedded`**: host the ORBIT broker in-process instead of requiring a standalone
broker deployment.
- **`rose setup`**: interactive first-time wizard that provisions broker TLS cert/token and
IRI/SFAPI credentials, then verifies everything end-to-end by bringing up a real ORBIT
endpoint — ending with a ready-to-paste `remote:` block on success.
- **Configurable HPC wait timeout**: `rose setup --wait-timeout SECONDS` and the spec's
`remote.target.endpoint_timeout_min` control how long ROSE waits on HPC job/endpoint
registration; a warning is printed with the default used when neither is set.
- **CLI reference docs**: a new Command-Line Interface page documenting `rose run` and
`rose setup` and their flags.
- **Two canonical example specs** under `examples/spec/` — `sequential_python` (Python
tasks, sequential learner) and `parallel_shell_learners` (shell tasks, parallel
learners) — each with a local (`workflow.yaml`) and a `--remote` (`workflow-remote.yaml`)
variant, together covering Python/shell, local/remote, and sequential/parallel.

### Changed
- **Unified async-iterator API for parallel learners**: `ParallelActiveLearner.start()`,
Expand All @@ -27,6 +44,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Shared `_stream_parallel` helper**: The internal `asyncio.Queue`-based fan-in pattern
is extracted into a single module-level async generator in `rose/learner.py`, eliminating
identical code that was previously duplicated across all three parallel learner classes.
- **`rose/bootstrap.py` and `rose/remote.py` reorganized into a `rose/remote/` package**
(`rose/remote/bootstrap.py`, `rose/remote/execute.py`), matching the existing
`active_learning`/`reinforcement_learning`/`spec` subpackage convention.
`from rose.remote import run_remote, run_setup_wizard` continues to work unchanged via
the package's re-export.
- **Default `--local` execution backend** switched from the deprecated
`DragonExecutionBackendV3`/`dragon_v3` to `DragonExecutionBackend`/`dragon` (rhapsody
deprecated the former). Also fixed `--backend`'s help text, which incorrectly stated the
default was `concurrent`.
- **PyPI distribution renamed** from `ROSE` to `rose-surrogate-explorer` (the `ROSE` name was
already taken on PyPI by an unrelated package). `import rose` and the `rose` CLI command
are unaffected. Added `license`/`classifiers` metadata to `pyproject.toml` for the first
PyPI release.
- **Documentation site restyled** to match RHAPSODY's mkdocs-material conventions (flat
header/tabs, forced heading color, pill buttons, `pymdownx.highlight`) while keeping
ROSE's rose/pink brand palette.

### Fixed
- `typeguard` (used directly by `rose/learner.py`'s `@typeguard.typechecked`) is now an
explicit dependency instead of only being available transitively through
`radical.asyncflow`/`rhapsody-py`.
- Several broken internal documentation anchor links in `spec-api.md`/`rose-aas.md`
(double-hyphen slugs that never matched mkdocs' actual generated anchors).
- A broken inline `<figure>` positioning hack on the docs homepage.

### Deprecated
- **`ParallelActiveLearner.teach()`**, **`ParallelReinforcementLearner.learn()`**, and
Expand Down
11 changes: 5 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,16 +27,15 @@ For tutorials and walkthrough notebooks please check [here](examples)
import asyncio

from rose.metrics import MEAN_SQUARED_ERROR_MSE
from rose.al.active_learner import SequentialActiveLearner
from rose.al import SequentialActiveLearner

from concurrent.futures import ProcessPoolExecutor

from radical.asyncflow import WorkflowEngine
from rhapsody.backends import RadicalExecutionBackend
from rhapsody.backends import ConcurrentExecutionBackend

async def main():
execution_engine = await RadicalExecutionBackend(
{'runtime': 30,
'resource': 'local.localhost'}
)
execution_engine = await ConcurrentExecutionBackend(ProcessPoolExecutor())

asyncflow = await WorkflowEngine.create(execution_engine)
acl = SequentialActiveLearner(asyncflow)
Expand Down
3 changes: 3 additions & 0 deletions _typos.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[default.extend-words]
# "aas" appears in rose-aas.md / "ROSE as a Service" links — not a typo of "as"/"ass".
aas = "aas"
73 changes: 73 additions & 0 deletions docs/getting-started/cli.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Command-Line Interface

Installing ROSE (`pip install .`) installs the `rose` command. It has two
subcommands: `run`, for executing a workflow YAML, and `setup`, an
interactive wizard for getting remote HPC execution working for the first
time.

```bash
rose --help
```

## `rose run`

```bash
rose run <yaml> --local [--backend NAME]
rose run <yaml> --remote
```

`<yaml>` is the path to a ROSE workflow spec. Exactly one of `--local` or
`--remote` is required — passing both, or neither, is an error.

- **`--local`** runs the workflow on this machine via a
[rhapsody](https://github.com/radical-cybertools/rhapsody) backend.
`--backend NAME` selects which one (default: `dragon`).
- **`--remote`** runs the workflow on a remote HPC target through the ORBIT
broker, spec-driven — no separate bootstrap step. It requires a
`remote.target` block in the YAML; see [ROSE as a Service](../user-guide/rose-aas.md)
for the concept and
[`remote.target`](../user-guide/spec-api.md#remotetarget-bootstrapping-the-endpoint-for-rose-run-remote)
for the field reference. If `remote.target` is missing, `rose run
--remote` exits immediately with a message telling you where to add it.

## `rose setup`

```bash
rose setup [--wait-timeout SECONDS]
```

An interactive, first-time wizard that gets `rose run --remote` working
end to end. It asks only simple questions — facility, account, names — and
handles everything technical itself: broker TLS cert/token setup, IRI/SFAPI
credentials, and the remote Python environment.

You choose one of two modes:

- **IRI / SFAPI** — everything runs from this computer; no SSH to the HPC
system needed. Works today for NERSC and OLCF.
- **PsiJ** — jobs submit directly from an HPC login node you're already
on; works on any SLURM/PBS system, but the wizard needs to run there.

The wizard verifies the setup by actually bringing up a real ORBIT
endpoint on the target resource and waiting for it to register — the same
code path a real `rose run --remote` takes, not an approximation. On
success it prints a ready-to-paste `remote:` YAML block and the exact
follow-up command, `rose run your_workflow.yaml --remote`.

`--wait-timeout SECONDS` controls how long the wizard waits for HPC jobs
(the environment check and the endpoint-verification job) to come up.
Default is 300 seconds; if you omit the flag, the wizard prints a note
saying it's using the default and how to change it. On a busy queue, raise
it: `rose setup --wait-timeout 900`.

See [ROSE as a Service](../user-guide/rose-aas.md) and
[`remote.embedded`](../user-guide/spec-api.md#remoteembedded-run-without-a-standalone-broker)
for what the resulting `remote:` block means.

## Exit codes

| Code | Meaning |
|------|---------|
| `0` | Success. |
| `1` | General failure — `rose setup` didn't complete, or `rose run --remote` is missing `remote.target`. |
| `2` | Argument error — `--local`/`--remote` both given or neither given. |
2 changes: 1 addition & 1 deletion docs/getting-started/dry-run.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import os
import sys
import asyncio

from rose.al.active_learner import SequentialActiveLearner
from rose.al import SequentialActiveLearner

from radical.asyncflow import WorkflowEngine

Expand Down
3 changes: 3 additions & 0 deletions docs/getting-started/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ Refer to the [Installation Guide](installation.md) for detailed instructions bas
A dry run allows you to simulate the execution of your tasks locally without making any changes to the behavior of your ML workflow.
It's a way to verify your setup is working on HPC before applying it to prevent time and resource waste.

!!! tip
See the [Dry Run](dry-run.md) guide for a full step-by-step walkthrough.

## Q3: What if I have an issue?
If you have any additional questions or an issue, feel free to reach out to our support team by opening a Github [ticket](https://github.com/radical-cybertools/ROSE/issues).

Expand Down
22 changes: 16 additions & 6 deletions docs/getting-started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ machines, please refer to the following link: [RADICAL-Pilot Supported HPC Machi
## For Linux and macOS 🐧

1. Clone the latest version from the [official website](https://github.com/radical-cybertools/ROSE).
```
```bash
git clone https://github.com/radical-cybertools/ROSE.git
```
2. Run the following commands to install ROSE and its dependencies:
Expand All @@ -21,19 +21,19 @@ machines, please refer to the following link: [RADICAL-Pilot Supported HPC Machi
1. Download the Windows WSL installer from the [official website](https://apps.microsoft.com/detail/9pdxgncfsczv?hl=en-US&gl=US).
2. Setup you WSL user name and password.
3. Make sure you have Python 3.9 or higher in your WSL as follows:
```
```bash
python --version
```
4. create new pip virtual env:
```
```bash
python3 -m venv rose_env
```
5. Activate the env:
```
```bash
source rose_env/bin/activate
```
6. Clone the latest version from the [official website](https://github.com/radical-cybertools/ROSE).
```
```bash
git clone https://github.com/radical-cybertools/ROSE.git
```
7. Run the following commands to install ROSE and its dependencies:
Expand All @@ -42,4 +42,14 @@ machines, please refer to the following link: [RADICAL-Pilot Supported HPC Machi
pip install .
```

If you encounter any issues, refer to the [Issues Section](https://github.com/radical-cybertools/ROSE/issues).
## Verify Your Install

```bash
rose --help
```

This should print the `rose` command's usage. See the [Command-Line Interface](cli.md)
page for what `rose run` and `rose setup` do.

!!! note
If you encounter any issues, refer to the [Issues Section](https://github.com/radical-cybertools/ROSE/issues).
6 changes: 3 additions & 3 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## What is ROSE? 🌹

ROSE: RADICAL Orchestrator for Surrogate Exploration (ROSE) toolkit is a framework designed to enable the concurrent and adaptive execution of simulation, surrogate training, and selection tasks on High-Performance Computing (HPC) resources. ROSE is a Python package that provides tools for developing active and reinforcement learning (AL and RL) methods for scientific applications. It enables users to define simulation and surrogate training tasks and automatically manage their execution on HPC resources via a predefined set of Learning Policies (Learners).
ROSE: RADICAL Orchestrator for Surrogate Exploration (ROSE) toolkit is a framework designed to enable the distributed, concurrent and adaptive execution of simulation, surrogate training, and selection tasks on High-Performance Computing (HPC) resources at a scale. ROSE is a Python package that provides tools for developing active and reinforcement learning (AL and RL) methods for scientific applications. It enables users to define simulation and surrogate training tasks and automatically manage their execution on HPC resources via a predefined set of Learning Policies (Learners).

ROSE also includes tools to facilitate the selection of the most effective surrogate model for a given simulation based on performance metrics.

Expand All @@ -9,7 +9,7 @@ ROSE leverages [**RADICAL-Cybertools**](https://radical-cybertools.github.io), a

## Why ROSE? 🚀🚀🚀
ROSE allows you to enable, scale, and accelerate your learning workflows across thousands of CPU cores and GPUs effectively and efficiently with just a few lines of code.
ROSE is built on the [**RADICAL-AsyncFlow**](https://radical-cybertools.github.io/radical.asyncflow/) and [**RADICAL-Pilot**](https://github.com/radical-cybertools/radical.pilot) runtime system, a powerful execution engine that enables the distributed execution of millions of scientific tasks and applications such as executables, functions and containers effortlessly.
ROSE is built on the [**RADICAL-AsyncFlow**](https://radical-cybertools.github.io/radical.asyncflow/) and [**RHAPSODY**](https://radical-cybertools.github.io/rhapsody/landing.html) runtime system, a powerful execution engine that enables the distributed execution of millions of scientific tasks and applications such as executables, functions and containers effortlessly.

**Preemption-safe on HPC.** Every completed iteration is written to disk before the next one starts. If the job is killed mid-run, all completed iterations are already on disk — inspect them, resume from the last checkpoint, and never rerun a finished iteration.

Expand Down Expand Up @@ -40,7 +40,7 @@ ROSE is built on the [**RADICAL-AsyncFlow**](https://radical-cybertools.github.i

- **Heterogeneous Task Execution** on GPUs, CPUs, with MPI and/or sequential tasks.

<figure markdown="span" style="position: relative; left: 600px; top: -160px">
<figure markdown="span">
<img src="assets/nsf_logo.png" alt="Image title" width="120">
<figcaption>NSF Funded Project (#2212550)</figcaption>
</figure>
4 changes: 4 additions & 0 deletions docs/integrations/clearml.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,10 @@ For **parallel learners**, `state.learner_id` is included automatically. The tra
each state as a separate `series` inside the same scalar title, making per-learner curves
directly comparable without any user code.

!!! note
No user code is required to get this overlay — `state.learner_id` is already populated
by the parallel learner framework before the tracker ever sees the state.

### Task tags — logged in `on_stop`

| ClearML tag | Value |
Expand Down
Loading
Loading