This repository contains the source code, data, and scripts supporting our EMSE'26 submission "Scaling Syntactic Breaking Change Analysis in Java with Roseau". Should the submission be accepted, we will finalize this repository and archive it on Zenodo.
The paper evaluates Roseau 0.7.0-SNAPSHOT against japicmp 0.25.4 and Revapi 0.28.4.
This repository is organized as follows. Subdirectories contain specific instructions for running the different tools and scripts.
- jezek-dataset: This directory contains our refined and extended version of Jezek and Dietrich's benchmark (310 cases)
- libraries-dataset: This directory contains the scripts we used to collect popular Java libraries from Maven Central
- results: This directory contains the pre-computed accuracy, performance, and longitudinal results together with Jupyter notebooks presenting the analyses and plots we use in the paper
- benchmark: This directory contains the analysis pipeline, including:
- Running Roseau, Japicmp, and Revapi on the improved accuracy dataset and collecting their accuracy
- Running the JMH benchmarks to measure their runtime performance
- benchmark/walk: the configuration of the longitudinal study (RQ3)
| What | Where | |
|---|---|---|
| RQ1 Accuracy | Static-tool and LLM commands in benchmark | data + notebook: benchmark/analysis |
| RQ2 Performance | java -jar target/benchmarks.jar -rf json in benchmark |
data + notebook: results/bench |
| RQ3 Longitudinal | BatchGitWalker on the git-walk branch of alien-tools/roseau, driven by benchmark/walk/walk.yaml |
data + notebooks: results/longitudinal/walk/notebooks |
| RQ3 Preliminary study | uv run python measure_build_times.py in benchmark/walk/build-times |
data: library-build-times.csv, notebook: build_times.ipynb |
JDK 25 and Maven 3.9+ for the Java pipeline, and uv for the Python notebooks and scripts. Reproducing the LLM measurements also requires OpenAI and Anthropic API keys.
All Python code (notebooks in results and benchmark/analysis, scripts in libraries-dataset) lives in a single uv workspace rooted at this directory. There is one pyproject.toml per component and a single uv.lock / .venv at the root, so every component resolves to the same dependency versions.
uv sync # create ./.venv with every component's dependencies (run once, from the repo root)
uv run jupyter lab # open the notebooks (works from the repo root or any component directory)
uv run python parse_mvn_top_repo.py # run a script (from its own directory)Python 3.13 is pinned in .python-version; uv installs it automatically if missing. To add a dependency, edit the relevant component's pyproject.toml and re-run uv sync.