Oversubscribe test forks in CI to cut wall clock time

[timtebeek]

Why

:test dominates CI. On the most recent main build (9m47s total):

Phase	Duration
Runner setup, checkout, JDK, Gradle setup	~15s
Gradle config	~13s
:compileJava	48s
:jar / :javadoc / :sourcesJar	~5s
:recipeCsvValidate	20s
:compileTestJava	10s
:test	8m09s

Eliminating every task except :test would save only ~1.5 min, so :test is the only phase worth attacking.

The problem

RewriteJavaPlugin sets maxParallelForks = availableProcessors(), and ubuntu-latest is 4 vCPU, so 4 forks. Their finish times:

Executor 2: 3m47s
Executor 4: 4m55s
Executor 5: 7m27s
Executor 3: 8m07s   <- wall clock is set by this one

That's 24m16s of executor time. Evenly packed across 4 forks it would be 6m04s, so roughly two minutes is fork imbalance alone.

Gradle hands out whole test classes, and 47 of the 267 test classes resolve Maven poms over the network. A couple of long, latency-bound classes landed late on two forks, and there was no work left for the other two to steal. There's no single slow test to fix — 1778 tests at ~1.1s of executor time each, largest inter-test gap 41s and tapering smoothly.

The change

Run 1.5 forks per core in CI. Finer granularity packs better, and the CPU-bound classes can use the cores that the network-bound classes are blocking on.

Heap drops to 1500m so the extra forks fit: 6 x 1500m is ~9g of a 16g runner, where 6 x 2g would have been tight. GitHub scales runner RAM at 4g per core, so 1.5 forks/core at 1500m stays in budget on larger runners too.

Local dev is untouched — the plugin's availableProcessors() / 2 and the 2g heap still apply when CI is unset. Verified both paths resolve as intended.

Not done here

Moving to an 8-core runner is the bigger lever (:test to ~4 min, total ~5m30s), but runs-on: ubuntu-latest is hardcoded in openrewrite/gh-automation/.github/workflows/ci-gradle.yml and needs an input added there. Happy to follow up if we want it.

Sharding across matrix jobs isn't worth it — each shard re-pays ~1m40s of setup and compile, and since shards start simultaneously they all miss the remote build cache, landing around 4 min for far more complexity.

[timtebeek]

Marking draft — the first CI run showed this change is a no-op on parallelism, and I want to fix it properly before this merges.

maxParallelForks is capped by org.gradle.workers.max

Gradle's worker-lease system bounds total concurrent workers, including test executors, by org.gradle.workers.max, which defaults to the core count. Setting maxParallelForks = 6 on a 4-core runner silently gets forced back to 4. Gradle says so directly — reproduced locally:

:test.maxParallelForks (21) is larger than max-workers (2), forcing it to 2

The CI run confirms it: still only 4 executors (IDs 1–4), same as baseline. The only change that actually took effect was the heap drop to 1500m.

The run took 27m38s, but not because of this change

16 minutes of that was the build-cache upload to Develocity hanging:

10:01:33  I/O exception (java.net.SocketException) caught when processing request
          to {s}->https://community.develocity.cloud:443: Connection timed out

Tests finished at 09:45:42; the build sat on that socket until it timed out at 10:01:33. Unrelated to this PR, but worth knowing it can happen — when it does it dwarfs every optimization discussed here.

Actual test numbers

	baseline	this run
executor finish times	3m47, 4m55, 7m27, 8m07	4m42, 4m42, 5m23, 10m07
total executor time	24m16s	24m54s
:test wall clock	8m09s	10m07s

Same total work and same 4 forks, but the tail moved from 8m07 to 10m07 purely on which fork drew the slow classes. Run-to-run variance is ±2 min, which is the same size as the effect I was trying to measure — so any future attempt here needs several runs, not one.

Where that leaves it

Oversubscribing forks requires also raising org.gradle.workers.max, and the only place this repo can set that is gradle.properties, which would apply to local dev too. Meanwhile the 8-core runner raises availableProcessors() and so lifts both limits together, with no build-script change at all — that now looks clearly like the better lever.

[timtebeek]

Closing — this change cannot do what it set out to do.

maxParallelForks is silently capped by org.gradle.workers.max, which itself defaults to the core count, so raising forks from tasks.test { } alone is a no-op. Gradle states it outright:

:test.maxParallelForks (21) is larger than max-workers (2), forcing it to 2

The CI run confirmed it: still 4 executors, unchanged from baseline.

Raising the core count lifts both limits at once and needs no build-script change, so the fix belongs in the shared workflow instead: openrewrite/gh-automation#111 adds a runs_on input (defaulting to ubuntu-latest, so it is a no-op until a repo opts in).

Two findings from the investigation worth keeping:

• Run-to-run variance is ±2 min. Two runs of the identical 4-fork configuration gave :test times of 8m09s and 10m07s, purely on which fork drew the slow classes. That is the same magnitude as most effects worth chasing here, so future timing work needs a median of several runs.
• The remote build cache can hang. One run spent 15m48s blocked on a community.develocity.cloud:443 upload before timing out, turning a ~10 min build into 27m38s. Harmless when rare, but it dwarfs every optimization discussed here when it happens.