Skip to content

Exit ephemeral runners on broker acknowledge job-not-found#4540

Open
rentziass wants to merge 5 commits into
mainfrom
rentziass-fix-ephemeral-runner-ack-404-exit
Open

Exit ephemeral runners on broker acknowledge job-not-found#4540
rentziass wants to merge 5 commits into
mainfrom
rentziass-fix-ephemeral-runner-ack-404-exit

Conversation

@rentziass

@rentziass rentziass commented Jul 6, 2026

Copy link
Copy Markdown
Member

Summary

  • classify broker acknowledge 404 / "Job not found" responses so the runner can recognize them explicitly
  • exit ephemeral runners cleanly when acknowledge hits that terminal condition
  • preserve the existing best-effort acknowledge behavior for non-ephemeral runners

Scope

  • this PR is intentionally scoped to self-hosted ephemeral runners
  • it reuses the existing ephemeral cleanup path when acknowledge returns broker job-not-found
  • it does not broaden behavior for other single-use modes in this change

Testing

  • cd src && ./dev.sh layout
  • cd src && ./dev.sh test

Fixes #4312

rentziass and others added 2 commits July 6, 2026 11:01
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 6, 2026 09:02
@rentziass rentziass requested a review from a team as a code owner July 6, 2026 09:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds explicit classification for broker acknowledge “404 / Job not found” responses and uses that classification to let single-use runners (ephemeral / --once) exit cleanly instead of continuing to poll in a terminal condition.

Changes:

  • Add a new RunnerRequestJobNotFoundException and throw it from BrokerHttpClient.AcknowledgeRunnerRequestAsync when the broker returns a “Job not found” 404.
  • Update the listener runner loop to exit early for single-use runners when acknowledge hits the job-not-found terminal condition, while preserving best-effort behavior for persistent runners.
  • Add L0 coverage for the new broker classification and the single-use vs persistent runner behaviors.
Show a summary per file
File Description
src/Sdk/WebApi/WebApi/BrokerHttpClient.cs Adds job-not-found detection and throws a dedicated exception on acknowledge.
src/Sdk/WebApi/WebApi/Exceptions/RunnerRequestJobNotFoundException.cs Introduces a new exception type representing the acknowledge terminal condition.
src/Runner.Listener/Runner.cs Exits single-use runners on job-not-found acknowledge and ensures ephemeral cleanup triggers.
src/Test/L0/Sdk/RSWebApi/BrokerHttpClientL0.cs Adds unit tests validating acknowledge error classification behavior.
src/Test/L0/Listener/RunnerL0.cs Adds runner loop tests for single-use exit vs persistent continuation on acknowledge job-not-found.

Review details

  • Files reviewed: 5/5 changed files
  • Comments generated: 3
  • Review effort level: Low

Comment thread src/Runner.Listener/Runner.cs Outdated
Comment thread src/Test/L0/Listener/RunnerL0.cs Outdated
Comment thread src/Test/L0/Sdk/RSWebApi/BrokerHttpClientL0.cs
@rentziass rentziass changed the title Exit single-use runners on broker acknowledge job-not-found Exit ephemral runners on broker acknowledge job-not-found Jul 6, 2026
@rentziass rentziass changed the title Exit ephemral runners on broker acknowledge job-not-found Exit ephemeral runners on broker acknowledge job-not-found Jul 6, 2026
@rentziass rentziass changed the title Exit ephemeral runners on broker acknowledge job-not-found Exit single-use runners on broker acknowledge job-not-found Jul 6, 2026
@rentziass rentziass changed the title Exit single-use runners on broker acknowledge job-not-found Exit ephemeral runners on broker acknowledge job-not-found Jul 6, 2026
rentziass and others added 3 commits July 6, 2026 11:54
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment on lines +260 to +263
if (IsAcknowledgeJobNotFound(result.StatusCode, brokerError))
{
throw new RunnerRequestJobNotFoundException(brokerError.Message);
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this merge with the switch above somehow? since they are both kind of checking the brokererror and throw exception.

return statusCode == HttpStatusCode.NotFound &&
brokerError?.StatusCode == (int)HttpStatusCode.NotFound &&
!string.Equals(brokerError.ErrorKind, BrokerErrorKind.RunnerNotFound, StringComparison.Ordinal) &&
string.Equals(brokerError.Message, "Job not found", StringComparison.OrdinalIgnoreCase);

@ericsciple ericsciple Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you help me understand more about this?

Is there an error kind we could rely on, instead of message?

I'm worried relying on the error message might be fragile. That's why error kind exists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Self-hosted runner gets stuck in active state, blocking queued jobs across multiple repositories

4 participants