Skip to content

.NET: Improve Hosted Agent LRA Resilient Recovery Sample - #8082

Open
Roger Barreto (rogerbarreto) wants to merge 4 commits into
microsoft:mainfrom
rogerbarreto:feature-resilient-sample
Open

.NET: Improve Hosted Agent LRA Resilient Recovery Sample#8082
Roger Barreto (rogerbarreto) wants to merge 4 commits into
microsoft:mainfrom
rogerbarreto:feature-resilient-sample

Conversation

@rogerbarreto

Copy link
Copy Markdown
Member

Motivation & Context

Make the .NET Foundry hosted workflow resilience sample easier to follow and show why downstream services must tolerate repeated calls during recovery. Recovering workflow progress and replaying response events do not prevent an unfinished step from being executed again.

Description & Review Guide

  • What are the major changes? Simplify the countdown executors and use one E2E flow for Crash and Shutdown. Run a SQLite-backed idempotent service in a separate process through the same E2E executable's --idempotent-service mode. Share IdempotentServiceClient between the hosted workflow and E2E. Replace the output converter's checkpoint callback with explicit checkpoint items handled by AgentFrameworkResponseHandler, add coverage for the marker, and update sample documentation and XML remarks. Ignore local launch settings for the Claw hosted sample.
  • What is the impact of these changes? Makes the checkpoint coordination and process lifetimes explicit while preserving the existing recovery model. Repeated service calls reuse stored results instead of inserting duplicate operations. The client displays replayed output and reports the stored operation count. The examples explicitly distinguish service idempotency from exactly-once workflow execution and are not production implementations.
  • What do you want reviewers to focus on? Checkpoint event ordering and cancellation/shutdown handling, the service remaining alive across hosted process replacement, and the distinction between replayed text and repeated downstream effects.

Related Issue

N/A

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix). A workflow keeps the label and title prefix in sync automatically.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The hosted deployment has an unavailable sibling project dependency, and the E2E can mask failures while reporting success.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Simplifies resilient workflow recovery while demonstrating idempotent downstream operations through a SQLite-backed service.

Changes:

  • Introduces explicit workflow-checkpoint items and handler persistence.
  • Adds crash/shutdown recovery scenarios with an idempotent service.
  • Updates deployment configuration, documentation, and tests.
File summaries
File Description
OutputConverterWorkflowTests.cs Tests checkpoint item emission.
OutputConverter.cs Adds explicit checkpoint boundaries.
AgentFrameworkResponseHandler.cs Persists checkpoint items in order.
Using-Samples/README.md Updates resilience sample reference.
Using-E2E-Resilience.csproj Adds SQLite and ASP.NET dependencies.
ServerProcess.cs Manages child-process output and lifetime.
ServerManager.cs Coordinates builds, services, and recovery.
Using-E2E-Resilience/README.md Documents recovery and idempotency flow.
Using-E2E-Resilience/Program.cs Runs crash and shutdown scenarios.
IdempotentService.cs Implements the SQLite-backed service.
Hosted-Workflow-Resilient-Long-Running/README.md Documents revised workflow behavior.
Hosted-Workflow-Resilient-Long-Running/Program.cs Integrates idempotent operations and shutdown handling.
HostedWorkflowResilientLongRunning.csproj References the shared client project.
azure.yaml Configures service endpoint and scope.
.env.example Adds local idempotency settings.
IdempotentServiceClient.cs Adds the shared HTTP client.
ClawAgent.Hosted/.gitignore Ignores local launch settings.
Directory.Packages.props Pins Microsoft.Data.Sqlite.
Review details
  • Files reviewed: 18/18 changed files
  • Comments generated: 5
  • Review effort level: Balanced

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

</PropertyGroup>

<ItemGroup>
<ProjectReference Include="..\Hosted_Shared_Contributor_Setup\Hosted_Shared_Contributor_Setup.csproj" />
Comment on lines +134 to +136
int operationCount = await idempotentService.GetOperationCountAsync(serverManager.OperationScope, cancellationToken);
Console.WriteLine($" Idempotent service contains {operationCount} completed operations.");
serverManager.MarkSucceeded();
Comment on lines +109 to 112
catch
{
if (this._preCrashTexts is not { Count: > 0 } preCrashTexts)
{
return false;
}

this._recoveryIncludesSnapshot ??=
string.Equals(text, preCrashTexts[0], StringComparison.Ordinal);
if (this._recoveryIncludesSnapshot is not true)
{
return false;
}

if (this._recoverySnapshotIndex >= preCrashTexts.Count)
{
return false;
}

if (!string.Equals(
text,
preCrashTexts[this._recoverySnapshotIndex],
StringComparison.Ordinal))
{
throw new InvalidOperationException(
"The response snapshot returned during reconnection did not match the pre-crash output.");
}

this._recoverySnapshotIndex++;
WriteOutput("restored", text);
return true;
}

private static void WriteOutput(string phase, string text)
{
Console.ForegroundColor = phase == "recovered"
? ConsoleColor.Green
: ConsoleColor.DarkGray;
Console.WriteLine($" {phase,-9} > {text}");
Console.ResetColor();
}
}

internal sealed class ServerProcess
{
private readonly Process _process;
private readonly Task _outputPump;
private readonly Task _errorPump;

private ServerProcess(Process process, TextWriter logWriter)
{
this._process = process;
this._outputPump = PumpAsync(process.StandardOutput, logWriter, "stdout");
this._errorPump = PumpAsync(process.StandardError, logWriter, "stderr");
Console.WriteLine(" The connection was interrupted.");
}
cancellationToken);
return;
}
// The first turn of the workflow is a single message that contains the countdown start value with added of 10 units.
if (!string.IsNullOrEmpty(update.Text))
{
await writer.WriteLineAsync($"[{source}] {line}");
Console.WriteLine($" {update.Text}{(textUpdates.Contains(update.Text) ? " > Repeated due to abrupt crash and recovery replay (idempotency matters here)" : "")}");
@rogerbarreto Roger Barreto (rogerbarreto) changed the title .NET: Simplify resilient workflow recovery and demonstrate service idempotency .NET: Improve Hosted Agent LRA Resilient Recovery Sample Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs .NET Usage: [Issues, PRs], Target: .Net

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants