Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
214 changes: 199 additions & 15 deletions apps/server/src/orchestration/Layers/ProviderCommandReactor.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2449,28 +2449,144 @@ describe("ProviderCommandReactor", () => {
});
});

it("reacts to thread.turn.interrupt-requested by calling provider interrupt", async () => {
it("interrupts the provider and cancels pending user input", async () => {
const harness = await createHarness();
const now = "2026-01-01T00:00:00.000Z";

await Effect.runPromise(
// The interrupt command's timestamp predates every request below, so a
// synthetic cancellation carrying it would fold closed before the request
// it closes. The reactor must stamp cancellations with server time.
const now = "2025-12-31T00:00:00.000Z";

const appendUserInputActivity = (input: {
readonly commandId: string;
readonly activityId: string;
readonly kind:
| "user-input.requested"
| "user-input.resolved"
| "provider.user-input.respond.failed";
readonly payload: Record<string, unknown>;
readonly createdAt: string;
}) =>
harness.engine.dispatch({
type: "thread.session.set",
commandId: CommandId.make("cmd-session-set"),
type: "thread.activity.append",
commandId: CommandId.make(input.commandId),
threadId: ThreadId.make("thread-1"),
session: {
threadId: ThreadId.make("thread-1"),
status: "running",
providerName: "codex",
runtimeMode: "approval-required",
activeTurnId: asTurnId("turn-1"),
lastError: null,
updatedAt: now,
activity: {
id: EventId.make(input.activityId),
tone: "info",
kind: input.kind,
summary:
input.kind === "user-input.requested"
? "User input requested"
: input.kind === "user-input.resolved"
? "User input resolved"
: "Provider user input response failed",
payload: input.payload,
turnId: asTurnId("turn-1"),
createdAt: input.createdAt,
},
createdAt: now,
createdAt: input.createdAt,
});

const questions = [
{
id: "continue",
header: "Continue",
question: "Continue?",
options: [{ label: "Yes", description: "Continue the turn." }],
multiSelect: false,
},
];

await Effect.runPromise(
Effect.gen(function* () {
yield* harness.engine.dispatch({
type: "thread.session.set",
commandId: CommandId.make("cmd-session-set"),
threadId: ThreadId.make("thread-1"),
session: {
threadId: ThreadId.make("thread-1"),
status: "running",
providerName: "codex",
runtimeMode: "approval-required",
activeTurnId: asTurnId("turn-1"),
lastError: null,
updatedAt: now,
},
createdAt: now,
});

// Answered before the interrupt, so it must not be cancelled again.
yield* appendUserInputActivity({
commandId: "cmd-user-input-requested-1",
activityId: "activity-user-input-requested-1",
kind: "user-input.requested",
payload: { requestId: "user-input-request-1", questions },
createdAt: "2025-12-31T23:59:57.000Z",
});
yield* appendUserInputActivity({
commandId: "cmd-user-input-resolved-1",
activityId: "activity-user-input-resolved-1",
kind: "user-input.resolved",
payload: { requestId: "user-input-request-1", answers: { continue: "Yes" } },
createdAt: "2025-12-31T23:59:58.000Z",
});

// Response already failed as stale, so it is dead and must not be cancelled.
yield* appendUserInputActivity({
commandId: "cmd-user-input-requested-stale",
activityId: "activity-user-input-requested-stale",
kind: "user-input.requested",
payload: { requestId: "user-input-request-stale", questions },
createdAt: "2025-12-31T23:59:58.500Z",
});
yield* appendUserInputActivity({
commandId: "cmd-user-input-respond-failed-stale",
activityId: "activity-user-input-respond-failed-stale",
kind: "provider.user-input.respond.failed",
payload: {
requestId: "user-input-request-stale",
detail: `Stale pending user-input request: user-input-request-stale. Provider callback state does not survive app restarts or recovered sessions. Restart the turn to continue.`,
},
createdAt: "2025-12-31T23:59:58.750Z",
});

// Still open when the interrupt lands; the provider resolves it for
// real while handling the interrupt, so no cancelled duplicate may be
// appended on top of the real resolution.
yield* appendUserInputActivity({
commandId: "cmd-user-input-requested-2",
activityId: "activity-user-input-requested-2",
kind: "user-input.requested",
payload: { requestId: "user-input-request-2", questions },
createdAt: "2025-12-31T23:59:59.000Z",
});

// Still open when the interrupt lands and untouched by the provider,
// so the reactor must cancel it.
yield* appendUserInputActivity({
commandId: "cmd-user-input-requested-3",
activityId: "activity-user-input-requested-3",
kind: "user-input.requested",
payload: { requestId: "user-input-request-3", questions },
createdAt: "2025-12-31T23:59:59.500Z",
});
}),
);

harness.interruptTurn.mockImplementation(() =>
Effect.asVoid(
Effect.orDie(
appendUserInputActivity({
commandId: "cmd-user-input-resolved-2-during-interrupt",
activityId: "activity-user-input-resolved-2",
kind: "user-input.resolved",
payload: { requestId: "user-input-request-2", answers: { continue: "Yes" } },
createdAt: "2026-01-01T00:00:01.000Z",
}),
),
),
);

await Effect.runPromise(
harness.engine.dispatch({
type: "thread.turn.interrupt",
Expand All @@ -2485,6 +2601,74 @@ describe("ProviderCommandReactor", () => {
expect(harness.interruptTurn.mock.calls[0]?.[0]).toEqual({
threadId: "thread-1",
});
await waitFor(async () => {
const readModel = await harness.readModel();
const thread = readModel.threads.find((entry) => entry.id === ThreadId.make("thread-1"));
return (
thread?.activities.some(
(activity) =>
activity.kind === "user-input.resolved" &&
(activity.payload as { requestId?: unknown }).requestId === "user-input-request-3" &&
(activity.payload as { cancelled?: unknown }).cancelled === true,
) ?? false
);
});

const readModel = await harness.readModel();
const thread = readModel.threads.find((entry) => entry.id === ThreadId.make("thread-1"));
const activities = thread?.activities ?? [];

// The real mid-interrupt resolution stands alone; appending a cancelled
// duplicate here would re-close a question the user actually answered.
expect(
activities.filter(
(activity) =>
activity.kind === "user-input.resolved" &&
(activity.payload as { requestId?: unknown }).requestId === "user-input-request-2",
),
).toEqual([
expect.objectContaining({
summary: "User input resolved",
payload: { requestId: "user-input-request-2", answers: { continue: "Yes" } },
}),
]);

// Only the still-open request folds closed, exactly once.
const cancelledRequestIds = activities
.filter(
(activity) =>
activity.kind === "user-input.resolved" &&
(activity.payload as { cancelled?: unknown }).cancelled === true,
)
.map((activity) => (activity.payload as { requestId?: unknown }).requestId)
.toSorted();
expect(cancelledRequestIds).toEqual(["user-input-request-3"]);

// Its cancellation carries server time, so it orders after the request
// despite the earlier interrupt command timestamp.
const request3Activity = activities.find(
(activity) =>
activity.kind === "user-input.requested" &&
(activity.payload as { requestId?: unknown }).requestId === "user-input-request-3",
);
const cancelledActivity = activities.find(
(activity) =>
activity.kind === "user-input.resolved" &&
(activity.payload as { requestId?: unknown }).requestId === "user-input-request-3" &&
(activity.payload as { cancelled?: unknown }).cancelled === true,
);
expect(request3Activity).toBeDefined();
expect(cancelledActivity).toMatchObject({
summary: "User input cancelled",
payload: {
requestId: "user-input-request-3",
cancelled: true,
},
turnId: "turn-1",
});
expect(Date.parse(cancelledActivity!.createdAt)).toBeGreaterThan(
Date.parse(request3Activity!.createdAt),
);
});

it("starts a fresh session when only projected session state exists", async () => {
Expand Down
95 changes: 95 additions & 0 deletions apps/server/src/orchestration/Layers/ProviderCommandReactor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import {
EventId,
type ModelSelection,
type OrchestrationEvent,
type OrchestrationThread,
ProviderDriverKind,
type ProjectId,
type OrchestrationSession,
Expand All @@ -16,6 +17,7 @@ import { isTemporaryWorktreeBranch, WORKTREE_BRANCH_PREFIX } from "@t3tools/shar
import * as Cache from "effect/Cache";
import * as Cause from "effect/Cause";
import * as Crypto from "effect/Crypto";
import * as DateTime from "effect/DateTime";
import * as Duration from "effect/Duration";
import * as Effect from "effect/Effect";
import * as Equal from "effect/Equal";
Expand Down Expand Up @@ -49,6 +51,10 @@ import { GitWorkflowService } from "../../git/GitWorkflowService.ts";
const isProviderAdapterRequestError = Schema.is(ProviderAdapterRequestError);
const isProviderDriverKind = Schema.is(ProviderDriverKind);

// Server-side timestamps for reactor-appended activities. Client-supplied
// command timestamps can predate the thread activity they must follow.
const nowIso = Effect.map(DateTime.now, DateTime.formatIso);

type ProviderIntentEvent = Extract<
OrchestrationEvent,
{
Expand Down Expand Up @@ -275,6 +281,43 @@ function stalePendingRequestDetail(
return `Stale pending ${requestKind} request: ${requestId}. Provider callback state does not survive app restarts or recovered sessions. Restart the turn to continue.`;
}

function pendingUserInputRequests(
activities: OrchestrationThread["activities"],
): ReadonlyArray<{ readonly requestId: string; readonly turnId: TurnId | null }> {
const openRequests = new Map<string, TurnId | null>();
const ordered = [...activities].toSorted(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium Layers/ProviderCommandReactor.ts:288

When user-input.requested and user-input.resolved share a createdAt timestamp, an ID ordering where the resolution sorts first causes pendingUserInputRequests to delete the request and then reopen it. The function therefore returns an already-resolved question as pending, so an interrupt appends a false cancellation. Use a durable append sequence/order for equal-timestamp lifecycle events instead of activity.id.

🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/orchestration/Layers/ProviderCommandReactor.ts around line 288:

When `user-input.requested` and `user-input.resolved` share a `createdAt` timestamp, an ID ordering where the resolution sorts first causes `pendingUserInputRequests` to delete the request and then reopen it. The function therefore returns an already-resolved question as pending, so an interrupt appends a false cancellation. Use a durable append sequence/order for equal-timestamp lifecycle events instead of `activity.id`.

(left, right) =>
left.createdAt.localeCompare(right.createdAt) || left.id.localeCompare(right.id),
);

for (const activity of ordered) {
const payload =
typeof activity.payload === "object" && activity.payload !== null
? (activity.payload as Record<string, unknown>)
: null;
const requestId = typeof payload?.requestId === "string" ? payload.requestId : null;
if (requestId === null) continue;

if (activity.kind === "user-input.requested") {
openRequests.set(requestId, activity.turnId);
} else if (activity.kind === "user-input.resolved") {
openRequests.delete(requestId);
} else if (activity.kind === "provider.user-input.respond.failed") {
const detail = typeof payload?.detail === "string" ? payload.detail.toLowerCase() : "";
if (
detail.includes("stale pending user-input request") ||
detail.includes("unknown pending user-input request") ||
detail.includes("unknown pending user input request") ||
detail.includes("unknown pending codex user input request")
) {
openRequests.delete(requestId);
}
}
}

return [...openRequests].map(([requestId, turnId]) => ({ requestId, turnId }));
}

function buildGeneratedWorktreeBranchName(raw: string): string {
const normalized = raw
.trim()
Expand Down Expand Up @@ -366,6 +409,36 @@ const make = Effect.gen(function* () {
),
);

const appendCancelledUserInputActivity = Effect.fn(
"ProviderCommandReactor.appendCancelledUserInputActivity",
)(function* (input: {
readonly threadId: ThreadId;
readonly requestId: string;
readonly turnId: TurnId | null;
readonly createdAt: string;
}) {
const commandId = yield* serverCommandId("user-input-cancelled");
const eventId = yield* serverEventId();
yield* orchestrationEngine.dispatch({
type: "thread.activity.append",
commandId,
threadId: input.threadId,
activity: {
id: eventId,
tone: "info",
kind: "user-input.resolved",
summary: "User input cancelled",
payload: {
requestId: input.requestId,
cancelled: true,
},
turnId: input.turnId,
createdAt: input.createdAt,
},
createdAt: input.createdAt,
});
});

const formatFailureDetail = (cause: Cause.Cause<unknown>): string => {
const failReason = cause.reasons.find(Cause.isFailReason);
const providerError = isProviderAdapterRequestError(failReason?.error)
Expand Down Expand Up @@ -1194,6 +1267,28 @@ const make = Effect.gen(function* () {

// Orchestration turn ids are not provider turn ids, so interrupt by session.
yield* providerService.interruptTurn({ threadId: event.payload.threadId });
// Some providers discard their callbacks without emitting a matching
// resolution event. Close those requests once the interrupt succeeds,
// deriving from a fresh read: the provider can append real resolutions
// while handling the interrupt. Cancellations get server-side timestamps
// so they order after the request activity even when the interrupt
// command predates it.
const postInterruptThread = yield* resolveThread(event.payload.threadId);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium Layers/ProviderCommandReactor.ts:1276

The interrupt path can append a user-input.resolved cancellation before the provider's real resolution, leaving a false cancellation and duplicate resolution in the durable activity log. resolveThread at this point may run before the callback fiber resumes and ProviderRuntimeIngestion projects the resolution; coordinate with callback completion or make cancellation resolution idempotent.

🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/orchestration/Layers/ProviderCommandReactor.ts around line 1276:

The interrupt path can append a `user-input.resolved` cancellation before the provider's real resolution, leaving a false cancellation and duplicate resolution in the durable activity log. `resolveThread` at this point may run before the callback fiber resumes and `ProviderRuntimeIngestion` projects the resolution; coordinate with callback completion or make cancellation resolution idempotent.

if (!postInterruptThread) {
return;
}
yield* Effect.forEach(
pendingUserInputRequests(postInterruptThread.activities),
({ requestId, turnId }) =>
Effect.flatMap(nowIso, (cancelledAt) =>
appendCancelledUserInputActivity({
threadId: event.payload.threadId,
requestId,
turnId,
createdAt: cancelledAt,
}),
),
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale snapshot double-cancels

Low Severity

pendingUserInputRequests runs against the thread snapshot taken before interruptTurn. Claude and Cursor already settle open prompts and emit user-input.resolved during interrupt, so the reactor still appends a second user-input.resolved with cancelled: true for requests that are no longer open.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit b6e1edf. Configure here.

});

const processApprovalResponseRequested = Effect.fn("processApprovalResponseRequested")(function* (
Expand Down
Loading