Skip to content

feat: Add Bounded Concurrent BYOM Workspace Leases - #167

Merged
danny-avila merged 8 commits into
mainfrom
danny-avila/byom-concurrent-workspace-leases
Sep 9, 2026
Merged

feat: Add Bounded Concurrent BYOM Workspace Leases#167
danny-avila merged 8 commits into
mainfrom
danny-avila/byom-concurrent-workspace-leases

Conversation

@danny-avila

@danny-avila danny-avila commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

I added opt-in bounded concurrency for independent BYOM native workspace roots, while retaining serial compatibility and workspace-specific failure containment.

  • Negotiate up to eight lease slots within the Code API and machine ceilings; retain serial wire behavior when either side does not enable concurrency.
  • Schedule the oldest eligible workspace from bounded admission, serialize each root, and preserve the legacy aggregate lock during concurrent work.
  • Bind claims, acknowledgements, settlement, cleanup, and reservations to their slot and assignment owner.
  • Retain durable per-root quarantine after uncertain mutations while allowing healthy roots to continue.
  • Add repeatable --workspace id=path registration, reject overlapping roots, and bound the persistent native executor cache.
  • Document opt-in deployment, root selection, and explicit quarantine recovery without changing approval or network policy.

This change is confined to Code API and @librechat/code. LibreChat's workspace selection work is independent; callers must preserve the registered workspace IDs. Concurrent Docker/runtime sessions and same-root file-level parallelism are not enabled.

Change Type

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Testing

  • Passed focused admission/slot reservation tests and existing cleanup/finalization regressions.
  • Passed store/worker integration cases for independent roots, same-root exclusion, and uncertain mutation quarantine while a healthy root completes.
  • Passed focused CLI overlap/slot-limit tests and legacy/missing negotiation receipt tests.
  • Passed cancellation/expiry during local cleanup handoff, replica ceiling enforcement, active-capacity-change rejection, and native pool exhaustion/eviction regressions.
  • Passed package TypeScript build, focused service TypeScript checking, and diff whitespace checks.
  • Verified real HTTP bridge + isolated Redis + two native SRT processes on macOS: independent one-second commands overlapped, persisted separate workspace files, and killing one executor quarantined its root without stopping the healthy root.
  • Verified a 12-command live burst across both roots completed with same-root exclusion and no workspace-busy failures.
  • Used source-based focused test selection because the Codegraph selection token was unavailable. Rely on CI for full suites and platform matrices.

Test Configuration

Set CODEAPI_BRIDGE_MAX_WORKSPACE_LEASE_SLOTS=2 on all API replicas and start the updated worker with --workspace-lease-slots 2, a primary directory, and an additional non-overlapping --workspace second=/path/to/second. Defaults remain serial. Existing pairing, approvals, and sandbox network configuration remain required.

Checklist

  • My code adheres to this project's style guidelines
  • I have performed a self-review of my own code
  • I have commented in any complex areas of my code
  • I have made pertinent documentation changes
  • My changes do not introduce new warnings
  • I have written tests demonstrating that my changes are effective or that my feature works
  • Local unit tests pass with my changes

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current PR head 322d780. Confirm that this exact commit is the reviewed commit and ignore findings that apply only to earlier heads. Focus on slot ownership, mixed-version negotiation, cancellation/settlement races, per-root quarantine, and native process lifetime.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-09T04:34:02.979445Z 6ebad00 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current PR head e7abf6d. Confirm this exact reviewed commit. This is the second and final requested cycle. The update fixes the legacy heartbeat regression, bounds cleanup handoff waits on cancellation/deadline, shortens aggregate lease expiry after slot release, and covers allocation failures before native dispatch. Real Redis + HTTP + native SRT verification passed concurrent commands and executor-death isolation. Please focus on lifecycle ownership and mixed-version safety.

@danny-avila danny-avila changed the title feat: Add Bounded Concurrent BYOM Workspace Leases 🛤️ feat: Add Bounded Concurrent BYOM Workspace Leases Sep 9, 2026
@danny-avila

Copy link
Copy Markdown
Collaborator Author

Self-review follow-up pushed as e3348dc: enforce the local Code API replica ceiling again at dispatch/lease consumption, test drain-before-capacity-change and successful local-cleanup handoff, and remove unrelated formatting churn. Focused service tests (12), worker handoff/negotiation tests (5), and typechecks pass. The real Redis + HTTP + native SRT rerun passed two concurrent commands (~1.53s), a 12-command queued burst with same-root exclusion, injected pre-assignment lease transport recovery, and executor-death quarantine isolation. Two review requests have already been made, so I am not triggering a third. The in-progress requested review targets e7abf6d, not this follow-up commit.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e7abf6d603

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/code/src/cli.ts
Comment thread packages/code/src/worker.ts Outdated
Comment thread packages/code/src/cli.ts Outdated
@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the exact current head 4882def and confirm the reviewed commit. The review cap has been lifted. Previous findings are addressed; especially scrutinize independent post-settlement quarantine, receipt lifetime/ownership, reset replay protection, concurrent cleanup and mixed-version behavior. Focused tests, live Redis/HTTP/native-SRT fault injection and CI tests pass.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4882def260

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/code/src/worker.ts Outdated
Comment thread service/src/bridge/store.ts
Comment thread packages/code/src/cli.ts Outdated
@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current PR head 760ab7f. Confirm that this exact commit is the reviewed commit and ignore findings that apply only to earlier heads.

This round makes result finalization plus local cleanup a two-phase root release, bounds local cleanup, retries ownership receipts while isolating lane-local transport failures, and keeps reset-only registration unready. Focused transition/failure-injection tests and real Redis/HTTP/native-SRT canaries pass.

@danny-avila

Copy link
Copy Markdown
Collaborator Author

Additional live verification on unchanged head 760ab7f: actual Redis + HTTP bridge + native SRT passed a post-unlink cleanup hang (five-second production bound), retaining the affected root fence while a subsequent healthy-root command succeeded. Independent one-second commands overlapped (~1.52 seconds total), and a 12-command burst preserved same-root exclusion. Process-death and post-unlink exception canaries also passed. All CI checks are now green. These were isolated local tests on an ephemeral HTTP port and Redis Unix socket, not deployment acceptance tests.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 760ab7f3ff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/code/src/worker.ts Outdated
Comment thread service/src/bridge/store.ts
@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review the current PR head 51540fe. Confirm this exact reviewed commit and ignore findings applying only to earlier heads.

Both latest P1s are fixed: every advertised slot keeps polling after lane-local receipt failures, and identical rejected-settlement retries finish pending root finalization. Regressions cover every quarantine response lost after server commit, sustained transport outage without re-executing the failed command, and finalization failure after the dispatcher exits. 17 focused service tests and both typechecks pass. Latest main (#168 scratch hardening) is merged; real Redis/HTTP/native-SRT verification passes with all quarantine responses dropped, same-root serialization, parallel roots, and healthy-root continuation.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 51540fe8fe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/code/src/cli.ts
Comment thread packages/code/src/worker.ts
Comment thread packages/code/src/native-pool.ts
Comment thread service/src/config.ts
@danny-avila

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review exact current head 6ebad00 and confirm the reviewed commit. All four latest findings are addressed: mapped guards gate serial readiness (including fallback negotiation); clean-rejection receipt failures use lane-local quarantine handling; known-clean executor failures retire the child without command replay; Compose forwards slot ceilings to both API consumers. Focused tests (13 package, 7 service), Compose render checks, typechecks, and the real Redis/HTTP/native-SRT lost-quarantine-response canary pass. Please audit remaining transition/rollout invariants, ignoring findings that apply only to old heads.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 6ebad00dea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila
danny-avila merged commit 71308b5 into main Sep 9, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant