Skip to content

Copilot chat: multi-agent UI, Plan/Execute questions, progress, cleanup - #1473

Merged
olsonanl merged 157 commits into
BV-BRC:alphafrom
cucinellclark:alpha-hardening
Oct 29, 2026
Merged

olsonanl merged 157 commits into
BV-BRC:alphafrom
cucinellclark:alpha-hardening

Conversation

@cucinellclark

@cucinellclark cucinellclark commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

From cucinellclark/bvbrc_website:alpha-hardening into BV-BRC/BV-BRC-Web:alpha (up to date with
alpha, v3.60.7). 145 commits plus upstream merges; 58 files, +10.4k / −20.2k lines, nearly all under
public/js/p3/widget/copilot/ and resources/copilot.css.

This is the frontend for the multi-agent BV-BRC Copilot. It needs the matching Copilot API gateway
(BV-BRC-Copilot-API, branch alpha-hardening) and the Python orchestrator behind it. Older gateways
don't have the routes it calls (execution questions, session stop, workflow-watch status).

What users get

  • Full-page chat at /view/Copilot, linked from the header (Tools & Services → AI → Chat). The
    floating window hides while the page is open.

  • Plan / Execute mode. Nothing that writes (jobs, groups, group edits) runs without consent. In
    Plan mode the assistant prepares the job or group and asks one Yes / No / Other question
    (ExecutionBlockedCard):

    • Yes runs exactly what was shown.
    • No records the answer.
    • Other sends a change request.

    The answer survives reloads.

  • Plans (PlanCard) with approve / skip / retry. A step that creates a group or submits a job
    pauses on its question, waits for submitted jobs, and continues when they finish.

  • Clarifying questions as a card for every agent (ClarificationChips). The answers go back to
    the agent that asked.

  • Live progress: a collapsed progress card while a turn runs, a "What I did" summary under each
    answer (ProgressView), and the view follows the card until the answer starts (TurnFollow).

  • Job completion messages appear in the open chat without a reload.

  • Workspace paths in answers become chips that open the workspace browser:

    • quoted or backticked paths keep their spaces;
    • links are URL-encoded per segment, so # and spaces work;
    • code blocks are left alone.
  • Attachments: images (upload, paste, screenshot), text files and PDFs, plus a workspace file
    picker (CopilotWorkspacePathPicker).

  • Large downloads from answers fetch every row with cursor paging (CursorDownload). In
    Chrome/Edge they stream to disk; the dialog explains the in-memory limit elsewhere.

  • Smaller fixes:

    • jump-to-latest button, with "New response" when a reply finishes off-screen;
    • Stop works even before the first event arrives;
    • "Running" shown in the sidebar for background turns;
    • titles come from the first user message;
    • a new user's first chat is registered;
    • markdown images render as links (no automatic fetch);
    • the Copilot button shows when logged out, with a log-in prompt.

Removed

Code with no caller, or features the backend no longer supports:

  • Legacy cards and their adapters: the data / workspace / jobs grid cards with their explorer
    adapters, CopilotToolHandler, the side-panel chat (ChatSessionContainerSidePanel,
    CopilotInputSidePanel), and the RAG dropdown.
  • The old workflow form engine: WorkflowEngine.js (2,700 lines, imported but never
    constructed), its workflowForms/, and ServiceValidationRules.js.
  • Hidden panels: the Files / Workflows / Images panels, which could no longer be opened but still
    fetched data on every chat switch. With them went their stores and the selection system.
    • The workflows "selection" was filled automatically from the session's job ids and sent with
      every message as items the user had selected. It no longer is.
    • Image chips in the input box are unchanged.
  • Uncalled code: about 25 uncalled methods, and about 1,700 lines of CSS that only the removed
    code used.

Changes outside the Copilot folder (please review)

These are small and all Copilot-related; no other page's behaviour changes.

File Change
widget/viewer/Copilot.js The full-page chat viewer (was a stub)
app/p3app.js Removed the old chat rectangle button; the floating button hides on /view/Copilot
views/bv-brc-header.ejs "AI → Chat" menu entry
widget/GridContainer.js, widget/JobManager.js Dropped imports of the removed side-panel chat
widget/WorkspaceObjectSelector.js showHidden no longer throws when window.App is not set yet
store/ChatSessionsMemoryStore.js Shared paging state for the session sidebar, and page merges deduplicated by session id

Testing

  • Jest: 56 tests under public/js/p3/widget/copilot/tests/ pass.
  • Release build (node dojo/dojo.js load=build --profile release.profile.js --release)
    completes. Its 28 errors are the pre-existing kinds: jest test files that aren't AMD modules, and
    libraries outside the build such as markdown-it, leaflet and molstar. None are in changed modules.
    The previous build of this branch reported 29; the difference is a deleted test file.
  • Module loading: every p3/ module dependency resolves.
  • ESLint: changed files have no new no-undef / no-unused-vars errors compared with alpha.
  • Manual: run on our dev and public test sites (dev-8 / dev-6) with real users through
    October 2026.

🤖 Generated with Claude Code

cucinellclark and others added 30 commits February 28, 2026 11:20
Replace all direct workflow engine REST calls to use GoWe's API:
- submitWorkflowForExecution: POST /submissions (was /workflows/{id}/submit)
- getWorkflowStatus: GET /submissions/{id} (was /workflows/{id}/status)
- getWorkflowById: GET /workflows/{id} (same path, unwrap GoWe envelope)
- getBatchWorkflowStatus: parallel individual calls (GoWe has no batch endpoint)
- config.js: workflow_url points to gowe.software-smithy.org
- All 4 fallback URLs updated from dev-7.bv-brc.org to gowe.software-smithy.org
GoWe uses separate workflow_id (registration) and submission_id
(execution) identifiers. Status checks hit /submissions/{id} and
require the submission_id, not workflow_id.

- CopilotToolHandler.js: Extract submission_id from SSE payload
- ChatMessage.js: Store submission_id after submit, use it for
  Check Status button and when opening workflow dialog
- WorkflowEngine.js: Store submission_id in execution_metadata
  after submit, use it for status polling
- CopilotApi.js: Fix JSDoc on getWorkflowStatus
cucinellclark and others added 24 commits October 15, 2026 09:45
Conflicts resolved:
- copilot CopilotApi / CopilotDisplay / ChatMessage / WorkflowsExplorerAdapter:
  ours (the fork rewrote or removed the code upstream's Phase 0a codemod
  edited), then upstream's convention applied to our code: the 30 inline
  Authorization headers in CopilotApi and the one in WorkflowsExplorerAdapter
  now use p3/auth/authHeaders (authHeader(), same value as the old idiom).
- GridContainer: our dependency list (no ChatSessionContainerSidePanel) plus
  upstream's ../auth/authHeaders.
- package.json / package-lock.json: upstream's (ours only bumped the version).
- archaeopteryx-js submodule: upstream's 3.28.0 (ours was an older commit).

Checked: AMD dependency/parameter alignment in every merged file (all 133
authHeader callers bound to auth/authHeaders), node --check on the 176
changed p3 files, upstream jest suite 53/53, copilot tests 11/11.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The title generator was sent every message in the chat, including the
assistant's reply (tool output, links, job cards), so titles described the
answer instead of the request. Now only the first user message is sent,
cut to 2000 characters for long pastes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_submitPlanAction (Approve, Continue, plan answers) never published
ChatSession:TurnStarted/TurnEnded, so the session card's Running label and
its active_job_id poll never started for plan turns. Same end rules as a
typed message: aborted or disconnected streams and 409s leave the label to
the poll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…off-screen

Auto-scroll is off while a reply streams, so a long answer grew below the
view and nothing said it had finished. A round arrow over the bottom of the
message list now shows while there is content below; when a turn in this
chat ends (ChatSession:TurnEnded) and the user is not at the bottom it
becomes a "New response" pill. Click scrolls smoothly to the end; reaching
the bottom, switching chats or panels clears it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng questions

- Delete asks for confirmation; a restored chat that was deleted starts a
  new chat instead of failing.
- Edit Title renames the clicked chat in place; Report Issue reports the
  clicked chat.
- Clarifying questions come back after a reload until answered; missing
  required answers are marked; Enter submits; text set as plain text.
- Empty messages are not sent; the saved Text Size applies in the
  full-page view; Approve returns after Edit Plan -> Save / Cancel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The data API returns at most 25,000 rows per request, so a download link
in an answer saved the first 25,000 rows without saying so (Salmonella
genomes: 25,000 of 57,445). A plain click on a data-API download link
(TSV / FASTA, http_download=true) now pages through the whole result in
the browser with cursor(<mark>) and X-Cursor-Mark, using the user's
token, and saves one file (one TSV header). A progress dialog with
Cancel; above 500,000 rows it asks first. Middle-click and copy-link
still give the plain link.

CursorDownload.js (UMD, testable), ChatMessage._wireCursorDownloads /
_runCursorDownload, tests/CursorDownload.test.js.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the planning agent's clarification questions showed as the chips
card; another agent's questions were dropped (2026-10-20, "I want to
download influenza protein data" got only "I have some questions before
I can proceed."). The backend now sends any agent's questions as the
card with the asking agent on the payload (and plan_step for a paused
plan step).

- ClarificationChips: header by agent ("Before I go on" vs planning's
  "Before I create a plan"); publishes agent and planStep with answers.
- ChatMessage: passes agent / plan_step from the card payload.
- CopilotInput: answers to a non-planning agent are one turn pinned to
  that agent (original request plus the answers, stored as
  user_clarification); a paused plan step's answers go through the
  plan-step reply path. _submitPlanAction takes a target agent.
- CopilotDisplay: a card is answered once any later user message exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…alog wraps

Where the browser has the File System Access API (Chrome, Edge), a click
on a data download link opens the Save dialog first and each 25,000-row
page is written to the chosen file as it arrives, so memory stays at one
page and there is no 500,000-row question; a cancelled or failed
download discards the file. Firefox and Safari keep the in-memory path.

CursorDownload.run takes an optional sink; saveOptions() gives the Save
dialog's suggested name and type. The progress dialog has a fixed width
and wrapping text (the size question stretched it past the window).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the in-memory path only (no File System Access API: Firefox, Safari),
the progress dialog notes that the download is held in memory until it
finishes and that Chrome or Edge write the file as it downloads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The round button did not center its content (no justify-content, a
min-width wider than the glyph) and the arrow was a font glyph whose
side bearings and vertical metrics vary by font. Now a 32x32 circle with
centered content and an inline SVG arrow centered in its 16x16 box; the
"New response" pill keeps its padding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- p3app: the floating button is created for logged-out visitors too.
  Removed chatButtonWidget(), which built a ChatButton it never placed on
  the page (dead).
- ChatButton: each click checks the login; logged out, a small dialog
  ("Please log in to use Copilot.") with Log in (Topic /login, the site's
  Sign In dialog) and Cancel. No chat window, no API calls.
- viewer/Copilot: the floating button is hidden on /view/Copilot also
  when logged out (one prompt), and its Login Required dialog has a Log
  in button.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The workflow monitor writes a finished job's message to the chat in
Mongo, but its SSE push reaches a browser only while a chat turn is
streaming (4 of 220 completion messages up to 2026-10-20). Now, while
the chat on screen has an active workflow watch, ChatSessionContainer
reads /session/<id>/workflow-watches every 30 s; when a job finishes it
re-reads the chat's messages 3 s later (after the turn if one is
streaming). It checks on chat switch, after each turn and on activation,
and stops when no watch is active, the chat changes, the container is
inactive (hidden floating window) or destroyed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edit_group (new backend tool) changes an existing group's members and
always asks first. The prepared-actions card labels it Group, asks
"Update this group?" / "Yes, update it", and "Run these N group actions?"
when a set mixes creates and edits.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The "New response" pill was set only by ChatSession:TurnEnded, which the
workflow-watch poll's refresh never fires, so a completion message that
appeared off-screen gave no signal. The poll now refreshes through
_refreshMessages(..., announce=true): showMessages compares with the
messages already shown for this session and, when the list grew, follows
it if the user was at the bottom or lights the pill otherwise. The
RefreshSession topic and other callers are unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Testing session 2026-10-16 BV-BRC#5: users saw one replaced line ("Planning
next step...") for most of a 1-3 minute turn, and "Using **undefined**"
for the routing and agent-start events.

The gateway now sends each progress update as a step (step_id, phase,
status, label, detail, result, took_ms) and saves them on the assistant
message as agent_metadata.steps. CopilotSSEEventHandler merges them by
step_id; the status message carries `steps` and `started_at`;
tool_selected / tool_executed no longer render.

ProgressView.js renders the same steps in one of four variants: line,
list (default), collapsed, timeline. The choice is a browser preference
(copilot-progress-style), set from the floating window's options dialog
next to Text Size or with the toggle on the status card; a change
redraws every open chat. Finished answers get a collapsed "What I did"
section (live from the last status message, after a reload from the
saved steps).

Tests: tests/ProgressView.test.js (merge, rows per variant, summary,
preference).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Clark, after comparing the four variants: the collapsed one-liner while
a turn runs plus the "What I did" list under each answer. The toggle on
the card and the options-dialog dropdown are gone. line / list /
timeline stay in code behind the localStorage override
copilot-progress-style (no UI), so an option can return as one dropdown.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Brings in the 429/503 throttle handling (retryRequest, throttledRequest,
ThrottleBanner), the vendored bvbrc_js_client (replaces our symlink into
node_modules), releases v3.60.6 and v3.60.7.

Conflict in P3JsonRest.js: our cursor pagination vs upstream's retry
wrapper. Kept the retry wrapper; the cursor code now reads X-Cursor-Mark
from lastXhr (the winning attempt), since results.ioArgs no longer exists
once the request goes through retryRequest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After sending a message the progress card grew below the visible area and
users had to scroll down every time: CopilotDisplay re-renders the list on
every progress update and does not follow new output. Now a turn
(ChatSession:TurnStarted, typed messages and plan actions alike) scrolls to
the bottom and follows each re-render until the answer starts (the view is
then on its first lines), the user scrolls up on purpose (wheel/keys/touch/
scrollbar + leaving the bottom), the turn ends, or the session changes.
Decisions in TurnFollow.js, with jest tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ChatMessage resolved message.tool_call for card renderers that no longer
exist, plus two unused workspace-listing helpers; CopilotApi.replayToolCall
posted to a gateway route removed on 2026-10-08; CopilotToolHandler.js was
no longer loaded.  The gateway stops storing tool_call (same day).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chip stopped at the first space ("/ARWattam@patricbrc.org/BV-BRC Workshop/..."
became a chip for ".../BV-BRC"), and chip links used the raw path, so a "#"
in a folder name cut the URL.

- A path inside "...", `...` or <...> is one path, spaces included, when the
  delimited text is exactly a path; the chip replaces the delimiters.
- Unquoted, a path continues across a space only when another "/" follows
  (spaces between folders); a space in the last folder still ends it.
- Inline code whose whole text is a path becomes a chip; code blocks and
  links never do.
- The workspace path button inserts the path in double quotes; removing an
  input chip removes its quotes too.
- Chip links encode each segment (util/encodePath's rule, copied because
  that module is AMD-only).

WorkspacePathUtils gets the UMD wrapper for jest; 21 tests.
Plan: current_plans/workspace_path_chips.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p F3, F4, F5, G13)

- F3/G13: WorkflowEngine.js (imported by CopilotDisplay, never constructed),
  its workflowForms/ (form adapters, ServiceFormRegistry and generated field
  defs) and ServiceValidationRules.js; ~1,000 lines of copilot.css that only
  they used (workflow-*, embedded-form, three unused keyframes).
- F4: methods with no caller anywhere in public/js/p3 or views: the four
  CopilotApi plan wrappers for the removed gateway routes, getUserPrompts /
  savePrompt / searchRagChunkReferences / getCurrentStreamState /
  isQueryAbortableTool, CopilotDisplay _unwrapReplayResultPayload /
  _formatSize / _formatTimestamp, CopilotInput setModelText / _escapeHtml /
  getExecutionMode / _renderWorkspacePathTokenEditor,
  ChatMessage._buildWorkspaceBrowserUrl, PlanCard._cancelPlan,
  ChatSessionContainer.whenInitialized, ChatSessionTitle enable/disableEditing,
  CopilotSSEEventHandler getSteps / getStatusMessageForRemoval.
- F5: source_tool was copied onto messages and never read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ction system (cleanup F1, F2, D1)

- F1: the panels could not be opened since the tab buttons were removed,
  but every session switch still fetched /get-session-files (removed from
  the gateway, so a 404). Gone: CopilotDisplay's files/workflows/images
  containers and renderers, setActivePanel, ChatSessionContainer's
  session-files state, fetch, paging and realtime insert, _setActiveTab,
  SessionFilesExplorerAdapter, WorkflowsExplorerAdapter, SessionFilesStore
  (+ test), CopilotApi.getSessionFiles, the session_file_created case (the
  gateway never sends it).
- F2: the four selection stores, the selection merge/bind code in
  CopilotDisplay and ChatSessionContainer, and CopilotInput's
  selected* fields, indicators and request params. Workspace items and jobs
  were always empty; the workflows "selection" was filled automatically
  from the session's workflow_ids (GoWe workflow and submission ids) and
  sent as selected_workflows on every turn in a session with jobs, so those
  ids reached every agent prompt as items the user had selected. The
  backend selected_items path stays (D2).
- Image attachments: the input-box chips (CopilotInput) are unchanged;
  the hidden images panel copy and its change callback are gone; switching
  chats still clears attached images.
- Also: WorkspaceListingWidget.js (referenced nowhere), the unused
  include_files option of getSessionMessages and the workflow_ids /
  workflow_grid fields of ChatSession:Selected, addWorkflowToSession (no
  caller), and ~650 lines of copilot.css only these used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The default service URLs (www.bv-brc.org instead of p3.theseed.org) and the
workflow_url default stay out of the upstream PR; deployments set what they
need in p3-web.conf.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e PR)

PageGrid, P3JsonRest and CursorPagination return to upstream alpha.
Cursor paging (291f42d) changed every PageGrid on the site to
Previous/Next with no page numbers, and nothing in the Copilot code uses
it; offset paging works at any depth on the production API (rows 48,000
of 49,429 in 0.74 s). It can be offered upstream as its own change.
Copilot's large downloads (CursorDownload.js) page the API directly and
are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@olsonanl

Copy link
Copy Markdown
Contributor

#1475, #1474 and #1476 are now all on alpha (plus the v3.60.8 release), so this PR will
need a merge of alpha. I test-merged it to see what you're actually facing.

One conflict, one file, two hunks — both in the AMD block of GridContainer.js. No
function bodies conflict.

The resolution is not the usual "keep both sides", because this PR deletes
copilot/ChatSessionContainerSidePanel.js and alpha still imports it. Take alpha's list and
drop that one entry:

  './copilot/CopilotApi', './copilot/ChatSessionOptionsBarSidePanel',
  '../auth/authHeaders',
  '../util/QueryDescriptor',
  './download/UnifiedDownloadWizard',
  './download/SaveSearchDialog',
  '../util/joinTooBroad', './JoinTooBroadMessage',
  './ColumnsGenome', './AdvancedSearchFields', '../util/constructMetadataName', '../store/GenomeMetadataStore'
], function (
  …
  CopilotAPI, ChatSessionOptionsBar,
  authHeader,
  QueryDescriptor, UnifiedDownloadWizard, SaveSearchDialog,
  joinTooBroad, JoinTooBroadMessage,
  ColumnsGenome, AdvancedSearchFields, constructMetadataName, GenomeMetadataStore
) {

Everything else from alpha stays, including the searchColumnDefinitions block that follows
the factory opener — that's #1474's and unrelated to this PR.

I applied exactly that locally and verified: 43 deps / 43 params aligned, file parses,
and npm test passes 163/163 with all three feature sets present (your PlanCard,
#1475's joinTooBroad, #1474's searchColumnDefinitions).

One leftover worth knowing: line ~497 still has a commented-out
// this.chatPanel = new ChatSessionContainerSidePanel({. Harmless since it's a comment,
but you may want to clear it while you're in there given the module is gone.

After resolving, worth a quick arity check — a dep/param mismatch shifts every argument
after it and fails at runtime, not at merge:

node -e "const s=require('fs').readFileSync('public/js/p3/widget/GridContainer.js','utf8');
const m=s.match(/define\(\[([\s\S]*?)\]\s*,\s*function\s*\(([\s\S]*?)\)/);
const d=m[1].split(',').filter(x=>x.trim()).length, p=m[2].split(',').filter(x=>x.trim()).length;
console.log(d===p ? 'aligned ('+d+')' : 'MISALIGNED '+d+' deps vs '+p+' params')"

# Conflicts:
#	public/js/p3/widget/GridContainer.js
@olsonanl
olsonanl merged commit c2e1944 into BV-BRC:alpha Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants