Skip to content

Commit ffc7572

Browse files
iteloclaude
andauthored
feat(wizard): report how far a run gets (#68)
The wizard is a funnel — connect, install SDK, install plugin, analyze, choose a mode, run the agent — and nothing tells us where developers stop. Add analytics for the whole run. Every phase the app enters is reported as wizard_screen_viewed, so the screen sequence is the funnel and the last one reported is where a run ended. Each decision along the way gets its own event: which connect method was picked, whether a key was already there, whether an install failed, whether the developer took the recommended mode, how each agent step went. wizard_run_finished closes the run with the screen it ended on. That event is emitted from the app's unmount cleanup as well as from a clean finish, which is what makes a Ctrl-C show up as outcome: 'abandoned' on the screen the developer walked away from rather than as silence. Events go to Seam, which forwards them to PostHog: a published CLI cannot hold a PostHog key, so the key, sampling, and rate limiting stay server side. The ingest endpoint is unauthenticated by design — the earliest and most interesting events happen before the developer has connected any key, so requiring a credential would blind exactly the drop-off this measures. Events are in PostHog's own capture shape so a batch forwards straight through, and carry workspace_id (with a workspace group) once a run has connected. The wire protocol is documented on postWizardEvents; the Seam side of POST /seam/wizard/v1/events does not exist yet, so until it does every batch is a dropped 404 — which is exactly how any other delivery failure behaves. Nothing can fail a run: track() neither throws nor blocks the render, a batch that cannot be delivered is dropped, the flush timer is unref'd so analytics never holds the process open, and a run posts nothing at all until startAnalytics() has been called — which is why --help, --version, --debug-screen, and the tests are silent. Runs are attributed to an anonymous install id kept in the host CLI's settings, and a host that cannot store one still reports under a per-process id. What is deliberately not sent: no API keys or key fingerprints, no project paths, no source code, no agent output, and not the free-text note the developer writes for the agent — only its length. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 1185ba0 commit ffc7572

11 files changed

Lines changed: 962 additions & 18 deletions

File tree

‎README.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -159,6 +159,7 @@ The wizard is an [Ink] application under `src/lib`:
159159
- `app.tsx` is the state machine driving the run,
160160
one phase per step, and holds all of the rendering.
161161
- `steps/` holds the logic for each step, with no UI in it.
162+
- `analytics.ts` reports how far a run got (see [Analytics](#analytics)).
162163
- `version.ts` holds the package version reported by `--version`.
163164
It ships a `0.0.0` placeholder that `prepack.ts` replaces with the
164165
version from `package.json` when the package is packed,
@@ -181,6 +182,48 @@ View them with
181182
$ npm run
182183
```
183184

185+
### Analytics
186+
187+
The wizard is a funnel, and `analytics.ts` measures how far a run gets
188+
and where it stops.
189+
Every phase the app enters is reported as `wizard_screen_viewed`,
190+
each decision along the way gets its own event,
191+
and `wizard_run_finished` closes the run with the screen it ended on —
192+
including `outcome: 'abandoned'` for a run left with Ctrl-C
193+
or a closed terminal.
194+
195+
Events are queued and posted in batches to Seam,
196+
which forwards them to PostHog:
197+
a published CLI cannot hold a PostHog key,
198+
so the key, sampling, and rate limiting all stay server side.
199+
The endpoint is unauthenticated,
200+
because the earliest and most interesting events happen before the
201+
developer has connected a key at all:
202+
203+
```
204+
POST {endpoint}/seam/wizard/v1/events
205+
{ "events": [{ "event": "…", "distinct_id": "…", "timestamp": "…", "properties": {} }] }
206+
```
207+
208+
Events are in PostHog's own capture shape so a batch can be forwarded
209+
straight through.
210+
`distinct_id` is an anonymous install id
211+
kept in the host CLI's settings — it identifies an install, not a
212+
developer — and `properties.workspace_id` is present once a run has
213+
connected.
214+
215+
Reporting is best effort and never affects a run:
216+
`track` does not throw or block the render,
217+
a batch that cannot be delivered is dropped,
218+
and a run posts nothing at all until `startAnalytics` has been called,
219+
which is why `--help`, `--version`, `--debug-screen`, and tests are
220+
silent.
221+
222+
Nothing sent identifies the developer or their code:
223+
no API keys or key fingerprints, no project paths, no source code, no
224+
agent output, and not the free-text note the developer writes for the
225+
agent — only its length.
226+
184227
### Source code
185228

186229
The [source code] is hosted on GitHub.

‎src/lib/analytics.ts‎

Lines changed: 258 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,258 @@
1+
import { randomUUID } from 'node:crypto'
2+
import { platform, release } from 'node:os'
3+
4+
import {
5+
postWizardEvents,
6+
type SeamWorkspace,
7+
type WizardAnalyticsEvent,
8+
} from 'lib/api.js'
9+
import type { Sdk } from 'lib/steps/detect-project.js'
10+
import { readInstallId, writeInstallId } from 'lib/store/index.js'
11+
import seamapiWizardVersion from 'lib/version.js'
12+
13+
/**
14+
* Usage analytics for a wizard run.
15+
*
16+
* The wizard is a funnel — connect, install, analyze, integrate — and what this
17+
* measures is how far a developer gets and where they leave. Every phase the app
18+
* enters is reported, so a run that stops shows up as the screen it stopped on.
19+
*
20+
* Events are queued and posted in batches to Seam, which forwards them to
21+
* PostHog (see `postWizardEvents`). Nothing here can fail a run: `track` never
22+
* throws, never blocks the render, and a batch that cannot be delivered is
23+
* dropped. Until `startAnalytics` has run, every call is a no-op — which is what
24+
* keeps tests and one-off invocations (`--help`, `--debug-screen`) silent.
25+
*
26+
* What is deliberately not sent: no API keys, no key fingerprints, no project
27+
* paths, no source code, no agent output, and not the free-text note the
28+
* developer writes — only its length.
29+
*/
30+
31+
// A batch is posted a beat after the first event in it, so a burst of events
32+
// (a phase change plus the decision that caused it) travels as one request.
33+
const FLUSH_DELAY_MS = 1000
34+
const MAX_BATCH = 20
35+
// A guard against an unbounded queue if every post fails: the oldest events are
36+
// dropped rather than held forever.
37+
const MAX_QUEUE = 200
38+
// Long string properties are truncated rather than dropped, so an event stays
39+
// readable without carrying an agent's error dump into PostHog.
40+
const MAX_STRING_LENGTH = 500
41+
42+
export type RunOutcome = 'completed' | 'abandoned' | 'error'
43+
44+
interface Run {
45+
install_id: string
46+
started_at: number
47+
// Merged into every event: what stays true for the whole run.
48+
common: Record<string, unknown>
49+
// Every phase entered, in order, including repeats.
50+
screens: string[]
51+
queue: WizardAnalyticsEvent[]
52+
timer: NodeJS.Timeout | null
53+
// Posts are chained so a batch cannot overtake an earlier one, and so a flush
54+
// can await whatever is already in flight.
55+
inflight: Promise<void>
56+
finished: boolean
57+
}
58+
59+
let run: Run | null = null
60+
61+
/**
62+
* Begin a run: resolve the anonymous install id and record what is true of the
63+
* whole run. Call once, before the app renders. Events tracked before this are
64+
* dropped.
65+
*/
66+
export const startAnalytics = async ({
67+
command,
68+
}: {
69+
command: string
70+
}): Promise<void> => {
71+
run = {
72+
install_id: await resolveInstallId(),
73+
started_at: Date.now(),
74+
common: {
75+
session_id: randomUUID(),
76+
wizard_version: seamapiWizardVersion,
77+
command,
78+
node_version: process.version,
79+
os_platform: platform(),
80+
os_release: release(),
81+
is_ci: isCi(),
82+
},
83+
screens: [],
84+
queue: [],
85+
timer: null,
86+
inflight: Promise.resolve(),
87+
finished: false,
88+
}
89+
}
90+
91+
/** Queue one event. Fire and forget: it neither throws nor blocks. */
92+
export const track = (
93+
event: string,
94+
properties: Record<string, unknown> = {},
95+
): void => {
96+
const current = run
97+
if (current == null) return
98+
99+
current.queue.push({
100+
event,
101+
distinct_id: current.install_id,
102+
timestamp: new Date().toISOString(),
103+
properties: sanitize({
104+
...current.common,
105+
seconds_since_start: secondsSince(current.started_at),
106+
...properties,
107+
}),
108+
})
109+
110+
if (current.queue.length > MAX_QUEUE) {
111+
current.queue.splice(0, current.queue.length - MAX_QUEUE)
112+
}
113+
if (current.queue.length >= MAX_BATCH) {
114+
sendQueued(current)
115+
return
116+
}
117+
scheduleFlush(current)
118+
}
119+
120+
/**
121+
* Report the screen the run just moved to. The screen sequence is the funnel:
122+
* `screen_index` orders it, and the last one reported is where the run stopped.
123+
*/
124+
export const trackScreen = (screen: string): void => {
125+
const current = run
126+
if (current == null) return
127+
current.screens.push(screen)
128+
track('wizard_screen_viewed', {
129+
screen,
130+
screen_index: current.screens.length - 1,
131+
})
132+
}
133+
134+
/**
135+
* Attribute the rest of the run to the workspace it connected to. The id (not
136+
* the name) rides along on every later event, and on the workspace group so
137+
* PostHog can roll a workspace's runs up together.
138+
*/
139+
export const setAnalyticsWorkspace = (workspace: SeamWorkspace): void => {
140+
if (run == null) return
141+
run.common['workspace_id'] = workspace.workspace_id
142+
run.common['workspace_is_sandbox'] = workspace.is_sandbox
143+
run.common['$groups'] = { workspace: workspace.workspace_id }
144+
}
145+
146+
/** Attribute the rest of the run to the SDK it is setting up. */
147+
export const setAnalyticsSdk = (sdk: Sdk): void => {
148+
if (run == null) return
149+
run.common['sdk'] = sdk
150+
}
151+
152+
/**
153+
* Close the run with how it ended, summarizing the whole funnel in one event.
154+
*
155+
* `abandoned` is the interesting one: it is what a Ctrl-C or a closed terminal
156+
* leaves behind, carrying the screen the developer walked away from. Only the
157+
* first call counts, so the app can report a clean finish and still call this
158+
* unconditionally on unmount.
159+
*/
160+
export const finishAnalytics = (
161+
outcome: RunOutcome,
162+
properties: Record<string, unknown> = {},
163+
): void => {
164+
const current = run
165+
if (current == null || current.finished) return
166+
current.finished = true
167+
track('wizard_run_finished', {
168+
outcome,
169+
last_screen: current.screens.at(-1) ?? null,
170+
screens_seen: [...new Set(current.screens)],
171+
screen_count: current.screens.length,
172+
duration_seconds: secondsSince(current.started_at),
173+
...properties,
174+
})
175+
}
176+
177+
/**
178+
* Post everything queued and wait for it to land. Called once on the way out,
179+
* where it is the only thing that blocks — briefly, and bounded by the request
180+
* timeout — so the last events of a run are not lost to process exit.
181+
*/
182+
export const flushAnalytics = async (): Promise<void> => {
183+
const current = run
184+
if (current == null) return
185+
sendQueued(current)
186+
await current.inflight
187+
}
188+
189+
/** Forget the run. For tests, and for a second run in one process. */
190+
export const resetAnalytics = (): void => {
191+
if (run?.timer != null) clearTimeout(run.timer)
192+
run = null
193+
}
194+
195+
const scheduleFlush = (current: Run): void => {
196+
if (current.timer != null) return
197+
const timer = setTimeout(() => {
198+
sendQueued(current)
199+
}, FLUSH_DELAY_MS)
200+
// The wizard must exit when the developer is done, not when analytics is: an
201+
// unreferenced timer never holds the process open.
202+
timer.unref()
203+
current.timer = timer
204+
}
205+
206+
const sendQueued = (current: Run): void => {
207+
if (current.timer != null) {
208+
clearTimeout(current.timer)
209+
current.timer = null
210+
}
211+
if (current.queue.length === 0) return
212+
const batch = current.queue.splice(0, current.queue.length)
213+
current.inflight = current.inflight
214+
.then(async () => {
215+
await postWizardEvents(batch)
216+
})
217+
// A batch that cannot be delivered is dropped. Analytics never reports
218+
// itself to the developer, and never fails a run.
219+
.catch(() => {})
220+
}
221+
222+
// A new install id is written back so the next run is recognized as the same
223+
// install. Where the host cannot store one, the run still reports — under an id
224+
// that lives only as long as the process.
225+
const resolveInstallId = async (): Promise<string> => {
226+
try {
227+
const existing = await readInstallId()
228+
if (existing != null) return existing
229+
const installId = randomUUID()
230+
await writeInstallId(installId)
231+
return installId
232+
} catch {
233+
return randomUUID()
234+
}
235+
}
236+
237+
const secondsSince = (startedAt: number): number =>
238+
Math.round((Date.now() - startedAt) / 1000)
239+
240+
const sanitize = (
241+
properties: Record<string, unknown>,
242+
): Record<string, unknown> => {
243+
const sanitized: Record<string, unknown> = {}
244+
for (const [key, value] of Object.entries(properties)) {
245+
sanitized[key] =
246+
typeof value === 'string' && value.length > MAX_STRING_LENGTH
247+
? `${value.slice(0, MAX_STRING_LENGTH)}…`
248+
: value
249+
}
250+
return sanitized
251+
}
252+
253+
// Runs from a pipeline are real, but they are not developers dropping off, so
254+
// they are flagged rather than filtered out here.
255+
const isCi = (): boolean => {
256+
const ci = process.env['CI']
257+
return ci != null && ci !== '' && ci !== '0' && ci !== 'false'
258+
}

‎src/lib/api.test.ts‎

Lines changed: 43 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import { beforeEach, expect, test, vi } from 'vitest'
1+
import { afterEach, beforeEach, expect, test, vi } from 'vitest'
22

33
const { get, post } = vi.hoisted(() => ({
44
get: vi.fn(),
@@ -15,10 +15,17 @@ vi.mock('@seamapi/http', () => ({
1515
},
1616
}))
1717

18-
import { exchangeWizardInferenceToken, getWorkspaceForApiKey } from './api.js'
18+
import {
19+
exchangeWizardInferenceToken,
20+
getWizardEventsUrl,
21+
getWorkspaceForApiKey,
22+
postWizardEvents,
23+
} from './api.js'
1924

2025
beforeEach(() => vi.clearAllMocks())
2126

27+
afterEach(() => vi.unstubAllGlobals())
28+
2229
test('uses the workspace SDK and its raw client', async () => {
2330
const workspace = {
2431
workspace_id: 'workspace-1',
@@ -48,3 +55,37 @@ test('uses the workspace SDK and its raw client', async () => {
4855
})
4956
expect(post).toHaveBeenCalledWith('/seam/wizard/v1/session', {})
5057
})
58+
59+
test('posts analytics events to Seam unauthenticated', async () => {
60+
const fetchMock = vi.fn(async () => new Response(null, { status: 200 }))
61+
vi.stubGlobal('fetch', fetchMock)
62+
const event = {
63+
event: 'wizard_run_started',
64+
distinct_id: 'install-1',
65+
timestamp: '2026-01-01T00:00:00.000Z',
66+
properties: { wizard_version: '0.0.0' },
67+
}
68+
69+
await expect(postWizardEvents([event])).resolves.toBeUndefined()
70+
71+
const [url, init] = fetchMock.mock.calls[0] as unknown as [
72+
string,
73+
RequestInit,
74+
]
75+
expect(url).toBe(getWizardEventsUrl())
76+
expect(url).toBe('https://connect.getseam.com/seam/wizard/v1/events')
77+
expect(init.method).toBe('POST')
78+
expect(init.headers).toEqual({ 'content-type': 'application/json' })
79+
expect(JSON.parse(String(init.body))).toEqual({ events: [event] })
80+
// No credential: the events the wizard cares most about happen before the
81+
// developer has connected one.
82+
expect(JSON.stringify(init)).not.toContain('authorization')
83+
})
84+
85+
test('rejects when Seam refuses an analytics batch', async () => {
86+
vi.stubGlobal('fetch', async () => new Response(null, { status: 503 }))
87+
88+
await expect(postWizardEvents([])).rejects.toThrow(
89+
'Seam analytics returned 503',
90+
)
91+
})

0 commit comments

Comments
 (0)