feat(mastra): add Openlayer observability exporter for Mastra - #222
Open
viniciusdsmello wants to merge 19 commits into
Open
feat(mastra): add Openlayer observability exporter for Mastra#222viniciusdsmello wants to merge 19 commits into
viniciusdsmello wants to merge 19 commits into
Conversation
…rter Design for OPEN-12306: an OpenlayerExporter for Mastra's observability system, shipped as the `openlayer/integrations/mastra` subpath export. Every mapping claim in the spec was verified against the live Openlayer OTLP endpoint rather than inferred: - row input/output is built only from gen_ai.input.messages / gen_ai.output.messages in semconv 1.38 `parts` shape; OpenInference, traceloop, gen_ai events and gen_ai.prompt/completion all yield empty I/O - Mastra emits that pair only on MODEL_GENERATION spans, so agent_run and workflow_run roots land with empty input/output — the defect the exporter's rewrite shim exists to fix - tool spans map natively and must not be rewritten - session/user come from session.id / user.id - cost, provider and tokens are normalized server-side on this path - the endpoint parses protobuf only Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
- rename exporter-level span filter to `dropSpanTypes`; `excludeSpanTypes` already exists on ObservabilityInstanceConfig and two identically-named knobs at two layers would be ambiguous. Document which layer owns what. - drop the tags rewrite rule: openlayer.tags, tags and mastra.tags were all probed and none is promoted to a row column, so a rewrite adds nothing. - add a live-test assertion for MODEL_STEP spans, which the capability guard will fire on since they carry mastra.model_step.input but no gen_ai messages. Measure the shape rather than discover it in a user's trace. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
Six TDD tasks implementing the design spec: message coercion, span rewriter, OTLP exporter + deps, OpenlayerExporter, example + docs, and a live end-to-end test. Self-review applied two fixes: - single super() call via a resolveExporterConfig helper, so the `name` field initializer does not depend on downlevel-class emit order across two branches - README step carries the actual copy rather than a description of it Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
Adds examples/mastra-tracing.ts exercising both Mastra root span types (agent run, workflow run) plus a tool call, session/user metadata via tracingOptions, and an explicit observability shutdown before exit. Documents the integration in README.md (installation, zero-config and explicit configuration, session/user attribution, composing with other exporters, the excludeSpanTypes/dropSpanTypes filtering split, and troubleshooting). examples/package.json now points its "openlayer" dependency at file:.. (its old ^0.22.2 pin predates this integration and could never resolve it) and adds the Mastra + ai-sdk/openai + zod deps needed to run the example.
createStep(agent) (equivalent at runtime to createStepFromAgent) runs the agent through Mastra's internal agent-step engine path, which threads the workflow's tracing context into the nested run. The prior hand-written step called agent.generate() directly from inside execute(), which has no way to receive a parent tracing context and so started its own independent trace — visible server-side as two sibling rows sharing a timestamp instead of one nested trace. Verified via the Openlayer rows API: the workflow row's steps tree now contains the agent run, its model calls, and its tool call as descendants, with no more sibling agent-run row for the same workflow execution.
Same runtime behavior as the prior createStep(agent) overload (createStep dispatches to createStepFromAgent internally for an Agent argument), but importing it explicitly by name reads clearly instead of relying on overload resolution. Re-verified via the Openlayer rows API that the workflow row still nests the agent run, its model calls, and its tool call as descendants.
Exercises the real path (Mastra agent -> OTLP export -> Openlayer ingest -> row read-back) and asserts non-empty input/output, session id, and nested tool-call hierarchy against a live row. Skips cleanly without credentials. Investigated adding SpanType.MODEL_STEP to DEFAULT_DROP_SPAN_TYPES (measured 4/7 steps in a live trace derive from it) but reverted: dropping the span does not reparent its children, so a live run lost the nested tool_call span and toolResult entirely. Documented the finding in index.ts and README instead of applying the drop. The live test's final assertion (openlayer_cost > 0) fails on purpose: Mastra's OpenAI Responses API calls report gen_ai.system as "openai.responses", which has no entry in Openlayer's cost table (verified directly against the live cost API), so cost never normalizes. This also affects examples/mastra-tracing.ts, which uses the same openai(...) call shape. Left failing rather than weakened so the gap stays visible; fixing it means normalizing the provider slug before export, out of scope here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
… price Mastra (via the AI SDK) reports gen_ai.provider.name as "openai.responses" for OpenAI Responses API calls, which has no entry in Openlayer's cost table even though the bare "openai" slug prices the same model - confirmed directly against https://llm-costs.openlayer.com/v1/costs/... (openai/gpt-4o-mini-2024-07-18 prices, openai.responses/... 404s). Add an explicit, individually-verified PROVIDER_SLUG_ALIASES map in spanRewriter.ts (openai.responses, openai.chat -> openai; anthropic.messages -> anthropic; google.generative-ai -> gemini) rather than a generic strip, since a naive dot-strip breaks other providers (e.g. "google" alone 404s where "gemini" prices). Unknown slugs pass through unchanged - a wrong alias is worse than a missing cost. Live test's openlayer_cost assertion now passes for real; unit tests added for the rewrite/passthrough/no-side-effects behavior. Also: document why SpanType.MODEL_STEP stays out of DEFAULT_DROP_SPAN_TYPES (dropping it orphans the nested tool_call span rather than reparenting it) in the live test's comments, note the --experimental-vm-modules requirement in the README troubleshooting section, and explain createStepFromAgent's fixed {prompt}/{text} contract in the example. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
Reviewer found a Critical false positive: openlayer_inputs is a list of
declared input-variable *names*, not content, so toContain('prompt')
passes whether or not the prompt column carries anything - the same bug
class already caught once for get_weather -> functionName.
Kept the existing openlayer_inputs check (still valid structurally) and
added a real content assertion on row['prompt'] (the actual user
message we sent). Audited every other assertion in the file for the
same shape-vs-content gap:
- openlayer_output: tightened to check the array is non-empty and its
first entry has non-empty content (an empty array is truthy and
stringifies to "[]", not "{}", so the prior check would pass against
it).
- steps hierarchy: tightened from toContain('functionName') /
toContain('toolResult') (field-name-only checks) to the literal
"functionName":"getWeather" pair plus "tempC" - the tool's actual
return value, which unlike "get_weather" does not appear anywhere
else in the trace.
- model/provider: audited and left as toBeTruthy() - nothing in this
path can produce a wrong-but-truthy value for either, and provider
is server-prettified so pinning it to a literal would be brittle.
Live test still fully passes with the tightened assertions verified
against a real fetched row.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
Fourth instance of the same structural-vs-content false positive in
this file: openlayer_output[0].content is a JSON-stringified envelope
({"text":...,"files":[]}), not the answer text. Checking only that the
envelope string is non-empty still passes against
'{"text":"","files":[]}' (22 characters) - exactly the empty-output
shape this test exists to catch.
Parse the envelope and assert on the real text: non-empty, and
contains "24" (the tool always returns tempC: 24, so a real answer
must mention it - same robustness class as the tempC check already in
the steps assertion). Fails loudly with the raw content if it is ever
not JSON, rather than silently passing past a parse failure.
Live test still fully passes against a real fetched row.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
…ocs accuracy Applies all 11 findings from the whole-branch "merge with fixes" review of the Mastra observability exporter, in one pass: - liftIdentity now accepts string | number ids (a numeric userId/sessionId was silently dropped before) - fixed a 5th false-positive test in mastraGenaiMessages.test.ts (role matched the default, and the assertion compared against the input's own reference) - documented and tested OPENLAYER_OTEL_ENDPOINT - moved `exporter:` after `...config` in index.ts's spread to remove a latent footgun the Omit<> type only blocks at compile time - replaced a misleading otlpExporter.ts comment with the verified isolation argument (each OtelExporter owns its own SpanConverter / fresh attributes per convertSpan() call) - moved prototype-spy restores into afterEach in two test files so a mid-test failure can't leak spies into later tests - documented why client_tool_call is absent from TOOL_SPAN_TYPES - fixed the README install line/peer-dependency count to match package.json - generalized the dropSpanTypes/MODEL_STEP orphaning note into the general "dropping a span type does not reparent its children" rule - corrected a false claim in the design spec about server-side cost normalization, and referenced the PROVIDER_SLUG_ALIASES mitigation - added a console.debug miss signal for unmapped dotted provider slugs, so a silent $0-cost defect can no longer pass unnoticed Verified: unit tests (48/48), tsc --noEmit, eslint, prettier, and the live end-to-end test against the real Openlayer OTLP endpoint all pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
The design spec's Goal snippet (the first code a reader sees) and the plan's Global Constraints both still showed the old, non-resolving subpath `openlayer/integrations/mastra`. The build regenerates dist/package.json's exports map from a directory scan and discards the hand-authored `./integrations/mastra` alias, so the real, shipped path is `openlayer/lib/integrations/mastra` (already correct in README.md, examples/mastra-tracing.ts, and src/lib/integrations/mastra/index.ts). - design spec: fixed the one occurrence in the Goal code block in place. - plan: left the plan's code blocks untouched (it's a historical record of what was actually dispatched to implementers) and instead added one line to Global Constraints noting the correction and pointing to the now-accurate sources. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
tests/integrations/mastraExporter.live.test.ts previously statically imported @ai-sdk/openai (and other Mastra packages) at module scope. @ai-sdk/openai ships ESM-only with no `require` export, so a plain `npx jest tests/integrations/mastraExporter.live.test.ts` (what ./scripts/test runs, with no NODE_OPTIONS) crashed at load time with "Cannot use import statement outside a module" before it.skip could ever run — breaking the no-credentials skip guarantee CI depends on. Fix: moved every import of @ai-sdk/openai, @mastra/core, @mastra/core/agent, @mastra/core/tools, @mastra/observability, and zod from static top-level imports into lazy `await import(...)` calls inside the test body. it.skip never executes the body, so the dynamic import never happens, and the file now loads and skips cleanly with no flag and no credentials — verified directly (not assumed; the file's own prior comment claiming a lazy import() would hit the same wall was checked and found false). The source-relative OpenlayerExporter import is unaffected: it already loads fine under plain CJS Jest (mastraExporter.test.ts proves this). --experimental-vm-modules is still required to actually run the live assertions once credentials are set. Deliberately not adding the flag to scripts/test or jest.config.ts, which are shared infrastructure for the whole SDK test suite. Updated the file's header comment and the README troubleshooting entry to describe the new, narrower flag requirement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
These are process artifacts, not shippable SDK documentation. The design rationale they carried — the measured OTLP mapping rules, why the rewrite shim exists, and the notable decisions — is summarised in the PR body and in the module doc comments, which is where a reader of this repo will actually look. Matches existing repo practice: src/lib/integrations/claudeAgentSdk.ts already cites a docs/superpowers/specs/ path that is not committed here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
OpenlayerExporter, a first-class Mastra observability exporter, so Mastraapplications can send agent, workflow, model, and tool traces to Openlayer with a
small configuration snippet.
Closes OPEN-12306.
With
OPENLAYER_API_KEYandOPENLAYER_INFERENCE_PIPELINE_IDset, that is the wholeintegration.
Why a rewrite shim was needed
@mastra/otel-exporteralready emits OTel GenAI semconv v1.38 — the conventionOpenlayer's OTLP ingest reads. But it only sets
gen_ai.input.messages/gen_ai.output.messagesonMODEL_GENERATIONspans. Openlayer builds a row's input andoutput from the root span, which instead carries
mastra.<type>.input/.outputand is ignored.
Posting a Mastra-shaped trace unmodified produced correct hierarchy, cost, tokens,
latency and error status — and
openlayer_output: {}withopenlayer_inputs: [].So this exporter is a gap-filler, not a translator.
@mastra/arizeconverts gen_ai →OpenInference and
@mastra/langfuseconverts gen_ai →langfuse.*; we convert nothingand only fill in what Mastra omits. The mapping rules were established by probing the
live endpoint, not inferred:
gen_ai.input.messages+gen_ai.output.messages(1.38parts)gen_ai.prompt/gen_ai.completioninput.value/output.valuetraceloop.entity.input/.outputOpenInference is deliberately not targeted — the issue lists it as a reference, but
it was measured to yield empty rows against Openlayer.
What ships
src/lib/integrations/mastra/genaiMessages.tspartsmessagessrc/lib/integrations/mastra/spanRewriter.tssrc/lib/integrations/mastra/otlpExporter.tsOTLPTraceExportersubclass applying the rewrite inexport()src/lib/integrations/mastra/index.tsOpenlayerExporter, config resolutionsrc/lib/integrations/mastra.tsexamples/mastra-tracing.tsDependencies are optional peers only — nothing added to
dependencies, so consumerswho don't use Mastra pay nothing.
Notable decisions
openlayer/lib/integrations/mastra.scripts/utils/postprocess-files.cjsregenerates
dist/package.json'sexportsfrom a directory scan and discardshand-authored subpaths, so
openlayer/integrations/mastrawould 404 on a real install.A flat facade module makes the module reachable via the generated
./lib/*wildcard,matching the convention every existing example already uses.
This also means the shipped
./integrations/claude-agent-sdkexport is dead today —latent only because it is undocumented. Worth a separate issue.
gen_ai.provider.name = openai.responses,which has no Openlayer cost-table entry, so every user following the docs would have
gotten silent $0 cost.
PROVIDER_SLUG_ALIASESmaps verified dotted slugs to realones; unknown values pass through untouched, and a debug log fires on an unmapped dotted
slug so a future miss is visible rather than silent.
MODEL_CHUNKdropped by default, viadropSpanTypes— Mastra emits one span perstreaming chunk. Deliberately not named
excludeSpanTypes, which already exists onelayer up in Mastra's own config.
MODEL_STEPspans are kept even though they are ~57% of steps: dropping a span typedoes not reparent its children, and dropping these silently lost the tool-call span.
Testing
negative cases), and config resolution.
asserts on the row fetched back: non-empty input content, non-empty output text,
cost > 0, tokens, model, provider, session id, and the tool call nested in the steps tree.
The live test is the only guard against the empty-I/O defect this integration exists to
fix — every other test uses synthetic data and cannot see it. Five structural-vs-content
false positives were found and fixed while building it, including assertions that would
have passed against an empty row.
Running it live requires
NODE_OPTIONS=--experimental-vm-modules(@ai-sdk/openaiisESM-only); its imports are dynamic so the suite still skips cleanly under a plain
yarn test.Pre-existing issues found, not fixed here
./integrations/claude-agent-sdkis unreachable in the published package (sameexportscodegen cause as above).tests/openai-tracer.test.tsfails onmain— a function-call output assertion inopenAiTracer.ts, untouched by this branch.yarn testcannot start its mock server in a fresh checkout:.stats.ymlcarries noOpenAPI spec URL, so the
tests/api-resources/*suites cannot run.🤖 Generated with Claude Code
https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp