Skip to content

feat(mastra): add Openlayer observability exporter for Mastra - #222

Open
viniciusdsmello wants to merge 19 commits into
mainfrom
vini/open-12306-integration-add-mastra-observability-integration
Open

feat(mastra): add Openlayer observability exporter for Mastra#222
viniciusdsmello wants to merge 19 commits into
mainfrom
vini/open-12306-integration-add-mastra-observability-integration

Conversation

@viniciusdsmello

Copy link
Copy Markdown
Contributor

Adds OpenlayerExporter, a first-class Mastra observability exporter, so Mastra
applications can send agent, workflow, model, and tool traces to Openlayer with a
small configuration snippet.

Closes OPEN-12306.

import { Mastra } from '@mastra/core';
import { Observability } from '@mastra/observability';
import { OpenlayerExporter } from 'openlayer/lib/integrations/mastra';

export const mastra = new Mastra({
  observability: new Observability({
    configs: {
      openlayer: {
        serviceName: 'my-service',
        exporters: [new OpenlayerExporter()],
      },
    },
  }),
});

With OPENLAYER_API_KEY and OPENLAYER_INFERENCE_PIPELINE_ID set, that is the whole
integration.

Why a rewrite shim was needed

@mastra/otel-exporter already emits OTel GenAI semconv v1.38 — the convention
Openlayer's OTLP ingest reads. But it only sets gen_ai.input.messages /
gen_ai.output.messages on MODEL_GENERATION spans. Openlayer builds a row's input and
output from the root span, which instead carries mastra.<type>.input / .output
and is ignored.

Posting a Mastra-shaped trace unmodified produced correct hierarchy, cost, tokens,
latency and error status — and openlayer_output: {} with openlayer_inputs: [].

So this exporter is a gap-filler, not a translator. @mastra/arize converts gen_ai →
OpenInference and @mastra/langfuse converts gen_ai → langfuse.*; we convert nothing
and only fill in what Mastra omits. The mapping rules were established by probing the
live endpoint, not inferred:

Root-span shape Row input / output
gen_ai.input.messages + gen_ai.output.messages (1.38 parts) ✅ populated
gen_ai.prompt / gen_ai.completion ❌ empty
gen_ai span events ❌ empty
OpenInference input.value / output.value ❌ empty
Traceloop traceloop.entity.input / .output ❌ empty

OpenInference is deliberately not targeted — the issue lists it as a reference, but
it was measured to yield empty rows against Openlayer.

What ships

File Purpose
src/lib/integrations/mastra/genaiMessages.ts Pure: coerce any Mastra value into semconv 1.38 parts messages
src/lib/integrations/mastra/spanRewriter.ts Pure: every attribute-mapping rule
src/lib/integrations/mastra/otlpExporter.ts OTLPTraceExporter subclass applying the rewrite in export()
src/lib/integrations/mastra/index.ts OpenlayerExporter, config resolution
src/lib/integrations/mastra.ts Flat facade so the module is importable (see below)
examples/mastra-tracing.ts Runnable agent + workflow example

Dependencies are optional peers only — nothing added to dependencies, so consumers
who don't use Mastra pay nothing.

Notable decisions

  • Import path is openlayer/lib/integrations/mastra. scripts/utils/postprocess-files.cjs
    regenerates dist/package.json's exports from a directory scan and discards
    hand-authored subpaths, so openlayer/integrations/mastra would 404 on a real install.
    A flat facade module makes the module reachable via the generated ./lib/* wildcard,
    matching the convention every existing example already uses.
    This also means the shipped ./integrations/claude-agent-sdk export is dead today
    latent only because it is undocumented. Worth a separate issue.
  • Provider-slug normalization. Mastra reports gen_ai.provider.name = openai.responses,
    which has no Openlayer cost-table entry, so every user following the docs would have
    gotten silent $0 cost. PROVIDER_SLUG_ALIASES maps verified dotted slugs to real
    ones; unknown values pass through untouched, and a debug log fires on an unmapped dotted
    slug so a future miss is visible rather than silent.
  • MODEL_CHUNK dropped by default, via dropSpanTypes — Mastra emits one span per
    streaming chunk. Deliberately not named excludeSpanTypes, which already exists one
    layer up in Mastra's own config.
  • MODEL_STEP spans are kept even though they are ~57% of steps: dropping a span type
    does not reparent its children, and dropping these silently lost the tool-call span.

Testing

  • 48 unit tests across three suites — message coercion, every rewrite rule (including the
    negative cases), and config resolution.
  • A live end-to-end test, skipped without credentials, that publishes a real trace and
    asserts on the row fetched back: non-empty input content, non-empty output text,
    cost > 0, tokens, model, provider, session id, and the tool call nested in the steps tree.

The live test is the only guard against the empty-I/O defect this integration exists to
fix — every other test uses synthetic data and cannot see it. Five structural-vs-content
false positives were found and fixed while building it, including assertions that would
have passed against an empty row.

Running it live requires NODE_OPTIONS=--experimental-vm-modules (@ai-sdk/openai is
ESM-only); its imports are dynamic so the suite still skips cleanly under a plain
yarn test.

Pre-existing issues found, not fixed here

  • ./integrations/claude-agent-sdk is unreachable in the published package (same
    exports codegen cause as above).
  • tests/openai-tracer.test.ts fails on main — a function-call output assertion in
    openAiTracer.ts, untouched by this branch.
  • yarn test cannot start its mock server in a fresh checkout: .stats.yml carries no
    OpenAPI spec URL, so the tests/api-resources/* suites cannot run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp

viniciusdsmello and others added 19 commits August 27, 2026 13:40
…rter

Design for OPEN-12306: an OpenlayerExporter for Mastra's observability
system, shipped as the `openlayer/integrations/mastra` subpath export.

Every mapping claim in the spec was verified against the live Openlayer
OTLP endpoint rather than inferred:

- row input/output is built only from gen_ai.input.messages /
  gen_ai.output.messages in semconv 1.38 `parts` shape; OpenInference,
  traceloop, gen_ai events and gen_ai.prompt/completion all yield empty I/O
- Mastra emits that pair only on MODEL_GENERATION spans, so agent_run and
  workflow_run roots land with empty input/output — the defect the
  exporter's rewrite shim exists to fix
- tool spans map natively and must not be rewritten
- session/user come from session.id / user.id
- cost, provider and tokens are normalized server-side on this path
- the endpoint parses protobuf only

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
- rename exporter-level span filter to `dropSpanTypes`; `excludeSpanTypes`
  already exists on ObservabilityInstanceConfig and two identically-named
  knobs at two layers would be ambiguous. Document which layer owns what.
- drop the tags rewrite rule: openlayer.tags, tags and mastra.tags were all
  probed and none is promoted to a row column, so a rewrite adds nothing.
- add a live-test assertion for MODEL_STEP spans, which the capability guard
  will fire on since they carry mastra.model_step.input but no gen_ai
  messages. Measure the shape rather than discover it in a user's trace.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
Six TDD tasks implementing the design spec: message coercion, span
rewriter, OTLP exporter + deps, OpenlayerExporter, example + docs, and a
live end-to-end test.

Self-review applied two fixes:
- single super() call via a resolveExporterConfig helper, so the `name`
  field initializer does not depend on downlevel-class emit order across
  two branches
- README step carries the actual copy rather than a description of it

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
Adds examples/mastra-tracing.ts exercising both Mastra root span types
(agent run, workflow run) plus a tool call, session/user metadata via
tracingOptions, and an explicit observability shutdown before exit.
Documents the integration in README.md (installation, zero-config and
explicit configuration, session/user attribution, composing with other
exporters, the excludeSpanTypes/dropSpanTypes filtering split, and
troubleshooting).

examples/package.json now points its "openlayer" dependency at
file:.. (its old ^0.22.2 pin predates this integration and could never
resolve it) and adds the Mastra + ai-sdk/openai + zod deps needed to
run the example.
createStep(agent) (equivalent at runtime to createStepFromAgent) runs
the agent through Mastra's internal agent-step engine path, which
threads the workflow's tracing context into the nested run. The prior
hand-written step called agent.generate() directly from inside
execute(), which has no way to receive a parent tracing context and
so started its own independent trace — visible server-side as two
sibling rows sharing a timestamp instead of one nested trace.

Verified via the Openlayer rows API: the workflow row's steps tree now
contains the agent run, its model calls, and its tool call as
descendants, with no more sibling agent-run row for the same workflow
execution.
Same runtime behavior as the prior createStep(agent) overload
(createStep dispatches to createStepFromAgent internally for an Agent
argument), but importing it explicitly by name reads clearly instead
of relying on overload resolution. Re-verified via the Openlayer rows
API that the workflow row still nests the agent run, its model calls,
and its tool call as descendants.
Exercises the real path (Mastra agent -> OTLP export -> Openlayer ingest ->
row read-back) and asserts non-empty input/output, session id, and nested
tool-call hierarchy against a live row. Skips cleanly without credentials.

Investigated adding SpanType.MODEL_STEP to DEFAULT_DROP_SPAN_TYPES (measured
4/7 steps in a live trace derive from it) but reverted: dropping the span
does not reparent its children, so a live run lost the nested tool_call span
and toolResult entirely. Documented the finding in index.ts and README
instead of applying the drop.

The live test's final assertion (openlayer_cost > 0) fails on purpose:
Mastra's OpenAI Responses API calls report gen_ai.system as
"openai.responses", which has no entry in Openlayer's cost table (verified
directly against the live cost API), so cost never normalizes. This also
affects examples/mastra-tracing.ts, which uses the same openai(...) call
shape. Left failing rather than weakened so the gap stays visible; fixing it
means normalizing the provider slug before export, out of scope here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
… price

Mastra (via the AI SDK) reports gen_ai.provider.name as "openai.responses"
for OpenAI Responses API calls, which has no entry in Openlayer's cost
table even though the bare "openai" slug prices the same model -
confirmed directly against https://llm-costs.openlayer.com/v1/costs/...
(openai/gpt-4o-mini-2024-07-18 prices, openai.responses/... 404s).

Add an explicit, individually-verified PROVIDER_SLUG_ALIASES map in
spanRewriter.ts (openai.responses, openai.chat -> openai;
anthropic.messages -> anthropic; google.generative-ai -> gemini) rather
than a generic strip, since a naive dot-strip breaks other providers
(e.g. "google" alone 404s where "gemini" prices). Unknown slugs pass
through unchanged - a wrong alias is worse than a missing cost.

Live test's openlayer_cost assertion now passes for real; unit tests
added for the rewrite/passthrough/no-side-effects behavior. Also:
document why SpanType.MODEL_STEP stays out of DEFAULT_DROP_SPAN_TYPES
(dropping it orphans the nested tool_call span rather than reparenting
it) in the live test's comments, note the --experimental-vm-modules
requirement in the README troubleshooting section, and explain
createStepFromAgent's fixed {prompt}/{text} contract in the example.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
Reviewer found a Critical false positive: openlayer_inputs is a list of
declared input-variable *names*, not content, so toContain('prompt')
passes whether or not the prompt column carries anything - the same bug
class already caught once for get_weather -> functionName.

Kept the existing openlayer_inputs check (still valid structurally) and
added a real content assertion on row['prompt'] (the actual user
message we sent). Audited every other assertion in the file for the
same shape-vs-content gap:

- openlayer_output: tightened to check the array is non-empty and its
  first entry has non-empty content (an empty array is truthy and
  stringifies to "[]", not "{}", so the prior check would pass against
  it).
- steps hierarchy: tightened from toContain('functionName') /
  toContain('toolResult') (field-name-only checks) to the literal
  "functionName":"getWeather" pair plus "tempC" - the tool's actual
  return value, which unlike "get_weather" does not appear anywhere
  else in the trace.
- model/provider: audited and left as toBeTruthy() - nothing in this
  path can produce a wrong-but-truthy value for either, and provider
  is server-prettified so pinning it to a literal would be brittle.

Live test still fully passes with the tightened assertions verified
against a real fetched row.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
Fourth instance of the same structural-vs-content false positive in
this file: openlayer_output[0].content is a JSON-stringified envelope
({"text":...,"files":[]}), not the answer text. Checking only that the
envelope string is non-empty still passes against
'{"text":"","files":[]}' (22 characters) - exactly the empty-output
shape this test exists to catch.

Parse the envelope and assert on the real text: non-empty, and
contains "24" (the tool always returns tempC: 24, so a real answer
must mention it - same robustness class as the tempC check already in
the steps assertion). Fails loudly with the raw content if it is ever
not JSON, rather than silently passing past a parse failure.

Live test still fully passes against a real fetched row.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
…ocs accuracy

Applies all 11 findings from the whole-branch "merge with fixes" review of the
Mastra observability exporter, in one pass:

- liftIdentity now accepts string | number ids (a numeric userId/sessionId was
  silently dropped before)
- fixed a 5th false-positive test in mastraGenaiMessages.test.ts (role matched
  the default, and the assertion compared against the input's own reference)
- documented and tested OPENLAYER_OTEL_ENDPOINT
- moved `exporter:` after `...config` in index.ts's spread to remove a latent
  footgun the Omit<> type only blocks at compile time
- replaced a misleading otlpExporter.ts comment with the verified isolation
  argument (each OtelExporter owns its own SpanConverter / fresh attributes
  per convertSpan() call)
- moved prototype-spy restores into afterEach in two test files so a mid-test
  failure can't leak spies into later tests
- documented why client_tool_call is absent from TOOL_SPAN_TYPES
- fixed the README install line/peer-dependency count to match package.json
- generalized the dropSpanTypes/MODEL_STEP orphaning note into the general
  "dropping a span type does not reparent its children" rule
- corrected a false claim in the design spec about server-side cost
  normalization, and referenced the PROVIDER_SLUG_ALIASES mitigation
- added a console.debug miss signal for unmapped dotted provider slugs, so a
  silent $0-cost defect can no longer pass unnoticed

Verified: unit tests (48/48), tsc --noEmit, eslint, prettier, and the live
end-to-end test against the real Openlayer OTLP endpoint all pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
The design spec's Goal snippet (the first code a reader sees) and the plan's
Global Constraints both still showed the old, non-resolving subpath
`openlayer/integrations/mastra`. The build regenerates dist/package.json's
exports map from a directory scan and discards the hand-authored
`./integrations/mastra` alias, so the real, shipped path is
`openlayer/lib/integrations/mastra` (already correct in README.md,
examples/mastra-tracing.ts, and src/lib/integrations/mastra/index.ts).

- design spec: fixed the one occurrence in the Goal code block in place.
- plan: left the plan's code blocks untouched (it's a historical record of
  what was actually dispatched to implementers) and instead added one line
  to Global Constraints noting the correction and pointing to the
  now-accurate sources.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
tests/integrations/mastraExporter.live.test.ts previously statically
imported @ai-sdk/openai (and other Mastra packages) at module scope.
@ai-sdk/openai ships ESM-only with no `require` export, so a plain
`npx jest tests/integrations/mastraExporter.live.test.ts` (what
./scripts/test runs, with no NODE_OPTIONS) crashed at load time with
"Cannot use import statement outside a module" before it.skip could ever
run — breaking the no-credentials skip guarantee CI depends on.

Fix: moved every import of @ai-sdk/openai, @mastra/core, @mastra/core/agent,
@mastra/core/tools, @mastra/observability, and zod from static top-level
imports into lazy `await import(...)` calls inside the test body. it.skip
never executes the body, so the dynamic import never happens, and the file
now loads and skips cleanly with no flag and no credentials — verified
directly (not assumed; the file's own prior comment claiming a lazy
import() would hit the same wall was checked and found false). The
source-relative OpenlayerExporter import is unaffected: it already loads
fine under plain CJS Jest (mastraExporter.test.ts proves this).

--experimental-vm-modules is still required to actually run the live
assertions once credentials are set. Deliberately not adding the flag to
scripts/test or jest.config.ts, which are shared infrastructure for the
whole SDK test suite. Updated the file's header comment and the README
troubleshooting entry to describe the new, narrower flag requirement.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
These are process artifacts, not shippable SDK documentation. The design
rationale they carried — the measured OTLP mapping rules, why the rewrite
shim exists, and the notable decisions — is summarised in the PR body and
in the module doc comments, which is where a reader of this repo will
actually look.

Matches existing repo practice: src/lib/integrations/claudeAgentSdk.ts
already cites a docs/superpowers/specs/ path that is not committed here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AX7bmWUeckGwETCQMyt8sp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant