Guardrail calibration datasets used to build and evaluate the safety layer of Sonder, SonderMind's LLM-powered chat-based care companion. We're releasing them for teams building AI in mental health and adjacent sensitive domains — safety testing infrastructure should be a shared foundation, not a competitive moat.
Read the technical blog post for the background.
Sensitive content: these scenarios contain realistic depictions of crisis, trauma, self-harm, and abuse. Content warnings are in each file header.
input_guardrails_scenarios.yaml(200 scenarios) — testing whether an input guardrail detects problematic user messages: self-harm, crisis, abuse disclosure, and more. Each scenario encodes whether the guardrail should fall into one of three tiers:"No Issue","Show Resources & Continue"(e.g., trauma disclosures), or"Static Block"(e.g. active crisis, security threats).output_guardrails_scenarios.yaml(100 scenarios) — testing whether an output guardrail catches harmful AI responses: clinical overreach, inappropriate suggestions, hallucinations, bias/stigma, and security issues.
Every scenario is a single or multi-turn conversation with a boolean expected result and category labels. Output scenarios also include a reason and feedback describing a corrected response. See the file headers for the full schema.
Calibration benchmarks, regression tests on every model/prompt change, and per-category coverage analysis. The YAML is simple and framework-agnostic.
These cover the automated guardrail calibration layer only. They were developed in close collaboration with licensed clinicians, but they are not a comprehensive safety suite. Our broader program also includes production data annotation, clinical review, and red-teaming.
- Why we're sharing the safety tests behind Sonder
- Open-sourcing SonderMind's guardrail calibration datasets
These datasets are provided "as is" without warranty of any kind. SonderMind makes no representations regarding their suitability for any particular use. They are a starting point for safety calibration, not a complete safety solution. Use is at your own risk and is not a substitute for independent safety testing, clinical review, or professional judgment.
@dataset{sondermind_guardrails_2026,
title = {SonderMind AI Coach Safety Guardrail Datasets},
author = {SonderMind},
year = {2026},
version = {1.0.0},
url = {https://github.com/sondermind/sonder-guardrail-evals}
}- MindEval — Sword Health's multi-turn benchmark for LLMs in mental health support
- HealthBench — OpenAI's evaluation framework for health-related AI systems
- VERA-MH (Spring Health) — a simulated multi-turn conversational benchmark for mental health LLMs, focused on safety