midojo-suites

Example benchmark suites for the MiDojo agent red-teaming framework.

Suites

• minibank — a banking-domain suite with 12 MCP tools, 4 user tasks, and 8 injection tasks targeting fund transfers, PII exfiltration, compliance bypass, and privilege abuse.
• weather — a minimal weather suite, useful as a starting point for writing your own suite.

Usage

pip install git+https://github.com/asago-ai/midojo.git
pip install git+https://github.com/asago-ai/midojo-suites.git

# start the servers
midojo-serve --suite midojo_suites.minibank
minibank-real-mcp-serve --port 8083
minibank-fake-mcp-serve --port 8082 --upstream-url http://localhost:8083/mcp
minibank-a2a-agent --litellm-api-url <url> --litellm-api-key <key> --litellm-model <model>

# run the ladder
midojo-run \
  --suite midojo_suites.minibank \
  --protocol a2a \
  --agent-url http://localhost:8000 \
  -ut user_task_1 \
  -it ladder_a -it ladder_b -it ladder_c -it garak_hijack

Writing your own suite

Use the minibank suite as a reference. Your suite's __init__.py needs two things:

from pathlib import Path
from midojo.yaml_task_suite import YAMLTaskSuite

task_suite = YAMLTaskSuite("my_suite", suite_yaml_path=Path(__file__).parent / "suite.yaml")

# optional — only used by --protocol ogx and --protocol openai
SYSTEM_MESSAGE = "You are a ..."

Then reference it by dotted path: --suite my_package.my_suite.

Roadmap

This repo is the long-term home for MiDojo suite authoring:

• Community suites — domain-specific suites (HR, finance, code assistant, etc.) shared as inspiration and reference patterns. Since suites require a real MCP server backed by your own environment, they're starting points to adapt rather than drop-in solutions.
• Suite authoring tooling (midojo-gen) — a guided CLI that takes your existing tools and generates a suite skeleton: parse your tools.py and produce suite.yaml + fake_mcp.py ready to run
• Translator — convert artifacts from other red-teaming frameworks (e.g. agent-policy-redteam) directly into MiDojo suites with a single command (poc implemented here)

The goal is to make "bring your own suite" a 5-minute experience rather than a day of reading documentation.