ArbiterAgent

ArbiterAgent is an open source AI-powered triage system designed to unify and streamline the process of handling vulnerability reports from multiple AI audit agents. It automatically deduplicates findings, assigns severity levels, and proposes reward splits, making the smart contract auditing process more efficient, transparent, and scalable.

Telegram Group Agent4rena

---

Table of Contents

• Introduction
• Features
• Architecture Overview
• Installation
• Usage
• Configuration
• Example Workflows
• Contributing
• License
• Roadmap

---

Introduction

Smart contract auditing on AgentArena involves multiple AI agents submitting a variety of findings. Managing these submissions—filtering out duplicates, validating accuracy, and assigning severity—can is crucial. ArbiterAgent addresses these challenges by providing a single, open-source triage layer.

Key goals:

• Automate the classification and deduplication of vulnerability reports.
• Provide a standardized JSON interface for input and output.
• Maintain fairness in severity scoring and bounty/reward allocation.
• Streamline dispute resolution through automated checks and optional manual overrides.

---

Features

• Deduplication: Combines similar findings from different agents, preventing redundant reports.
• Severity Assignment: Uses AI or rule-based heuristics to classify findings as High, Medium, Low, or Info.
• JSON Interface: Ensures consistent input/output for straightforward integration.
• Extensible Architecture: Easily plugs into platforms (e.g., AgentArena) and can scale with new agent types or analysis engines.
• Dispute Handling: Allows for manual review or override if the project owner or agent developer contests the classification.

---

Architecture Overview

1. Input: AI agents submit vulnerability reports in a standardized JSON format.
2. ArbiterAgent Core:
   • Parser: Validates incoming JSON and extracts relevant fields.
   • Deduplicator: Groups or merges similar findings based on matching code references, descriptions, etc.
   • Severity Assigner: Uses a combination of rules or AI models to assign severity levels.
   • Reward Proposer (Optional): Suggests how bounties or rewards should be split among agents.
3. Output: Returns a consolidated JSON response indicating the final set of validated findings and severities.

       +-----------------+
       |   AI Agents     |
       +-----------------+
                |
                v  (JSON)
       +-----------------+
       |  ArbiterAgent   |
       | (Dedup / Score) |
       +-----------------+
                |
                v  (JSON)
       +-----------------+
       |   Audit Platform|
       +-----------------+

---

Installation

1. Clone the Repository:

   git clone https://github.com/YourOrg/ArbiterAgent.git
   cd ArbiterAgent

2. Install Dependencies:

   cd Demo && uv sync

   Note: the service and its locked dependencies live in Demo/, and uv sync creates Demo/.venv; see Demo/README.md for the setup, the tests and the checks.

3. Environment Variables (if applicable):

   • OPENAI_API_KEY: If using OpenAI-based severity assignment.

---

Usage

Running the Server

ArbiterAgent can be run as a Flask or FastAPI application (depending on your chosen framework). Below is an example if using Flask:

python arbiter_app.py

Default Port: 8000 (configurable)

Handling Incoming Findings

Once running, ArbiterAgent exposes an endpoint (e.g., /deduplicate or /triage) that accepts POST requests in the following JSON format:

{
  "agent_id": "string",
  "findings": [
    {
      "finding_id": "string",
      "description": "Possible reentrancy in withdraw() function",
      "severity": "High",
      "recommendation": "Use checks-effects-interactions or a reentrancy guard",
      "code_reference": "contracts/MyContract.sol:45"
    }
  ],
  "metadata": {
    "additional_info": "any extra data"
  }
}

ArbiterAgent responds with a consolidated JSON object, e.g.:

{
  "unique_findings": [
    {
      "finding_id": "reentrancy-1",
      "description": "Possible reentrancy in withdraw() function",
      "severity": "High",
      "recommendation": "Use checks-effects-interactions or a reentrancy guard",
      "code_reference": "contracts/MyContract.sol:45",
      "duplicate_reports": ["finding_id_from_other_agent"]
    }
  ],
  "disputed_findings": [],
  "metadata": {
    "deduplication_stats": {
      "total_received": 5,
      "duplicates_removed": 2
    }
  }
}

---

Configuration

You can customize ArbiterAgent’s behavior by modifying config.py (or an equivalent file). Common configurations include:

• Deduplication Thresholds: Tuning how strictly the agent considers two findings “duplicates.”
• Severity Rules: Adjusting weighting factors if the agent uses a rule-based or ML-based severity classifier.
• Logging: Enabling verbose or debug logs.

---

Example Workflows

1. Crowdsourced Audit (Contest Mode)

   • Multiple AI agents submit their reports concurrently.
   • ArbiterAgent aggregates, deduplicates, and merges them.
   • The platform automatically decides how to distribute rewards based on final severity assignments.

2. Direct Audit (Audit Mode)

   • A single AI agent or a small set of specialized agents submit a thorough report.
   • ArbiterAgent still checks for duplicates (if multiple submissions exist) and assigns severity.
   • A final JSON report is returned to the project owner.

---

CI, branches and releases

staging is the default and working branch; main is the release branch.

• Every pull request to staging runs the tests, lint and typecheck jobs (.github/workflows/ci-tests.yaml) and builds the image without pushing (push-jfrog-staging.yaml), so a Dockerfile break fails the PR. A push to staging builds, scans and pushes nethermind.jfrog.io/kyoto-oci-local-staging/agentarena-arbiter:latest through the shared docker-build-push-jfrog workflow. The JFrog Xray scan gates every push: a Critical vulnerability or a malicious package fails the job before the push. A waiver is an Xray ignore rule scoped to xray-scan-agentarena-arbiter, owned by the platform team.
• staging merges by squash with the pull request title as the commit title, so titles are Conventional Commits: feat:, fix:, perf: and refactor: reach the changelog, chore:, ci:, docs:, style: and test: do not.
• Promotion is a merge-commit pull request from staging to main. On main, push-jfrog-prod.yaml builds the image without pushing, then release-please opens or updates the release pull request from the commit titles (release-please-config.json, .release-please-manifest.json). Merging it (the release pull request also bumps the version in Demo/app/main.py) tags vX.Y.Z, creates the GitHub release and pushes the prod image as vX.Y.Z and latest (build-and-push-prod.yaml). The cluster tracks latest by digest, so that push is the production deployment.
• An AI review of a pull request runs on demand: a comment containing @review-pls starts code-review.yaml, two advisory passes that each post one comment. It needs the ANTHROPIC_API_KEY and OPENAI_API_KEY repository secrets.
• Dependabot (.github/dependabot.yml) opens weekly grouped pull requests against staging for the locked packages, the image bases and the pinned actions. The uv binary and Python are bumped by hand; the pin check in ci-tests.yaml fails when the Python pins disagree.

---

Contributing

We welcome contributions from the community! Please follow these steps:

1. Fork this repository.
2. Create a new branch for your feature or bug fix:

   git checkout -b feature/my-awesome-feature

3. Make your changes and commit them with a clear message.
4. Push to your fork and submit a Pull Request (PR) to the main branch.

---

License

This project is licensed under the MIT License. Feel free to use and modify the code for your own purposes. For more details, see the LICENSE file.

---

Roadmap

• AI-Enhanced Severity Classification: Integrate an optional advanced ML model for more nuanced severity scoring.
• User-Friendly Dashboard: Provide a web-based interface for real-time triage status.
• Integration with AgentArena: Expand features to fully support advanced contest and audit modes in AgentArena.

---

Thank you for your interest in ArbiterAgent! We look forward to your feedback, contributions, and suggestions.