Multimodal LLM Message Router

🏆 Achievement: Top 14% Global Finish (Ranked 278 / 1,983) with a score of 64.7/100 in the HackerRank Orchestrate AI Hackathon (August 2026).

🚀 Project Overview

This repository contains my submission for the HackerRank Orchestrate Hackathon. The objective was to build a highly resilient, multimodal AI system capable of analyzing incoming messages (text, image, audio) and intelligently deciding whether to:

• notify: Interrupt the user immediately (e.g., highly urgent or personal messages).
• digest: Save for later (e.g., standard promotional or non-urgent updates).
• mute: Silently block the message (e.g., spam, phishing, or high-risk scams).

The routing decisions were based on historical user interactions, behavioral context, group metadata, and strict numerical confidence boundaries.

📊 Evaluation Metrics & My Results

Submissions were evaluated via a strict hidden grading script (evaluate.py) across four distinct categories. I achieved a total score of 64.7 / 100, placing me in the Top 14% globally.

1. Code Quality (21.6 / 30)
   • Graded on architecture, modularity, and cleanliness.
   • My solution strictly separated concerns (main.py, llm_router.py, rules.py, evidence.py) for a highly maintainable, production-ready script.
2. Output CSV Validation (18.0 / 30)
   • Graded against a hidden test set containing complex, adversarial edge cases.
   • I achieved 96.67% accuracy on the local dev set and guaranteed exactly zero schema/type errors by clamping the LLM's raw output programmatically.
3. AI Judge Interview (16.8 / 30)
   • An automated AI evaluated the architectural decisions and reasoning used to build the pipeline.
4. Chat Transcript Logging (8.3 / 10)
   • Graded on process documentation and AI-agent collaboration tracking via a strict logging format.

🧠 Technical Architecture & Key Features

1. Resilient LLM Fallback Pipeline

To survive extreme rate-limiting and API instability during processing, the llm_router.py automatically rotates through 10 distinct Gemini 1.5 Flash API keys. If Gemini fails entirely, the system gracefully falls back to a Groq LLaMA-3 model, ensuring the batch pipeline never crashes on a single rate-limit error.

2. Heuristic Pre-Routing (Safety Engine)

Before spending API credits on an LLM inference, the rules.py engine scans the message for high-risk phishing tactics, suspicious URLs, and urgent manipulative language (e.g., "blocked in 2 hours", "verify now"). This instantly tags the payload with severity flags to anchor the LLM's logic.

3. Programmatic Confidence Clamping

The Hackathon evaluation script required mathematically rigid, non-overlapping boundaries for the final confidence scores:

• notify: [0.85, 0.91]
• mute: [0.81, 0.87]
• digest: [0.78, 0.84]

Because LLMs are probabilistic generators and occasionally hallucinate raw numerical boundaries, I built a deterministic clamp in Python. The LLM dictates the action (e.g., "mute"), and the Python script forcibly clamps the confidence score exactly within the allowed boundary, eliminating 100% of formatting/schema regressions.

4. Historical Context Retrieval

The evidence.py script mimics a lightweight retrieval system, extracting the user's past reactions to similar senders/businesses to inform the LLM whether the user historically appreciates or ignores these notifications.

📂 Repository Layout

.
├── certificate.png                   # Official HackerRank completion certificate
├── problem_statement.md              # Full challenge statement
├── README.md                         # You are here
├── requirements.txt                  # Python dependencies
├── code/                             # Core runnable solution
│   ├── main.py                       # Batch orchestration script
│   ├── llm_router.py                 # Multi-model LLM routing and confidence clamping
│   ├── rules.py                      # Rule engine heuristics and safety flags
│   ├── evidence.py                   # Historical evidence retrieval
│   └── evaluation/
│       └── evaluate.py               # Local dev-set evaluation script
└── dataset/                          # Datasets and Media

⚙️ How to Run

1. Install dependencies:

   pip install -r requirements.txt

2. Configure API Keys: Create a .env file in the root directory:

   GEMINI_API_KEY=your_primary_key
   # You can also add GEMINI_API_KEY_2, _3, etc., for rotation
   GROQ_API_KEY=your_fallback_key
   GEMINI_MODEL=gemini-1.5-flash

3. Execute the Pipeline:

   python code/main.py

   This reads dataset/messages.csv, processes them, and writes the compliant schema to dataset/output.csv.