Skip to content

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

43 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Investigation System - RAG-Based LLM Decision Support

Interactive investigation simulation for comparing Sparse, Dense, and Hybrid retrieval on the same Indonesian investigation questions.

The active ID v3 workflow, migration instructions, offline checks, and experiment commands are documented in Research realignment handoff. Earlier benchmark reports are historical and must not be combined with v3 results.

Project Structure

.
├── app/                    # Next.js app directory
│   ├── api/               # API routes
│   │   ├── cases/         # Case management endpoints
│   │   ├── sessions/      # Session management
│   │   └── ai/            # AI interaction endpoints
│   ├── page.tsx           # Main entry point
│   └── layout.tsx         # Root layout
├── components/            # React components
│   ├── CaseSelection.tsx  # Case selection UI
│   └── InvestigationDashboard.tsx # Main investigation interface
├── lib/                   # Utility functions
│   ├── db.ts             # Prisma client
│   ├── embedding.ts      # OpenAI embedding generation
│   ├── rag.ts            # RAG retrieval logic
│   └── case-data.ts      # Case data format specification
├── prisma/               # Database schema and migrations
│   ├── schema.prisma     # Database schema
│   └── seed.ts          # Database seed script
└── README.md

Setup Instructions

1. Environment Setup

Create .env.local with required variables:

DATABASE_URL="postgresql://admin:root@localhost:5432/thesis?schema=public"
OPENAI_API_KEY="your-openai-api-key"
NEXT_PUBLIC_API_URL="http://localhost:3000"

2. Install Dependencies

npm install

3. Generate Prisma Client

npm run prisma:generate

4. Create Database Schema

npm run prisma:migrate

5. Seed Database with Cases

npm run seed

By default this loads the Indonesian ID v3 case (150 evidence records and 30 scenarios) into an isolated database. Embeddings require the explicit --embed flag; seeding never clears existing research data.

6. Start Development Server

npm run dev

Visit http://localhost:3000 in your browser.

Database Schema

Core Tables

  • cases: Investigation cases

    • id, title, description, createdAt, updatedAt
  • suspects: Case suspects

    • id, caseId, name, profile
  • evidence: Investigation evidence with vector embeddings

    • id, caseId, type, content, embedding

Investigation Tracking

  • investigation_sessions: Player sessions with RAG toggle

    • id, caseId, ragEnabled, createdAt, updatedAt
  • investigation_logs: Player actions during investigation

    • id, sessionId, action, target, createdAt
  • ai_interaction_logs: AI responses with metrics

    • All prompt/response content
    • Timing metrics (retrieval_time_ms, llm_response_time_ms, total_response_time_ms)
    • Token usage (prompt_tokens, completion_tokens, total_tokens)
    • Hallucination detection flags
    • Historical action-score fields retained for prior runs; active runs use retrieval and RAGAS metrics

Creating Custom Cases

Use the standardized case data format in lib/case-data.ts.

Example JSON structure for AI case generation:

{
  "case": {
    "title": "Case Title",
    "description": "2-3 sentence description"
  },
  "suspects": [
    {
      "name": "Suspect Name",
      "profile": "Background and motive"
    }
  ],
  "evidence": [
    {
      "type": "witness_statement|forensic_report|cctv_log|financial_record|email_message|location_report",
      "content": "Specific evidence details"
    }
  ]
}

To add custom cases:

  1. Generate case JSON using lib/case-data.ts format
  2. Add to sampleCases array in prisma/seed.ts
  3. Run npm run seed

API Endpoints

Cases

  • GET /api/cases - List all cases
  • GET /api/cases/[id] - Get case details with suspects and evidence

Sessions

  • POST /api/sessions - Create investigation session
    • Body: { caseId: string, ragEnabled: boolean }

AI Interaction

  • POST /api/ai/interact - Get AI response
    • Body: { sessionId, caseId, prompt, ragEnabled }
    • Response includes: response text, timings, token usage, log ID

Features

Investigation Interface

  • Case selection with RAG mode toggle
  • Evidence inspection sidebar
  • Suspect profile viewing
  • Predefined investigation prompts
  • Custom free-text prompts
  • Real-time AI responses

RAG System

  • Vector embedding of all evidence using OpenAI embeddings
  • Semantic similarity search via pgvector
  • Injection of relevant evidence into LLM context
  • Investigation history retrieval

Evaluation Metrics

  • Response Timings: retrieval_time_ms, llm_response_time_ms, total_response_time_ms
  • Token Usage: prompt_tokens, completion_tokens, total_tokens
  • Hallucination Detection: boolean flag + reason field
  • Correctness Scoring: optional 0-1 score field
  • Auto Logging: all metrics captured per interaction

Development

Prisma Studio

npm run prisma:studio

Access database GUI at http://localhost:5555

Build for Production

npm run build
npm start

Technology Stack

  • Framework: Next.js 15
  • Language: TypeScript
  • Styling: Tailwind CSS
  • Database: PostgreSQL with pgvector
  • ORM: Prisma
  • LLM: OpenAI API (GPT-4 Turbo)
  • Embeddings: OpenAI text-embedding-3-small

Research Contribution

This system enables evaluation of RAG's impact on:

  1. Decision accuracy in LLM recommendations
  2. Hallucination reduction with grounded context
  3. Contextual consistency through evidence retrieval
  4. Investigation workflow efficiency

The RAG toggle allows A/B comparison between standard LLM and retrieval-augmented approaches using identical case data and investigation scenarios.

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages