Distributed Real-Time Log Processing & Alerting Platform

A production-grade, microservices-based log observability platform built in Go, designed to ingest, process, search, and alert on application logs at scale (100K+ logs/minute target architecture).

Inspired by Datadog, Splunk, ELK, Grafana Loki, and New Relic — simplified for learning, portfolio, and system design interviews.

---

Architecture Overview

flowchart TB
    subgraph Clients
        APP1[Payment Service]
        APP2[API Service]
        APP3[Mobile Backend]
    end

    subgraph Edge
        GW[API Gateway<br/>Rate Limit · JWT · CORS]
    end

    subgraph Services
        AUTH[Auth Service]
        ING[Log Ingestion]
        PROC[Stream Processor<br/>Worker Pool]
        ALT[Alerting Engine]
        QRY[Query Service]
        NOTIF[Notification Service]
    end

    subgraph Messaging
        K1[(logs.raw)]
        K2[(logs.processed)]
        K3[(alerts.triggered)]
        KDLQ[(logs.dlq)]
        KRET[(logs.retry)]
    end

    subgraph Storage
        PG[(PostgreSQL)]
        REDIS[(Redis)]
        ES[(Elasticsearch)]
    end

    subgraph Observability
        PROM[Prometheus]
        GRAF[Grafana]
        JAEGER[Jaeger OTLP]
    end

    APP1 & APP2 & APP3 --> GW
    GW --> AUTH
    GW --> ING & QRY & ALT
    ING --> K1
    K1 --> PROC
    PROC --> ES
    PROC --> K2
    PROC --> K3
    K2 --> ALT
    ALT --> PG & REDIS
    ALT --> K3
    K3 --> NOTIF
    NOTIF --> SLACK[Slack / Webhook / Email]
    PROC -.-> KDLQ
    ING -.-> KRET
    GW & PROC & ING --> PROM --> GRAF
    GW & PROC --> JAEGER

Loading

Services

Service	Port	Responsibility
API Gateway	8080	Routing, rate limiting, correlation IDs, auth proxy
Auth	8081	JWT, RBAC, API keys, user registration
Ingestion	8082	Validate & publish logs to Kafka
Processor	8083	Worker pool, anomaly detection, ES indexing
Alerting	8084	Dynamic rules, regex matching, threshold alerts
Query	8085	Full-text search via Elasticsearch
Notification	8086	Slack, webhooks, email dispatch

Quick Start

Prerequisites

• Go 1.21+
• Docker & Docker Compose
• Make (optional)

Local Development

# 1. Copy environment
cp .env.example .env

# 2. Start infrastructure + all services
make docker-up

# 3. Wait ~60s, then seed data
make seed

# 4. Ingest sample logs
curl -X POST http://localhost:8082/v1/logs \
  -H "Content-Type: application/json" \
  -d '{
    "logs": [{
      "source": "payment-service",
      "level": "error",
      "message": "Payment failed: connection timeout",
      "host": "pay-pod-1",
      "environment": "staging"
    }]
  }'

# 5. Search logs
curl "http://localhost:8085/v1/logs/search?q=timeout&source=payment-service"

# 6. Create alert rule
curl -X POST http://localhost:8084/v1/rules \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Payment Errors",
    "pattern": "(?i)payment.*(failed|timeout)",
    "severity": "critical",
    "threshold": 3,
    "window_sec": 60,
    "channels": ["slack"]
  }'

Build Locally (without Docker)

make deps
make build
# Start infra only:
docker compose -f deployments/docker/docker-compose.yml up -d postgres redis kafka elasticsearch
make migrate
./bin/gateway  # repeat for each service

Observability URLs

Tool	URL
Grafana	http://localhost:3000 (admin/admin)
Prometheus	http://localhost:9090
Jaeger	http://localhost:16686
Elasticsearch	http://localhost:9200

---

Project Structure

cmd/                    # Service entrypoints (7 microservices)
internal/               # Business logic (clean architecture)
  gateway/ ingestion/ processor/ alerting/ query/ auth/ notification/
  repository/ bootstrap/
pkg/                    # Shared libraries (kafka, es, worker pool, jwt...)
api/                    # OpenAPI + gRPC proto definitions
configs/                # Migrations, service configs
deployments/            # Docker, K8s, Helm, Prometheus, Grafana
docs/                   # Full documentation (15 guides)
scripts/                # migrate.sh, seed.sh, setup-local.ps1
tests/                  # Unit, integration, load tests

---

Concurrency Model

The Stream Processor is the concurrency hot path:

1. Kafka consumer reads logs.raw (partition-parallel across pods)
2. Messages submitted to a bounded channel worker pool (pkg/worker)
3. Backpressure: TrySubmit → blocking Submit when buffer full slows consumer
4. Each worker: validate → anomaly score → aggregate → ES index → publish logs.processed
5. Graceful shutdown: context.Cancel → drain pool → close Kafka consumer

Goroutine 1..N (workers) ← channel(buffer=4096) ← Kafka ConsumeClaim
                              ↑ blocks when full (backpressure)

---

Key Engineering Decisions

Decision	Rationale	Tradeoff
Kafka for ingestion	Durable buffer, replay, horizontal scale	Ops complexity vs Redis streams
Async ES indexing	Decouples write latency from API	Near-real-time search (~1s)
Worker pool vs unbounded goroutines	Memory bounds under spike load	Slight latency under saturation
Redis for alert counters	Fast sliding-window thresholds	Extra dependency; not durable counts
API Gateway reverse proxy	Single TLS/rate-limit entry	Extra network hop
DLQ + Retry topics	Failure isolation & reprocessing	Topic proliferation

---

Documentation Index

Document	Path
Architecture	docs/architecture.md
API Reference	docs/api.md
Database Schema	docs/database-schema.md
Kafka Topics	docs/kafka-topics.md
Deployment Guide	docs/deployment.md
Kubernetes	docs/kubernetes.md
Docker Setup	docs/docker.md
Local Development	docs/local-development.md
Scaling Strategy	docs/scaling.md
Security	docs/security.md
Performance	docs/performance.md
CI/CD	docs/cicd.md
Monitoring	docs/monitoring.md
Incident Handling	docs/incident-handling.md
Interview Prep	docs/interview-prep.md

---

Makefile Commands

make build          # Build all 7 services
make test           # Unit tests with race detector
make bench          # Benchmarks
make docker-up      # Full stack via Docker Compose
make docker-down    # Tear down
make migrate        # Apply PostgreSQL migrations
make load-test      # k6 load test (requires k6)
make k8s-deploy     # Deploy to Kubernetes (dev overlay)
make helm-install   # Helm chart install

---

License

MIT — Built for educational and portfolio purposes.