Project Wedjat (gr_sat)

This repository hosts the Project Wedjat codebase, an end-to-end pipeline for satellite telemetry analysis. It features offline telemetry processing, per-satellite anomaly-model training, synthetic-fault benchmarking, a robust FastAPI backend, and a premium SvelteKit + Tailwind dashboard.

Key Documentation

• Technical Details & Architecture: Deep dive into the system's design, ML models, and normalized telemetry structures.
• WEBSITE_PLAN.md: Web interface, Docker orchestration, and UX integration plan.
• .gemini/GEMINI.md: Active project context and agent instructions.

Current Repository Status

Core Pipeline & Backend:

• SatNOGS data pipeline: fetch -> decode -> normalize -> train -> benchmark.
• Python-based Kaitai Struct decoders (currently supporting UWE-4).
• Online streaming runtime for packet-by-packet autoencoder inference.
• FastAPI REST backend for telemetry streaming and model metrics.
• Containerized MQTT broker and TimescaleDB stack.

Frontend Dashboard:

• Svelte 5 frontend built with TailwindCSS and Observable Plot.
• Websocket-driven live telemetry feed with real-time anomaly highlighting.
• ML diagnostic views showing standardized deviation (Z-Score) and reconstruction errors.
• Exploratory Data Analysis (EDA) views for payload stats and orbital mechanics.
• Satellite pass prediction and operations console.

---

Web Dashboard & UI

View Dashboard (Dark Mode)

View Dashboard (Light Mode)

Architecture

View System Architecture (v3)

View ML Pipeline & Docker Components

---

Quick Start Guide

1. Install Prerequisites

We use Pixi for the core Python environment, and Bun for the frontend.

Linux / macOS:

curl -fsSL https://pixi.sh/install.sh | bash
curl -fsSL https://bun.sh/install | bash

Windows (PowerShell):

powershell -ExecutionPolicy Bypass -c "irm -useb https://pixi.sh/install.ps1 | iex"
powershell -c "irm bun.sh/install.ps1 | iex"

2. Setup the Project

Clone the repo and install dependencies:

git clone https://github.com/Mohamed-Badry/wedjat.git
cd wedjat
pixi install
cd frontend && bun install && cd ..

3. Configure API Keys

To download telemetry, you need a SatNOGS API Token.

1. Log in to SatNOGS Network.
2. Go to Your Profile / Edit and copy your API Token.
3. Create your config and paste your token:

   cp .env.example .env

4. Optional Docker Dev Stack

For the MQTT broker, TimescaleDB, FastAPI backend, and simulator:

docker compose up --build

---

Usage & Task Lifecycle

We use just (installed automatically by Pixi) to run all common tasks across the stack.

Recommended: Enter the Pixi environment first:

pixi shell

Phase 1: Data Pipeline (The Lab)

Fetch, decode, and normalize telemetry into structured, ML-ready numerical arrays.

just fetch                  # Download telemetry (interactive)
just fetch --norad 43880    # Download specific satellite
just process                # Run decode + normalize pipeline (interactive)
just process --norad 43880  # Process specific satellite

Phase 2: Machine Learning Pipeline

Train Autoencoder (VAE) models on the processed data and benchmark them.

just train --norad 43880            # Train VAE model for a specific satellite
just benchmark --norad 43880        # Run synthetic-fault benchmark
just train-benchmark 43880 100      # Train for 100 epochs, then benchmark immediately

Phase 3: Operations & Frontend

Run the minimal backend wedjat runtime and the beautiful SvelteKit dashboard.

just wedjat --norad 43880 --help  # Run backend anomaly wedjat
just frontend-dev                   # Start frontend SvelteKit development server (localhost:5173)
just frontend-build                 # Build the frontend production bundle
just frontend-preview               # Preview the production build

To see all available commands, run just or just --list.

---

Production Deployment & Ingestion

The project utilizes Docker Compose profiles to cleanly separate cloud resources from edge devices. Two deployment scripts are provided in the root directory:

1. The Cloud VPS (./deploy_vps.sh)

Spins up the TimescaleDB, PyTorch Backend, and the production Bun SvelteKit frontend.

• Closed Ports (Secure Setup): Do not expose port 1883 (unencrypted MQTT), 9001 (WebSocket MQTT), 5432 (PostgreSQL), or 8000 (FastAPI) to the public internet. They are closed or bound strictly to 127.0.0.1.
• Caddy reverse proxy (TLS/HTTPS): Caddy terminates TLS and routes standard traffic to the correct containers. Add this block to your /etc/caddy/Caddyfile on the VPS to support secure real-time telemetry publishing:

  wedjat.space {
      # Secure MQTT over WebSockets (WSS)
      handle /mqtt {
          reverse_proxy localhost:9001
      }
  
      # FastAPI API & WebSockets
      handle /api/* {
          reverse_proxy localhost:8000
      }
  
      # Svelte frontend
      handle {
          reverse_proxy localhost:5173
      }
  }

2. The Edge Device / Laptop (./deploy_edge.sh)

Spins up the local data buffer and Mosquitto broker on your antenna PC or Raspberry Pi.

• Scope: It now only boots the local broker container. This keeps the environment clean and avoids replaying simulated historical frames when setting up a fresh ground station.
• Publishing Telemetry: Your local SDR software (like gr-satellites) publishes decoded frames to your local broker on localhost:1883.

---

📡 Ground Station Data Ingestion Schema

To send decoded packets from your SDR antenna station, connect to the secure broker at wss://wedjat.space/mqtt on Port 443 (via the Caddy reverse proxy) and publish to the topic telemetry/live/{norad_id}.

The payload must be a JSON string structured exactly as follows:

{
  "norad_id": 43880,
  "timestamp": "2026-07-06T03:02:38Z",
  "raw_frame": "8A8A8A8A8A8A608A8A8A8A8A8A6103F025821E130201...",
  "station_id": "laptop_sdr_station_01",
  "snr": 15.4
}

• norad_id: (Integer) The NORAD catalog identifier of the satellite (e.g., 43880 for UWE-4).
• timestamp: (String, ISO-8601 UTC format) The exact time the packet was observed.
• raw_frame: (String, Hex-encoded) The raw binary frame output from the demodulator.
• station_id: (String, optional) Ground station identifier.
• snr: (Float, optional) Signal-to-noise ratio of the signal.

---

Complete Data & Directory Structure

Project Wedjat is strictly organized to separate raw data pipelines, machine learning models, core logic, simulated environments, and user interfaces.

.
├── data/                       # Local data storage (ignored by git, populated by scripts)
│   ├── raw/                    # Stage 0: Raw JSONL files fetched directly from SatNOGS DB API
│   │   ├── 43880/              # e.g., UWE-4 raw telemetry records
│   │   └── ...                 # Other satellite NORAD ID directories
│   ├── interim/                # Stage 1: Kaitai-decoded telemetry CSVs (unaltered fields)
│   └── processed/              # Stage 2: SI-unit normalized telemetry CSVs mapped for Machine Learning
├── frontend/                   # Bun + SvelteKit Web Dashboard
│   ├── src/                
│   │   ├── lib/                # Shared internal components
│   │   │   ├── api.ts          # Strongly-typed fetch wrappers for the FastAPI backend
│   │   │   ├── components/     # UI elements (charts, tables, layout components)
│   │   │   └── chart-theme.ts  # Unified design tokens mapping CSS variables to SveltePlot
│   │   └── routes/             # File-system based router pages
│   │       ├── (dashboard)/    # Authenticated/Main App layout group
│   │       │   └── dashboard/  # Dashboard views:
│   │       │       ├── live/   # Real-time packet watcher feed
│   │       │       ├── ml/     # Model telemetry root cause analysis & reports
│   │       │       ├── eda/    # Exploratory Data Analysis (sensitivity, stats)
│   │       │       ├── operations/# Satellite/model status summaries
│   │       │       └── inspector/ # Ground-truth telemetry variable explorer
│   │       └── (landing)/      # Public facing landing pages
│   ├── tailwind.config.ts      # Tailwind design configuration (Amethyst themes, plugins)
│   └── package.json            # Bun dependencies (SvelteKit, Tailwind, Lucide, Observable Plot)
├── src/                        # Python Backend & Shared Library Code
│   ├── api/                    # FastAPI Dashboard Endpoints
│   │   ├── main.py             # Entrypoint: handles REST routes, artifacts, and summary stats
│   │   └── Dockerfile          # FastAPI service container definition
│   ├── gr_sat/                 # Core domain logic ("The Shared Core")
│   │   ├── decoders/           # Kaitai Struct adapters (e.g. uwe4.py, registry pattern)
│   │   ├── telemetry.py        # Central abstractions for telemetry frames
│   │   ├── models/             # Shared PyTorch/Scikit models (VAE, Autoencoders, Scalers)
│   │   └── utils/              # Data parsing and time manipulation helpers
│   └── simulator/              # Edge Simulator Environment
│       ├── edge_node.py        # Hardened Edge Simulator (TLS MQTT + Local CSV playback when disconnected)
│       └── Dockerfile          # Simulator service container definition
├── scripts/                    # Executable Pipeline & ML Scripts
│   ├── fetch_training_data.py  # Pipeline Stage 0: Fetches SatNOGS historical API data
│   ├── process_data.py         # Pipeline Stage 1+2: Decodes & normalizes raw telemetry into ML arrays
│   ├── train_model.py          # Pipeline Stage 3: Trains VAE models on processed satellite data
│   ├── generate_faults.py      # Pipeline Stage 4: Injects synthetic faults and benchmarks models
│   └── wedjat_runtime.py     # Minimal deterministic online runtime for stream processing
├── notebooks/                  # Interactive Jupytext Notebooks for Prototyping
│   ├── uwe4_pipeline_eda.py    # Multi-scale EDA, hyperparameter sweeps, statistical profiling
│   └── telemetry_inspector.py  # Visual ground-truth debugger
├── docs/                       # Formal Documentation & Architecture
│   ├── spec/                   # Allium formally verified domain system contracts
│   ├── architecture_diagrams/  # Architecture assets & Typst layouts
│   └── slides.typ              # Typst presentation slides
├── models/                     # Persisted local ML artifacts (Scalers, PyTorch Models, Metadata)
├── mosquitto/                  # MQTT Broker configuration and TLS certificates
├── tests/                      # Python `unittest` regression coverage
├── .gemini/                    # Agentic context mandates & conversation brain (`GEMINI.md`)
├── .agents/                    # Custom Allium and domain skills for Antigravity
├── docker-compose.yml          # Docker development stack (MQTT, DB, API, Frontend, Simulator)
├── justfile                    # Centralized task runner (Makefile alternative)
├── pixi.toml                   # Conda-compatible environment & dependency manager
├── pyproject.toml              # Ruff linting & Python metadata
└── README.md                   # You are here

---

Adding a New Satellite Decoder

The decoder system uses satnogs-decoders (Kaitai Structs) for binary parsing and a registry pattern for satellite-specific logic. To add support for a new satellite:

1. Create src/gr_sat/decoders/<satellite>.py
2. Subclass BaseDecoder and implement decode() + adapt()
3. Register with @DecoderRegistry.register(NORAD_ID)
4. Import in src/gr_sat/decoders/__init__.py

Currently Supported Satellites

Satellite	NORAD ID	Decoder	Status
UWE-4	43880	decoders/uwe4.py	Primary target, ~7 months of data
CUTE	49263	decoders/cute.py	9600 bps GMSK, Active NASA Astrophysics payload