Skip to content
garamahqPublic

About

No description, website, or topics provided.

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

46 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

CloudMesh πŸš€

Production-grade infrastructure auditor linking Cloudflare DNS records to Hetzner Cloud virtual servers.

logo

Build Status License Conventional Commits Release Please


1. System Overview

CloudMesh is a production-grade infrastructure auditing tool that matches active Cloudflare DNS A records to Hetzner Cloud virtual servers by resolving and matching public IP addresses. It identifies orphaned DNS records (subdomains pointing to missing servers), aggregates monthly cloud costs per project, checks domain expiration dates, pushes telemetry to a Prometheus/Pushgateway stack, and provides a modern, interactive React/Vite dashboard.


2. Core Features

  • Extended Resource Mapping: Integrates and maps Cloudflare DNS zones (supporting both A and AAAA records) to multiple Hetzner Cloud resource types, including Virtual Servers, Load Balancers, and Floating IPs.
  • Network Security Port Audit: Concurrently scans ports 22 (SSH), 80 (HTTP), 443 (HTTPS), and 3389 (RDP) on all resolved IPs using a high-performance multi-threaded socket verification pool.
  • Stale VM & Cost Optimization Engine: Detects financial waste and potential subdomain takeovers by recommending removal of dangling DNS entries and idle/unmapped Hetzner resources.
  • Interactive SVG Topology Graph: Renders a dynamic node connection map (Domain -> Cloudflare Record -> Public IP -> Hetzner Resource -> Project) showing orphaned nodes in crimson, clickable resource nodes, and cost-weighted projects.
  • Drift & Historical Diff Engine: Computes modifications (added, removed, or changed records/resources) compared to prior audit runs, with a client-side selector for arbitrary historical snapshot comparison.
  • DNS & HTTP Latency Telemetry: Measures DNS resolution speed and TCP connection latency asynchronously for all subdomains.
  • Scheduled Monitor (Daemon Mode): Runs continuously inside containers or servers at a configurable interval (MONITORING_INTERVAL_SECS) to automate ongoing scans.
  • Advanced Filtering: Filters records on the fly by Cloudflare proxy states, DNS type (A/AAAA), wildcard formats, cost ranges, projects, and domain expiration health.
  • Disk-Persistent WHOIS Cache: Automatically caches domain registration lookups in reports/whois_cache.json to prevent rate-limit blocks and speed up execution.
  • API Robustness & Retries: Utilizes a customized connection pool configured with a 5-retry policy and exponential backoff to handle transient errors (429, 500, 502, 503, 504).
  • Label Sanitization (Data Safety): Masks sensitive metadata keys (e.g., key, token, secret, auth, password) in VM labels as ******** before exporting reports.
  • Multi-Channel Alerting: Dispatches summary metrics and domain expiration warnings to Slack channels (supporting PDF uploads) and Google Chat webhook cards.
  • Prometheus Telemetry: Pushes metrics (running counts, processing durations, domain mappings, server uptimes) to Prometheus Pushgateway.

3. Directory Structure

β”œβ”€β”€ docs/                     # Detailed guides (Deployment, Configuration, Troubleshooting)
β”œβ”€β”€ frontend/                 # React / Vite TypeScript dashboard application
β”‚   β”œβ”€β”€ src/                  # React dashboard component codebase
β”‚   └── public/               # Static assets & generated data.json source
β”œβ”€β”€ reports/                  # Generated HTML, PDF, and JSON audit artifacts (gitignored)
β”œβ”€β”€ src/                      # Modular Python package engine
β”‚   β”œβ”€β”€ api/                  # Cloudflare and Hetzner API client components
β”‚   β”œβ”€β”€ core/                 # Matcher calculations and persistent cache adapters
β”‚   β”œβ”€β”€ notifications/        # Slack and Google Chat webhook integrations
β”‚   β”œβ”€β”€ reports/              # Print HTML and JSON file writers
β”‚   β”œβ”€β”€ config.py             # Configuration validation and environment loader
β”‚   β”œβ”€β”€ metrics.py            # Prometheus metric registry declarations
β”‚   └── main.py               # Orchestrator flow
β”œβ”€β”€ docker-compose.yml        # Telemetry containers (Prometheus & Pushgateway)
β”œβ”€β”€ prometheus.yml            # Prometheus scrape targets configuration
β”œβ”€β”€ script.py                 # Thin backward-compatible script wrapper
└── PROJECT.md                # System specification and architecture design doc

4. Getting Started

Quick Start (Linux Deployment)

For a rapid setup and run on a Linux server (e.g., target directory /home/cloudmesh), execute the following:

cd /home/cloudmesh

# 1. Stop old services
docker compose down --remove-orphans || true

# 2. Set up virtualenv (only needed once)
python3 -m venv venv
./venv/bin/pip install --upgrade pip
./venv/bin/pip install -r requirements.txt

# 3. Configure credentials (only needed once)
cp .env.example .env
nano .env        # Fill in CLOUDFLARE_TOKEN, HETZNER_TOKEN_1, etc.

# 4. Create output directories
mkdir -p reports/snapshots
mkdir -p frontend/dist/snapshots

# 5. Start infrastructure
docker compose up -d

# 6. Run the audit
./venv/bin/python script.py

Prerequisites

  • Python 3.8+
  • NodeJS 18+ & pnpm
  • wkhtmltopdf (optional, required to export PDF reports)
    • Ubuntu/Debian: sudo apt-get update && sudo apt-get install wkhtmltopdf
    • macOS: brew install wkhtmltopdf
    • Windows: Download from wkhtmltopdf.org and add to system PATH.

1. Setup Python Environment

Create a virtual environment and install engine dependencies:

# Create and activate environment
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt
# Or manually:
pip install requests pdfkit prometheus_client python-dotenv python-whois

2. Configure Credentials

Copy .env.example to .env and fill in API credentials and daemon configurations:

CLOUDFLARE_TOKEN=your_cloudflare_api_token
HETZNER_TOKEN_1=your_hetzner_project_api_token
HETZNER_PROJECT_NAME_1=Hetzner_Project_Name_1
PUSHGATEWAY_URL=http://localhost:9913

# (Optional) Scheduled Daemon Mode Configuration
MONITORING_INTERVAL_SECS=300       # Run audits every 5 minutes (set to 0 or omit to run once)
MAX_HISTORICAL_SNAPSHOTS=10        # Maximum history snapshots to persist for diffs and sparklines

5. Execution & Usage

Step 1: Run the Audit Engine

Execute the Python script to fetch, map, and export the infrastructure mapping reports:

python script.py

This script validates credentials, computes mappings, and outputs:

  • reports/mapping.html: Static printable report.
  • reports/data.json & frontend/public/data.json: Raw structured audit data.
  • reports/whois_cache.json: Persistent WHOIS queries.
  • reports/mapping_YYYYMMDD_HHMMSS.pdf: Archived PDF report.

Step 2: Launch the React Dashboard

Run the frontend dev server to visualize findings interactively:

# Navigate to frontend and install dependencies
cd frontend
pnpm install

# Start local server
pnpm dev

Open http://localhost:5173 in your browser. The dashboard automatically syncs with frontend/public/data.json on page refresh or load.


6. Telemetry Stack (Prometheus)

To collect execution metrics, spin up the telemetry containers:

# Start Prometheus and Pushgateway
docker compose up -d

# Verify services
# Prometheus target page: http://localhost:9912
# Pushgateway metrics: http://localhost:9913

7. Contributing

  1. Fork the repository.
  2. Create a feature branch (git checkout -b feature/name).
  3. Commit changes using standard Conventional Commit guidelines.
  4. Submit a Pull Request.

8. License

This project is licensed under the MIT License - see the LICENSE file for details.

About

No description, website, or topics provided.

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages