📐 Smart Depth CRA2M Tracker

Repository: smart-depth-cra2m-tracker
Author: Rolando Cortez (Computer Vision Developer)

---

📋 Overview

Smart Depth CRA2M Tracker is a plug-and-play 3D perception module using just a monocular RGB camera. It enables absolute distance estimation and real-time tracking without the need for LIDAR or stereo vision — ideal for edge robotics and smart monitoring.

🔑 Main Components:

• YOLOv10: robust object detection.
• ByteTrack: multi-object tracking.
• Depth Anything v2: relative depth map estimation.
• CRA2M: converts relative depth to metric using ArUco/Charuco markers and solvePnP.
• 1D Kalman Filter: smoothing and continuity in depth estimation.
• Graphical Interface: live controls for tracking, scaling, and averaging settings.

---

🧠 Architecture and Internal Flow

graph LR
A[RGB Frame] --> B[YOLOv10 Detector]
B --> C[ByteTrack Tracker]
C --> D[Bounding Box Center]
A --> E[Depth Anything Map -every 4 frames-]
D & E --> F[Patch Sampling -median-]
F --> G[CRA2M Scale Estimation]
G --> H[Z Real Estimation]
H --> I[Kalman Filter per ID]
I --> J[Final Depth + Visualization]

Loading

• Relative depth is calculated at the center of the bounding box using a patch of 10% its size.
• CRA2M: custom calibration-based scaling from relative to real-world metric depth using ArUco markers + PnP, enabling depth in meters from monocular input.
• The metric depth of each tracked object is updated by applying the model and smoothing with a per-ID Kalman filter.

---

🗂️ Project Structure

smart-depth-cra2m-tracker/
├── gui/                   # User interface (PyQt backend in video_worker.py)
├── depth/                 # CRA2M, estimation and Kalman tracking
├── detection/             # YOLOv10 wrapper
├── tracking/              # ByteTrack + Q/R tuning
├── utils/                 # Central patch, buffers, scaling
├── scripts/               # Main runners: webcam vs OAK-1
├── checkpoints/           # Models: YOLOv10, DepthAnything
├── data/                  # Calibrations
├── config.py              # ByteTrack config loader

---

🚀 Basic Execution

With webcam:

python scripts/run_cra2m_tracking.py

With OAK-1:

python scripts/run_cra2m_tracking_oak.py

---

🖥️ Graphical Interface

Here’s a demo of the interface in action. It shows real-time object tracking, depth estimation, and scaling visualization.

---

🎥 Click here to watch on YouTube

Features:

• Select depth model (vits, vitb, vitl)
• Switch between linear, quadratic, cubic, spline scaling, etc.
• Visualize detected markers and their rvec, tvec
• Change averaging type: z_rel, z_real, or none
• Use optional buffers and stabilization per ID

---

⚙️ Requirements

• Python 3.9+
• PyTorch (depends on CUDA version)
• OpenCV with GUI (opencv-python, not headless)
• Models:
  • depth_anything_v2_vitl.pth, yolov10s.pt...

Installation:

pip install -r requirements.txt

Common GUI issue fix:

pip uninstall opencv-python opencv-python-headless
pip install opencv-python==4.8.0.76

---

🖥️ GUI Features

• Real-time visualization (RGB + Depth + Metric)
• Select:
  • Camera (Webcam / OAK-1)
  • YOLO and Depth models
  • Scaling method (linear, cubic, spline, etc.)
  • Smoothing strategy (Kalman, averaging)
  • Scale type used (z_rel, z_real, none)
• Tune:
  • Inference frequency
  • Central patch size
  • Buffer length
  • Kalman parameters (Q, R) per ID
• Display:
  • Depth per object ID
  • CRA2M-generated scale models
  • FPS, resolution, active markers

---

📏 Technical Details

🔍 Detection and Tracking

• Detector: YOLOv10 (PyTorch)
• Tracker: ByteTrack with YAML config
• Association via IOU and score

🌊 Depth Estimation

• Monocular model: Depth Anything v2
• Takes median from a central patch of the bounding box
• Configurable frequency: apply every N frames

📐 CRA2M Scaling

• Uses ArUco/Charuco and solvePnP to compute real distances
• Compares the z_rel of those points with their z_real to derive scaling factors
• Fitting methods:
  • linear, quadratic, cubic, log, exp, spline
• GUI shows real-time fitting status

📉 Kalman Filter

• One independent 1D filter per object (ByteTrack ID)
• Dynamically tuned Q and R using multiple strategies:
  • Based on deviation, prediction, or hybrid models
• Maintains depth stability between inferences

---

🛠️ Use Cases

• Smart monitoring systems
• Mobile robots without LIDAR
• Depth estimation on edge devices
• HMI applications where low-cost proximity sensing is needed

---

📈 CRA2M Scaling Details

• Based on ArUco / Charuco and solvePnP
• Supports single marker, Charuco, or multi-marker mode
• Generates scales:
  • linear: a * z + b
  • quadratic: a * z² + b * z + c
  • spline: smooth interpolation
• Evaluated in real time and viewable in GUI
• Validates range and consistency of each scale

---

💡 Technical Contributions

This project showcases the following applied skills:

• Designed and integrated a multi-model CV pipeline combining detection, monocular depth estimation, metric scaling, and object tracking.
• Applied camera geometry and PnP estimation (ArUco/Charuco + solvePnP) for real-world distance calibration.
• Developed CRA2M, a custom depth scaling framework supporting linear, polynomial, and spline mappings.
• Built a real-time GUI (PyQt) with modular configuration, allowing dynamic model switching, scaling modes, and per-ID Kalman smoothing.
• Focused on practical engineering: real-time performance, usability, modularity, and deployability on edge devices.

⚙️ Development Note

• Dynamic Adjustment of Q and R: The adjust_qr function has been designed to dynamically adjust the Q and R parameters of the Kalman filter, but it is currently not connected to the backend of the system. The code and necessary functions are present in the project, but they are not yet invoked or integrated into the main flow. This feature is planned for future improvement and will be completed in upcoming versions. Why this approach?

---

🤝 Credits

• YOLOv10 (THU-MIG)
• Depth Anything V2 (ISL)
• ByteTrack
• Full interface and integration: Rolando Cortez García
• CRA2M (including the quadratic, spline, exponential, and logarithmic functions) is an original implementation by Rolando Cortez García

---

📄 License

MIT License

---

📨 Contact

Rolando Cortez García
Email: rolscg@gmail.com
GitHub: rolandocortez
LinkedIn: rolando-cortez

---

“Detect, scale, track — one frame at a time.” 🚀