Skip to content
SieamShahriarePublic

About

A CLI tool for converting long form content into short form vertical videos

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

YouTube Video Processing Pipeline

An automated pipeline that transforms long-form YouTube videos into engaging short-form content with AI-powered segment selection, automatic cropping, and burned-in captions.

Features

  • AI-Powered Content Analysis: Uses Google Gemini AI to identify the most engaging segments from video transcripts
  • Automatic Video Cropping: Intelligent person tracking and reframing to 9:16 aspect ratio for social media
  • Caption Generation: Automatic speech-to-text with MLX Whisper
  • Smart Editing: Removes silence and keeps only captioned segments
  • Subtitle Styling: Burns styled captions directly into the final videos

Prerequisites

System Requirements

  • macOS: Optimized for Apple Silicon with MPS acceleration
  • Windows: Supported with modifications (see Windows Setup section)
  • Linux: Should work with CUDA/CPU setup
  • Python 3.10+ (recommended)
  • FFmpeg installed on your system
  • Conda package manager

API Keys Required

Create a .env file in the project root with:

GEMINI_API_KEY=your_google_gemini_api_key_here
YOUTUBE_API_KEY=your_youtube_data_api_key_here

Getting API Keys

  1. Google Gemini API: Visit Google AI Studio
  2. YouTube Data API: Visit Google Cloud Console

Installation

Step 1: Create Conda Environment

# Create a new conda environment named ShortGen
conda create -n ShortGen python=3.10 -y

# Activate the environment
conda activate ShortGen

Step 2: Clone and Setup

git clone https://github.com/SieamShahriare/ShortGen.git
cd video-processing-pipeline

Step 3: Install Dependencies

pip install -r requirements.txt

Step 4: Install Additional Tools

For macOS:

# Install FFmpeg
brew install ffmpeg

# Install MLX Whisper (Apple Silicon optimized)
pip install mlx-whisper

# Install yt-dlp
pip install yt-dlp

For Windows:

# Install FFmpeg - Download from https://ffmpeg.org/download.html
# Add FFmpeg to your system PATH

# Install OpenAI Whisper (since MLX is macOS only)
pip install openai-whisper

# Install yt-dlp
pip install yt-dlp

Step 5: Create Environment File

Create your .env file with the required API keys

Windows Setup Instructions

Additional Windows Requirements

  1. Install FFmpeg:

    • Download from FFmpeg official website
    • Extract to C:\ffmpeg\
    • Add C:\ffmpeg\bin to your system PATH
    • Verify installation: ffmpeg -version
  2. Install Visual Studio Build Tools:

    • Download Visual Studio Build Tools or Visual Studio Community
    • Required for some Python packages compilation

Windows-Specific Code Modifications

1. Update saveCaptions.py for Windows:

# Replace MLX Whisper command with OpenAI Whisper
WHISPER_COMMAND = [
    "whisper",  # Changed from "mlx_whisper"
    "--model", "base",  # Changed from "mlx-community/whisper-base-mlx"
    "--output_format", "srt",
    "--language", "en",
    "--no_speech_threshold", "0.5",
    "--word_timestamps", "True",
    # Remove MLX-specific parameters
]

2. Update autoCrop.py for Windows:

# At the top of the file, modify environment variables
os.environ.pop('PYTORCH_ENABLE_MPS_FALLBACK', None)  # Remove MPS settings
os.environ.pop('PYTORCH_MPS_HIGH_WATERMARK_RATIO', None)
# Keep only:
os.environ['NPY_USE_BLAS_ILP64'] = '0'
os.environ['VECLIB_MAXIMUM_THREADS'] = '1'

# In detect_people function, change device
def detect_people(frame, model):
    # Change device from "mps" to "cpu" or "cuda" if available
    detections = model(frame, device="cpu", half=False)[0]  # Also set half=False for CPU

3. Update requirements.txt for Windows:

# Remove or comment out macOS-specific packages
# pyobjc>=9.2  # macOS only

# Add Windows alternatives
openai-whisper>=20231117
torch>=2.0.0+cpu  # or +cu118 for CUDA support
torchvision>=0.15.0+cpu
torchaudio>=2.0.0+cpu

Windows Installation Commands

# Create conda environment
conda create -n ShortGen python=3.10 -y
conda activate ShortGen

# Install PyTorch for Windows (CPU version)
conda install pytorch torchvision torchaudio cpuonly -c pytorch

# Or for CUDA support (if you have NVIDIA GPU)
# conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

# Install other requirements
pip install -r requirements.txt

# Install Windows-specific whisper
pip install openai-whisper

# Install yt-dlp
pip install yt-dlp

Quick Start

Run the complete pipeline:

python main.py

The program will prompt you for a YouTube URL and automatically process it through all stages.

Manual Step-by-Step Execution

  1. Extract Segments (AI Analysis):
python getSubSegments.py

Enter a YouTube URL when prompted. The AI will analyze the transcript and identify 10 engaging segments.

  1. Download Segments:
python cutVidBySegments.py

Downloads the identified segments from YouTube.

  1. Generate Captions:
python saveCaptions.py

Creates SRT caption files for each segment using MLX Whisper.

  1. Cut by Captions (Remove Silence):
python srtCutter.py

Removes parts of videos without speech, keeping only captioned segments.

  1. Auto-Crop Videos:
python autoCrop.py [--preview]

Tracks people in videos and reframes to 9:16 aspect ratio. Use --preview to see the processing in real-time.

  1. Burn Captions:
python burnCaptions.py

Adds styled captions directly to the videos and saves final output.

Output

Final processed videos are saved in the output/ directory with:

  • 9:16 aspect ratio (perfect for TikTok, Instagram Reels, YouTube Shorts)
  • Burned-in styled captions
  • Automatic person tracking and centering
  • High-quality H.265 encoding

Configuration

Caption Styling

Modify the STYLE variable in burnCaptions.py:

STYLE = (
    "Fontname=Helvetica-bold, Fontsize=25, BackColour=&H00000000,"
    "OutlineColour=&H000000&,BorderStyle=1,Outline=1,Shadow=0,"
    "PrimaryColour=&HFFFFFF&,Alignment=2,MarginV=80"
)

Whisper Settings

Adjust transcription parameters in saveCaptions.py:

WHISPER_COMMAND = [
    "mlx_whisper",
    "--model", "mlx-community/whisper-base-mlx",
    "--output-format", "srt",
    "--language", "en",
    "--no-speech-threshold", "0.5",
    "--word-timestamps", "True",
    "--hallucination-silence-threshold", "1.0",
    "--max-words-per-line", "3"
]

Auto-Crop Settings

Modify tracking and reframing parameters in autoCrop.py:

  • target_aspect_ratio: Change output aspect ratio (default: 9:16)
  • LOST_SUBJECT_THRESHOLD_SEC: How long to wait before centering when person is lost
  • DeepSort tracking parameters for better person detection

Directory Structure

project/
├── main.py                 # Main execution script
├── getSubSegments.py       # AI-powered segment analysis
├── cutVidBySegments.py     # YouTube video downloading
├── saveCaptions.py         # Caption generation
├── srtCutter.py           # Silence removal
├── autoCrop.py            # Person tracking and cropping
├── burnCaptions.py        # Caption burning
├── requirements.txt       # Python dependencies
├── .env                   # API keys (create this)
├── segments.json          # Generated segment data
└── output/               # Final processed videos

Temporary Directories

During processing, these directories are created and cleaned up automatically:

  • segments/ - Downloaded video segments
  • captions/ - Generated SRT files
  • captionsCut/ - Processed captions
  • DeadCut/ - Videos with silence removed
  • reframe/ - Auto-cropped videos
  • styledSubtitles/ - ASS subtitle files

Troubleshooting

Common Issues

"No transcript available"

  • Some YouTube videos don't have automatic captions enabled
  • Try a different video or check if captions exist on YouTube

Platform-Specific Performance Issues

  • Windows: MLX Whisper not available; use OpenAI Whisper instead (slower but works)
  • Windows: MPS acceleration not available; use CPU or CUDA if available
  • macOS: Optimized performance with MPS and MLX Whisper
  • Linux: Should work with CUDA acceleration for better performance

MLX vs OpenAI Whisper

  • MLX Whisper (macOS only): Faster, optimized for Apple Silicon
  • OpenAI Whisper (cross-platform): Slower but more compatible

Device Compatibility

  • The code automatically detects and uses the best available device
  • Modify device settings in autoCrop.py if needed:
    • "mps" for Apple Silicon
    • "cuda" for NVIDIA GPUs
    • "cpu" for CPU-only processing

FFmpeg Errors

  • Ensure FFmpeg is properly installed and in your PATH
  • On macOS: brew install ffmpeg

API Rate Limits

  • YouTube API has daily quotas
  • Gemini API has request rate limits
  • Wait and retry if you hit limits

Environment Activation

# Always activate the conda environment before running
conda activate ShortGen

# To deactivate when done
conda deactivate

Cross-Platform Compatibility

  • macOS: Full feature support with MLX optimization
  • Windows: Requires code modifications (see Windows Setup section)
  • Linux: Should work with minimal modifications

Performance Tips

  1. Preview Mode: Use --preview flag with autoCrop.py to monitor processing
  2. Model Selection: Adjust Whisper model size based on accuracy vs speed needs
  3. Video Quality: Longer videos take significantly more time to process

Dependencies

Key libraries used:

  • ultralytics - YOLO object detection
  • deep-sort-realtime - Multi-object tracking
  • opencv-python - Computer vision operations
  • google-generativeai - Gemini AI API
  • youtube-transcript-api - YouTube transcript extraction
  • ffmpeg-python - Video processing
  • yt-dlp - YouTube video downloading

License

This project is for educational and personal use. Ensure you comply with YouTube's Terms of Service and copyright laws when processing videos.

Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Test thoroughly
  5. Submit a pull request

Support

For issues and questions:

  1. Check the troubleshooting section
  2. Review the error messages carefully
  3. Ensure all dependencies are properly installed
  4. Verify your API keys are correct

Note: This tool is designed for creating transformative, fair-use content. Always respect copyright laws and platform guidelines when using processed videos.

About

A CLI tool for converting long form content into short form vertical videos

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages