An automated pipeline that transforms long-form YouTube videos into engaging short-form content with AI-powered segment selection, automatic cropping, and burned-in captions.
- AI-Powered Content Analysis: Uses Google Gemini AI to identify the most engaging segments from video transcripts
- Automatic Video Cropping: Intelligent person tracking and reframing to 9:16 aspect ratio for social media
- Caption Generation: Automatic speech-to-text with MLX Whisper
- Smart Editing: Removes silence and keeps only captioned segments
- Subtitle Styling: Burns styled captions directly into the final videos
- macOS: Optimized for Apple Silicon with MPS acceleration
- Windows: Supported with modifications (see Windows Setup section)
- Linux: Should work with CUDA/CPU setup
- Python 3.10+ (recommended)
- FFmpeg installed on your system
- Conda package manager
Create a .env file in the project root with:
GEMINI_API_KEY=your_google_gemini_api_key_here
YOUTUBE_API_KEY=your_youtube_data_api_key_here- Google Gemini API: Visit Google AI Studio
- YouTube Data API: Visit Google Cloud Console
# Create a new conda environment named ShortGen
conda create -n ShortGen python=3.10 -y
# Activate the environment
conda activate ShortGengit clone https://github.com/SieamShahriare/ShortGen.git
cd video-processing-pipelinepip install -r requirements.txtFor macOS:
# Install FFmpeg
brew install ffmpeg
# Install MLX Whisper (Apple Silicon optimized)
pip install mlx-whisper
# Install yt-dlp
pip install yt-dlpFor Windows:
# Install FFmpeg - Download from https://ffmpeg.org/download.html
# Add FFmpeg to your system PATH
# Install OpenAI Whisper (since MLX is macOS only)
pip install openai-whisper
# Install yt-dlp
pip install yt-dlpCreate your .env file with the required API keys
-
Install FFmpeg:
- Download from FFmpeg official website
- Extract to
C:\ffmpeg\ - Add
C:\ffmpeg\binto your system PATH - Verify installation:
ffmpeg -version
-
Install Visual Studio Build Tools:
- Download Visual Studio Build Tools or Visual Studio Community
- Required for some Python packages compilation
1. Update saveCaptions.py for Windows:
# Replace MLX Whisper command with OpenAI Whisper
WHISPER_COMMAND = [
"whisper", # Changed from "mlx_whisper"
"--model", "base", # Changed from "mlx-community/whisper-base-mlx"
"--output_format", "srt",
"--language", "en",
"--no_speech_threshold", "0.5",
"--word_timestamps", "True",
# Remove MLX-specific parameters
]2. Update autoCrop.py for Windows:
# At the top of the file, modify environment variables
os.environ.pop('PYTORCH_ENABLE_MPS_FALLBACK', None) # Remove MPS settings
os.environ.pop('PYTORCH_MPS_HIGH_WATERMARK_RATIO', None)
# Keep only:
os.environ['NPY_USE_BLAS_ILP64'] = '0'
os.environ['VECLIB_MAXIMUM_THREADS'] = '1'
# In detect_people function, change device
def detect_people(frame, model):
# Change device from "mps" to "cpu" or "cuda" if available
detections = model(frame, device="cpu", half=False)[0] # Also set half=False for CPU3. Update requirements.txt for Windows:
# Remove or comment out macOS-specific packages
# pyobjc>=9.2 # macOS only
# Add Windows alternatives
openai-whisper>=20231117
torch>=2.0.0+cpu # or +cu118 for CUDA support
torchvision>=0.15.0+cpu
torchaudio>=2.0.0+cpu# Create conda environment
conda create -n ShortGen python=3.10 -y
conda activate ShortGen
# Install PyTorch for Windows (CPU version)
conda install pytorch torchvision torchaudio cpuonly -c pytorch
# Or for CUDA support (if you have NVIDIA GPU)
# conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
# Install other requirements
pip install -r requirements.txt
# Install Windows-specific whisper
pip install openai-whisper
# Install yt-dlp
pip install yt-dlpRun the complete pipeline:
python main.pyThe program will prompt you for a YouTube URL and automatically process it through all stages.
- Extract Segments (AI Analysis):
python getSubSegments.pyEnter a YouTube URL when prompted. The AI will analyze the transcript and identify 10 engaging segments.
- Download Segments:
python cutVidBySegments.pyDownloads the identified segments from YouTube.
- Generate Captions:
python saveCaptions.pyCreates SRT caption files for each segment using MLX Whisper.
- Cut by Captions (Remove Silence):
python srtCutter.pyRemoves parts of videos without speech, keeping only captioned segments.
- Auto-Crop Videos:
python autoCrop.py [--preview]Tracks people in videos and reframes to 9:16 aspect ratio. Use --preview to see the processing in real-time.
- Burn Captions:
python burnCaptions.pyAdds styled captions directly to the videos and saves final output.
Final processed videos are saved in the output/ directory with:
- 9:16 aspect ratio (perfect for TikTok, Instagram Reels, YouTube Shorts)
- Burned-in styled captions
- Automatic person tracking and centering
- High-quality H.265 encoding
Modify the STYLE variable in burnCaptions.py:
STYLE = (
"Fontname=Helvetica-bold, Fontsize=25, BackColour=&H00000000,"
"OutlineColour=&H000000&,BorderStyle=1,Outline=1,Shadow=0,"
"PrimaryColour=&HFFFFFF&,Alignment=2,MarginV=80"
)Adjust transcription parameters in saveCaptions.py:
WHISPER_COMMAND = [
"mlx_whisper",
"--model", "mlx-community/whisper-base-mlx",
"--output-format", "srt",
"--language", "en",
"--no-speech-threshold", "0.5",
"--word-timestamps", "True",
"--hallucination-silence-threshold", "1.0",
"--max-words-per-line", "3"
]Modify tracking and reframing parameters in autoCrop.py:
target_aspect_ratio: Change output aspect ratio (default: 9:16)LOST_SUBJECT_THRESHOLD_SEC: How long to wait before centering when person is lost- DeepSort tracking parameters for better person detection
project/
├── main.py # Main execution script
├── getSubSegments.py # AI-powered segment analysis
├── cutVidBySegments.py # YouTube video downloading
├── saveCaptions.py # Caption generation
├── srtCutter.py # Silence removal
├── autoCrop.py # Person tracking and cropping
├── burnCaptions.py # Caption burning
├── requirements.txt # Python dependencies
├── .env # API keys (create this)
├── segments.json # Generated segment data
└── output/ # Final processed videos
During processing, these directories are created and cleaned up automatically:
segments/- Downloaded video segmentscaptions/- Generated SRT filescaptionsCut/- Processed captionsDeadCut/- Videos with silence removedreframe/- Auto-cropped videosstyledSubtitles/- ASS subtitle files
"No transcript available"
- Some YouTube videos don't have automatic captions enabled
- Try a different video or check if captions exist on YouTube
Platform-Specific Performance Issues
- Windows: MLX Whisper not available; use OpenAI Whisper instead (slower but works)
- Windows: MPS acceleration not available; use CPU or CUDA if available
- macOS: Optimized performance with MPS and MLX Whisper
- Linux: Should work with CUDA acceleration for better performance
MLX vs OpenAI Whisper
- MLX Whisper (macOS only): Faster, optimized for Apple Silicon
- OpenAI Whisper (cross-platform): Slower but more compatible
Device Compatibility
- The code automatically detects and uses the best available device
- Modify device settings in
autoCrop.pyif needed:"mps"for Apple Silicon"cuda"for NVIDIA GPUs"cpu"for CPU-only processing
FFmpeg Errors
- Ensure FFmpeg is properly installed and in your PATH
- On macOS:
brew install ffmpeg
API Rate Limits
- YouTube API has daily quotas
- Gemini API has request rate limits
- Wait and retry if you hit limits
Environment Activation
# Always activate the conda environment before running
conda activate ShortGen
# To deactivate when done
conda deactivateCross-Platform Compatibility
- macOS: Full feature support with MLX optimization
- Windows: Requires code modifications (see Windows Setup section)
- Linux: Should work with minimal modifications
- Preview Mode: Use
--previewflag with autoCrop.py to monitor processing - Model Selection: Adjust Whisper model size based on accuracy vs speed needs
- Video Quality: Longer videos take significantly more time to process
Key libraries used:
ultralytics- YOLO object detectiondeep-sort-realtime- Multi-object trackingopencv-python- Computer vision operationsgoogle-generativeai- Gemini AI APIyoutube-transcript-api- YouTube transcript extractionffmpeg-python- Video processingyt-dlp- YouTube video downloading
This project is for educational and personal use. Ensure you comply with YouTube's Terms of Service and copyright laws when processing videos.
- Fork the repository
- Create a feature branch
- Make your changes
- Test thoroughly
- Submit a pull request
For issues and questions:
- Check the troubleshooting section
- Review the error messages carefully
- Ensure all dependencies are properly installed
- Verify your API keys are correct
Note: This tool is designed for creating transformative, fair-use content. Always respect copyright laws and platform guidelines when using processed videos.