AI Video Generator

Overview

This is a local video generation pipeline, not a diffusion app. Given a JSON script plus narration audio, it scrapes reference images from Google with a Playwright-based scraper (proxy rotation and CAPTCHA handling built in), re-ranks them with OpenAI CLIP for semantic fit, auto-frames faces with YOLO11x segmentation, then composes the final MP4 in MoviePy and FFmpeg using hardware encoders (NVENC, AMF, QSV). A local Ollama LLM plans cinematic effects per segment from a 24-effect library, and Real-ESRGAN handles upscaling. The UI is Gradio, running on the local machine.

Highlights

• Playwright scraper for Google Images with proxy rotation, CAPTCHA handling, and scroll pacing.
• CLIP-based semantic re-ranking of scraped images to drop off-topic results before editing.
• YOLO11x segmentation for face-anchored framing so subjects stay centered through cuts.
• Ollama-driven effects director picking from a 24-effect cinematic library per segment.
• Hardware-accelerated 1080p encoding (NVENC, AMF, QSV) with parallel chunk rendering.

Challenges and approach

• Google image scraping broke often due to CAPTCHAs and anti-bot measures. The fix was a manual or retry fallback path layered on top of proxy rotation, so a failed batch could be rescued without restarting.
• CLIP sometimes ranked visually similar but topically wrong images highly. Adding a re-query step with a sharper prompt when the top score fell below a threshold recovered most misses.
• Panning across images of different resolutions produced black bars. A uniform-scale normalization before panning kept the framing consistent.