Distribution is the bottleneck for almost every early product I've worked on. Founders record a podcast or a keynote, and the clips that would actually travel stay buried in an hour of footage because nobody has ten hours a week to spend in Premiere.
AI Reel Agent was my attempt to automate the whole path: raw long-form video in, vertical captioned clips out, no human touching a timeline.
What follows is the pipeline and the three problems that made it hard.
The pipeline
Upload → transcribe (word-level) → plan cuts → render → deliver
Each stage is a discrete state in Firestore, and that detail matters more than it looks. A user who uploads a 30-minute video and sees nothing for four minutes assumes the app is broken. A user who sees TRANSCRIBING → PLANNING → RENDERING → UPLOADING tick past knows the machine is working. Same wall-clock time, completely different product.
The pieces:
- FastAPI backend orchestrating ingestion, validation, and stage transitions
- faster-whisper for transcription with word-level timestamps
- Gemini as the edit planner, ranking candidate segments
- FFmpeg, driven programmatically, for cropping, padding, audio normalization, and burned-in captions
Word-level timestamps change what you can build
Most speech-to-text gives you sentence-level timing, which is fine for a transcript and useless for captions that pop word by word. Running faster-whisper with CTranslate2 gave me word-level output, dropped transcription latency about 4x compared to a cloud speech API, and removed a per-minute cost line entirely.
The catch: raw Whisper output drifts. Over a 20-minute video, captions can slide 200–400ms out of sync — enough that it looks subtly wrong without the viewer being able to say why. Voice activity detection filtering plus calibrating against syllable cadence pulled it back into lock for the full length of a clip.
Letting a model choose the cuts
This is the part people expect to be magic and it mostly isn't. I fed Gemini semantic transcript chunks with their timestamp boundaries and asked it to rank segments on hook potency, emotional inflection, and whether the segment stands on its own without the surrounding context.
The critical constraint is that the model chooses from timestamps I gave it rather than generating timestamps itself. A model asked to invent a cut range will confidently hand you one that starts mid-word. A model asked to pick between candidate windows that already align to real speech boundaries cannot make that mistake. Constrain the output space and a lot of "hallucination" problems stop being problems.
FFmpeg is where theory meets reality
Three things bit me here.
Concurrency. Spawning FFmpeg encodes per job crashed memory on a standard instance almost immediately. Video encoding does not share nicely. A strict bounded concurrency queue — a small fixed number of simultaneous renders, everything else waits — is the entire fix.
Framing. A 16:9 recording puts the speaker off-center; a naive center-crop to 9:16 produces a clip of someone's shoulder. Face-detection heuristics to crop around the active speaker, with blurred background pads as fallback, made the output look intentional.
Codecs. This one cost me a day. Videos from mobile screen recordings arrive with variable framerates, and FFmpeg progressively lost audio sync across the clip — fine at the start, half a second off by the end. The fix was a normalization pass before anything else touches the file: force constant framerate with ffmpeg -vsync cfr and standardize audio to 48kHz. Now every file entering the pipeline looks the same to every downstream stage.
Normalize at the boundary. Every media pipeline I've built eventually learns this lesson the expensive way.
Results
- 3–5 vertical clips from a 30-minute input in under 4 minutes
- Editing turnaround down from ~90 minutes of manual work
- Zero manual caption timing adjustments
What I'd change today
The render worker should live on serverless GPU (Modal, or ECS on Fargate with GPU) rather than a fixed instance. Rendering load is spiky by nature — a creator uploads four videos at once and then nothing for two days — and that's exactly the shape that fixed capacity handles worst.