One idea in, one YouTube Short out: an AI video pipeline on a single 16 GB GPU

TL;DR

• What: a solo-built pipeline that turns one typed idea into a finished, uploaded YouTube Short — script, consistent characters, animation, voice, sound, edit, upscale and SEO — with a human approval step before every expensive stage.
• The hard part: it runs on one 16 GB GPU with 31 GB of RAM and a personal budget, so hardware and cost limits drove most of the design.
• Abandoned a model on evidence: after ~13 failed renders I dropped LongCat-Video, wrote down why, and picked HunyuanVideo 1.5 because it avoided each of LongCat's failure modes.
• Read the model before tuning it: at cfg = 1.0 negative prompts do nothing, so I rewrote the video constraints as positive statements and moved the quality work to the first frame.
• A workaround became the bug: prompt rules written for the weak local model ruined a $2.36 hosted render, so the prompts now change with the backend's capability — and paid backends get an automatic cost guard.
• Four levels of "done": built, unit-tested, live-verified, render-validated — and anything not yet validated is labelled that way.

At a glance

The problem

Short-form video channels live on volume. Producing a 60-second Short by hand means writing a script, generating consistent characters, animating each shot, recording or synthesising voice, finding sound effects and music, cutting it all together and writing the metadata — for every single video.

I wanted a system where I type one idea, review a draft, press approve, and get back a finished video I would actually publish. Not a demo that produces one lucky clip, but a pipeline that produces a coherent sequence of shots with the same characters, sound that lands on the action, and a human checkpoint before every expensive step.

The constraint that shaped everything

Three numbers drove most of the decisions in this project:

• 16 GB of VRAM — one render at a time, and most state-of-the-art video models do not fit.
• 31 GB of system RAM — smaller than the working set of the model I started with.
• A personal budget for hosted renders — every paid generation came out of my own pocket.

A lot of the interesting engineering was making a system that wanted a datacentre run on one desktop.

Architecture

                idea
                 │
        ┌────────▼─────────┐
        │  React control   │  draft review · timeline editor · models · analytics
        │  room (Vite)     │
        └────────┬─────────┘
                 │ REST
┌────────────────▼──────────────────────────────────────────────┐
│ NestJS                                                        │
│                                                               │
│  Scenario Engine ── LLM (Gemini / Groq / Claude / MUapi)      │
│     idea → art style + location + cast + ordered scenes       │
│     + per-scene render profile, dialogue, SFX, music mood     │
│  RAG memory (pgvector) ── what this channel already made      │
│  Profile resolver ── validate against installed models        │
│  Character reconciler ── reuse or generate reference images   │
│                                                               │
│  RabbitMQ (prefetch = 1, manual ack)                          │
│   video.generate ──► Video Worker ── ComfyUI / OpenRouter /   │
│        │   (each scene enqueues the next)          Muapi      │
│   video.postprocess ──► FFmpeg assembly                       │
│   [ AWAITING_UPSCALE — human gate ]                           │
│   video.upscale ──► upscale + frame interpolation             │
│   metadata.generate ──► title / description / hashtags       │
│   ──► YouTube upload ──► analytics fed back into prompts      │
└───────────────┬──────────────────────────────┬────────────────┘
                │                              │
         PostgreSQL + pgvector          ComfyUI on the GPU

The pipeline, step by step

1. Draft. The LLM expands the idea into an editable draft: art style, location, cast and a scene-by-scene script, with preview portraits. I can rename, re-roll or rewrite anything before approving.
2. Reconciliation. Each character and the location get a reference image — reused from the library if the character already exists, generated otherwise.
3. First frame per scene. SDXL composes the opening frame, identity-locked to the scene's focus character and the location through IP-Adapter. Several candidates are rendered and CLIP picks the most on-model one.
4. Animation. That frame seeds an image-to-video model. Scenes render one at a time, in order.
5. Post-production. FFmpeg aligns clips to the voiceover, places sound effects at precise points, ducks the per-scene music, burns captions and the end card, and concatenates.
6. Human gate. The assembled video stops at AWAITING_UPSCALE. I watch it, edit it in the timeline editor, and only then pay for the upscale.
7. Upscale + interpolation to the final resolution and frame rate.
8. Metadata + upload. The LLM writes SEO metadata; the video is uploaded through the YouTube Data API, and channel analytics flow back in.

Key decisions

1. A sequential cascade instead of a parallel worker pool

With one GPU, parallel scene renders just fight over VRAM. Each scene job is consumed with prefetch = 1 and, when it finishes, enqueues the next scene. This also made continuity possible later: scene N can start from the literal last frame of scene N − 1, because scene N − 1 is guaranteed to exist.

ComfyUI runs natively on the host rather than in Docker — GPU passthrough on Windows was friction with no payoff. Postgres, RabbitMQ and the Kokoro TTS server stay in Compose.

2. The video model arc: abandoning a path on evidence

This is the most instructive decision sequence in the project, because it contains a migration I abandoned.

The symptom. Starting a render pegged the disk at 100% and froze the machine. The top disk consumer wasn't ComfyUI — it was System: Windows pagefile thrash. Wan 2.2 14B fp8 is a dual-expert model whose working set is ~33 GB (two 14 GB UNets plus a 5 GB text encoder) on a machine with 31 GB of RAM. ComfyUI was evicting and re-reading ~14 GB from disk on every expert swap.

The options I weighed: GGUF-quantise Wan (real cure, but needed an uninstalled node and a re-exported graph), buy 64 GB of RAM (cost), cap WSL2 memory (a stopgap — 33 is still more than 31), or move to a smaller single-expert model. I applied the stopgap and picked LongCat-Video (~20 GB).

LongCat failed ~13 times. LoRA-merge segfaults, then the discovery that it is text-to-video with no start-frame image-to-video — the pipeline's core requirement — then VRAM OOM, then a broken offload path in its third-party wrapper. No quantised variant existed. I deleted the 21.9 GB of weights and wrote the dead end down explicitly as "do not retry on this 16 GB card", with the reasons, so it couldn't be re-entered later.

The replacement was selected against the failure modes, not a leaderboard. HunyuanVideo 1.5 was chosen because it avoided every one of them: 8.3B parameters in fp8 (~8 GB), true image-to-video, and native ComfyUI support with no third-party wrapper. First validation: a 33-frame 480p image-to-video render in 191 s, no OOM — and it incidentally fixed an open "the video is lifeless" complaint.

3. Reading the model card: negative prompts were doing nothing

HunyuanVideo 1.5 is CFG-distilled and runs at cfg = 1.0. At cfg = 1.0 the unconditional branch of classifier-free guidance is unused — so negative prompts have mathematically zero effect on the video stage. Every "no extra limbs, no bipedal cats, no morphing" negative in the codebase had only ever been binding on the SDXL first frame.

That one finding redirected the quality work:

• Constraints for the video stage were rewritten as positive assertions ("quadrupedal, all four paws on the ground").
• Quality investment moved to the first frame, which anchors the whole image-to-video clip.
• "Turn CFG up" was recorded as an anti-pattern — raising it breaks this model.

4. A hallucination roadmap, ordered by cost

Grounded in research on hallucination in video generation, I turned the fixes into an ordered roadmap — cheap, high-impact levers first, expensive ones last — each mapped to a concrete place in the code:

1. Dense recaptioning of scene prompts
2. Negatives rewritten as positive assertions
3. First-frame hardening
4. Video best-of-N with a verifier
5. VLM verify-and-regenerate
6. Structural conditioning (ControlNet)
7. Sampler knobs
8. Regional multi-character lock
9. Preference-pair logging for future DPO training

Two ideas were explicitly excluded unless the first six plateaued (a JEPA verifier and hybrid render/simulate) — scope discipline written down at the time.

I also defined a standard regression prompt: a cat, three red apples, one blue bowl, a single bounce. It stresses object counting, colour binding, physics and identity drift at once, so every change could be judged against the same scene.

Later, blind "render N, pick the best" was replaced by budgeted search against a verifier: render candidates one at a time and accept the first that clears the bar. Easy scenes cost one render; only hard scenes spend the budget.

5. Consistency across scenes and across frames

Keeping a Short coherent is two separate problems:

• Scene-to-scene (look). One global art style; the IP-Adapter reference is chosen per scene from the character who is actually in it. An early bug locked every scene to the primary character — the result was, in my own bug report, "the squirrel has a cat head". Fixing the per-scene focus selection fixed it.
• Frame-to-frame (motion). The LLM marks each scene with continues_previous. Continuous scenes start from the previous clip's last frame; real cuts get a freshly composed, identity-locked frame. One rule is baked into the prompt: a scene that shows the aftermath of an off-screen impact must never chain from the previous frame, because that frame still shows the unbroken state.

Motion needed tuning in the other direction too. The first round of stabilisation overcorrected into "the cat only looks and walks; the camera only gets wider". The cause was partly a bug — the generated motion_hint was stored but never sent to the video model — and partly prompt design. Every shot now needs one strong action verb, and passive verbs and slow zoom-outs are banned.

6. The "turn-based RPG" render: when a workaround becomes the bug

I made three video backends switchable at runtime from the UI: local ComfyUI (free, slow), OpenRouter (per-second pricing) and Muapi (per-generation, cheap draft tier).

The first hosted render of "Tesla vs Edison" cost $2.36 and came out like a turn-based RPG: two characters posing alone, never in the same frame, never touching. The cause was my own prompt rules. To protect the weak local model, the scenario prompt said keep impacts off-screen, move one subject at a time. On a frontier hosted model those rules forbade exactly what a fight scene needs.

The fix was structural: prompts now fork on backend capability.

The rationale is written next to the code with an explicit note not to merge them back into one doctrine.

7. Cost as a design input

8. Autonomous, but never unsupervised at the expensive steps

• Draft review before anything renders.
• Timeline editor: reorder, cut and trim scenes, per-track volumes, per-scene music and voice, re-render a single clip. Edits are stored as an edit-decision list (JSON) and applied only on Re-assemble.
• No auto-resume of GPU work on boot. Stale queue messages are parked, not processed — I may want to start something else. Resume is an explicit endpoint.
• Stop a running task is a persisted flag, not a queue purge: because the cascade is sequential, the in-flight scene finishes and the next one is simply never enqueued.

A hazard I hit more than once and finally documented: Save is not apply. There are two independent payload builders, and both have to honour the edit list — otherwise a full re-render silently throws your edits away.

9. Sound that lands on the action

Sound effects are placed at a fractional position within each scene (at: 0.42 of the clip, multiplied by the real duration at render time), and near-simultaneous cues are spaced at least 0.3 s apart so they sequence instead of stacking.

After the clip renders, a vision pass watches it and re-times, rewrites or drops each cue against what actually happened on screen, then matches it to the existing SFX library by fuzzy search before generating anything new. Music is planned per scene; consecutive scenes with the same track play as one continuous bed.

10. Content decisions driven by data

• Channel memory with RAG. Instead of a "last five titles" hack, each channel has pgvector memory of what it already published, with isolation between channels verified live.
• The niche steers tone, not subject. The channel niche was hijacking ideas — the channel's recurring cat kept appearing in videos about something else. Rule: the idea is authoritative; the niche only sets the vibe.
• SEO rewritten from real analytics. Views on the real channel had collapsed from 1.1k–21k per video to 9–56. An analysis named four causes: dormancy, no hashtags in titles, descriptive titles instead of curiosity hooks, and no audio hook in the first second. Three were fixed in code, including a first-second audio-hook rule pushed into the scenario prompt itself.
• A real bug from that work: descriptions were truncated at 4,900 characters, but YouTube's limit is 5,000 bytes — and emoji are 3–4 bytes each. The clamp is now byte-based.

Constraints you only find by measuring

None of these appear in an architecture diagram, but each one cost real debugging time:

How I define "done"

The project distinguishes four levels of done and never conflates them:

Notes routinely say things like "build + 38 tests green; render-side behaviour untested on GPU". This distinction earned its keep: two defects once reached me after a feature was reported as building with passing tests — a stale validation allow-list and a parser that didn't recognise one provider's response shape. Green builds proved nothing about either.

Honest state

• Not everything is render-validated. Per-scene music, seamless loops and the newest LLM provider are built and unit-tested but haven't been through a live paid render.
• One first frame can lock only one identity. Two characters in the same shot still drift; the fix is a regional multi-character lock (built, needs GPU-side validation) or reference-to-video on a hosted backend.
• Frame-to-frame coherence is still the weakest axis, and it's being escalated at the model level rather than declared solved.
• Some third-party integrations use best-effort model identifiers behind configurable endpoints, because the API shape couldn't be confirmed from the docs.

How I worked with AI

I built this with Claude Code as a research, implementation and record-keeping partner, and kept the decisions on scope, cost and architecture myself — several times against its recommendation. The working pattern that emerged:

1. Start from a concrete defect in real output ("the dog jumps and shrinks to a molecule"), then chase it to the model, config or code that caused it — never patch the symptom.
2. Frame choices as a few options with costs attached, then decide.
3. Ground the big moves in something external: a research paper for hallucination, the channel's own analytics for SEO.
4. Put the dollar figure on the table before spending it, and add a guard the moment an unplanned cost shows up.
5. Ship in verified increments — the unit suite grew from 9 to 90+ tests over the project.
6. Keep a dated decision log with its staleness stated, including notes like "do not retry this" and "don't merge these back into one", so a future refactor can't quietly undo a hard-won lesson.

What I'd take to the next project

• Constraints are a design spec. The 16 GB card chose the queue model, the video model, the cost guards and the human gates.
• Select replacements against failure modes, not benchmark scores.
• Read the model before tuning it. One line in a model card invalidated a whole class of tuning.
• Compensating rules have a shelf life. A workaround for a weak component becomes a bug when the component gets stronger — tie such rules to the capability they compensate for.
• Measure what the filter claims to do. PSNR and dB readings found problems that "it looks fine" never would have.