Building MusicVision with AI: How I Actually Work

AIClaudeMusicVisionsoftware engineeringagentic

Why I embrace it

A friend of mine, a developer I respect, told me recently that the flood of AI-generated code is making him anxious. He sees the human role in software shrinking, and he asked me, fairly directly, why I lean into AI so hard. Instead of a quick answer, I want to show him how I actually work, using the project I know best.

MusicVision is an open-source pipeline that turns a song into a music video. Audio and lyrics go in; the system segments the song into scenes, generates reference images for each one, renders video clips with one of two video engines, upscales them, and assembles a rough cut plus an edit list for DaVinci Resolve. It runs on local hardware: two GPUs in my workstation and an LLM server on my LAN. It is not a toy. It has a forked video model with two dozen bug fixes, a multi-GPU memory plan, an asset library for character consistency, and over 300 unit tests.

I did not build it alone, and I did not let the AI build it alone either. Here’s the honest version of how it got made, and why I think the human role didn’t shrink. It moved.

Phase 1: Talking it through before writing anything

The project started as a conversation, not a codebase. I set up a dedicated workspace with Claude for architecture and planning, separate from the tool I use to write code. In that space we argued about goals before we touched implementation.

The early questions were mine to answer. Who is this for? I decided it was for creators who want deep control and real post-production integration, not one-click social clips. What makes it different from the commercial tools? Song-level workflow, per-scene engine choice, local model ownership, and export to a real editor. Which licenses can I live with? That one changed over time; I eventually relicensed the code to Apache 2.0 and documented which model weights carry non-commercial restrictions.

The AI’s job in this phase was to be a fast, well-read sparring partner. It surveyed models I hadn’t tried, laid out trade-offs, and pushed back when an idea had holes. When I asked about character consistency, it explained the difference between LoRA training and IP-Adapter conditioning and where each one tends to fail. It could not tell me which trade-off mattered for my creative goals. That was my call, and it stayed my call.

Phase 2: Framework and spec sheets

Once the goals were settled, every major feature got a written spec before any code existed. The pipeline spec, the asset library spec, a reference doc for the HuMo video model, and later migration specs all live in the repo’s docs folder. On top of those sits one file, CLAUDE.md, that is the authoritative record of project state. When anything disagrees with it, it wins, and the disagreement gets flagged.

Drafting specs is where AI collaboration paid off most for me. I would describe intent and constraints in plain language; Claude would turn that into structured documents with data models, stage boundaries, and edge cases I hadn’t listed. Then I reviewed them, hard.

That review matters. While implementing phases 3 to 5 of the asset library, we found the spec contradicted itself about the order in which models had to load. The draft read smoothly and confidently, and it was wrong. A human reading carefully would have caught it before handoff, and now I read every spec with that in mind before code gets written from it.

The specs also encode architectural rules that I set and the AI enforces. All pipeline logic lives in core modules, never in the API or the UI. Every engine registers in one file that is the single source of truth for frame limits. All duration math uses integer frame counts, never seconds, so audio and video can’t drift apart. Those are judgment calls about what the system should be. Once written down, the AI applies them more consistently than I ever would by memory.

Phase 3: Building it iteratively, together

Implementation happens in a different tool. I hand a finished spec to Claude Code in the terminal and work through it in small, reviewable steps. I clear its context before each new plan so stale assumptions don’t leak in, and I keep the context during active debugging so it remembers what we just tried.

The AI writes most of the code. It is very good at the parts that would drain me: Pydantic models with backward-compatible migrations, CLI and API wiring, test scaffolding, and refactors that touch twenty files at once. It keeps the whole test suite running without a GPU or even PyTorch installed, because I set that as a rule.

The truth of a generative video pipeline, though, is only visible on the GPU, and that loop is mine. I use ComfyUI as an experimentation lab: I test parameters against real reference images and seed-locked scenes before anything gets committed to the codebase. That’s where the most important finding in the project came from. The upstream HuMo default for a sampling parameter called sigma shift is 8.0. My own testing showed that values between 1 and 2 give clearly better results for singing performance shots. No amount of reading the paper would have told either of us that. I had to watch the clips.

The same pattern played out with IP-Adapters. The AI researched the candidates and their integration paths; I ran a bake-off on real images. We landed on XLabs v2 through the standard diffusers path. One alternative had no diffusers support and broke multi-reference conditioning; another had weights whose license restricted the generated output. The research was the AI’s strength. The verdict, based on what I saw and what I could legally ship, was mine.

When the AI and I disagree, the evidence decides. I’ve overruled it on taste, and it has spotted problems I’d overlooked. Neither of us is right by default.

Who does what

Over the life of this project, the division of labor is pretty clear to me. It isn’t about who is smarter. It’s about who is better positioned for each kind of decision.

Work Who leads Why
Goals, audience, licensing, scope Me Values and context the AI doesn’t have
Surveying models, papers, libraries AI Breadth and speed
Turning intent into a structured spec AI drafts, I review Fast structure, but it can be confidently wrong
Architecture rules Me They encode what the system should become
Boilerplate, wiring, refactors, tests AI Tireless and consistent once rules exist
Judging output quality on real clips Me Taste, and eyes on the actual result
Debugging from logs and stack traces Together It reads fast; I know what changed in the real world
Hardware reality Me Only I know the GPU order changed twice

That last row is a small example with a big lesson. The physical order of my GPUs has changed twice as I swapped hardware. Code that assumed a fixed device index would have broken silently. The fix was a rule: never hardcode device indices, auto-assign by VRAM at runtime. The AI implemented it in minutes. Knowing it was needed came from me standing next to the machine.

Rules I’d give any developer

  1. Don’t expect the AI to make judgments you can make better. Taste, priorities, ethics, licensing risk, and “does this actually look good” are yours. The AI will happily offer an opinion. Treat it as input, not a verdict.
  2. Don’t bog yourself down in minutiae the AI handles better. If you’re hand-typing your fortieth Pydantic field or wiring another endpoint, you’re spending your scarcest resource, attention, on the cheapest work. Delegate it and spend the time where your judgment counts.
  3. Write it down before you build it. A spec turns your intent into something the AI can follow and you can check. Most of my best AI sessions started from a good document.
  4. Review the AI’s confident output hardest. Fluent isn’t the same as correct. The load-order contradiction in my spec read perfectly well.
  5. Encode your judgment as rules. “Integer frames, never seconds.” “No pipeline logic in the UI.” “Never hardcode a GPU index.” Once a rule exists, the AI enforces it everywhere, every time, without getting tired.
  6. Let evidence settle disagreements. Seed-locked A/B tests, real reference images, metrics. Neither you nor the AI gets to win an argument by being sure.
  7. Decide which problems need a human and which are just code. Put people where judgment lives and automate the rest. In MusicVision, I approve the storyboard and iterate on each scene; stitching sub-clips together is a code problem, not a review step.
  8. Extend autonomy in steps you can audit. I’m building toward an agent that drives the pipeline itself, but trust ratchets up only as reliability is shown.

The role isn’t shrinking. It’s moving.

I understand the anxiety. If you define your value as the number of lines you type, AI is a threat, because it types faster than any of us. But typing was never the hard part. The hard part is knowing what to build, why, for whom, and whether the result is any good. That work hasn’t gone anywhere. If anything, there’s more of it, because the AI makes it possible to attempt projects one person couldn’t have finished before.

MusicVision is a one-person project that would have needed a small team a few years ago. I’m still the architect, the tester, the person with taste, and the one accountable for what ships. I just spend far less of my day on work that never needed me.

So my advice to my friend, and to any developer feeling the same way: don’t compete with the AI at what it does well, and don’t hand it what you do better. Learn where that line sits for you. Then take full advantage of both sides of it.

MusicVision is on my GitHub, linked from the About page. The code, specs, and CLAUDE.md are all there if you want to see how it fits together.

This post was drafted with AI help and edited by me, the same way MusicVision gets built.