AI Music Needs More Knobs: YouAndOrchestra Makes Conducting Agents a Compositional Exercise

TLDR: AI music production is shifting from one-shot prompt generation toward agentic co-creation that prioritizes granular control and iterative feedback, exemplified by projects like YouAndOrchestra. This evolution transforms AI from a simple output machine into an interactive collaborative instrument, allowing users to guide the "conductor loop" with persistent, provenance-aware oversight.

Note: YouAndOrchestra creator Yusuke Shibui will present "The Conductor Pattern: Multi-Granularity Feedback for Creative Agents" on Friday, September 11, at AGNTCon/MCPCon in Tokyo.

The first wave of AI music was impressive but also bland. Nearly every polished track emitted by an AI music factory ended up blending into the ether. Lots shipped and trended on Spotify. But none actually broke through. (Try to remember the last AI-generated song you loved and put on endless play.) The missing ingredient, like in all creative processes, was and remains taste: the difference between a solo blues guitar riff played over a Marshall tube amp and through a less soulful digital box. The notes might look the same to the model, but the effect was different.

The next generation of AI music tools takes a more layered approach, employing a team of agents to help produce a more distinctive and personalized sound. The personalization comes not from agent diversity but from giving the human more dials and knobs to turn in the process. Concurrently, access to music "stems," formerly limited to those with deep pockets or big studio catalogs, is becoming an AI-powered commodity. This unlocks a mythical substrate of sound that makes remix culture both more malleable and more approachable: a better bridge between what has come and what can be. The agents are beginning to become session musicians and editors that amplify and codify your taste rather than push-button machines for less memorable finished content.

Beyond the Prompt: Finding the Music Between and Around the Notes

The saying in music is that its not the notes but everything else that makes a song great. The first generation of generative music tools were stochastic note machines that said "Give me a prompt and I'll pump out a song." It was too easy. Describe a genre, mood, voice, and lyric idea, and receive a plausible three-minute track that sounded fine on the surface. The process is technically amazing and mind-boggling compared to state-of-the-art even a few years ago, but remains creatively thin. The creator has very few handles. If the bridge feels wrong, the chorus is too eager, or the bass should drop out for two bars before the vocal returns, "make it cooler" is not a real musical instruction.

To get to the next level, the next generation of music AI has to work more like an instrument, a studio, or a group of collaborators – not a band, collectively, but a band of agents. You play them and they play back. You reject, redirect, isolate, revise, and layer or an instrument or a specific thread of sound. You amp up this part, tone down that part. There are infinite shades of blue you paint against an aural canvas. Just like in an all-night studio session or an impromptu living room jam, taste emerges through accumulated interactions and iterations — or just via serendipity — not through one allegedly perfect prompt. This is the music future that the open source tool YouAndOrchestra is trying to package and deliver.

Three Layers of AI Music Capabilities

Today, there are three overlapping forms of AI music creation. The first is finished-track generation: describe a song and choose among results. Suno and Udio are the obvious examples. The leverage is enormous; the fine control is not. The second is intelligent production tooling: systems that separate, repair, remix, master, or extend existing audio. Moises, BandLab Splitter, and Suno's own stem tools belong here. These give the creator real handles, though not necessarily musical intent. The third is agentic co-creation: a developing category in which a person directs an evolving work across roles, versions, critique, and local edits. That is where projects such as YouAndOrchestra, CoComposer, and their successors are headed.

The middle category is having a major impact, largely because it makes formerly expensive and rare things cheaper (and probably heading towards very cheap). The music primitive of note is the "stem", which is akin to a "track" but contains only a single isolated sound element. Distinctive stems from legacy tracks, like the sitar-guitar riffs on Stevie Wonder's "Signed Sealed Delivered" or Beck's "Loser." Stem separation used to be challenging, highly dependent on recordings that were a label-controlled asset. Now Moises can isolate vocals and instruments from a mix. BandLab Splitter can split uploaded audio into vocal, bass, drum, and other stems. And Suno offers extraction of up to 12 tracks plus a stem player for remixing. All of this makes adding "taste" with AI both cheaper and easier, because musicians can now pattern match off the precise sounds they want to revise or remix.

That changes the relationship to music from "consume a fixed recording" to "open a recording and start manipulating its parts," which makes remix a high-speed joyride. To be clear, this is not just for aspiring producers. A singer can practice against an instrumental. A DJ can build a mashup. A songwriter can pull out a vocal, replace it, and rebuild the arrangement. Anyone can not only fire up Beck's sitar riff but layer in a violin riff that they captured from a trad session at an Irish bar. Someone who knows the feeling they want but lacks the studio skills to chase it has a new way into the work.

But stem separation is also a perfect example of why agents matter. Technology gives you pieces and choices but can never tell us what to do with them. An agentic music environment could understand an instruction such as: keep the tension of the first chorus, but make the final chorus feel more restrained and haunting. Bend the D in the final riff just a little bit further. This sort of AI-enabled flow-state needs a persistent project state, a model of musical structure, the ability to make localized edits, and a user who remains the final judge.

The Conductor's Dance: Negotiating with the Machine

That is why YaO is interesting. It turns a vague creative request into a negotiated process. Specialized agents propose a composition, critics assess it, and the human can intervene globally, at a section, or down at a particular bar, beat, or instrument. The novel bit is not "seven agents," which is conventional, and can produce a familiar kind of committee art that is only marginally better than a whole-cloth music engine. What makes YouAndOrchestra (YaO) cool is giving the user a way to say, this is not it, but this small part is, and make this part more like that — while preserving the reasoning trail so the system can explain and potentially honor that preference later. In essence, it makes making music a "forkable, versioned" experience.

Here's how it all works. The project's conductor loop generates, evaluates, adapts, and regenerates. Its critic can catch structural problems before the user has wasted time listening to a fully rendered failure. More unusually, every note carries provenance. Ask why the chorus modulated to the relative minor and the system can show the causal chain: which role made the proposal, which musical rule it followed, what feedback redirected it, and how the final decision landed.

That is not merely a nice feature for engineers. Creative work needs this kind of memory. The usual agent trace tells you which API was called and when. That may be useful for debugging. It does not tell a musician why the work got sadder, why the piano disappeared in the second verse, or whether the system heard the feedback that the chorus felt too clean. In a serious creative system, provenance is not audit paperwork. It is the record of how a work became itself, more like watching the amazing Beatles documentary than sifting through conventional log files. This is the hole in the music universe YouAndOrchestrate is attempting to fill.

People are now surrounded by AI-generated music, especially functional, background, short-form, and platform-targeted music, but they are not necessarily consciously choosing it or finding it artistically satisfying. A lot of it is competent on first listen and empty on the second. It mimics the surface signals of a genre but does not reliably sustain intention, surprise, or a point of view.

Just as the first pass at weird synthesizer sounds later yielded to an endless palate, the future of AI music creativity is not stopping at prompt-to-song. When agents become a band with ears and tools and stems, the editable work becomes the interface, and taste becomes the control system. AI is a tool, like any other. Like a paintbrush or a can of spray paint, its various iterations can paint beautiful sounds. The more nods and dials we figure out how to expose to the human mind, the better the music will be.

See it in practice: Yusuke Shibui will present "The Conductor Pattern: Multi-Granularity Feedback for Creative Agents" at AGNTCon + MCPCon Japan on Friday, September 11, in Tokyo. Register for the conference and add the session to your schedule.