I’ve recently been using iReal Pro on an iPad to practice improvisation. It’s a convenient way to play backing parts for a chord progression. But creating and editing progressions in the app is a frustrating experience. Unfortunately, iReal Pro, like most apps, is a walled garden. Short of filing a ticket and hoping it will one day be prioritized, there isn’t much I can do to improve the situation.
One of the themes we talk about at Ink & Switch is “situated software”, tools that are developed for the specific requirements of an individual or small community. I was interested in seeing if I could develop a patchwork tool that could not only solve my specific problem here but generally support my processes when writing progressions.
I’ve been thinking of different ways to quickly input chord progressions for a while. For example, some years ago I rekeyed and programmed a Planck keyboard to act as a MIDI controller for entering chords:
My first thought was I wanted a software equivalent of this keyboard with features for editing progressions and exporting them to both iReal Pro and MIDI formats. But since I find it easier to think in terms of the Circle of 5ths when writing, I decided to lay the chords out that way, placing the alterations in a separate section to the side. Here was my initial sketch:
Bringing over code I’d written for past music tools (and with a little help from my neighborhood LLM), I prototyped a version of the Progressions tool that was relatively user-friendly and well-suited to the iPad:
Sound modules
The prototype already met my initial requirements and was immediately useful to me. I could quickly enter in (and edit) a progression and export it to iReal Pro (thanks to the fact that others had reverse engineered the obfuscation algorithm). The walled garden was now significantly less constricting.
But the initial built-in synth didn’t sound very good. One of the design goals of Patchwork is to make it easy to compose tools together. So I started to think about how to support external tools as sound modules in Progressions.
My first thought was to come up with a simple protocol for playing sounds on an external tool. This would make it easy to take any sound-generating tool I might work on and make it compatible with Progressions, without having to design it with that use case in mind. Anything that implements the protocol would function as a “sound module” that Progressions could import.
As a first draft, I created a little self-contained triangle synth tool. Nothing special but it would prove out the idea:
It was straightforward to add an implementation of my protocol. Then it was a matter of adding support to Progressions for importing tools that spoke the protocol:
Great, it worked! But malleable software shouldn’t be limited to integrating with tools that play nice and do what you ask them. And anyway, we try as much as possible to avoid centralizing design decisions, which rules out relying on a sound module protocol that Patchwork tool authors are expected to respect in general.
I wanted to see how I could integrate with any sound-generating tool that someone might develop entirely independently of Progressions (and its sound module protocol).
Sound Module Adapter
You might not be surprised to hear that folks had already been building a variety of sound-generating tools at Ink & Switch. For example, Alex Warth had a patchwork tool that implemented his msynth synthesizer DSL. He’d demoed msynth internally a couple of times. But Progressions didn’t even exist then. It was a perfect test case.
Here’s his patchwork tool in action:
I wanted Progressions to be able to integrate with any sound-generating tool, even if I couldn’t edit the code, the project was abandoned, and the creator had gone into hiding.
There are many approaches you could take here. I decided to create a patchwork tool for building sound modules out of sound-generating tools that don’t speak the protocol. You could use it to create a new tool that wrapped a target tool and acted as an adapter for the sound module protocol. Then Progressions (or any tool that wanted to use the protocol to play sounds) could import that.
The sound adapter basically does two things: it translates protocol calls into function calls in the wrapped tool (making use of noteOn, noteOff, and stopAll functions defined in the UI) and it intercepts the AudioContext and destination node connections so that the wrapped tool (unknowingly) uses our AudioContext and destination node for playing sounds. Here it is wrapping Alex’ msynth tool (and bringing in the patch he demoed):
Keep in mind that msynth is completely unaware it’s operating as a sound module. It doesn’t have to know anything about Progressions or our protocol.
Keyboard Mapper
This was a great step forward. Progressions could now use a wide variety of external tools to generate sound. And I added support for building up a library of sound modules so you could easily load them into new projects.
But not every sound-generating tool is a playable instrument. For example, months ago chee rabbits had demoed a sampler tool with a “beep” sample:
What if we wanted to build a sound module out of a collection of independent sounds created by one or more external tools? I mean, of course we want that, right?
Translated to the context of Progressions, why not create a sound module that can map each key to a distinct sound created by an external tool that has no idea we exist? Next stop: a Keyboard Mapper tool.
Like the adapter, Keyboard Mapper wraps a chosen tool type and takes noteOn/noteOff/stopAll wiring code. But instead of wrapping a single doc, it provides a piano keyboard where each key maps to a separate automerge doc of the wrapped type. It allows you to create a new doc per key, or copy an existing key’s doc to all keys to start from its settings:
Once you have your keys mapped, it’s a simple matter of pressing play:
Progressions doesn’t know the keyboard mapper is calling a bunch of different docs (or tools) under the hood. And the tools don’t know they’re part of a sound module.
Reflections
The malleable software essay envisions “a software ecosystem where anyone can adapt their tools to their needs with minimal friction”. Patchwork made it relatively easy to compose tools in ways their authors didn’t expect, in the pursuit of my own somewhat idiosyncratic goals. But my approach highlights a tension in the design space between two models of tool development (what we might call “independent” vs. “coordinated” approaches).
The coordinated approach is the more traditional model for integrating tools via agreed-upon assumptions (like standards, contracts, or APIs). The independent approach assumes that tools are built in isolation, constrained only by the minimal requirements for building a tool that can run in Patchwork. What are some of the tradeoffs?
Since I had no shared assumptions to rely on, I ended up depending directly on the implementation code to write the wiring logic for each wrapped tool. In practice, this would require freezing any wrapped tools in place (even across the equivalent of patch updates) since changes to their implementations might break our adapter code. It also means you have to examine and understand these implementations well enough to write the wiring logic (and this has to be repeated for each external tool you want to wrap).
In the coordinated approach, as long as the protocol is fixed, we don’t need to worry about external tools’ implementations. But since in reality protocols evolve, we do need to collectively ensure either that external tools implement a compatible version with ours, that changes to the protocol are backwards-compatible, or that we have something like the right lenses to convert between versions. This also means all the developers of the tools we want to compose would have to agree to those protocols, which is a significant impediment to freely composing tools in the patchwork ecosystem as a whole.
Of course, we can imagine a middle ground where tool developers document at least some API, making it simpler to write adapters without depending on implementation details. Or what about lenses between protocols that have sprung up in isolation? I’m really just scratching the surface here.
These experiments with composing independent tools weren’t meant to provide a clean solution, but to highlight some interesting design questions for future work. In any case, I have a new suite of tools I’m using on a regular basis, with the potential to keep expanding as more folks develop tools for Patchwork.