dear mimi
so imagine this. you’re in a folder. the folder is a(n infinite) canvas. all your files are there, live and living and usable, editable with your tools. now you invite a friend (me) into the folder, and we both see the same canvas. as you move files around, they move for me. when you play a video, or listen to a sound, i hear it too. we’re in one space.
what do you think of that?
do you remember when we were in the back of a taxi in Chinchón and i asked “hey what does ■■■■■■■ do that you couldn’t?” and then you started listing all the features of ■■■■■■■ and how they work and what models they use? and you told me they take care of your microphone and speaker setup stuff, they detect meetings. the system integration, and the packaging. that’s the product. i think we can do this. or, like, a much worse version of it that is also a million times cuter. 2026 is the year of softer software on cuter computers. so is 2027. actually all the years now i think.
okokok so imagine this: we’re in a video call. your face is up there, and mine’s down here, and then over here on the left there’s an empty box. we’re talking. we’re chittering, we’re chattering. we’re yap-yap-yapping through an idea for a tool and there’s a nosey little Large Language Model running, listening, building it.
as you talk in a call, a local speech-to-text model running in your browser is
transcribing everything you say. it writes it into the call’s document (on a
.transcription field) with your name before it irc style (<mimi>). we neatly
sidestep the speaker problem by having each person only transcribe themselves,
and automatically pop their own name before their messages. the LLM is prompted
by every new transcript line. it gets the whole log as context. it generates
the code for the tool we’re talking about, and pins it in the chat in a place
we can see it. we focus on the design.
what was that thing Hestie mentioned recently? the Perlis quote.
“A programming language is low level when its programs require attention to the irrelevant”
— Alan Perlis
yes, that’s the one. you know, i think by that definition, anything other than talking to you about a design while a machine builds it is low level.
this is the closest i’ve felt to concentrating only on the relevant: what is this, who is it for, how does it feel, how does it help? the Magic of Patchwork™ already means i don’t have to think about storage or sync, with its no frills DOM-first javascript function tool interface. and these new LLMs are very good at javascript.
the local transcription is special. we’re editing a shared underlying Automerge doc. we’re collaborating on the document together and our voices are the instrument. you can turn off transcription and you won’t be transcribed, because everyone’s only transcribing their own voice. analyzing only your own voice feels so much less creepy than the current default of analyzing the voices of everyone else, often without explicit consent.
we end up with a shared artifact of the conversation. you can also open it up in a text editor tool and edit it when it makes mistakes. you could make a tool that lets us annotate the chat with comments and drawings and links to other things.
i really like this shared artifact thing. a chat log is a shared artifact too. oh, that reminds me. i also built another thing. the code was all generated by the war criminal Claude, and i might want to try a clean room write by hand later once we’ve felt through the UI and UX. who knows. maybe we don’t do that anymore. but i do know that it feels inhumane for any person to ever read code generated by an LLM. and if we’re ever going to want to read this code, somebody should write it. but maybe we don’t want to do either anymore.
nevertheless, you can make text chat rooms now too. like discord or IRC or whatever. you can make one like any other document, invite whoever you like. send messages, voice notes, meatspac.es-style 2s gifs. did i ever tell you about meatspac.es? it’s where i met lb.
you can have a call in the chat. you can send in a patchwork link
and it’ll embed it as an editable tool. who needs codeblocks when we can have a
text editor. you can also invite a computer. if you start a message with
@computer it’ll prompt an LLM. the LLM gets the whole chat log as part of its
context. if there’s a call attached to the chat, it’ll get the call transcript
as context too.
the LLM has some preprompt stuff to make it really favour making and pinning a Patchwork tool. if you send a voice note that starts with “computer,” it’ll be treated as a prompt. you can go full star trek captain janeway seven of nine earl grey hot. sending voice notes to make tools is about the most fun i’ve had using a computer since making that Habbo Hotel hacking client in Visual Basic 6 when i was a teenager.
when a tool has been pinned, the LLM has access to the tool running in an
iframe.
that gives it a safe place it can evaluate code, and investigate the rendered
DOM.
the llm gets fed
console.log,
console.error,
window.onerror
etc back in as prompt, which lets it get into a debugging loop by itself. i
watched as it generated a whole TfL-aware centrepoint picker. text entered into
the search field, button clicked. at one point it returned to the chat and sent
“The TfL api is really hard to use!” and then carried on.
you gotta put the llm inside the execution environment. i think that’s why they work so well with bash, it’s inside the execution environment and can prompt itself with stdio. the closer we can make the loop, the less the distinction between what is written and what runs, the better it gets. maybe we should be running all this in Squeak.
both of these things, the call and the chat, can be brought into the space folder viewer. you know, the other day when you text me asking about the price per person and i started doing pizza math it annoyed the fuck out of me that i couldn’t throw a calculator up beside us in discord and open up luma and sketch out the numbers live. i had to do it all on the calculator app on my phone, log into the ticket site alone on my computer, and then type it all up to you afterwards like it’s the Victorian times.
but here in space, we can talk and invent calculators and start using them together, and we can draw right on the tablecloth, and when we’re done we can make a new chat and a new call and leave all that behind as an artifact that shows our whole little digital moment. meetings are these old transcripts surrounded by a handful of links and files.
there’s something real cute about how people behave when they’re talking in the folder. like, because we all know everyone’s eyes are on the video a person’ll pan and grab a file from somewhere else on the canvas, drag it over and shake it “i mean this”.
anyway you should come work here so we can make all of this good.
~ chee