Programmable Ink
- title
- DrawDeck
- dated
- Morning Yawn, Twenty Twenty Six
DrawDeck is a small speculative design study in composable dynamic behaviour.
DrawDeck is is very much ‘not the thing’. The choices that I made are quite obviously wrong for a context like playbook. The idea is to try to “take a sideways step in design space” by relaxing the constraints and requirements we have for playbook. Instead, my approach here is to take a speculative and slightly absurd premise, which I then extrapolate according to it’s own internal logic.
This technique has a few advantages. Firstly, it gets us out of dead ends without an obvious path forward. Secondly, it moves us away from strong attractors (local optima).
I think of this project as providing a stepping stone. It provides ideas that we can try to horizontally transfer into PlayBook.
Demo
There is a Demo that you can play with. Because it relies on the Claude API, you have to provide your own API key to use it. Don’t expect it to “work” in any real sense.
Walkthrough
In DrawDeck you get a bunch of blank index cards. You can pick up a card and draw and write freely on it with a black pen.
If you place cards near each other, they form a pile. Piles are shown with a grey bounding box that you can see when hovering over a pile. You can freely add and remove cards from the pile. You can move the whole pile at once by dragging the grey bounding box. If you put two piles near each other, they merge to form a single pile.
Runestones
In DrawDeck, dynamic behaviour comes from Runestones; small square tiles that house different animistic spirits. Each stone has a different personality with an intrinsic desire or effect. To use the magic inside a stone you place it near a card or pile. In this prototype, there are three different kinds of stones (though, one could imagine many others): Sensing stones, Organising stones and Cursing stones.
Sensing Stones observe and respond to drawings on cards.
For example, if you place the Perception stone next to a card, it will tell you what it sees on the card, by producing a new card.
Similar stones include Divination, which responds with an obscure forecast.
Imagination, which responds with an oblique suggestion.
Inquiry, which you can use to ask questions about what’s on the card (more on this later)
By default, cards are produced somewhere nearby, but you can give it a specific target location.
A stone will either produce a new card if there isn’t one, or replace a card if it decides the old one needs to change.
If you place any of these stones next to a pile, they will produce a second pile.
At any point in time, you can simply pick up a card, remove it from a pile and put it somewhere else. This will upset the spirit inside the stone, but it will not resist. It has no choice but to produce a new card to replace the old one, restoring balance in the universe once more.
Another type of stones are Organising Stones. These act only on piles. The spirits inside these stones desire piles to be organised in some way.
Alignment desires cards be neatly aligned in a vertical stack.
Composition
By placing different stones together, you can start to loosely compose more complex behaviours.
Stones can communicate with each other through ‘dreams’. This mechanism allows you to attach some information to card. Rather than putting new cards “on the table”, you can tell a stone to put them into the dream of another card.
This allows you to do something like the following:
Use the Inquiry stone to ask if the checkbox is checked. Then, tell the stone to put the answer into the card’s memory.
When we create a pile, the stone will now attach its answer to each card respectively.
We can then use the Sorting stone to sort the cards.
Dreams remain underdeveloped. Initially, my idea was to have ‘dream space’ be a parallel hidden world that, for the most part behaves much like the regular environment. (Allowing you to put cards and stones inside other cards).
Conflicts
Some stones are not friends. Their personalities are intrinsically incompatible with each other. If you place both of these stones near the same pile they’ll respond by starting a fight.
Curses
Curses are modifiers that change properties or behaviours. If you place a Cursing stone near a card, as the name suggests, it will curse that card.
For example, The Dimnation is a curse that makes everything on the card grey-out:
Curses are represented by little tokens. You can pick up these tokens, and place them on a card. You can also remove them, this lifts the curse. You can also move the curse to another card, which transfers the curse.
Thoughts & Conclusions
- Piles are a useful mechanism
Perceptually, placing two objects makes them form a group (gestalt). Computers usually require formal grouping. Piles are a useful middle ground between formality and informality. It gives us some of the benefits of a formal group without explicit grouping. The pile can just as easily be broken, or merged with another pile.
- Stones run “when they feel like it”, and execution is very slow.
The system is highly non-deterministic. Stones sometimes trigger by themselves, sometimes you have to bump them to trigger. Because stones wait for each other, effects take time to cascade through the system. On the surface, this seems problematic, but it feels surprisingly good. Part of this is the explicit animation, which makes the system feel more alive and less like a machine. It also makes the state of the system quite ledgeble.
- Being able to mess with “program state” at any time feels really good.
If you think of a stack of cards like an ‘array’ in a regular language, it’s kind of non-sensical to be able to just pick up an element and move it somewhere else. However, in drawdeck, it feels quite natural. The user acts on the same level and timescale as the program. The stones function as a kind of “self-healing” mechanism. The user can intervene in the process at any time, and the system will just adapt to it.
- Natural language queries feel really good.
Over the years, we’ve tried to approach the problem of spacial queries in various ways, with ideas like ‘flux’, or by tracking sameness by duplication. While still fallible, using a visual reasoning model allow us to sidestep a lot of the complexities in one fell swoop. We don’t need to rely on some sort of indirect metric, we can just ask a question directly. I think there’s something about asking the question that implies some context that the model can use to reason about what it’s looking at. (i.e. “is the checkbox checked” implies there is a checkbox).
Assuming image understanding in models will improve, this seems like a pretty good bet for how to approach spacial queries in the future.
One open question is where to draw the line between using models and using more traditional approaches. You could imagine for example, combining models with a more mechanical QSR parser, which bottoms out at asking a model to categorize something into a few buckets: yes/no, A/B/C, same/different. Alternatively, we could simply throw the whole page over the wall and ask the model to interpret the whole thing.
- The mechanism behind Curses feels like a really elegant solution to the ‘override’ problem.
We’ve long been thinking about the tradeoff between mutating a value, v.s. overriding it. The former is more imperative, but it doesn’t always make sense to mutate a value, especially if the change is just temporary, and we want to be able to revert back to the original value. The curse model seems like a really elegant solution to this problem. It allows us to have temporary overrides that can be easily reverted, and it also gives us a nice mental model for how these overrides work, and allowing the user to intervene in the process by picking up and moving curses around.
- The exact boundaries around stones and their composition doesn’t feel quite right.
Combining an ‘observe’ stone with a ‘sort’ stone makes sense from a programming perspective, but it relies on card having memory as an intermediary. As a possible alternative, we could put the observe mechanism directly into the sorting stone. I could just use the natural language “sort the cards by whether the checkbox is checked” and skip the whole memory step. i.e. maybe observing is just a skill that all stones should have? Another alternative could be to have a more explicit binding mechanism where the observe stone feeds directly into the sorting stone, without needing to go through the cards memory.