title

Informal ink augmentation

dated

Fall 2023

The very top part of our big use case map contains drawing activities where semi-structured ink and simple augmentations could be helpful. Particularly, these are examples where most drawing tools (muse, tldraw, etc.) rely on using built-in components like cards and arrows.

There are two well-understood approaches, each with different tradeoffs:

Toolbar with built-in tools

• Pros: Can capture some of the drawing feeling, but still adds some indirection
• Cons: Adds chrome to the screen, requires built-in primitives

Library of components

• Pros: User can create their own components
• Cons: We lose the pen-and-paper feeling. The user needs to context-switch in order to find and drag in components.

Both of these solutions feel unsatisfying in light of our project goals. We would prefer not to have different tools and the use of a toolbar has often been marked as “obviously wrong”.

One way to break out of this “double-bind” could be to use a tagging & recognition strategy. With this, the user could mark a set of strokes as having some behaviour after drawing them, rather than forcing the user to decide upfront. This could allow the user to maintain the feeling of pen & paper (not breaking context and flow), and then gradually enrich the strokes by adding structure only when needed.

To this end, we started exploring the possibility of weakly augmented ink: how many affordances can we get that usually require things like a “box” and “arrow” tool?

This experiment attempts to see how much of that augmentation we can get by having a very general recognizer, and have that recognition augment the selection affordance in the style of ScanScribe (Saund).

Marcel implemented a very general (domain unspecific) recognizer that is able to distinguish between a number of perceptually salient features: enclosed shapes, detailing strokes (like writing) and connectors (arrows). The recognizer is relatively simple and surprisingly reliable.

We built this recognition into the selection system. A single tap will select a single stroke, and a double tap will select a perceptually salient group. This allows the user to get simple card-like affordances without using a card component or even explicitly marking something as a card. The system will remember the last grouping that you selected, so this gives us group-like affordances without explicit grouping.

This is a concrete implementation of an approach to recognition that plays a useful yet non-invasive and non-destructive role in the system. On the one hand, the recognizer is simply speeding up the selection process, so we don’t rely on it and you can always fall back on using a more explicit lasso selection gesture. On the other hand, the speedup does qualitatively change the feeling of interacting with ink. Hand-drawn shapes start to feel more like things rather than merely a bundle of separate strokes.

Combine this with some of the earlier ink deformation primitives, and we could have a pretty good box & arrow-style whiteboard without needing explicit construction or built-in primitives.