Success of spreadsheets
Making end-user programming powerful and ubiquitous is a central goal within our lab’s broader vision of making better tools for thought. Spreadsheets are a widely successful end-user programming tool, and in particular we are inspired by their support of gradual enrichment and programming in the moment.
Spreadsheets support gradual enrichment by allowing you to quickly put text and numbers in arbitrary boxes as if using a pen and graph paper, do basic math as if using a calculator, move things into neat rows and columns when you find yourself making comparisons, and add more formula-driven cells and labels if/when necessary as your model evolves—incrementally building dynamic complexity over time. You might eventually polish the formatting, clearly label input cells, and turn the model into a reusable piece of situated software. Each bit of effort to refine the model is a small investment made in a moment when the payoff is immediate.
When software allows adding dynamic behavior as part of gradual enrichment, it supports programming in the moment. Spreadsheets support dynamic behavior with live, reactive, declarative formulas you can place anywhere you would otherwise have data. They treat data and formulas equally: where they appear in the interface, how they are edited, and how they reference each other. This interchangeability supports programming in the moment by allowing you to keep working with the same tools, in the same mindset—thinking only about the specific thing you are modeling—even though you have switched to programming. They don’t force you to stop and switch to your programmer hat.
Through this lens, spreadsheets stand out as one of the more successful and prevalent instances of end-user programming. They give a broad range of users access to computational power while maintaining flexibility through the use of the familiar and adaptable grid—what Bonnie Nardi calls a visual formalism.
Spreadsheets are immensely popular for uses that fit neatly within a grid of cells (and have been creatively misused enough to prove their flexibility), but they are not the end of the road for end-user-programmable media.
Despite the spreadsheet’s computational advantages, most people don’t reach for a spreadsheet the moment they want to think through a problem or communicate an idea.
Enduring value of pen and paper
Despite an overabundance of other tools, many people still turn first to pen and paper for ideating, note taking, sketching, and more.
There is something special that happens in the brain when making marks by hand, and a large body of research suggests it has various advantages, whether for learning and psychological development, retention of material, or activating more pathways in the brain.
What the hand does, the mind remembers.
We believe these attributes contribute to pen and paper’s staying power:
Flexible & free-form
While word processors constrain text to predefined lines, on paper you can write anywhere. You aren’t limited to writing text either—you can draw diagrams, sketch images, and doodle in the margins. You can combine these styles of expression freely, without stopping to change tools or select formatting or ask a system for permission.
You can pick up a pad of paper and begin writing without hesitation, and without working through any system of menus or dialog boxes. This is essential for fluidly expressing and thinking through ideas—working at the speed of thought. This is also essential when integrated into a larger activity, like two people having a conversation over a pad of paper or a student taking notes alongside practicing an instrument.
Intimate and tangible
You can touch and feel a pen and paper, hold them, and move them around—there is nothing mediating your interaction. You don’t have to take care to avoid touching the wrong place and altering or losing your work, as is often the case with a digital tablet. You receive full tactile feedback even down to feeling the friction of your writing implement dragging across the paper’s surface. This results in a strong sense of direct manipulation and working with the material.
Informal & “sketchy”
The natural grain of digital media is to create precise and polished artifacts, but the fidelity of the tool you use should match the maturity of the idea you’re working with. Nascent ideas are tiny, weak, and fragile. They require attention, nimbleness, and open-ended thought to survive. When a tool pushes you into inappropriate levels of precision, it can slow you down and suppress creativity: you want to quickly throw a box on the screen but by the time you’ve chosen the drop-shadow depth and rounded-corner radius, that whiff of an idea has dissipated. Handwriting allows looseness, quickness, and incremental refinement—attributes hard to come by in the world of computers.
Connected with human life
Pen and paper are deeply entangled into our lives and shared cultures. Most people have been making marks on paper since infancy. Even if we don’t consider ourselves to be talented artists, we all have distinct and personally meaningful ways of expressing ourselves with ink. Simply put, it feels good to mark on paper.
Digital pen and paper, in the form of tablet/stylus interfaces, have extended the power of their traditional counterparts with sophisticated editing interactions like copy-paste, undo, and infinite zoom, while trying to keep some of paper’s unique charms.
We want a digital pen and paper that maintain as many of the above attributes as possible, but with all the benefits of a dynamic digital medium. The most important being programmability—describing new behaviors for a sketchbook to take on, automating and processing and simulating.
What does it mean for a sketcher to program as they sketch? How might this new power aid them in their life and work? While we can’t foresee all the possible uses of a tool that doesn’t exist today, we can take inspiration from uses of traditional pen and paper to imagine a few use cases that hint at a larger set of possibilities:
- An event planner draws up a list to pair things that need to be done with people available to do them. After a bit of connecting lines and boxes, they add some behavior to highlight the remaining unmatched people/items to make them easier to see. Later on, they need to do this again for a different event, so they turn this first version into a template for easy re-use.
- A furniture designer sketches patterns for a table leg. After they decide they love a pattern that repeats a vine figure in a spiral down the leg, they automate the hand-drawn spiral pattern so they can try out different vine shapes instantly as they sketch. This helps them explore the design space quickly and stumble onto entirely new designs.
- An engineer on a circuit-design team is trying to illustrate to a colleague how their signal processing might be flawed. They roughly sketch what an incoming signal might look like, and then program in a model of what the circuit will do. The engineers take turns sketching on the page, trying out different signals and models, coming to a shared understanding of the problem.
These three examples share an interesting feature—they all illustrate programming in the moment while sketching. This is our ambitious vision for a programmable sketchbook.
However, this vision raises multiple layers of research questions at once:
- What mental model or programming concepts are the right fit for interacting with hand-drawn marks?
What features of a formula or programming system are important? Is a formal language even exposed?
- How should an interface be designed for programming in the moment while inking?
How do you interact with dynamic drawings? How do you see relationships between ink and the program? What affordances could make pen and tablet programming ergonomic? How do you edit a program “with pen in hand”?
- What might you make with a programmable sketchbook?
What are the most interesting use cases for this tool and how will it feel to interact both with the tool and with your creations? Which uses should the natural grain of this tool encourage?
A common research approach would be to start by designing a programming system from first principles. But because we believe the uses for this tool will be novel—and therefore unforeseeable—we don’t want the design to arbitrarily constrain the potential usage space.
Consequently these questions form a dependency stack in the reverse order: the use cases and aspirational feel of the tool determine the right design for the programming/editing interface, and in turn the way you interact with the models you build determines the right mental model and formula/programming system design.
If we want to start at the beginning of the dependency chain to explore use cases and the feel of a tool like this, we need to build a vertical slice through this whole stack to produce a working prototype we can actually play with.
For the bottom two layers we chose naive/incidental solutions—whatever was pragmatic, expedient, or best guess from our previous research experience in this area—so we could spend most of our time on the top layer.
The essence of the Inkbase project was to experience that moment of making a sketch, enhancing it with dynamic functionality, and interacting with the result.
We hoped to unlock the next phase of this line of research by discovering interesting use cases, starting to understand the required ergonomics, and developing better intuition for how the tool should feel.
While we haven’t seen work that directly achieves Inkbase’s goals, there has been a plethora of relevant work over the years, spanning from the 1960s to the present. A selection of projects we have taken ideas and inspiration from:
To answer the research questions above, we built Inkbase, a native Swift application for the Apple iPad and Pencil, implementing a programmable sketchbook:
Below is a basic primer on Inkbase’s features followed by a sampling of things we made with it.
Inking and selecting
To support gradual enrichment, Inkbase is a sketchbook first—by default the pencil inks on the canvas and touch/drag does nothing:
Selection is a quasi-mode—while holding two fingers down on the canvas, touch an object to select or use the pencil to lasso a selection. Ink can be moved around the canvas by dragging it with a finger or pen while holding down an additional finger, or by just dragging objects that are already selected:
Objects and properties
Inked paths are the basic building blocks of everything you can do in Inkbase. We wanted them to be as open-ended and malleable as possible—they are live vector objects with dynamic properties. Selecting an object opens its property inspector:
All properties are live—including the original path as vector data—and can be inspected and edited:
Additional arbitrary properties can be added:
Properties can have a value or can be an expression:
We wanted the reactivity of spreadsheets where the dependencies between properties are tracked by the system and values are automatically recalculated when necessary.
Inkbase uses a homegrown flavor of Lisp. In addition to being simple to implement (which allows us to easily interoperate between the core engine functions and end-user code), everything in Lisp is an expression (everything returns a value), which composes nicely with the idea of reactive properties. This is similar to spreadsheet formulas which always return a value into the cell.
Accessing properties is done with the
? function (with a shortcut button available in the Inspector for quick insertion of a property reference).
Property Lookup using
(? object property) returns the value of the given property on the given object, and records the dependency in our reactive dataflow system. Now when that value changes, anything depending on it will be updated as well. This is analogous to referencing another cell in a spreadsheet formula using its name, e.g.
Inkbase objects always exist somewhere on the page. This makes accessing them by their position very natural.
We created a spatial query system where the user can ask for objects in a specific region, inside a specified path, or in a general direction (e.g. “to the right of…"):
The results of these queries are also reactive (refresh based on changes on the page), so they compose with the rest of the property system:
Thus we have programmable ink:
Lastly, our reactive system also allows for side-effects—pushing new values into the dependency graph “from outside”, using the
Property Setting using
(! object property value) sets the given property on the given object to the given value. This is equivalent to the user opening a property inspector and typing a value, but done programmatically.
We can use this, for example, to make all of the marks to the right of this box react to its
Sampling of use cases
“What is this for?” is a central question for this kind of work. We built models with Inkbase in the pursuit of our central goals of discovering interesting use cases and gaining intuition for the ergonomics and feel of the tool. Below is a sampling of these, each pointing to a different type of use which feels promising and underserved by existing tools.
A class of use cases we find interesting are common uses for notebooks/sketchbooks where the system can provide helpful automation or bookkeeping. A basic example is the habit tracker, a motivational tool for establishing or maintaining habits. They revolve around the emotional and visual satisfaction of completing an activity and lengthening the streak, or the discomfort of breaking one.
You could make a habit tracker to, for example, increase your consistency at exercising every day. You write the name of the habit, draw boxes representing each day, and mark each box if you completed a workout that day:
The key feature of a habit tracker is that you can easily visualize the length of the current streak. You could add behavior to your drawing to color the boxes of the current streak red so it’s easier to see—particularly helpful once you’ve been tracking for a while and the page is full:
This can be done the same way we showed in the Inkbase primer above, by setting the box
stroke property to
black depending on whether it is part of a streak, and determining that with a set of spatial queries (see discussion of spatial queries for more implementation details).
Now when you miss a day and the streak resets, it’s clearly visible (and extra painful—more motivating):
You could make a streak even more clear by drawing a bracket underneath:
You could make the size and location of the bracket dynamic, so it changes in real-time as you use the habit tracker:
This can be done simply by setting the bracket’s left and right to the same value as the left of the leftmost streak box and right of the rightmost streak box, respectively. When you mark a box, it triggers recomputation of the streak, which trigers recomputation of the bracket’s bounding box, which automatically transforms the bracket.
Paper notebooks are a great place to sketch out designs, though they remain static. The only way to adjust them is to draw them again, and there’s no way to simulate behavior. Sometimes you want to build a model and play with it to better understand the constraints or behavior of a system.
User interface design often starts with a sketch to work out general layout and eventually actual pixel positions of various elements. Perhaps you have a constrained screen size—like this reprogrammable audio effect box with a small OLED screen—and you want to determine the right sizes and spacing for the three UI tab indicators at the bottom:
You could start with a quick sketch of the basic layout:
The upper lines represent margins, the middle bracket is the width of the tab section, the bottom bracket represents the total screen width, and the boxes are tabs.
You could add dynamic layout constraints by connecting the margins to the screen width to make them equal and keep the tab area in the middle of the screen, and size the tabs to match the width of the tab area:
To explore the model more fluidly you could make a slider to control the margin size dynamically:
You could explore different numbers of tabs without redrawing the sketch:
Finally you could add labels for widths so you can take the values and implement them in your UI:
Learning and epistemic actions
Sketching can be used to think through and better understand complex concepts. We find ourselves wishing our sketchbook could be used to draw concepts and add dynamic behavior so we can manipulate the concepts—take epistemic actions—to increase our understanding.
Perhaps you are learning the first principles of computation, studying transistors and logic gates, but don’t understand why a NAND gate is important.
You could sketch basic gates, like AND and OR:
You could bring them to life with dynamic behavior so they can sense inputs and produce correct outputs (see spatial queries discussion for more detail):
You learn that a NAND gate is a combination of AND and NOT, but you could construct both yourself to play with them, compare them, and see the internal state to improve your mental model of how they work:
You learn that NAND gates are significant because all other gates can be built from them, but you want to see this in action yourself. You could construct, for example, an AND gate from two NANDs and test its behavior:
This project was inspired by this logic gates Etoys demo, which allows children to play with simulations of circuits using a bin of reusable components that react to what they can “sense” (what’s visible on the canvas):
Although math is typically viewed as a tool of precision, people who work with quantities often want to work “sketchily”. When exploring a space of possibilities or explaining a big idea, they want to think quickly and loosely and not get bogged down in unnecessary detail.
There’s a reason so many innovations and discoveries have been based on back-of-the-envelope calculations—paper is great for this mode of thought. We hope it’s possible for computers to expand on paper’s capabilities, speeding the tedious or error-prone parts of mathematical sketching and helping sketchers explore further.
Sketching is also used for teaching, as it aids explanation of complex concepts and allows for the creation of visual aids on the fly as you speak. The low fidelity of sketching is key. It allows focus to remain on the big picture concepts rather than the details, as well as creation of bespoke visual aids tightly fitted to the audience and working at the speed of conversation.
But you often want dynamic behavior—you want to draw a sketch and interact with it or have it react. To do math on sketchy objects, you need “sketchy math”.
Perhaps a friend is learning basic physics. They’ve learned how velocity and acceleration are defined in terms of derivatives, but they want to better understand the concepts. You could sit down with a tablet and talk it through.
You could draw a graph of a bouncing ball:
You could annotate this graph as you explain:
“At this point, the ball is moving up, which we know because this line slopes up. And at this point, the ball is moving down, because the line slopes down.”
“That means the derivative here will be positive, and the derivative here will be negative. Let’s have a look.”
Here you grouped the axes to have them recognized as special axes and straightened out, drew a triangle—a derivative symbol—and it was detected as a triangle and colorized to show it has special behavior, and drew and grouped a new set of axes to hold the output of the derivative operation. The computed derivative of your bouncing ball sketch is now displayed below.
See sections on recognition and microworlds below for more detail on how this was implemented.
“As we expected, the derivative is positive here and negative here.”
Shown above is a critical feature: you can continue to use the sketchbook as a sketchbook, without interfering in the computations.
“So that derivative is velocity. If we take the derivative again, we get acceleration…”
These operations are all live and reactive, so you can manipulate or re-sketch the paths and see the results as you draw.
“Let’s draw a more typical path for the ball by erasing our old path and drawing a new one. It looks like the velocity is slowly decreasing, and that gives us a more-or-less constant negative acceleration. Gravity!”
The physics example uses a triangle operator to compute derivatives. The chart prototype was also built with support for a more advanced operation, convolution. Convolution has an intimidating symbolic definition:
But in practice, convolution has an intuitive feeling. Convolution “smears out” one function, using a second function—the kernel—as a template for the smearing. Below, we invoke convolution with a hand-drawn star, then vary the shape of the kernel graph by stretching its width. Watch as the function’s two humps become blended together in the output:
This is an example from statistics. Suppose the distribution of wingspans in a population of birds is bimodal—it has two humps. When we take samples of this distribution, there will be some error inherent in our wingspan measurements, as represented by the bell curve of the kernel. This convolution model demonstrates that if the error level is too high, the two humps will blend into one and the bimodality of the birds will be missed.
In both of these examples—physics and statistics—playing with hand-drawn graphs helps the sketcher explore “what if?” scenarios in their domain, while building intuition for the underlying math. The computer contributes to the sketcher’s exploration by providing instant feedback. A system using the traditional computing approach of asking the user for algebraic expressions for the input curves would be tedious to use and distract from the underlying intuition. A computational sketchbook is a natural setting for this sort of “sketchy” exploration.
In research we often have under-constrained design spaces to explore. We sometimes use the ingredients analogy at the lab for thinking about the design of a tool—some ingredients are essential to the nature of the thing (apples for apple pie), some are necessary for it to work (leavening agent for bread), some don’t work well together (lemon juice and milk—unless curdling is intended), some don’t taste good together (garlic and chocolate), and some need to be used judiciously (ginger).
Through our design iterations and use case exploration we found certain ingredients were key to the feel we wanted Inkbase to have, and others were required by some or many of our experiments. Below are three that surfaced repeatedly.
Working with ink on a canvas naturally encourages thinking spatially: location is intentional for every mark you make, as are the spatial relationships between marks. As a result, querying the canvas for other marks ended up being a core part of almost every model we made, making them a likely key ingredient of Inkbase.
For example, in the habit tracker we define whether a box is marked as whether it contains another mark, as determined by
Whether a box is part of a “streak” is more complex—only the marked boxes in the most recent (rightmost) unbroken chain are part of a streak. This can be expressed with some (spatially) recursive logic: I am part of a “streak” if I am marked and either the box to my right (using
query/r) is part of a streak, or I am the rightmost box:
(if (? me :streak) "red" "black")
(? me :marked)
(? (? me :to-right) :streak)
(? me :rightmost)))
(first (query/r me 100))
marked value with
(query/r me 100) and a recursive dependency to determine if part of a streak.
While our spatial query library isn’t exhaustive—we added query functions as needed while using the tool—we did accumulate quite a few functions and helpers. Notably all are implemented in user-space (with the exception of path containment which uses the performant native iOS polygon library).
Our naive queries did run into some challenges with reality, however: natural strokes don’t always align with the logical containers needed for behavior. For example, when marking boxes by hand, you don’t necessarily stay within the lines:
Realistic example of hand-marked boxes.
There isn’t an easy way to determine whether the 2nd box above is checked:
(query/inside me) will match nothing because the mark isn’t fully contained inside the box’s polygon, and
(query/intersects me) will match the mark both for the 2nd and 3rd boxes because it overlaps both. What we actually want here is fuzzier querying such as finding marks that are mostly or primarily inside a path rather than strictly so (also the “boxes” aren’t always square or even closed paths).
Or imagine you draw a grid the common way by first drawing a large box and then slicing it with a series of vertical and horizontal lines:
Realistic example of drawing a grid of boxes.
How do we query the contents of the 3rd box in the 2nd row? It isn’t an object, but rather a negative space between other marks. We don’t have a convenient way to do the polygon clipping needed to break the resulting grid cells into separate paths to contain their own logic or for use with spatial queries.
Helper functions could make it much more concise to scope spatial queries to relevant objects or groups, or to slice a set of objects by another set—sort of the spatial equivalent of common array operations, but for polygons/paths. As you might split a string by spaces to get an array of words, you might want to split a large containing mark (the outer box above) by a set of small marks (the vertical/horizontal lines). Further work could make spatial queries more powerful.
Another area of interest is providing more visualizations to aid the user in constructing and understanding spatial queries. In the logic gates example, each gate “senses” its inputs and outputs using simple spatial queries:
(first (query/tl me 100)) ;; top left
(first (query/bl me 100)) ;; bottom left
(first (query/r me 100)) ;; right
(? (? me :input1) :state)
(? (? me :input2) :state))
(! (? me :output) :state (? me :state))
It then visualizes the results of the spatial queries by drawing lines between the gate and the inputs/output. Because the queries are live and reactive, you can see the inputs/outputs “attaching” to the gate when they get within the query distance:
(? me :center)
(? (? me :input1) :center))
(? me :center)
(? (? me :input2) :center))
(? me :center)
(? (? me :output) :center))
Visual feedback for spatial relations turned out to be an important part of this experiment—seeing the connections update live helps understand the simulation.
Inkbase programs are able to add shapes to the canvas dynamically, as seen in this example, using a visual style distinct from user-drawn ink. We took to calling this the “computer voice”. One possible direction is to double down on the computer voice, providing default visualizations for all spatial queries and other program information. Another direction is doing the opposite—allowing only user-drawn marks on the page. We played with both approaches in our experiments but don’t yet have an opinion on this.
Coming into this project, we had mixed feelings about the role of ink/stroke/shape recognition. On one hand, interesting dynamic behaviors require that the computer have some understanding of the intent of marks, and recognition of inked marks feels like a natural way to interact with a programmed system. If this intent isn’t provided by recognition, it would have to be provided by a non-ink-based interaction—such as manual tagging or duplicating from a template. If it’s a design goal for the sketcher to spend most of their time drawing with a pen, recognition feels necessary.
On the other hand, there are few experiences in computing more frustrating than trying to be understood by an opaque system that claims to be “smart”. Mistakes in recognition—which will, realistically, be quite common—leave the user feeling powerless and mystified. This is directly opposed to our design principles: users should have total control over and visibility into their working environment.
This tension was apparent in our use cases. For example, in real-world use of Inkbase, the habit tracker wouldn’t be programmed onto a blank canvas, but rather added over time as gradual enrichments to a pre-existing tracker you’ve been using for a while. This means implementing it over an existing page that’s already partially filled out:
Example of real-world bullet journal from Super Mom Hacks.
Imagine wanting to add dynamic behavior to this page at this point. You would need to be able to differentiate, for example, between the marks making the boxes and the marks filling them in. Shape recognition is probably necessary to handle this realistic case, but we aren’t sure how to resolve this tension between recognition and having a completely reliable tool with maximum user agency.
Perhaps selecting and then grouping or layering would allow us to separate these marks, but it’s hard to do post hoc (whether done by the user manually or by the system automatically—even more challenging), and grouping or layering adds undesirable interface chrome and modes. This approach also wouldn’t allow the dynamic behavior to be automatically applied to new boxes you draw subsequently, whereas recognition would.
We also need to differentiate between strokes with different meanings—perhaps you want to make different marks in different boxes and do different things depending on the marks. See the photo above: some boxes have a
X in them or next to them, some are filled in fully or partially diagonally, and some have a combination of the two.
We currently support basic shape recognition that would allow us, for example, to detect and ignore vertical lines as part of the background, or we could recognize a specific mark for filling boxes in (say a single-stroke or two-stroke “X”, though recognizing multi-stroke shapes isn’t currently supported), but these would need more work to be reliable and ergonomic to use in this way. We are generally skeptical that this can be made reliable.
Another example is from sketchy math, which leans more heavily on recognition than some of our other examples.
Sketchy math example using the $1 Unistroke Recognizer to identify a triangle.
Specifically, there are two instances of recognition:
- When a pair of axes are drawn and then grouped, the computer recognizes them as axes.
- When any shape is drawn that looks sufficiently like a triangle or star, the computer recognizes it as a derivative or convolution operator.
Behind the scenes, these use two different methods:
- Detecting axes works according to simple, bottom-up, purpose-specific heuristics: if a mark is sufficiently straight (
lineiness > 0.8), it is recognized as a line. If a mark recognized as a line is sufficiently horizontal/vertical (
-15º < angle < 15ºor
75º < angle < 105º), it is recognized as a horizontal/vertical line. If a group is created consisting of one horizontal line and one vertical line, it is recognized as a graph.
Example of a mark recognized as a line (
lineiness > 0.8) and as horizontal (
75º < angle < 105º).
- Detecting triangles and stars uses a generic stroke recognition system, the $1 Unistroke Recognizer. This works by comparing drawn marks to template shapes.
Triangles and stars must be drawn in a very specific way to be recognized, which illustrates the reliability challenges with shape recognition. This works well for a demo in which the presenter is practiced in drawing triangles and stars to the recognizer’s liking, but doesn’t lend itself well to organic use. Factors which might reduce this tension:
- Bottom-up, purpose-specific heuristics provide greater visibility into recognition and greater moldability. If a pair of axes fails to be recognized, the sketcher can determine why. Maybe they'll discover the x-axis wasn't recognized as horizontal because it was too tilted. In response, the sketcher can change the way they draw their axes, or change the code to accommodate their own style. This is possible because the recognizer lives in an open programming environment with the sketcher, rather than an inaccessible black box.
- Implicit recognition combined with explicit tagging can be a friendly compromise. For instance, the charts system does not try to recognize every pair of lines as a pair of axes. If it did, either every drawn “t” would have a chance of being detected as a pair of axes, or the axis-detection heuristics would become so complex as to be unworkable. Instead, in our system the sketcher invites the computer to recognize the diagram by explicitly grouping it, rather than letting the recognizer run continuously and indiscriminately over the whole canvas.
- Despite their disadvantages, “black box” algorithms like the $1 Recognizer can be made friendlier. For instance: if the sketcher draws what they think is a star, and the recognizer doesn't recognize it as such, the sketcher should be able to label their mark as a star, adding it to the recognizer's database of known shapes. The next time they draw a star, it is likely to be similar to the last star they drew, and more likely to be recognized. This reflects an advantage of end-user programming: you don't need to teach the computer how to deal with the idiosyncratic styles of every possible user. You just need to teach it how to deal with you, molding it as you go.
Most of our examples were built entirely on the iPad, using Inkbase’s interface. Sketchy math was not. Much of the code that runs it—parsing ink-drawn paths and computing derivatives and convolutions—was written on a laptop in a conventional text editor and then merged into the Inkbase code.
This certainly signals limitations of Inkbase. Building larger, more technical software systems in Inkbase becomes extremely difficult for many reasons, from the poor ergonomics of typing with an on-screen keyboard, to the absence of important programming constructs needed to organize larger-scale systems.
But these limitations are not so damning, and the story of the chart system actually illustrates an important use case for Inkbase: that of the microworld, where powerful building blocks are available to be combined in unanticipated ways.
The charts system is a microworld, in that it gives the sketcher access to charts and operations that transform them, and they can combine these building blocks to explore many different domains and contexts. Perhaps it’s acceptable for some building blocks to be made outside of the system if they can be creatively used within it.
At first glance, the pattern of pre-built microworlds seems like it stands in contrast with programming in the moment, but we think the two can be combined in interesting ways: in Inkbase, the pre-built microworlds are used inside of an environment that is itself a fully programmable system. That means you are not limited to using the components of the microworld. Using the spreadsheet-like reactive substrate, you can connect them to the larger world outside the microworld, or even to other microworlds.
One small example:
Combining end-user programmed behavior with the sketchy math microworld components.
Here we spin a segment around and compute its intersection with a charmingly hand-made circle. As it moves, the y-coordinate of this intersection is plotted on a graph below (as a function of the segment’s angle), and the derivative of this plot is computed. As trigonometry students might expect, the y-coordinate makes a sine curve and its derivative makes a cosine curve.
To make this happen, the sketcher wrote a few lines of code to compute the intersection and feed it into a chart. Then the chart microworld sprung into action, and did its derivative magic. Multiple modalities of computational sketchbook use can be combined, because they share an underlying communications layer—the reactive spreadsheet.
Findings and open questions
Working with the material
The broad concept of working with the material gestures at a different way of using computers that feels more like working with your hands in the real world and less like poking at a model through a glass pane. Interfaces encouraging the user to play with on-screen content seem to flow easily and naturally into progressive refinement and programming in the moment, which makes this way of working compelling.
This sense of intimacy and working with the material could be enhanced by a number of improvements to the tool, especially around making your work always feel safe (such as unlimited undo) and making as much of the program as possible be visible on the canvas and manipulable with the same hand-tools as ink.
Hand-drawn reactivity feels great & new
A central goal of the project was to see what it feels like to make hand-drawn marks, enhance them with dynamic functionality, and interact with them. Playing with Inkbase and the various demos we built felt unique among our experiences with computing media. Using a pen to interact with a lively reactive system is exciting. Even on its own this encourages us to keep investing in this and related lines of research.
Dynamic annotations can be meaningful
Working with Inkbase, we felt hints of how small amounts of dynamism—even just annotations or bookkeeping aids—could aid our thinking, without interfering with our sense of working with the material.
We believe this is an area with large potential to make thinking tools more valuable. In the pursuit of personal agency in the use of tools—the tool being guided by the user rather than the user being guided by the tool—there is an important distinction between providing helpful information versus just doing the work for you. In the habit tracker example, annotations provide a clearer picture but the user still marks boxes and makes decisions—as opposed to a system that automatically queries your calendar and produces a visualization of your habits.
An inspiring project on this front is INK-12, which explores how the combination of digital tools and freehand drawing can support teaching and learning mathematics in upper elementary school. In the example below, a student visualizes a multiplication problem as an array, and can use a digital tool to slice the array into smaller chunks that are easier to multiply. The computer shows the sizes of the arrays and ensures they always sum to the original whole, but it’s the student who decides where to cut them, moves them around on the canvas, and ultimately still does the math.
INK-12 investigates learning mathematics through a combination of drawing and representational tools with dynamic assistance from the computer. Similar to our mental bookkeeping example above, the computer can provide helpful annotations and check the math while still allowing freedom of thought through freehand sketching. Several interesting examples are shown here.
Grouping/compound objects needs work
We consistently found a need to group multiple strokes into a compound object in order to have useful spatial query results or computed attributes—getting the width of a group of marks, for example. We also wanted to use grouping for logical containers or encapsulation of behavior, like when a large set of marks represents a single todo item.
We designed interface affordances for grouping and prototyped methods for working with collections of marks concisely but didn’t arrive at an acceptable solution. Part of this is a programming model issue, where working with a group or collection requires mapping or iterating through them, a set of operations which felt increasingly like it deserved a specialized interface.
Our experience working with Inkbase indicates significant investment is needed in working with groups/collections for a tool like this to be truly useful.
The programming model is the least solved part of Inkbase. The ergonomics of programming in it are clearly terrible: heavy text editing on a tablet is bad, even worse if you’re editing Lisp, inspector panes are cluttered, and much of the program is hidden from normal canvas view. We believe that a wholly different way of thinking about and expressing the “program” is what’s called for, as opposed to trying to make a more ergonomic program editor for the iPad.
Good tools for thought are as much about helping you manage your headspace as anything else, and adding dynamic behavior just as needed allows you to stay in your current headspace rather than switching to a programming one. We’ve been convinced programming in the moment is the right direction for tools for thought since long before we started this project. However, Inkbase showed us both some specific results we’d like to emphasize in future tool design and some acute tensions that need further study:
Toolkits as a byproduct are great
We found it easy with Inkbase to build a reusable toolkit as a byproduct of working on the problem at hand. We might throw together a slider to scrub a value in a model or an operator to center an object between two input marks. We would then use that mini-tool again later on a different model by duplicating it to another page and swapping out its inputs/outputs with a simple insertion of a property using the
! shortcut buttons. This pattern is not only satisfying, but also productive in the amount of saved time and mental focus.
Shared behavior & reusability needs work
While it was easy to re-use a toolkit by rudimentary duplication, there isn’t a correct-feeling method for sophisticated re-use like sharing behavior between many objects. Exploring a formal system for encapsulation and re-use (i.e. prototypes, class/instance, mixins, particles vs fields, etc.) is a rich area for future work.
Creation of new objects reactively is unsolved
Creating new marks/objects reactively doesn’t easily fit the current programming model, and we struggled to find a coherent design for it. Imperative object creation is one possible direction, but doesn’t feel correct in this model, which wants to be declarative and reactive. Other areas worth exploring are creation of objects via spreads à la Apparatus, or other means by which objects can be created indirectly by manipulating a group or collection, such as a shape whose area is automatically filled with smaller shapes.
We opened with the question “what would be possible if hand-drawn sketches were programmable like spreadsheets?” We still don’t know the answer, but we found interacting with Inkbase to be extremely compelling—even with its limitations. Moving from sketching by hand to adding dynamic functionality, all within the same environment, is an exciting way of working we haven’t experienced elsewhere.
There are many areas enumerated above that need improvement or deeper research in order to move toward an actually usable tool like this, the largest of which revolve around the fundamental programming model. We hope to explore other approaches to the programming model in future projects.
We welcome your feedback: @inkandswitch or [email protected].
Thanks to all who gave feedback and guidance throughout the project: Marcel Goethals, Peter van Hardenberg, Geoffrey Litt, Ivan Reese, Toby Schachman, Alex Warth, and Adam Wiggins.