Crosscut

Drawing Dynamic Models

Crosscut is a stylus & tablet tool for drawing dynamic models of your thoughts. It aspires to be a thing to think through. In Crosscut you sketch, cobble, tinker, and play toward a deeper understanding. Begin by using the drawing tools to sketch something you’re thinking about, as you would with pen & paper. Then, add dynamic behaviors with a programming system that lives right inside your sketches. Each dynamic sketch is a reusable part that you can nest inside other sketches, enabling you to build elaborate models from simple pieces.

As you learn about Crosscut, you’ll be able to play with interactive demos we’ve included in the essay. Note however that Crosscut was designed specifically for a multitouch tablet & stylus. The embedded demos are simplified to work with a mouse and fit within the essay.To try them out for yourself, please make your browser window at least 960px wide, or open the essay on a larger screen.To try them out for yourself, please open the web version of this essay at inkandswitch.com/crosscut.

We welcome your feedback: @inkandswitch or [email protected].

Thinking, modeling, and computers

Thinking doesn’t just happen in our heads. When faced with a problem, we rarely sit perfectly still and just think. Instead, we interact with the world around us. We move around the room, talk to other people or to ourselves, write notes, draw diagrams, build prototypes, act out scenarios. We externalize the thinking process. Each externalized thought forms a model—a simplified representation that allows us to preserve, communicate, or reflect on the thought.

Computers have given us many new ways to work with our thoughts. Digital tools for professional drawing (CAD), music production (DAW) and video editing (NLE) allow us to create with amazing precision and to quickly explore multiple ideas. If we learn to program, we can even create richly dynamic simulations of our world and the worlds of our imagination.

But the most versatile tool for externalizing our thoughts isn’t any piece of software—it is the pen & paper. Why is this? Can we design a new tool to help us think, with the simplicity and versatility of pen & paper, but also with the dynamism of software?

Thinking with models

When Watson and Crick discovered the structure of DNA they didn’t just conjure up a structure in their minds, and afterwards build a physical model to demonstrate it. They built a physical model to explore how the different parts of the DNA molecule could actually fit together, and only by doing so came to discover and understand the double-helical structure of DNA.

Models aren’t just the end result of the thinking process. Making models can be an integral part of the process of understanding. The act of externalizing, communicating, and reflecting through models changes and advances one’s inner thoughts, surfacing new realizations.

James Watson (left) and Francis Crick with their DNA model. (source)

James Watson (left) and Francis Crick with their DNA model. (source)

Thoughts move quickly, so the tools we use to make our models must be fast to use and readily available.

Model of Sagrada Famillia by Antoni Gaudí—modeling an optimal arch by following a hanging chain. (source)

Model of Sagrada Famillia by Antoni Gaudí—modeling an optimal arch by following a hanging chain. (source)

When it’s frictionless to externalize your thoughts, you can create many small variations and reflect on the consequences of each change. Just as you might enter into a conversation with a friend to discuss an idea—to untangle its different aspects and consequences—so too, by externalizing your thoughts using a tool like pen & paper, you enter into a kind of conversation with the tool and the models you make with it.

By engaging in a conversation (with a person, a piece of paper, or a computer), thinking becomes a feedback loop. We don’t experience the world only by observing it. We poke and we prod at it. Thinking is a process of affecting and being affected; acting and understanding.

Modeling with computers

Computers offer clear advantages when it comes to working with external representations of our thinking:

We’ve seen a proliferation of software that leverages these capabilities—tools for word processing, 3d modeling, music production, and digital illustration, to name but a few. These software tools allow us to express the content of our thoughts, without having to necessarily understand how the computer actually does the things it does.

But note that these tools are focused on a specific, narrow range of media. They have interfaces and features catering to a preselected form of expression. They are molds that you may pour your thoughts into. They rarely allow you to create ad hoc models with unexpected structures—and they certainly don’t encourage it.

Moreover, these tools have a formal, businesslike focus on producing results. They are tools rich with features for polishing your work, preparing something that is ready to be shared, aligned with the technical and cultural expectations for how such a work ought to appear.

As creative people interested in externalizing all kinds of thought, we want tools for exploring the fringes of the possibility space, and for playing with a wide range of low-fidelity representations, not just tools for polished products in predetermined forms.

Of course, if it’s flexibility and creative freedom you want, you could always learn how to program and then write your own software. Programming languages allow you to create rich and dynamic models, or tools for working with models, in virtually any form you desire. But programming demands that you represent concepts as long swaths of abstract symbols in text files. You are no longer thinking in terms of the problem you’re trying to solve, or the thoughts you want to express and reflect on. Instead, you’re forced to think in terms of the computer—functions, and data. Imagine trying to discover the structure of the DNA molecule while at the same time worrying about control flow, syntax, and error handling. The concerns imposed by programming are incidental to the problem you’re trying to solve.

These are the current trade-offs. On the one hand, you have tools that allow you to work directly with the content of your thoughts. But those thoughts are constrained to preselected forms, and the tools have an unfortunate focus on producing polished results. On the other hand you have programming, giving you the freedom to use the full dynamic power of computation in any way you please. But this power comes at a great cost. You may no longer work directly with the content of your thoughts. Instead, you must work in terms of the computer.

Can we make different trade-offs? Can we unify the directness of pen & paper with the dynamism of software?

Design principles

The following set of principles guided our work from the outset.

How it should feel

We wanted something that doesn’t feel like programming.

In short, instead of molding your problem to the machine, the machine should be molded to fit your problem.

Importantly, while we didn’t want this to feel like programming, we did want to preserve the power of computation—interactivity, abstraction, encapsulation, iteration—and create an alternative, visual interface to it.

We wanted the building of models to resemble tinkering and bricolage. You should feel as if you’re thinking through the tool, and playfully exploring the problem space.

How it should function

Additionally, we followed a set of intuitions about the technical aspects of the project.

Inspirations

Beyond the metaphysical yearnings and design goals explored above, our work builds on generations of existing research.

The overlap of drawing and computing

These projects offer ways to visually express computation—building dynamic models, data visualizations, and interactive diagrams by directly manipulating graphical representations. They show alternative interfaces to the very same set of computational ideas behind conventional text-based programming.

Sketchpad, Ivan Sutherland, 1963
Drawing Dynamic Visualizations, Bret Victor, 2013
Apparatus, Toby Schachman, 2015

A bin of parts

These projects show how building software can be about recomposing visual elements. By carefully thinking about interfacing between components, the building of dynamic models (ThingLab) or situated software (Fabrik) is merely a matter of joining smaller parts into bigger wholes.

ThingLab, Alan Borning, 1978
Fabrik, Dan Ingalls et al., 1988

Programming as a medium for thinking

There’s a flavor of programming that’s about code as a thing to think through to solve problems at hand, instead of code being merely a means to make software that in turn fulfils some purpose.

While the aesthetics of these inspirations aren’t relevant to us, their different relationship between thinking and code is an important reference point.

VisiCalc, Dan Bricklin et al., 1979 (and spreadsheet software in general)
Jupyter, Fernando Pérez et al., 2014 (and notebook software in general)

Non-inspirations

Finally, it’s worth mentioning a few projects that seem relevant or similar to our work. These projects are interesting in their own right, but aren’t well aligned with one or more of our goals. Identifying them as non-inspirations early helped us hone and focus our work throughout the project.

First, most visual programming languages, which feature colorful blocks of code or nodes connected by wires—Scratch, Max/MSP, Origami, vvvv, etc. They give you a canvas on which to spatially organize your symbolic code, and at their best, show helpful visualizations of program state.

Second, most live editor tools, which synchronize a textual code editor on one side with a visualization of the code’s output on the other—Sketch-n-Sketch, Glisp, the inspector in web browsers, etc. They allow you to see how parts of the textual code relate to parts of the visualization, and at their best, allow changes on either side to affect the other.

In both of these types of tools, you’re still programming with code—manipulating abstract symbols, which indirectly produce some output that exists separately from the code. What we want are tools that let us work directly with a graphical representation of our problem of interest, not with some form of code necessary to express that problem to the machine.

Crosscut

We’re now ready to introduce Crosscut, a tool for drawing dynamic models.

To explore the basic concepts of the tool, we’ll build a Varignon parallelogram—a visual proof that connecting the adjacent midpoints of the sides of any quadrilateral will always form a parallelogram.

💡 Try dragging any of the black-and-white circles. Notice how the solid black lines drawn between midpoints always stay parallel.

Concrete Ink

At the surface level, Crosscut is a very simple vector drawing application: you can draw only points and straight lines. Crosscut is meant to be a dynamic alternative to pen & paper, so we call these drawings ink.

There are two kinds of ink in Crosscut. The first is colored black, and we call it concrete ink. You’ll use these black points and lines to concretely draw representations of your thoughts.

A concrete point on its own (left), and a concrete line connecting two concrete points.

A concrete point on its own (left), and a concrete line connecting two concrete points.

One of the most basic interactions in Crosscut is snapping. Drag one point close to another and they’ll snap together. You can drag the point away to abort the snap, or lift the stylus (shown as a red dot in the video below) to create a connection. This is useful for drawing and for adding dynamic behavior, which we’ll explore next.

Meta ink

Imagine trying to make one of the concrete points move dynamically. The status quo approach would be to give it a symbolic name like point1, and then manipulate that symbol with code.

function update() {
  point1.x = Math.sin(Date.now());
  requestAnimationFrame(update);
}
update();

As we explained in earlier sections, this approach corrupts the types of thinking you can do while working with dynamic models. It forces you to work symbolically instead of visually, even though the model you’re making is in a visual medium. Worse, you need to express the dynamic behavior in terms of how the machine works, not in terms that are more relevant to the model or the behavior itself.

In contrast, we designed Crosscut to let you add dynamic behavior directly to your concrete representations. This is accomplished with a second type of ink, rendered in yellow, which we call meta ink. In addition to being something you can draw with, meta ink creates a dynamic relationship between the things it touches. We’ll explore this relationship in depth in the next section.

From left to right: a meta point, meta line, and sin meta node.

From left to right: a meta point, meta line, and sin meta node.

Crosscut comes with a selection of prebuilt meta nodes that can be connected via meta lines to manipulate the concrete ink things you draw. Meta nodes make up the low-level programming interface within Crosscut. They have a passing similarity to other node-and-wire programming tools, with some crucial differences we’ll explore shortly.

The first meta node we will examine is the property picker. This meta node has a rounded left side containing a snappable meta point, which should be connected to concrete points. On the right, the node has a flat side with another snappable point, which should be connected to other meta nodes (most of which also have flat sides, a design clue that they’re meant to be connected to one another).

In the example below, the property picker nodes are connected to value meta nodes, which simply display a numeric value.

Using property picker and value nodes to measure the x and y coordinates of a concrete point.

Using property picker and value nodes to measure the x and y coordinates of a concrete point.

Using meta ink and a property picker meta node, you can extract information about concrete points by directly connecting to them. This might seem like a small detail, but is actually one of the core ideas of the project. Pulling values directly from concrete representations allows you to always see what you’re doing—something that is lost as soon as you start hiding values behind abstract identifiers.

Dynamism and equality

As mentioned above, meta ink creates a dynamic relationship between the things it touches. Specifically, meta ink establishes an equality. All points connected by meta lines will have the same value, much in the way that all directly connected points in an electrical circuit have the same voltage. We can illustrate this with a basic example.

💡 Try dragging either of the concrete points. Notice that their vertical (y) position can be changed freely, but their horizontal (x) positions are kept equal.

Here, the two property picker nodes select the x values of the two concrete points, and then the x values are made equal by the meta ink lines connected on the right.

You can build more complex relationships using additional meta nodes. For example, a concrete point can be animated by taking the sine of the current time.

Recall that we saw this same example earlier, only expressed a little differently. Here it is again, for stark contrast.

function update() {
  point1.x = Math.sin(Date.now());
  requestAnimationFrame(update);
}
update();

Meta nodes allow you to define relationships between concrete points. For example, the two concrete points below share an x position, and are kept one unit apart on the y-axis.

💡 Try dragging either of the concrete points.

Reusable parts in the library

Recall that as we explore the basic concepts of Crosscut, we’re building toward the construction of a Varignon parallelogram.

With what we’ve already seen, we can build a key piece of that geometric proof: a line with a midpoint.

💡 Try dragging any of the three concrete points. Notice that the meta node logic maintains the midpoint relationship no matter how you move the points.

We could rebuild this same construction three more times to get four separate lines with midpoints, but this is tedious. Instead, let’s do something very common when programming and turn it into a reusable component.

The status quo way to make this midpoint reusable would be to turn it into a function or a class, give it a name, expose some parameters, and call it by the name any time we want to use it. There are some issues with this. For one, we’re back to indirectly using named things rather than directly using tangible things. Second, when the function requires parameters, you have to to fill them in—and to fill them in, you have to know what kinds of values are meaningful. Types can be helpful here, but only to a point—you still need to know what the values mean to the function. All of this, again, forces you think more about the technicalities of the tool, instead of thinking with it.

Our approach is to flip this situation on its head. You don’t explicitly mark something as reusable—every drawing you create is a reusable part.

You can pinch to zoom-out from the drawing interface, which will take you to a library of parts. Here you’ll find all the parts you’ve created, plus all the parts and meta nodes that Crosscut includes by default, ready to be reused and remixed in the creation of new parts.

The library is a freeform canvas. You can place your parts wherever you like, grouping them together visually in whatever way makes the most sense to you.

There’s no button to create a new empty part. Instead, you create a new part by duplicating an existing one, and Crosscut includes a selection of helpful building-block parts to get you started. This duplicate-only approach encourages remixing and iterative making. This also makes it natural to create a “template” part for yourself with whichever things you use most frequently.

Encapsulation without lamination

You can take any existing part and drag it inside another to nest the first part inside the second.

When you nest one part inside another, its meta ink is hidden. Crosscut also draws a dashed yellow bounding box around the nested part to signify that it is, in fact, a part.

Here, we’ve nested the midpoint line from above inside an otherwise empty part.

💡 Try dragging any of the three concrete points. Notice that the midpoint logic still applies, even though the meta ink is hidden.

You can still see and manipulate the concrete ink of the nested part. Meta ink still imbues the concrete ink with dynamic behavior, even though it is hidden. This places a particular emphasis on concrete ink as the material for representing the matter of your thoughts. Meta ink, and meta nodes in particular, are just for the low-level implementation of dynamic building blocks—an alternative to code that should fade away as you build up an elaborate model from simple parts.

Importantly, you can still snap concrete and meta ink to the concrete points of a nested part. The part may be nested, and its meta ink may be hidden, but it still exists in the same conceptual space as the things around it.

💡 Try dragging any of the concrete points. Notice that there are both concrete and meta lines connecting things inside and outside the nested part.

Most tools that offer some notion of nesting will wrap an impregnable boundary around the things being nested. We refer to this as lamination. It is a superfluous formalism that breaks the flow of thinking about the problem at hand.

Here’s an example from Max/MSP, a popular visual programming tool:

The blue rectangle is a bpatcher—a mechanism for embedding one Max/MSP patch into another. You can still see the nodes inside, but you can't connect wires to them or move them around.

We designed Crosscut so that it can have encapsulation without lamination. Nested parts give you encapsulation—they allow you to create an easily reusable piece of concrete ink with its own internal behavior. The visual elements in Crosscut have important semantic qualities, and encapsulation doesn’t diminish that.

We are now ready to build the quadrilateral proof in an ergonomic manner—by dropping in four midpoint parts, snapping them together, and drawing concrete lines between adjacent midpoints.

We’ve now shown how Crosscut can be used to turn a handful of graphic primitives into reusable building blocks, and how to combine those into a richly interactive model. This kind of composition is common when programming with static text, but Crosscut lets you do it while working with live, tangible things.

Next, we’ll look at how the things you draw in Crosscut can be imbued with more dynamic behavior.

Sharing values with ports

There are often situations where you want to share meta ink values between a nested part and its containing part. While you can connect meta ink to a nested concrete point from both inside and outside the nested part, using concrete ink as an interface for sharing values doesn’t make much sense. We need some other way to connect the meta nodes inside a nested part to the meta nodes outside it.

For this, we introduced port meta nodes. A port is like the inverse of a property picker. Instead of selecting a single property (like x or y) from a more complex element, they let you create a new property that can later be selected.

Consider a more general version of the midpoint line from above, where now the middle point can be shifted between the two endpoints.

💡 Try dragging any of the concrete points. The middle point will always stay at a spot 70% of the way between the two endpoints.

In the example above, the node labelled lerp is a port. The port has no value by default, so it’s connected to a value node set to 0.7 for the sake of our example.

Recall that a nested part is given a dashed yellow bounding box. You can snap meta ink to this box. If you connect a property picker to the bounding box of a part, any ports inside that part can now be selected.

In this example, we’ve nested the part from the example above, and are animating the lerp property from outside.

As you’ve probably noticed, this approach runs counter to our goals. It introduces naming, which we sought to avoid. Worse, it’s a move away from tangible representations, in two ways. First, these port names aren’t visible from the outside of a nested part until you connect a property picker. Second, this approach connects visible meta nodes outside a nested part to invisible meta nodes inside it, so you can’t see the whole system at once. All of these issues could be addressed or avoided entirely by enriching the GUI.

Repetition with repeat and spread

Repeating things with controlled variation is one of the fundamental powers of the computer. The status quo here is some form of for i in ... loop, where you have to not only understand and map the abstract symbols within the loop body to their imagined result, but you also have to imagine how varying i will change the result.

With Crosscut we explored ways to make repetitions more tangible, without the need to simulate them in our heads to understand what’s going on. We offer two takes on this, albeit both with some issues.

The first is repeat. This meta node allows you to clone a part any number of times. On its own it would be pretty much useless, as it would spawn the same part over and over in the same place. To modify each clone distinctly, they are each given a numerical index value by the repeat node. Inside the part, the index can be accessed using a port, and the part can then manipulate itself so that its concrete elements change as the index changes.

That’s a complicated description, so let’s walk through an example. Below, there is a nested part of a line with a height property. We’ve set up some meta nodes that will give this line a height based on the sin of the x position of one of its concrete points. The example above also includes props named a, b, and index. In a moment, we’ll use these to manipulate this example from outside. For now they have no effect, and the math nodes to which they’re connected will just use sensible defaults.

💡 Try dragging one of the two concrete points left and right. Notice how the other point moves up and down like a sine wave.

To better see the sine wave formed by this part, we nest it inside another part, and connect it to the repeat node. We then connect the repeat node to a numeric value, specifying the number of clones to create.

This example includes three ad hoc sliders—concrete points connected to y property pickers that you can drag up and down to manipulate numeric values.

💡 Try dragging the ad hoc sliders for the a, b, and repeat parameters. (Note that the repeat slider resets the position of the other sliders—this is a bug, which we discuss near the end of the essay.)

In our experience, this approach using repeat and index feels too much like conventional programming. You have to first build the part that will be repeated—imagining how changing index will impact it—and then use it in some other part to see the result. This is the sort of playing computer in your head we are trying to avoid.

To improve on this, we made a second attempt at repetition with a spread meta node. With the spread node, you connect one end to the part to be cloned, and the other end to the calculations that will change with each clone. With the original repeat node, the logic that created variation with each clone lived inside the nested part, in a separate context from the logic that created the repetitions. With the spread node, the varying logic lives right in the same space as the logic creating the repetitions.

In the below example, we use the same line with a height prop that we started with for the repeat node example above. We also use the same sine wave logic, but this time these nodes are connected directly to the spread node.

💡 Try dragging the ad hoc sliders for the mul and sin nodes.

Repetition is a great enabler of dynamism. Sadly, neither approach we took was a resounding success within our prototype. The repeat node is disembodied and decontextualized. The spread node composes poorly, and performs spooky action at a distance by manipulating nested parts in a way where the exact nature of the manipulation is unclear. Finding richer mechanisms for repetition will be an important direction for future work.

Spatial queries

Both repeat and spread, in abstract sense, are a way to turn meta ink values into concrete ink representations. Spatial queries are the inverse: a way to turn concrete ink into meta ink values.

Every nested part in Crosscut is automatically given a special count property, which you can access by connecting a property picker to the dashed yellow bounding box. If the part includes any concrete lines that happen to form a polygon, the count property will equal the number of concrete points that are enclosed within this polygon.

💡 Try dragging the concrete points in and out of the rectangle, or moving the rectangle itself, and notice how the count updates.

The spatial queries as they exist in Crosscut right now are very limited. They only count concrete points, and only within a specified space. But, they compose nicely with the rest of the system. You can dynamically move concrete points in and out of the polygon, or move the points forming the polygon, or the part that contains it. Everything remains alive.

Consider this playful example of a simple step sequencer, with a sliding playhead that counts concrete points on the bottom left, banging a “drum” on the right.

💡 Try moving the individual concrete points at the bottom to change the rhythm. At the top, drag the concrete point in and out of the "play" rectangle to play/pause.

Notice that the playhead is just the line with a variable midpoint position that we saw earlier, connected to the line with a height property, connected to a square made out of two height lines and two similar width lines. By building up from simple parts we can quickly create rich, dynamic models. Notice too how little meta ink we had to add to turn these component parts into a step sequencer. This recombination of concrete parts is how Crosscut best enables you to focus on your thoughts, not on the workings of the computer.

Findings and open questions

Building with parts

We wanted working in Crosscut to not feel like programming. Building models by reusing and remixing existing parts is where we came closest to achieving that feeling. Typically, you only need a small amount of additional ink to make two parts work together—a few concrete lines to complete the drawing, some meta ink to unify behaviors, that’s all it takes. Ideally, we would want to snap together most parts without the need for any additional meta ink, but this is something that still needs to be explored.

The breadth of constructs you can build with just a few simple primitives is surprisingly wide. While implementing Crosscut, we continually tested it by building an ever-expanding variety of little models, tools, and toys. We built Bézier curves and wild geometric forms, time-based devices such as clocks and sequencers, a dice roller, a simple electrical diagram, a temperature converter, snappable logic gates, curve plotters, and more. See the appendix for demos of some of our constructions.

The program is the drawing

Crosscut is one of the few programming systems that puts live-running code and tangible data (the concrete ink you use to make a model) together in the same space. What’s more, Crosscut allows you to work with both of them using the same metaphors and direct-manipulation tools.

This is in contrast to the traditional separation between the code and the running program. There are several prominent tools that have attempted to reduce this separation. HyperCard placed its code editor on the “back side” of the card, which was accessed from inside the space where the running program lived. In Crosscut, there’s no “back side”. We achieved this unification by designing a simple graphics system for concrete ink first, and then designing the supporting meta ink system to work with it by directly drawing/wiring from the meta to the concrete.

There’s never any uncertainty about where dynamic behavior comes from. If you see a yellow line touching something, that thing will be manipulated dynamically, and you can follow the yellow line to see exactly what drives the dynamism. In principle, a screenshot gives you enough information to recreate an entire system and its exact state—the program is the drawing.

Drawing lines to concrete representations as a way of pointing to things has much smaller cognitive overhead than having to pick a name for each thing you’d like to refer to. What’s more, it feels fun to work in this way. Snapping is a very immediate and playful primitive, and allows you to construct both “data” and “code” in this environment using very familiar primitives.

No syntax or runtime errors

Crosscut is both the programming environment and an execution environment. There’s no code-compile-run loop. Everything you make is always there, alive and ready to respond. A valuable consequence of this design is that there are no syntax or runtime errors.

const point1 = Point(1, 2);
point2.x = 10;

Consider the above code. We create a variable named point1, and then try to manipulate a variable named point2, which doesn’t exist. It’s not possible to make this mistake in Crosscut. Since you always have to wire directly to an existing embodied representation, you can’t request something that doesn’t exist.

In Crosscut, every value that flows through meta ink is either a concrete point or a number. Every meta node can work with both types of values. This means it’s impossible to get type errors when constructing graphs. Additionally, all meta nodes have default values for each input/output snap point. These two features keep your construction live and responsive even while it’s only partially implemented. We believe that through careful design it’d be possible to maintain this property while supporting additional data types.

You can still easily get an unexpected result, but you’ll always get some result. In our experience, this approach promotes tinkering and feels much closer to sketching than to programming.

It’s worth noting that runtime errors can be avoided with systems like a static type checker, pushing the problem into compile time. The result is that your program is often in an in-between state where it won’t even start until all errors have been resolved. This “if it compiles, it works” approach is the opposite of what we were trying to achieve with Crosscut.

Bidirectional evaluation

It feels delightful to wire a relation between two objects and have the relation work exactly as you expect. You can experience the relationship between parts very immediately by wiggling any of the points. This causes the other interconnected points to wiggle in concert. This works even with complex relations between them, allowing you to get an intuitive feel for the systems behavior

This is achieved using a technique called bidirectional evaluation. When something concrete changes, that change is propagated through the network meta ink and meta nodes, updating everything connected to the source of the change. When a change flowing through a meta node enters from the left or top snap point, a forward implementation of the meta node’s functionality is used. When a change flows in from the right snap point of a meta node, a separate backward implementation is used.

Implementing bidirectional evaluation isn’t as straightforward as it may seem. In part this is an inherent consequence of the unidirectionality of many math functions. In part this is due to our primary focus being on the features of Crosscut that allow composition via parts, with the exhaustive particulars of meta ink being a less relevant thing to focus on. Our implementation was good enough to validate our use case and satisfy our goals, but had some frustrating shortcomings, which we’ll outline here.

Firstly, a limitation inherent to math: not all meta nodes have a meaningful backwards interpretation. For example, the range of sin is only defined for values [-1, 1]. If you connect a concrete point to the right snap point of a sin meta node, and drag this point outside the range, there are a number of different things that could happen. In our prototype, when backward evaluation would have an unclear meaning, the meta node will simply stop the flow of change through it in that direction. A more thought-out approach would be preferred here.

Secondly, a limitation we could have solved with design: even though meta nodes work forwards and backwards, there’s a preferred directionality to them—input on the left, output on the right, constant on the top. It can be very hard to interpret the directionality of a constructed graph. What does it mean for the computation to go one way or the other? This is something that can be learned by experience, but it’s not well communicated by the system. In our prototype, any meta node that has a different effect depending on the direction of flow, no matter how subtle, has snap points that are drawn as little triangles pointing in the forward direction. A richer implementation might use a stronger visual cue, like animations that show the occurrence and direction of data flow.

Thirdly, a limitation we could have mitigated with engineering. It’s possible to construct graphs that are perfectly valid but evaluate incorrectly. For example, cyclical graphs are especially likely to produce wrong or counterintuitive results.

Updating the value on the right will cause the add operation to run backwards. This will in turn trigger the add operation to run forwards again, causing a loop. The result is unexpected and mathematically incorrect.

Updating the value on the right will cause the add operation to run backwards. This will in turn trigger the add operation to run forwards again, causing a loop. The result is unexpected and mathematically incorrect.

Simple spatial queries

In our Inkbase project, spatial queries showed a lot of promise, which is why we wanted to continue exploring this idea with Crosscut. The highlight of our new implementation is just how low-ceremony it is: queries happen automatically for any nested part that contains a closed polygon; the result is simply a count of points. This brutal simplicity allowed us to implement a working version quickly. But because spatial queries only give a simple count, the uses are rather limited. In the end, we mainly used queries in Crosscut to build switches and toggles.

There were a few ideas that we considered but didn’t have time to explore:

Repetition of things in space

Repetition is one of the most significant capabilities that computers can offer us. Finding a way to do repetition within Crosscut was an important goal. The repeat and spread meta nodes are two attempts at repetition of things in space, but both leave a lot to be desired.

The repeat node feels very much like programming—exposing the index property inside a part is obviously bad. It usually means that every time you want to do some repeating, you need to design a bespoke part to be repeated. It doesn’t encourage reuse of existing parts in a generic, ad hoc way.

The spread node feels like a step in the right direction, by flowing repeated logic through meta ink on the same level as the rest of the construction. But this can still be confusing to work with, and doesn’t provide enough visual feedback.

With both repeat and spread, we felt the need to show a connection between the node and the repeated parts. We used a dashed yellow line, but this wasn’t something we spent any time designing. Even with this dashed line, there is still a sense of spooky action at a distance, something we wanted to avoid in the project.

Future research should look for more ways to do repetition—both repetition of things in space, and other kinds of repetition too—while adhering to the design principles of Crosscut.

Conclusion

One of our earliest goals for the Crosscut project, before we even arrived at the idea that it should be a tool for drawing dynamic models, was to look for potential key ingredients that ought to exist in a tool that allowed programming through drawing. We expected that spatial queries and repetition would be such ingredients, and they probably are, though we weren’t able to explore them as much as we’d hoped. What we did find, through building the prototype, letting various people experiment with it, and reflecting on it in this essay, is that building models by combining simple, programmable parts feels wonderful.

The combination of meta ink directly connecting to concrete ink with encapsulation without lamination and everything is a part really sings. It shouldn’t come as a surprise—composition of simple parts is a major theme of conventional programming and countless other creative practices. But what excites us here is how this composition was achieved while adhering to our specific design principles, despite the fact that the principles push against common programming practice. In Crosscut, you are always in touch with your concrete things while dynamically combining them. The computer’s processing helps you with the calculations it’s good at, while the touch and stylus interface gives you a sense of immediacy that we’d never really felt in a programming tool before.

We will continue exploring these ideas, and we encourage you to explore them too. If you find things that feel like key ingredients, please share them with us. There seems to be a lot of enthusiasm for reimagining programmable tools with an emphasis on concrete representations, and a newfound energy in the pursuit of tools for thought. We’re excited to see what comes next.

We welcome your feedback: @inkandswitch or [email protected].

Thank you to everyone who offered feedback and guidance throughout the project: Blaine Cook, Peter van Hardenberg, Josh Horowitz, Scott Jenson, Martin Kleppmann, James Lindenbaum, Geoffrey Litt, Kevin Lynagh, Rae McKelvey, Omar Rizwan, Chris Sun (daiyi), Eileen Wagner, Adam Wiggins


Appendix: Examples

Designing a bookshelf

I moved apartments, and now I have this 5.5' by 10' wall for which I’d love a custom bookshelf. I’ll use Crosscut to quickly sketch the wall dimensions using width and height lines, and then drop in a rectangle and a spread to see which configuration I like the most.

💡 Try moving the three ad hoc sliders at the top right underneath the W (width), H (height) and N (number of shelves) to change the properties of the bookshelf design.

Crosscut was a tightly-focused research prototype, but it’s easy to imagine ways it could be broadened and enriched. For example, you could drop a photo of your wall next to this construction, and then play with parameters in a more grounded context. This shows how Crosscut could grow into a powerful thing to think through: import some reference materials, roughly sketch new ideas, drop in ready-made parts to make the sketches dynamic, and then explore possibilities.

Joinery for laser cutting

I’m working on a custom box design to be made out of multiple plexiglass parts I have left over in my workshop. I’m trying to figure out good proportions for the joinery teeth, so I make myself a little playground to explore what “looks” good. My approach was always more art than science anyway…

💡 Try moving the two ad hoc sliders on the right to change the height and width of the joinery teeth.

In addition to importing existing media, another easily imagined improvement to Crosscut would be support for exporting your work, for instance as an SVG. That would allow Crosscut to be used as a tool for building visual prototypes that could be shared with other people or other tools. It’s important to remember that we want to push back against tools for thought being too production-oriented, so any export feature would need to be designed with care to maintain the essential sketchy feel of Crosscut.

Rolling dice

I want to play a board game with my friends, which requires d4, d5, and d6 dice. But we can’t find them anywhere! Not a problem—I can quickly construct my own configurable dice roller with a couple of parts and some ingenuity.

💡 Look for the lone concrete point at the lower left. Drag that point in and out of "ROLL" box to roll the die. Change the y ad hoc slider at the top to adjust the range of the rolled values, which represents the number of faces on the die.

Crosscut’s approach to immediacy and concrete representations feels like a natural fit for situated software. If you want to write some software to solve a simple problem for a few friends in the room with you, you shouldn’t have to go down to the basement computer, create a repo, write a program, compile it, debug it, deploy it, then come back upstairs to find everyone has gone home. You should be able to plunk an iPad down in the middle of the table, collaboratively sketch out what you want, add a tiny bit of dynamic behavior, and within a minute or two get back to board game night. This scenario—sitting around a table, playfully collaborating on a dynamic sketch with friends or family—feels profound to us, and is not addressed by any currently popular approach to programming software.

Understanding the Bézier curve

I’m teaching an introductory course to computer graphics, and we’re currently studying the Bézier curve. I find that it helps the students gain an understanding of the material by modeling it together as a group. We start by building an innocuous construction from a bunch of lerp lines.

💡 Try moving the ad hoc slider on the right to change the interpolation value. Observe how the center point traces the path of a Bézier curve. You can also move around any of the other points to affect the shape of the curve.

Then, we spread over them using the middle point of the highest-order (innermost) lerp, which leads to an “aha!” moment for the students.

💡 Move any of the outer lerp-line points to change the dotted path traced by the inner spread points.

Crosscut was inspired by understanding through building, a process of gaining an intimate feeling for an idea by building examples and objects to think & play with. It’s only natural that Crosscut works well for building examples from graphics and geometry, so they make for easy demos. But we also want to be able to build other things, too. One major omission from our prototype which, again, it would be easy to imagine adding is text. Crosscut’s directness and dynamism would be lovely for working with poetry, interactive fiction, semantic mapping, and all sorts of other explorations in the space of language. We considered multiple approaches to text, including building a system of lettering and recognition on top of our vectors and queries, but didn’t have the opportunity to test these ideas with a proper design and implementation effort.

Converting temperatures

I wanted to understand how temperatures in Celsius and Fahrenheit relate, so I made a bidirectional temperature converter. Crosscut is inherently bidirectional, so I only needed to throw together two sliders with a mul and an add meta node. By sliding the numbers up and down on either side I can see what the temperature is in the other unit.

💡 Drag the points on either side (labelled C or F at the bottom) to change the temperature.

This example is an interesting one to end on. It’s similar to the die roller, but different from all the other examples, in that it’s less like an externalized thought—some idea you have that you want to reflect on—and more like a tool that enables you to do something else. Sure, the act of building a temperature convertor allows you to gain an intuition for how the two scales relate. But after that learning through building is done, you’re left with an artifact that can serve a useful function. When we started working on Crosscut, we weren’t sure if it should be focused on tool-building or model-building. As the project progressed, we gravitated toward the latter. This means you can imagine something like Crosscut, some stylus & tablet tool for drawing dynamic things, but designed with an emphasis on building tools that enable you to do something else rather than models that are a thing to think through. The difference is subtle, but it is its own path that leads to a separate set of discoveries just waiting to be made.