title: Ink Deformation - a review
dated: Q3 2025
related: Deformation

Ink deformation

Freehand digital ink is an incredibly direct and fluid input method: It allows you to quickly jot down your ideas without any need for formalization.

One of the main benefits of using digital ink over traditional pen & paper, is the ability change the properties of ink after you’ve drawn it, rather than needing to redraw from scratch. This ability also opens the door to end-user-programmability, as we can dynamically change properties based on computations.

There’s a number of different changes you might want to make such as changing the color or position of a drawing, which we understand pretty well.

But, once you’ve drawn a stroke you might also want to change its shape. That is, to deform it. This could be relatively minor manual tweaks, such as resizing a box to make more space inside, or changing where an arrow points. It could also be more complex dynamically driven behaviour, such as a parametric sketchy CAD model, or perhaps bending a characters’ arm.

Freehand v.s Formal

Of course, if we were using a vector graphics tool, deforming shapes would be easy. Vector drawing tools provide a highly editable format that allows you to tweak positions exactly. Points, lines & bezier curves make a good match for computational dynamic behaviour, and we’ve explored end-user-programming with these primitives extensively in Crosscut.

However, for the most part, vector graphics lack the directness and fluidity of hand-drawn ink, instead requiring a lot of detailed and finicky manipulation.

We can separate representations along two axis: the first is how neat a drawing is. A back-of-the-napkind sketch is less neat than an illustration in a textbook. The second, the extent to which it is computational or dynamic. A CAD model is more dynamic than a textbook. We’re looking for a solution in the top right quandrant. Informal, but dynamic.

Is there some way that we can maintain the directness and fluidity of hand-drawn ink, while adding the ability to manipulate shapes dynamically? Ideally, we’d do this by using formality on demand, allowing the user to add structure only when needed.

We believe that this is a crucial ingredient for programmable ink. But, it has so far proven tricky to get right. We want a model of freehand ink deformation that resolves this tension between freehand sketches and formal geometry. We want a system that allows the user to deform drawings both manually and programmatically, creating a smooth onramp between the two.

There’s a wide array of approaches to ink deformation, but, broadly speaking, we can break the problem down into three main steps:

Finding some kind of control structure; Something that the user can manipulate more readily than messy, hand-drawn ink strokes.
Finding a mapping between the messy geometry and the simplified control structure.
When deforming the control structure, use the mapping to deform the original geometry.

The big open questions in relation to programmable ink are:

How to create the control structure, either automatically/manually.
How to make the control structure match the underlying meaning of your sketch.
The meaning of a sketch is implicit and depends on the intent of the author. Solving this automatically in the general case is a capital-H Hard problem. Doing it manually is probably the only solution that works in all cases.
Finding a mapping between control structure and geometry. Either automatically in way that correctly maps to the conceptual model, or, manually in a way that doesn’t require a lot of complex actions, or, some hybrid approach that allows the user to tweak the mapping.
Supporting a wide range of deformations from simple (formal) geometric shapes (lines, arcs, curves, rectangles etc), up to complex, higher level deformations like bending a character’s arm.

Below is an overview of the different techniques ` considered so far. There is overlap between the different categories, so don’t take the categories too strictly.

Simplified geometry

The simplest, and perhaps most “obvious” way to deform ink is to turn ink-strokes into simplified vector geometry. The resulting splines can then be deformed like you would do in a vector-drawing application; by manipulating handles.

There’s significant overlap here with stroke simplification, and in fact, many vector drawing programs implement a ‘pencil’ tool that allows you to draw freely, and then manipulate the control points of the spline to deform the geometry.

Algorithms for curve fitting are readily available though they aren’t necessarily trivial to get right. Producing a simplified stroke that looks good, does not mean that the resulting control points are useful for deformation. Figuring out the right number of control points requires heuristics and parameter tuning.

Figma’s pencil tool allows you to draw bezier curves (right). Even though the shape was drawn with sharp corners, the tool creates rounded corners as a result of simplification. The resulting control points do not allow you to deform the shape asif it were a rectangle (left)

More importantly though, this approach really only works well for “clean” ink strokes, where the geometry is relatively simple and the number of resulting control points is low. In a sketchy environment, where the ink is messy and there is over-drawing, this approach can quickly become unwieldy. Because the mapping is per-stroke, it basically doesn’t work for higher level deformations like bending a character’s arm or twisting a shape.

In the Habitat project we explored a slightly more sophisticated approach which distinguishes between straight lines (with sharp corners) and curves. It also automatically joined separate strokes at the end-points. This more closely maps conceptual control points to drawn ink geometry.
One possible way to avoid stroke simplification entirely is to make a distinction between formal and informal ink Formal ink would effectively be what we consider to be vector graphics, while informal ink is purely freehand geometry.dy. Because the mapping is per-stroke, it basically doesn’t work for higher level deformations like bending a character’s arm or twisting a shape.

Nudging, Pushing and Pulling

A more direct way of deforming ink is to push and pull on it as if it were material, we’ve explored several approaches in this bucket. The main advantage of this approach is that it requires very little in the way of control structure (usually just the position of your finger).

String: We’ve tried several times to explore the idea of deforming ink by treating ink strokes as a piece of string. By applying forces to the string (pushing & pulling) you can then deform ink strokes.

Szymon explored the idea of using an IK-chain to deform ink strokes.
Ivan, similarly, explored a variant that feels more like a spider web.

Nudging: A nudge tool typically works by applying a force to the geometry with a distance falloff from the cursor or touch point, which creates a smooth deformation effect. Nudging is useful for fixing small mistakes, but not really for large scale deformation.

The Concepts app has a “Nudge” tool, which warps the ink by influencing the position of the ink. It’s also called “magnet tool”, or “falloff editing” in other software.
Early experiments by Szymon also explored nudging NURBS and b-splines

Noodlify: Finally, Marcel came up with the idea of noodlification which, rather than pulling on a single point, uses selection to pick up a whole chunk of ink. Any strokes that are sliced by the selection are then deformed by stretching them at the cutting point. This gives you much more control over which parts of the drawing stay stable, and which parts are deformed. Technically, this isn’t actually deforming strokes at all, as it’s just inserting new ink between to the stroke segments.

Scale, stretch & warp

A very simple way to deform ink is to simply scale or stretch it along a single axis. This is perhaps the most straightforward and common way to deform shapes, though it has obvious limitations.

Many tools give you a bounding box that you can use to scale and stretch shapes. Some programs like Procreate even allow you to rotate the bounding box relative to the selection, which means you can stretch your drawing along any arbitrary axis (pretty neat).
The Inkling prototype simply stretched ink along a single axis, which was surprisingly effective. Though this only really works in limited cases, and we considered it a temporary ‘cop-out’ rather than a final solution.

The most advanced version of this, often called Warping, or Mesh Warp, is a technique that allows you to deform shapes in a flexible way. It involves creating some kind of “cage” around the drawing, in most cases a rectangle.

Tools like Procreate & Photoshop have a tool called “Mesh Warp”, which allows you to warp using a 3x3 grid.
In 3D modeling tools this is also known as Lattice deformation.

Skeletal deformation

Skeletal deformation is the most common technique used in 2d and 3d graphics to deform shapes. It works by creating a skeleton using “bones”: simple geometry that is superimposed onto the actual geometry. Each point in the original geometry is then mapped to one-or-more bones, which are used to influence the position of the point. The bones can be moved and rotated, and the points in the geometry will deform accordingly.

Typically, the influence of each bone on a point is defined by a weight, which determines how much each bone affects a point. Each point is transformed by each bone’s matrix, scaled by the points’s weight for that bone. The final position is the sum of those weighted transformed positions. This is called linear-blend skinning. It is a very efficient technique that allows you to create complex deformations, and so it is common in games.

Of course, in-and-of-itself this does not give us a full deformation system, as we still need a way to define the bones and their weights. In most software this is done almost entirely manually, which is effectively a skilled job. A lot of the nuance comes from how the bones are structured (rigging), and how the weights are assigned, a process which is called weight painting.

The big challenge for us then, is finding an approach that allows us to surface these things without requiring complex UI or a lot of manual work.

Automatically generating bones (rigging), or automatically assigning weights (weight painting) are both active research areas, and there are plenty of published papers on the topic.

For automatic weights, a common set of techniques is called bone heat weighting, volumetric heat diffusion skinning, or harmonic coordinates.

Blender3d implements bone heat weighting as a feature.
Marcel implemented a variant of bone-heat weighting, which is what we’re currently using in Playbook. The heat diffusion step is relatively expensive, but is a massively paralelizable operation, which makes it suitable for GPU implementation.
A different technique is described in Bounded Biharmonic Weights for Real-Time Deformation. This gives great, even somewhat physically plausible results, though it seems significantly more complex to implement.

Automatic rigging seems the more difficult problem, and seems to rely much more on heuristics and machine learning. Any known techniques are also less obviously applicable to our context, and the more useful results are to be found in Sketch simplification. Some papers that might be relevant here are:

Free-form deformation

A combination of skeletal deformation and mesh-warp, you can generalize your cage-shape to arbitrary closed polygons, which is sometimes called free-form deformation. There are many possible ways to compute mapping between geometry and cage, including: distance fields, barycentric coordinates, mean value coordinates, wachspress coordinates, green coordinates, harmonic coordinates. It’s relatively straight forward to do this for convex hulls, though it’s significantly more complex for concave geometry.

Marcel explored some of this in the Etui study.

Physically accurate deformation

There are a number of different techniques that try to approximate physically accurate deformation of shapes, as if they were made of some sort of material. These techniques typically involve some kind of solver that computes the deformation based on the forces applied to the geometry.

The simplest way to implement this is as a mass-spring system, which is sometimes called Soft-body dynamics. The geometry is represented as a set of points connected by springs. When you apply a force, the solver computes how the points move based on the forces applied to them and the spring constants. This gives a squishy, bouncy effect which makes it popular in games and simulations.

Marcel implemented a simple mass-spring deformation system.

Unfortunately, mass-spring systems are also very unstable, making them unsuitable for our purposes. Though there are techniques which improve stability such as shape matching. There are other approaches that are more stable, the most well known is described in As-rigid as possible deformation

Tools such as Photoshop implement ARAP and typically call it Puppet Warp.

You can push this further to get more realistic physics using techniques such as Position Based Dyanamics. Some examples here, and here

The main drawback of all these techniques is that they require running a solver, making them very computationally expensive, which is not ideal in a real-time environment. Especially compared to the other forms of deformation which are increadibly cheap in comparison. Secondly, it’s not uncommon to want deformations that go beyond “physically accurate”, in which case a lot of these techniques are prone to numerical instability.