The stylus should:
- Feel good in your hand.
- Have low inking latency (<25ms).
- Nice-to-have: a stylus barrel button or eraser for stylus tool selection.
- Prefer to avoid: stylus that needs pairing or frequent recharge.
The A4 paper notebook is a tool beloved by creative people. It’s a way to externalize thoughts. It’s comfortable to use at a desk, in a coffee shop, or at the library. It's calm and reflective, with blank pages quietly inviting you to sketch, write, and develop your ideas.
Computers, by contrast, are often inhumane in their form factors.
In pursuit of a more ergonomic interface, the computing industry has spent decades perfecting the touchscreen tablet. Devices like the iPad or Microsoft Surface, when paired with a stylus, offer a notebook-and-pen posture that seems promising for digital work.
But tablets have yet to find their footing as devices for creative professionals to do their most important work. At the Ink & Switch research lab, we explored the question “Why not?” through experiments such as a digital sketchbook and a science manuscript annotator.
But app prototyping on these platforms has proven to be surprisingly difficult because it’s hard to get the control we need to try truly new ideas. Tight coupling of the hardware, operating system (OS), and development tools means we cannot pick those things in isolation but must commit to an entire platform ecosystem. Those ecosystems typically come with rigid user interface (UI) paradigms including system-wide gestures that you can’t override, technical limitations in areas like background processes or handling wake-from-sleep, and build distribution that discourages rapid prototyping.
This article documents what we’ve learned on this journey. We hope it might be helpful for other human-computer interface researchers or industrial research labs with similar aims.
One weakness of today’s tablet platforms is that their UIs are hand-me-downs from desktop and smartphones.
Desktop computers are the gold standard for pro computing. Yet many of classic abstractions such as hierarchical filesystems, overlapping windows, cut-and-paste, cursors/focus, and shortcut keys are a poor fit for a tablet. These metaphors are aging and need a refresh to fit with the modern, cloud- and touchscreen-centric digital world of today.
The design language of smartphones, on the other hand, is all about compactness and on-the-go usage. They are optimized for one-handed use. But work-oriented apps on a tablet should use all your facilities (including both hands) for a high-bandwidth connection between your brain and the device.
Similarly, notifications, badges, and other “engagement-driving” interface elements from the mobile design paradigm are the opposite of what you want in a creative tool which protects your state of flow.
A productive tablet-based application will require some fresh interface ideas.
We believe that creativity and productivity are enhanced by tools that are a joy to use. Part of that is how they feel, physically and emotionally — hence our first set of criteria for the platform.
The tablet should:
The stylus should:
Our team is a research lab, and we need to build and test many ideas in rapid succession. Hence the question of how we will make and share software on a platform provides another set of important criteria in selecting the platform.
Our engineering team is willing to use any language, IDE, and build toolchain as long as the development experience is smooth and enjoyable. A mature language runtime, good editing tools, and an active community are important.“Community” in development tools is hard to measure, but Stack Overflow answers, conferences, books, and engineers available for hire are indicators.
Also key is an easy way to distribute builds to test users. The product team is happy when they can offer continuous delivery of software to our users, and the engineering team is happy when they can get quick feedback on code shipped to production.
Given that we know tablets are not yet great tools for productivity, the whole point of our research experiments is to try some alternate-universe UI ideas change that. As such, we need the ability to make changes to the way the entire device works.
At a minimum, we need to take over the entire screen and gesture space, with no OS chrome, control panels, or system-wide gestures getting in the way. We want to prevent screen rotation and we don’t want OS notifications to pop up over our app.Many OSes offer a kiosk mode or single-app mode which can help with this.
Beyond that we want capabilities like the ability to control exactly what appears on the lock screen and what the wake-up flow is like. We often need the capability to run background processes both while the app is awake and while the device is sleeping for background syncing or other housekeeping. And we want as close to direct access to the stylus and finger inputs as possible, including any hardware buttons like stylus barrel buttons or physical buttons (volume, power, etc) on the edge of the tablet.
In short, we needed to be able to make changes to how the device works, while staying as much as possible in application space rather than trying to write a new operating system from the ground up.
Here’s our comparison of the four platforms we tested in detail in 2017 and 2018.
|Surface||iPad Pro||Tab S3||Pixelbook|
|Dev stack||C#, Visual Studio||Swift, Xcode||Kotlin, Android Studio||Typescript, React|
|Builds via||UWP||TestFlight||APK||Chrome Apps|
|OS control and hackability||ok||poor||ok||great|
In brief: the Surface is reasonably good on most counts; the iPad is a study in contrast between truly spectacular benefits and truly awful flaws; Android is mediocre across the board; and Chrome OS shows great promise but isn’t quite there today.
Read on for the details.
For a company who only entered the productivity hardware game six years ago, Microsoft’s Surface products are surprisingly compelling.
The tablet is a nice size and shape. The stylus has reasonable latency, replaceable tips, no pairing and minimal recharging needs, and a great bonus feature of an eraser on the back. And the Surface Go offers a low-cost option.
Our engineering team was up and running with C# and Visual Studio very quickly despite not having much prior experience with these tools. The Windows platform has always made developer experience a priority and it shows.
The Windows Ink API is easy to work with. Our team had a basic single-user sketching app working in a matter of hours. Later we had to break out of the high-level APIs and make our own wet/dry stroke renderer, but this was straightforward to do.
Build distribution on Windows has historically been via sending
.EXE files to users. We opted to use the newer Universal Windows Platform (UWP) which is Microsoft’s attempt to modernize their build distribution, a sandboxed and app store-friendly approach. We distributed builds via an app package in a shared folder. In the future we would try distribution via Windows Store package flights.
By far the biggest downside of the Surface is the baggage of the Windows platform. This includes unskippable, multi-hour download/install/reboot cycles that happen with surprising frequency when waking up the device.
Another example is the frequent and seemingly impossible-to-disable system notifications informing you that Windows Defender has scanned your system for viruses; advertisements for Microsoft’s web browser; and news headlines injected directly into the user’s workflow when trying to run an application.
Instant-on is poor. If the device is connected to power and no system updates are pending, it wakes up reasonably quickly (<2s including Windows Hello facial recognition). But when disconnected from power it often takes ten or twenty seconds for the system to resume from suspended state. This is the legacy of desktop-era operating systems and system architectures.We used the Intel line of Surface tablets. In theory, the newer Snapdragon-based tablets incorporate true mobile architecture with instant-on.
Surface is a promising platform that deserves attention. But the Windows baggage is the cement shoes that drag the whole thing down. We hope the team at Microsoft might try something bold, like offering an official version of the Surface running Ubuntu or other user-friendly Linux distribution.
Android is a modern mobile operating system with everything you expect from such: app sandboxing, perfect instant-on, and goodies like a SIM card slot for a always-on cellular network connection.The Tab S3 uses Wacom EMR technology. The stylus contains a circuit that resonates on a particular RF frequency sensed by a thin PCB glued onto the back of the tablet’s screen. This is passive and thus requires no power to the stylus.
Big thumbs up for Android build distribution. Build an APK and email it or post to a Slack channel, where users can tap on it to install immediately on their device. Or use the Play Store to do alpha and beta channel releases, with only a minimal review process and lots of controls for easily adding users to your channels.
Even better: you can hypothetically can get total control over a device using Android OEM unlock or a custom build of the operating system.
In theory, Android was the perfect platform for us, and the Tab S3 was a solid tablet/stylus hardware choice.
The lackluster elements of this platform are pervasive.
Some on our team thought the Tab S3 didn’t feel so good in the hand. Android’s UI is a design-by-committee mess, with a the crazy-making homescreen, three different note-taking apps installed by default, and dozens of notification icons in the task bar. We tested the stylus latency as poor.
On the development side, we found Android Studio and Kotlin to be serviceable but generally unappealing. We worked with a handful of experienced Android developers and couldn’t get any of them to say they thought the developer experience on the platform was joyful, or even good. The best most would say is that it’s “greatly improved” since the introduction of Kotlin and subsequent Android Studio improvements.Android’s developer community has produced many helpful add-on tools such as the Genymotion device simulator or Stetho for debugging.
It appears that Android tablets have failed to find much footing in the market and many speculate that Android on tablets is dead.
Android seems to offer the perfect combination of a true mobile platform, a big hardware ecosystem, mature development tools, great build distribution options, and a hackable OS. We really wanted to love it for our research experiments.
Unfortunately, mediocrity pervades every part of the hardware, OS, and dev experience. Combine with the platform seemingly on its way to sunsetting and we ruled Android tablets out.
The tablet feels great in your hand. It has an amazing screen with 120hz (double the standard 60hz rate of most desktop and mobile computers) refresh and a 240hz sampling rate on the stylus.120 frames-per-second is rare outside of gaming and the iPad. We’d be curious to see how a creative app might be able to take advantage of the high refresh rate.
iOS is the trend-setting and defining operating system of our era. Instant-on is near-perfect. The UI is smooth and responsive at all times.
On the development side there’s Xcode, the extensive and well-documented iOS APIs, the device simulator, and the Swift programming language — all mature and highly useable. Apple has put as much effort into its developer experience as it has its user-facing experience over the past decade since the launch of the App Store and it shows.
TestFlight is a reasonable method for build distribution once your developer accounts and test users are set up. The process of pushing a build (especially with an automated tool like Fastlane) and having users get the new version twenty minutes later offered a solid continuous delivery story.
iOS is the ultimate walled garden, with Apple exerting a total stranglehold over every user’s ability to create and distribute software.
Build distribution on iOS is a tangled clustermess of Apple developer accounts, key signing, and laggy, incomprehensible iTunes Connect workflows. I have personally worked on a half-dozen major iOS projects, all of which had one or more experienced iOS devs on the team. In every case it has been days or sometimes weeks before we have a reliable build process set up. I’ve now spent several hours of my life waiting on hold or talking to confused salespeople trying to get a DUNS number set up just so that my team can have a shared Apple developer account… a process I will have to repeat for the next app/company.
iTunes Connect is a terrible admin interface, with long page-load times and poor information architecture. Setting up test users requires multiple steps of inviting them to your team, waiting for them to reply to invitation emails, then inviting them to the app which in turn requires them to install two different apps (TestFlight and your app) on their device.
Apps added to the beta function in TestFlight expire after 60 days. Our team has written many small research experiments that I’d love to continue using on my devices, but the cost of maintenance is too high. Erosion-resistant it is not.
A fundamental function of computers is the ability for users — that’s you and me — to write our own software. And if we wish, to share that software with one or more other people, even if it’s just a few friends and colleagues. iOS does not have this function in any meaningful way, and as a result it is hard to imagine the iPad ever being more than a DVD player for apps.
The contrast here is extreme. The iPad, iOS, and Xcode/Swift are a pinnacle of humanity’s achievements in computing hardware, operating systems, and developer tools. At the same time, Apple’s chokehold over software creation is a depressing nadir for computing empowerment and freedom.
At an experimental research lab like Ink & Switch where we need freedom to create many small apps and test them with small sets of users, iOS is a troubled path.
Google’s Chrome OS is somewhat of a dark horse entrant to this showdown. It’s massively dominant in the education market, with over 50% market share of US elementary and high schools. But it’s largely out-of-view for regular consumers.
Chrome OS is based on Gentoo Linux. This is fantastic — bringing together a powerful, hackable open-source OS with a great community onto a modern, mobile platform with capabilities like instant-on and app sandboxing. Add low-cost hardware to the mix and it gets quite interesting.
Web development is native development on Chrome OS. This means we can build the app much as we would for a web app or on Electron. But then that app can be installed and run as a local piece of software!The Pixelbook uses an active stylus which injects RF signal into a capacitive sensor. Active styluses require a battery and accuracy not yet as good as Wacom. The Pixelbook stylus uses a AAAA battery, which determines the stylus thickness.
Build distribution is a dream with the web. Web apps are easy to deploy and the user needs only refresh their browser. We opted to package our app as a Chrome App for a true native experience. It’s easy to “Load as unpacked” or package as a
.CRX which can be sent via any email or Slack. It’s also straightforward to publish in the Chrome web store.
The Linux subsystem for Chrome OS works well. It’s a joy to open a Linux terminal on this device and using apt-get to install packages.The Linux subsystem is a lightweight VM running inside the Chrome OS host. This means you need to do special things to access the Linux subsystem from the main OS. We found this to be a reasonable way to bring desktop-style capabilities to a sandboxed mobile architecture, but the end-user experience is different from (say) a Thinkpad running Ubuntu.
The Chrome OS base is Chromium OS, and we were delighted to find excellent instructions for building from source and installing onto your own device via a USB drive. We got the feeling that Google learned from the successes and failures of Android OEM unlock and developed a more straightforward way to hack on the OS. It can be tested under KVM or even do live deploys via Cros Deploy. A+ for OS hackability here.
Unfortunately the great promise of a powerful Linux combined with mobile platform capabilities and a web technologies app development story did not bear out for us.
Web standards and APIs don’t provide the depth of access and control we needed, such as the stylus API mentioned below. But there is no “native” to fall back to. And web technologies encode a tremendous amount of legacy concerns.
Unlike the true tablet form factor of the iPad and Surface (even the Surface Book has a detachable keyboard), the Pixelbook’s tablet mode was a bolt-on. The keyboard on the back is confusing when holding the tablet clipboard-style. Our team experienced a lot of variability in stylus sensitivity, with some members needing to push so hard the screen bends.
This is a laptop with a touchscreen, and that confounded our abilities to built a great tablet-and-stylus-first experience. In theory the upcoming Slate might solve some of this problem.Trying to use a two-in-one device on a daily basis makes it clear that this is an ergonomics trainwreck. that Apple is on to something in saying a touchscreen laptop makes no sense.
Stylus latency is acceptable. A bigger problem is that the pointer events API claims to be agnostic to the hardware, but in practice has many pre-cooked assumptions about the types of gestures the app developer might want to support. For example, we wanted to do hybrid finger/stylus gestures to take full advantage of the tablet+stylus form factor. But we’d receive unexpected and undocumented
pointercancel events, most likely related to palm rejection.
We partially worked around this with a hack, but we were were forced to scale back the richness of the interactions we could support. The lack of developer control over something as fundamental as touch and pen input is indicative of how immature and unprepared Chrome OS and the web are for tablet+stylus applications compared to other platforms.
Chrome Apps, though on the surface similar to Electron, were consistently frustrating with their limitations. We struggled to get a separate backend thread for our app, trying approaches such as Service Workers, the app’s “Background Page”, and eventually hacking things with a WebView.
OS control is potentially unlimited since you can fork Chromium OS, but we were disappointed by lack of customization possible without a fork. For example, we could not access the barrel button on the stylus, nor is it possible to remap the Google Assistant key on the keyboard. Not allowing users to override or customize behavior in support of your product’s “branded experience” is a depressing indicator that the platform is not intended for serious use.
On the development side, we never found a comfortable workflow for the edit/build/run cycle. With an iPad or Android tablet you can use the simulator or connect via a cable to your development machine. No such thing exists on Chrome OS, and our team variously tried editing directly on the machine, syncing builds from a development machine via rsync/ssh, and running the Chrome App directly on the dev machine. All had significant problems.
Chrome OS and the family of devices shows great promise. Unfortunately the experience for building apps with sophisticated gesture interfaces is not there yet. We were left worried that Google is not very focused on making it a great platform for third-party developers,.
We won’t be using Chrome OS for future research projects, but we’ll be cheering the efforts of the team on bringing Gentoo Linux to mobile hardware.
A few other options we evaluated in less depth:
In 2018 there are many excellent hardware options for tablet and stylus. But the hardware alone is not enough: the platform, including operating system and development toolchain, is a key part of the question.
What will we use for future experiments? The Surface and the iPad are the only two viable options. The choice between them comes down to whether we want to suffer through the pain of daily interactions with Windows and its legacy baggage, or daily struggles with the Apple walled garden of software development. Future-facing, we will keep an eye on Chrome OS, Flutter, and Fuchsia as high-potential entrants into the tablet game.
And as to whether the tablet+stylus can ever be a good tool for work, this remains an unsolved problem for our industry.