Up and Down the Ladder of Abstraction

We'll start with an in-depth example — designing the control system for a simple car simulation. Our goal is to write a set of rules that allows the car to navigate roads, such as the one to the right.

What inputs does our system have? Let's say that our car has a very simple sensor which can only detect three states: the car is on the road, left of the road, or right of the road.

How do we design the rules for the car to follow? We don't understand this system well enough to predict its behavior, so we need to prototype. We start with the simplest possible design, and thoroughly explore it. As we develop a feel for how it works, we'll get ideas for improvements. We try an idea, explore again, and keep iterating. Our design and our intuition should grow side-by-side.

The simplest possible rule here might be to turn right when off-road to the left, and turn left when off-road to the right.

To the right is an algorithm which encodes this strategy.

Click Tap to the right to find out how this algorithm behaves.

I suspect we all know someone whose driving is accurately modeled by this algorithm.

Above, we watched the system evolve in real time. A realtime view is useful for getting a direct, visceral sense of a system's behavior, especially for systems that are meant to be experienced by humans, such as visual effects.

However, real time is also quite limiting. Imagine a film editor who has to watch the entire film from the beginning with every edit, and cannot even pause or rewind. That would be absurd. Yet, most of our interactive systems are designed in this fashion.

To understand a system, we must be able to explore it. To explore, we must be able to move freely, under our own control.

Mouse over Drag the slider at the right to control time explicitly. Notice that we can easily simulate realtime playback simply by moving the mouse over dragging the slider at a steady rate. But we also have the ability to quickly skim over it, or stop at interesting events and examine them carefully, or quickly jump between interesting events and compare them.

A designer needs direct, interactive control over the independent variables of the system. We must not be slaves to real time.

In this algorithm, the car turns at a rate of 2° per step. Why 2°? Because we needed to choose some value, and with no understanding of the system's behavior yet, we made a wild guess.

Wild guesses are okay! We need to start somewhere. But we also need a way to easily and fluidly adjust our guesses, see the consequences, and develop a sense for how they affect the behavior.

The turning rate should be a parameter of our algorithm. And like time, we must be able to interactively control our parameters.

We've put our variables under interactive control, but it's difficult to get a clear sense of how the parameter affects the car's motion. We need a broader view.

Imagine looking for parking in a busy city, winding up and down the streets, hoping you'll chance upon an empty spot. Now imagine looking from a helicopter above the city. You'd instantly see every spot available! On the ground, you are limited to seeing a single street at any moment. From the air, you can see every street at once.

Likewise, our representation above, at any given moment, only depicts the system at one particular time. But we can go higher, and concoct a representation that statically depicts the system across all time.

Here, instead of showing the car, we show the car's entire trajectory. Mouse over Drag the slider, and notice how much easier it is to see patterns. For instance, the system exhibits severe oscillation for turning rates between 1° and 2°, and that oscillation appears to "tighten" around the road as the parameter approaches 2°. This pattern is difficult to notice from the ground.

There is no longer a time control — the function of the slider (to vary time) has been absorbed into the static visual representation. This picture represents the system for all time, not one particular time. We say that we have abstracted over time.

The representation in the previous section was fully concrete — the picture was drawn with every variable having a single, specific value. The representation here is abstract — it depicts the system for any value of the time variable.

By abstracting over a variable, we've taken our first step up the ladder of abstraction. Enjoy the view.

Standing on the first rung of the ladder, we get a much broader view of the behavior. But this benefit comes at a cost. We can no longer see the car! We can no longer viscerally experience the car's motion, and apply our common sense toward analyzing the system.

In previous sections, we first controlled time. This allowed us to interactively explore times of interest. We then abstracted over time, with a representation that depicted the system for all time instead of any particular time. This gave us a broader view with which to understand the system's behavior.

We now need to do the same for the algorithm. We've already controlled the algorithm, in that we can interactively adjust the turning rate and see the results. The next step is to abstract the algorithm, with a representation that depicts the system for all possible algorithms instead of any particular one.

Here's a different representation at this rung of the ladder. This plots the time to completion (number of steps the car takes to reach the end of the road) versus the turning rate. Here, we have chosen a metric — a number that summarizes the behavior of the system in some interesting way.

In general, there are often multiple ways of abstracting over a given set of variables, and each one will bring out different patterns. The more ways we have of looking at the system, the most insights we can gather.

The examples in the previous section are both second-level abstractions — they are both representations of the system for all time, instead of some particular time, and all algorithms, instead of some particular algorithm.

These representations reveal interesting high-level patterns. But they are too abstract to explain those patterns. For example, we observed that most of the trajectories remain in a tight bundle around the bend, but then spread apart on the straightaway. Why? In order to discover these causal relationships, we need to climb down a step on the ladder, and compare individual concrete instances.

Algorithms run on data. For many systems, that input data comes from some sort of environment. A video game character responds to the actions of its opponent. A building's smart lighting system responds to the presence and movement of people through the rooms. In our example here, the data consists of the shape of the road.

So far, we've been examining the algorithm's behavior on one particular road. But our goal is an algorithm that handles any road it encounters.

One approach is to randomly generate various roads, and see how the algorithm performs on them. With good high-level metrics, we can learn something about the overall performance of the algorithm, and where it's strong and weak. When done somewhat systematically, this approach is called "Monte-Carlo analysis".

However, it's difficult to deliberately explore a random space that doesn't offer well-defined dimensions to move in. In designing this algorithm, we want to understand why it behaves as it does for various data.* * This is not necessarily the case for all algorithms. For some algorithms, we don't expect to understand why or how they work. Machine-learning and genetic algorithms, in particular, might design their own rules and associations which we never actually see. (These algorithms obviously present their own challenges for designers!) This essay considers systems whose behavior emerges from human-authored rules, and thus human understanding is an essential part of the design process. It's thus very helpful for our data to have parameters that we can control, and thereby explain.

An arbitrary road could look like almost anything. In order to tame this data space, we choose some aspect of the road which we suspect is significant — an aspect that reflects some challenge that our algorithm will face. Our algorithm is currently built around a fixed turning rate which determines how sharply the car turns. We might therefore suspect that the sharpness of the bend in the road will play an important role.

Now that we are able to adjust the road, we can dig into our hypothesis that the trajectories "reflect" off the edge of the road at the same angle that they hit it. Is this exactly true, or an approximation? By examining many examples, can you understand why a fixed turning rate leads to this behavior?

We've abstracted over time, the algorithm, and the data. Are we done?

Of course not. Our goal is to design a good algorithm, and this algorithm is clearly terrible.

But that's okay! We intentionally started with the absolutely simplest possible algorithm. By thoroughly exploring it, we now understand why and how it's terrible. The exploration has given us three things:

• We have an understanding of the system. Now, this isn't necessarily a logical, conscious understanding! We learn a lot by studying and playing with these representations, even if we can't put our insights into words. What we are doing here is feeding the pattern-recognizer in our brains, and building up an intuition for how the system behaves. As the system gets more complex, we'll make subtle connections to patterns that we've seen before, and we can follow our instincts through the space of possibilities.
• We have ideas for improving the algorithm. If we had any ideas at the outset, they would have been mere guesses. The ideas we get as we explore are responses to real behavior and real failures of the real system. They are much more likely to be fruitful.
• We have a set of interactive visual tools that will continue to serve us throughout the design process. When we change the algorithm, we can immediately see the effects of that change. To design quickly and confidently, we need to be able to try out ideas and verify hypotheses as fast as we think of them.

So far, we've been looking at a toy example. Real-world systems may be more complex, but they all share the same general anatomy: an independent variable (such as time), a structure (such as an algorithm), and a dataset (such as an environment).

• The independent variable is usually time. This is our way of thinking about causality — a system's state depends on its previous states in time. Even for systems that are normally expressed with multiple independent variables, such as heat diffusion or wave propagation, we typically think of the system as evolving over time.

  "Time" needn't be real time. Time might be discrete unitless steps, as with our car example above, or the stages of an industrial process, or a sequence of messages in a communications system. Backtracking and divide-and-conquer algorithms have their own peculiar notions of branching time, which may be useful to visualize.

• The structure is the set of rules that the system obeys. In a board game or legal system, the rules are in prose. In software, they form an algorithm. In a physical system, they are typically equations implied by coupled components, according to the laws of the governing theory such as Kirchhoff's laws or Maxwell's equations. If the system evolves over time, these are often differential or difference equations. If the system varies over space, they may be partial differential equations or cellular automaton rules.

To understand a system, we must explore it. The three features above — time, structure, and data — define the territory. We explore this territory by controlling parameters, abstracting over them, and stepping down from these abstractions.

• We start by taking control. This forces us to clarify what our parameters are in the first place, and lets us develop a concrete sense for how the parameters affect the system.

  We want direct, interactive control of each parameter, so we can go forward and backward, stop, and jump to arbitrary positions. This allows us to travel freely around the territory, investigate areas of interest, and pose and verify hypotheses as quickly we think of them.

• We step up to an abstraction by devising a representation that depicts the system across all values of a parameter. This gives us a broad view of the system's behavior, and allows us to discover high-level patterns.

  The appropriate visualization varies, and often there are multiple good visualizations, each offering a useful perspective. We can look for metrics that summarize some aspect of the behavior. We can also consider transformations that make it easier to visually compare multiple states of the system.

Unfortunately, development environments generally don't support this process. Most are actively hostile to it. We live in primitive times.

If a language requires a "compile" or "refresh" to show the results of a change, it even denies us interactive control. Some languages are marketed as "sketchbooks", but a real sketching environment would, at the very least, offer basic interactive adjustment like to the right.

Perhaps someday this will change. Perhaps IDE makers will focus on dynamic exploration instead of static analysis, rich visualization instead of line debugging. Perhaps language theorists will stop messing around with arrows and dependent types, and start inventing languages suitable for interactive development and discovery.

Until that glorious day, it is our sad but unavoidable responsibility as system designers to build our own tools.

The good news is that the sorts of visualization and control presented here are not especially difficult to implement. They can be crude, ugly, and fragile. They can crash twice a day. They simply need to serve as scaffolding for our understanding.

Video games are perhaps our culture's deepest interactive systems, so it shouldn't be surprising that game developers have been building these tools into their games for a long time. On-screen metrics and diagnostics are the norm, and live adjustment of parameters is not uncommon.

At the other end of the scale are the sciences. Despite the fascinating models and high stakes, interactive control and dynamic visualization are shockingly rare. It's an appalling fact that "scientific computing" is a synonym for "high-performance number crunching" — despite Richard Hamming's vehement warning when founding the field: The purpose of computing is insight, not numbers.

It's unacceptable that our tools for understanding game worlds outperform those for understanding the real world. Until civilized scientific tools become available,* * ... scientific investigators must take responsibility for creating their own explorable visual environments.

An "abstraction", as used in this essay, represents a system for all values of a variable, instead of any particular value. However, we've only considered one particular form of abstraction so far — visual abstraction — instead of the more common form of symbolic abstraction.

Here is a symbolic abstraction — a mathematical expression that represents the system over all time.

Both are perfectly valid abstractions. Symbolic abstraction is the traditional form, because it runs well on the pencil-and-paper operating system that was dominant for hundreds of years.* * "Mathematica" is an emulation of the pencil-and-paper operating system on the computer operating system. It's like running a command line terminal in a window, to emulate a VT100 on your Mac. Both emulators offer some extra features — scrollback and selection in Terminal, solving and plotting in Mathematica. But the way we think about our work remains the same. Visual abstractions were difficult to generate until recently, and are still difficult to make interactive.

As a result of its long and deep history, symbolic abstraction offers certain benefits. Symbolic expressions can be reasoned about, composed, and decomposed in powerful ways. This allows us to logically prove properties about our systems, such that an invariant always holds or a failure state is impossible. We can also generalize our solutions to apply to many types of systems, as well as implement them efficiently.

However, it is often impractical or impossible to derive a symbolic expression that represents the behavior of most systems. We thus tend to focus on the tiny and contrived subset of systems that happen to yield to symbolic analysis. Even within that subset, symbolic analysis often involves struggling through a deluge of symbols, only to arrive at a solution that is difficult to interpret and offers little insight.

The most important issue is whether our abstractions are useful. Can we discover and explain high-level patterns? Do they generate insights that guide our design? Do they spark ideas? Do they lead to great systems?

Both symbolic and visual abstractions can do so. But the human brain is fundamentally a pattern-matching machine, easily outperforming any machine we can build — for certain kinds of patterns. With visual patterns, we win hands-down. With logical inferences, we're slow and clumsy.

We are human beings, and our tools should play to our strengths.

A ladder is a simple and useful metaphor, but it has only one dimension — step up, step down. Here are some richer ways of thinking about the abstraction process, which correspond to other concepts you may be familiar with.