Introduction (2 min)

This talk is about a particular kind of media, which is "media for thinking in." And it's about a particular kind of thinking, which is understanding systems.
"Science" is understanding a system in the world. "Engineering" is building a system in the world, where the challenge is often understanding what we're building.
Media are our thinking tools. Our representations of a system are how we understand it.
To understand or build new complex systems, we need powerful new representations, and we need a powerful new medium in which to work with these representations.
Today's representations were designed for the medium of paper. This talk will show examples of new representations for systems, and offer hints as to what a new medium might be like.

Demo 1 — Seeing behavior (3 min)

This first example shows a scientific paper. The example is about showing the behavior of a system. (For more information, see Scientific Communication As Sequential Art.)
The authors have a rich picture in their heads, which they'd like to transmit to the reader's head. But paper is a "low-bandwidth channel", and much information is lost.
This redesign of the paper uses words and pictures, tightly intertwined. Words describe the structure of the algorithm, and pictures depict the behavior of the algorithm.
Readers are shown the state of the system at each step of the algorithm, instead of simulating it in their heads.
Readers are shown relationships between variables, instead of reconstructing them in their heads from expressions.

Demo 2 — Seeing the entire behavior (3 min)

This example shows an environment for working with electronic circuits. (For more information, see Inventing on Principle, 23:22.)
To understand a system, it's not enough to see one variable at a time. This example is about seeing all variables at once.
In this environment, the designer can see all voltages at once, visually overlay any voltage on all other voltages to compare, and adjust parameters to see how the behavior responds.
To show current, the symbolic representation of each component is replaced with a visual representation of the component's data.
This representation conveys the same structural information as a conventional schematic, but it conveys behavior as well. It's built out of live data, not dead symbols.

Interlude — Thinking the unthinkable (8 min)

Richard Hamming suggested that, like sounds we cannot hear, there could be thoughts we cannot think.
But how do we even know about such sounds? We build tools which adapt unhearable sounds to our human senses.
Likewise, thinking tools adapt unthinkable thoughts to our human minds. Two of the greatest such tools are writing and mathematical notation.
To build a thinking tool, it helps to understand how people think. Jerome Bruner considered three mentalities — interactive, visual, and symbolic.
Pencil-and-paper constrained us to mostly simple symbolic representations. In the new medium, representations can draw on all three mentalities simultaneously.

Demo 3 — Multiple representations of behavior (3 min)

This example is from signal processing, and is about multiple representations for the same behavior. (For more information, see Explorable Explanations.)
As the designer adjusts parameters, five different ways of interpreting the behavior are shown simultaneously. Each representation offers its own unique insights.
By seeing how the different representations dance together, the designer builds associations between them. This leads to intuition — the ability to "feel your way" around a system.

Demo 4 — Interacting with behavior (5 min)

This example shows an environment for working with difference equations. The example is about directly interacting with behavior as opposed to just parameters.
Defining and seeing transformations of the system's variables.
Selecting parts of the behavior, measuring them, and seeing reductions of them.
Using search to dynamically select parts of the behavior that meet a condition.
And plotting any measurement against any parameter. This abstracts over the parameter, giving a representation of not just one particular system, but an entire family of systems. (For more information, see Up and Down the Ladder of Abstraction.)

Demo 5 — Linked representations (4 min)

This example shows an environment for working in the Nile programming language. The example is about representations linked together. (For more information, see Nile on Github.)
This is the code for a vector graphics engine. Both the language and code are well-designed, but it's still difficult to understand them because you can't see the behavior of the system.
In this environment, the data in the pipeline is visualized at each stage.
Pointing to any object highlights where it came from in previous stages, where it went in later stages, and the lines of code that produced it. The representations all work together.
Different types of data use different visualizations. You can understand the gist of the how the system works by ignoring the code and just looking at how the data was transformed.

Demo 6 — Creating representations (8 min)

A medium for working with many dynamic representations must provide a good way to create all these representations. (For more information, see Drawing Dynamic Visualizations.)
Visualizations should be created by thinking visually. Hand-drawing is direct and expressive, but not dynamic. Code is dynamic, but coding is indirect, symbolic, non-visual.
This example is a tool in which the designer specifies a picture by directly drawing on a canvas, as in Illustrator. But drawing actions can then be parameterized with variable expressions.
The picture can therefore depend on data. With different data, we see a different picture. These pictures can thus be used as representations in systems.
Designers aren't confined to standard chart types, but can draw ad-hoc representations to explore whatever they need to understand about a particular system.

Closing (3 min)

Summary of the principles and examples presented in this talk.
All of the examples I've shown here are hints. They are nibbling at the corners of a big problem — what is this new medium for understanding systems?
We must get away from pencil-and-paper thinking. Even when working on the computer, we still think in representations that were invented for the medium of paper.
Especially programming. Programming languages are written languages — they were designed for writing.
We have an opportunity to reinvent how we think about systems, to create a new medium. I don't know what that medium is, but if you'd like to help find it, let me know.