Substroke was a research language for drawing dynamic (data-dependent) pictures.
The description given here was intended as a brain-dump of a work-in-progress. The work-in-progress is no longer in progress as such, but ideas and insight from this exploration will find their way into future work on dynamic drawing.
This description is an admittedly poor explanation of the language, especially lacking in the motivation for each of the features, but I hope you can glean some inspiration from the high-level ideas and general philosophy.
If you get overwhelmed by minutia, try going straight to the examples. (If you hunger for even more minutia, see the details page.)
Contents:
It is not currently possible to draw dynamic pictures.
By draw, I mean construct via direct manipulation of the picture itself, with a clear view of the picture being constructed. By dynamic picture, I mean a graphic that can vary, in a fully general way, according to a set of parameters.
Most things that appear on a computer screen can be considered dynamic pictures. The most interesting cases to me are information graphics and technical drawings. Some examples:
Currently, creating such pictures requires an awkward dance of drawing components or mockups in a static drawing tool (Photoshop, Illustrator, etc.) and then blindly manipulating them in a textual programming language. Unlike drawing on a canvas, textual programming offers no visual feedback -- the programmer must imagine the effects of each code element, and maintain a complex mental mapping between lexical and graphical concepts. This indirection severely limits artistic freedom.
Furthermore, most visual artists cannot handle the textual abstraction. The result is poorly-designed pictures drawn by programmers, and pictures tediously designed by an artist/programmer pair.
I believe that much valuable data, particularly in scientific applications, is unvisualized or poorly-visualized, because visual artists lack the means to create effective visualizations.
For a more detailed motivation, see Designing a Design Tool in Magic Ink.
Substroke is a visual language for drawing dynamic pictures.
The language is intended to be a "Scheme for pictures". That is, the intent is a simple, elegant framework, using simple, fundamental constructs, that allows fancy functionality to be implemented in libraries.
This is a research project. It is ambitious and exploratory. It is more important that the language inspire descendants than be directly usable itself. It is more important to change the way people think about graphics programming than to create the end-all graphics tool.
Principles:
This section gives a tour of the language. Slightly gorier details are are on the details page.
Substroke is conceptually a transform language. The artist thinks in terms of transforming one picture to another:
In the top-level case of transforming a set of data into an information graphic, the initial "picture" might be a block of text:
A chain of transforms is called a strip (in analogy to a comic strip or film strip). Each picture in the strip is called a panel. Strips should appear comfortable to artists -- if you were to take a series of snapshots as an artist drew a picture, it might look like this.
A function is a strip with a name. The function below doesn't offer any abstraction -- it is essentially a thunk:
The function below is abstracted, because it takes a direct object. The first panel in the strip operates on the direct object:
The function can be applied elsewhere. Wherever it is applied, the previous panel is used for the direct object:
A function can have properties. These properties are themselves functions, which may or may not have direct objects. The function's strip can refer to the properties. The properties can refer to each other.
When a function is applied in a panel, its properties can be overridden by parameters:
The function below has no strip of its own -- it has only properties. Such a function is called an object constructor:
When this function is applied, it returns an object that holds the given properties. An object is rendered visually by evaluating its Appearance property.
Below each panel in a strip, there is a description of the object in the panel. By default, the description is the name of the object's constructor.
Each panel holds a single object. The description "4 shapes" is short for "Group of 4 shapes". A Group is an object that has a "First" property and a "Rest" property. The "Rest" property, if it exists, is another Group.
A property can be extracted from an object via the possessive syntax. Below, within "Pentahair", we extract the Hair property from the Pentapair object using "Pentapair's Hair":
Members of a Group can conveniently be extracted by index. One or more indices can be provided. The syntax "#7,etc" refers to member #7 and all subsequent members.
To refer to the object in the previous panel, we can use the pronoun It. "Its" and "It's" are synonyms.
We can refer to an object in any panel by giving the panels labels. A label can be used anywhere within the function's strip or properties (even in earlier strips!). Below, the second panel is labeled "Greenie", and is used in the fourth panel.
A label (including "It") can be applied like a function. By giving parameters to this function, we can "set" the object's properties. (Note that objects are immutable, so by "set", we mean "reconstruct the object with some properties overridden".)
There is an object named Nothing. Nothing has no properties and no appearance. If you try to extract a non-existent property from an object, you get Nothing.
Sometimes a function must do one of several things conditionally. We can supply multiple definitions of a function, with some properties marked (required):
When a function is applied, the required properties in its first definition are evaluated. If any of them evaluates to Nothing, the next function definition is tried.
If none of the definitions pass their requirements, the function returns Nothing.
The direct object is implicitly required. This allows many recursive functions to be written without an explicit base case. In the function below, the recursive invocation automatically returns Nothing if Source's Rest is Nothing, thereby terminating the recursion:
If a base case is needed, we can provide a second definition which does not take a direct object.
It is possible to give an object new properties via application:
However, this isn't typically useful. Due to lexical scoping, existing properties and new properties cannot see one another.
To add or override properties that can interact with existing properties, we can extend an object. In the function below, the "Base" property is marked (extended):
When a Hexapair is constructed, the "Base" property is evaluated, and the properties from the resulting object are (conceptually) copied to Hexpair. It behaves (aside from some scoping subtleties) as if we had defined it like so:
There are now two definitions of "Head", but Hexapair's definition appears first, and thus dominates. (Although if Hexapair's Head had required properties that failed, it would "fall through" to the Pentapair's Head. That is called conditional override.)
Hexapair inherits Pentapair's functionality, but provides a new "Face" property:
Extension is dynamic -- extended properties can be a function of the direct object or other parameters. This allows us to create mix-ins. Suppose we have a function called "brushstroke" which takes Shapes and renders them with an organic brushstroke:
We can mix this functionality into any object with a mix-in:
Not only does this give any object a brushed appearance, it also adds a "Brush Width" property to the object, which controls the effect:
Any number of properties may be marked (extended). Higher ones take precedence.
With extension, scoping is not strictly lexical. But if we think about extension as copying one definition into another (ie, as a sort of run-time macro), we can reason about it similarly. In particular, functions "close over" (capture) externally-defined names as in a lexically-scoped language.
Multiple objects can be tracked independently through a strip, by using scratch layers. Layers other than layer 1 are tinted a particular color:
The color of a function application indicates which layer it takes its direct object from. The (n) prefix indicates which layer it sends its result to.
Layers allow multiple objects to be viewed in the same space. They are helpful when multiple objects interact:
Property linking binds one layer to a property of another layer. When either layer changes, the other layer is updated.
Generic linking allows two layers to be bound via a linking function. When either layer changes, the function is called to update the other layer.
Linked layers are extremely important -- they allow the artist to adjust an object by "selecting" parts of it and manipulating the parts on the canvas. Generic linking is especially important -- it allows one to define various "interfaces" through which an object can be manipulated. But I'm having trouble right now coming up with good example pictures. Linking is explained better on the details page, and it's used a lot below; hopefully you can get the idea.
Substroke is a language based on extending immutable records. It is quite similar in that respect to the research language Piccola (which I discovered only after I did most of the work here!).
Substroke is a pure language. Functions have no state or external effects. In fact, there is no notion of "time" -- there are only dependencies.
Substroke is a lazy language. Computation proceeds by working backward from the output, along a dependency graph. Properties are evaluated only when "needed". (Where "is needed" might mean "is applied as a function", "has a property extracted", or "is a required property".)
Despite being a pure language with immutable data, the comic-strip syntax provides the comfortable familiar feeling of mutating a picture, even allowing objects' properties to be "set".
I have explored most of the language's features using micro-examples. It is difficult to come up with complete macro-examples, because libaries play such a major role, and library design is a can of worms that I haven't opened yet.
Below are a few library functions which might be important.
Distances and angles are represented with paths, not numbers. However, functions are available to convert when necessary, along with a little syntax sugar:
Path intersection:
Shape algebra:
Move, resize, rotate:
"replace" returns its "With" property, but uses the direct object's location and coordinate transform.
"replace each" is similar to the conventional "map", but respects the location and transform as above:
"replace each" can take an optional predicate:
"select" is similar to the conventional "filter". (It can also be used as a linking function, as shown above.)
Midpoint, subpath, tangent, normal. (The percents should be visual, but I don't know how yet.)
Translating an object along a path.
Tweening shapes.
Using a Path as a mapping curve. (Below might be an "ease" curve.)
The Path object has a Points property, which is a Group of Points or Paths. These represent Bezier points/vectors. The Path's shape can be manipulated by manipulating this property. (Layer-linking is especially helpful.) The key idea is that reshaping is done with plain old geometry functions -- move, resize, rotate, etc. (A Shape is the same as a Path, but it's closed.)
Even color can be represented and manipulated geometrically. Below, color is represented by a vector in colorspace. Rotating the vector changes hue, moving it horizontally changes saturation, and moving vertically changes brightness. (Note that the "colorspace" linking function provides an interface for changing the color, and the actual Color property may be implemented differently.)
The IDE looks and feels almost exactly like a conventional vector drawing tool, such as Illustrator. It is intended to be used with one hand on the mouse or stylus, and the other on the keyboard. Most Substroke-related functionality is specified through drawing or dragging. Typing is normally only used for naming properties and labels.
Simple manipulation functions can be specified by manipulating the example in the appropriate panel. Below, the artist resizes and moves the blob using the conventional gestures, and the IDE automatically creates the "scale" and "move" panels. Note that, as panels are added to the right, the entire document pans to the left. This keeps the manipulated panel in exactly the same position, so the artist can create a series of panels simply by drawing in place.
Below, the artist goes back and rotates the first panel. A new panel is inserted in the second position, and the examples in subsequent panels update appropriately.
Most drawing tools provide "smart guides" which "snap" dragged objects into alignment with other objects. If the artist drags an object into some alignment, the IDE creates a panel which applies the alignment:
Many common dynamic relationships can be specified in this way. It also encourages the artist to draw explicit guides on scratch layers, which makes the program easier to read and modify later.
And because Bezier points, colors, and (hopefully) most other things are represented geometrically, they too can be "programmed" simply by dragging objects on the canvas:
The artist can create new "blank" panels simply by drawing a box:
The artist can factor a range of panels into a property by selecting them and dragging them to an empty area.
Above, the name "new property" can be renamed in either place it appears, and the other will update automatically. (That is, the IDE remembers a binding that is stronger than just a name.)
Similarly, one or more members of a Group can be factored by dragging them off:
All library functions are listed in a scrolling pane to the side. The library functions are categorized for easy browsing, and the pane also has a live search-box, which searches by name, tags, and description/documentation, in that order. Functions are shown as before/after pairs with short descriptions. I'm not sure how parameters are documented.
The artist uses a library function simply by dragging it from the library to the appropriate strip.
Below is an eye that can follow a target. The eyeball is constrained to lie within a given outline.
Like any unfamiliar language, this may seem overwhelming at first. The way to read it is to (initally) ignore all the text above and below each panel. Just look at the property names and example pictures as if you were reading a comic strip. Glance up to the function name when you don't understand what's happening in a given panel.
Below is a gear shape. The tooth width (and number of teeth) is controlled by the "Tooth Angle" property.
Below are Substroke implementations of some standard list-munging functions. There are many possible implementations for each of these; I've tried to come up with ones that reflect a (nascent!) "Substroke Style".
The above function, when given a Group, returns the first n members, where n is the size of the "As Many As" parameter. Notice that no base case was needed -- when either the direct object or "As Many As" becomes Nothing, the recursion will stop. Also notice that the return value was not constructed from scratch, but was obtained by "setting" the input's Rest property (by linking to "Its #2,etc"). This might be important if the input is not just a raw Group, but has been changed in some way (via a geometric transform, a mix-in, ...).
The above function applies its "with" parameter to each member of a Group. It would conventionally be called "map". Notice how layer linking makes the strip easy to read -- the affected parts of the group "light up" in red before and after they change.
The above function returns the members of a Group that fulfill a given predicate. The predicate is passed as the "If" parameter. In the example, it is filtering the members that intersect a particular shape.
Note that the recursive calls to "filter" assume that the overridden "If" parameter is maintained, and does not need to be passed explicitly each time. I don't know whether this is correct behavior. Probably not.
This section discusses the internal implementation of an interpreter. A user of the language would never see this.
I have implemented a toy intepreter, mostly as a way of discovering technical flaws and inconsistencies. The compiler accepts a program in substroke-text format. It translates it to a substroke-core data structure, which is then translated a little more into an internal structure which is executed. Consider the following program:
The substroke-text for this would be:
constructor:
function [source]:
hello [label]
pass
with:
param
its prop
2.1 from1
3=1's linky
1.2 backto1
subproperty:
sub
property (required):
foo
daddy (extended):
bar
This is translated into the following substroke-core data structure (shown here with Lua syntax):
constructor = {
_extended = "daddy",
_required = "property",
function = {
_eval = "panel_6",
source = "panel_0",
label = "panel_1",
panel_0 = "nothing",
panel_1 = {
_apply = "hello",
panel_0 = "panel_0",
},
panel_2 = {
_apply = "pass",
panel_0 = "panel_1",
with = {
_eval = "panel_1",
panel_1 = {
_apply = "param",
panel_0 = "panel_0",
},
},
},
panel_3 = {
_apply = "panel_2.prop",
panel_0 = "panel_2",
},
panel_4 = {
_apply = "from1",
panel_0 = "panel_3",
},
panel_5 = {
_apply = "panel_3.linky",
panel_0 = "panel_3",
},
panel_6 = {
_apply = "backto1",
panel_0 = "panel_4",
},
panel_7 = {
_apply = "panel_6.linky",
panel_0 = "panel_6",
},
subproperty = {
_eval = "panel_1",
panel_1 = {
_apply = "sub",
panel_0 = "panel_0",
},
},
},
property = {
_eval = "panel_1",
panel_1 = {
_apply = "foo",
panel_0 = "panel_0",
},
},
daddy = {
_eval = "panel_1",
panel_1 = {
_apply = "bar",
panel_0 = "panel_0",
},
},
},
Note that panels and properties are one and the same here. The direct object is passed through the property "panel_0". Layers don't exist at this point; the only evidence of layers is which "previous panel" is passed as the direct object. (panel_6, for instance, takes panel_4 as direct object.) Note also that panel_7 was generated automatically, because a linked layer was changed.
A strip has the meta-property _eval = "panel_n", which indicates that evaluating the strip means evaluating the final panel. A function application has the meta-property _apply = "function_name". An object constructor has neither.
The interpreter is fairly simple. (It's also stunningly inefficient, due to the prevalence of deep copying.) Below are the two primary functions:
function eval (obj)
if obj._name then return evalName(obj._name, obj._parent) end
if obj._extended then return eval(extend(obj)) end
if obj._apply then return applyName(obj._apply, obj._parent, obj) end
if obj._eval then return evalProperty("_eval", obj, obj) end
return obj
end
function apply (func,params)
if func._name then return applyName(func._name, func._parent, params) end
if func._apply then return apply(eval(func), params) end
local obj = copy(func)
for name,value in iterateProperties(params) do
obj[name] = copy(value, params._parent, params)
end
return eval(obj)
end
See the details page.