[mattnewport]

Experience with tools like Blender or Nuke and particularly with visual programming in games engines is actually where a lot of the better informed dislike of visual programming comes from.

The biggest problem with these tools is scalability and maintainability. You will hear many stories in the games and VFX industries of nightmare literal spaghetti code created with visual programming tools that is impossible to debug, refactor or optimize.

Visual programming seems easy for very small examples but it doesn't scale. It has no effective means of versioning, diffing or merging and usually lacks good mechanisms for abstraction, reuse and hierarchical structure. It doesn't have tooling for refactoring and typically lacks tooling for performance profiling.

Some of these problems seem to be more fundamental and others like they could potentially be addressed with better tooling but that tooling never seems to emerge.

I've got a lot of experience with shader programming and have never found node based shader editors to be better than text over the long term, although there are some nice visual debugging features which are rarely implemented in text based IDEs (though I have seen it done). I've also found visual scripting to all too frequently get out of hand and have to be replaced with code due to being unmaintainable, undebuggable or unoptimizable.

I think there is possibly fertile terrain to explore in trying to get some of the benefits of visual programming approaches while avoiding all these downsides but many of us have been burned enough to be very skeptical of the majority of visual programming systems that don't even try to fix the worst problems.

[vvanders]

> code due to being unmaintainable, undebuggable or unoptimizable.

I would argue that the shader editor in UE3 had none of these properties. It showed you cycle count and each step of the graph visually for debugging.

Also, I don't mean to be blunt but you aren't the target for those tools. Where they shine is when you have a level artist that needs to make a small tweak to how a shader looks. With those systems you don't have to loop in a dev to make it happen. You still need a solid tech artist to make sure things don't get out of hand and they're not a tool for every problem but in the domains where it aligns you see 10x gains on a regular basis.

[nineteen999]

I think rather than the shader editor (which uses a similar-but-different node based interface) they are referring to the Blueprint programming system in UE4 which effectively wraps C++.

It's extremely powerful, but it comes at a cost because it's nigh impossible to create diffs between different versions of Blueprints AFAIK.

I did see this (https://forums.unrealengine.com/community/community-content-...) the other day which looks pretty cool, essentially being able to embed some C++ at the Blueprint level, but I don't think that will necessarily solve the problem.

[zsmizzle]

They added a visual diff/merge tool for Blueprints in Engine version 4.7 (See "Blueprint Merge Tool" here: https://docs.unrealengine.com/en-us/Support/Builds/ReleaseNo...)

[nineteen999]

That is neat, I'd never noticed that. Although in the context i've heard it discussed, it was more in the vein of "can't generate textual diffs/patch with them, as you can with C++".

[heavenlyblue]

>> It's extremely powerful, but it comes at a cost because it's nigh impossible to create diffs between different versions of Blueprints AFAIK.

Why not? What makes their representation undiffable?

[nineteen999]

I could be wrong but I believe the Blueprint assets are stored in a binary representation. So that rules out vanilla tools like patch/diff for the most part.

See the sibling poster, they did actually add a visual diff/merge tool in a much earlier version and I missed that.

[infinite8s]

Ascii is a binary representation. It's just that we've built up a lot of tooling around being to visualize/manipulate that representation. There's no reason that similar tooling couldn't arise for other binary representations

[nineteen999]

I'm not really sure what your point is. The point is ASCII/UTF-8 are a binary representation that is easily parsed by humans, which is why we use it for writing code. Sure, if you want, dump both files with xxd and do a diff on that instead.

There's been ways to diff binary files forever. Doesn't mean it is a great idea to store source code in that manner though. With UE4 you're not really supposed to be able to edit the Blueprint "code" outside the UE4 Editor.

[marcosdumay]

> It has no effective means of versioning, diffing or merging

Color solve this one problem just like they solve the text equivalent better than annotations. This is not fundamental.

> lacks good mechanisms for abstraction, reuse and hierarchical structure

VLSI circuit designers have some abstraction mechanisms that are not that bad. This one is fundamental, but it's not as bad as most people say.

> It doesn't have tooling for refactoring and typically lacks tooling for performance profiling.

I can't imagine how those could be fundamental problems either.

In principle the largest problems of visual programming are our lack of capacity of understand complex images, compared to text, and the added information that most visual languages add (making the complexity worse) (that do not apply to things like GUI designing). I don't see any other showstopper, but those two are really bad.

[dkersten]

> our lack of capacity of understand complex images, compared to text

I'm not sure I agree with this. Not as a fundamental law, at least. I find complex text pretty difficult to understand. Sure, I can read the words or even expressions, but to really understand how everything is connected and how the data and control flows through what's written, I find that incredibly difficult. A large part of programming is keeping that contextual information in my head so that I don't have to re-evaluate it all again.

Which is the reason why I love pen & paper and boxes & lines when I try to map out some code, a data structure, algorithm or idea. Obviously everyone is different, but for me, when things get too complex, I reach for images and diagrams to help me understand and form a clear mental model of what's going on or what I want.

Having said that, I've yet to find a visual programming language that does this. When I used Max/MSP a number of years back, I found it helped me think in "code" as I could map out ideas visually, reducing the need for pen and paper, but it had a ton of shortcomings. It gave me a glimpse of what it could be though, and I think the problems could be solved to a point where I can skip pen and paper altogether. We're not there yet, but if you think about how much time and effort went into making textual languages, its no wonder visual languages are so far behind. Also, its clear that not every task is well suited to visual languages, so the best environment would need to be a mix of both.

[anichale]

Good point about complex images. I do not think this issue will be solved anytime soon. An individual's mental image of a complex idea like inheritance or a data structure is subjective. Tying a specific visual GUI to a complex model (that many users would agree upon) is a very difficult design problem. Text allows the user to interpret the model using their own imagery.

[wires]

I don't think I agree. We know that there are diagrams that come from category theory that behave very well, this is what statebox uses. It enforces certain constraints, so we pivot everything around this and see if we can recover some form of programming from it.

certain cases are still much better done in text and statebox itself is written in text, so nobody is claiming programming itself should be 100% pictures, but then again, I think it can be done and I think there is merit to it.

after all, text is also some sort of picture; on the screen but also in your head

[blevin]

Very much agree on these points. In my experience the pragmatic path for larger projects has been a hybrid of keeping visual programming graphs self-contained -- roughly within the "asset" boundary -- and then using a hierarchical format with textual representation to assemble those assets and drive their interface parameters. This is roughly how animation studios use USD (openusd.org), for example. Arguably this is just a strategy of combining a couple domain specific languages, with edges where the optimal tradeoffs of each domain flip. It's a very interesting question whether these limitations of visual programming are essential complements of their benefits, or if there is some good way to provide better abstraction, re-use, etc. Certainly Houdini, Nuke, Katana, etc. all provide limited forms of those things (ex: Houdini OTL's; scripts that version-bump nodes and try to auto-upgrade the parameters), and they do see lots of use in industry.

[wires]

so I started my studies in animation actually, I used a lot of those tools, you can think of statebox as applying category theory to restructure aftereffects or https://en.wikipedia.org/wiki/Shake_(software)

when done right, we claim, you can target many different things ("semantics")

what we claim is something like, the compositional aspect of many such node based systems can be described as a certain type of mathematical object (monoidal category) ~ we can build an editor for that and then map that "dsl" to particular targets (image processing, state machines, etc)

[tomc1985]

> The biggest problem with these tools is scalability and maintainability.

This has been my biggest complaint with both Unreal and NI Reactor... you can build some brilliant things, but not without creating a mess of connections that becomes very difficult to reason about, let alone work on. A lot of production "visual programming" diagrams are spaghetti code, without comments or a changelog you can study.

While written code tends to the same way (just try diagramming the class tree in most software), at least we have tools for dealing with and reasoning about it in text form.

[crankylinuxuser]

Indeed, spaghetti is a big deal. However, in systems like [Node-Red, Antimony, Apache NiFi] you can make function nodes that abstract out the actual data vs the function. And then, your function primitive can be called and instantiated for the particular purposes.

I know in Antimony CAD, you can even instantiate a function, edit the function's flow or individual python subroutines, and the delta is saved for that instance.

I know the hardest graphical programming I've had was with the Lego Mindstorms ev3. There were multiple "simplifications" that removed functions along with a nigh-unusable gui (the if block was this huge encapsulating thing, and other branch codes also had onerous things on screen).

[wires]

the tools for reasoning about text can be limited if the text is in a language unsuitable for reasoning

this is why we work in a typed, purely functional setting.

spaghetti is difficult to deal with, for starters, you need "compositionality", so that you get no undefined or emergent behaviour.

then second you need some form of "graphical macros", or a "meta-language" for diagrams, code that generates diagrams or "higher order functions" for diagrams.

[zozbot123]

> ...and usually lacks good mechanisms for abstraction, reuse and hierarchical structure. It doesn't have tooling for refactoring...

Arguably, the whole point of having a diagrammatic representation with good formal properties is to provide new mechanisms for accomplishing these things. Of course these mechanisms, features etc. won't quite resemble the ones that are in use with text-only languages.

[setr]

Read it as refactoring, or equivalent; the described functions all fall under organizational and editing tools, both of which are necessary in visual and text environments. Lacking decent mechanisms would make a full buy-in difficult for any larger project, regardless of the precise manner in which its done. Without it, you have a write-only language

[wires]

100%

this is the point exactly

[candiodari]

You shouldn't generalize like this. Even in massive projects visual programming can be enormously helpful. Nobody seriously tries to script game levels in an IDE. Nobody tries to match sprite parameters, which eventually become code, to image files in an IDE. Nobody tries to design GUIs in IDEs.

The right tool for the right job. For 50% of a game's code visual programming is absolutely the right tool. For some other parts it probably isn't.

[mattnewport]

You seem to be using a different definition of visual programming than the normal one. The article and my comment are talking about visual programming as a graphical representation of code / logic rather than a textual one. Examples would be things like Unreal's Blueprint visual scripting, node based shader editors (Unreal has one of those too) or the Petri nets described in the article. Visual / GUI tools like sprite editors, level editors or GUI builders are not what is usually meant by visual programming.

Unity is a very popular game engine that doesn't have an official visual programming solution (they're previewing one in the very latest version). Unity has a powerful level editor that is used to lay out the levels in a GUI tool but no visual scripting / programming tools. The majority of Unity games that currently exist therefore do all level scripting in C# code. Many other games engines have no visual scripting solution and all level scripting is done in either a scripting language like Lua or in some cases in C++ code. Unity has sprite editors, visual GUI builder tools etc. but those are not what is generally meant by "visual programming". The closest Unity has had until recently was its graphical animation state machine editor.

[wires]

awesome comments, really cool to read all of this.

anyway, I would argue both are valid examples is graphical progamming, but they happen at different levels.

the "node based" tools usually define some sort of function or system, ie. a "type"; for this you need category theory to describe how the diagrams look and this is not what any editor I know does, but it means a whole of difference.

And the map editors are for defining "terms of a type", given a definition of a "map datatype" there is a graphical way to edit it.

when we talk about graphical programming we are initially focussing on the first, well defined graphical protocol definitions. you can think of it as type checked event sourcing, where the "behaviour" or "type" is described by a (sort of) graph representing a (sort of) state machine)

but we have relatively clear idea's to extend this to the second case as well.

The difference with other (older) approaches is that in the last 20 years a lot of mathematics appeared dealing with formal (categorical) diagrams or proof nets, etc. that we leverage. I claim we (the world) now finally really understand how to build visual languages that do no suck.

hence statebox :-)

[mattnewport]

Things like a GUI designer or level editor map a 2D or 3D domain to a 2D or 3D-projected-to-2D space. A 3D animation editor maps a 4D domain to a 2D projection of a 3D representation plus a timeline representing the 4th time dimension. These mappings are natural, intuitive and work well generally.

Visual programming tools attempt to map logic to a (usually) 2D domain where there is no natural or intuitive general mapping. The representation has both too many degrees of freedom (arbitrary positions of nodes in 2D space that are not meaningful in the problem domain) and too few (connections between nodes end up crossing in 2D adding visual confusion due to constraints of the representation that don't exist in the problem domain).

I've been exploring colored Petri nets for our product and they do seem to have promise for certain use cases though so I do think it's an interesting area to explore.

[Nigredo]

> Visual programming tools attempt to map logic to a (usually) 2D domain where there is no natural or intuitive general mapping. The representation has both too many degrees of freedom (arbitrary positions of nodes in 2D space that are not meaningful in the problem domain) and too few (connections between nodes end up crossing in 2D adding visual confusion due to constraints of the representation that don't exist in the problem domain).

In general this is true, but the diagrams we use at Statebox are different in the sense that there is a completeness theorem between the diagrammatic language and an underlying mathematical structure (a category). In this case the mapping is sound by definition.

Also, it is worth stressing that our diagrammatic calculus is topologically invariant, meaning that the position of diagrams in space is meaningless, everything that matters is connectivity. This is also the approach originally used by Coecke and Abramsky in the field of Categorical Quantum Mechanics, which is getting huge success to define quantum protocols :)

[tomc1985]

Why does category theory magically transform node diagrams into something usable from something not? Unreal blueprints/Reactor schematics/whatever are quite fine in their current form, even if their usage falls apart in advanced constructions. Is statebox going to magically make huge node-and-graph-designed programs reasonable?

Your writeup didn't convince me that "category theory" adds any significant value, and neither does it help inform as to what category theory actually is. How does statebox improve upon existing node-based programming implementations?

CS formality and big words misses the point of visual programming entirely, in that it is to simplify the process of software creation to make it more approachable to non-programmers. Unless your UX is absolutely top-notch you are going to lose these novice users as they struggle to deal with the constraints without a good reason or UX to do so.

Also- The memetastic design of statebox's main page is a pretty big turnoff :(

[wires]

hi tomc1985

> Why does category theory magically transform node diagrams into something usable from something not? Unreal blueprints/Reactor schematics/whatever are quite fine in their current form, even if their usage falls apart in advanced constructions. Is statebox going to magically make huge node-and-graph-designed programs reasonable? Your writeup didn't convince me that "category theory" adds any significant value, and neither does it help inform as to what category theory actually is. How does statebox improve upon existing node-based programming implementations?

nothing magical, just good engineering and UX design and solid theoretical underpinnings.

cat. th. does add value: there are many ways to build diagrams and build syntaxes for diagrams, but they are not all equivalently powerful or general. but it turns out that there are diagrams that _are_ suitable, and this is what we use.

It will improve upon existing diagram tools in that it gives a formal theory of how they work, so you can really build huuge diagrams and still be sure everything works.

I didn't write the blog post, but I could try to write one about the value of category theory, because it is often misunderstood. It is however very abstract and takes the mind a while to see the value off, which is not so easy to convey.

> CS formality and big words misses the point of visual programming entirely, in that it is to simplify the process of software creation to make it more approachable to non-programmers. Unless your UX is absolutely top-notch you are going to lose these novice users as they struggle to deal with the constraints without a good reason to UX to do so.

oh, yeah this is something often misunderstood, we are not trying to target novice developers (yet). we need to develop a lot of stuff and CS formality is right now still the simplest way to understand the system. I mean, we are not trying to be arrogant or puffy or something, but for instance the way we realise our compilation is with "functorial semantics". we have a functor between categories that does the trick. We could call it something else, but it doesn't help (at this stage).

anyway, if we do our job well then all the category theory would be under the hood and you just get a nice UX for coding with diagrams.

> Also- The memetastic design of statebox's main page is a pretty big turnoff :(

opinions differ :) I thought it was quite funny 2 years ago and many people thought so as well and then it got turned into this homepage.

at the moment we don't really have time to spend time on the site, but it will def. be changed in the future

[tomc1985]

You're making a lot of generalizations for a post that says not to make generalizations.

Lots of people got their starts programming a GUI in their IDEs, how can you say "nobody" would do this?

[goatlover]

> Nobody tries to design GUIs in IDEs.

Smalltalk would be an exception. The graphical elements are provided for you to make your own inside the image on the fly, or you can modify the IDE as needed.

[V-2]

> Nobody tries to design GUIs in IDEs

Design, or encode? Because visual designers in the latter sense typically have many shortcomings, and are usually used mostly for preview purposes.

[mcphage]

> Experience with tools like Blender or Nuke and particularly with visual programming in games engines is actually where a lot of the better informed dislike of visual programming comes from.

Maybe the better informed dislike, but the bulk of the dislike in general is usually of the form "this kind of thing never works, nobody uses it", when in fact it is being used, in multiple disciplines.

[wires]

Great points,

> scalability and maintainability > easy for very small examples but it doesn't scale.

this is very true, that is why we do it differently; we clearly define the semantics of our diagrams and take guidance from category theory in this. this is different from other graphical languages; we try to assume the minimum but then guarantee you that some stuff is always preserved.

think of it like deterministic, pure functional programming, but with diagrams.

> usually lacks good mechanisms for abstraction, reuse and hierarchical structure. > f versioning, diffing or merging

very important points, we try to address this by having everything based on immutable, persistent data structures with built in content addressing; similar to git for instance.

diffing and merging is very complicated and still research but there are many hints that this can be done

[snidane]

Obligatory link to show scalability issues in visual programming. https://scriptsofanotherdimension.tumblr.com/

[emmanueloga_]

I think those examples are a bit misleading in that nobody actually prints 100,000 lines of code and looks at them all at once. So those examples are showing probably the whole "program" at once, which is impressive and looks daunting, same as a big textual code base.

Rather, in order to understand a certain aspect of the system, I imagine picking just one of the nodes and then asking the IDE to show me, say, the immediate inputs. Some of these may be semi-hidden (code folding!), others may show more detail, etc.

BTW, I've been thinking it would be great if these systems had a textual representation in the vein of Graphviz' dot language. So one could have the best of both worlds. For diffing, a simple textual diff could do, but one could come up fancier semantic diffs in the same vein semantic diffs for code or XML exist.

[dkersten]

When I used Max/MSP a few years ago, a lot of people's code snippets looked like this, however, an important thing to remember is that the target audience for most visual programming tools are not trained software engineers -- ie they don't know about encapsulation and abstraction and all of the other concepts we take for granted.

I found Max/MSP code could look extremely tidy, as I would group my functionality together in a similar way as I would in a textual language (short single-purpose functions and such). You can write horrible spaghetti code messes in textual languages too, its just we've used enough textual languages to have learned how to abstract our code and how to organise it for maintainability. Its not an inherent feature.

[mbrock]

At least there you can see how convoluted the systems really are, at a glance...

[wires]

love that blog, great anti-examples, thanks for sharing!

we are really trying to avoid coming to this style of visual programming

write once, read never

[atoav]

And this is why you can easily script things with python also in blender.

For simpler usecases the nodes still make a ton of sense (e.g. compositing, shaders, ...) And I say that as somebody who loves CLI and programming.