[chubot]

It is indeed great, and probably one reason you don't see it more is that it's a ton of work! That's 5K lines of hand-written code! (view source and it's at the end of the page)

    $ wc -l gears.js base.js 
     4135 gears.js
      904 base.js
     5039 total

I've wanted to make visualizations like this for my blog. Writing a blog post takes me around 10 hours, which is a fair amount of effort.

I believe that the visualizations will take more time than writing in general -- i.e. it would be more like 20 hours, bringing the total to 30.

If you consider that it's 4K lines of code (assuming the base library is reused), it's very easy to see that it could take 20+ hours. Probably more like 40-80 to be honest.

I guess there is that other thread that says people only write 10 lines of code a day, which would make it 400 hours ;) I don't think that applies here but it could be closer to 400 than 10 or 20.

----

I also wonder if you can save time by using a less "hand-written" style (e.g. d3.js, which seems to be on everyone's wishlist to learn).

My inclination is also to go "hand-written" rather than using a bunch of JS libraries. I think you get a better result that way. It's interesting to see that I'm not totally wrong -- the thing everyone praises ends up being very hand-written. And it's smooth and fast, etc.

----

another edit: I also believe one reason that this visualization is good is because there's no build process in the JS. The author clearly just edits the code and refreshes the browser. You need that kind of fast edit-view loop to make good visualizations.

IOW, consider using plain JS for blog posts. They are documents and not apps.

[amitp]

Having been down this road, yes, it does take time, but you can save some time by using SVG and some DOM library. Although canvas is faster, most diagrams don't really need it, so I use SVG unless I really need to switch to canvas. SVG also adjusts for screen dpi automatically.

The things I like:

1. Reactivity (ObservableHQ, Vue.js, hyperactiv.js, etc.). There's usually some underlying data and then a corresponding visualization. These reactive systems let you modify the underlying data and then the visualization updates automatically. You don't have to figure out which diagrams to update when. Even easier: just redraw everything every time you change anything.

2. Some easier way to write the DOM (d3.js, jsx, vue, lit-html, etc.). Since I'm writing a blog post with html, I usually prefer writing my js-in-html (vue) rather than html-in-js (jsx) but try both directions and see which you prefer.

3. No build step. This is especially important when I want to update a page years later and don't want to figure out which build tools I was using in 1997 or 2007 or 2017. I want my pages to last for decades, and I still update my pages from 25 years ago.

I tried recreating one of the gear page diagrams in ObservableHQ https://observablehq.com/@redblobgames/unwind-circle-example . It's not a lot of code. There's a slider, there's a loop to generate the lines, and there's the output svg. Whenever you move the slider it recalculates the output.

I admit that I'm not using ObservableHQ much for my own projects because I want more of a "hand-written" style. I used d3.js for my older pages and vue.js for my newer pages. Vue's reactivity and templates save me probably a factor of 2 or 3 over d3.js.

[chubot]

Thanks for these tips -- I've seen your visualizations and it's nice to hear from people who have done it!

It's cool that you were able to reproduce the diagram quickly and in a small amount of code. It looks a bit foreign to me, probably because I don't know much about SVG (or canvas for that matter). And as I understand it Observable is almost another language on top. (I do know HTML, CSS, and JS pretty well, but there's still a gap.)

Do you ever mock visualizations up in a WYSIWIG tool, or do you always use web technologies in a text editor?

Doing it programmatically has advantages when you need to make 30 similar diagrams, as in this post about gears.

But I also feel WYSIWIG tools could help in prototyping to avoid throwing away a lot of code that wasn't properly conceived of. That is, implementing the visualization is only part of the huge amount of work; the other part is designing it of course. And in many cases the design effort is probably larger.

For example, I have wanted to write an article about regexes, visualizing NFAs and DFAs. I find that some programmers have trouble with the idea of nondeterminism, which is more of a mathematical thing. A subtitle would be something like "A trie is a special case of a DFA".

This post has some nice diagrams, and you can easily imagine them being interactive and more approachable:

https://swtch.com/~rsc/regexp/regexp1.html

(in fact a few months ago else I posited that a textual summary of these great but dense posts would be useful too)

I can sort of imagine what I want to visualize, but I also think there will also be many false starts. Though maybe a pencil and paper is sufficient. I'm not sure I will get to it, but this polished and smooth gears post got me thinking again! Using something reactive like vue rather than doing it "vanilla JS" is also probably something I should look into as well.

[amitp]

SVG is declarative. You write <circle cx=300 cy=400 r=100 fill=red/> to make a red circle at (300,400) with radius 100. You can then change any of these properties and the system will redraw automatically. Canvas is imperative. You write ctx = canvas.getContext('2d'); ctx.fillStyle = "red"; ctx.beginPath(); ctx.arc(300, 400, 100, 0, 2 * Math.PI); ctx.fill(); and the system will draw that circle. You handle redraw yourself by redrawing everything on that canvas, so you need to keep track of it all. The DOM helpers (React, Vue, etc.) help you with SVG but not with Canvas.

I usually mock visualizations on paper! I'm interested in using WYSIWIG interactive diagram tools like http://aprt.us/ but I never seem to get into them. When I started, making the visualizations was the largest part of the work, but now I've gotten better at it, and making the explanation is now the part that takes the most time.

After paper, I often use SVG to make a non interactive diagram, either by hand, or in inkscape. One of my guiding principles is that the page should be usable without interaction, so the static diagram is a test of that. If it looks promising I can then gradually transform it into an interactive diagram. For example if I had the circle above, and I am using vue.js, I can change it to <circle :cx=x :cy=y r=100 fill=red/> (note the ":" before attributes), and then vue will fill those values in from the object I give it. I can give it {x: 300, y: 400}, and any time I change x or y, it will automatically redraw. I can then hook up x or y to a slider to try out the interactivity. This allows me to start with a static diagram, gradually add interactivity, and then build reusable abstractions that I can apply to multiple diagrams. ObservableHQ and React/JSX allow something similar, with slightly different syntax.

I'd love to see an article about regexes with interactive diagrams. There's a standalone diagram tool https://regexper.com/ and an interactive tutorial https://regexone.com/ but neither is an essay in the style of the Gears article. If you're pursuing this, let me know at [e-mail] and I can send over more resources.

[TeMPOraL]

Observable is amazing! Some time ago, I prototyped half of the 2.0 version of a company's product in it in a scope of a few days; that's how nicely the notebook interface and reactive programming fit together.

I didn't realize Vue can work without a build step; that's actually great. While I so far avoided using any JS on my tiny little blog, I'd really like to do some interactive explanations. I'll check this workflow out. Thanks!

Last but not least, I'm a great fan of your articles! Keep up the great work!

[spankalee]

Observable's templating is directly inspired by lit-html, which also works without a built step, and works great with SVG.

[kragen]

Thanks for sharing these tips! I didn't realize you were trying out ObservableHQ too.

[amitp]

I love ObservableHQ! I think it's a neat model that fixes a lot of things I didn't like about notebooks. I don't use it much though. It makes the simple things easier but it makes the complex things harder. Examples:

1. I often start making a diagram interactive by adding a slider connected to some state. The slider/state is in one "viewof" cell and the diagram is in another cell. But for more polished work I often want to use direct manipulation, where you move something around in the diagram itself. In my usual JS+Vue code it's a small amount of extra work. But in ObservableHQ it seems like a lot of work: the diagram has to modify state defined in a different cell (breaking the simple spreadsheet model in my head), and the diagram is being re-rendered while it has event handlers active.

2. I often start out by making one diagram. This is nice and easy. But then I want to make multiple diagrams with some shared state. In my usual JS+Vue code I can lift the diagram code into a function, pass in a shared object for shared state, and instantiate objects for non-shared state. In ObservableHQ those properties are in cells, and I can't generate multiple cells programatically (afaik).

I also would prefer to host things myself, both because of longevity, but also because I sometimes work offline (e.g. in a park or on a train). And the biggest reason I'm not using ObservableHQ is that I'm so much slower editing text on it than I am in Emacs. So I continue to use ObservableHQ for some simple projects/prototypes but I don't do a lot with it.

[kragen]

Yeah, the centralization aspect is a big turnoff for me. I'd thought about the direct-manipulation issue (the rather crude visualization of skeletonization I linked below benefits a lot from it) but not the multiple-projections problem. Can't you put the entire state in one cell and then create multiple cells that render some reduced dimensionality projection of it? Or do you mean that there's no way to factor out the aggregation of three such projections into a reusable unit?

I recently walked my girlfriend through your explorable on populating landscapes with Perlin noise btw. It was great.

[kragen]

Writing a small program takes much less effort per line than writing a large program, especially if it doesn't have to be maintainable and extensible, especially by other people. http://canonical.org/~kragen/sw/dev3/circlegrid.html only took me three hours of JS hacking, and it's like 200 lines of code. The simplified version of the basic COCOMO model in David A. Wheeler's "SLOCCount" is Person-Months = 2.4 * (KSLOC1.05), which suggests a budget of 0.44 person-months for this code, about two weeks, which is clearly too high; we've all written and debugged a few hundred lines of code in a day on some occasion after getting out of the larval stage. This probably would have been faster if I hadn't been so out of practice with DHTML.

Another thing, though, is that some things are easier than other things. Usually in a programming job you have to do the easy things and also the hard things. This brings down the average. If you're writing a bunch of blog posts for fun, though, you can just publish the ones where good visualizations came out pretty quickly and discard the others that are much more effort for less return.

I feel like visualizations using d3.js are still pretty "hand-written".

[chubot]

That's true, and like everything I'm sure writing JS visualizations gets faster with practice, and you become more familiar with the tools.

Still, after finishing the article, I'm impressed by not just the the number visualizations, but by their legibility and smoothness. I can't in any universe imagine it taking less than a week of full-time work. If you told me it took a month of full-time work I wouldn't be surprised at all.

----

I liked this concept of "interpretive labor" and that's what I'm really getting at:

https://distill.pub/2017/research-debt/

There’s a tradeoff between the energy put into explaining an idea, and the energy needed to understand it. On one extreme, the explainer can painstakingly craft a beautiful explanation, leading their audience to understanding without even realizing it could have been difficult. On the other extreme, the explainer can do the absolute minimum and abandon their audience to struggle.

I've been writing for public consumption for about 3 years and really feel that tradeoff. When I put effort into some writing or explanation, the result is better. People tell me it clicked, etc.

And I would say you can go a lot further than you think in bridging the gap. This article is evidence of that! Lots of people here are saying they wished they had this in high school, etc.

3Blue1Brown's videos are another example of that. I was fairly good at math in high school but if those videos existed then, I would have probably gotten a lot further.

----

This ties in nicely to your Dercuano notes which I have had open in my browser for awhile! There are many interesting topics there. But I also feel like you went to the other extreme and there's a lot of interpretive labor involved in reading them :)

Why did you decide to polish and publish the "memory models" post? I'd like to see more posts like that. (I could send you the notes that jumped out at me if interested.) Whenever I've taken the time to polish some writing, I've never regretted it. For some reason it feels annoying to start (probably because I have to clear my brain of other things), but when I'm doing it it's fun, and when I'm done, it's worthwhile.

[kragen]

> Still, after finishing the article, I'm impressed by not just the the number visualizations, but by their legibility and smoothness

I think the author probably had to try a lot of things before hitting on such a well-balanced set of visualizations: not too much detail, not too little, not too many degrees of freedom, not too few, usable on hand computers, usable on laptops. But I think that if you gave someone the web page to look at and asked them to make similar visualizations, it would probably take them a day or two.

> I would say you can go a lot further than you think in bridging the gap.

You might be right, but I'm kind of cynical about it. I think it's easy for people to fool themselves into thinking they understand things after seeing a good explanation. Nova on PBS resulted not in rapidly advancing physics from the generation of kids who grew up watching it, but instead Deepak Chopra and The Secret — the fruit of the illusion of fluency without the corresponding skill.

Worse, the educational system is in many ways optimized to promote the illusion of fluency. You have textbooks of the variety Feynman complained about, with the wakalixes. You have semester-long self-contained courses, discouraging spaced learning. You have pre-announced big-bang exam dates for those courses, so students game the system by cramming to get better grades. In many cases you have multiple-choice tests to make grading easier, so guessing the teacher's password is the highest-return gaming strategy, even if often insufficient by itself. The whole educational experience is compressed into a degree program of four years or thereabouts, further discouraging spaced learning. Students rarely attend any classes they aren't getting credit for, even though this is technically permitted at most universities I've visited. It's a commonplace observation that students forget almost everything they "learn" within a few years. So the idea students get of learning is very different from what learning is; schooling systematically distorts their ability to evaluate whether they are learning or not. (Paul Graham has explored this theme from a different angle in http://www.paulgraham.com/lesson.html as well.)

How would we distinguish a universe in which this beautiful post on gears successfully transmitted learning to its readers from a universe in which it only transmitted the illusion of learning to them? In both universes the post is popular. In both universes people say things like "I wish I had this in high school!" But in one universe people are able to do things they couldn't do before; perhaps design gear trains for 3-D printing that show up in Hackaday projects, or perhaps laser-cut unusual gear-tooth profiles with different pressure angles or different depthing/ripple tradeoffs or something. But that could take a while, and it might be really hard to trace back. Is there a lower-latency indicator we could observe, even if it's something hackable like school exams?

I don't want to sound like I'm above all this. I feel like I have new gear knowledge from reading this post, despite having read hundreds of pages about gears and watched hours of gears videos previously; it happens to be something I can state, namely that the pressure angle always needs to point at the point of tangency of the pitch circles. But I'm not yet sure if that's really true or what happens if it's false (you can definitely design gears for which it is false). I'd need to struggle with the problem for a while in order to really understand it.

(A different learning effect would be that people learn that posts like this are very popular and make more of them, having been inspired by this one. I'm pretty sure that will happen whether or not people learned things about gears from this post.)

> This ties in nicely to your Dercuano notes which I have had open in my browser for awhile! There are many interesting topics there. But I also feel like you went to the other extreme and there's a lot of interpretive labor involved in reading them :)

Thanks, that's flattering! I'm glad you're enjoying it. Are you using the HTML version or reading the raw Markdown?

> Why did you decide to polish and publish the "memory models" post?

I don't know. I think it was just random chance. I enjoy doing that kind of thing but I don't do it very often. Even memory-models is pretty unfinished and has some parts that are pretty raggedy.

> I could send you the notes that jumped out at me if interested.

Sure, that would be wonderful! If you wanted to commit to commenting on further drafts within some timeframe, that would probably help motivate me to work further on them.

----

I want to be careful to disclaim a particular interpretation here. In "Real Programmers Don't Program in Pascal" it says, "If it was hard to write, it should be hard to read." I don't believe that. I don't believe that if something was hard to discover, it should be hard to learn. I think we should make learning as easy as possible, and I agree with the post you linked that this benefits from improved explanations, and that improving those explanations takes a great deal of work.

But I have been burned many times even by my own illusion of understanding, let alone those of Freud and Chopra, to the point that I am wary of it. It is well that we remember, as Euclides said, that there is no royal road to geometry; a good tutor can save a student from remaining in error and from failing to notice the importance of something they could learn, but ultimately the student is the one that must do the hard work of constructing the knowledge within their own mind.

[chubot]

I'm with you that there can be an illusion of fluency, but I don't think that casts any doubt on the value of the article.

Most people will read it and say "fun" (including me, since I don't work with gears) and maybe 5-10% will go on to do something else with it, but that's working as intended.

I agree you can't really say you know something without testing the knowledge. You need to do more work to test whether you know it or not, but having the concepts and words at hand is a prerequisite for that. I'm certain if I were to actually work with gears I would come back to this article. (I would probably also learn where it falls short in practice, but I would learn that about any resource AFAICT.)

-----

Testing your knowledge is one reason my brain flipped from math to programming over 20 years. Math is hard to test but programming is easy to test.

However, one thing I found surprising is that publishing tests your knowledge, but writing does not. I would categorize Dercuano as writing but not publishing at the moment.

It tests it in exactly this way:

https://news.ycombinator.com/item?id=22033792

Blogging is a public act. Anyone can read this. When I write a blog post, I imagine my supervisor, a respected colleague, or a future employer reading my explanation. These imagined readers force me to ask myself honestly if I understand what I am writing.

I found the same thing while writing my posts on https://www.oilshell.org. If someone digs this up in 5 years, is it going to look dumb? And of course simply not knowing something is not dumb, but pretending you know something you don't is dumb (likewise with promising something you can't deliver, which I've been careful not to do).

I can see there are a lot of great ideas in your notes, but I have 100% certainty that polishing those ideas will lead to a better understanding, more ideas, and forcing a focus. The writing alone alone doesn't cut it. (I know because I also have 3382 personal wiki pages with notes and URLs accumulated over 15+ years with overlapping research!)

I would definitely comment on drafts. I was paid to review the second edition of Effective Python last year, and am also reviewing a book for a friend currently, so I have some experience with that. I'm most interested in the posts on parsing, automata, languages, compilers, and (parallel) algorithms -- I have less experience with graphics. I didn't know you were working on HAMMER -- that line of research is also in my wiki pages and I have several thoughts about it. I'll send you a mail with the ones that jumped out at me.

[kragen]

I've responded to your email! Let me know if it got spam-filtered.

[1wheel]

> another edit: I also believe one reason that this visualization is good is because there's no build process in the JS. The author clearly just edits the code and refreshes the browser. You need that kind of fast edit-view loop to make good visualizations.

You can go a step further and reload code without refreshing the page. This makes it even easier to make the sort of fiddly tweaks that visualizations require while avoid change blindness.

https://roadtolarissa.com/hot-reload

https://github.com/1wheel/hot-server

[kragen]

In some cases I've exposed sliders to enable a 60Hz feedback loop on those fiddly tweaks. In other cases Firebug or its modern successors already expose enough.

[32gbsd]

This. The time and effort it takes to put such a visualization together for a target audience that might possibly just gloss over the content is immense. Its like making a multi-million dollar movie about people who have no money just to show thier situation. Instead of giving them the money directly. And on top of that you need all the requirements to actually be able to view it in the latest browser.