[Verdex_2]

If you're an explorer and you live in a small valley, then you will slowly begin to believe yourself a better explorer as time goes on. But one day you will get to the edge of the valley and look out onto a vast landscape and believe that you have become a worse explorer.

Checkout the volume of N-balls [1]. As the number of dimensions in a "sphere" become higher the "volume" of the sphere becomes smaller when you compare it to a hypercube of the same dimensional count. This is another way to saying that as the number of dimensions increases the closer the "halfway" point of the volume of the N-ball gets to the edge of the N-ball.

Programming (and software engineering by extension) involves a lot of orthogonal issues. We can imagine this so be a really high dimensional space. My theory is that whenever you're in situation that can be represented by a high dimensional space you will learn a bit of knowledge and think you know something. Then you'll learn a bit more and discover there's more to learn. The more you learn the farther the horizon moves away. It will feel like you're getting worse. Just like finding out you live in a valley.

What's really happening though is that your increase in skill is allowing you to realize how small your world was all along. Keep learning and trying and you'll get better. It will however, take a while.

[1] - https://en.wikipedia.org/wiki/Volume_of_an_n-ball#Low_dimens...

[jasode]

>We can imagine this so be a really high dimensional space. My theory is that whenever you're in situation that can be represented by a high dimensional space you will learn a bit of knowledge and think you know something. Then you'll learn a bit more and discover there's more to learn. The more you learn the farther the horizon moves away. It will feel like you're getting worse.

That is all true but I don't think the n-dimensions analogy applies to the particular essay the author wrote.

He didn't look back on his accumulation of learning and suddenly notice that it was smaller part of a larger knowledge space. (E.g., programmer stumbles across "monads" and suddenly he realizes that's just the tip of the iceberg in a never-ending horizon of category theory.)

Instead, his essay is lamenting the all-too-common "joy vs drudgery" in projects. It's a similar sentiment that aspiring writers who don't finish their "great novel" have. He then grades himself on his lack of motivation (words like "lazy") to stay on top of programming tasks that are not pure intellectual fun. On his idiosyncratic standard of "programming competence", he wonders if he is a "worse programmer".

Of course, he can choose whatever "success metrics" he wishes to apply to his personal projects but some of us might think that staying on top of documentation (that nobody will read) or adding features to Lily (that a large population of programmers haven't adopted) can bring undue self-criticism.

If we are to plow ahead into boring unpleasant tasks, there usually has to be a compelling motivation -- e.g. we think the personal project is the basis of a great business startup... or the code is part of a PhD paper due next Friday, etc.

[TheOtherHobbes]

Maybe there's a simpler explanation: FOSS projects can easily turn into unpaid work, and devs tend to burn out.

Projects that start as fun programming exercises eventually take on all the aspects of a full-time job with "customer" support, project planning, maintenance, bug fixing, and so on.

It can become very draining, and if the project isn't unusually popular or high profile and/or there's no other dev support and/or the team isn't solid and mutually supportive it can be understandably difficult to stay motivated.

One of the insidious things about FOSS is the way that devs can make themselves feel as if they ought to be doing all of these things, sometimes in addition to a full-time job.

The problem isn't that they're not up to the task, it's that the expectation isn't very realistic.

[devoply]

Then don't do it. If no one is paying you, you owe no one anything. You have no obligation. Walk away. Our society and economy have lots and lots of money. If the way system works can't justify paying you for your work, then you should not be doing that work. At least that's the way I see it. At this point FOSS is big business, if you are not getting paid, then you should not be bothering with it.

Get someone with money to pay you for it or don't do it. The end.

[westoncb]

I think I get part of the N-ball analogy, but I'm trying to piece together the rest...

So it seems like one problem/feature with high dimensional spaces is that you have more regions which are orthogonal to one other. So, it's possible to have quite a bit of motion, which to a human mind might give the impression that one is doing quite a lot—but then you might discover a new way of turning around and find that space extends in another direction, too, at which point you realize you covered a smaller fraction of the available space than you previously believed.

What I don't get is how the volume of an N-ball with fixed radius going to zero as N goes to infinity plays into the analogy. When you say, "when you compare it to a hypercube of the same dimensional count," I don't get what in the programming analogy would link to the N-ball rather than the high dimensional hypercube. I also get the algebraic justification for this property of N-balls, but I still have no real geometric intuition for why it would be the case.

Could you clarify what's meant by, "as the number of dimensions increases the closer the 'halfway' point of the volume of the N-ball gets to the edge of the N-ball"? Not sure what's meant by 'halfway' point here, or edge of the N-ball. My guess is that you're talking about distribution of volume, like if you have a tapered cup that gets small at the bottom, most of the volume will be at the top.

[Verdex_2]

Checkout the generic formula for any number of dimensions [1]. The function in the denominator is a factorial (except extended to work on decimals instead of just integers). The functions in the numerator are exponents.

Factorials grow much faster than exponents, so with an N-Ball we expect the "volume" to be small if we were to compare it to the corresponding hypercube when the dimension and when the diameter / length are all equal. After all we're comparing approximately diameter * R ^ D / D! and length * R ^ D. So if you put an N-Ball into a hypercube with length = diameter, what you'll discover is that the N-Ball is tiny when compared to the hypercube. Which means we have the weird and counter intuitive conclusion that all the "volume" of a hypercube is in the corners. Try calculating the volumes yourself using a high dimension like 100. Even better plug the whole thing into a script so you can generate a table and see the effect get progressively worse as you increase the dimension count.

There is also a similar effect on the N-Ball itself. Imagine drawing a smaller circle inside of a circle where their centers are the same such that the inner circle contains exactly half of the area of the outer circle. Now imagine doing this in a sphere. Now a 4D sphere. Now a 5D sphere. What you'll find is that as you increase the dimension count, the radius of the inner circle will get longer and longer. Which means that half of the "volume" of an N-Ball slowly moves closer and closer to the outer edge of the N-Ball as you increase dimensions.

Here's the punch line. Imagine that the N-Ball represents all there is to know about a topic. You begin in the center of the N-Ball and in order to learn more you have to walk to the outer edge of the N-Ball. Also imagine that you're only able to see what's right in front of your face. In the beginning at the center there is very little volume to look at. It will appear to be easy to understand what's going on. But as you walk to the edge of the N-Ball, the amount of volume that you're able to see increases. When you get right to the edge of a very high dimension N-Ball you will notice that you only know half of what is available to know.

Simple topics are N-Balls with low dimensions. It's easy to "walk" your way to knowing half of everything there is to know. Complex topics are N-Balls with high dimensions. Programming (and by extension software engineering) is really high dimensional. The more you learn the more you notice there is to learn. When you walk to the edge of what seems possible you find yourself swimming in a sea that dwarfs you. But in the beginning it looks like there isn't really all that much going on.

[1] - https://en.wikipedia.org/wiki/Volume_of_an_n-ball#The_volume

[westoncb]

Cool—that makes sense! Thanks.

I've been thinking a little more about how the space of software engineering knowledge might be structured though, and one thing I'm betting is that it wouldn't be an N-ball, but instead some complex sponge looking thing (because the extent of software engineering knowledge isn't equidistant in all directions/aspects). Ordinarily I'd think the N-ball would be fine to use as an idealization, but since the hypercube behaves so differently, I bet the idealization would end up changing the character of the situation.

I'm also trying to consider what the analogy of a plane would be in software engineering knowledge. Let's say it's on two dimensions: X and Y; and let's say X is knowledge relating to text editor usage, and Y is knowledge about unit testing. It seems like the XY plane is mostly full of empty space, since most facts (single points) from either field are unrelated to the other. So, if we have a pure text editor fact X=5,Y=0 —what legitimate fact might exist at X=5,Y=1? It would have to be some single, constant piece of text editor knowledge which meaningfully combines with a unit testing fact. There may be some points in this plane on the line X=5 which are valid, but since all unit testing facts don't have a meaningful connection to text editing, there will be gaps.

Additionally, it seems like the way you'd navigate the space would be different than walking a single contiguous path. Some of our knowledge is map-like and can help give at least sketchy knowledge of the boundaries of the field without walking to them step by step. And some facts are dependent on knowing other first so that you have 'pass through' in a particular order, but a lot of the facts about a single subject are still unordered, so you would be doing a lot of teleporting.

Just for amusement, here's another idea for an alternate way of spatially structuring knowledge: each general principle is some high dimensional object of its own; each axis corresponds to a parameter to the general principle. So you have a general structures like: ProgrammingLanguage(syntax, semantics, runtime, standard_library) —each of the parameters is a different axis, and you always get a meaningful ProgrammingLanguage back if you change to a different position on some axis (seems like this structure is recursive though since each of the parameters could be a general principle itself, so I guess these objects are nested?).

When you learn things you are jumping around between these non-intersecting objects (which are even in differently structured spaces) without realizing, until you inductively discover these general principles/categories. Additionally, the speed with which you move around inside an object increases dramatically once you acquire knowledge of its general structure (since learning new facts which are just different instantiations of already known general facts is easier for us).

I wonder if there is some kind of 'meta-space' that would be useful for describing the relationships between all these objects which exist in spaces of different dimensions. Also not sure about the geometric significance of the 'general principle objects' being recursive...

Edit: seems like the recursive/nested 'sub-objects' would just be lower dimensional slices of the whole higher dimensional 'general principle object'...

[Verdex_2]

Yeah, I like the way you're going. I haven't spent much time thinking about how to fit in software engineering, but I did try to think about how to structure programming issues (which is probably the first step if you want to eventually get to a software engineering).

https://www.sep.com/sep-blog/2017/04/25/objective-code-quali...

It's missing a conclusion / wrapping it all up (which hopefully I can get to soonish), but is otherwise mostly done. [I should probably augment it with motivations and examples.] I'm not sure that this is the definitive answer, but more like this is my attempt to structure some of the things I've been encountering.

EDIT: Sorry about the length. I'm not sure there's a good way to shorten these sorts of ideas.

[westoncb]

Interesting! I'm on "Problem Analysis: High Dimensional Spaces" so far. I'll probably just leave a comment there if I end up with something to say since we'll probably drift too far off topic if continuing here ;)

Edit: I finished but couldn't find a way of leaving a comment on your blog. I thought it was an interesting read—but yep, definitely needs that conclusion! Hard to say much without that. Is your goal to use the ideas to create a new metric for how difficult a code base is to work with?

[debaserab2]

Isn't this effectively the same as the Dunning Kruger effect (as it applies to this situation)?

https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

[Verdex_2]

More or less. I'm suggesting a mechanism for why something like the Dunning Kruger effect might occur and where we are likely to see it occurring.

[debaserab2]

That's a very interesting way of explaining it from a perspective I hadn't considered.

[anonymoushn]

In the paper, ability is positively correlated with perceived ability. So we would need some other effect to explain people saying they got worse over time, or people's perceived ability decreasing over a time period in which their ability improved.

[lacampbell]

To use your analogy, when I start exploring outside the valley, I think to myself - "wow, a year ago I only knew this valley. Now I'm getting to know this new area! I've come a long way and I feel great I've made progress!".

I tend to focus on the fact I am constantly expanding my knowledge, instead of worrying about all the stuff I don't know yet.

I still consider myself a generalist, however. I can't say I'm an expert in any one area. Maybe that's why I feel the way I do.

[mpfundstein]

great way of thinking. its all about the journey

[Philomath]

Wise words!

This certainly applies to programming but I've felt it too in my Aerospace career. I used to feel like the smartest guy, I always knew how to do things or learnt them really fast, and I couldn't imagine where the horizon was. Now, the more I see, the further I feel the horizon is, and it's pretty frustrating as it makes me feel less intelligent.

Your comparison helped me to see things a bit clearer and to be a bit more motivated.

[hinkley]

It's not always that way though. We often suffer as if we have unique problems, but through house repairs (got a nice long chat with a couple GCs) and extracurriculars, I find that a lot of my problems look a good bit like other people's problems. I think the surface area thing is overblown.

There are always stories about innovators who simply apply established solutions from problem space A to problem space B and everyone is amazed. The surface area problem certainly exists for the entire domain of human knowledge, but domains pretty much exist to wrangle problems that are manageable in the first place, even if it makes you myopic in the process.

I get frustrated in interviews because everyone is just so convinced that the latest utility module is so new and unique and it's really just a retread of five other things. It might have some interesting semantics that change how you use it, but even that isn't always the case.

Plus ça change, plus c'est la même chose.

I've just started telling people that 'how' ages faster than 'why'.

[edem]

It is more like traversing a tree. The more you learn (the deeper you go) the more branches you can explore which leads to the conclusion that the more you know the more you realize how much you still don't know.

[disposablename]

TL;DR "The more you learn, the less you know"