[vsnf]

Kind of tangential, but who are these people who are so comfortable with disassembling a high level language binary, reading assembly, and then making statements about branch prediction and other such low level esoterica? I've only ever meet people like that maybe two or thee times in my career, and yet it seems like every other blog post I read in certain language circles everyone is some kind of ASM and Reverse Engineering expert.

[keyle]

They're a dying breed. We're forgetting how to look under the hood and understand "why something works".

Case in point, I'm slowly being replaced by Salesforce muppets for all my projects at work. They're little code monkeys with amazon ebook type knowledge, projects cost 20x more and I look like the mad scientist for speaking the truth. The products are worse in every possible metrics, I'm not crazy. The politics at play is the reason why I'm losing ground, not logic.

Cabinet designers are being replaced by Ikea flat pack artists in the software world. All we can do is stand by and watch.

And in regards to this blog, when Medium eventually go, that knowledge will go too. Blogs have died, personal websites as well, and their ability to be found in Google is almost non-existent.

Sorry I don't have anything more positive to add, except maybe that they're still there, slowly being alienated by the modern tech world!

[mcv]

> We're forgetting how to look under the hood and understand "why something works".

Partly because that's often not what we're supposed to do; the stuff under the hood "just works" and we're meant to use it to write features, not worry about optimising the stuff that happens under the hood.

And partly it's because the stuff under the hood is increasingly weird and bizarre. Branch prediction is weird, and I still don't understand why that extra print statement changes the branch prediction. Why does it predict `v > maxV` is true when the alternative is to print something, but it doesn't predict that when the alternative is to do nothing?

Is it because printing is expensive, and therefore the branch predictor is going to strongly prefer avoiding that? It's weird that we'd basically have to deceive our code into compiling into a more performant form.

I don't want to have to second guess the compiler.

[lordlimecat]

In my world "just works" means "we blew away all of the security controls and best practice to get this thing hobbling across the finish line."

I see COTS products using ldap memberof queries without LDAP_MATCHING_RULE_IN_CHAIN and stating definitively in their documentation that nested groups are bad (despite decades of best practice).

I see product documentation recommending authenticating against LDAP instead of kerberos, despite the underlying libraries having full kerberos support.

I see sslverify: no, and flags to ignore SSH TOFU warnings, and recommendations to avoid SSH gssapi-keyex (WHY?????), and security approached by buying ever more products creating ever more complexity.

Yes, things "just work" in a horrible, 'youre stuck with your vendors forever' sort of way that results in lengthy outages every 6 months due to mounting, intractable technical debt. But things don't have to be this way, you just need people who are willing to ask "why" or "is that necessary" or "can it be better".

[michaelmior]

> I don't want to have to second guess the compiler.

No one wants to have to second guess the compiler. And it's almost never the compiler, until it is. As a grad student, I remember someone else in our lab spending a long time on a particular issue until he tracked it down to a not yet reported bug in gcc.

99% of the time you can rely on the compiler (or the language runtime, or popular library X), but we also need to be able to identify the rare cases where the lower levels of the stack are broken and how to either fix or work around them.

[chii]

> not worry about optimising the stuff that happens under the hood.

no one is asking you to do this optimization all the time. But you need to know what and how, if you were to be tasked. Then as you get more senior, you will have to also be able to judge whether it is possible, or worth the effort to devote to the optimization, should a scenario arises.

> second guess the compiler.

you aint second guessing it, you're examining the output, and understanding it. Then potentially recognizing when or if the compiler is outputting bad stuff. That's not second guessing.

[pjc50]

> I still don't understand why that extra print statement changes the branch prediction.

The article is under-informing you: they should have shown the full disassembly for both loops, not just the bit in question. I suspect the explanation as given is slightly wrong, but there's no way to disprove it without replicating the setup myself.

[gv83]

> We're forgetting how to look under the hood and understand "why something works".

But under the hood there is a hood. And under that hood there is another hood. And under that hood there is a brand new car you don't know how to open the hood, and so on.

I cannot devote my life to know everything or I won't be able to provide for my family.

[keyle]

If you don't understand "why something works" then you will certainly not be able to answer "why it doesn't work".

Every high level library is an abstraction and abstractions are leaky.

I'm not saying you have to be able to write assembly by hand. I'm saying you should understand why your code runs 10x slower in docker.

[rajamaka]

I'm interested in what's under the hood but after working full time scrubbing the bonnet, my motivation is mostly gone.

[flohofwoe]

You don't need to know every little detail, but having at least a basic understanding how the machinery works (and keeping that knowledge somewhat uptodate), and also being able to verify that the machine actually does what you expect it to do is useful for all programmers, because it lets you write code that works in harmony with the hardware instead of fighting it.

(besides, learning to read a bit of assembly code never hurt anyone)

[_joel]

"It's hoods, all the way down..."

[MrYellowP]

No. Flat out wrong. There is only a small amount of fundamental understanding that needs to be acquired to actually know what your code is actually doing.

Knowing how things work fundamentally is what matters the most, as everything else can be inferred and deduced from that. This is universally true. When you know how something works, you can build up on that.

There's no value for a programmer below how processors work with registers, caches and bus accesses. The most fundamental interface between the programmer and the computer are the registers, caches and interrupts, which make everything else happen.

Understanding how these are being used practically and properly covers a LOT of ground to build up upon.

On the other side you have the people who are fully dependent on compilers to do their job for them. They don't write code for the machine, they write code for the tool.

They don't actually know what they're doing, they have no idea how it works and they believe that the compiler optimizes it all properly anyway, as if it's magic, with no fucking clue what the fuck they're actually doing, while patting themselves on the back for living in dependency.

Why do programmers need garbage collection? Because people have no idea how not to need it. Why are pointers considered bad? Because some assholes decided that telling everyone they're too complex to use was a smart idea instead of teaching them properly, which, of course, it fucking wasn't.

If people would actually know how to do proper memory management, we'd had far less security issues from the get go instead of requiring patches for software that was written in a stupid way, ignorant of memory mapping, pages, etc ... you know, fundamental knowledge and understanding of what's actually happening.

It is considered brilliant to use mmap for faster, more memory efficient file access ... which is absolutely ridiculous, because it exposes how little programmers actually know about the fundamentals!

Using mmap for sharing information between processes and reading and writing files has been my go-to since forever, simply because every other way is literally doing it wrong.

Anyone saying that not every programmer needs to know all of this is missing the point. Of course you don't need to, but the status quo is built upon exactly that line of thinking.

Are you better off being able to, at least moderately, repair your car yourself, or are you better off depending on someone else to do it? The answer is obvious, and the same applies to literally everything.

Less dependency should always be the top priority, because less dependency leads to better understanding and increased freedom ... but we're living in a world that's doing the exact opposite.

Gonna stop writing now, because there's no end to this.

[gv83]

I never had the chance to work at the lower levels you seem to have worked with; in any case I don't disagree in principle with your points (and understand some part of the fundamental understanding you mention), but less dependency leads to wheel reinventing which is completely unsustainable in the long run, specially if you need to maximize the value you bring to the table. It has been like this for every field; how can we call this, industrialization?

[Chris2048]

> Cabinet designers are being replaced by Ikea flat pack artists in the software world

That analogy doesn't really work if the new projects "cost 20x more" upfront, unless you mean long-term associated costs.

[Waterluvian]

It’s what evolution looks like.

My grouchy mother in law also laments that they teach typing in school and not handwriting.

Lamentable? I guess. But it’s not realistic to hope people just learn more every generation.

Despite the tradition of “kids these days with their frameworks and libraries!” complaining, you want to see this evolution.

[dikei]

> They're a dying breed. We're forgetting how to look under the hood and understand "why something works".

Well, layering abstraction is the most effective way to deal with complexity. Our brain is too limited to know everything.

The people, who understand "under the hood", probably won't know much about what's "above the hood", like writing a website or mobile app. Therefore, we need experts on every layers.

[buro9]

This feels semi-normal to me... just have the curiosity to ask "why?" and the bias-to-action to move to "I'm going to find out".

You encounter far far more dead-ends than anyone ever says, and every unsolved mystery is a mild nerd snipe, an open case, that years from now you'll see someone else explain something you realise it answers that question from years prior.

For me, the hard bit is not over-indexing on this... you learn things, but biasing too much for them is a sure fire way to over-engineer or increase complexity to the point where something is now worse for you knowing something. But once in a while that tiny thing you learned years before is a 20% savings across the board with associated performance increase and everyone wondering how on Earth you could possibly have made those jumps.

Also related... incidents. "Why" and "I'm going to find out" is the best way these things don't recur in future. A high degree of observation and understanding is a happy engineer life as it can improve what can often be the most stressful parts of the work (on-call, etc).

That XKCD comic about everyone learning something for the first time factors too... there is stuff you know that others do not, share it.

[djtango]

I remember someone saying the difference between Physics and Computer Science is that in CS we are the masters of the universe - there are no laws of Physics that bind us.

For me that means that in our world of computers there is infinite curiosities to discover. (Not that the same isn't true for the natural world too)

[matwood]

> "Why" and "I'm going to find out"

And it drives me nuts dealing with people who don't think this way. I'm not a jerk about it, but my personality is 'lets what all we can figure out the why'.

[totetsu]

https://imgs.xkcd.com/comics/ten_thousand.png for those who haven't heard of it.

[Chris2048]

I always felt this one was a bit smug when applied to occupations (which it often is). People paid to do a job should know the basics of that occupation.

[teddyh]

Canonical link: <https://xkcd.com/1053/>

[ColinWright]

My first machine was a TRS-80. My first large program was a compiler from TRS-80 BASIC to Z80. I subsequently disassembled the ROM in the machine to figure out how things worked.

These skills stay with you, and if you read articles like this then you can keep broadly up-to-date with the insanity that is current CPUs. Things like pipe-lines, branch prediction, and different levels of cacheing are optimisations that you can acquire as you go.

If you're an auto-didact web developer then you never have the opportunity to learn these skills, or the need to do so.

I know a lot of people who are comfortable with doing this, but in my case it's a generational thing. If you want to do it then you can. It's not hard, it's just a different skill from those you already have, though sufficiently related that you wouldn't be starting from scratch.

But starting with modern CPUs can be hard. Learning the basics from older, simpler CPUs can help. Doing some kind of embedded programming might be the way to get started, or working on an emulator.

As always, YMMV.

[matwood]

It's odd to me to see people who don't want to understand the next level from any current level. A front end person who doesn't care how the backend works, or a backend person who doesn't care how the database works, etc... Obviously there is only so much time in the day, but being comfortable digging down to whatever level is necessary to debug something is a valuable skill.

Not sure if it's still asked, but reminds me of a class interview question. "When a user presses a button on a web page, describe in as much detail as possible what happens."

[pohl]

A compiler from what to Z80? (I think there’s a word missing in what you wrote.) I started on a Model I and I’m curious about your compiler. I miss some aspects of those old machines. It was great, as a learner, that the ROM and a DOS like L-DOS was small enough to fit in your head. It can be intimidating how large everything is today.

[ColinWright]

Now edited ... thank you.

The compiler was from a limited subset of TRS-80 BASIC to Z80. It was written in its own subset, so it could compile itself.

I had 16K of RAM, and that had to fit the BASIC source, the compiled version, and then that compiled version compiled the BASIC source to another compiled version in a different location, which could then be saved on tape.

Interesting times, and so foreign to today's experiences that it's hard for people to follow the details. I could write it up, but I'm pretty sure no one would derive any value from reading it. I sometimes wonder if the hand-written syntax flow charts and first draft are still somewhere in my piles of undiscarded papers. I doubt it ... it as 1979 when I was doing it, and I've moved continents since then.

[pohl]

That sounds like a valuable exercise. I wrote a word processor. There was an issue of 80 Micro that had a simple buffer in assembly (I think they called it Scripy) and I wrote some BASIC around it for various other functions, so that I could keep a journal and write essays for school. A compiler sounds ambitious in comparison.

[deaddodo]

I think you'll find that the deeper you go into "traditional" computer scientists, the more you'll find the problem-solvers, hackers and tinkerers that post these types of blogs. Especially in odd cases where a random print statement doubles your profiled performance.

That being said, of all the people at all of the tech companies I've worked at, maybe ~5% of them had this sort of mentality and drive to execute on it.

[distcs]

I don't think I'd call "branch prediction" as "low level esoterica". It is a basic fact about how CPUs are implemented since many decades now. I learnt these things in my university coursework. Any module on CPU or computer system architecture is going to teach you all this stuff. But I'm sure you could learn these things from books on this topic too.

[mcv]

I didn't, and frankly, half of the articles I read about it make me think branch prediction is a bug. I mean, I know it's meant to improve performance, which is great, but it has to make assumptions about what's going to happen before it knows it, and those assumptions are going to be wrong. How wrong? How can we con it into making better assumptions? Suddenly programming becomes about second guessing the compiler.

And remember Spectre and Meltdown? Security vulnerabilities caused by branch prediction. If I recall correctly, the pipeline was executing code it wasn't meant to execute because it's executing it before it knows the result of the check that decides if it has to execute it.

Programming is a lot easier if the actual control flow is as linear as I'm writing it.

My broad takeaway of the whole ordeal is that I'm basically avoiding if-statements these days. I feel like I can't trust them anymore.

[layer8]

Without branch prediction and speculative execution, performance would always be as bad as with a misprediction. So every correct prediction is a win, and that’s why it’s being done.

> My broad takeaway of the whole ordeal is that I'm basically avoiding if-statements these days.

That seems like an overreaction. For most applications, cache locality and using the right algorithms is more important, if performance is an issue at all.

[Galanwe]

> it has to make assumptions about what's going to happen before it knows it, and those assumptions are going to be wrong. How wrong

Speculative/optimistic techniques are not limited to branch prediction, you encounter various forms of it pretty much everywhere, without knowing it.

* Your hard drives have a read ahead buffer, fetching in advance future data, just because most reads are immediately followed by an other one for the data just after.

* Your CPU instructions are pre fetched, because most instructions are _not_ jumps, so you will 99% of the time just execute the instruction right after it.

* Instruction reordering where your compiler will decide that future instructions not affected by previous instructions could be run in advance.

* Overall any kind of caching is some form of optimistic technique. You are preparing future results.

If you think about it, optimistic/speculative techniques are ubiquitous, and used even at _high_ abstraction levels.

The famous python mantra of "better ask for forgiveness than permission" embodies that spirit. It encourages a coding style of "try: do() except: nope()" rather than "if check: do()".

Standing "against optimistic/speculative techniques" is standing against transactions, rollbacks, caches. It's just not a viable line of thinking IMHO.

[superb_dev]

You shouldn’t be thinking about the branch predictor this much. The predictor is pretty good at guessing, unless you need to eek every last shred of performance you can ignore it

[flohofwoe]

> My broad takeaway of the whole ordeal is that I'm basically avoiding if-statements these days. I feel like I can't trust them anymore.

Such broad "optimization rules" usually don't make much sense on any compiler created in the last 25 years. An optimizing compiler will analyse and optimize your program's control flow at a very high level, not just naively insert a conditional branch for each if (and that's also true in "low level" languages like C)

If you actually care about such things, you really need to look at the compiler output and incrementally tweak your code (but then other compilers or even compile options might destroy such tweaks again), there hardly is a single optimal source code solution across CPU architectures or even just different CPU models of the same architecture (just don't do any obviously stupid things which make the life of the compiler harder than need be)

[lordlimecat]

....And then feel OK resorting to ChatGPT for the explanation.

Seriously that threw me, and maybe it makes sense in this context but it seems strange for someone with such an apparent depth of technical knowledge leaning on an LLM for anything.

[layer8]

It seems they didn’t want to bother coming up with their own explanation, but then why not just link to https://en.wikipedia.org/wiki/Branch_predictor?

[xdavidliu]

pasting LLM answers is sort of a tongue-in-cheek rhetorical flourish these days

[2-718-281-828]

> but who are these people who are so comfortable with disassembling a high level language binary, reading assembly, and then making statements about branch prediction and other such low level esoterica?

i don't see any assembly here. the analysis is done by using a profiler. a very common tool available for most programming languages.

https://timotijhof.net/wp-content/uploads/2020_profiling_fig...

[kuroguro]

There's some bias as topics like these are often on top of HN. Extrapolate the 2 or 3 people you've met to all the programmers in the world - that's how you get some amazing in depth blog posts every other week.

[Smaug123]

There are many ways you can get to this point (e.g. I just sort of picked this kind of thing up), but an example of a course which is designed precisely to give you these skills is Casey Muratori's "Performance-Aware Programming".

[sailorganymede]

These people genuinely care about their craft to the point they ask: what happened. They then look into it.

People who tend to care tend to talk about what they care about. So, you wind up getting a lot of blogs who sort of self select themselves to do this kind of stuff. And everyone feeds off that energy and gets better :)

[rofo1]

There are many engineers that can do this, it's just that writing javascript pays 3 times or more so the focus goes there :)

If you visit #c / #asm on any popular IRC network, you'll find a lot of skilled people that can do this routinely.

[crvdgc]

This topic among others was covered by a Computer Architecture class back in my university. That class is mandatory for all CS students. Though I have to add that it's never been brought up after I started to work.

[amirxyz]

I think it depends on what niche you're in. I'm in the storage business and while not everyone, but quite a few of the people in my group would not be afraid of running a disaseembler, or talk about branch prediction, or unaligned integer access. `likely` and `unlikely` are very common in the code, and so on. AFAIK linux kernel people are very similar, and I assume a few other niches of system programming as well.

[BiteCode_dev]

The HN bubble doesn't have only disadvantages.

[secondcoming]

It’s not all that uncommon outside of web stuff

[layer8]

People who maintain a blog and whose blog posts are widely circulated are already highly selected. Conversely, people not comfortable with that stuff will also tend to be less comfortable writing a technical blog.

Also, if you read the other subthreads, there are a number of points to be criticized about this writeup.