[ibains]

We want software engineers who understand compilers and query optimizers and systems.

This requires good theoretical foundations with solid engineering skills.

It is extremely hard to get these. Most CS majors don’t know much CS - top grads from top US schools undergrads are clueless usually.

That’s why you get projects like Hadoop- a large scale systems project written in Java instead of systems languages - because the engineers are poor quality in top companies as well.

[likpok]

I don't know that your example is a fair one. I'm not a huge fan of java, but I have run it in anger (and specifically abused the crap out of hadoop).

I do not think that being written in C++ would have solved any of the problems we had with it. The biggest issue was the thread replication problem on writes (each write is replicated 3x, and each operation is done synchronously with a thread. Lots of parallel writes will mean tens of thousands of threads). Async was known and possible at the time, and I think even fixed upstream.

Java doesn't share machines well (at the time it never returned memory to the operating system, this is fixed now), but your hadoop machines shouldn't be shared. If you want to run a monolithic server, java will do just fine. The garbage collector will work fine unless you do dumb things (which I've also done!)

[BeetleB]

> We want software engineers who understand compilers and query optimizers and systems.

I worked in that space once. You get paid more doing web development.

[jjnoakes]

Money isn't everything though. I've passed up many higher-paying cookie-cutter jobs for lower-paying but more-fulfilling and interesting problem-solving jobs.

Of course, web development can be interesting and problem-solving for some, depending on the job and the person. I just mean to say that money isn't the only criteria that some use to evaluate jobs.

[BeetleB]

True, but the top comment on this thread was about high end engineers being paid high salaries. Compilers is a good counterexample.

[preordained]

What is the work you are doing that requires this sort of fundamental knowledge? Just curious, as it seems like most of the business (read: monetary) value is in working much higher in the stack, leveraging the bedrock software already out there. Is there present moving and shaking happening at that level?

[giantg2]

It's because schools teach to the market. How many schools teach COBOL? There are still companies that want to hire COBOL resources, but most schools aren't going to teaching because the market is too small. It's the same for query optimizers and compilers, only even more pronounced by their very nature - these are things that a very small number of people create because you want a standard or consistent experience with this level of tool. It's like my cryptography course - I'll never use it because the best practice is to use the industry standards.

Another reason you don't find many people with this experience is that there aren't many jobs for it. Going back to my cryptography class example, if I don't have the need to use the info after taking the class, then I'm not going to remember it.

That said, there should be plenty of candidates who are willing and capable to learn this stuff if companies would actually train people.