[PaulDavisThe1st]

I would say that a better analogy is that you've already got an experienced crew, some swinging hammers, some lugging nail guns and everyone's really quite good at what they do. But the plans from the architect are just really hard to build. You and the crew can do it, for sure, but the issues are not going to better or worse if the guys use the table saw to crosscut some of the framing elements or someone gets one of those new german tool widgets. I mean, sure, the tools might make a small difference to the process, but the overall experience will be dominated by the fact that the thing is just hard to build.

[bslatkin]

Let's keep going with the building analogy. You're running a cabinet making company. You've got all kinds of hand tools and power tools. You build jigs to make certain repetitive tasks faster. Then a ridiculously difficult design comes in for you to build. You and your crew are flummoxed by its complexity. Suddenly, someone offers you a CNC machine that you've never used before. What seemed hard is now easy. The nature of solving problems with a CNC is different. Using a CNC presents other challenges. But you have entered a new realm of what's possible. Analogies are a lot of BS but hopefully that gets across the flavor of what I'm talking about.

[PaulDavisThe1st]

It's a fine analogy. The only problem is that I don't recall ever coming across the equivalent of a CNC machine for the class of problems I face in my work.

Thread-safe lock-free sparse integer-to-integer map? No CNC for that.

Translating time between two domains, one of which is linear and monotonic, and the other is non-linear and non-monotonic. No CNC for that.

Generating and caching the right versions of different segments of audio waveforms at different zoom scales, in multiple threads? No CNC for that.

I could go on, but you get the point.

What tends to be more like a CNC machine are libraries. For example, realizing that you need some sort of reference-counting system for lifetime management, preferably combined with pointer-like behavior ... and then discovering boost::shared_ptr (later to be std::shared_ptr) ... now that's like getting a new CNC machine. But it doesn't require a new language (and realistically, it didn't even require the library - the library just made it possible to not implement it locally).

I think what I'm really trying to say is that I rarely come across problems where I think that the kind of help offered by the putative "new CNC machine aka new language" is anywhere nearly as substantive as the help offered by an actual CNC machine to a cabinet making company. Put differently, the new tool (language) still leaves the problem essentially as hard as it was before.

p.s. a good friend runs a high end wood shop, and I'm fairly aware of the impact their first CNC machine made to what they could do.

[kilgnad]

When the web became popularized there problem of writing really fast concurrent servers that can handle 10k connections without the overhead of 10k threads.

This problem is arguably harder than the example problems you gave and this problem was solved by language primitives that now exist in basically every popular language. These primitives, when used basically change the nature of the language they are used in.

These primitives (async await) are more than libraries. They intrinsically change the nature of your code. (Though technically they could be made into libraries for languages without async await it's just the syntax would be extremely busy)

This only occured because the web was popular and the specific problem of servers and IO changed from a specific problem to a general one. So when someone wants to create a new language it's to attack a general problem.

Your issues in your example look to be somewhat domain specific, so new languages won't really help you in these specific areas you need to handle.

>Generating and caching the right versions of different segments of audio waveforms at different zoom scales, in multiple threads? No CNC for that.

I would say that for this example there are enough general issues here that modern languages CAN help you with. For example do you want to program in a language that can guarantee with helpful static error messages that your code will never have a data race or a seg fault or a buffer overflow or a dangling pointer?

Well there's a language that can help you here. In the same vein I've seen languages go even further then this and guarantee that the compiler will never ever let you write code that will make your program crash.

I think we can both agree that these general features that improve safety WILL make the issues you face easier.

[PaulDavisThe1st]

Based on my experience (and I was doing web stuff starting in 92), this is a mistelling of the tale you're trying tell.

The problem was called "the thundering herd": if you had N threads all sleeping/waiting on a condition, and then that condition was raised/signalled, there were no OS primitives available that would wake only a single thread. Instead they all woke up, tried to get whatever work was available, only one succeeded, the rest go back to sleep. Incredible waste of cycles. These days, you can signal a condition in a way that will only wake a single thread that is waiting on it. Problem solved, for every language, without language modifications.

This was NOT fixed at the language level. It was fixed by adding new OS-level primitives that did the right thing.

Async-wait is another wrinkle in this, but for those of us old enough to remember life before pthreads, that was already effectively taken care of using threads (whatever the API) and existing OS-level sleep/wait primitives.

Doing this without threads is popular among the cool kids these days, but that's even harder than doing it with threads. Consequently, various languages have wrapped this sort of code into builtins in the language. Yes, that makes thread-less async wait easier to code, but it doesn't actually address the design problems where you might be using thread-less async wait to accomplish something.

The problem with waveform caching is not data races etc (though of course, those issues are hard enough). It's figuring out what you should cache and when. The best answers vary depending on user behavior, so you need an adaptive approach that isn't particularly linear, and you also a need way to recognize when user behavior means you should clear out everything in the cache and start over.

[eternalban]

Your recollection is spot on.

Re. CnC, how about something like:

    @thread-safe @lock-free @sparse
    map map[K,V] 

(Let's not get hunged-up syntax.)

> What tends to be more like a CNC machine are libraries.

If a language or library gives you composable semantics, you may have a programming CnC on your hand. CnC requires minimally one degree of disconnect between the artefact and the artisan. A language compiler/runtime (or library) that applies composable semantics to logical & computational abstractions.

[PaulDavisThe1st]

> (Let's not get hunged-up syntax.)

But the syntax is precisely the only thing a language can give you that a library cannot.

[kilgnad]

I'm referring to a similar overlapping problem called C10k. http://www.kegel.com/c10k.html

This was fixed at the OS level. But the usage of new system calls involved fundamental shifts in basically every language to account for the new paradigm.