It's analogous to how functional programming expressed through languages like lisp is the classical foundation of spreadsheets.
I believe that skipping first principles (sort of like premature optimization) is the root of all evil. Some other examples:
- If TCP had been a layer above UDP instead of its own protocol beside it, we would have had real peer to peer networking this whole time instead of needing WebRTC.
- If we had a common serial communication standard analogous to TCP for sockets, then we wouldn't need different serial ports like USB, Thunderbolt and HDMI.
- If we hid the web browser's progress bar and used server-side rendering with forms, we could implement the rich interfaces of single-page applications with vastly reduced complexity by keeping the state, logic and validation in one place with no perceptible change for the average user.
- If there was a common scripting language bundled into all operating systems, then we could publish native apps as scripts with substantially less code and not have to choose between web and mobile for example.
- If we had highly multicore CPUs with hundreds or thousands of cores, then multiprocessing, 3D graphics and AI frameworks could be written as libraries running on them instead of requiring separate GPUs.
And it's not just tech. The automative industry lacks standard chassis types and even OEM parts. We can't buy Stirling engines or Tesla turbines off the shelf. CIGS solar panels, E-ink displays, standardized removable batteries, thermal printers for ordinary paper, heck even "close enough" contact lenses, where are these products?
We make a lot of excuses for why the economy is bad, but just look at how much time and effort we waste by having to use cookie cutter solutions instead of having access to the underlying parts and resources we need. I don't think that everyone is suddenly becoming neurodivergent from vaccines or some other scapegoat, I think it's just become so obvious that the whole world is broken and rigged to work us all to the grave to make some guy rich that it's giving all of us ADHD symptoms from having to cope with it.
I'm not sure about the rest of your comment, but we would likely still want GPUs even with highly multicore CPUs. Case in point: the upper-range Threadripper series.
It makes sense to have two specialized systems: a low-latency system, and a high-throughput system, as it's a real tradeoff. Most people/apps need low-latency.
As for throughput and efficiency... turns out that shaving off lots of circuitry allows you to power less circuitry! GPUs have a lot of sharing going on and not a lot of "smarts". That doesn't even touch on their integrated throughput optimized DRAM (VRAM/HBM). So... not quite. We'd still be gaming on GPUs :)
Ya I winced after I wrote that and am considering taking a break from posting for a while. I'm just really tired of suffering the status quo because many people love it if it aligns with their worldview, while people like me who see things a little differently seem condemned to struggle in isolation. I've always admired Tesla, not Edison, but the Edison way of doing business seems to win out and suppress the people on the fringe.
As for GPUs, they are useful for languages like GNU Octave (MATLAB) and Julia. Although there's a code smell with that. The matrix languages that would benefit most from SIMD seem to be the ones that aren't hardware accelerated on GPUs. Something has gone wrong:
That was over 10 years ago and I doubt the situation is any better today. Misaligned incentives.
I feel that the missing link is direct hardware access to the GPU ALUs. Without being able to transpile CPU code to GPU code and vice versa to provide bare metal multithreading and vector operations, we miss out on a whole branch of computer science. We can't do the interesting experiments that I always rant about, which would make stuff like genetic algorithms borderline trivial. We're stuck with cookie cutter solutions like OpenGL, Vulkan, Metal, etc.
Yet we still praise Nvidia, even though like Microsoft/Intel in the 1990s, they're probably most responsibly for stifling innovation in multicore computing.
For what it's worth, you're right about the latency/throughput tradeoff. I don't mind stuff like Intel's integrated graphics or Apple's M line of processors with integrated GPU and NPU cores for raw throughput and efficiency. I just think they're silly because they operate at the wrong layer: 1) CPU 2) multicore 3) GPU. There's no layer 2. Why is that?
> while people like me who see things a little differently seem condemned to struggle in isolation
Haha, you should consider founder life then. I think many founders feel like this.
The issue with transpiling to GPUs is that it is really hard to do that in a sensible, performant way. Something something a sufficiently smart compiler...
There is plenty of @jit in Python, and there's TornadoVM for JVM.
Nvidia's CUDA has been around since the 2000s and is somehow still best way to program for GPUs still. Probably the best SIMT-oriented stack at all, even. From my perspective, it's everyone else stifling themselves.
I realized after responding that I forgot what I meant to add - that I shouldn't have brought neurodivergence into the conversation. I believe that proponents of the status quo are vilifying outside-the-box thinking and the innate sense of justice prevalent in people on the spectrum because they feel threatened by them. But that's a generalization that makes assumptions about people and isn't constructive, so I'm sorry for saying that.
Anyway, ya I can see how founders get pretty well ostracized by the business community. I tried to be one back when I thought that shareware mailbox money meant winning the internet lottery. Statistically, it's highly unlikely that anyone can make it on their own now. It's not just that the system is rigged, but that founders who exit often take the money and run rather than changing the systems of control that they managed to overcome. To me, rising wealth inequality and deliberate ignorance of survivor bias indicate that the situation is getting worse, not better.
You're right about the sufficiently smart compiler, I hadn't heard that term in a while. I believe that functional programming (FP) and imperative programming (IP) are at opposite ends of the spectrum, and that FP is about simple formal determinism while IP is about easy informal convenience. I also believe that IP can never be fully optimized, due to the halting problem, exponential complexity from mutability, etc. Whereas FP might be optimizable because it can be statically analyzed. Usually IP code is broken down into FP segments which can be optimized, then maybe mutable code gets approximated with impurity or monads or left as a bottleneck. I dream of writing an IP language where all variables are const and there are no pointers or references, that's performant by way of copy-on-write and other techniques where the runtime does the heavy lifting instead of humans, so that it can be converted to/from FP and fully optimized. Without that, I don't see a practical way to convert CPU code to GPU code.
So it's really 2 problems that are unlikely to be solved any time soon:
1) we're using the wrong hardware
2) we're using the wrong software
The status quo won't fix this, and the game is rigged to prevent independents from fixing this, and that's why I'm not seeing any movement on these problems after observing them for over a quarter century.
AI will likely make things worse by glossing over the real work of fixing the fundamentals. Rather than writing provably correct code in an FP language like lisp and verifying it with behavior-driven development (BDD) integration tests for example, we'll trust the IP code written by AIs and skip verifying it.
So it's been over for a long time, and getting to be even more over than it was. Defeatism is the safest bet now, which I find heartbreaking. And that's what people will hear when they read my words, even though I'm fighting for a revolution in how we write code and how fast it runs. That's why it's over.
SQL on GPUs is definitely a research classic, dating back to 2004 at least: https://gamma.cs.unc.edu/DB/