[dh997]

Having been an HPC sys admin pre and post cloud, this unicorn comes around about every ten years... many previous startups failed because they tried to push expensive custom hardware with fancy interconnects, but most importantly, there isn't a need because scaling apps horizontally instead of vertically is usually more powerful, cheaper and more reliable as evidenced by Google's dominance. Plus, Rocks Cluster, Hadoop, Kafka, Kinesis, iPython, Erlang/OTP, Akka and more already make these sort of things moot because deploying a resilient cluster has not been as impressive as summiting Mt. Everest for a decade now. It's feels like trying to sell a magical product (cluster resource management) to get a major advance for free (single process system image). Maybe they can make the file system, ram, network devices, packets, firewall rule and everything else magically work, but fault handling and latency are big concerns that rapidly increase in complexity, orthogonal to anticipating understanding of how to most efficiently schedule what a process wants done when it would be simpler to code for parallelism such as MapReduce / actors from the beginning.

[sz4kerto]

> there isn't a need because scaling apps horizontally instead of vertically is usually more powerful

No, it's isn't more powerful per se. There are many problems that simply cannot be scaled horizontally at all because the data exchange pattern between nodes. However, the infrastructure/hardware behind horizontal scaling is simple, and it's basically unlimited.

[dh997]

There's only so many TB of ram / disk and cores that can fit in one box for sensible cost, whereas it's usually far cheaper and less limited to just add more boxes, plus the SPoF of one giant box.

It usually indicates developer/provider laziness / cheapness / lack of imagination when people say something "can't" be done. Have they even tried? Most real problems are diagonalizable. Plus any shop using RDMA and fast infiniband instead of Ethernet dont have to worry as much about node-to-node latency.

There is no magic solution for all use cases, but scaling veritically tends to encounter budget or technical limits where a little forethought beforehand could have scaled using modern tools instead of shifting burden down the stack.

If you're LMAX, scale up... looking for ET in radio signals, scale out.

[taneq]

If these problems can't explicitly be scaled horizontally by the developer, why would this system be able to scale them horizontally by doing it behind the scenes? Any limitations of data exchange between nodes would still apply.

[dh997]

Most of these single system image clusters have used custom, fast interconnects akin to Inifinband or hypertransport to ameliorate locality issues (latency and bandwidth)... Without such, it's neigh impossible without pinning all resources for a "process" to one physical box.

Good luck ever getting more than one box worth of performance or resources out of a single application instance.

Hence MapReduce, actors, promises, message-passing, etc.