Hacker News new | ask | show | jobs
by slap_shot 3250 days ago
I'm a co-founder of a stealth-stage company that helps data analysts/data engineers build data pipelines. Every "task" that can be done in our framework is essentially just an image that can be reused over and over with different settings.

We deploy these tasks across Kubernetes clusters on AWS, GCP, and Azure.

Since these tasks are schedule irregularly and are short lived, we had to do a lot of work to dynamically scale the nodes up a head of their demand and down after, and we typically have to pay for at least 10 minutes of usage no matter how quickly the job finishes.

This "pay-by-the-second" will be a huge win for us. Most of our tasks deal with S3/Redshift or GCS/BigQuery, do we can't immediately use this. But as we onboard more clients working with Azure Storage/Data Lake/Data Warehouse I see some big operational gains for us.

Here's hoping we see similar developments across the other major cloud providers. Very impressed with Azure's development in the last 3 years!

1 comments

Check out Hyper.sh too. It abstracts away the whole datacenter -- you can use `hyper` instead of `docker`, basically. You don't need to think about VMs ever as a concept, containers run directly on the hypervisor. And they have Hyper Func, an AWS Lambda-like alternative that uses images. And per second billing.

On the downsides, they're small and they have one data center, and they're not Microsoft. But their tech is open source.

We don't want to compete with the big providers, instead we open source the tech to enable more container-native clouds, where the world will become a seamless (portable) network for containers (different clouds are different ports with the same image spec and API).