1) I don't use "work-in-progress" things in production (unless I own it and wrote it) and I have a hard time recommending others do so.
2) It's not just events in your application you have to worry about, but environmental stuff as well. SNS is the lingua franca of AWS and you've gotta put it somewhere, with reliability guarantees. That increases the amount of orchestration you have to do outside the safe-and-happy confines of k8s itself, and that makes my job way harder than having only-one-road-to-plow.
At my current company we use ECS/Fargate, but it's only really so we don't have to manage servers; each ECS service has one task and is independently managed. And it interacts directly with the baseline cloud provider so there's no weird jank at the edges of our compute vs. our datastores. The containers are a convenience here--I'd use EC2 just as readily. But I see no real value to using k8s instead and haven't at any of my consulting gigs to date, either. The k8s environment is both incomplete and at best kinda-incompatible with the things that actually matter, like your data, and TBH it casts real doubt on the reason to use it.
So I have no doubt that SQS is more mature than knative. In my case I am developing an actual product IN k8s (with around 40 new CRDs) , so my goal is to use as much of the platform as possible (including istio and knative) and avoid cloud specific services.
The value with k8s is that you can abstract the underlying hardware by using only k8s objects.
"The underlying hardware" has side effects. You can't exactly ignore them. And your cloud provider has non-substitutable stuff. Just as an example from my day today, by hewing to k8s you lose stuff like Athena and Glue for data analytics and ETL. I'm sure somebody out there would like to charge me 10x what AWS does to do a worse job of it inside of k8s, but no. So instead you get a Frankenstein of Terraform/Pulumi/CloudFormation and (poor and poorly expressive) k8s configuration and you've geometrically increased the complexity of your system, your failure cases, and the challenge of solving a problem when you're under the gun.
So your product might be fine, but everything that exists to feed it--be it monitoring and alerting or business analytics or security or even hands-on operational control surfaces--is going to be worse for it and create marginal drag every step of the way (either in terms of labor or money).
More and more (and your comments are reinforcing this position, TBH) I get the "it's a slick five-minute demo" argument out of k8s, much the same way that Docker brainwormed people long before it was good or useful. I have a cluster at home, and it's fine, but it's for play. I can't afford systemic drag from immature and possibly-wrong tech choices where I've gotta make money, though.
Thanks for the feedback. I do not think that k8s is bad bet long term as you paint it to be. Of course it is less mature than AWS. But the point is 5 years from now.
The problem with AWS is the price and the latency. Everything is fine until you get the bill. But by than you are completely locked to that architecture. The same apply to all the public cloud providers, not just AWS.
So as I see it, for new applications which are based on micro services, and want some day to become self managed, K8S is the only long term good bet.
I gotta ask, because now you're firmly in my backyard--what's the biggest cloud spend you've ever been in charge of? I ask because "Cloud costs" keeps coming up as this bugbear reason to use k8s and it isn't a real concern for the 99th percentile of applications. An application that's expensive when running directly against a cloud provider's APIs will remain expensive when running in k8s, if not moreso because of k8s's steadfast refusal to pay attention to the bin packing problem. The galaxy-brain thought on HN is that cloud providers are so much more expensive than OVH or Hetzner or whatever--it's literally meaningless. People cost a lot. Even an inefficient use of AWS doesn't cost very much. By the time you're at the point where your cloud spend exceeds one FTE, you should probably have forty and the wins it gives you should be self-evident or you've screwed up somewhere else (and that "somewhere else" is probably your business plan).
I've moved nontrivial systems from AWS to GCP and in the reverse direction. It's a job done in Terraform/Pulumi and while a competently written application or set of services needs some work to do the move it's work you are likely to do once at most. (Emphasis on at most. The overwhelming, overwhelming majority of companies are way better off going multi-region in a single cloud provider than going multi-provider. Multi-cloud is for the rich and the silly.) The underlying cloud provider doesn't matter very much when you can pay somebody like me to come in for a month or two and help you make your application an actual citizen of the platform you want to use and leverage its efficiencies properly. The "good long term bet" is abstract interfaces in your code--the hype-driven cycle of the new-and-shiny means there's a nontrivial risk that k8s is no longer sexy enough to blog about by the time that "oh, we now need to move to a new provider!" even matters to you.
(I am contractually obligated not to step in the microservices pothole. It's a good way to waste development time and not ship, though.)
> I ask because "Cloud costs" keeps coming up as this bugbear reason to use k8s and it isn't a real concern for the 99th percentile of applications.
AWS VMs tend to cost between 3 to 4 times more than equivalent VMs offered by smaller no frills service providers such as Hetzner and Scaleway.
You may argue that you don't mind paying a hefty premium for a service that has plenty of competing offers, or that some high-level service provided by AWS is nice to have, but it's hard to argue in favour of needlessly spending 3 to 4 times as much to provide the same service, or get somewhere around 30% of the cloud computing resources for the same price tag.
So more expensive is on a relative basis and taking into account the egress traffic from the cloud. My workload involve training machine learning models and serving them. Training on the cloud is 20X more expensive, and you cannot use commodity GPU (banned by Nivdia).
Moreover, My platform offer automl, which basically trade data scientist time for compute time. However, since I need to train 100's or 1000's of models, this can become very expensive, very fast.
Since I am not sure what my customers load will be like, I want to give them the option to move between clouds or on prem.
For the long term prospects of kubernetes. For me it is clear the kubernetes have found PMF and it is now at the first 1/4 or 1/3 of the S curve. IBM is all in, VMWARE is all in, Azure is all in, Gcp is all in.
Also, what are the alternatives ? Do you agree that containers are better than jar files or manual deployment options?. Do you agree that having a CI/CD pipelines with fully automatic unit/func tests is better than throwing code over the wall to some QA department?
So if containers are better packaging/deployment architecture, they need to be managed/monitored etc.
For micro services. In my case, the data since part is written in python, while the control plane (kubernetes operators) is written in go. So micro services are actually a very neat solution to a polygot product.
2) It's not just events in your application you have to worry about, but environmental stuff as well. SNS is the lingua franca of AWS and you've gotta put it somewhere, with reliability guarantees. That increases the amount of orchestration you have to do outside the safe-and-happy confines of k8s itself, and that makes my job way harder than having only-one-road-to-plow.
At my current company we use ECS/Fargate, but it's only really so we don't have to manage servers; each ECS service has one task and is independently managed. And it interacts directly with the baseline cloud provider so there's no weird jank at the edges of our compute vs. our datastores. The containers are a convenience here--I'd use EC2 just as readily. But I see no real value to using k8s instead and haven't at any of my consulting gigs to date, either. The k8s environment is both incomplete and at best kinda-incompatible with the things that actually matter, like your data, and TBH it casts real doubt on the reason to use it.