[threeseed]

Before the cloud took off Hortonworks and Cloudera owned the Big Data market.

They both offered a Hadoop distribution but had different strengths e.g. Hortonworks had fine grained access control, Cloudera had a better SQL product with Impala.

Then AWS came along and built their own which was significantly cheaper and more flexible as you could easily scale your cluster up/down. And so companies moved to it when they over time began to move to the cloud.

The Hortonworks/Cloudera response to this threat was to put away their differences and merge together.

Over time Big Data has evolved from being Hadoop centric to being much more ML/AI focused i.e. not just manipulating and querying the data but doing something interesting with it. And AWS, Azure, GCP have really jumped in with a whole suite of products that are tightly integrated with the rest of their cloud offerings. And it's a large part of what differentiates their offerings so they compete very hard.

So Cloudera has no choice but to do things that cloud providers won't or can't do: (1) focus on non-cloud or multi-cloud and (2) offer a much more integrated and cohesive solution.

But having spent 10+ years in this space and deployed many Hadoop clusters I can tell you that Cloudera is going to struggle. Companies that I never thought would move to the cloud e.g. banks are figuring out the security and regulatory challenges and eagerly moving across. And so it's going to be a Cloudera versus Amazon/Google/Microsoft which is an impossible fight.

[wardb]

Competing with Amazon/Google/Microsoft on their own cloud is...ehm...good luck with that indeed. I believe they should have partnered with them early on (real partnership, like a premier offering, not the rubber stamp / marketplace partnership).

[manigandham]

It can work, as that's exactly what Snowflake has done and it's one of the fastest-growing SaaS companies today.

A good product is more valuable than a partnership.

[wardb]

Good products, but don't discount their go-to-market strategy.

IIRC MS/Azure is an early Databricks investor and their sales folks were heavily incentivised to sell it. They also pushed Snowflake in the early days until they had a competing product and their relationship status was upgraded to 'it's complicated'.

[8ytecoder]

Databricks as well. Azure sells both their own version and “Azure Databricks”

[bostonsre]

AWS EMR is still pretty pricey compared to free ambari/cloudera running on ec2. Although, there is a lot of time and effort that needed to be put into automation that uses those ambari/cloudera hadoop management layers. After they merged, they got really aggressive and made moves that effectively killed each of the free versions. They definitely put another nail in the coffin of hadoop. Spark on kubernetes is pretty gorgeous and has been a successful route out of pricey hadoop infrastructure for my company.

[8ytecoder]

It need not have been this way. All three major providers do work with third party vendors all the time. They could have been the Databricks. Or even better a fully managed solution on the cloud as well (like Snowflake).

[mathattack]

They will struggle as on-pre-K only. They are trying to get to the cloud. The cloud vendors struggle some because their PAAS products aren’t as good. Lift and Shift doesn’t work. It’s Lift and Whoops and Refactor.

[mcny]

> Lift and Shift doesn’t work. It’s Lift and Whoops and Refactor.

I’d like to learn more. What doesn’t work? What can vendors do to make it easier? My understanding is that lift and shift doesn’t mean one and done, no grueling manual testing required.

Once you set the lift and shift as long as the source schema doesn’t change, you could run it as often as you’ll like as you deploy, test, fix?

[swyx]

thank you! i have no history in this space so can't ask followups except to observe that the tendency of ML/AI to reward the "big gets bigger" phenomenon is exemplified here. I don't feel too great about that but also don't have ideas for a better system.