[aventrix]

Interesting. My initial instinct is to compare them to MongoDB which IPO'd about a year ago. They seem to be doing quite well since, but then again, who hasn't in the tech sector?

I will say I very much enjoy Elastic's suite of products, I can't say the same for MonogoDB.

[hendzen]

There are key differences between MongoDB and Elastic.

MongoDB (the company), has been very strategic about defending themselves from Amazon. Likely due to the AGPL licensing on MongoDB (the database), Amazon doesn't offer a hosted MongoDB - you have to buy MongoDB Atlas which runs on multiple cloud providers, including AWS, GCP and Azure. Coincidentally, Atlas subscriptions are the fastest growing part of MongoDB Inc's revenue, which is why investors are bullish on the company.

Elastic (Apache licensed) has actually had their lunch eaten by Amazon to a much larger degree. Many businesses are purchasing hosted ES instances directly from Amazon via Amazon Elasticsearch service. The problem is bad enough that Elastic Inc had to write a marketing blog post about it [0].

Say what you will about MongoDB (the database), but MongoDB Inc. is the most successful open source software-based company since Red Hat. You can see it in the outperformance of $MDB compared to similarly timed open-source IPOs such as Hortonworks ($HDP) and Cloudera ($CLDR).

[0] - https://www.elastic.co/blog/hosted-elasticsearch-services-ro...

[mooreds]

I can't speak for others, but I was looking to set up a quick elastic search cluster and Elastic's hosted offering beat the pants off of AWS ElasticSearch in terms of usability, polish, ease of getting started and security-by-default.

Now, I don't have a lot of data in the cluster, so we'll see how performance is when that happens, but would definitely recommend Elastic's offering.

[FridgeSeal]

We recently moved our stuff at work from AWS to Elastic Cloud and it’s excellent.

Performance is miles and above better, there’s actual security options on the clusters and there’s way more features.

[busterarm]

Huh?

Elastic Cloud runs in either AWS or GCP. You get to choose but it's just between those. Unless you mean you were using AWS's ElasticSearch service... which is known to not be all that great.

You could build ES clusters on AWS directly on EC2 and it would be cheaper than Cloud, unless what you're really looking for is that support contract.

I spent a few months of my life learning ES inside and out and would choose this option every time. One of the best tech investments I've ever made.

[mooreds]

> You could build ES clusters on AWS directly on EC2 and it would be cheaper than Cloud, unless what you're really looking for is that support contract.

Or you are willing to trade money for dev time. Setting up and running the ES cluster on EC2 will take time and energy (source, I did it ). Whereas setting up the ES cluster on either the Elastic Search hosted service or AWS ElasticSearch is really just clicking some buttons.

However, I'm sure the right solution also depends on the amount of data you are using and the problem you are trying to solve.

[busterarm]

I can largely agree with the last statement. The problem I find with the hosted services is that people just tend to throw data at it without really knowing anything about cluster/node/shard performance. This typically blows up in their face at some point and even though Elastic Cloud has nice sliders in place for spend-more-money, they still have to go through the rebalancing pains (if they didn't write-lock the cluster outright). That should never happen.

If you have strictly time-series data and are willing to make compromises around retention window, hosted elasticsearch is an easy conclusion. Invariably though, I find that where performance is a real concern, you have to carefully plan what data you're indexing and that requires operations knowledge around elasticsearch. At that point you almost might as well host it yourself anyway.

Also if X-Pack was something I _had_ to have, and tbh there are free alternatives for all of its good features, I would consider Elastic Cloud.

I'm even a customer of theirs on some clusters, but lackluster support and pushy upselling experiences have made me want to move out of it.

[Rapzid]

An issue I had with AWS was that even after they launched VPC ES clusters, they still hid the node IPs in the API output(I guess they patched ES?!). Our use case as a data lake meant that we had severely degraded performance with the hadoop/spark drivers than one would expect form it.

[toomuchtodo]

> Elastic (Apache licensed) has actually had their lunch eaten by Amazon to a much larger degree. Many businesses are purchasing hosted ES instances directly from Amazon via Amazon Elasticsearch service. The problem is bad enough that Elastic Inc had to write a marketing blog post about it [0].

Also the reason Redis Labs made the controversial decision to change their licensing (and they reference Elastic in it) [1].

[1] https://redislabs.com/blog/redis-license-bsd-will-remain-bsd...

> Cloud providers have been taking advantage of the open source community for years by selling (for hundreds of millions of dollars) cloud services based on open source code they didn’t develop (e.g. Docker, Spark, Hadoop, Redis, Elasticsearch and others). This discourages the community from investing in developing open source code, because any potential benefit goes to cloud providers rather than the code developer or their sponsor.

[busterarm]

A big part of it is Elastic's fuckup.

It's a big deal for a lot of companies to be able to pay the cloud bill in one place. Elastic used to provide the availability to pay for the Elastic cloud subscription through AWS' Marketplace. Elastic stopped this when they changed their pricing at the beginning of 2017 and we (my last employer) were negotiating a contract with them at the time. AWS billing was a hard requirement for us because it made it a lot easier on us from the finance side. We walked and I know a lot of others who did too.

[toomuchtodo]

It’s not Elastic’s fault your accounts payable process is deficient.

[busterarm]

I'd be surprised if even 20% of companies' accounts payable process is sufficient.

The point is that AWS removes friction and Elastic chose to add friction. They're chasing big Enterprise hard and it doesn't seem to be working out.

[toomuchtodo]

It’s a business decision of course. Forge your own path or be beholden to your revenue model dictated by AWS.

Big Enterprise will not require billing through AWS (disclaimer: employed at a large enterprise, buying ~$900k software license currently). They’ll have an accounts payable department, a procurement process, and someone to cut a check or ACH to Elastic monthly/annually.

[toast0]

It's not elastic's fault, but it is elastic's problem if they can't get paid.

[itronitron]

Can someone explain why a company would pay a cloud provider extra to manage an open source API for them when they could just run it in the cloud themselves?

[toomuchtodo]

Cheaper than spending the time doing it yourself until you’re paying people to do it for you.

Time is the ultimate non-renewable resource. Throw dollars at the problem when you have them and it’s cheaper than your time.

[itronitron]

I think a lot of companies have a hard time managing that transition as they scale up though.

[achompas]

Another thing to note, along with sibling comments: AWS Elasticsearch is limited in its capabilities. For example, you are blocked from relying on custom ranking models to rank your documents.

[jillesvangurp]

There are a lot of limitations with Amazon's hosted offering. Mostly you will find out about them when your cluster needs maintenance (upgrades, outages, etc.). Elastic cloud is a much better product in terms of features, support, performance, documentation, price, etc. I've used both and ran my own clusters as well. Elastic Cloud is my default option these days.

[angelirizarry]

Can you say more about these limitations? Right now I'm using a pretty big custom ranking function to rank my documents in Elasticsearch (using a function_score query), but maybe I'm misunderstanding.

[achompas]

Sorry, I was a bit vague. Queries are fine, but you cannot, say, use a scikit-learn model trained on user activity.

[nodesocket]

I setup a 3 billion plus document ES cluster using AWS Elasticsearch Service. As a whole it works really well, and it's nice not to have to manage individual EC2 instances. With that said, there are some very annoying things. Previously if you modified the access policy of the ES cluster, it would completely rebuild the entire cluster (create all new nodes and copy all data), which can take many hours.

Recently I turned off some slow query and slow index advanced logging options, again AWS completely rebuilt the entire ES cluster when these options surely should be able to be applied without copying all data.

[scarface74]

With that said, there are some very annoying things. Previously if you modified the access policy of the ES cluster, it would completely rebuild the entire cluster (create all new nodes and copy all data), which can take many hours.

From what I can tell, it still does. I can’t comment on the length of time though. I did it before I had many documents and it was a very small Dev instance.

[nodesocket]

I think in March[1] they announced this improvement.

[1] https://aws.amazon.com/about-aws/whats-new/2018/03/amazon-el...

[czbond]

Really great reply - thank you for sharing. I was just researching the company more for investment. I agree also with regards to Horton/Cloudera. They had so much promise - but felt out of favor.

[d4l3k]

To be fair, both MongoDB and Elasticsearch have had serious issues as pointed out via the Jepsen tests.

https://aphyr.com/posts/317-jepsen-elasticsearch

https://aphyr.com/posts/323-jepsen-elasticsearch-1-5-0

Though, all those issues have been fixed for both MongoDB and Elasticsearch so it's not necessarily fair to judge them based off past performance.

https://www.elastic.co/guide/en/elasticsearch/resiliency/cur...

https://www.mongodb.com/mongodb-3.4-passes-jepsen-test

[Fiahil]

These jepsen tests cover elasticsearch 1.5, a large number of issues have been progressively fixed. We're now at ES 6

[aisofteng]

Note that 3 and 4 were skipped; ES 2 was followed by ES 6.

[Argorak]

ES 2 was followed by ES 5.

[xaranke]

The results for ElasticSearch are from 2015, curious to see how relevant they are now especially given that the team has been working on addressing these issues in the interim.

[tschellenbach]

Also elastic back then was used as a place where you copy your data, not your main data store. (Their positioning changed later) MongoDB was supposed to be used for your primary data store.

[Thaxll]

Those Jepsen are not that useful, I can point to you Postgres issues where it would corrupt the DB, does it means you shoudn't use Postgres?

[baq]

They're very useful - these posts compare actual functionality with the project's documentation.