[vosper]

> Do any other entrenched software projects come to mind?

Elasticsearch is underrated here, IMO. Yes, there are alternatives for simple fulltext search. But there’s a lot more it can do (adhoc aggregations incorporating complex fulltext searches, with custom scripted components; geospatial; index lifecycle management) and if you’re using those features, there’s nothing else comparable.

It’s pretty stable, too, once you’ve got the cluster configured. We don’t have outages due to problems with Elasticsearch.

[FridgeSeal]

To provide an opposing viewpoint here: ES and it’s monstrous API and resourcing requirements are a pain to manage and run. It’s a product that has pivoted in so many directions that it’s just become a bit of a mess. I don’t want a full-text search engine that also has graphs, ML, some bizarre scripting feature, log management, etc all stapled in on top. Geospatial and other analytic stuff I’d rather use a dedicated OLAP db like Redshift or ClickHouse.

I’m currently evaluating typesense vs ES for a fts project and typesense is winning so far by simply be “not painful” to deal with.

[vosper]

> I don’t want a full-text search engine that also has graphs, ML, some bizarre scripting feature, log management, etc all stapled in on top

Sure, so use something else. I do need (most all of) that at my work (plus the horizontal scaling), and there's no competition. I know we're not the only ones.

Also, there's nothing bizarre about the scripting feature. There are several options for scripting, it's very flexible, and it suits implementing custom logic when you need it.

And, I'm not saying ES is perfect! I'm saying that there's a set of use-cases that only ES (to my knowledge) can fulfil, and that's complex aggregations also involving complex full-text search, over tera/petabytes of data. Clickhouse can do aggregations, but doesn't have anything close to the search chops (again, to my knowledge).

[jeff-davis]

I don't know about elasticsearch specifically, but I'm skeptical of special-purpose systems for databases.

They are great in some cases and terrible in others, and over time, use cases push database systems into their worst cases. Use cases rarely stay in the sweet spot of a special-purpose system.

That being said, if the integration is great, and/or the special system is a secondary one (fed from a general-purpose system), then it's often fine.

[vosper]

I’m not sure I fully understand your comment (databases that are special-purpose and evolve out of a sweet spot, or special-purpose systems using databases in worst-case ways?).

I certainly wouldn’t say ES is the former. We use it for some conplex things that (AFAIK) no other (publicly available; I don’t what eg Twitter or Google has going on) system could provide at the scale we need. Everything we’re doing is well within the realm of what ES is built for, and it’s the only system built for it. It’s not perfect, but most of our performance issues could be solved by scaling out, where query or index optimization isn’t tractable.

[jeff-davis]

I interpreted (misinterpreted?) your comment to be suggesting ES for wider use cases.

[bradleyjg]

It’s frustrating to need a run-time team for a piece of infrastructure, especially one sold as IaaS.

It’s totally understandable that you’d need developers to have expertise in patterns and anti-patterns, as well as needing an expert to set things up in the first place, but you shouldn’t have to have a dedicated ES monitoring / tuning / babysitting team like Oracle DBAs of yore. That you do, means it isn’t there yet as a product.

[Mike8435234]

ES doesn't need a "run-time team". It just works.

[FridgeSeal]

It absolutely does not “just work”, there’s so much to configure and then get-right for your use-case that you almost certainly need people with a solid understanding of the JVM + ES. Let alone fixing it when something inevitably breaks.

[Mike8435234]

No more than any other database. I mean relative to SQL Server, Postgres, MongoDB or any other database. There's no extraordinary difficulty to manage ES above any other production system. It is very usable out of the box, and needs minimal tuning for many use cases. Of course some uses cases will require additional tuning and maintenance, sometimes quite a lot if you have a very large system, JUST LIKE ANY OTHER DATABASE SYSTEM.

In our case for a small website serving the general public (a few tens of thousands of requests per day) it just worked OOTB with hardly any tuning or maintenance at all.

[samsquire]

Elasticsearch requires lots of hand holding if you have a cluster. Sounds like you're talking about a single instance.

Especially if an index goes down and you need to kick it to continue indexing.

[Mike8435234]

We had a cluster. It was low maintenance. Any clustered / distributed database will require maintenance. At my current job we have SQL Server and there's a shitload of admin/maintenance required for that.