[arafalov]

Nice introduction. A good ramp-up from basic splitting to ranking.

It does need to be said that when Lucene had a set of features this small, it was also pretty tiny. And, if those are the needs, one could still download it and it will probably run on modern JVM:

https://archive.apache.org/dist/lucene/java/

lucene-1.4.3.jar 2004-11-29 14:13 316K

It is a bit bigger of course, but that's because it already had stemmers, multilingual support, multiple Query Parsers including phrase, RAM and Disk directory implementations and a bunch of other advanced features one can easily see by unzipping the jar file.

[finikytou]

it amaze me how much is about creating an eco-system to scale and handling complexity and how software grow to answer that instead of trying to split into two different piece of software. Many people/companies/websites just need those basic features when search is not a core essential capability

[arafalov]

Well, Solr and Lucene projects are right now in the process of splitting up (after joining for versions 3-8).

And, you could have always used Lucene directly as an embedded search library. Amazon does, for example, for their customer facing search as their scalability patterns do not align with either Solr or Elasticsearch.

And if you do use Lucene directly, you can choose just the libraries that apply to your use case. I think at minimum, you can get away with maybe 3 jars (core, queryparser, analyzers-common) and that's just over 5Mb for the latest Lucene, which includes things like: http://blog.mikemccandless.com/2021/03/open-source-collabora...

Sometimes, there is a disconnect between implementation that is very flexible and messaging that just shows 'use everything' garden path.

[fakedang]

> Amazon does, for example, for their customer facing search as their scalability patterns do not align with either Solr or Elasticsearch.

Can you explain this to a non-programmer?

[arafalov]

That is a good challenge, as I am sure their reasons were extremely technical. Twitter also uses Lucene and also in different ways. And I am not at Amazon or Twitter, so I just go by their public presentations, quite technical ones.

But basically I suspect it is like this.

Vast Majority of actual search magic (vector analysis and heavy math and optimized index management) is in Lucene. But Lucene does not care much about getting data into the system (json, csv, xml, etc) or getting it out. It just has internal representation of documents and fields. Plus it does not do any sort of multi-index management, needed for scale.

So, both Solr and Elasticsearch build on top of Lucene to do the user-friendly and scaling parts. Solr allows to send data in multiple formats, Elasticsearch sticks to JSON. Solr has multiple way to build search queries (url params, xml, json, changed over time), Elasticsearch sticks to JSON. Solr is a bit rigid about how schema (field collection and type definition) is done, Elastisearch is a bit more hands-off. Solr has composable pre-processor chains and explicit field analyzer chains, Elasticsearch focuses more on logs and has external pre-processors and custom scripting language. Solr uses Apache Zookeeper to coordinate distributed state, Elasticsearch rolled their own. Solr has one particular way to split data into replicas and shards and throw them from one server to another, Elasticsearch has another. Solr is perfectly happy to run a single node=server=collection=core. Elasticsearch starts with fully distributed cloud setup.

All of these are trade-offs on top of actual search. That's why most of the consultants work with both Solr and Elasticsearch, the most difficult search optimization concepts are same for both.

Bloomberg, I think, uses Solr directly and builds on top of that. They contributed Machine Learning ranking to Solr, for example. They also use Elasticsearch for log analysis, I believe.

Amazon's needs are different again. Their data comes in through different routes (direct from databases?), their multi-tier replication and sharding strategy have different needs, they don't need user-facing user interface, etc. So, they take Lucene directly, and build the rest on top. And then contribute back to Lucene, which makes it available to both Solr and Elasticsearch. Everybody benefits in the end.

[bryanrasmussen]

>Many people/companies/websites just need those basic features when search is not a core essential capability

and some companies convince themselves that search isn't essential to them and they end up with awful search not much better than an SQL like query and when they do UX evaluations of how people use their site say look - nobody uses our crappy search let's not put any time into fixing it!

[finikytou]

agreed. but lets be honest to operate an elasticsearch for instance. You need a sizeable team. its not a one man job.

[arafalov]

You can use embedded Lucene if you want great search but no UI/schema management and you are happy to code the integration. You can start with 3 jars (5Mb) in size and add additional jars as your needs expand (e.g. multilingual processing).

You can use a single node Solr, if you want UI and send data in multiple formats and may eventually need to scale.

You can use Elasticsearch if you know you are starting big and deal with scaling from the start.

You have the choice. It is not just clustered Elasticsearch vs code your own.

[bryanrasmussen]

well, I'm pretty good at all the non operations aspects of ElasticSearch, so in my experience if you have ES in some sort of managed cloud instance - like searchly, doesn't have to be AWS or elastic - it's a one man job - and also in my experience not necessarily full time devoted to just es.

May vary dependent on if you have a lot of data, but if so you probably have people to handle lots of data problems anyway.