[nzadrozny]

Sure, that's a fair point. Data consistency reliability in ES and Lucene will only get better over time.

But I personally suspect Lucene won't ever get away from the dreaded "just reindex." And to the larger point, I think recent resurgent interest in data stores and distributed systems have shown pretty clearly that there is no holy grail. No single data store can provide all the semantics necessary for all use cases. Maybe not even for most use cases. There are just too many tradeoffs to consider.

Believe me, I earn a living hosting Elasticsearch, so I'd love to see it become a robust primary data store. There are some use cases where it actually does make sense—just look at the amazing traction ES is experiencing for storing and indexing time-series data.

But as a general-purpose primary store, I'm not really holding my breath. Maybe I'm just becoming battle-worn and bitter. I would love to be proven otherwise over the next few years!

[dclara]

I'd like to learn from you about "general-purpose primary store". Do you mean for storing any type of data? Here is what I think regarding the case you brought up in the previous post:

ES is suitable for full-text based document indexing for enterprise level or any websites, which means they have a reasonable amount of data to be indexed in a given timeframe. A complete re-indexing won't not take for a couple of days.

So the basic idea behind the NoSQL database is to dump the data into the database quickly and return, so you can see very fast response for insert and delete. Then it will load the data into the memory to process for real-time retrieval which also produces fast response from select. I'm not sure about update.

If the data volume grows, they quickly add shards or make the number of pre-shards big enough to allocate enough memory resources to handle the queries or let the OS to swap the memories by adding more server nodes.

So if you want to use NoSQL database, you must be bound with the system requirement and make your application fit into that and take the most advantage from it. Otherwise, if you are running high structured data store, better to use relational database.

Another point is: if the documents are collected from the web like search engine, NoSQL will not fit for the large volume of data and relational database is also used to store the indexed data for fast retrieval. I guess this is what you meant "general-purpose primary store".

Correct me if I'm wrong.

[berkay]

I think the distinction made it this comment is valuable, and echoes our experience. ES is not (at least so far) suitable as a general purpose data store, but it is suitable (and very good) for more than search. For some use cases, it's the best available.