[deweller]

I don't have facts to back up my opinion - It is formed by memories of articles I've read.

I believe CouchDB is a better choice for very large data sets because of its design.

+ CouchDB uses a Map Reduce design that I believe would scale better over very large data sets. + CouchDB always stores data in a consistent state on disk. You can literally pull the plug on the server at any time and the data will never be inconsistent.

MongoDB is geared for performance and is a great bridge between a relational database and a high-performance No-SQL database. But I don't recall that it's strength is handling large datasets.

[ethangunderson]

Comparing map/reduce in Mongo and Couch is really apples and oranges. They are designe to do two different things. i.e data processing vs building views.

Mongo is designed from the ground up to deal with large datasets. Take a look at their sharding architecture.

[catch23]

I guess it depends what you consider "very large". If you're talking about multi-petabyte, then I'd probably use hdfs, but otherwise mongodb might fit. I hear craigslist uses mongodb to store their data since 1997, which is a fair amount of data I believe.

[jzawodn]

Uhm. The data goes back that far but the usage is far more recent than that.

[Wijnand]

1997, are you sure?

[simonw]

See http://blog.mongodb.org/post/5545198613/mongodb-live-at-crai...

[duggan]

When he says "store their data since 1997" I believe he means "store data dating from 1997 on" - slight confusion in choice of wording!

[alnayyir]

CouchDB is better'ish for larger datasets, but not for arbitrary scaling. MapReduce in CouchDB requires dumb full-scans if you're not just refreshing an existing view.

Arbitrarily large data is the exclusive domain of hadoop/hypertable/cassandra AFAIK atm.

[tmcneal]

To be fair CouchDB is very explicit that to get any sort of performance, everything must be a view. "Ad-hoc queries" (i.e. queries that are written on the fly instead of uploaded as a view) are clearly stated as "for development only".

Where CouchDB really falls flat is for write-heavy applications. The default configuration in CouchDB is to not reindex a view until it has been read. When a read occurs, any new data in a view that was added since the last read must be re-indexed by executing the map/reduce functions on that data. If you're writing frequently to CouchDB but not reading a lot (as in a data warehouse) the first query you run is going to be extremely slow, since it will need to run map/reduce on a lot of new data. CouchDB doesn't distribute work to multiple nodes like Hadoop, and I've found even simple reduce functions to slow down re-indexing by a factor of 10. I think CouchDB has settings now to update the index on commit, or you could always run a cron job to regularly query the view and force a reindex, but it's still going to be slow.

BigCouch (https://cloudant.com/#!/solutions/bigcouch) might be a potential choice for data warehousing, since it advertises full compatibility with the CouchDB API but offers distributed map/reduce like Hadoop/Hive/etc. I haven't used it though.

[alnayyir]

Couch is definitely a lot more honest about their limitations than mongo or riak, but my experiences make me hesitant to recommend it to anyone not intimately familiar with those limitations.

[jchrisa]

This is part of what we are addressing with Couchbase Server, an autosharding rebalancing Couch fronted by memcached. For K/V read and write we measure microsecond latency.

We are currently optimizing the views for cluster access, but the design goal is to offer at least the query performance CouchDB offers on small datasets, even on very large clusters.

More info: http://blog.couchbase.com/couchbase-server-2-0-tour-and-demo

[alnayyir]

Is it going to have the same/similar multi-replication for enterprise only limitation that Riak has or is this going to be funded in some other way?