Hacker News new | ask | show | jobs
by bonzoesc 4756 days ago
Assuming you are using Riak's default configuration each range query hits 1/3 of the cluster, which could get pretty hairy on large clusters that have lots of requests. Also, there is no pagination, so if an index has a million objects you'll have to be prepared to wait even if you only want the first part of the query.

You could solve this by putting a sort value in the key and using a range query, but this wouldn't work if you want the most recent items keyed with time, because the items could be unevenly spaced back in time.

Pagination is coming soon; it's in riak_kv master already, but in buyer-beware #yolo territory.

LevelDB is also supposedly slower than Bitcask, the default backend, but I'm not sure if this is still true.

Bitcask is faster when all the keys fit in memory: it's designed to load any value with a single disk seek. LevelDB can't make that guarantee, but neither can Bitcask with too many keys for available memory.

I've been trying to think of ways around these problems. A simple thought I had was to simply cache the response as pages in Riak. Although this introduces new problems like how to know how often to reset the cache, too often and I may as well not have this cache, too infrequently and users get stale data.

Caching is one of the two hard problems in software engineering (along with "naming things" and "off-by-one errors"), so good luck :) If you're not opposed to running a separate service, Memcache is what I'd use.

1 comments

Any thoughts on when the pagination goes live? I can't find any information on it online. Memcache would be a good choice, but I am wondering, if I have a few secondary indexes with over a million indexes each, wouldn't continually recreating this cache irreparably bog down the cluster?
I believe it's part of Riak 1.4, which is our next release; no date yet.

The number of entries in a 2i isn't going to bog down querying it any more than lots of objects bog down LevelDB. Make sure your indexes have the right content with the right cardinalities and it shouldn't be a problem.

If you want to drop in to #riak on freenode tomorrow (I'm in the America/New_York time zone) I'm brycek in there.