|
|
|
|
|
|
by written-beyond
570 days ago
|
|
|
Code quality is pristine, really great job! I see that you've used protocol buffers, can you expand on why? I am aware of the benefits it offers but I think it adds a bit of mental overhead initially due to it being an additional type system you have to understand. Also why are you using pebble exactly? I was interested in seeing how you're managing your geo databases because that's usually the most mind numbing part of handling analytics if your cloud provider doesn't add that information into the request header already. However, I can't understand why you'd use pebble over something like sqlite. |
|
|
> Why protocol buffers ?
They are very good for defining API boundaries, in vince we only use them for configuration and admin structure. We use Roaring Bitmap based storage, so fundamental units persisted are Bitmap containers.
> Also why are you using pebble exactly?
Well, vince is write heavy and any LSM based key value store would have been nice. It happens pebble is the best option for us.
Also, we don't use transactions (We batch writes and use snapshots for reads). Combining with the fact we rely on pebble batch Merge api.
The merge api allows us to do efficient updates. Since we only store bitmap containers, when doing update we just do a container union of observed values of a key.
Bitmap unions are pretty fast and efficient.
I hope I covered all your questions.