[TheOtherHobbes]

I don't understand this argument. Is it really likely that a spaghetti nest of Python, json, Hadoop, and Elasticsearch is easier to understand and to maintain than maybe 100-200 lines of self-contained Go code with minimal (no?) outside dependencies?

Or that new hires are going to think bit counting and XOR are ninja CS voodoo, while everyone already knows how to Elasticsearch?

How about cost? What's the monthly all-in cost for each approach?

This is a nice satire. It points out why modern software can be faddy, inefficient, fragile, hard to maintain, and expensive to run, with no obvious benefits (including "easy to understand", which seems to be code for "I already know how to use all those dependencies and APIs, so let's assume new hires will too.")

[opticalfiber]

Not at all. I can appreciate an elegant, problem-specific solution as much as anyone else. The point is that there are always tradeoffs.

What's the query interface like for the bit array? Does it even have one? Seems like you'd have to sit down and add more code whenever you wanted to know something new about the data. Would you write your own query layer? Would you eventually need a dedicated team to maintain it? You may say that is an extreme conclusion, but I've seen this very story play out multiple times in large dev organizations.

As amorphic pointed out, indexing the data with ES makes it easy to access even for non-technical users. Each layer of abstraction comes with its own costs – of course – but also its own benefits. Tradeoffs.

[pdkl95]

> What's the query interface like for the bit array?

I don't see any request for a query interface in the list of requirements.

The point of the packed-bit array is that you don't need any kind of query interface other than a few accessor macros. It's a memory-mapped array of 64-bit integers; you just index into the array and use a macro to mask out the bits you're interested in.

The "old-timer" solution - which actually solves the stated problem instead of a hypothetical future problem with lots of extra requirements - can be written in a few pages of C. It shouldn't take more than an hour, perhaps two (plus some time for testing).

If and only if the requirements change0 to something significantly more complex would something like a query interface make sense (YAGNI). The task as stated just isn't very hard.