[wyc]

Projects tend to gain more and more functionality to match the new workloads they're being used to accomplish, and it must certainly be a difficult decision for project visionaries. Do I listen to my users and implement features that will solve their new woes, but in return accept increased complexity and higher learning barriers? Complexity sucks, but it's even harder to say no to users in pain.

I wonder if this opens such projects up to disruption, in the original sense of the word. I've seen the same teams forgoing apache for nginx, then forgo nginx for haproxy when it matches their needs. With additional layers of complexity, there may be accompanying opportunities for "simple but good" projects to gain traction.

[antirez]

I hate complexity, and because of that I spent a lot of time creating the Redis modules subsystem so that I can take a very small core. However general data structures like streams are, in my opinion, not bloating Redis, which is still, in the "streams" branch, at just 85k lines of code. For two reasons: 1) Data structures are self contained beasts in Redis, they do not interact with other features to multiply complexity. Other features are instead like that, for instance expires interact with replication, AOF, scripting and so forth. 2) IMHO bloating Redis is to add too narrow use-case specific things insdie it. But as Redis has lists, hashes, sets, ... the "log" really was part of the general purpose things that Redis was lacking. As a result of this lack, people used something else adding complexity inside their code in order to model the same problem with a wrong tool, like sorted sets or lists or Pub/Sub, which are good for certain things but not for time series or certain events streaming tasks. Btw the fact that after around 8 years we are at 85k likes of code, totally understandable by a single individual in a matter of weeks, positions Redis as one of the simplest simple software projects out there.

Another data point, streams are if not completely, at least 70% done as we talk, and yet `git log --stat unstable..streams` reports:

12 files changed, 1323 insertions(+), 10 deletions(-)

We'll end at 2000 lines of code I guess more or less. Just to say, Redis is at a complexity scale very far from many other things we are used to these days.

[barrkel]

It's also why software is a pop culture. Sophistication and completeness is seen as complexity and cruft by each successive generation, who start something new and simple.

I don't think it's very avoidable. Tech is genuinely getting incrementally better, but it's usually in a sawtooth pattern.

[freshhawk]

I'll agree that sophistication and completeness is often seen as complexity/cruft but it also always comes with actual cruft as well since improvement is incremental and breaking APIs is annoying.

My favorite aspect of this cycle is when some features in the complex software become seen as so useful as to be required and standard, so when the new simpler version is created they have to figure out a novel way of providing that useful functionality in a simple and elegant way. And they do it, sometimes knowing they have made a significant advance and sometimes without knowing.

[scott_karana]

Do you have any examples of that second case?

Sounds too good to be true... but I'd love to be proven wrong :)

[derefr]

I'd agree in general, but Redis is architected in a way where these features aren't really new "layers", but rather just horizontal modular additions.

A new data structure + associated commands being supported in Redis is like, say... a new filesystem being supported in the Linux kernel. It's a few files that you could just avoid compiling in if you didn't want them, and which add code-paths that are never run unless you intentionally use that specific new thing.

[takeda]

> I'd agree in general, but Redis is architected in a way where these features aren't really new "layers", but rather just horizontal modular additions.

> A new data structure + associated commands being supported in Redis is like, say... a new filesystem being supported in the Linux kernel. It's a few files that you could just avoid compiling in if you didn't want them, and which add code-paths that are never run unless you intentionally use that specific new thing.

Isn't it exactly how features are implemented in Apache? Most people think that Apache is too bloated, but the same people also leave a lot of those modules enabled without trying to spend time to understand if they actually need them.

[derefr]

I made that very specific comparison for a reason: most "plugin systems" for software (like Apache) act sort of like audio VSTs—they can insert themselves anywhere in the "processing chain" of a request, making following the logic more complex.

Redis modules, like filesystem drivers, are comparatively simple: they register a set of commands that they respond to, and each such command is handled exclusively by that module when received. Less like Apache plugins, more like scripts in a cgi-bin. You don't need to know what else is in the cgi-bin besides your own script, because everything there is entirely independent.

[jankotek]

Look at the source code. Redis is very simple, single threaded, primitive networking... I would hardly call it complex even with this extra feature.