[0xbadcafebee]

> it is really simple to create the basic "database". You can start by using the dictionary data structure that comes with whatever programming language you're using and slap a web API on top of it.

Better yet: do it in C. There's no "dictionary" object type so you have to make it yourself. You'll soon learn a whole bunch of fallacies about how those "dictionaries" actually work. After you spent a good deal of time doing that, you can switch to authentication/authorization, logging, storage, tracing, API management, resource quotas, and a raft of distributed computing issues.

I recommend basing it on Consul, it has a better general model than etcd.

[SilasX]

Relatedly, I loved microcorruption.com, a RE/CTF challenge where you have to find exploits at the assembly level to bypass a lock. Some of the problems have hashtables implemented in assembly that you have to work through with a debugger, and it forces you say e.g. "okay, that register is storing the number of bits needed to store the [log of the number of] hashtable elements, which is then masked over the hash function output to know what bucket it needs to look in".

[avl999]

I disagree with the C advice. If you don't want to rely on higher level language primitives like Dictionaries and so on there is nothing stopping you from rolling out your own implementations of these data structures in a more sane language (Go, Python, Java, Clojure whatever, anything but C). The project is challenging enough without having to fight segfaults and undefined behaviors.

[iainmerrick]

Writing it in C is good advice if you want to learn C (or you already know C but want to understand it better). It’s bad advice if you don’t want to learn C.

Why would you want to learn C? To better understand the machine at a fairly low level. I think there’s still a lot of value in that. I’ve found that programmers who never learned C often don’t fully understand how memory management works, for example (not that that necessarily makes them bad programmers!)

Most other non-garbage-collected languages would do the trick, like Rust or C++. But C arguably still has special value in that it’s a lot simpler than either of those -- no higher-level constructs or abstractions to distract you. Maybe Zig will be able to take over that role.

[avl999]

Learning C however is completely orthogonal to learning and building challenging projects (in this specific instance a distributed datastore) that the OP is talking about. It is not going to make you understand the domain better or provide any other tangible benefits. If anything it is going to add negative value by distracting you from $TheChallengingConceptYouWantedToLearnInTheFirstPlace

If I am working on a challenging project recreationally to stretch the limits of my comfort level- say a compiler or a distributed system, the last thing I want is the cognitive overload of trying to deal with C unless I am a C expert already. When working on such projects you should be focused on grokking the problem being solved as opposed to worrying about language idiosyncrasies.

If you want to learn C, it should be done in an environment where it is decoupled from trying to learn another extremely challenging idea. So I definitely would not recommend someone interested in learning about distributed systems attempt to build one in C. If you are trying to learn C you probably build something that you are quite familiar with so that your focus is on learning the language and its concepts as opposed to trying to implement/understand the ideas behind Paxos while also trying to learn safe memory management and pointer tricks.

[0xbadcafebee]

I agree that the complexities of C will add to the challenge and workload and make the project take much longer. But certain aspects of C can raise interesting questions that you might not face otherwise. Some of those things will be about algorithm design, some about language design, some about computing in general. Especially if you spend most of your time in high-level languages, it can really help to have this new perspective. Some of it makes you question modern software development.

Also, I learned a lot more about C (and computing in general) by working on complicated projects rather than simple ones. I had built all sorts of simple C projects, but one complicated one taught me twice as much as I'd previously learned. It took me a long time, but it was all valuable because I got to learn what designs didn't work for C and which did.

Anyway, it's up to the reader to decide how much of their time to invest and what they want to get out of it. If you don't want to write it in C, don't, but I personally think the incidental lessons are some of the most valuable.

[iainmerrick]

Agreed -- writing toy programs generally doesn't teach you a whole lot about a new programming language. To learn it in a deep way you need to use it properly, on real problems.

Although, if you're mainly interested in getting something done rather than using it as a learning experience, using an unfamiliar language is probably a bad idea. You'll learn a lot about a new language, but you probably won't build anything you could actually ship.

[FartyMcFarter]

> Most other non-garbage-collected languages would do the trick, like Rust or C++

Only as long as you avoid any fancy libraries that do a lot of the work for you.

Granted you can use fancy libraries in C too...

[0xbadcafebee]

It's worth it to try to implement in a low and high level language, to see the huge differences in how they can be implemented. We all know "C is fast" but it takes a project like this to go "OH. It's because the high level language is abstracting me into a corner." It was a trip when I first realized how much high level languages can limit you.

[ghoward]

I'm not sure I agree with you. Writing a hash map implementation in C made me a much better programmer as far as performance goes because I understand the machine better.

[glouwbug]

I've been back porting parts of the C++ STL to C, so it may have a dictionary type now, depending on who you ask!

https://github.com/glouw/ctl

[CBLT]

I'll second this suggestion. Most of my toy projects aren't in C, but every time I design software in C I learn something.