[vishnugupta]

You get hands on experience at a place

1. That has a functioning app

AND

2. Business is growing rapidly bringing more customers than the system can handle.

In other words you learn on the job by getting your hands burnt. I got lucky to have joined such a startup. Learnt a lot, from fixing DB queries, designing asynchronous order processor, using CDN etc. I worked on scaling up the full stack including stuff like connection pools.

So your best bet is to join such a startup. You will learn a lot by handling real user traffic. And also scaling isn’t homogeneous. For example, you make different trade offs scaling a search application Vs scaling a payment processor. So business use case and business domain does matter.

[jiggywiggy]

I've got my hands burning over the last 12 years in 4-5 places like that.

From web & app, frontend, to backend, databases and server management.

Yet I still find it hard to grasp good system design.

At best I prefer things to be as simple and clear as possible, that's about all I'm certain off.

Most systems are already so established if they get going, you are mostly patching. Once you start patching all effeciency goes out the window, especially if those who build v1 are gone. Hard, and most often a bad idea, to overhaul a system to something more stable & efficient.

Now that I think of it, probably my biggest learnings are what not to do, not what to do.

[brabel]

> you learn on the job by getting your hands burnt.

That's how it works in new fields where there's no good theorical foundations yet. I think OP is asking if we already know how to avoid getting hands burnt in the first place, as in many other fields of engineering where you don't just learn on the job and get yourself burnt a few times.

[capableweb]

This is why I think "software engineering" is more of a creative field than a "hard engineering" one. More akin to painting or singing. You need practice in order to improve solving problems in the real-world, you can't just read up on it and become better than people who've actually implemented useful things in real scenarios.

[brabel]

I think that in the area of system reliability, specifically, there's quite a lot of theory that's useful and often ignored by practitioners, though. Saying you need to learn everything on the job is akin to cowboy system design. There's a lot of collective knowledge you should absolutely use when designing large systems, and ignoring that body of knowledge by assuming the only way is learning on the job is a mistake.

I believe not long ago, medicine was like that. Clearly, things can, and should, become more standardized and that will not in any way encumber the creativity that is also a big part of our field.

[tootie]

This only works if you work someplace that has complicated problems and/or architecture astronauts who like to overcomplicate everything. Every small, mid and even most large-scale tech shops are able to execute on goals using off the shelf software. How do we scale payments at my org? We pay Stripe. How do we scale search? We pay Alogolia. How do we scale our public facing web sites? We pay Amazon. I've been in architects/director roles at multiple orgs for over ten years and everything they ask at FAANG system design interviews is still stuff I've only read about.

[tleilaxu]

To add something further to this excellent comment:

Join a start up with a legacy code base, e.g. dodgy old PHP that they’re pulling out into modern domain services.

Within a year you’ll be able to analyse performance and scalability issues and refactor them in your sleep.

[beebmam]

If you want to build your skills, sure go ahead.

But working at a company like this is not a good way to build your wealth. You're far better off working at a big tech company that has awesome benefits or a fresh startup. Startups that have been around long enough to have a legacy code base aren't a place to become wealthy.

[capableweb]

> But working at a company like this is not a good way to build your wealth

Common software engineering mistake: focusing on solving the wrong problem.

OP wants to practice hand-on system design. OP never writes about "building wealth" and doesn't even mention money anywhere. So where do you get that this is about building wealth?

[xiwenc]

Not everyone aims to be wealthy. Most companies do pay a decent salary (specially in europe) with good work life balance. High paying jobs often correlates with high stakes which means potentially higher level of stress.

[tomduncalf]

I don’t think the high paying job = big stress correlation is as strong as you might think. Sure a few places are known for bad work life balance but (especially if you are employed in Europe) I would guess many big companies offer better WLB than startups - in fact, I’m sure the pace might be “too slow” compared to smaller companies in many teams.

There are plenty of valid reasons to not want to work at big tech co’s, but if you’re discounting it for expected stress level alone I’d do a bit more research as I don’t think it’s as true as you think. Obviously read reviews of Glassdoor/Blind to get a sense of the company you’re applying to…

[throwaway7173]

I think the comment meant "wealth" in the pg sense https://www.paulgraham.com/wealth.html

[nathias]

Not necessarily, the biggest most rapidly growing project I was with only taught me anti-patterns, but I guess that's something you need to learn too ...