[efxhoy]

I work at a 60 dev shop so not a startup per se.

Our biggest problem is feature environments, or actual integration tests where multiple services have to change. Because infra is in its own repo in terraform and the apps have their own repo we don’t have a good way of creating infra-identical environments for testing code changes that affect multiple services. We always end up with some hack and manual tweaks in staging.

Data engineering is another problem, managing how to propagate app schema changes to the data warehouse is a pain because it has to happen in sequence across repo borders. If it was all one repo and we got a new data warehouse per PR it would be trivial.

Not trusting CI to hold secrets is another. As soon as we do anything in CI that needs “real” data we need to trigger aws ecs tasks, because circleci has leaked secrets before so we don’t trust them and keep all our valuable secrets that can access real data in aws ssm. The more complex the integrations the harder they are to test.

If we had a monorepo I think this type of work would be much easier. But that comes with its own set of problems, mainly deployment speed and cost.

If there was a way to snapshot all our state and “copy” it to a clean environment created for each PR that the PR could then change at will and test completely, end to end, that would be the dream.

[mdaniel]

> Our biggest problem is feature environments, or actual integration tests where multiple services have to change

OT1H, :fu: terraform, so I could readily imagine it could actually be the whole problem you're experiencing, but OTOH it is just doing what it is told, so that's why I wanted to hear more about what, specifically, the problem is? too many hard coded strings? permission woes? race conditions (that is my life-long battle with provisioning stuff)?

this whole Ask HN is nerd sniping me, but I'm also hoping that we genuinely can try and find some "this has worked for me" that can lift all boats

[efxhoy]

The main problem is state, it’s too big and scattered. If it was just one db that knew everything that would be easy enough to snapshot, but there’s multiple dbs and s3 buckets and messages on queues and its all spread out. We could, with a bunch of work to make stuff actually portable, spin up a complete new environment, but a lot of the work that happens is related to synchronizing state and since all the state is scattered we can’t make an atomic clone of it.

Then of course there’s always the issue of people taking shortcuts, “i will test this once then we know it works then i’ll just hardcode this thing”. Making stuff truly impotent and portable is extra work for a “nice to have” of a feature env. YAGNI, until you do. For most devs they’re happy to have something that works and they can move on and ship the next thing.

Personally i’m a big fan of monoliths because they supposedly make this stuff easier. Then again i’ve never worked on a huge one and my colleagues much prefer spinning up an “isolated independent loosely coupled” service to adding it in the main app.

[oldandboring]

> Our biggest problem is feature environments

I'm always baffled to see how many shops claim to either not have this problem, or sidestep it. Every project I've ever worked on has had multiple enhancements/fixes in flight at the same time, they need to be tested and deployed independently of each other, on their own timelines. For this we need story branches and a fast way of deploying different story branches to different test environments. If you're merging everything into kitchen-sink "dev", "staging", "test" branches because someone drank the Gitflow kool-aid, your confidence goes way down that a specific story branch is "ready to go" and that production will behave exactly like dev and test. And, as you mention, accomplishing the story-branch approach across multiple repos (assuming the change is large enough to affect multiple repos) sounds like swimming with crocodiles.

[efxhoy]

I think it could be done with a monolithic app with a single db and truly stateless external services in a single repo. I only have experience from a single shop though and that’s not how the stuff we build ends up.