[ryanbrunner]

But in the case we're talking about, where you're reliant on an external service that can change underneath you, "world looks like X" is genuinely not under your control. It feels like pretending that it is will lead to just as many failures as acknowledging it's inherent volatility.

[stephen]

Agreed! And, to me, record/replay is still pretending like it's controllable, b/c even if you decouple for replays, records will always be a PITA.

My depressing solution is to just not even try to automate tests against the upstream system and instead invest in test builders/DSLs that make mocks/stubs on both sides as pleasant as possible.

And when bugs slip through, make sure to update your stubs/mocks on both sides to prevent the regression.

To me this gets the most agility and reliability, and will be a test suite that developers don't hate 1-2-5 years down the road.