[mikekchar]

I also don't like recording frameworks for TDD (and similarly dislike using fixtures). However, the place where a recording framework really pays for itself is in isolating changes in protocols. I've often had to interface with, as you put it, uncontrollable upstream systems. These are systems that are not mine -- they are upstream services from other companies that I have to interact with and I have no control over. Often these systems are badly built and they play fast and loose with the "protocols".

In these case I like to have an adaptor layer and use a recording framework to "test" the adaptor. That way I can occasionally rerecord my scenarios and be notified if something important has changed. Normally what happens is that my service stops working for some unknown reason. I rerecord the adaptor scenarios and usually the reason pops out very quickly. All the rest of my code is coded against the adaptor and I stub it out in their tests (which I can do reasonably well because I control it).

[m12k]

I've worked for a while with similar scenarios of needing to integrate with systems beyond my control (e.g. Stripe, Slack, Google), and though I still don't have a good setup for it, I've come to the conclusion that a two-pronged approach would be ideal: Record/and/or stub the calls and responses from the external services so your normal tests are run entirely without external networking (like what Polly allows you to do). But also set up a server to periodically validate the responses from the external services against those that have been recorded, and alert you to any changes in their protocol/behavior. I've yet to see any middleware to tackle the latter (though granted, I haven't been looking too hard yet either)

[stephen]

> isolating changes in protocols

That makes sense.

Ideally protocols are declarative/documented/typed, e.g. Swagger/GRPC, so you can be more trusting and not need these, but often in REST+JSON that does not happen.

> All the rest of my code is coded against the adaptor

Nice, I like (the majority?) of tests being isolated via that abstraction.

Although, if "the protocol" is basically "all of the JSON API calls my webapp makes to the backend REST services", at that point do you end up adapter scenarios (record/replay recordings) for basically every use case anyway? Or do you limit it to only a few endpoints or only a few primary operations?

[mikekchar]

Yes, I like to have scenarios for all of the end points. It definitely doesn't reduce the number of tests :-) The advantage is in isolating the protocol from the operation of the application.

The main complaint I've seen for this approach is, "We shouldn't be testing the other person's system". There is wisdom in that advice, but it really depends on how much you depend on the 3rd party service and how much downtime you can tolerate. For example, I'm working in the travel industry right now and we often rely on small services that nobody has ever heard of. If we can't use the service then we can't sell anything and our site is essentially down. If it happens frequently (and with a lot of these travel services, they often break things weekly if not daily), then your site is not viable. In that case I'll exercise the protocol as much as I can. However, we also talk to marketing services, etc. If that breaks, and it takes a day or two to get it back up, then it's not a major problem -- our marketing effort might be a day late, which is unfortunate, but not game breaking. In that case I'll usually have a smoke test or two.