|
|
|
|
|
|
by jeremyloy_wt
849 days ago
|
|
|
> I’m going to take issue with to you’re use of the word never here Perfectly fair. I shouldn’t have stated it as an absolute. If I could still edit it, I would change my statement to “don’t default to a mock” > Would you mind explaining what you mean here Gladly. Martin Fowler provides really clear definitions for the different types of test doubles. https://martinfowler.com/articles/mocksArentStubs.html Maintaining that your Fakes are correct takes work. An easy way that I’ve found to do that is to run the tests against a “real” component and the Fake component with the exact same set of assertions and set up. If that test breaks, then you know that consuming code should also break |
|
|
I haven't used fakes so take these critiques with a grain of salt, but I have two concerns with fakes:
1. The process you're describing to verify fakes does take work, and sounds suspiciously like it veers into testing your tests. That's probably worth it for applications like the space shuttle or an X-ray machine that need an extreme degree of reliability, but seems pretty overkill for the kinds of applications that make up most of the work in the software industry.
2. More often than not, I can't think of a useful fake for most things I'd use mocks for. The in-memory database mentioned by Fowler is an exception: you're skipping writing to disk, which saves a lot of time. But in most cases, the fastest implementation of a unit is... the unit, because if it wasn't the fastest you'd use the fake instead of the unit.
And all this sort of sets aside the larger issue which is that there are plenty of cases where the unit you want to test isn't the interface between A and B, so having a real B in that test just adds unwanted dependencies. The ideal here is that if you break the interface between A and B, one test fails--the test of that feature of the interface between A and B--which tells you exactly where the bug is. With an LLM, for example, this could look like you calling the real LLM and just testing that the call doesn't call any errors and returns syntactically valid output (even if the output is unpredictable, you can verify the syntax). If you're using fakes everywhere, then a failure of the interface between A and B is going to cause failures in a bunch of A tests, which makes debugging harder, not easier.
The case of the in-memory database makes sense because a general purpose fake (the in-memory database) can be written by a third-party and act as an effective mock for all the places you're using the DB--it's easier to use the in-memory database than to mock all the places where you call the DB (and notably, I've never seen anyone write a bunch of tests to test the in-memory database as you describe being a good idea for fakes). And in most cases, I'm using mocks because it's not easier to use the real thing.