[mabbo]

What bothers me about this article is that it doesn't really present alternatives or real-world examples of how to do what the author is saying.

> Use real instances of managed dependencies in tests.

Is the author suggesting I should spin up an entire instance of Oracle to run a single 50ms unit test? Because I often want to run my unit tests very frequently as I develop, ensuring I'm still green, etc. And my code may depend on an large, complex database. I could have a continually running instance just for my unit tests, but that's expensive (time, money, hosts, etc) and means I can't run my unit tests if the wifi goes down.

If we're talking about large, complete systems running very large suites of system tests as part of a CI/CD pipeline, then sure, I'm on board. Let's use a real database. But there's lots of other small, simple tests that absolutely don't need anything more than a mock while I work on business logic bugs.

And no, I'd not like to get into a debate of "you shouldn't bother to write small unit tests". I find them very useful.

[grey-area]

When you're developing, aren't you running a db server with a dev database for that? Just add another separate test db and use that for tests where appropriate.

Also, most unit tests don't need a db, it's more useful for integration tests.

[amw-zero]

Hitting a database in tests is an order of magnitude slower and also involves worrying about a lot more code, i.e. the database engine. You may or may not want to take on that burden.

[grey-area]

I use the db for some tests, it's totally fine, because it's limited to those tests which actually need db access (most don't). It's also closer to the actual application behaviour than mocks and doesn't need to be maintained.

[pc86]

If you're using a Repository pattern or similar you'll have a lot of unit tests where a database or db provider is an injected dependency.

[amw-zero]

What I got out of the article was that, in unit tests, you should stub the response of a database query. However, you shouldn't directly assert that some database API method is called, because that is one additional level of coupling.

What I don't understand is, a stub to me is almost the same level of coupling as a mock. Your test setup needs to know about the dependency on what you're stubbing. So when you want to change the interface of the dependency or use a different one, or remove the dependency, your tests have to change.

It's not like you can add a query to an operation that didn't have one previously and the result will magically appear without connecting to a real database. So you have to modify the tests to provide the stub. Seems like a logical contradiction to me.

[bluejekyll]

Right, the testing pyramid. Your largest set of tests should be a foundation of unit tests that run quickly. Then integration and or functional tests, which is the section this is talking about.

They are still necessary, then move up the stack and you get into the end-to-end tests of a fully running system.

The reason isn’t only speed but also signal to noise ratio. The further up the testing pyramid you go, the less clear it becomes where errors were introduced.

[pydry]

However, the further down the pyramid you go:

* The more unrealistic the tests generally become (larger % of false positives - tests that fail when they shouldn't - and false negatives - unit tests that simply don't catch bugs at all).

* The less reusable the test infrastructure becomes. Stubbing/mocking individual method calls to the database is an ongoing cost of development whereas building scripts to start the database and shut it down is an investment cost that pays dividends.

On the whole I think the pyramid idea enforces a wrong view that there is a "right" mix of "test levels" across any project. The best mix is determined by the kind of bugs and code you have (integration vs logical) and the kinds of abstractions you need or already have (in general, the worse your abstractions, the higher level you need your tests to be).

A lot of projects are best done with 100% integration tests while others can be done with 100% unit (especially small, self contained, simple-to-interact-with code bases that are 99% about calculations/logical decision making).

[amw-zero]

I've started to agree with this view of the test pyramid as well. This is a great video overall, but here's a part where Aslak Hellesoy (creator of Cucumber) is talking about how a better way to think about the test pyramid is as a [spectrum of speed & reliability](https://www.youtube.com/watch?v=PE_1nh0DdbY&t=12m55s)

If a test isn't a pure unit test, where all collaborators are stubbed out, but it is still fast and reliable, it is still a very valuable test. Possibly preferable since testing multiple collaborating objects / functions provides more confidence than just testing one by itself.

[bluejekyll]

With all the projects I've worked on, I've always found unit tests to be the best possible place to catch bugs and errors, because it's faster and easier to diagnose the root cause.

That said, they don't cover all test cases, which is why it's a pyramid. What I have found is that when an bug arises and is caught in an integration test, it's often beneficial to create a unit test that helps catch the same error before it you get to the integration test area, not always, but definitely if you have something that fails more than once in an area. Unit tests should never have false negatives, there's something wrong with the test if that is happening.

That being said, tests are designed around the code that is being tested. As technical debt and refactoring of existing code happens, you do often need to rework tests. Many people allow tests to go unrefactored, and they become their own set of technical debt, but that doesn't mean that they don't have value.

[pydry]

>With all the projects I've worked on, I've always found unit tests to be the best possible place to catch bugs and errors

Have you considered that that might be due to the nature of the projects you've worked upon rather than the nature of unit tests themselves?

>it's often beneficial to create a unit test that helps catch the same error before it you get to the integration test area, not always, but definitely if you have something that fails more than once in an area.

It depends upon what the bug was. Sometimes it's possible. Sometimes "replicating" it with a unit test is expensive and largely pointless since the unit test won't catch that class of bug in the future and will break as soon as you change the code (e.g. I've seen people try to create unit tests that mimic race conditions before and the results were horrendous to read, pointless, and didn't even catch race conditions).

>That being said, tests are designed around the code that is being tested. As technical debt and refactoring of existing code happens, you do often need to rework tests. Many people allow tests to go unrefactored, and they become their own set of technical debt

The higher level and the more behavioral the tests are, the less they have to be changed when the code is refactored and the more confidence that they give you that the code actually works afterwards.

The absolute worst situation to be in is with a bunch of unit tests that are tightly coupled to code that needs refactoring. Those unit tests' breakages signal nothing except that you've changed some code and they demand expensive repairs before breaking again in the future - again, because an API endpoint was refactored, not because a bug was introduced.

[bluejekyll]

> Have you considered that that might be due to the nature of the projects you've worked upon rather than the nature of unit tests themselves?

Yes, and I have yet to see a set of code that doesn't benefit from some set of unit testing.

> I've seen people try to create unit tests that mimic race conditions before and the results were horrendous to read, pointless, and didn't even catch race conditions

I didn't say put in unit tests that are crap code that don't do what they're supposed to do. If the test doesn't catch what it is supposed to (i.e. you wrote the test and it didn't reproduce the error), then it's a worthless test.

> The higher level and the more behavioral the tests are, the less they have to be changed when the code is refactored and the more confidence that they give you that the code actually works afterwards.

> The absolute worst situation to be in is with a bunch of unit tests that are tightly coupled to code that needs refactoring.

Unit tests should be tied to the code they are testing. In general, if the API of the code changes, then the unit test will need to change, this is a no brainer.

It sounds a bit like we're talking about language level deficiencies vs. testing issues. Folks working with languages that do not have strong types, and a way to validate them, definitely makes it harder to maintain when the API changes.

For typed languages that allow you to quickly discover usages of an API, it is far easier to maintain unit tests, as they tell you immediately what has changed and what needs to be updated, before ever running the tests.

[kerkeslager]

> The reason isn’t only speed but also signal to noise ratio. The further up the testing pyramid you go, the less clear it becomes where errors were introduced.

I disagree. A failing unit test doesn't even necessarily indicate that an error was introduced: if a unit doesn't do what it's supposed to but the user doesn't see that, then an error wasn't introduced. Sure, when a unit test fails there's often an error, but if an end-to-end test fails, there's always an error--E2E tests are testing from the user's perspective, so what they're testing is actually errors. (This is assuming that both the unit tests and the E2E tests are correctly written).

You're positioning unit tests as a debugging tool, but I'd argue that there are much better debugging tools: REPLs and debuggers give you a lot more information than a unit test, and allow you to ask new questions quickly.

I don't want to come across as being anti-unit tests. On the contrary, I think unit tests are highly valuable. But I don't think the value comes from gathering information, debugging, or even catching bugs (in most cases). I think the value comes from a few things:

1. TDD forces you to design units for reuse from the start. Immediately you're using the code in two contexts: the application and the unit test. So right away your code is inherently reusable (in a binary sense) because de-facto you've re-used it. Reusability is more complicated than that (it's really more of a spectrum than a binary) but having at least two uses from the beginning pushes you toward the reusable side of the spectrum.

2. Unit tests act as living documentation for units. It is often unclear by reading code what the code does, because production concerns such as performance and security can lead you to do things in seemingly complicated ways. But unit tests don't have these concerns (at least not in the same way) so you can write code in unit tests that clearly communicates what a unit does. And unit tests don't fall out of sync with code like plain text documentation does.

3. TDD is incredibly motivating. Moving red->green on a quick cycle takes advantage of the dopamine reward system to increase productivity.

[barrkel]

Unit tests on code that has no dependencies (and I mean no dependencies, neither injected nor direct) is great.

Unit tests on code with dependencies (whether injected so that they can be mocked, or directly referenced so that they end up more like mini integration tests) are less excellent. They're brittle, inhibit refactoring, and either don't test as much as you think they do (if mocking dependencies) or are slow (if not mocking).

The further up the testing pyramid you go, the less work it is to refactor things, because you don't need to rewrite as many tests. OTOH test are more complex to write and take longer to run.

And now I get to my point: I don't think the blanket statement of "your largest set of test should be ... unit testst that run quickly" is well-founded. There are trade-offs, and they shouldn't be trivially waved aside.

[sanderjd]

Do people usually mock the database at the integration / functional level? I haven't seen that... I interpreted this to mean the unit test level, which similarly to the thread starter, seems crazy to me.

[bluejekyll]

If you think of the DB like a service, then yes, people sometimes mock it. IMO, the danger comes when you start trying to mock things and act like a DB, i.e. you create some in-memory store to act like a DB. That's dangerous because then you're potentially introducing issues that are completely unrelated to the DB operations, and therefor not testing anything of value.

But it can be far cheaper to generate data through a mocked interface, than say fill a DB with data, and test against that data-set. Obviously there are ways to structure your code such that the DB isn't part of the data flow at all, but sometimes existing code structure isn't perfect.

[stickfigure]

IMO, the classic test pyramid with integration tests at the narrow top and unit tests at the wide bottom is inverted. We test software that way because integration tests are hard and unit tests are easy, but integration tests provide more value than unit tests. It's the streetlight effect: "Did you lose your car keys here?" "No, but the light is much better here."

I built some framework code so that integration tests as easy to write as unit tests. Almost all of my tests are "integration tests". I never mock the database. My test harness clones a template database at the start of each test; that database is maintained with migrations just like every other database. I run against test accounts at all the 3rd party services I use. It's not super fast - a full CI run takes about 15 minutes. And occasionally a 3rd party service will cause a test to flake. But it's fast enough and most importantly it's thorough.

I still have some unit tests that run without the expensive setup harness, but they're for components that have a lot of algorithmic complexity. Anything that touches the database gets a real instance.

This works pretty well with Postgres, which is free and I can run locally and has a fast db clone operation. YMMV with other databases.

[Tainnor]

That's not the sole reason for avoiding too many integration tests. The main reason for that IMHO is combinatorial explosion of code paths which you'd have to test for if you wanted to be thorough.

[tybit]

A big part of the authors unit testing philosophy is that you should separate IO from business logic.

That way you can unit test without mocks and without heavy real dependencies either, and leave that for integration tests.

[davedx]

> Is the author suggesting I should spin up an entire instance of Oracle to run a single 50ms unit test?

So what we do where I work is for unit tests, we mock services and repositories, unit tests don't go anywhere near the database.

For integration tests, we use an in memory database.

BUT! Be careful before you embark down the path of running integration tests against a different database than the one you use when running the application. There are SO many pitfalls, nasty bugs, and other warts along that road. EntityFramework alone has quite some weirdness there. Expect these kinds of integration tests to cost a lot of developer effort to build and maintain. For us, it took months of effort to get these kinds of tests working usefully.

Personally after working on applications that have a solid test pyramid, I would recommend:

* Write unit tests as per the standard advice (bottom of pyramid kind of quantity), but try and keep it sane (don't go for 90% coverage just for the sake of hitting an arbitrary number; don't pointlessly unit test your framework/libraries/other dependencies)

* Write some integration tests where they really add value (interaction between 2 or a few complicated components in your system, for example places where the state of a component changes a lot depending on input from another component). Make sure when you start out writing an integration test that it doesn't turn into an "almost" end-to-end test along the way. They have a habit of doing this and it can really cost you later. Integration tests should still be focused.

* Write end-to-end tests that test as much of your system as possible, including a database (preferably the same vendor; one approach is to truncate all tables before each run). IME it is very good here to have one e2e test that covers a big chunk of functionality in one run, than lots of separate tests covering different things. Why? Because no matter how "different" the things being tested by the e2e test there will still be LOTS of overlap by its very nature, and changing anything where that overlap is will tend to break ALL your e2e tests. Not a fun workflow.

One last comment on E2E tests. This is more opinionated. But try and limit E2E tests to things that are really business critical (e.g. your "sign up" process and your "renew subscription" process, but not every single form in your dashboard). This is just a cost-benefit thing. E2E tests help, but they also slow you down, and the more you have, the more slowly you will be able to change your software. Sometimes minor bugs slipping through is an acceptable cost if it means you release 1 week sooner.

[matwood]

> For integration tests, we use an in memory database.

I think this can work for running the tests locally when speed is what you want, but you should still have the tests run on the actual database on commits. It requires keeping a database running just for tests to run (it can be a smaller instance and startup/shutdown on demand), but it will save so much pain later.

[davedx]

Yes I thought I covered that here:

"BUT! Be careful before you embark down the path of running integration tests against a different database than the one you use when running the application. There are SO many pitfalls"

[dtech]

With docker and the other arguments the article points out I find there's very little value in an in-memory database. Just use either a stub/mock/dummy for unit test or spin up the real thing for integration test.

[pc86]

I've yet to have a good testing experience when using anything more than a trivial database if it has EntityFramework on top. The de facto standard seems to be Sqlite for unit tests, which causes some blocking issues if you have the same database name in two different schemas, or composite primary keys, to use my two most recent examples.

[regularfry]

I find it utterly bizarre that this should still be a current topic of contention. Running unit tests against a local database was a norm in Rails a decade ago, because the framework made it trivial to set up.

Yes, there were knock-on performance questions that wanted answering - how do you avoid the database setup costs for those tests that don't care about it, for instance, the "fast Rails test" movement was a big thing - but by and large those were solved problems by the time I stopped writing Rails code professionally around the 5.1 era.

The answer is, of course, no, you don't spin up an entire instance of Oracle to run a single unit test. You run against a local instance, and you use whatever tricks your-RDBMS-of-choice gives you to make resetting the test tables to a known-good state extremely fast. That way you can have your tests running continuously, giving you fast feedback as you develop, only suffering db overheads when you actually need to run tests that hit it.

If your chosen stack makes it difficult to do this, it's worth asking why.

[lifeisstillgood]

If I am honest, at this point I would rather not use a mock, but a stub / injection.

I know this gets horrbily pedantic but its easier to see in code

def difficult_function(dbconn, a,b,c): dbconn.execute(Select * from tbl) <comlicated stuff invlvoing results set and a,b,c>

I would not want to mock the dbase at this point. Please can we instead do this

def difficult_function(dbconn, a,b,c): resultset_as_dict = dbconn.execute(select * from tbl) insider_function(a,b,c,results_set_as_dict)

def insider_function(a,b,c,results_set_as_dict): This can now be tested without mocks quite easily.

I think if you are doing 'difficult' stuff with a exernal database you are de facto, writing integration tests.

In fact i would say anything involving a database MUST be treated as a integration test. If it takes ten minutes thats fine - its an integration test.

If you want fast and external connections, use sqllite as part of your testing CI suite. But dont moan.

and do not use Mocks.

[BenoitP]

> should spin up an entire instance of Oracle

I'm sort of doing that at the moment, with postgres and series of tests. But it is still useful for unit testing too: pull up the right CREATE tables, INSERT your test data, execute your tests, then drop tables (lots of safeguards here), and repeat. The container loads up in 1.5 sec, and all my tests (~100) are done in 10 seconds.

It's been great in my use case. I'm on postgres, Java, and using testcontainers. They have handy containers available, not limited to RDBMS and not even limited to databases; and here is Oracle Express edition, which should be enough for most tests:

https://www.testcontainers.org/modules/databases/oraclexe/

[je42]

yep. the test pyramid helps figuring when to use what test setup. https://martinfowler.com/bliki/TestPyramid.html

[jaitsu]

Writing unit tests is debatable?

[commandlinefan]

On HN, it's not debatable: you should definitely do it. Everywhere I've ever worked, it's also not debatable: no time or effort should be "wasted" writing unit tests, that's what QA is for.

[swat535]

> no time or effort should be "wasted" writing unit tests, that's what QA is for.

This is not my experience at all, everywhere I worked unit tests are required. I don't agree at all, writing unit tests is not a waste of time, this sentiment comes from people who don't know how to avoid brittle tests.

It's impossible for QA to test every single path in your code. Unless you wanna cover all of those in your slow end to end tests?

[commandlinefan]

Oh, you're preaching to the choir here, pal. It's incredibly frustrating. I have reason to hope, though... when I first started programming in the 90's, I would fight with my co-workers about using version control (I insisted on it, they said it was a waste of time). Now, people who oppose version control are off selling life insurance or whatever happens to people who probably shouldn't be writing computer programs.

[ashtonkem]

That argument gets floated occasionally.

The utility of a unit test suite is roughly proportional to the amount of in-process computation you do. “Glue” systems which mostly transform from one protocol to another usually need more integration systems than unit, making skipping unit tests not disastrous.

[squiggleblaz]

On legacy code? Alas. The amount of changes you might have to make to make the legacy code unit testable versus the benefit of unit tests can be exceedingly significant.

For instance, if the implementation involves calling up a bunch of value objects from a database, each of which do the same, and all of the code is inhouse so there is no standard mock or stub libraries available, adding unit tests is tantamount to rewriting the whole system without tests.

Existing codebases can also be too complicated for a few people to formalise into unit tests. The algorithm itself might be simple, but again, to discover that, you need to rewrite the whole system without tests. (You can add tests, of course, but when you're adding tests, you're making an assertion you cannot prove. Since you don't know what the code is meant to do, you don't know whether the tests are complete or even accurate.)

Once you're coding in a world where tests passing or tests failing has almost no predictive value on success or failure in release, you're working in a self-fulfilling prophecy where the code will never be tested because the tests literally make things worse.

Many codebases clearly do not have any automated tests at all, but unit tests can be the hardest to add onto a system after the fact.

[Tomis02]

Unit tests are code, and all code has a cost. Writing unit tests without any benefit is harmful behaviour. Real world example: testing dumb getters/setters.

[packetlost]

That's why I write large sweeping tests that make coverage metric to up, but don't take too much time to write or maintain /s

[allendoerfer]

Writing getters and setters is harmf... - okay, I will stop before things escalate in here!

[Tomis02]

If you live in OOP land and have a natural aversion to directly accessing the data then by all means, write all the getters and setters you want. But testing that "int getX() { return X;}" actually returns X is pathological behaviour. It's harmful because 1) it costs to write 2) it costs to read 3) it takes up space, distracting from more important tests 4) it costs to run 5) prevents you from easily making changes (it's a brittle test). Can't really see any benefits; if you manage to screw up writing a dumb getter, how do you trust yourself that the test code isn't wrong as well?

[uk_programmer]

Yes. For a lot of smaller (web) applications where it is obvious if it isn't working correctly it just become a time sink. Everything in engineering is a set of trade-offs.

With regards to TDD specifically. It isn't for everyone and many consider it to be a bit of a cult. While I don't consider it a cult, It doesn't work with how I personally solve problems. I normally for example get something extremely rough working and then iterate until I consider it to be perfect and then write my tests to define how it should behave.

[jerf]

Oh, yes. We've still got a bunch of Real Programmers [1] out there. Real Programmers have moved on in my lifetime from insisting that everything should be written in assembly to insisting that they're Real Programmers enough to handle C directly and anyone who writes undefined behavior or buffer overflows or memory management issues just aren't Real Programmers enough and should put down their keyboards and walk away in shame, perhaps taking up goat farming or plumbing.

In all seriousness, their numbers seem to be diminishing. But you certainly can't expect the Real Programmers to write test code. Why should a programmer write test code when they know they didn't write any bugs?

[1]: http://catb.org/~esr/jargon/html/R/Real-Programmer.html - I'd say as time goes on, this term is relative.