[Poefke]

The target audience for unit tests is not the client, it is the developer. Unit tests allow you to change code with more confidence. 100% code coverage is not a useful aim, you should aim for 100% confidence in your code. Unit tests can also function as example code, that can't get out of date, since then the tests will fail.

Testing for quality assurance is a different thing, usually called acceptance testing and sometimes including regression testing. One easy way to make clients angry is by re-introducing bugs that were fixed in earlier releases.

[kstrauser]

A few years ago, I got tired of my company talking about plans to upgrade our main app from Python 2 to Python 3, until one weekend I just did it. The tests caught a million little changes that I worked through until everything passed. Come Monday, we were on Python 3 and I took a couple workdays off to play video games.

I wouldn’t have dared even start if I didn’t have confidence in our test suite.

[eesmith]

I was not so lucky! I had a good test suite, but there were a lot of places where I mixed up strings and bytes. I had to add new parameters to specify in Unicode encoding/decoding options, new tests to handle those failures, and new APIs so I could have one function return bytes and another return strings.

Plus, I had C extensions, which had to be updated to handle changes in the Python/C API, including places where Python 2.7 could handle both Unicode and bytes in the ASCII subset:

  >>> u"bbcf".decode("hex")
  '\xbb\xcf'
  >>> "bbcf".decode("hex")
  '\xbb\xcf'
  >>> b"bbcf".decode("hex")
  '\xbb\xcf'

but under Python 3 required more work:

  >>> bytes.fromhex("BBCF")
  b'\xbb\xcf'
  >>> bytes.fromhex(b"BBCF")
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: fromhex() argument must be str, not bytes

AND, I needed to support both Python 2.7 and Python 3.5+ on the same code base.

AND, I needed to support various third-party tools that had their own different migration paths for how to handle the transition.

[eesmith]

As the author points out, "What constitutes a unit? Is it a function? A class?"

If you test to a function, or a class, then your tests imply that function or class must be present, with that specific API.

In my experience, most people write unit tests for internal implementation details that are irrelevant to the business domain. They end up inhibiting code change rather than encouraging change, because anything changes often require re-evaluating each failing test to see if it was meaningful in the first place - and if 100s of tests are no longer meaningful, it's easy to skip the couple of tests which are true regression tests.

As the author writes, full end-to-end tests "are often slow, cumbersome, hard to debug and tend to be flaky."

Instead, find the internal interfaces which tied to the business logic ("something that delivers value to a paying client"), and write the tests to that. You can use unit test frameworks for that sort of functional testing.

[wakawaka28]

>If you test to a function, or a class, then your tests imply that function or class must be present, with that specific API.

So what? Every implementation will have some basic structure. That structure can be modified if needed. The point of a unit test is not to posit that any particular thing exists, but that the things that do exist actually work.

>In my experience, most people write unit tests for internal implementation details that are irrelevant to the business domain. They end up inhibiting code change rather than encouraging change, because anything changes often require re-evaluating each failing test to see if it was meaningful in the first place - and if 100s of tests are no longer meaningful, it's easy to skip the couple of tests which are true regression tests.

If the internal implementation is not observable without a bunch of other bullshit, these tests can help instill confidence that the stuff actually works. If you don't test, or test the overall system, it takes much longer. There is such a thing as a pointless test but it is far more common in my experience to have stuff that isn't covered at all by tests. If your biggest problem is that you have to delete some tests that you made obsolete, that's perfectly ok.

>As the author writes, full end-to-end tests "are often slow, cumbersome, hard to debug and tend to be flaky."

Those types of tests are not unit tests.

>Instead, find the internal interfaces which tied to the business logic ("something that delivers value to a paying client"), and write the tests to that. You can use unit test frameworks for that sort of functional testing.

It isn't only the business logic that needs to be tested. Anything that is cumbersome to test "enough" in the overall system ought to be unit tested. At work I'm faced with a series of structures that are cumbersome to test in isolation and in totality. If I had unit tests I could make changes at least 3x faster.

[eesmith]

> Every implementation will have some basic structure.

Setting aside functional vs OO paradigms, even if you have a simple helper function like:

  def removeprefix(s, prefix):
    if s.startswith(prefix):
      return s[len(prefix):]
    return s

do you write tests for it? Or do you write tests for the higher level routines which call it? I think most write tests for the function.

If you write tests for the function then you hinder future refactoring from removeprefix(s, prefix) to s.removeprefix(prefix) once you switch to a Python which implements s.removeprefix (3.9, I think?)

For example, in red-green-refactor TDD, you are not supposed to change the tests when you refactor.

What you've ended up with are tests for the specific structure, and not the general goals.

> If you don't test, or test the overall system, it takes much longer.

Good thing neither I nor the linked-to author makes either of those arguments.

> it is far more common in my experience to have stuff that isn't covered at all by tests

Which is why I use coverage-based method to identify what need more tests. Coverage is a powerful tool, and easily misused.

> If your biggest problem is that you have to delete some tests that you made obsolete, that's perfectly ok.

The biggest problem is that you decide to not refactor because there are so many tests to delete, and you have to figure out if the failing test really is okay to delete (because it was implementation specific) vs. one that needs to be updated (eg, because it handles a regression case).

> Those types of tests are not unit tests

Correct! And no one said they were.

> At work I'm faced with a ...

That's fine. You do you.

[wakawaka28]

>do you write tests for it? Or do you write tests for the higher level routines which call it? I think most write tests for the function.

I try to not test things that are that simple. But I wouldn't fault anyone for writing a small number of test cases for it.

>For example, in red-green-refactor TDD, you are not supposed to change the tests when you refactor. > >What you've ended up with are tests for the specific structure, and not the general goals.

I'm not familiar with this methodology. But if your refactoring requires changing the interfaces then it must require changing the tests. It may be ideal to have such rigid interfaces in some cases, but I don't think you have to be so committed to one methodology. Big changes require big testing.

>If you write tests for the function then you hinder future refactoring from removeprefix(s, prefix) to s.removeprefix(prefix) once you switch to a Python which implements s.removeprefix (3.9, I think?)

How? Is the old code supposed to stop working because a new function was added? You aren't obligated to delete perfectly working and tested code. And if the new version of Python breaks it somehow, the tests will tell you quickly compared to finding out in a bigger test.

>Which is why I use coverage-based method to identify what need more tests. Coverage is a powerful tool, and easily misused.

Coverage is fine. But I mean, coverage is for people who already committed to testing every line of code. Not every project has unit tests in the first place so coverage is a moot point.

>The biggest problem is that you decide to not refactor because there are so many tests to delete, and you have to figure out if the failing test really is okay to delete (because it was implementation specific) vs. one that needs to be updated (eg, because it handles a regression case).

This work is very easy compared to fixing the real product in most cases. If you didn't write a bunch of frivolous tests it isn't that much work.

>>Those types of tests are not unit tests > >Correct! And no one said they were.

Well this is a discussion about unit tests. If you want to talk about other types of tests you need to be clear about that.

>That's fine. You do you.

I was just pointing out that unit tests are not always feasible to do. But I wish they were. The stuff I see at work was not designed by me, so it's not my fault it is infeasible to test at least.

[eesmith]

> You aren't obligated to delete perfectly working and tested code.

You aren't obligated to write unit tests either.

If your goal includes long-term maintainability then should refactor for simplicity and consistency. Removing unneeded code and unneeded tests is a good thing.

> coverage is for people who already committed to testing every line of code.

Much as I would like it, I don't have 100% coverage. I use coverage to identify and prioritize which areas of code need more tests. Some are more important than others.

I also use it to identify code which is no longer needed, like workarounds for third-party-package V2021 which have been fixed in late releases, and I have no need to support V2021 any more.

> If you didn't write a bunch of frivolous tests it isn't that much work.

My observation is that people write a bunch of frivolous tests.

> If you want to talk about other types of tests you need to be clear about that.

I am talking about how to design unit tests.

Someone else here pointed to Robert Martin's essay on exactly this topic, at http://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrava... .

"Design the structure of your [unit] tests to be contra-variant with the structure of your production code. ... The structure of your tests should not be a mirror of the structure of your code."

The issue I'm describing is what Martin characterizes as "the Fragile Test Problem" with some unit test approaches.

[wakawaka28]

>If your goal includes long-term maintainability then should refactor for simplicity and consistency. Removing unneeded code and unneeded tests is a good thing.

I agree with this in principle, but I think people disagree about what is needed. One person might look at a good set of tests and think it's a waste of time, but if it actually hits the right points then it's probably worth keeping. I think if unit tests are feasible and can hit a bunch of cases that would be hard to recreate otherwise, then unit tests are far superior.

>"Design the structure of your [unit] tests to be contra-variant with the structure of your production code. ... The structure of your tests should not be a mirror of the structure of your code."

That's a nice ideal. But as with the refactoring issue, it may be impossible to test a thing without recreating some of that structure. That's why we have mocks.

[taylodl]

Depends on how your unit tests were designed. You need to avoid the Fragile Test Problem. Bob Martin wrote about this a while back: http://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrava...

[rTX5CMRXIfFG]

There needs to be a rule against having to read entire articles or YT videos just so that a reader can understand the point that a commenter is trying to make, but which they cannot explain in their own words, no? It’s very anti-discussion, and how do we even know to what extent commenters are quoting or agreeing with their sources?

[taylodl]

Your unit tests should be contravariant to the code its testing so that way your code can be refactored and your existing unit tests can be used AS-IS to test the refactored code.

The Fragile Test Problem arises when your unit tests are covariant to the code its testing, such that a significant refactor breaks the tests - not because the code is necessarily wrong, but the structure of the code is no longer what the unit test was expecting.

[bluGill]

While the link in question isn't a good article, for complex subjects like this a long article is much better. The article linked should take several days to create if you are a great writer (again, I would create in a different way, but the content deserves days or even weeks of work to create), and thus cannot fit in a comment box except in a reduced form that doesn't do the idea justice.

[osigurdson]

Agree. The "what about the client" trope is a complete non-argument. If unit tests make code better, the client maybe wants that. The argument should purely be based on if they are helpful.

[pc86]

I'll take it a step further than this. It doesn't matter if the client wants unit tests or not. If the developer has been hired to do the job, it's going to be up to that developer whether or not they feel unit tests will make the code better. If they do, they should do them. If they don't, they shouldn't.

Caring about the client's opinions re: unit tests is a little like caring about their opinion re: the interior color of your vehicle. Sure we can talk about it if it comes up over a beer some Friday evening but we're certainly not going to change anything based on the conversation.

[bluGill]

Most clients should care about code that is long term maintainable even if it isn't in the contract. Thus tests are good. As a professional engineer (I'm intentionally invoking professional in context of other industries where the term has legal implications on quality, even though it doesn't exist for software it should!) you should care about that

[blitzar]

If it is in the contract and specs (either way) then you do what you are paid to do.

[pc86]

Well of course. That's not relevant to the larger point, though.

[mewpmewp2]

How do you measure 100% confidence? Pretty sure I have never reached 100% confidence, but I haven't had a metric to track it so I have no data to prove it.

The only possible way to reach near 100% is I think by not writing any code, and by not building anything.

I do think 100% code coverage goal doesn't mean much, since you can potentially achieve it using 0 assertions. Or you make 1 assertion per test, leaving 95%+ expected results unasserted. Assertions could also be wrong.

[pfix]

You measure how many of your deployments ship broken code. I've worked in contexts where test coverage was sufficient enough to be confident that when deploying there wouldn't be any issues apart from really novel unexpected stuff you wouldn't have tested anyway.

I've also worked in contexts wher I didn't have that confidence.

I would say you know when you're in that environment and directly measuring doesn't really help as it's more a matter of culture than technical metrics.

[pfix]

Because it's linked in another post right now:

If you feel confident that renovate[1] pull requests can be merged without review on a green pipeline, you have 100% confidence in your tests ;-)

My first context was that way.

[1]: https://www.jvt.me/posts/2024/04/12/use-renovate/

[mewpmewp2]

What do you mean by "broken code"? Any sort of code that yields in a bug? Any sort of bug? Because interpreting directly broken code implies to me some sort of compilation error. Also measuring that seems very complicated, depending on how you interpret "broken code". E.g. a bug could influence 100% of the customers, 5% of the customers, 0.1% of the customers and only when certain conditions happen, it could be a combination of deployments introducing this. It could be a bug that can easily be worked around by a customer. It could be a bug that only arrives 6 months down the line, because some dependency changed, and you didn't have proper handling for it, even though it was expected that you had. E.g. you are calling a third party API that alerts you in advance they are planning to add extra error code in the system in 6 months, so you should probably also include it, but you don't and then system bugs due to that.

The problem could arrive 2 years in the future when scale increases and the code wasn't built to handle it, even though it was expected.

You take 100 of your past deployments and see how many resulted in some sort of a bug? E.g. if there's 0, then you have 100% confidence?

> any issues apart from really novel unexpected stuff you wouldn't have tested anyway.

I mean that's one thing that makes it impossible to have it 100% for me at least.

> I would say you know when you're in that environment and directly measuring doesn't really help as it's more a matter of culture than technical metrics.

I may have been in wrong environments throughout my whole life, but I also have trouble imagining a perfect environment where there's 100% confidence.

[ludovicianul]

Confidence can be achieved using https://en.wikipedia.org/wiki/Mutation_testing. this involves running an engine that mutates your code and check if you have at least one assertion failing.

[HeyLaughingBoy]

I agree. The post completely misses the point of unit testing in the first place by drawing a relationship to the features being implemented.