[eb0la]

> you have shocked people when they find out that 100% test coverage doesn't mean that you really have a bug-free codebase.

This applies specially to business people.

100% test coverage just means you tested that s*t with all you expected...

...but your software does not live isolated.

Some external system may have a bug, or may inject some unforeseen values, or a solar flare might flip the value of a bit in your system and crash your software.

[yakubin]

I see completely different problem. Just because there are branches in your code, doesn't mean there are branches in its inputs. And just because there are branches in the inputs, doesn't mean they are reflected in the code. So you may have 100% test coverage, but be testing branches which aren't going to be taken, at the same time completely missing the branches that are in the inputs, which you'll fail to handle. Example:

  fn abs(x: i32) -> u32 {
      x as u32
  }

  fn test_abs() {
      assert_eq!(5, abs(5));
  }

Boom, 100% test coverage! But the tests are actually very low quality. There is a bug when x is negative. That's why property-based testing is nice. It uncovers branches in the inputs (although the task of generating a representative set for the inputs is sometimes non-trivial).

After discovering the bug, one may even write a branchless implementation of this function for performance without updating the test, and it will still be 100% coverage. But the arithmetic has "logical branches" which do not look like ifs, instead they generate qualitatively different results for different inputs.

[pydry]

I tend to find logical/algorithmic bugs overemphasized. Property tests are nice and work very well for extremely complex algorithmic code that has simple inputs and outputs but most code I write at work simply isnt that.

In school I wrote parsers. At work I have done it but it's rare.

In fact if you write commercial code you'll often find that the code that you rely on that is like that weirdly gets concentrated in open source libraries which you will be testing only indirectly.

Certainly when writing commercial code I find that the majority of bugs lurk in the interstitial spaces between subsystems I am integrating or in misunderstandings about how the overall system or subparts of it are supposed to behave.

And property tests are not much help there and unit tests are often a hindrance (because theyre as likely to bake in wrong assumptions).

TDD is some help with this but only if A) it's paired with BDD to exorcize specification bugs and B) done with integration tests that exercise all parts of the system together.

[mannykannot]

The sort of programs you are writing about can easily exhibit the sort of problem that yakubin is giving an example of - here's a comparable example:

  wasBusinessDay(date): Date.dayOfWeek(date) not in (Date.Saturday, Date.Sunday)

Which is as trivial as yakubin's example, but can fail for more, and much more nuanced, reasons - and no, you are not, in general, going to find an off-the-shelf answer to the question of whether your particular business was operating on a given day.

Both your claim that a big problem is things not working as they are supposed to[1], and the fact that the same old basic security problems get repeated in every new platform that comes along, show that libraries have not, in practice, been any sort of panacea for business software.

[1] I suspect you are referring to application code rather than library code, but either will do for the point I am making here.

[IanCal]

I think property tests work well on larger things, with a super basic assertion they're essentially just fuzzers. You can point hypothesis at a swagger spec and let it test your API like this.

I built a property testing library back in the day when I made a library for creating UIs. I was then able to write tests like

* Given an arbitrary UI

* And an arbitrary up/down/left/right list of user direction (this was the only way of navigating)

* If they press a direction and the focus moves, pressing the opposite direction takes them back to where they were before

This uncovered bugs in interfaces like this with 3 items, bottom left is in focus and you press right. Now press left, users probably want to go back to the bottom left rather than top left.

    ┌─────┐  ┌─────┐      ┌─────┐ ┌──────┐
    │     │  │     │      │     │ │      │
    │     │  │     │      │     │ │      │
    └─────┘  │     │      └─────┘ │      │
             │     │ ────►        │focus │
    ┌─────┐  │     │      ┌─────┐ │      │
    │focus│  │     │      │     │ │      │
    │     │  │     │      │     │ │      │
    └─────┘  └─────┘      └─────┘ └──────┘

Another one was

* Given an arbitrary list of API calls to add/remove/change the UI and user direction presses

* There is always only one item in focus, or no items at all

This actually uncovered a specification bug. We had two requirements

1. Always have an item in focus if there are any that can be in focus

2. If you delete all the items, then add one back in, it's not focussed

Those conflict, but we never noticed, and even had passing unit tests for both cases.

I think property tests can map very nicely to the level of system description that we typically want. I'd love to see larger integration with BDD tools.

[yakubin]

Maybe it's true for CRUD apps. I was speaking from the experience of writing compilers. From what I've read, logical/algorithmic bugs are also common in gamedev. I'm most interested in algorithmic code though, so that may be a bias.

[pydry]

Logical and algorithmic code (& bugs) were a lot more common in gamedev before the onset of the open source game/physics engines. It has followed a similar evolution to the rest of software development whereby core algorithmic components tend to get shared in generic, battle hardened OSS systems like Unity. Most games will rely on their APIs rather than implement their own physics or rendering engines and this is only becoming more common over time.

College students who start out by writing toy compilers, quicksort implementations and physics engines will often get a distorted view of what professional development is actually like.

[yakubin]

Someone has to write those libraries you're using, and they need to test it. Maybe you're satisfied with having a job consisting of gluing libraries together, but not everyone is like that and there is no need for being patronizing towards them.

[geysersam]

> maybe you're satisfied...

If anyone is patronizing here it's you.

[jerf]

abs on machine-sized integers has another easy-to-make mistake: what's the absolute value of the signed byte -128?

You face some hard decisions on that one. None of the error handling paradigms are good at handling the class of numeric errors that people really, really want to pretend don't exist because no matter what kind of error they throw they're a major pain to deal with correctly.

[yakubin]

I'd agree, if the result type was the same as the input type. However, notice that the function I wrote returns an unsigned type, so in the case of byte-sized types it would be fn abs(x: i8) -> u8, i.e. there is enough space for 128. Otherwise, good point.

[tuyiown]

100% test coverage only covers one dimension of the code, the lines. The other dimension is the data range, and proper coverage is generally not measured.

So 100% test coverage you still have tons of possible codepaths that can go sideways, way before including external variability.