Hacker News new | ask | show | jobs
by thibran 85 days ago
I'm doing propety-based test since years for frontend stuff. The hardest part is, that there is so much between the test inputs and the application under test, that I find 50% of the time problems with the frontend test frameworks/libs and not in our code.
3 comments

And sometimes you find errors in code that absolutely should never have errors: I found an (as of yet not-root-caused) error in sqlite (no crash or coredump, just returns the wrong data, and only when using sqlite in ram-only-mode). Had to move to postgres for that reason alone. This is part of the reason why I have a strong anti-library bias (and I sound like a lunatic to most colleagues because they "have never had a problem" with $favorite_library -- to which my response is: "how do you _know_?"[0], which often makes me sound like I'm being unreasonably difficult).

Sometimes, only thing you can do is let the plague spread, and hope that the people who survive start showering and washing their hands.

[0]: I once interviewed at a company that sold a kind of on-prem VM hosting and storage product. They were shipping a physical machine with Linux and a custom filesystem (so not ZFS), and they bragged about how their filesystem was very fast, much faster than ZFS or Btrfs on SSDs. I asked them, if they were allowed to tell me how they achieved such consistent numbers. They listed a few things, one of which was: "we disabled block-level check-summing". I asked: "how do you prevent corruption?". They said: "we only enable check-summing during the nightly tests". So, a little unsettled, I asked: "you do not do _any_ check-summing at any point in production"? They replied: "Exactly. It's not necessary". So, throwing caution to the wind (at this point I did not care to get the job), I asked: "And you've never had data corruption in production"? They said: "Never. None". To which I replied: "But how do you _know_"? My no-longer-future-coworker thought for a few seconds, and realization flashed across his face. This was a company that had actual customers on 2 continents, and was pulling in at least millions per year. They were probably silently corrupting customer data, while promising that they were the solution -- a hospital selling snake-oil, while thinking it really is medicine.

> I found an (as of yet not-root-caused) error in sqlite (no crash or coredump, just returns the wrong data, and only when using sqlite in ram-only-mode).

You should report this to the SQLite developers - they are very smart and very interested in fixing SQLite correctness bugs!

Did you report that SQLite bug? What's the company's name?
Didn't get around to reporting it (huge backlog of tasks). Luckily I am working on a project that _has_ to support SQLite, so if I run into the bug again, I'll report it.

I don't believe that I can tell you the name of the company (they made me sign some NDAs, before the interview, and I have no clue how enforceable those are). Also, this was in 2019, so I would be shocked if they did not fix the problem by now -- especially after I interviewed there (plus I can't be the only one to have noticed this, since).

That said, you have a few data-points if you want to try to triangulate them yourself: physical vm-hosting and storage product, existed since at least 2019, used linux kernel as hypervisor, custom FS, international customers across 2 continents. All of those data-points are my recollection from 2019.

Hell, I’d hire you on the spot.
Are you talking about user flows and multiple interactions that are happening and data exchange that PBT before that wasn't able to address?
PBT allows us to test more combinations without writing hundreds of tests. Yes, it's about user flow inside a single module of our gigantic application.
Interesting. What kind of properties are you checking?
I use quicktheories (Java) and generate a determistic random test scenario, then I generate input values and run the tests. This way I can create tests that should fail or succeed, but differ in the steps executed and in the order with "random input".
OK. What kind of problems do you hit from third-party libraries with that?
Escaping problems or wrong handling of non-visible characters.