| Most tool companies making claims about their tools show a shocking lack of knowledge about testing. This generally guarantees that their tools are dismissed by serious professionals. That still leaves a pretty substantial market among credulous wishful thinkers, of course. But as a tester, I would like to see a tool that isn’t just more bullshit. For this to happen you will have to explain: - What exactly is your product designed to do? What kind of products can it be applied to test? - What do you mean by the word test? Humans test in many ways and levels. Do you simply mean “exercise code while detecting crashes?” Because that’s a tiny part of testing. - Code coverage is not the only kind of coverage. So how do you automatically achieve state and data coverage? I’m guessing you don’t, but hoping you will surprise me. - Test oracles come in all shapes and sizes. One of the reasons I say testing cannot be automated is that I can easily demonstrate that a human tester cannot fully specify their own oracles, and thus cannot write code to implement them, either. So, how does your product recognize a bug when it sees it? - How much human handholding is needed to operate your product? - Testers think critically about how users interact with the product as users attempt to fulfill their purposes. This guides practical testing. I haven’t yet seen any product that thinks critically. ChatGPT can’t. So how does your product cope? - When the product under test changes, what does your product do? - Can your product EXPLAIN its test coverage (other than reporting code coverage, which is a poor indicator of good test coverage)? - Say I have a product that sends the user through a multimodal questionnaire (including the use of animated screens that guide the user through measuring heartrate) and then produces a diagnosis of possible illnesses. Can your product tell if the diagnosis was correct in relation to the original intent of the logic that is documented in Jira tickets and Slack conversations? Will it generate questions about any of that, the way a real tester does? |