[dirtbag__dad]

> Tests created a similar false comfort. Having 500+ tests felt reassuring, and AI made it easy to generate more. But neither humans nor AI are creative enough to foresee every edge case you’ll hit in the future; there are several times in the vibe-coding phase where I’d come up with a test case and realise the design of some component was completely wrong and needed to be totally reworked. This was a significant contributor to my lack of trust and the decision to scrap everything and start from scratch.

This is my experience. Tests are perhaps the most challenging part of working with AI.

What’s especially awful is any refactor of existing shit code that does not have tests to begin with, and the feature is confusing or inappropriately and unknowingly used multiple places elsewhere.

AI will write test cases that the logic works at all (fine), but the behavior esp what’s covered in an integration test is just not covered at all.

I don’t have a great answer to this yet, especially because this has been most painful to me in a React app, where I don’t know testing best practices. But I’ve been eyeing up behavior driven development paired with spec driven development (AI) as a potential answer here.

Curious if anyone has an approach or framework for generating good tests

[suzzer99]

I've always thought that writing good tests (unit, integration or e2e) is harder than the actual coding by maybe an order of magnitude.

The tricky part of unit tests is coming up with creative mocks and ways to simulate various situations based on the input data, w/o touching the actual code.

For integration tests, it's massaging the test data and inputs to hit every edge case of an endpoint.

For e2e tests, it's massaging the data, finding selectors that aren't going to break every time the html is changed, and trying to winnow down to the important things to test - since exhaustive e2e tests need hours to run and are a full-time job to maintain. You want to test all the main flows, but also stuff like handling a back-end system failure - which doesn't get tested in smoke tests or normal user operations.

That's a ton of creativity for AI to handle. You pretty much have to tell it every test and how to build it.

[whattheheckheck]

Use tla+ and have it go back and forth with you to spec out your system behavior then iterate on it trying to link the tla+ spec with the actual code implementing it

Pull out as many pure functions as possible and exhaustively test the input and output mappings.

[gck1]

The problem with specs for me is always with boundaries. How many specs do you have for a complex project? How do they reference each other? What happens when requirements cross boundaries?

And finally, how do you address spec drift?

[michaelsalim]

Personally i think the challenge of testing never really changed with AI. You need to know what you want to specifically test before writing/vibe coidng with it. Otherwise it'll just manufacture tests that always passes and are of 0 value.

If some component doest benefit from being extensively tested, then it's still the same today. The difference is now it's so easy to generate something, no matter how useless it is. Worse part is, no one cares. Test passes, it doesn't affect production, line coverage increases, managers think the software is more tested, developers just let a prompt do everything. It's all just testing theatre.

I think E2E is the more important than ever. AI is pretty good at getting the local behaviour correct. So unit tests are of less value. Same can't be said for the system as a whole. The best part is, AI is actually pretty good at writing E2E tests. Ofc, given that you already know what you want to test