[The_Colonel]

That honestly sounds like a downside. Having to verify HTML as an output (in comparison to e.g. JSON) is brittle since it will often change just for presentation reasons. Having a clear and somewhat stable API (which is a boon for testing) is IMHO a strong reason for the SPA model.

[seanwilson]

> Having to verify HTML as an output (in comparison to e.g. JSON) is brittle since it will often change just for presentation reasons.

For HTML tests, if you target elements by their ARIA role, semantic tag, or by matching against the text/label/title the user would see, it should be robust to a lot of presentation changes (and does some accessibility checks too). It's much more robust than e.g. `body > div > div .card > h2` which are sure to break in tests and slow you down when you make changes later.

See Playwright locators for example:

https://playwright.dev/docs/locators

Not sure if this is what you meant, but you can't rely on only the JSON because you can't assume it's been properly transformed into the expected HTML.

[recursivedoubts]

htmx moves the data -> html transformation to the server side and thus should be more testable

[yawaramin]

It's standard practice in the frontend world https://jestjs.io/docs/snapshot-testing

[sensanaty]

In my experience snapshots have largely fallen out of vogue, at least as far as them blocking deploys goes. They're usually flaky, change often, are slow to test and it's way too easy to have false positives which lulls people into complacency and just regenerating the snapshots as soon as they reach a failure. I'm guilty of that myself, unless I can spot the breakage immediately I pretty much just regen the snapshot by default subconsciously because of how often I've had to do that for false positives

[yawaramin]

Then what's to stop you from making any unit test pass by just changing the expectations? The best testing technique in the world can't save us from developer error.

[Hasu]

The problem with snapshot tests is that they tend to fail when something is changed.

One property of good tests is that they only fail when something is broken.

Snapshot tests aren't really automated tests, because you have to manually check the outputs to see if failures are genuine. It's a reminder to do manual checking but it's still not an automated process.

> The best testing technique in the world can't save us from developer error.

Sure, but using a testing technique that increases developer error is unwise, and snapshot testing has done that every time I've seen it used, so I don't use it anymore.

[yawaramin]

> they tend to fail when something is changed.

Then fix the test so that it fails only when something breaks? Do people not fix flaky, overly broad, or incorrect unit tests? How is a snapshot test any different?

[DecoySalamander]

A fix here would be to drop snapshot testing altogether, since being flaky and overly broad is a natural result of dumb diffing of your app's output.

[Hasu]

I'm not sure you understand what snapshot testing is. It's a form of testing where you render frontend components and take a "snapshot", storing the output and saving it for later, then on the next test run, the component is rendered again and compared to the saved snapshot.

Any change that modifies the rendering of any component under test will break snapshot tests. It's literally just a test that says, "Is this render function still the same as it used to be?"

[sensanaty]

Do you have experience with snapshot tests? If not, you should try it out for a bit and see what we mean, you'll quickly run into the situation we described.

If you introduce a new feature and have a new button on the page, for example, the snapshot test will fail (by necessity, you're changing what's on the page) because the snapshot doesn't match the current version with the new button, for example. So, what you have to do is regenerate the snapshot (regenerating it will always make it pass).

The problem is in my experience, that these kind of tests are massively unreliable. So you either fiddle with tolerances (most testing libraries let you set a threshold for what constitutes a failure in a snapshot) a lot, or you manually have to hunt down and pixel peek at what might've happened.

As an example of what usually happens is that, at some middle step of the test an assert fails because some new code causes things to render slightly out-of-order or delayed in some way. Ultimately the final product is good and if it was an intelligent system it would give it a pass. The unit tests pass. Even the integration and E2E tests pass, but the snapshot tests fail because something changed. The chance that this test failure is a false positive is much, much higher than the chance of it being a legitimate failure, but let's assume it's legitimate like the good citizens we are and try to hunt it down.

So, you go through the snapshots, 1-by-1, setting breakpoints where you think things might break - note that this process by itself is painful, slow and manual, and often running snapshot tests on your local machine can break things because things get rendered differently than how they would in CI, which leads to an extra set of headaches - What happens here is that you're testing an imperfect reflection of what the actual underlying app is, though, because a lot or most of it will be mocked/hoisted/whatever. So sifting through what is a potential breakage, and what is just the quirk of the snapshot test is basically a gamble at best. You spin up the app locally and see that nothing looks out of the ordinary. No errors to be seen. Things look and behave as they should. You have your other tests, and they all pass. So, you chalk it up to a false positive, regen the snapshots, and go on with your day.

The next day you add some other thing to the page. Uh oh, snapshots fail again!

Rinse and repeat like this, and after the 50th time sitting there waiting for the snapshot to be captured, you decide there are better ways to spend your limited time on this mortal plain and just say "Fuck it, this one's probably a false positive as well" after taking a 2 minute look at the page locally before hitting the repacture command again.

I legitimately don't think I've ever actually seen a non-false positive snapshot test failure, to be honest.

[eddd-ddde]

Not really unstable since you generate it and there's no browser modifying it. However you'd still lack client functionality testing.