Hacker News new | ask | show | jobs
by ilyt 1278 days ago
I remember when we were telling devs to stop returning java 503's with stacktraces to the user.

Devs fixed it by returning 200's with stack traces.

And as page was ESI stitched together on Varnish, when they fucked up there wasn't just a stacktrace, but a bunch of different ones in various parts of the page.
1 comments
ywei3410 1278 days ago
I really feel this is an infra problem rather than a dev problem though; the reverse-proxy should strip 5xx response bodies before the egress no?
link
ilyt 1278 days ago
Eh, depends, for some apps 503 is legit response that should be returned to customer/app, in other cases it's app being badly designed. We did that few times when it made sense but in most it does not. There is no error code for "down for maintenance" so 503 was also used sometimes for that purpose (although we recommended devs to just fail healthchecks so loadbalancer displays its own error page in those cases)

The other problem (let's just say devs were not... that great with architecture) was that they were getting exceptions in the logs without attached URL and other metadata so they kinda wanted to get that exception directly on the webpage.

I'll paint you a picture of how shoddy some stuff was: they were using a templating language to generate JSONs (and had many bugs in it too) for years, instead of just natively encoding some JSON in Java, coz of party shoddy architecture, partly years-long war between frontend and backed dept. They migrated to Git in... 2018 I think ? No CI/CD of any sort till recently.

End result (of them insisting that they will send the exception to frontend to be seen) was actually exceptions being signed and encrypted blob wrapped with a bunch of JS that gathers all the errors (incl. errors that JS on site might've done), adds all the metadata it sees from "browser perspective" and sends it to monitoring endpoint where that is shoved into ES cluster.

Sort of retarded version of distributed tracing that is now in vogue... done somewhere in 2013. But it did catch a bunch of bugs that were "only" showing to users in browser

link