Ask HN: How do I debug my back end in production?

[parthi]

We're a small startup running Node for our backend. Using a combination of LogDNA, FullStory, Sentry and Mixpanel across frontend and backend.

Problems I face:

1. Can't trace user journey: User reports a bug. I look through logs with a timestamp of the user report, but can't trace the user journey, especially with lots of asynchronous operations going on in parallel

2. Insufficient logging: I can identify the error being thrown but can't identify the cause. Would want to see logs of surrounding code which we maybe didn't log

To solve this, I've resorted to adding logging around the suspected problem path, pushing to production and asking the user to try again (very annoying for them)

Any best practices I'm missing or is this really the best way to be debugging issues in prod?

[zadkey]

I'm not familiar with nodejs backends specifically, but for web applications in general it should be possible to run a local copy of the web application with the same code that is being used in production and have it use the prod database connection. And then you can locally step through what is an equivalent of what is in prod with all of your normal local development tools and debuggers.

If you get this working, you can follow along steps to reproduce and hit debug breakpoints etc.

[juangacovas]

This is the way, but connecting to production database is not always an option.

Let's say there is a problem when adding some type of record... You want to avoid, if possible, the creation of "test" records in production while reproducing the bug.

That's why I think we all end having some sort of inhouse tool that grabs a full snapshot of the database, or if not possible because of really big databases, just a sufficient dump of relevant data for the bug at hand (ie. just rows from some date, just some "customer" data, etc. to be able to mostly mimic the data environment).

Do whatever is needed to do your job: squash the bug. To do it you need to be able to reproduce it NOT in production (or you're lost in guess-temptative-fixes in production).

[zadkey]

It's true you can't always connect to prod backend db. Sometimes it's possible if you have a QA or DEV environment you can load it with prod data and test with that.

[XCSme]

Do you get the stack trace of the error?

[parthi]

Sentry provides local stack traces but not data which is hard to reproduce in dev