[MSFT_Edging]

Another disagreeing voice, but I try to employ problem speculation when some problem arises. I'm working in a cross-company, cross-team project, where everyone's input interacts in interesting ways. When we come across a weird problem, getting folks together to ask, "what does x issue sound like its caused by?". This gets people thinking about where certain functionality lives, the boundary points between the functionality, and a way to test the hypothesis.

It's helped a dozen times so far essentially playing 20 questions and being able to point to the exact problem and have it resolved quickly.

This is a semi-embedded system. FGPAs, SoCs, drivers, userspace, userspace drivers, etc. Lots of stuff to go wrong, speculation gives a place to start.

[organsnyder]

Speculating is a great way to prioritizing what to investigate, but I've worked with many senior engineers (albeit not in embedded) that have made troubleshooting take longer because they disregarded potential causes based on pattern-matching against their past experiences.

[razorfen]

This has become a personal debate for me recently, ever since I learned that there are several software luminaries who eschew debuggers (the equivalent of taking an oscilliscope probe to a piece of electronics).

I’ve always fallen on the side of debugging being about “isolate as narrowly as possible” and “don’t guess what’s happening when you can KNOW what’s happening”.

The arguments against this approach is that speculation and statically analyzing a system reinforces that system in your mind and makes you more effective overall in the long run, even if it may take longer to isolate a single defect.

I’ll stick with my debuggers, but I do agree that you can’t throw the baby out with the bathwater.

The modern extreme is asking Cursor’s AI agent “why is this broken?” I recently saw a relatively senior engineer joining a new company lean too heavily on Cursor to understand a company’s systems. They burned a lot of cycles getting poor answers. I think this is a far worse extreme.

[organsnyder]

For me, it's about being aware of the entire stack, and deliberate about which possibilities I am downplaying.

At a previous company, I was assigned a task to fix requests that were timing out for certain users. We knew those users had more data than the standard deviation, so the team lead created a ticket that was something like "Optimize SQL queries for...". Turns out the issue was our XML transformation pipeline (I don't miss this stack at all) was configured to spool to disk for any messages over a certain size.

Since I started by benchmarking the query, I realized fairly quickly that the slowness wasn't in the database; since I was familiar with all layers of our stack, I knew where else to look.

Instrumentation is vital as well. If you can get metrics and error information without having to gather and correlate it manually, it's much easier to gain context quickly.

[MSFT_Edging]

To me, it's the method for deciding where I put the oscilloscope/debugger.

Without the speculation, where do you know where to put your breakpoint? If you have a crash, cool, start at the stack trace. If you don't crash but something is wrong, you have a far broader scope.

The speculation makes you think about what could logically cause the issue. Sometimes you can skip the actual debugging and logic your way to the exact line without much wasted time.

[sigbottle]

Its probably different depending on how much observability you have into the system.

Hardware, at least to me, seems impossible to debug from first principles, too many moving parts from phenomenon too tiny to see and from multiple different vendors.

Software is at a higher level of abstraction and you can associate a bug to some lines of code. Of course this means that you're writing way more code so the eventual complexity can grow to infinity by having like 4 different software systems have subtly different invariants that just causes a program to crash in a specific way.

Math proofs are the extreme end of this - technically all the words are there! Am i going to understand all of them? Maybe, but definitely not on the first pass.

Meh you can make the argument that if all the thoughts are in the abstract it becomes hard to debug again which is fair.

That doesn't mean any one is harder than the other and obviously between different problems in said disciplines you have different levels of observability. But yea idk