[Veserv]

Multiplexing onto a single thread is sufficient to observe and record concurrency errors. If that is not sufficient, then I am assuming you want to observe and record errors caused by true parallelism. If so, then you need a full memory trace. That restricts you to either hardware that supports full memory trace connected to a hardware trace probe or instrumented software full memory trace.

The former basically only exists for embedded boards and the latter does not exist (at say less than a 10x slowdown) for Linux or any other common desktop operating system as far as I am aware.

[matu3ba]

So the only way to trace probe consumer desktop CPUs and possibly GPUs is by the hardware vendors them-self or specialized facilities/laboratories? For Intel I can find https://www.lauterbach.com/supported-platforms/architectures..., but nothing for trace probing AMD or later versions.

[Veserv]

You can trace consumer desktop CPUs using instrumented software full memory trace, but that requires OS + debugger + compiler support which is not available for Linux, Windows, Mac, etc.

You can trace hardware that exposes trace functionality usually via a debug port of some kind. Many chips have trace functionality in their production design, but no debug connector is physically present in off-the-shelf boards (to reduce manufacturing cost). You can usually physically modify the board to get access to this functionality which is routinely done when porting software to a new chip/board.

Trace functionality comes in two major flavors, control flow trace and memory trace. Control flow trace only records control flow, so the contents of memory are unknown which is not very useful for your desired use case. Memory trace records memory accesses, so the contents of memory are known. Unfortunately, memory trace is very resource intensive, so most systems that support trace only implement control flow trace. As far as I am aware, it is very unlikely that any desktop or server CPU has memory trace.

The major manufacturers of trace probes and solutions that I know of are Green Hills Software, Lauterbach, and Segger.

[matu3ba]

Thanks for your write-up, this is very helpful for understanding.