It absolutely is fraud. I worked on one of those big video exchanges and we'd sometimes see ads run through our own system multiple times. We couldn't spot it until after the ad ran because of the way the ad actually plays. It ends up a bunch of black box Russian dolls, where the spec says you call methods on the ad unit you are playing, which will just wrap another ad unit, which in turn wraps another ad unit.
We could piece together what happened sometimes after the fact, but not reliably.
As an example, I'm currently diagnosing an issue where an ad refuses to play in our player as it incorrectly identifies itself as being adblocked. We load the advertiser provided ad (really a DoubleVerify fraud protection module) which loads a custom video player which eventually loads the advertiser's media file. The DoubleVerify script is downloading a bootstrap JavaScript file from their server, which then randomly chooses some other JavaScript to eval to check for whatever they think indicates fraud. It's this JavaScript, dynamically downloaded from their server (possibly even dynamically generated) that thinks our ad player is an ad blocker.
And that's just diagnosing a bug, now imagine if you were trying to find some malicious JS.
It was definitely worse when it was all flash. At least JavaScript has ubiquitous debug tooling.
VPAID 3.0 (or is it 4.0?) has some proposals to fix this. It makes fraud analysis a first class citizen of the spec, sort of like companion ads. This allows them to be downloaded separately, as well as cached.
We could piece together what happened sometimes after the fact, but not reliably.