[ealloc]

The article says the problem was a dropped frame from the camera, but that just further piques my curiosity:

Presumably they use some kind of Kalman Filter, but those are easy to program to account for missing frames, or frames at non-discrete timepoints, perhaps even for screwy camera images if the programmer had a reasonable prior for the likelihood of it happening. Kalman Filters by design account for measurement error.

[jcims]

It seems like the issue wasn’t that there was a dropped frame, it’s that the time slot for that frame got filled by the next frame, then every subsequent frame was off by one resulting in a persistent timestamp offset of the vision data from reality for the remainder of the flight.

I didn’t read into it too much so I may not have all the details right, but I think this is the gist of it.

[ealloc]

That would be a straight-up, avoidable software/hardware bug: The incoming timestamp is incorrect, and garbage in is garbage out.

That would make me curious how the timestamp error occurred: software, hardware? Camera or Navigation code? I assume they have very high standards, what was the process failure point?

[cm2187]

Or timezone bug! Always hard to test.

[tonyarkles]

Thank you for your comment, because it triggered an interesting chain of thoughts about a semi-related problem I’m working on at work.

Usually with a Kalman filter, you’re taking into account the spatial measurement error (gyro-measured roll rate error, accelerometer-measured acceleration error, etc) but I don’t think I’ve ever encountered a system that explicitly modelled sensor latency variation relative to timestamps. Based on the description of the problem they encountered here, I suspect what happened is that it lost a frame but didn’t adjust the “photo timestamps” appropriately; every frame that came along afterwards would have had an incorrect timestamp? Even if the Kalman filter was set up to handle “this photo was taken 20ms ago” when doing its forward integration, if they didn’t model “this photo was taken 50ms ago but is reporting that it was taken 20ms ago” then you’d pretty readily get the kinds of oscillation they were getting.

Edit: yeah, just like the sibling comment said :)

[doublerebel]

HoloLens provides a timestamp with every frame from each sensor, which can be used for sensor fusion outside of the system usage. Windows.Perception.PerceptionTimestamp can be used for either recorded data (e.g. camera) or for future predictions (e.g. predicted device position). The predicted latency is also used to adjust the render to ensure the viewer's perspective is correct even though the draw calls may be lagging slightly behind the viewer's position.