Part of the problem with timing variety is frameworks not always picking the right gpu/backend.
If you want to inspect or tweak the setup, be my guest at https://github.com/kvark/inferena