Speculative pre-positioning: off-path decode for stateful inference sessions

Y	Hacker News new \| ask \| show \| jobs

	Speculative pre-positioning: off-path decode for stateful inference sessions (arxiv.org)
	1 points by logotype 26 days ago

1 comments

With native support for sessions in an inference engine we can make use of idle GPUs... doing work!