Hacker News new | ask | show | jobs
by brutus1213 711 days ago
Scientist here and have book marked this article for close reading (so apologies for this question if it is discussed in the article).

I had a few brushes in RL (with collaborators who knew more RL than I did). A key issue we encountered in different problem settings was the number of samples required to train. We created a headless version of the underlying environment but could not make it go a lot faster than real-time. We also did some work to parallelize but it wasn't enough (and it was expensive). Is the TM related RL training happening in real-time or is it possible to speed it up? That seemed like the key problem to make RL widely used, but curious about your thoughts.
2 comments
donadigo 711 days ago
I'm not sure about your particular case, but if your environment really is headless, then it should absolutely be possible to run it a lot faster than realtime. It depends on what the environment is and if you have access to its source code (we do not have that in TrackMania so it's a lot harder). Either the environment is purposely throttling the amount of time it simulates, or it just takes so much time to simulate the environment that it's not possible to speed it up anymore.

We're lucky in case of TrackMania because it internally has systems to both set the relative game speed and also completely disable all rendering and just run physics. Linesight achieves about ~10x speedup where the most time spent now is in rendering game frames and running the inference on the network. They also parallelize training by running more game instances and implementing a training queue. For the "raw" speedup ratios, TM usually achieves about ~60x (one minute is simulated in one second) and I use this speedup to implement bruteforce functionality in the tool (coupled with a custom save states implementation).

link
msephton 711 days ago
It's possible to speed it up by running the game as fast as it can go (so, not limited as it normally is for human consumption). They talk about running it at 9x speed, so months of training could be done in 80 hours.
link