[benbye]

Hey! Author here! Yes, I completely agree with your take! I want to run another experiment around whether the JEPA can focus its distance metric on the task-relevant features. But I think this is also where world modeling starts to blend into RL or goal-conditioned learning, the world model can learn what is predictable, but an external objective still has to tell it what matters. The position probe was essentially a lightweight way of adding that task-specific weighting without retraining the representation.

Also I actually did try the experiment where we maximized the predicted X! It worked across open ground, moving Mario from x=40 to roughly x=360. But... it broke down at the first obstacle, it repeatedly chose to jump in place. The model predicted those jumps would increase X, but Mario’s real position stayed around x=342. It just couldn't get over the error in its learned jumping dynamics.

Thanks a bunch for reading :)))