| Nice job, i have also been working on a few JEPA based models during the last few months. Trying to make more efficient LLMs. I feel like you hit the main issues in the use of jepa models (well except collapse but sigREG more or less solves the collapse issue). The main issues in JEPA as i see it is pushing the latent space toward representing features that are needed for good planing. A thing which is especially a problem in hierarchical planing. You prime a JEPA world model to predict changes based on actions but you never really push it to use those actions. You simply hope that it will use them. If your latent is big enough and the actions effect on the world is simple enough it tends to work out but those qualifiers are not always small things. Secondarily finding actions for the higher level JEPA Predictors. LeWorldModel encodes multiple movements in to higher level actions. But this is a not a very good idea. It solves a basic issue with planing where the predictions degrade after a set nr of steps. But it does not solve the issue of higher level actions not actually being button presses. The higher level actions for your mario game version would be things like: get the coin, Kill an enemy or get to the end of this stage. You cant just encode many button presses in to those types of things. You need to discover those actions somehow. |