[omrjml]

Regarding dynamical systems, Neural ODEs hold promise as they are more analogous to the numerical solvers. I think with any dynamical system, purely data driven approach can be problematic because, as you say, the standard NN architectures are not great for it. You can however add physical constraints, e.g. minimize the Lagrangian as the objective function, to bring stability to these NN emulators.