NixOS 24.05 contains an older version of zed, as feature updates are generally not backported to stable NixOS releases. Try running the package from nixos-unstable instead.
Well that got rid of the core dump each time I quit, and it fixed the language server issues. So together with the portals configuration it seems to be working as well as it can.