The main challenges I have had with Flatpak is that for any apps that need to integrate with other components (e.g. I want my VS Code to use the same Node that I run from the Terminal) things get flaky. Its not so much that Flatpak does not support that kind of integration, but that many apps are not properly configured when packaged to take advantage of FP's integration features.... End result is still a poor user experience. Moral of the story: don't just shove your code in a Flatpak and call it a day! Take the time to do it right.
Gnome has a built in method that allows you to set your own permissions on where in the filesystem a Flatpak can access, in addition to more fine-grained control like Flatseal. For most programs this means limiting it's access to your Download directory or similar until portals become more widely adopted. There are also wrapper scripts you can use for when you double click a file it will temporarily give access to the file to VLC and revoke it afterwards - basically a poor man's portal.
Also you should look closer at what "your entire filesystem" means. There are a lot of areas of the file system that are completely blacklisted with no actual way to enable them regardless of configuration. Important things like /etc and /usr just to name a few.