Gnome has a built in method that allows you to set your own permissions on where in the filesystem a Flatpak can access, in addition to more fine-grained control like Flatseal. For most programs this means limiting it's access to your Download directory or similar until portals become more widely adopted. There are also wrapper scripts you can use for when you double click a file it will temporarily give access to the file to VLC and revoke it afterwards - basically a poor man's portal.
Also you should look closer at what "your entire filesystem" means. There are a lot of areas of the file system that are completely blacklisted with no actual way to enable them regardless of configuration. Important things like /etc and /usr just to name a few.