If you are willing to adapt to a different structure and workflow, you can filter the scope of git down dramatically with sparse checkouts (as @WorldMaker also mentioned).
Sparse-checkouts are amazing. I wrote some small tools that use dependency information in Flutter packages to drive a sparse-checkout. We use it at $dayjob now.