How theoretical is the math in a game engine? (I’m asking because I have no idea but I assumed it’d all be really applied linear algebra stuff, that sort of thing).
Not very theoretical. But with 'nerd sniping' focus I tended to create my own tech unnecessarily, and wasting too much effort to optimize things that do not necessarily need to be that much optimized. Nowadays I try to reuse other people's code as what I really enjoy is to create the end-product.
If you use an existing game engine all you need really is some decent Google-fu...
I've been commissioned to create some non-public VR experiences for a couple of people and all I needed during development was some basic high-school vector math (like literally addition, multiplication, very occasionally a cross prodcut or a dot product which I googled by their purpose [e.g. "how to find a tangent to two vectors" or "how to check if a vector is aligned with other vector"]).
You do use that math constantly especially in shader tricks which do some visual magic based on angles between stuff but the difficulty is not in the math itself at all.