Physics has plenty of holes. It’s a changing model of measurable quantities in our world. It doesn’t take a lot of brainpower to fit other systems either into the gaps in physics, or independent unmeasurable areas
Magic violates fundamental principles of our understanding of the world like causality or various conservation laws. You can't just squeeze that into the gaps.