[whatshisface]

Linear algebra (with diagonalization not just using gauss-jordan) could be pushed back to highschool for motivated students, and is in some countries. The coordinate system aspect of special relativity (the origin of time dilation and most of its "weird effects") only requires algebra. General relativity requires the full mechanisms of differential geometry but advances in things like differential forms are pushing this back to the undergraduate level. Overall I would say that it could be done but you would have to leave the unmotivated students behind.

[abecedarius]

Turtle Geometry gets as far as motion in curved spacetime using code in Logo. Dunno how many high schoolers have ever learned from it, but it's there. (It includes a nice concrete intro to vector algebra earlier, too.)

Re quantum mechanics without many prerequisites, I'm a fan of Feynman's book QED.