If you control the domain then yes you could. But if I want to put a link on my website to some SVG hosted elsewhere and I want it to be safe for you to open that link in a new tab then there's not really a way for CSP to protect you the user from the host deploying a malicious SVG.
Like opening a PNG in a new tab is harmless but opening an SVG in a new tab is opening a pretty substantial can of worms.
If your threat model is “I don’t want the image I’m hotlinking to be replaced with something else when opened in a new tab”, then no image format is safe.
Well as an example: Lets say I maintain a hypothetical appview for an atproto service and we support SVGs. Users can upload SVGs via our appview or directly to their PDS and we pick them up when the network propagates record updates.
So users can view SVGs embedded in our site and they are regular vanilla SVG images. But say the user copies a link to this image (which we serve via our site or a CDN).
They share the image to a friend via URL and their friend clicks the link opening it directly in firefox or chrome. Now all the scripts in the SVG can execute and the image can rewrite the DOM to present itself as a fake website prompting them to log into their bluesky/atproto account to view the content. So said friend types their credentials in and the script in the SVG sends that back to their C&C server.
Like opening a PNG in a new tab is harmless but opening an SVG in a new tab is opening a pretty substantial can of worms.