Accessibility is key IMO. And why it's sad the standard components don't get more love. Like why in 2020 can't select dropdowns do basic matching on any part of optiond instead of only leading characters? Or show filtered results as one types?
Also SVG has also some very ugly things in the spec. And you would have to build up every UI-Element by yourself. It would have nothing to do with SVG, if it would be like HTML.
Years ago I heard someone saying that SVG should have replaced what CSS does nowdays, i.e use it for anything graphical. Not sure how well it would have worked though. But it is vector graphics so maybe very well?