I must be missing something, this doesn't answer the question at all. Literally everything you said is true for just building the graph in JS/SVG. What does D3 give on top of that?
Maybe this will help, it creates an SVG graphic by default.
Consider how much JS you'd have to write to even start creating all the frameworks to build in native SVG structures... a lot.
I created html graphs with tables in the late 90s, made Flash graphs in 2000s, upgrades to numerous JS based charting libraries after this, building custom graphs the whole way.
I had looked at D3 so many times over the years when it first showed up online. But it "looked" complicated, no obvious "chart" library, etc...
Finally, when I tried D3, I was like "why the f didn't I start with this?"
Maybe I'm missing something, but it's literally impossible to achieve something with D3 that you can't achieve with JS+SVG, since D3 is a JS library that emits SVG. The question is how much work you can afford to do. You can always roll your own. But you're going to have to a lot of fussing around with things like figuring out where to put tick marks or grid lines, maybe you need to parse some CSV, or figure out how to label dates. There are also things like Voronoi diagrams which are, if not tricky, at least tedious and annoying to implement yourself. And then there's simple statistical operations like making histograms or calculating medians or other quantiles that, well, you can always implement those yourself.
Basically, D3.js strikes a good balance between ease of use and expressive power, that leans pretty far towards expressive power, while still being far, far less work than rolling your own.
If you do it yourself in straight JS, you'll very quickly start building the common tools of D3: various scaling mechanisms mapping the domain to the pixels, data binding against a set of DOM elements, converting an array of data values to the co-ordinates for the d attribute of an SVG path, etc.
Somewhere else someone described D3 as a hammer, and that's a good analogy, because if you don't start with a hammer, it's the first tool you'll make for yourself when you need to pound nails.
D3 particularly excels at graph animations and data transforms (as might be helpful to facilitate graphing). Its one of the most comprehensive graphing libraries around. Well written code that uses it looks like pure elegance. Its amazing how much functionality you can get out of so little code. It does take some time and a solid understanding of javascript and scope.
It's an API for building custom charts. It's a huge step up from having to fully 'roll your own'. If you have to do any kind of vis that isn't available in the various off the shelf libraries, you can put it together with D3 easier than you could if you fully DIY a solution. It's well thought out but you have to use it for that to be more obvious. This book (by the D3 author) got me started, it's worth the hassle of learning [0].
So d3 is actually a _set_ of libraries, of various kind, most of which happen to be useful for data visualisation. Through its richness, there's a good chance that at least _some_ of those libraries will fit your use case well enough. There's nothing stopping you from not using it, and in fact my most recent work was with Canvas, and the only thing I used was d3-scale, which aids mapping datasets into number ranges and selecting ticks.
In another case I might not care about it, but use d3-selection, a mechanism for creating a data-driven DOM tree — but it's pretty clear that you can achieve similar mechanics with React.
Importantly, you don't have to pull the whole thing into your dependencies just because you use a bit of it, and it's light on own dependencies. So just using a small subset of functionality is "cheap."
Consider how much JS you'd have to write to even start creating all the frameworks to build in native SVG structures... a lot.
I created html graphs with tables in the late 90s, made Flash graphs in 2000s, upgrades to numerous JS based charting libraries after this, building custom graphs the whole way.
I had looked at D3 so many times over the years when it first showed up online. But it "looked" complicated, no obvious "chart" library, etc...
Finally, when I tried D3, I was like "why the f didn't I start with this?"