I'd recommend starting from scratch. The whole point of emacs is it's super easy to hack and make your own. So you'd just end up "undoing" everything in a starter kit anyway. But you should definitely look at those starter kits to see what you can do.
I feel like doing that with doom emacs. I'm not sure what is a feature or a package and it feels like a bunch of things are overlapping in terms of functionality.
If you start with framework you might not learn enough to solve problems later, but using it later to simplify your config should be ok. For example ergoemacs http://ergoemacs.github.io/ is a nice way to make Emacs more ergonomic and easier to use(if having problems with default configuration).
I think the only thing I couldn't live without is evil mode. I'd install that right off the bat. I'm using doom but it feels like it's doing too much behind the scenes for my taste.