It doesn't feel tidy to have .cache or .config out at the root of home folder either. IMO, those should be pushed down into .local. At least with the xdg base spec you can do that!
That's the thing I always thought was weird: why isn't the default value for XDG_CONFIG_HOME ~/.local/etc, and XDG_CACHE_HOME ~/.local/var/cache? I guess they wanted to make it easier to find for people who wouldn't be familiar with the spec (i.e., most regular users).
Even with .config and .cache, the number of .folders in homedir stays constant vs the number of apps installed, which is the most important thing IMO. From there it could be argued that if you allow .local to exist at all and it only contains always the same three subfolders, you might as well move those up and benefit from shorter, more meaningful paths.