The CA isn't directly compromised so a third party couldn't generate any arbitrary certificate this way. Essentially though, assuming my understanding is correct, it would allow them to be a man-in-the-middle and take copies of the keys & certificates used by this tool, allowing them to use keys and certificates generated by that tool. Also, if such a tool is run by root (bad practise, but not uncommon practise) or other significantly privileged user, they potentially have access to far more.
Yeah, but that's more likely to be noticed in cert transparency and by the website operator, as there's either a duplicate cert in the log or the website server does not work.
If the CA can access your private key, then it can reuse (or worse, redistribute) it without anyone knowing.