Are they, though? Fair use only covers use for the purpose of criticism and parody (as in, making fun of the original work, not just making up your own lyrics to a popular tune).
> not using YouTube api
Even if you're not using their API, it still counts as a ToS violation. But since downloading YouTube videos is the main feature of your tool, there's not much you can do about it.
There are number of things that court would consider if its fair use, GIFs generally fall into that category, there haven't been a case brought to courts yet, but doesn't mean there won't be.
Also I like to add, I've build number of applications using an API only to regret the time I spent on it later. When you rely on external API, its really not your product and you don't own it, API owns you and every move you make. Instead, I would rather rely on a strong community for support such as YouTube-DL, even though there are issues and its not perfect, still much better than API route, just my two cents.