How could it not be? You "download" the stream every time you watch it. It's to everyone's benefit to cache it if you want to watch it more than once. That way you don't use that much bandwidth. You might be served less ads, but you shouldn't be watching them anyway.
The law can make a distinction between bits you downloaded to cache the video as you streamed it, and bits you downloaded to keep a copy offline. Of course, the result is similar- you have a copy of it on your hard drive somewhere- but the actions you took are different, your intentions are different, and the law can look at those things if it wants (which it does, all the time).
Probably only on creative commons-licensed videos (there are some). Otherwise, it might be an unlawful replication.
And yes, there will be arguments about how you're just saving to a file what otherwise comes in via the player. I honestly don't know the definitive answer. :)
Despite all the comments pretending this is all super ok scrapers are obviously not aboveboard. Sites may try to place limits on you or your account but it is unlikely you will go to court over a TOS violation unless you're really abusive.
Streamed bits might have different Colour than bits downloaded through youtube-dl, even if they're the same bits. It depends on your legal jurisdiction.