Completely agree. Comparing these numbers doesn't make sense; of course npm has more modules, but why would I care about having e.g. node's express framework available to my build tool/task runner?
You're right. You wouldn't care about everything in npm. The point is this: When using Gulp/Grunt, we have to search for a plugin. This greatly limits our selection.
I see your point, it makes sense. But I have to note that in gulp (at least) you aren't restricted to plug-ins. You can use "generic" npm packages, as long as they output a stream (e.g. browserify).