There's the rub. It all depends on how you define 'free'. Under the FSF definition, it isn't. They would (maybe) argue that this isn't a valid definition of 'free' because it would be like saying you're not 'free' unless you have the right to subjugate your neighbor.
That's why it's ok to run a non-free program on top of Linux or install a non-free driver, etc.