[donal]

Most definitely, otherwise you are really just eye-balling it and you don't want to find out that your small sample-set wasn't representative of your actual user-base.

This is important considering that during early stages you could have early adopters and other more tech-savvy types using your site more heavily than what you will end up with.

[Shadow84]

You don't necessarily need to give people a specific task.

For example, just ask them to discover the site, watch them in what they do, watch what they spend their time with, discover which problems they have while surfing your site. As always this can, of course, only be a part of your whole testing effort. Watching people solving tasks is also important.

Additionally I have to say, that I am not against A/B or multi-variate testing, in my experience it just is very time consuming (and therefore expensive) and the effort to get results is bigger than with direct testing IMHO. If you think you have the money or have exhausted the possibilities with direct usability testing, A/B or multi-variate testing is a valid option.