[goodwinb]

I'd caution making your survey "Hey would you use Web app A?" The Mechanical Turk users want to blaze through the survey as fast as possible and get paid. Clicking "Yes" gets them through the fastest and also lets them please the surveyor (the same bias as your friends or family).

Force them to make a choice. Present Web app A (real), Web app B (dummy), and Web app C (dummy). Make them rank which web app is best.

Set up the survey three ways: A, B, C. B, C, A. and C, B, A. Have a third of the sample take each survey. You would be surprised at the first choice bias with Mechanical Turk. Actually you wouldn't be surprised when you remember that these people just want to get the thing done.

Finally a good secondary survey is to make them rank order features for their worth. This helps find your MVP. (Read more about "conjoint analysis" if this interests you.)