[tomseldon]

Testing the change is one thing, but if the original people were hired and asked something very similar, you have an existing group selected based on that characteristic and so will likely do well.

e.g. if you have always done a whiteboard interview, then you've selected for people that do well in that type of scenario. Changing the question but keeping the format will likely just prove that those people are still suited to that type of task, but not that an otherwise good hire will be filtered out.

I'm not against whiteboard interviews or similar, but just pointing out that selection bias is an important factor here if you're judging effectiveness by comparing it to results for existing employees.