[rolandchang]

What would you suggest calling running pilot studies at schools (both affluent and lower-income), like Khan Academy has done with schools in Los Altos and Oakland (http://educationnext.org/can-khan-move-the-bell-curve-to-the...), and getting feedback and suggestions from the teachers who are implementing the pilot programs? Of course they could they do even more (although who of us really knows who they've spoken to and how much/what about), but it's a bit unfair to suggest that they haven't been working with educators.

The lessons they're learning from the massive amounts data they've been collecting from real students (ones outside the pilot programs, who are using their material in an uncontrolled, natural manner) also shouldn't be discounted, and I would argue that it might be as useful as talking to educators can be. It's definitely a Silicon Valley thing to put data up on a pedestal, but should it be valued any less than education research that can be hard to generalize from due to problems with experimental design (like giving extensive training to teachers in the experimental conditions when it's unlikely that most teachers who'll have to implement the same experimental curriculum will have the same sort of training/enthusiasm) or anecdotal evidence from teachers working with one or two classes?