Hacker News new | ask | show | jobs
by JamieLewis 4574 days ago
I really wish people wouldn't give this as much weight has it gets, there are so many problems with the methodology:

1. The raw data (answers, in particular the free response text) has never to my knowledge been published. 2. The algorithm used to select schools and pupils taking the test has never to my knowledge been published. (In fact it differs from country to country...or city..) 3. The test itself is taken in many different languages, which language version of the test is used in which country, has to my knowledge never been published. 4. They compare Shanghai, a relatively well off city with entire countries..without knowing the data in 1-3 we cannot know if this comparison is fair or makes any sense at all. 5. They administer the test at different times of the year relative to exam time...some teenagers would have just got back from summer break, others would have been deep revision, preparing for other exams - I would guess this has a statistically significant impact on performance.

Until they publish all the data so it can be analysed independently (and god forbid replicated), this isn't science or statistics...it's busy work.