Hacker News new | ask | show | jobs
by bbuffone 4906 days ago
I have to wonder about the testing results that were obtained. Testing the improvements of optimizations need to be done carefully; you need to make sure capture sufficient data to draw conclusions.

#1 -> You need to capture enough data samples for each location and browser #2 -> You need to capture data from a set of global locations #3 -> You need to capture data from the commonly used web browsers.

You can see a test run from a single location and browser using one sample (2.873) second for the time to interact.

http://www.websitetest.com/ui/tests/50e89376479876092f000012

but when you run the test over 17 location and run 5 samples for each locations. (6.4) second for the time to interact.

http://www.websitetest.com/ui/tests/50e893d7479876092f000016

There is a big difference between the one location and the multiple locations with 5 samples. Looking at just Washington with 5 samples; the time to interact is (4.1) seconds.

(Disclosure, I work at yottaa.com the provider of websitetest.com) For those people looking to verify optimizations are working (automated or hand-tuned) you should use websitetest.com to simplify the testing process. It makes running tests (multiple locations, multiple browsers, multiple connectivities) possible with one click and test results make it easy to draw conclusions.

--- All test data for the information in the comment is available through these links

Tests by browsers in Washington DC -> http://www.websitetest.com/ui/tests/50e89598479876124100000e