My team's approach has been to automate real devices and report on any regressions in terms of JavaScript evaluation, layout, paint, garbage collection, etc. Shameless plug: the team is growing and we're hiring an experienced front-end dev. https://nordstrom.taleo.net/careersection/2m/jobdetail.ftl?j...
Oh no Taleo, the corporate ATS blackhole where resumes go to die. I rather not apply with companies that use Taleo for various past bad experiences. The first being the resume and all the details that I type in Taleo goes into HR oblivion. The problem of job applicant tracking & recruitment is not yet solved.
I've recently been through a number of these damn things and have found SmartRecruiters[0] to be pretty solid. Clean UI and easy application flow. The companies using it have been good about actually responding to applications as well, which is always nice.
How are you measuring those metrics on real devices ? Do you use the Chrome tracing timeline ? I wrote a tool called http://github.com/axemclion/browser-perf that does something similar, so was curious !!