[tlb]

What drags the US down most in these rankings is "tertiary efficiency": roughly the fraction of people in grad school or with graduate degrees.

Ranking countries is a dodgy business, even more than ranking colleges. A different set of weights or ways of measuring things could give you totally different answers.

If you're looking for a way to claim the rankings are biased, you might argue that this up-ranks countries that value credentials over actual innovation. Or you might claim that these days, an undergrad education is enough to go out in the world and innovate and that countries that send more students through grad school are wasting their time. Or you might claim that the US is a developed country with a developing country attached, which drags down the averages. And probably California, NY, MA and a few other states considered independently would rank highly.

[electrograv]

When casual readers see results titled "Bloomberg Innovation Index" (myself included, a while ago), there's a tendency due to brand reputation (Bloomberg!) to believe that this is some scientifically and statistically rigorous analysis of a well-defined "innovation" that can be trusted. This is dangerous, and IMO disappointing (I know, I was naive and optimistic) regarding the statistical credibility of many "reputable" sources.

In reality (as you describe), these are completely arbitrary human-designed heuristic scores with most likely no statistical significance.

I really wish we could qualify these "rankings" with a more honest term , like:

"statistically useless, arbitrarily rated average of multiple human designed score scales, meant to loosely relate to some quality we want to measure, but in reality is more a game of politics and adversarial score optimization."

But that doesn't have the same 'ring' to it as "top country rankings in innovation".

[didibus]

It sounds a little like denial. I mean, Bloomberg does a great job, its pretty simple: "The 2018 ranking process began with more than 200 economies. Each was scored on a 0-100 scale based on seven equally weighted categories. Nations that didn’t report data for at least six categories were eliminated".

So its an equally weighted average of 7 categories. The data was reported by the nations themselves.

Now the key takeaway is that the US dropped out of the top 10. Comparatively, you can tell something is changing in the US causing a drop.

Metrics are just indicator, and can sometimes misrepresent the reality, but more often, there's truth in the metrics also. Its hard to say what the impact of this innovation score is, is it economic, or is it social, but clearly the score change is due to real realities changing.

[totalZero]

> Comparatively, you can tell something is changing in the US causing a drop.

The reported numbers from other countries, or the reported numbers from the US, could also be flawed. Just because numbers have changed relative to one another, doesn't mean that countries have done so as well. You are assuming a particular causal relationship when there are several plausible alternative theories available.

> more often, there's truth in the metrics

Metric isn't the right word. It would make sense for tangible qualities like area, population, and even GNP. But the measured quality here is "innovation," with an amalgam of other characteristics taken as the one true proxy thereof. It's a ranking, but it's not actually measuring innovation.

> clearly the score change is due to real realities changing.

It could also be a change in reported numbers. Or even random noise that tends to revert. These numbers may not even be relevant to the true seeds of innovation.

[toomanybeersies]

You've hit the nail on the head I think.

It's the change in relative rankings that is the useful data from these rankings, rather than the absolute ranking.

[oh_sigh]

Presumably OP is calling into question any such ranking, not just the recent ranking where the US dropped out.

> Comparatively, you can tell something is changing in the US causing a drop.

Yes, one of the factors they used to come up with this rating. Now - the question is, are those factors actually correlated with innovation?

[muninn_]

Sorry but not buying it. The selection of even 7 categories, let alone suggesting that such categories should be equally weighted is unreliable at best. The nations self-reporting the data is completely irrelevant and, again, at best unreliable.

Now, maybe the US isn't as innovative as Sweden is. Ok. What exactly does that mean? Why do I care if the percentage of graduate educated people is higher? How does that actually affect innovation? Are those people releasing new, globally-changing products and services? What are some examples?

What does it mean if Samsung has more US patents than any other company besides IBM? Is IBM more innovative than Google?

It's fun and popular to bash the U.S. (has been for some time) but I really don't see much meaning behind these rankings. It's not an in-depth study. Amazon has more criteria for picking a HQ. Do you really think Bloomberg can look at these '7 criteria' and come up with a meaningful estimation? No.

[watwut]

> Now, maybe the US isn't as innovative as Sweden is. Ok. What exactly does that mean?

It means exactly the same thing it meant when US was among top. There were people who were interested in it and sometimes happy about it. Those very same people are still interested, but this time wonder whether it means something is changing for worst.

[muninn_]

So it was meaningless still. There’s nothing to be interest about here unless unfounded clickbait is a hobby.

[rxhernandez]

You should apply this kind of logic to scientific papers too. It's infuriating that some papers manage to get published that have the type of errors that would at the very least gotten a letter grade taken off(if not an F) for papers I wrote in undergrad physics labs.

[electrograv]

Oh I absolutely do, and IMO this scientific irresponsibility is far more dangerous within fields that actually call themselves "sciences" -- at least these "indexes" don't claim to be scientific (even if some people erroneously assume otherwise, such as my former self). There are certain fields with the word "science" in the title that are notoriously awful in this respect. (I won't name names, just for the sake of simplifying the discussion, but it's not hard to find out.)

Even the notion "hard science" vs "soft science" is a very slippery slope, IMO. We should not seek to speak of "hard" vs "soft" science; instead, we should focus on distinguishing good from bad science. Even science performed on incredibly difficult and complex topics (with immensely numerous confounding variables, near impossibility of controlled trials/experiments, etc.) can still be done correctly! One simply needs to adhere to statistical rigor and qualify conclusions from data with appropriate levels of uncertainty, withhold conclusions with no predictive power or statistical significance, and publish meaningful negative results just as frequently as positive results.

This movement of lax scientific rigor within fields that call themselves "sciences" is incredibly dangerous and threatens to erode the credibility of all results/fields that call themselves "scientific" in the eyes of the general public -- the vast majority of whom do not have the time, energy, or ability to review each field and/or publication to understand how rigorous and honest it actually is.

When you read about problems of anti-intellectualism and public distrust in science, the first thing we should do is look to the "sciences" (and bad journalism) that justify this distrust.

[CandidlyFake]

You forgot the elephant in the room. Money. Everyday these writers have to make up the most outlandish "story/fake news" to get you to click to sell you ads.

"statistically useless, arbitrarily rated average of multiple human designed score scales, meant to loosely relate to some quality we want to measure, but in reality is more a game of politics and adversarial score optimization." while honest and true doesn't get the masses clicking. Honest and true doesn't get stories bumped to the frontpage of HN.

I do feel for these journalists. They are like daily vloggers who have to deal with the constant pressure of generating content every single day to make money. That's like clickbait is so rampant in both the traditional and social media.

[jandrese]

In other words it is about as rigorous as any other Internet listicle[1].

[1] https://en.wikipedia.org/wiki/Listicle

[TimPC]

These rankings need to account for mobility of high profile graduate students. For example some of the top CS programs in Canada have 50-75% of graduates leave for the US tech sector. Canada gets rewarded for it's STEM grads who end up working for US companies in the US market while the US gets punished for it's lower domestic STEM grad production. If the rest of the world is producing a fairly substantial number of educated people who work for US companies on US innovations does it make sense to say it's less innovative? Sweden is #2 in Large part because there aren't categories that measure Venture Capital Invested in Country or New Business Formation. Is it really the case that a few more net patents filed is more valuable to innovation than having companies like Dropbox, AirBnB and Stripe? (Feel free to pick your start-ups of choice here).

[electrograv]

Weighted score rankings are useless / statistically irrelevant, no matter how much careful hand-tuning we do. The only way to maintain accuracy and scientific responsibility in such things is to:

1) Formally define what is meant by "innovation" in terms of clearly measurable outcomes.

2) Measure this clearly-defined quality among all countries and many sample points through time.

3) Try to separate out explanatory variables for the quality being measured. Build these data driven statistical models to model this formally defined "innovation" quantity -- not using hand-tuned weights of various measures, as these "rankings" or "indexes" often do.

4) Try to predict a probability distribution of the "innovation" quality, using models developed in step 3.

Step 1 should be qualified with explanation that this human-designed definition is an imperfect, and that all results should be understood in the context of this formal definition.

Step 2 should be qualified with notes of any possible limitations in the sampling methodology (availability of data, etc.) and how this factors into error margins.

Step 3 should be qualified with sufficient explanation that it's a model of reality derived from data, and therefore risks overfitting/underfitting/etc. errors.

Step 4 should be qualified with an explanation that this is a prediction based on the above model fit, and therefore is subject to potential errors compounded by any of the previous steps.

That would be the scientifically/statistically responsible and rigorous thing to do. But I suppose I'm crazy to expect Bloomberg to aim for any level of rigor in these "indexes".

[chrisseaton]

> Formally define what is meant by "innovation" in terms of clearly measurable outcomes.

Surely you can see that this is not possible, and any attempted would be superficial and would be gamed anyway?

[electrograv]

“Innovation” is a very slippery term, so I agree defining it would be more difficult than other measures. So for simplicity, let’s use something a bit simpler, like a “quality of living index” or “overall human well-being index”.

It’s not feasible for a human to define such a metric formally upon ‘environmental variables’ (such as education stats, graduation rates, etc.) — quite obviously, as you say — yet trivial to define it as an “outcome measure”, where we directly measure the quantity in question (no matter how difficult or expensive to sample this variable).

To the “quality of living index” example: One could design a polling methodology to fairly reliably gauge people’s overall happiness and satisfaction in a country. This polling would be expensive, so we couldn’t do it super broadly or super frequently — and that’s why we use the subsequent steps described in my parent post (on forming statistical models to separate out connected variables that we can easily and cheaply measure to approximately model the “ground truth” happiness metric).

You can then use this “model fit” to predict this extremely expensive “ground truth” notion of individual happiness in this case, on a much more frequent and granular basis than would ordinarily have been feasible using a ground truth gathering method like a polling process.

[Numberwang]

There are more successful startups in Sweden per capita than in the US. Just get over yourself.

[logfromblammo]

I have seen enough "brilliant stuff that I can't talk about" in my career that I think you really can't ignore military/paramilitary R&D spending. But nor can you really measure the innovation that comes out of it, since countries are largely going to keep their coolest, newest stuff as national secrets.

The innovation you can see and experience in a product like Facebook or VKontakte is just the tip of the iceberg when it comes to the unseen tech from the three-letter agencies of many different countries, that are mining, scraping, and reducing the data.

But there's a lot of anti-innovation going on in that sector too, I'm sure.

[angarg12]

The fact that Germany ranks 4th in innovation makes this list not very meaningful to me. Germany is one of the most conservative and tech-averse countries I know, and they are struggling to get theirs businesses up to date.

[TheAdamAndChe]

Seeing how developed nations(particularly with service-based economies) depend on ever-increasing specialization to drive economic growth and efficiency, what makes you think the percentage of grad students and graduates _wouldn't_ be a good proxy for relative economic strength?

[mindcrime]

At first blush, the idea that comes to mind is this: Innovation and associated economic growth isn't the exclusive domain of grad students and graduates. Undergraduates can innovate, as do lots of people who are in private industry and not associated with academia at all.

Of course all that means is that it isn't a perfect proxy. It may well still be a good proxy. Tough to call.

[slededit]

Others can innovate but it really isn't their full time job

[simula67]

I thought "Tertiary education" includes undergrad : https://en.wikipedia.org/wiki/Tertiary_education

[ojbrien]

Here's the indicators that bloomberg uses for constructing the factor:

1. Gross tertiary enrollment ratio

2. Percentage of working-age population with advanced level of education

3. Annual new science and engineering graduates as a percentage total tertiary graduates

4. Annual new science and engineering graduates as a percentage of the labor force

[notatoad]

>Or you might claim that the US is a developed country with a developing country attached

huh?

[ddebernardy]

You've never been to central California or the Mid West? In contrast with the West and East coasts, they look about as developed as Belize - including in Hollywood movies, I dare add. It's like, there are a few mansions here and there, but there are also a lot of crack shacks and developing world worthy public infrastructure.

[muninn_]

"California is not on the coast because it doesn't fit my narrative where I bash poorer parts of the country"

[DaniFong]

In the USA we crucify our heroes and our workers and our ethnic and sexual minorities. LGBTQ people are constantly hounded, and people belief the worst rumors and smear articles about women, but nothing about men.

luckily my friends and I are black-shifting these deep blue pigs, who eat people, and are creating a new era where people are allowed to write their own story.

[11thEarlOfMar]

So how would one measure a region's demonstrated capacity for innovation, in a meaningful way?

[defertoreptar]

Imo focus on output of new technologies.

[phillc73]

I can see flaws in that potential approach too. For example, if a country has high R&D investment/capacity, but most manufacturing or commercialisation activities happen abroad, then simply measuring output of new technologies doesn't account for the R&D.

Using the Bloomberg list, Austria ranks 12th overall, but 5th on R&D. Their figure is dragged down by a 26th rank in High-Tech Density. Then, Germany's High-Tech density is ranked 3rd overall. Perhaps one could hypothesise that much of Austria's R&D effort ends up commercialised in Germany.

[defertoreptar]

Yeah that's a good point. You can look at where a company is incorporated, but they still may do R&D in different parts of the world. Could be resolved by taking the "new technologies output" of a company, but then dividing it to each country it does R&D in with proportion to the cost of that R&D.