If "type bugs are an obvious thing and happen all the time" and "static Typing reduces type related bugs" then it should be easy to demonstrate this empirically. However, "a review of all the available literature (up to 2014), show[s] that the solid research is inconclusive while the conclusive research had methodological issues."
Why would you need an empirical study for this? It’s trivially provable. Runtime exceptions in a language like JavaScript can arise from type mismatches. That’s impossible to do in a language like Java, because the compiler catches it before you ever run the program. This eliminates an entire class of bugs.
What you’re proposing here sounds like somebody saying “How do we know Rust results in less bugs than C++ without an empirical study?”. Even though, we _know_ Rust eliminates an entire class of memory related bugs. I say this as a C++ advocate too. Anytime I run into a memory bug, that’s a bug that would not have happened in Rust. Likewise, any time you run into a runtime exception due to a type mismatch (for example: expected an int not an object), that is a bug that would not have happened with a type safe language.
Edit: I also want to add that the metric is important. Is it number of bugs per line of code? What does that even mean? Assembly programs consist of many more lines of code because it’s more terse, but the number of bugs in assembly will probably be greater than a higher level language. Even though the large number of lines of code would probably push the metric down and make it seem like assembly has a low number of bugs per line of code. Because of this, bugs per line of code isn’t a useful metric.
The only way I could think of measuring this would be to have two feature for feature equivalent projects in two different languages and compare the number of bugs in each. But even that probably has a bunch of flaws.
I think you're right that static typing reduces bugs, but I am not convinced the reduction is significant or meaningful. If static typing has a significant effect, then why is the existing research so weak and inconclusive?
I don't understand your point about metrics and measurement. Are you saying the effect of static typing is so small that it is completely dominated by other confounding factors and thus cannot be measured?
My point about metrics is why I think the research is inconclusive. It’s very difficult to get a metric that’s meaningful in this context. If you said: this code base on average has 1 bug per 100 lines of code, that doesn’t say anything meaningful. If that code is assembly, that’s not very good because of how terse the code is. Whereas, if that code is Python or Ruby, that’s much better because of how concise those languages are.
Because of this, I feel like the only way to truly measure whether or not static typing has a significant effect would be to create two equivalent projects. Say you created stack overflow in Python and in C#. Then you could compare the quantity of bugs and see if it differs. But even this has problems because who knows how many bugs haven’t been caught? Is the code truly equivalent? Did the people who wrote the two codebases have slightly different experience resulting in differing number of bugs?
There’s too many variables in an experiment like this to conclusively determine whether or not static typing reduces the bugs. But, I don’t think that means that we can’t infer that eliminating a whole class of bugs is helpful.
Edit: the more I try to think about my reasoning the more I’m thinking it’s flawed. I think the answer to whether or not static typing reduces bugs is unknowable, but I strongly believe that it helps. Maybe we’ll get a study that isolates this metric one day :)
I think the important question is: at what cost? E.g., if it takes me 4x more time to write statically-typed code, and it saves me 10% fewer bugs (completely made up numbers here), is that worthwhile? Maybe, if I'm programming self-driving cars or autopilot software for aircraft. Probably not if I'm programming a web calendar for dog sitters.
But this is where the studies come in. Lots of people think this is true. And it seems perfectly reasonable. But there's really no research to back this up.
It's even worse than that. If it saves me 10% bugs per unit of code, but I have to write 20% more code, am I actually even ahead in the bugs department?