Indeed! This is an endemic issue in the computer sciences.
I don't think it's a bad way to approach knowledge dissemination by the way, it is however indicative of the problem of reprouducability and explainability in AI broadly.
I bring this point up simply to be another voice stating that we need more rigorous methodology in AI research if we are going to make advances that are focused first on knowledge, rather than primarily applicability of technology.
Way back before AI was cool, there was a good paper on this [1] that is very relevant to today.
Quoting from the abstract:
"There are two central problems concerning the methodology and foundations of Artificial Intelligence (AI). One is to find a technique for defining problems in AI. The other is to find a technique for testing hypotheses in AI. There are, as of now, no solutions to these two problems. The former problem has been neglected because researchers have found it difficult to define AI problems in a traditional manner. The second problem has not been tackled seriously, with the result that supposedly competing AI hypotheses are typically non-comparable."
I don't think it's a bad way to approach knowledge dissemination by the way, it is however indicative of the problem of reprouducability and explainability in AI broadly.
I bring this point up simply to be another voice stating that we need more rigorous methodology in AI research if we are going to make advances that are focused first on knowledge, rather than primarily applicability of technology.
Way back before AI was cool, there was a good paper on this [1] that is very relevant to today.
Quoting from the abstract:
"There are two central problems concerning the methodology and foundations of Artificial Intelligence (AI). One is to find a technique for defining problems in AI. The other is to find a technique for testing hypotheses in AI. There are, as of now, no solutions to these two problems. The former problem has been neglected because researchers have found it difficult to define AI problems in a traditional manner. The second problem has not been tackled seriously, with the result that supposedly competing AI hypotheses are typically non-comparable."
[1]https://link.springer.com/chapter/10.1007/978-1-4471-3542-5_...