[acdc4life]

I see BERT as non sensical. You need to be scientific and have a mathematical theory on how humans learn language, which is a multi disciplinary task requiring physicists, mathematicians, neuroscientists, cognitive psychologists and linguists. Benchmarks are useless, theories, models, experiments and testable predictions is how science progresses. You’re making a comment on cognitive science, and trying to imply that language learning in humans isn’t learned, but pre baked. The psychological, linguistic, evolutionary biology and neuroscience evidence doesn’t seem to corroborate. The evidence points stronger to humans having general learning and problem solving abilities. For instance, there was no evolutionary pressure for humans to be good at math or programming. I was not born knowing english or calculus or probability theory, these were learned abilities. Evolution favoured brain mechanisms that lead to behaviour for success in a rapidly changing world. Had I been born in ancient Rome as a farmer, I would learn to speak Latin, and learn how to be a successful farmer, instead of the physics, math, probability, computer, driving, reading skills that I learned in my life time.

[bumby]

>You’re making a comment on cognitive science, and trying to imply that language learning in humans isn’t learned, but pre baked.

You make good points, but this one isn’t quite what I meant. I didn’t mean we are trained by evolution for a particular language, but that evolution selected for a language skill. In other words, we are “pre-baked” with the ability to learn a language. It would be analogous to a DL model being trained for generalized regression but not a particular problem. To that extent, I think we’re saying the same thing. Some of the theories related to our generalized learning abilities postulate they stem from this base ability (our aptitude for music, for example, being a consequence of our language learning ability)

>I see BERT as non sensical. You need to be scientific and have a mathematical theory

This is a matter of contention. A lot of science progresses by starting with an empirical result that drives a change in theory rather than the other way around. I can’t remember who to attribute it to, but there’s a quote to the effect that “‘Thats odd’ is the most productive phrase in science, rather than ‘Eureka!’”

[acdc4life]

> evolution selected for a language skill

Yes it did, and we do know a lot about general principles behind, from different disciplines. This is still an early science. NLP research still hasn’t considered many important aspects of language learning that we have discovered in such a short period of time.

> starting with an empirical result

What makes you think these benchmarks are empirical? They were hand constructed to fit some objective, assuming that being good at the said objective is required for NLP tasks. Where’s the empirical experiments to validate the notion that said objectives lead to language? Science hasn’t worked this way, datasets aren’t constructed, you do an experiment and MEASURE it. Then you make models, try to explain the phenomenon, and test new ideas and validate your models. Can your model extrapolate new information and suggest new experiments to validate? I used the word extrapolate over predict intentionally.

[bumby]

It’s a little hard for me to follow your last paragraph but it sounds like you are confusing AI and AGI. I don’t think anybody using BERT is claiming the latter.

I assume the empirical results are the validation sets run. I.e., the tests that show it provides better results than the base rate. Again, it’s important not to conflate verifiable results with understanding the underpinnings of why it works. If you’re walking and I race you with my car over and over again, I can conclude my car is a faster mode off transportation without understanding anything other than “push right peddle to go faster”. My ignorance doesn’t invalidate the results