[tysam_and]

Yeah, it's a very silly article with wrong mathematical reasoning. Hinton is quite obviously talking about a much more information-theoretic approach to the process, but he's phrasing it in people-friendly terms.

What's a little more concerning to me is that people are reading and upvoting it. I think, because I have hopes and aspirations about working on some very hard problems and communicating them to the public at some point. And if this is the level of 'ooh, squirrel'! that we're going at, that the work that I make might get overshadowed by something really silly.

Perhaps an odd insecurity, but there it is, I think.

[nighthawk454]

Yeah I agree with that for sure. It’s so strange how the majority of the research folks appear to be on this ‘new shiny’ mentality at the expense of fundamentals. Especially for how new this field is, relatively. It’s not exactly like we’re all tapped out. Probably not even of low hanging fruit.

[tysam_and]

Yeah, if you read Hinton's long backlog... It's mindblowing. So many fresh concepts, just left there in the dust.

Highly encourage. There's a goldmine in there, I thinksies. <3 :')))))))))

[theGnuMe]

>information-theoretic approach to the process

Can you elaborate on this? I've studied some information theory and I don't see it.

[nighthawk454]

I think the analogy is something like: if you have a simple distribution over all words, then that's just word frequency. Obviously not a good predictor. The 'information' necessary to predict the correct next word contextually is just not there if you're predicting words in a vacuum. In order to be practically useful and predict the right words _in context_, the model must be conditioning off of more of the sentence/document (aka more information). So it should not be surprising that a 'glorified autocomplete' has some degree of "understanding" as it would be impossible for it to be any good as an autocomplete-er otherwise.

[theGnuMe]

That's not information theoretic, that's just conditional probability.

[tysam_and]

You might want to take another look at Shannon's paper, lol, this statement is quite contradictory. Probability _is_ the backbone of information theory, dude! It's quite incredible.

[nighthawk454]

it is conditional probability, but that is a fundamental concept used in information theory