|
|
|
|
|
|
by ben_w
1176 days ago
|
|
|
> What is the entropy per word of random yet grammatical text? More colourless green dreams sleep furiously in garden path sentences than I have > This does not help you one bit. Dunno, how many bits does ELIZA? I assume more than 1… |
|
|
That is what these 5-11bit estimates are about. Those would correspond to a choice out of 32 to 2048 options (per word), which is much less than there are words in english (active vocabulary for a native speaker should be somewhere around 10000-ish).
Just consider the XKCD "thing explainer" which limits itself to a 1k word vocabulary and is very obviously not idiomatic.
If you want your big if to produce credible output, there is simply no way around the entropy bounds in input and desired output, and those bounds render the concept absolutely infeasible even for I/O lengths of just a few sentences.
Eliza is not comparable to GPT because it does not even hold up to very superficial scrutiny; its not really capable of even pretending to intelligently exchange information with the user, it just relies on some psychological tricks to somewhat keep a "conversation" going...