Hacker News new | ask | show | jobs
by _t89y 822 days ago
Modeling language in a latent space is useful for certain kinds of analyses and certain aspects of language. It has its place as an empirical tool. That place is not the nuts and bolts of language itself. There are more suitable formalisms for this than directional magnitudes and BPE tiktokens.
2 comments
throwaway2562 822 days ago
Intuitively I agree with you, despite the unexpected ‘success’ of current approaches. What formalisms do you suggest?
link
_t89y 822 days ago
The Lambek calculus. Categorial grammars. Meanings are proofs. Not clusters of directional magnitudes in space.
link
jll29 822 days ago
Curiously, the upcoming third edition of Jurafsky and Martin [1], one of the two standard text books for NLP, places Context-Free Grammars, Combinatory Categorial Grammars, and logical meaning representations in its appendices on the companion Web site, no longer in the text book itself. Unthinkable only a few years ago.

[1] https://web.stanford.edu/~jurafsky/slp3/

link
_t89y 819 days ago
That's a really interesting thing to point out. NLP doesn't even work on language anymore. If it was adjacent to information retrieval before it is now a subfield of information retrieval. As long as it's grounded in Firth Mode natural language understanding, as it's called, can't really be a semantics.

I tried to create a Kaggle (TensorFlow Hub, TensorFlow Quantum) competition for motivating alternative formalisms but was unable to publish it because all Kaggle competitions must be evaluated with information retrieval metrics. Talk about a one-track mindset!

Today work in NLP advances by ``leaderboards'' and dubious, language-specific evaluation datasets that the same authors stand to benefit from when their proprietary model is praised for doing well on the evaluation criteria they invented a few months back. It validates the price hike for access to their proprietary models.

These formalisms that do work are at odds with Firth Mode, the preferred representation for Google (Stanford, OpenAI), so I guess we should be thankful they're still in the book. If you're interested in language, though, I'd suggest picking up a different book.

link
blackbear_ 822 days ago
What are those formalisms?
link