|
|
|
|
|
by syllogism
4323 days ago
|
|
But but...Why? I would've just written: def find_bigrams(input_list):
ngrams = []
last_word = '-EOL-'
for word in input_list:
ngrams.append((last_word, word))
last_word = word
return ngrams
I have trouble seeing the requirement to generalize to arbitrary n as important...If the data is big enough to want n >= 4, it's probably large enough that you'll write this in another language anyway. And n is unlikely ever to be larger than 5. |
|
And n of quite large degrees is not uncommon in hardcore natural language processing, or bioinformatics, both of which Python (wrapping Numpy and Scipy, usually) is heavily used for.
For instance, Chinese doesn't tokenize its words (all the characters are packed) which means you usually end up doing something like taking N-ngrams (of potentially large degree) on the character space, doing a lot of lookups into a dictionary and a language model, and seeing if you can get everything to "fit" so that all characters are accounted for and the resulting sentence makes sense.