[pwf]

Is there a bug in the train function? Using 'lambda: 1' for the default dict along with '+=' means that the first time a feature is encountered, the value is set to 2.

  In [1]: from collections import defaultdict

  In [2]: d = defaultdict(lambda: 1)

  In [3]: d['foo'] += 1

  In [4]: d['foo']
  Out[4]: 2

[jonas21]

I think the code is correct. To implement smoothing, you want to add 1 to the count of every word, regardless of whether it appears in the training data or not.

That is to say, a word that appears once should get a count of 2, and word that doesn't appear at all should get a count of 1.

[pwf]

That makes sense, thanks!

I should learn to read...

[justinpombrio]

This is a perfect example of where a (short) code comment would be helpful. The "lambda: 1" a notable piece of code, but it's hard to tell that at a glance.

[hnriot]

no it's not, it's perfectly clearly documented in code. The smoothing is common practice in this area of software engineering.

[dljsjr]

And yet this is written for people who aren't familiar with this area of software engineering. It's stated so right at the top of the article.

[hnriot]

nonsense, any software dev that can't follow a lambda that adds 1 should be taken outside and shot.

When Norvig says talks of regular folks, he means people with 1/10th his IQ, which is still the top 1%. Norvig is so far out there on the IQ scale that I find it funny when some noob says he's found a bug!