[iki23]

Thanks for the frozenset. Yet pickle makes the file 2 times bigger and 20 times slower to parse on my machine:

  >python wordlist.py -w wordlist -p wordlist.pkl
  >python wordlist.py -T wordlist wordlist.pkl 10 3
  2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)]
  words(wordfile): 142.50 best msec/loop
  words_slower(wordfile): 161.00 best msec/loop
  words_normalized(wordfile): 295.84 best msec/loop
  words_pickled(pickled): 2943.66 best msec/loop

The winner is:

  with open(wordfile) as wordlist:
      return frozenset(wordlist.read().split('\n'))

Source: https://gist.github.com/1059725

Wordlist: http://www.freebsd.org/cgi/cvsweb.cgi/src/share/dict/web2?re...

Do you have different results?