[nerfhammer]

better to pickle a frozenset and use that.

  import pickle

  # initialize:
  wordset = frozenset(line.lower().rstrip() for line in open('/usr/share/dict/words'))
  pickle.dump(wordset, open('/tmp/wordset.pkl', 'wb', -1))

  # when you want to use it:
  wordset = pickle.load(open('/tmp/wordset.pkl', 'rb'))

  'bear' in wordset # == True

[iki23]

Thanks for the frozenset. Yet pickle makes the file 2 times bigger and 20 times slower to parse on my machine:

  >python wordlist.py -w wordlist -p wordlist.pkl
  >python wordlist.py -T wordlist wordlist.pkl 10 3
  2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)]
  words(wordfile): 142.50 best msec/loop
  words_slower(wordfile): 161.00 best msec/loop
  words_normalized(wordfile): 295.84 best msec/loop
  words_pickled(pickled): 2943.66 best msec/loop

The winner is:

  with open(wordfile) as wordlist:
      return frozenset(wordlist.read().split('\n'))

Source: https://gist.github.com/1059725

Wordlist: http://www.freebsd.org/cgi/cvsweb.cgi/src/share/dict/web2?re...

Do you have different results?