Hacker News new | ask | show | jobs
by mdellabitta 3297 days ago
Hey, I'm wondering why you didn't consider using stopwords to prevent bloated inverted index entries fir words like 'the'?
1 comments

We don't use stopwords because we want to find all the best and complete matches. We don't want to ignore any of the words part of the search query.
You do use stopwords. Your most common unigrams are not in the index, by design. You just use your own stopwords.