Hacker News new | ask | show | jobs
by laurent123456 3570 days ago
They need to filter porn out of their search results (even for common queries like "hat", there's only porn) and perhaps be more resilient to SEO techniques since it looks like there's lot of spam on top results. Queries with common words such as "cat" return almost only irrelevant results.

I'd really like to see that kind of project working as a good alternative to Google, but as it is it's not really usable.
1 comments
deusu 3570 days ago
I hadn't even thought about that. But it should be pretty easy to do in post-processing. I just have to take a list of "porn" keywords. If none of them occurs in the query, but in a search-result, then that result gets downranked.
link
runarb 3570 days ago
If you want you can use the now defunct web search engine boitho.com list of adult words. It is avalibal at https://github.com/searchdaimon/adult-words .

We mostly filtered out porn by using a two word phrase method. There were a lot of edge cases because many potentially dirty concepts are made up of words that are not bad when used alone. For example a text can have both "girls" and "nude" in it without being vulgar, but if it has the phrase "nude girls" the chance for it being pornografic is much higher.

link
laurent123456 3570 days ago
Yes I guess filtering them out would at least make the website SFW, and it would make it easier to show it to people. The issue seems to happen mainly with common words (which results also appear to be polluted with heavily SEO-ed websites).

I've also searched for less generic things like "xperia z5" and the results looked good.

link
deusu 3570 days ago
I have the filter implemented now. It's not perfect yet, but it already filters out a lot of the NSFW stuff. Unless you explicitly search for it.

I'm gonna further improve this over the next days. Right now it's just a quick'n dirty hack. :)

link
rasengan 3570 days ago
It would be a interesting data point to see how many of the 2b pages indexed are adult. lol.
link
deusu 3570 days ago
I don't know. But I do know that the end-of-year statistics from search-engines about what people searched for, are complete BS. I have such a list for the German DeuSu page:

https://deusu.de/blog/2015-12-03-alle_jahre_wieder_wonach_de...

Warning! This is definitely NSFW! :)

link
wila 3569 days ago
LOL, seems that a big part of your users is searching for adult related things.

So if you filter out all the adult stuff you might make those users unhappy. Perhaps make it configurable?

Eg. add a checkbox for NSFW results or something better.

link