Hacker News new | ask | show | jobs
by mlevental 2930 days ago
what a weird question: who looks at a search engine and thinks yea hm that's trivial enough i could do it myself in a weekend?
1 comments

Seriously, creating an efficient scalable search engine is among the most difficult computer science problems. From stemming, to combined queries, to word tokenization, to handling various string collations, and language issues, and caching, and parallelization of work, and handling huge numbers of writes, there are so many tricky parts. I used to work for a search startup and I can answer the OP's question: do not try to write your own search engine. That work should be done by someone with a PhD and decades of experience. Even elastic which is great software has issues, such as not being transactional, having issues handling huge numbers of writes, ect.
It depends on the complexity, which language(s) you are using, and how you will parse the search string (which Context Free Grammar it will obey). Just recently, this was a task for me that ended up landing me an interview. A take-home problem. Even a simple search engine needs some clever reverse-indexing for speed. Add any sort of logic like And or OR that not even Google implements and now your parser has to work, and you have to be able to translate from tokenized parse tree with operators to results. It's a great learning exercise for someone with experience, initiative, and enough background with CFG parsing, building a reverse index, and set logic -- but without some key Computer Science building blocks it would end up being quite a challenge.