[nostrademons]

What would "extensions" to search look like? Google had "Subscribed Links" from roughly 2006 to 2009, which would let users opt in to receiving links from third parties for certain queries (eg. I added a Javadoc extension that would show me the official Javadoc when I searched for a Java class). Nobody used them. Search isn't a market like mobile phones: it serves an immediate, well-defined need, and there doesn't seem to be a need for third parties to jump in.

[Sven7]

Your comment highlights the problem. When people think search they think Google. Search is bigger than that and Google is in a way through its success and utility, limiting people's imagination when they think about search.

Just look at their menu bar...images, videos, flights, blogs, shopping, books, patents, apps.

Is that it?

Not to mention random. So we just sit around waiting for some benevolent god in Mountain View to say, you know what now let the mortals have...recipes.

If they want to expand that list to the infinite domains it should be covering, it is never going to happen with the resources they have. They need to open the index to tap into its full potential.

[magicalist]

I doubt any "index" exists in the way the words indicates. It's likely highly custom for how it's accessed. What would an API for that look like? Do you want to just be able to do random regexes (that would be awesome...I miss code search)? Do you just want a disk sitting somewhere with all the internet on it so that you can run custom programs on it?

Identifying what a recipe looks like and then providing a search interface that can figure out which of the millions of variations of some soup recipe is what a person is looking for and is more authoritative than others (and not some blog spam with minor (but random) alterations, or written by an amateur with no business in the kitchen) is a hard problem. Crawling isn't really the hard part. It takes a lot of hardware and time, but then you have all this data...that's when the hard part starts.

I'm interested in what others think a useful "index" API would look like, though.

[nostrademons]

If you just want a disk sitting around with the Internet on it, check out Common Crawl (http://commoncrawl.org/).

[DanBC]

I want a search that works for Usenet news. Yes, I understand that Usenet news is dead, but still, effective searching would be nice and Usenet search has been broken for ages and ages.

Another example is targeted search. For example, if I'm searching about mental health stuff I do want good quality sources, I don't want tabloid gossip about celebrities going into rehab.

Or sometimes I want to break out of the SEO trash, and have a bit of serendipity. Search for a term like [spectacles cases]. You get a lot of shops selling pretty much identical cases. What you want is a nicer way to preview those shops (because often the websites are god-awful and their own site searches are much worse than anything Google provides.) WAIT: I just tried this to make sure I was right, and Google have changed the way they show results like this. You get the same ads at the top, with heavily SEOd content links below, but now at top right there's a "shopping" section with links to different cases in different shops. So, that's much better now than it used to be.

Still, serendipity is fun. I remember when you used to be able to use Google to noodle around and find cool stuff. Now? Not so much. That's not Google's fault. The modern web is very different to what it used to be, but it'd be great if there was some way to get access to those smaller sites.