[Sven7]

Google needs to create a search market place around that index of theirs. They need to open it up.

Search is too big a problem for just a couple people at Mountain View to be working on. And at this stage with the amount of data being generated, no one really has the infrastructure to compete.

Hope they do it before the regulators make them. If Apple has succeeded at building a market place around their closed platform Google can too.

[rgrieselhuber]

I'm pretty sure that more than just a couple people work on search at Google. ;)

[Sven7]

More than a couple work at Apple too. But the 50 billion app downloads came from an ecosystem of millions of developers around the world.

[nostrademons]

What would "extensions" to search look like? Google had "Subscribed Links" from roughly 2006 to 2009, which would let users opt in to receiving links from third parties for certain queries (eg. I added a Javadoc extension that would show me the official Javadoc when I searched for a Java class). Nobody used them. Search isn't a market like mobile phones: it serves an immediate, well-defined need, and there doesn't seem to be a need for third parties to jump in.

[Sven7]

Your comment highlights the problem. When people think search they think Google. Search is bigger than that and Google is in a way through its success and utility, limiting people's imagination when they think about search.

Just look at their menu bar...images, videos, flights, blogs, shopping, books, patents, apps.

Is that it?

Not to mention random. So we just sit around waiting for some benevolent god in Mountain View to say, you know what now let the mortals have...recipes.

If they want to expand that list to the infinite domains it should be covering, it is never going to happen with the resources they have. They need to open the index to tap into its full potential.

[magicalist]

I doubt any "index" exists in the way the words indicates. It's likely highly custom for how it's accessed. What would an API for that look like? Do you want to just be able to do random regexes (that would be awesome...I miss code search)? Do you just want a disk sitting somewhere with all the internet on it so that you can run custom programs on it?

Identifying what a recipe looks like and then providing a search interface that can figure out which of the millions of variations of some soup recipe is what a person is looking for and is more authoritative than others (and not some blog spam with minor (but random) alterations, or written by an amateur with no business in the kitchen) is a hard problem. Crawling isn't really the hard part. It takes a lot of hardware and time, but then you have all this data...that's when the hard part starts.

I'm interested in what others think a useful "index" API would look like, though.

[nostrademons]

If you just want a disk sitting around with the Internet on it, check out Common Crawl (http://commoncrawl.org/).

[DanBC]

I want a search that works for Usenet news. Yes, I understand that Usenet news is dead, but still, effective searching would be nice and Usenet search has been broken for ages and ages.

Another example is targeted search. For example, if I'm searching about mental health stuff I do want good quality sources, I don't want tabloid gossip about celebrities going into rehab.

Or sometimes I want to break out of the SEO trash, and have a bit of serendipity. Search for a term like [spectacles cases]. You get a lot of shops selling pretty much identical cases. What you want is a nicer way to preview those shops (because often the websites are god-awful and their own site searches are much worse than anything Google provides.) WAIT: I just tried this to make sure I was right, and Google have changed the way they show results like this. You get the same ads at the top, with heavily SEOd content links below, but now at top right there's a "shopping" section with links to different cases in different shops. So, that's much better now than it used to be.

Still, serendipity is fun. I remember when you used to be able to use Google to noodle around and find cool stuff. Now? Not so much. That's not Google's fault. The modern web is very different to what it used to be, but it'd be great if there was some way to get access to those smaller sites.

[taliesinb]

It's probably smaller than we all think. Google has a lot of products, even after graveyarding a large number of them.

[taliesinb]

This is a great point. Gabe Newell makes the same point about the gaming industry: the short-term view is that gaming is about selling units. The long-term view is that gaming is a cultural activity that can be monetized in an almost infinite number of ways.

[new299]

Perhaps, but the model as also been tried and failed (Inktomi).