[fibertera]

What kind of legitimate uses are there for something like this? This is not a sarcastic question. It seems like an obvious spam magnet, but if people are using it legitimately wouldn't their sources already be providing an API or RSS key?

[antjanus]

I've my own use case for it and it will probably mirror other sites. I run my own blog and thus have ads and affiliate links there. The thing is, as good as Google Adsense is, it's shitty for my site and my topic (Web Dev).

What am I left with? Great affiliates like Team Treehouse, Lynda.com, framework themes, and Udemy. The problem is that none of those offer any kind of a good API. All they have is a link and possibly an image that they provide.

By using Kimono, I can scrape (but I don't) all of Udemy's programs, categorize them with custom categories, build a full-text search engine around it and serve relevant ads per post. For instance, my "Best Bootstrap Themes" post would yield "Learn bootstrap" udemy course and an on-the-fly-but-cached image for it thus serving relevant ads to my users.

Same goes for Lynda. If someone lands on "Why C# is a great language to learn" (one of my unreleased articles), my custom API built on top of scraped data could serve them with a "ASP.NET Essentials" course.

So why use something like this for framework themes? Take Wrapbootstrap.com, they have a great affiliate program. Using Kimono, you can easily get daily refreshes of their main page which usually has: sales priced themes, featured themes, and new rising themes. This way, you can serve users with an ad that has up-to-date prices and themes that are hot right now.

What about non-ad uses? You can create custom search, weighted according to YOUR metrics and build your own marketplace front and aggregate several sources in order to serve users with better content.

[kyriakos]

We use scraping to gather product prices from online shops for a price comparison site. I have permission from the sites who are not bothered to provide us with a price list other than their public website. Legal and necessary - so there is a market for this I believe, I am not sure about its size though.

[tomschlick]

Doing something like http://openstates.org/ is a perfect example. State government data is shitty most of the time and doesn't have a public api you can query so open states runs 50+ scrapers to get the data and normalize it.

[troels]

Very few companies can figure out how to provide proper api's. Unless it's part of their core business, it'll always be lacking.