Ask HN: Why developers hate scraping?

[dogreborn]

Over time we've built products like : adplexity.com (5million USD annnul revenue, 100K annual expense)

But why aren't there more developers interested in scraping products?

[onion2k]

The legality is complicated, the tech challenges are hard, and, to be honest, it just feels a bit off to profit from other people's work without giving back to them.

Question for you - why do you care what people think if you're making $4.9m profit a year?

[dogreborn]

In absense of competition, life is pretty boring.

A scraping company can only pickup limited number of ads from limited profiles.

If more companies start scraping, scam and advertisers will be caught faster and prosecuted once vigilantes use these tools to report fraudlent ads.

[rajacombinator]

5mil/yr on 100k/costs sounds pretty interested to me ... what exactly are you complaining of? Not enough API users? When I click your API tab, it just has a “contact for pricing” pop up. You’d probably get more interest if you had open documentation/some examples of use cases/upfront pricing.

[deytempo]

The fun part is knowing you are being scraped and adding tagged data values that you can scan for elsewhere on the web to see where your scraped data is going

[wprapido]

Says who? I particularly enjoy it and find it extremely entertaining and a great opportunity to brush up my skills, while making some money in the process.

[sh87]

1)Legality 2)Morality

[dogreborn]

2. About morallity, ad tech companies are hyper targeting people with ads which some evil advertisers take advantage of and bypass moderation.

Ads are targetted in such way that expert users and regulators do not see them but the naive ones get tricked.

If we scrap ads using different visitor profiles and list all ads, along with offensive/malware ads... wouldn't it help regulators moderate ads?

So is it still morally wrong to go against ad tech companies by scraping their scam/tricky ads?

[sh87]

I was referring to scraping content, not ads. Not sure what benefit ad scraping provides and to what end ?

[dogreborn]

1. Sell data to ad regulators

2. Sell data to competitors of the advertisers, turning online advertisement less effective in long term and easier in short term.

3. Sell ads data to activists etc...who will sue the company running misleanding ads and defrauding consumers.

[sh87]

I did not know there was a market for this.

[dogreborn]

Also what's the latest tech stack for easy scraping?

Most new developers are not able to scrap using regex, css selectors etc... So is there any new advancement in scraping field?