[robbles]

Your premise that the web is somehow less effective because you can't scrape data from pages easily doesn't make much sense to me.

Have you taken a look recently at the plethora of web APIs for just about every purpose? The modern way of collecting machine-friendly data from a server is through APIs and semantic content (RDFa, microformats, etc.).

Not through HTML / CSS / Javascript formatted pages which are made primarily for human consumption.

[cypherpunks]

Which only works for pre-intended uses, or standards that have substantial market share.

Again, read the literature on agents from the nineties, and see how diverse a technology tree was killed....