[cookiecaper]

Yeah, I think we're talking past each other a little bit here. It's understood that a company could stop offering some data that a company is dependent on altogether, but I think it's substantially less likely than a company threatening to terminate access to data that already exists (which is what happened here). I also think that if entrepreneurs know that they're free to gather any freely-available information, changes in the commercial offerings or access rules are less threatening.

Of course it's ideal to have a contract with a company that guarantees access to the data stream for a reasonable chunk of time, but the reality is that unless you're already a big shot, platform vendors like Twitter aren't going to give you the time of day for something like that.

If we take away the ability of companies to selectively allow access to data that's available for free to the general public, it gives these guys a fighting chance. The web is a publication platform. It's equally absurd to say "I'm publishing this novel and everyone except kbenson can read it" as it is to say "I'm publishing this data to the general public and everyone except kbenson can access it". You can try to stop someone from accessing your data, but certainly no court is going to consider it reasonable to assist a company in preventing kbenson from reading a widely published manuscript. Data access is substantially the same thing, and should be treated the same way. Keeping non-disruptive entrepreneurs on the same footing as everyone else makes innovation more secure.

[kbenson]

I was responding to "Big companies should not have the ability to destroy small businesses just because they changed their mind about something." as a general statement, since it was phrased in a general way.

> It's understood that a company could stop offering some data that a company is dependent on altogether, but I think it's substantially less likely than a company threatening to terminate access to data that already exists (which is what happened here).

I'm not sure sure. There's regular griping on HN about Google cancelling services, even though they are generally beta services, because people expect them to offered. Some of these people griping are people that built products on Google APIs that were discontinued.

> Of course it's ideal to have a contract with a company that guarantees access to the data stream for a reasonable chunk of time, but the reality is that unless you're already a big shot, platform vendors like Twitter aren't going to give you the time of day for something like that.

A contract? Maybe not. But if they have a clearly defined deprecation policy, that's a start. And if you are paying them, well that's a lot better, since they are incentivized to keep it going because of your (and hopefully others') money.

As for the rest (access to public API), I'm not really interested in arguing it, as we likely agree more than we disagree. :) The only caveat is that direct consumers of the API often don't behave like regular public users, and that may put a strain on the system and degrade performance for everyone, causing real problems and the need to spend real money to fix if you still want to allow unfettered public use.

[cookiecaper]

>The only caveat is that direct consumers of the API often don't behave like regular public users, and that may put a strain on the system and degrade performance for everyone, causing real problems and the need to spend real money to fix if you still want to allow unfettered public use.

I believe that sites can implement technical measures to "enforce" a fairly normal request profile. Third-party consumers will have to comply with that because it will be technically impossible not to do so. Beyond that, there is no need for a legal method to address access. If the data source is concerned about said consumers going through a code path that is more intense on the backend, they will accommodate with a lighter API. :)

[kbenson]

> I believe that sites can implement technical measures to "enforce" a fairly normal request profile.

Doesn't that make some projects and uses, where it needs more than what is regularly allowed, impossible? I think the natural response to that would be "well, we'll just charge them (something/more) and give them special access beyond our normal restrictions", and we're back at square one. Enforcing everyone gets the exact same access is essentially communist theory, not mean in an inflammatory way, but in that it really does enforce some level of mediocrity on everyone (to be fair, as insurance some bad actors makeing things untenable for the rest) in comparison to an approach that allows some companies to pay for for extra access to account for the extra burden which may result in new products that people want.

I'm for making publicly available data available to use as wanted, but I'm not really for enforcing arbitrary limits that prevent whole classes of useful products that people generally might want in the name of being fair, especially when "being fair" might mean contradictory things depending on the side you approach from. The fact that the data is generated by the users, and ownership may or may not reside with users or the aggregator, and current law may not be the best with regard to that just makes it all the more complex. :/

[cookiecaper]

I think we're having a communication issue here again. I do appreciate your willingness to continue the discussion, because it helps me find the points that are unclear and that need refinement, so thanks for that.

I don't think we should force companies not to have a paid API, that's all well and good. What I want to remove is their ability to stop scrapers from gathering the data from the generic public interface just because they don't like the way the scraper is using the information.

[kbenson]

Yeah, we're mostly in agreement on that. I don't believe in locking down data that is provided free of charge to the public in one manner from other methods of access (i.e. eyeballs vs scraping) in any way beyond normal usage rate limiting that applies equally to all.

As it pertains to this specific story, I think Twitter was well within their rights and norms to try to move a special cased user to some more traditional offering of their, or to at least make them justify their usage. I also think it's well within Emojitracker's rights and is acceptable to try to make a public stink if they think they are getting a raw deal. The only thing that I think leaves a bad taste in my mouth is that he did so prior to attempting a good faith negotiation with Twitter.