[vinceguidry]

The answer to this is to do human rate limiting.

Build your pipeline so that a human has to approve each automated action being taken. The difference between a bot making 5000 network requests in a day and a human making 100-200 semi-automated requests isn't a whole lot in terms of throughput, but makes an enormous difference in terms of quality control and not stepping on toes.

I really wish more companies would do this. Fully automated business procedures that make demands on human attention are just plain awful. Humans should be interacting with humans, machines should interact with machines. Machines can help the human, and the human can help the machines, but interaction points should only be between two of the same types of entity.

Every time I've ever suggested human rate limiting though, I get looked at like I'm a moron. Even when I build the entire workflow myself and tune it so it only takes a relatively tiny amount of time to clean massive amounts of data, people just don't want to do it. It's beneath them. Even when it creates a massive difference in the quality of the product / service you're offering.

[icebraining]

That's an interesting point. I'm not sure I would trust myself to do a better job than automated tests after a (short) while. It's not really a matter of being beneath me; for example, I don't mind doing repetitive manual labor (once in a while).

Just out of curiosity, what's the biggest job of this type you've personally handled?

[vinceguidry]

I made a brief, aborted attempt at a restaurant recommendation service. We wanted to hydrate our data with existing pictures of dishes from the restaurants sourced from Yelp and/or Google Image Search. After looking at that data, I realized that a human touch to picking the right images would make a huge difference in the service.

We're talking thousands of restaurants that we wanted pictures of food from, each of the restaurants had dozens of images we could pull. So tens of thousands of images needed to be sifted through, I figured with the right tooling, myself and my cofounder could put together something really nice that would only need an hour or so of maintenance a day to keep up.

So I built a pipeline that used very basic and easy to build and maintain 'dumb' Rails asset pipeline pages to present data for sifting. Go to the endpoint, it shows you the name of the restaurant and a bunch of images, you select one, type in a name for the dish, and it saves it to the database and puts up another page of images.

It took me bitching up a storm to get him to even look at it. He complained about how long he thought it would take, while I just got to work. Took maybe three weeks to prototype our app. One thing I learned in the process is that if you're looking at a bunch of Southern food, for some reason the picture of shrimp and grits always looks the most appetizing.

I was well on my way to classifying and figuring out novel ways to present the data when I had to make the determination that there wasn't good cofounder fit. So now I work with CNN.

But now all my side projects revolve around ways to get human attention to improve automated tasks. I suppose one of these days I'll get the right idea and/or the right cofounder and I'll give it another go.

There's a wealth of usable information out there on the web that one can build businesses on top of if one only wants to apply a little elbow grease to clean it and turn it into data. It's far easier to scrape data with a regular web browser with a custom browser extension than to try to build out headless infrastructure. But no one wants to do it.

[MrLeap]

Your story reminds me of a tool I wrote for helping lawyers classify a feed texts as a test set for a project.

Our main initiative was creating a heuristic based classifier (think lots of regex). At my own initiative, I trained ML classifiers while we worked on it. As development went on, the ML classifiers were rapidly catching up with the heuristic based one. Unfortunately it was kind of a one off data processing task, and when time ran out the regex machine was still in the lead.

I was modestly proud of the legalese DSL generator I wrote up. The lawyers didn't even know they were writing coffeescript as they typed out what documents were, what key dates were, etc. :D

That coffeescript formed the basis of our accuracy testing suite. It was as fundamental as it was huge. That team ended up creating a couple thousand tests in less than a month.

[ruairidhwm]

I'd be interested in hearing more about this. Fancy dropping me an email (address is in profile)?

[movermeyer]

A project I was working on was a much more useful bot. The number of affected repos for that problem was over 35,000 (although 95% of the value from fixing would be gained by only fixing the top 3,000 repos, when weighted by popularity)

The task of verifying correctness of a pull was much more time consuming than this one. Even if it only took 30 seconds to verify (optimistic), that's 290 hours. Which isn't necessarily all that much for an organization of verifiers (or Amazon Turk), but it is a lot for an individual.

Maybe that should be the cost. But perhaps some things you might be fine with letting a bot do (after manual verification of a statistical sampling, and thorough testing).

The project is currently on hold.

[enraged_camel]

I've been working in the business automation field for over 10 years. I strongly disagree with what you wrote.

The entire purpose of automation (meaning, automation in general, not just business automation) is to take humans out of the equation for mundane and repetitive tasks[1], and have them deal only with exception scenarios and edge cases cannot be properly handled by the machine (either by design or due to system limitation).

Guess what happens when you have humans approve each and every automated action like you suggest? You defeat the purpose of automation, and users end up hating the system because mundaneness and repetition are reintroduced, except in a different context.

[1]The reason you want to do this is because the more mundane and repetitive a task is, the more likely people are to make mistakes, and mistakes can be costly. In fact they are often more costly than the labor itself.

[amenod]

That very much depends on the work you are automating away. For much of the business automation I agree. Invoice processing? Automate. Drip campaigns? Automate. Measuring performance of something? Automate.

But - picking the right photo for some restaurant, as GP stated in another comment? Make good UI and perform manually. Alternatively, train NN in background - but it might not make sense in a startup world where the effort for this would be prohibitively high. In the end it's the photos that matter to your business, not that fancy photo picker algorithm that took ages to develop and that you can't sell to anyone else.

[enraged_camel]

>>But - picking the right photo for some restaurant, as GP stated in another comment?

That’s not a good example though because it is unlikely to be repetitive and mundane. It’s a decision most restaurenteurs make at most a few times for each restaurant.

[mnutt]

Not in all cases, but in this particular case I think the "stepping on toes" part applies. Assuming that some percentage of repo authors do not want your proposed changes, your automated work is now now creating manual work for them to close the PRs. You either need to signal to the repo author that you have put at least as much effort into opening it as they would have to in order to evaluate and close it, or your proposed changes need to be _really_ beneficial.

[sverhagen]

"The answer to this is to do human rate limiting"

So, PRs then?

[icebraining]

vinceguidry is saying the PRs should be reviewed before being foisted on the maintainers of the target repositories.