[eyelidlessness]

I don’t think the author is trying to insinuate that GitHub is in the wrong in any way. They explicitly say they understand the decision, and anticipated that it would happen.

I don’t want to quibble with the term “abuse”, because I think in this scenario it depends on whether intent is a factor and whether we should trust their stated intent. But depending on how you look at it, GitHub would be just as likely to benefit from hiring the author as they would from banning.

[ranting-moth]

Load testing someone else's system resulting in a manual staff intervention due to potential system destabilization at 6am?

It's a wordplay to call that anything else than abuse.

[marvin]

I don't know what happened behind the scenes. But I thought it was amusing when one of our mutual fund customers tried purchasing one share of each of the ~1000 funds we offered in our catalog. It obviously broke things; there were various UI things we hadn't tested for that kind of portfolio.

[eyelidlessness]

Okay, but I explicitly declined to quibble with the term. But I’ll go one further: I’ll concede the action qualifies as abuse. If there’s any quibbling worth quibbling, it’s between whether the author’s figurative abuse-hat was white or grey.

[adamckay]

> GitHub would be just as likely to benefit from hiring the author as they would from banning

For what purpose?

Creating an infinite loop that updates a file and commits it is hardly worthy of a job offer.

[eyelidlessness]

They seemed fairly surprised by the fact it happened, and let it go on for some time. Which strongly suggests they hadn’t considered such a load test on their own. If I had a budget/head count, I’d at minimum put out a feeler for a QA role.

[adamckay]

But the next largest legitimate repo is going to be something like 20M commits shy of that, so it seems like it's excellent that GH engineers only just started to care.

It's entirely possible that such a load test has been considered, but deemed non-realistic so not prioritised for some time. If I were running the QA team I'd be annoyed if time were spent on abusive destructive testing than realistic testing that real-world users may experience, especially because load testing like this would have to be on an identical environment to PROD so rather expensive.

It reminds me of that old QA joke:

A QA engineer walks into a bar and orders a beer. She orders 2 beers.

She orders 0 beers.

She orders -1 beers.

She orders a lizard.

She orders a NULLPTR.

She tries to leave without paying.

Satisfied, she declares the bar ready for business. The first customer comes in an orders a beer. They finish their drink, and then ask where the bathroom is.

The bar explodes.

[blowski]

In your joke, QA was only doing exploratory testing. Somebody - perhaps the builders, bar staff, or QA - should have also been doing integration testing for key user stories, and the system has failed because nobody ensured that was happening.

GitHub hasn't failed here - it continued to perform at normal levels for other users, so far as I can see, and they had an upstream process which caught the issue without the system failing. Maybe some exploratory testing had previously identified where that process should kick in, but without having an automated process since it was so unlikely to happen.

[rewmie]

> Which strongly suggests they hadn’t considered such a load test on their own.

Not really. GitHub has been around for over a decade. People bother with problems that have a realistic chance of happening. If GitHub didn't bothered to rate limit commits it means it was a potential issue that didn't manifested itself for over a decade.

People tend to bother about problems that happen. Otherwise everyone would be freaking out because of killer asteroids.

[sverhagen]

Where are you reading that they're surprised?

[eyelidlessness]

They asked with more than passable benefit of the doubt what the user intended. And they asked quite a ways after the user noticed local degradation. “Surprised” might be the wrong term, but it definitely doesn’t seem like a specific guard was in place for the scenario.

[rewmie]

Was there a guard needed? I don't think so. It seems GitHub didn't saw any degraded performance and barely noticed the issue, and odds are they presumed the author screwed up with their GitHub actions configuration. Once they determined it was plain old abuse, I'd guess some GitHub employee said "what a moron" and proceeded with his day.

[eyelidlessness]

What are you basing this on?

[apetresc]

Hire them? Why? There’s nothing technically clever or novel here. Anyone can create a shell script to generate random commits and push them. I’d bet even GPT-3.5 could handle that. Why should GitHub hire them?

[eyelidlessness]

> There’s nothing technically clever or novel here.

Nothing technically novel. But evidently it was at least a somewhat novel stress test execution for GitHub’s live systems, otherwise surely it would have been dealt with sooner and messaged with less benefit of the doubt to the user.

Investigating the limitations of something doesn’t have to be novel to be interesting. It’s been a while (I think), but for example there’s been plenty of praise here for Netflix’s own stress tests of its live systems. The tests are often really mundane, eg just shutting some stuff off or triggering known error conditions. It’s interesting not because the nature of the fault is novel, but because systems are complex and it’s a way to learn about their failure modes.

I’m also not saying GitHub should hire them (and kinda I doubt they’d want a QA offer based on reading other blog posts on their site). Just that a hire would plausibly be similarly beneficial to a ban.

[rewmie]

> Nothing technically novel. But evidently it was at least a somewhat novel stress test execution for GitHub’s live systems, otherwise surely it would have been dealt with sooner and messaged with less benefit of the doubt to the user.

Not really. This is boring stuff, and odds are they never bothered with it because a) it has no impact on operations, b) the blast radius of this doesn't go beyond the attacker's own repo, c) no moron with time to kill bothered attempting this stunt until now.

Probably now some low-level employee at GitHub needs to add a metric and an alarm to react to rate limits to prevent moron copycats from pulling this stunt for attention-seeking.

Not smart, not clever. Just boring vandalism.

[sergiotapia]

Right? Let me just write 22 million comments of random garbage on every HN thread, YC will surely hire me for that!

[eyelidlessness]

A more apt analogy, if we didn’t already know the general range where a given single comment thread degrades that thread (analogue to a local repo) and HN overall (analogue to the GH service), writing a bot to answer that specific question. It’s kind of wild that this yielded any new/not-widely-known information at all because it’s such an obvious thing to test. But apparently it raised at least some eyebrows on both fronts.

[sverhagen]

In load testing there's a difference between testing if something executes as specified versus testing where it breaks. I'm pretty sure that GitHub has tests to validate their performance specification. They may have tests even far in excess of that. They may not have tested where it breaks. They may have had a discussion like: what if someone tries such and such, and their answer may have been: we have good monitoring, we'll catch it before it gets out of hand, and lock out the user in question (which ultimately is what they did). In other words, spending engineering resources to determine the breaking point may not have been a priority. I'm not saying that I would agree with that in all circumstances, but it's their determination to make.

[eyelidlessness]

Sure, that all makes sense. It’s still true that someone stressing git and GH’s services in this particular way produced information that wasn’t especially redundant. Monitoring was good at catching it, but probably based more on service quality than on the actual thing under stress. Now there’s some data about the thing under stress, and if nothing else that allows some knob turns to calibrate monitoring. And if nothing else, that would more readily catch someone doing the same with nefarious purposes.

[intelVISA]

Alarmingly close to the secret criteria, Paul - he's the one.

[classified]

Everyone can make a website and use a database. Did everyone invent AirBnB?

Everyone can drive a car. Did everyone invent Uber?

You're underrating the non-technical factors.

[femto113]

Deliberately trying to create an extreme situation in order to find when/where/how a service breaks is inarguably "abuse" regardless of whether the intent was malign.

[eyelidlessness]

I addressed this in another downthread reply. Briefly, I agree. My quibble isn’t with the term “abuse”, only the nature of its intent.

[akerl_]

The intent doesn't seem particularly relevant.

[eyelidlessness]

Sure it does. Why would we even be discussing “GitHub predictably enforced rules in response to actions with no conceivable merit”?

[rat9988]

It is malicious as he knows he will harm the service to be able to draw whatever conclusion. This is not a case where the end justifies the means.

[eyelidlessness]

The first time I wrote and shared any kind of interactive code, it took approximately five minutes for someone to XSS it. At the time, I was pretty miffed too. After a polite explanation that the “abuse” was curiosity about defensive measures I’d taken, I understood pretty suddenly that there was a whole scope of programming I hadn’t even considered.

More than 20 years later, I still remember the enormous benefit that little bit of malice has bestowed on me and my career. And every time I’ve been on the receiving end of such an exploratory exploit since has been exponentially more appreciated.

[eyelidlessness]

I’ll add one more anecdote while I’m at it.

At a previous job I was aware of a potential vulnerability, voiced it rather loudly, but had a hard time getting the attention it deserved until I recognized it happened to coincide with a really high profile business-critical bug. I only recognized it because some jerks had previously fucked with much less important stuff under my purview, and I wanted very much to understand how they did it, and learned quite a bit by wanting to know.

I used those developed instincts to unfuck what would have otherwise resulted in at least contract terminations, if not lawsuits. And the recognition allowed me to correct almost every compromised datum, which also guarded every contractee from challenges to their license status and ultimately whether they could be subject to wholly different jurisdictional context.

I’m not going to disclose the nature of the vulnerability but the way the bug presented was time deltas based on time zone configuration. Hardly a novel problem, but nearly put a whole industry into peril and or conflict. Definitely was worth the attention.

And when communicating the problem suffered, I did what any self respecting hacker would do: I exploited the damn thing myself and showed how it was done.

[layer8]

I don’t think GitHub has been harmed. GitHub did the right thing by having mechanisms to disable repositories before they can cause real harm. The author merely tested out where that to-be-expected limit would be.

Arguably, it would be better if GitHub documented an explicit number of supported commits, so that one can know beforehand which usage scenarios the service is suitable for.

[rewmie]

> Arguably, it would be better if GitHub documented an explicit number of supported commits, so that one can know beforehand which usage scenarios the service is suitable for.

I don't agree. Clearly GitHub can easily handle this number of commits, and more. There was no real world limit being hit. There is no user impact or degraded performance.

This means that in practice there is absolutely no practical limit in GitHub.

Why document that? Are you planning on working on pushing more than 22 million commits into a project? And if you are, what stops you from sending an email to GitHub to clarify if it supports your extraordinary usecase?

It seems some people around here are desperate to find any flaw in the way GitHub handled this case of vandalismz and at best are grasping at straws.

[rat9988]

And what if github didn't have such mechanism? They should swallow the loss as they should have known better? And more importantly, as they weren't documented, how could the author be certain there is a safeguard and he won't cause any harm?

[eyelidlessness]

You are almost getting the value of this kind of curiosity!

[rat9988]

Now, imagine if you had harmless alternatives. Maybe it will help you expand your thinking.