If @homakov is finding security holes without access to Github repositories, imagine what he'd find if you had him code audit for a few days... He's clearly been going about this the proper white-hat way and ensuring holes are patched before open disclosure... what's there to lose?
On the flip side, you could go about doing what you're doing under the presumption nobody is maliciously targeting your user base. In this scenario, it's possible you have a couple bad actors that see a net benefit greater than your bug bounties and are silently stealing and selling supposedly secure code from your users. You could be supporting a hacker black market where they sell and trade codebases to popular online sites. Imagine how easy it would be for them to find vulnerabilities in these sites if given access to the source code.
I don't think that is a fair assessment of him, even then.
At any case, I hired him fairly recently for a security audit and he worked quickly, and was very effective (he found several important vulnerabilities and reported them in a crystal clear manner). He was also a pleasure to deal with (no bullshit stance, something I find enjoyable).
The 4000 USD for ~20 hours of work were definitely well spent!
The parent was asking why Github haven't hired him, not why nobody has hired him. If you remember, Github actually banned him for hacking the Rails account in his pentesting.
Yeah, the first Github and Rails exploit is the one that still sticks out in my mind. That kind of thing can be hard to shake, but it helps that you were quite young at the time. I'm happy to see you've matured a lot since then.
I think his behaviour was commendable - he tried many many times to warn, going to multiple people and projects, but they all ignored him - they were too busy being Gem installing Ruby hipster Brogrammers to consider security, and it bit them hard in the backside.
I'm incredibly interested in angling my career towards security and have no real experience.
Wouldn't it also be wise to keep people like him 'out of the loop', I imagine it's much harder to audit when they have access to internal code/architecture that would be difficult for an outsider to stumble-upon?
he gets paid $400/hr doing consulting for YC Companies and other startups and companies, he is from Russia, and now lives in Bangkok, when he becomes rich he wants to live in Hong Kong, pretty nice for a 20 year old, I don't see any glaring reasons to work for Github http://egorhomakov.com/
400 USD/hour is a great rate anywhere in the world, even the most expensive cities.
But abcd_f's comment is right about one-off 4-hour projects vs. long-term contracting. Non-billable time overhead spent on finding clients, negotiating contracts, mentally switching projects, or just sitting idle can negate the benefits of a high hourly rate.
Completely agree, GitHub private repos are a huge target. Even if you use 2FA, after login it's just a cookie that separates the good from the bad. How could GH improve that? Client-side SSL Certs?
I'm pretty sure many more codebases have been lost through failures to secure internal networks by corporate IT departments than through vulnerabilities in cloud hosting providers.
Important storage can be done 'in the cloud', but you need to audit and verify the cloud vendor is providing the proper controls. Just like you need to do 'privately'.
For code projects that are between me and a couple of other devs, none of whom are infrastructure security experts, I trust a company like Github a lot more than one of us trying to hack something together on a server.
With the exception that if you have three guys hacking something together a dedicated server or a box off your cable modem, with git tunneled over ssh using keys and a proper firewall, you'd probably be miles ahead. That might take you an afternoon to set up with almost no experience.
Not to say that it couldn't be compromised, but your not a target like github might be. If you're working with an enterprise level project with more complex auth and access methods, more users, performance and scaling needs, you'd need a real security implementation.
But hiring him offers no guarantee that he will be able to find any other bugs.
That's the beauty of bounties, it allows people to decide whether they want to do the right thing or not, if there was no bug bounty more people are just tempted to exploit the bug.
Github uses ruby on rails, which is a pretty mature framework, perhaps covering most of the common security pitfalls. Additionally, I assume github has excellent programmers because of the nature of their job.
Could someone explain in simple english, how did they overlook known & well documented bugs that got them hacked (e.g. Bug 3 about cross domain injection). I'm wondering if someone of Github's caliber can be hacked so easily, what about the rest of the masses developing web apps. Especially all those new crypto-currency exchanges popping up left & right.
I've been toying with Django. Reading through the docs makes me feel that as long as I follow the safety guidelines, my app should be safe. It feels as if they've got you covered. But this post rattles my confidence.
It's worth mentioning that Github has forked Rails and is working off their own private branch of Rails 2.3. Not saying that was relevant to this exploit, mind you.
I'm wondering if someone of Github's caliber can be hacked so easily, what about the rest of the masses developing web apps.
They're all pretty bad. SQL injection was a boondoggle for years until people wised up, or more likely moved to the then-newly-popular ORMs, but it still got Bell Canada recently. Target is #36 on the Fortune 500. That wasn't a webapp based attack, but even companies of their considerable resources still get security that wrong. Sure, you can tell yourself a startup is more tech focused and better positioned to get security right. But do devops building for server stacks and platforms they don't fully understand while pushing code multiple times a day really have both the skills and time to focus on security?
It's a dream job for developers, in some ways a lot more so than the big boys like Google and Facebook. They have a hiring pipeline any tech company would kill for. They probably don't have the deep security talent that say Google or Microsoft have, but they should have enough.
Really? I'm not so sure, AFAICT Github doesn't have any new or interesting problems to deal with. It's just a Rails app that's constantly developed on. You can do that, well, anywhere.
Having met several of them and spent a day in their offices (they gave Kiva engineering a tour day a couple years back), I can say they have an awesome company culture and space and great leadership and great brand recognition. Plus, people generally like them. I'd put it near the top of my list if I had one.
As briefly as possible? Infosec is hard. Most companies have virtually no security policies. Nobody listens. Black hats are ahead in the arms race and anyone who has decent knowledge (doesn't even have to be anywhere near on a level like Homakov or Zalewski) can pull off all sorts of exploits. Even if they don't strike the application itself, they'll get you through infrastructure that your application relies on. Look at how script kiddies like the SEA can pull off high-profile hacks through social engineering, domain and DNS hijacking.
It's assured that a ton of Rails apps are vulnerable, it's just that no one has found them, or more likely, is not publicly releasing or actively exploiting them.
Also, Rails doesn't address for all security pitfalls. Some of its mechanisms are actually underdeveloped and require rolling lots of checks by yourself, such as for proper session termination, IIRC.
This highlights to me that our infrastructure is horrendously overcomplicated. We have all these great abstractions, but you have to worry about bugs and exploits in every possible layer of every system. Even the simplest modern web-application has an enormous surface-area to secure, and that makes getting it "right every single time" damned near impossible.
This is a little myopic but understandable in the context of a discussion on HN. Infosec is hard,
but it is just one example of a bigger truth:
Defense is hard.
This comes up time and time again in any defensive discipline:
Over two decades the CIA had learned again and again that it could not hope to
defend against terrorists by relying solely on its ability to detect specific
attacks in advance. No matter how many warnings they picked up, no matter how
many terrorist cells they disrupted, at least some attackers were going to
get through. Officers in the CTC privately compared themselves to soccer
goalies: They wanted to be the best in their league, they wanted to record as
many shutouts as possible, but they knew they were going to give up scores to
their opponents. Ultimately, many of them believed, the only way to defeat
terrorists was to get out of the net and try to take the enemy off the field.[1]
The final sentence above highlights the one pecularity of InfoSec; you do not have any
offensive capabilities.
This is why I think some more work into client (or active) honeypots may be beneficial. If we can get an easy to install, auto updating honeypot that fights back, we may have a better offensive capability.
This may just end, like nuclear warfare, in MAD... But it would be great fun to watch!
No one gets it right every single time. No one. That's a completely unrealistic expectation. What you do is establish a bar, which you share with everyone who will use your software. Then you evaluate your efforts against that bar.
One of the keys to developing good software is hiring third-parties to conduct audits. A bug bounty program is one way to incentivize people who are already probing your software to take the next step and tell you about the bugs they find.
I'm pretty sure Egor's first language isn't English, so OK might mean 'meh it's alright' through to 'hey this is great'. I know a few non-native speakers who do similar things.
8 hours at 400$/hour will still only be 3200$ and he can presumably spend the remaining 4-3 hours doing more security analysis with less overhead, so it might still be cheaper to hire him as a consultant.
But they'd have to pay those $3200 without knowing if there were results. They might have to pay dozens of such consultants before one of them found bugs like this. Bug bounties, paid only on successful discoveries, are much cheaper.
But also much riskier. What if it transpires that the $4000 isn't enough? We know roughly what they're paying now, so when people find an issue like this they know they could sell it for much more.
@homakov finds 5 different bugs with github and manages to align them so that a bigger vulnerability is exposed in under 5 hours? That's amazing! I used to think I'm a fast delivery-focused developer but I'm probably just a fraction of how fast some people are.
How can I start learning about how to identify exploits like this? I know some basics about web application security and work as a software engineer on a day-to-day basis but security has always been a passion of mine and I have always wanted to be able to support myself through working on security alone (by collecting rewards through bounty programs, self-employed security consulting, working at a security consulting firm like Matasano, or some combination thereof) but I don't know where to start. I want to learn the ins and outs of web application security instead of just understanding the OWASP top 10 and having a strong interest in certain topics (like HTTPS/SSL vulnerabilities). When I read disclosures from people like Egor I grasp the steps they are taking to craft an exploit like this as they are explained but I don't know how to identify these exploits on my own.
Can anyone recommend some reading material or some first steps I can take to work towards moving to a more security-focus career?
Like a lot of other things, practice matters. OWASP has some deliberately insecure webapps which are meant to give people practice spotting and exploiting vulnerabilities (WebGoat, RailsGoat, PyGoat, probably others). There are also "capture the flag" competitions of the sort run every so often by Stripe; Matasano currently has one going as well, focused on embedded systems:
I'm the only that thinks that $4000 was very cheap on part of Github? a security hole like this on the wrong hands would have bring severe consequences to github, consequences so big that they would probably pay $1,000,000 USD for it to never happen. So maybe something in the $50-100K would sound more reasonable. Egor is a great hacker with no business sense? On the other hand, the publicity his service gets for this its probably worth more than $50-100K.
No you're not alone, considering this was a combination of security holes that allowed people to get read/write access to others repos, including private.
I'm really glad Github paid him, but reading what the exploit can do I really think he deserves more, sure they were a series of small exploits, but all together... they are pretty damaging in the wrong hands.
Send him an e-mail saying, "Hey, I sent you $100. I would deeply appreciate it if you spent it on your beverage of choice, or a nice dinner with a friend, rather than on necessities."
Charging $400/hour does not mean he does not need extra money. His nature of business is a short term projects, it's not like a regular web developer who has to work 40 hours a week for many month to finish a project, he only does audits which don't last long because of that you see this "high" (I personally don't think it's high) hourly rate.
It's actually a good strategy to price high hourly but over-deliver (doing lots of free work behind the scenes, or speculative unpaid work, etc.) -- rather than the market-clearing rate of ~100-150/hr, at least when you're trying to build a brand. At $400, he's clearly a specialist, and will get more interesting work; at $100/hr, you could hire him and just treat him like another developer, have him do cookie-cutter assessments, etc.
Personally, I think he'd make more money at $400-600/hr if he could also get some kind of manager to handle the interactions with clients; it doesn't seem to be what he enjoys, or is particularly good at.
(I've had drinks with him before, so probably the most effective way to accomplish my goal is to buy him drinks when I'm in town.)
At least in my experience, I donate to groups that do good work but aren't getting paid for it. I wouldn't donate to people who are being paid (quite handsomely, in this case) for their labor. Especially when he's already clarified that GitHub paid him more than he thought his time was worth.
Perhaps you could clarify that part in your future posts, to appease the Internet haters on both sides. "I do paid contract work. However I also spend lots of time fixing open source stuff for free. If you want to encourage me to keep doing the latter, here's how to donate."
Donate or don't donate, that's your call. But why are you complaining about him asking for a donation? Why try to "shame" him? What is he doing to harm you?
Not sure why you're viewing my comment with such hostility. I was mistakenly under the impression that most of his work is contracted / bounty. He's already clarified his reason for accepting donations below, and I understand. I just think the placement/wording was less than ideal.
I doubt Egor is being paid for posting these summaries to his own blog for all of us to see. Even if he weren't contributing code to various libraries and applications, these write-ups are a great benefit to everyone else who has yet to be a target.
Although you probably should factor in the possibility of several years of compulsory $0.30/hr labour, plus forfeiture of all your ill-gotten gains (and probably some healthily-gotten ones too, they're not so fussy)
And that's before legal costs and possible restitution.
Sure, I had a similar first reaction, but thought about it. If you have skills but haven't yet developed a deep-enough client base, you're in a quandary. You can't bill for $10/hour, or no one will take you seriously. You need perceived value, so you have to quote some reasonably high rate, even if you case-by-case discount it or work gratis.
(At least that's how I imagine it must work. I've never consulted.)
I totally believe that he's worth that amount of money. I'm sorry if you thought I was questioning that. I'm questioning the juxtaposition of his hourly rate with a request for donations.
I think the contract makes sense for clients, and the donation makes sense for other security researchers who want an incentive for him to keep publishing ideas.
Understood. But I imagine that his work isn't quite as "steady" as one might expect. He invests time by trying to find security exploits in hopes that the affected company compensates him. He doesn't set his price or even determine if he gets paid for his time.
I think that might be the rationale...or it might just be that he's found himself in a position where he can collect bounties AND donations :).
As soon as I saw the new bounty program the first thought through my head was "Any Github Hacking leaderboard without homakov at tthe top is an inaccurate one". Congrats on your newest discovery!
Impressive display of persistence, stringing together those vulnerabilities. I also see your English has gotten noticeably better :) Keep up the good work!
> Oh my, another OAuth anti-pattern! Clients should never reveal actual access_token to the user agent.
From what I understood by reading the OAuth RFC is that front-end intensive applications (a.k.a. public client) should have short lifespan access tokens (~ 2 hours) and the back-end takes care of reissuing a new access token when expired.
Can someone clarify on how to make a those calls from a front-end application without revealing the access token?
$400 is such chump change compared to the PR disaster that can come from exploited, or even just leaked, vulnerabilities. I honestly think any SaaS needs to have this somewhere in their budget once a year.
One more comment. Security flaws seem obvious, but getting security right is hard. It require a lot of testing and effort to get everything right. This kid Homakov has a talent for finding holes and seems that has his hard on right place ie. isn't abusing it.
Really good work @homakov and I suggest you should start a web-security-school or something of the sort. I'm sure there is money in that field and you would be able to keep traveling around the world while doing it.
Why is GitHub so hostile to this kid, just give him a job already! He obviously has deep understanding of how things work. I would feel better knowing he work for them.
I am consultant as well, I can be wooed with right offer and if I am interested in something. He obviously is interested in GitHub. I think they are still pissed off from last time when he found flaws.
I'd put this in the same category as mobile app dev. There are a few people making money by the truckload, plenty of people making a decent living, and lots of folks who strike out.
If it's something you're interested in, go for it. I just worry that people see this like the promise of gold in a faraway land and go rushing in, not thinking about the real distribution of success.
Remember that you only see the interesting stories and successful investigations. Before making such a decision you should try to arrange a chat with someone already doing comp-sec, and figure out how much time they spend on all the other stuff.
It would be great for educational purposes if a sample app was setup so this vulnerability could be tried on it. Most of the white hack vulnerabilities are fixed by the time white hat blog posts come out so there is no way to actually try them out.
Thanks for continuing to make Github safer for all, @homakov. Someday I might even host a private repo there again, but I haven't done that since your first mass assignment exploit. You continue to prove that my decision was a good one.
If we're shaming any code with security flaws, no one is free of shame. I'm excited by the bounty program, it's a great way to get things like this identified and responsibly disclosed
On the flip side, you could go about doing what you're doing under the presumption nobody is maliciously targeting your user base. In this scenario, it's possible you have a couple bad actors that see a net benefit greater than your bug bounties and are silently stealing and selling supposedly secure code from your users. You could be supporting a hacker black market where they sell and trade codebases to popular online sites. Imagine how easy it would be for them to find vulnerabilities in these sites if given access to the source code.
That, my friends, would be a catastrophe.