[natfriedman]

This is a cat and mouse game. We add code to detect and disable abuse – sometimes in very clever ways – and then the abusers come up with a new way of circumventing that detection. In order to prevent miners from creating long queues for legitimate free users of GitHub Actions, we have to stay on top of this all the time. So the miners are not just stealing CPU time, they are also stealing engineer time. Because without mitigations the miners will consume all available CPU, and because devising abuse countermeasures is, for whatever reason, a very powerful nerd snipe (including for me!). The sad thing is that it's displacing time that would be spent improving Actions in other ways.

[lmeyerov]

Actions are awesome... but scary as soon as you have a public repos, contractor, rogue employee, etc. They seem to go against security fundamentals. Ex: Actions should allow going into default-deny mode for all basic runtime capabilities and resource use, and only brought back on via RBAC. Today, it's not hard to steal npm/pip/etc creds or get into people's corp runners. Having gone through the browser security policy heyday, this is deja vu, except now for exposing the server side and supply chain.

Ex:

- do not run on any event.. unless user authorized for that event. Same for actions.

- separate out policies and users cannot edit policies unless authorized to do that

- do not get physical/logical resources (runners, disk quota, long runs, ...) unless given

- default-deny network outbound with url safe-listing

That way only trusted users can run them, and a bit harder for them to get hurt when there is a surprising action that they run

The next level would probably be something like sandboxing : allow anyone to run an action , but a sandbox mode can autofail if violated, and have explicit imports/exports to lock down for how it gets used.

A lot possible.. but need to invest in the basics first..

[slrz]

Getting a hold of someone's secrets is not possible just by doing a pull request. It's really only about resource usage, at least when the runners in question provide sufficient isolation (true at least for the Github-hosted ones, or we're all in big trouble).

Unfortunately, using self-hosted runners to provide additional capabilities not supported by Github-hosted ones is basically impossible (for public repos at least) as you can't restrict a runner to an organization or project. Set up a bare-metal runner and it will receive jobs from random forks.

[chatmasta]

> Getting a hold of someone's secrets is not possible just by doing a pull request

Only if you've configured the actions correctly. I would bet that there is a high number of repositories on both GitLab and GitHub with misconfigured CI pipelines where someone can submit a PR with `env | curl` to grab any secrets defined as environment variables.

[sofixa]

GitLab CI, and i suppose Github, allow marking env variables as secrets, making them more complicated to show / exfiltrate.

[chatmasta]

No, GitLab does not allow marking variables as secrets. They allow "masking" env variables, subject to a bunch of caveats, like your secrets not being multiple lines (e.g. a private key cannot be masked). Even then, the masking is just about log output -- it doesn't prevent a `env | curl` type situation. [0]

The correct mitigation is to ensure that any "secret" variables are marked as "protected" so they can only run on protected branches that are limited to pushes by maintainers. And you'll still need to make sure the masking works in the logs.

They do support integrating with Vault to access secrets in a CI job, but you need to pay them to use that feature. [1]

[0] https://docs.gitlab.com/ee/ci/variables/#mask-a-cicd-variabl...

[1] https://docs.gitlab.com/ee/ci/secrets/

[sofixa]

Re: Vault, you can read Vault secrets yourself via their JWT integration.

[brodock]

For GitLab if you don't have at least developer access to the repository (as in you are sending a MR from a fork) that will run in the context of your user, so you don't have access to any secrets configured upstream, etc.

If you have access to a repository you can customize the script to do whatever you want, but there will always be a trace tracking it back to you.

There is a discussion about ultimate security (access only when asked) Vs the convenience of self-service.

You can still avoid that by having people use a fork model, or triggering CD from an external project with tight access.

Putting a burocratic process between ICs will only limit their throughput as in Jenkins paradigm.

The better advice is don't hire people you can't trust

[lmeyerov]

Add one extra command ;-) These can be innocuous if buried in something like unit tests of configs or network behavior, or in a big pr:

logs: `env | base64`

network: `env | gzip | curl`

It should be easy to set most workflows to run sandboxed with almost no capabilities - no secrets access, safelisted network access, safelisted package manager accesses for top 10 langs, etc - so that testing someone's PR isn't scary, and runtime violations make loud noises. The whole 'just disable actions on fork PRs' thing is a great default, but ultimately a figleaf as it's not hard to get someone to run an action.

[sofixa]

env doesn't show secret/masked variables ;) you have to pass through echo and a file

[sorbits]

Do I understand correctly that the attacker forks a repository with GitHub Actions enabled, modifies the action, submits a PR, which makes GitHub run the altered action?

If so, I wonder if there is a legit need for running modified GitHub actions from non-collaborators?

Could also subject modified actions coming in via pull requests (from non-collaborators) to heavy resources constraints and timeouts.

[natfriedman]

The mitigations you suggest are all logical. However, there are legitimate reasons to run CI and tests for outside contributions without taxing maintainers with the cognitive load of having to evaluate whether each contribution is CI-worthy.

The attack vector in the article is not the main way miners try to steal CPU from the GitHub community. It's just an interesting one that the journalist chose to write about.

[teachtwolearn]

But when a PR is submitted that modifies an Actions workflow, shouldn't GitHub run the old unmodified workflow until that PR is accepted?

IIRC, they already treat the .github folder as a special case; you can't push modifications to workflow files with a personal access token. So why not ensure that an action or workflow will only run if it is checked into the base branch?

That wouldn't stop PRs from modifying scripts that the action runs, but the current behavior seems a bit counter-intuitive.

[lmeyerov]

If that action is "./run_tests.sh", which is a top use case, the attacker just changes "./run_tests.sh", so while I agree that's useful, it doesn't secure the typical case, and makes for a hard cost/value stance.

The threat models are probably more like 1. "make sure only the right people run actions" and separately, 2. "make sure authorized events/actions only use the expected capabilities." Both largely fail today.

[cortesoft]

Well the idea is that a person submits a PR, and the action runs to verify that the tests pass BEFORE the PR is accepted. You don’t want to wait until after the code is merged in order to see if tests still pass.

The issue is that even if you don’t allow changes to the actual action workflow, running tests gives an attacker the ability to run arbitrary code. They just need to add the code they want to run to the tests (e.g. have the tests mine crypto)

[wiredfool]

That makes it really hard to debug actions, and ensure that they’re working prior to merging.

[neolog]

> steal CPU from the GitHub community

from Microsoft*

[anoojb]

Thanks Nat. What if we made new GitHub Actions temporarily only available to users with a verified second factor?

Could temporarily reduce the population of abusers while we figure out a more sustainable strategy?

[chatmasta]

A TOTP code response is trivial to implement on the client. So if you wanted this to be meaningful, you would need to force users to use SMS 2FA, which is widely considered insecure. Not a great solution IMO.

[kerng]

I wonder if it would help forming an internal red team that could help stay a step ahead with such and related attacks and abuse scenarios by running such attacks against yourself?

[bko]

Can't you say the same thing about any defensive measures? The need for security is displacing developer time to be building out cool new features.

[natfriedman]

Yes. This is just a category of attack whose growth has been incentivized by rising crypto prices. All providers of free compute are experiencing some level of mining attack right now. Eventually a new equilibrium will emerge.

[knorker]

A new equilibrium that thanks to cryptocurrency speculators is a worse world.

This speculation is making some people rich, yes. But the amount of externalities is staggering.

Thanks to PoW nobody can provide a free compute anymore without getting owned, and of course the environmental impact of bitcoin alone is worse than when Saddam Hussein set oil wells on fire while retreating.

Let the world burn, and products rot to shit, as long as my HODL portfolio goes up. Cryptocurrency supporters really are sociopaths, worse than any hedge fund manager.

[pcthrowaway]

To be clear, these are very specific problems to proof of work. It's almost to the point where I'm in favour of banning or heavily taxing proof of work mining.

There are lots of things driving for crypto besides speculation though. Suggesting cryptocurrency supporters are sociopaths is quite simplistic, and overlooks the majority who are not involved in anything like this article discusses

[knorker]

> There are lots of things driving for crypto besides speculation though.

Like what? I mean aside from crime.

E.g. the "Venezuela argument" has been debunked, from my reading.

In my own life I've seen employees ask to be paid in bitcoin, because of the large fees involved in getting your salary and transferring it into Brazil.

So… those "fees" are mainly taxes, and not paying them is illegal.

To steal manned argument I guess is that some people actually do want to buy stuff online without a third party knowing who paid whom.

First of all BTC is terrible for that. But second, most people don't care if a third party knows they bought pizza. Especially since the pizza place still knows, and they need to keep records anyway, because tax laws.

Fourth, if they don't keep records then they can just say you didn't pay them. They can say "oh, that was not our wallet, you must have mistyped".

Fifth, if the pizza has poop on it you can't even reverse the charge.

Sixth, every pizza place will now be a money laundering scheme.

This is not what we want as individuals, nor as a society.

But yes, some people do want to buy a pizza online and not have a third party know. It's basically LARPing.

Buying a house or a car anonymously? That was made illegal on purpose.

Buying a big mansion anonymously? Well that's clearly at the very least tax fraud.

Anyway, I truly want to hear about any legit use case for cryptocurrencies that is not just LARPing, because as far as I've seen nobody in the 12 years since bitcoin launched has come up with one that actually makes sense.

> Suggesting cryptocurrency supporters are sociopaths is quite simplistic,

Yeah, it is. It's like the old expression "it's very hard to make someone understand something when their livelihood depends on them not understanding it", or something like that.

https://www.youtube.com/watch?v=hn1VxaMEjRU

But the amount of rationalization from cryptocurrency proponents I do think is sociopathic.

> and overlooks the majority who are not involved in anything like this article discusses

When you're in the mob you're still one of the baddies, even if you're not actually the one murdering.

I do blame every cryptocurrency supporter. They are complicit in making the world worse for their own profits. They have a moral obligation to recognize their supporting role in this, and to stop it.

[pcthrowaway]

In 2015 I moved to a different country. I needed to arrange to pay my security deposit for my place before I had moved. Guess what? After talking about it for a bit, it turned out bitcoin was the easiest way to do it. There was nothing illegal about this, and it saved us a giant hassle.

Cryptocurrency users aren't any more complicit in others using it for illicit activities than people who use cash. And you seem to be equating cryptocurrency with proof of work and bitcoin, which, again, are ignoring the majority of the real-world use cases.

[teitoklien]

So you’re blaming cryptocurrencies for ....

Proof Of Work ?.

A lot of em are already trying to shift to other viable alternative proofs.

Your analogy of calling Cryptocurrency supporters as sociopaths.

sounds similar to insulting Edison because he designed the inefficient incandescent bulbs , which consume waaay more energy compared to LEDs built these days.

How would it sound , if someone insults artificial light , just because of that ? .

Cryptocurrencies are perfectly good ideas/products.

Proof of Work’s viability isn’t.

The hate is aimed in the wrong direction.

[knorker]

> sounds similar to insulting Edison because he designed the inefficient incandescent bulbs

If he designed and sold that in 2021, trying to replace LED lightbulbs, and lying through his teeth to pump up the value of his lightbulbs, well then yes.

Can blockchain people please first come up with a use case, that isn't crime, before saying it's worth using more energy that many countries? (for a rounding error away from 0 number of transactions)

All the enumerated use cases are so naive and uninformed that clearly they've just been invented without knowledge about the field. "Move aside, expert in field, I'll just solve this with technology. No I don't need to hear you describe the issues".

[toomanybeersies]

Lots of countries have banned incandescent bulbs due to their abysmal energy efficiency.

Maybe the cryptocurrency industry needs some environmental regulation thrown at it?

[matkoniecz]

> Cryptocurrencies are perfectly good ideas/products.

for scammers and speculators

[teitoklien]

Try doing a transaction comparison between any payment service that offers foreign transactions , and compare it to cryptocurrency transactions and use it once to do a real trade/purchase , or try using smart contracts in crypto , for once in your projects.

You’ll know why it’s a great product/ service Once it’s environmental viability has been figured out .

Hating crypto , because news reporters only show you the scammers and the speculators , is blind hatred.

Crypto has/does much much more , than just act as a form of trading / speculation

Scammers often steal money in dollars too. Speculators often trade currency pairs with dollar in them.

Are you still gonna say people who use the dollar for daily transactions don’t exist ? That dollar is also a good idea only for scammers and speculators ?

[bko]

Bitcoin is priced pretty much on par with low cost energy consumption. It's not any more profitable to mine Bitcoin now in terms of energy per dollar worth of Bitcoin then before

[merb]

limit the action configurations file to be only editable by a configured set for a specific repo and give us a special folder like .github-action-commands or so that is scoped like that aswell..