Those things cost resources, and now you're introducing a new attack vector: open up a bunch of shit PRs, burn a lot of cash for the target organization.
You're right. It doesn't solve for all scenarios and doesn't block malicious actors.
I do believe, however, that it would have a meaningful impact on the "drive-by" PRs that keep being used as examples; the thoughtless, throw-spaghetti-at-the-wall PRs that do not have malignant intent behind them.
Many large OSS projects would have the resources to eat that cost with Donors, Sponsors, and OSS hand-outs. That's why I clarified in my original post because I know this is not a general solution.
The problem is you can get the LLM to iterate until it compiles and lints and even passes LLM review, but will that actually improve the quality of the contribution or just produce more line noise to mechanistically meet criteria?
To large complex projects often the kernel of an idea is the core value of a contribution, and it can take a lot of iteration to figure out how to structure it. Token bashing until CI is green does nothing to ensure the best approach is selected.
> The problem is you can get the LLM to iterate until it compiles and lints and even passes LLM review
Worst of both worlds with this, if you're doing it in a github workflow. You wind up effectively paying for the testing/validation layer of someone else's irresponsible LLM use.
For sure, but that's not what I was referring to in my posts. I'm specifically referring to the callout that the contributions are so low quality they don't even pass linting or compile.
I could have been more explicit on that nuance, I suppose.
That's why you sandbox. You can mitigate most low-hanging DoS fruits by running your server side hooks in a per-tenant cgroup that limits CPU and memory usage. One tenant per public key for trusted contributors, and one general-purpose tenant shared by all new/unknown contributors.
Can't you prevent pushing from the client side with pre-commit hooks? I would expect a hook to fire on the developer's computer that prevents them from even committing/pushing (unless they nuke the hook in their local repo copy).
You have to manually install hooks in your local repository. They aren't propagated as part of the repo. Git has intentionally made hooks require a very explicit opt-in.
I do believe, however, that it would have a meaningful impact on the "drive-by" PRs that keep being used as examples; the thoughtless, throw-spaghetti-at-the-wall PRs that do not have malignant intent behind them.
Many large OSS projects would have the resources to eat that cost with Donors, Sponsors, and OSS hand-outs. That's why I clarified in my original post because I know this is not a general solution.