[zeeveener]

I'm personally amazed that _Large_ OSS projects don't have the appropriate automation in place to prevent non-compiling or non-linter-passing submissions.

- Hooks (although there's no clean way to enforce they be "installed" on a clone), GHA Workflows (or their equivalents on other forges).

This might be my bias showing, but these are items I would consider table-stakes for a project of a certain size / level of popularity.

It feels like a lot of the "AI is shit at contributing" problems could be addressed in part by better automated checks and balances.

[jmcqk6]

Those things cost resources, and now you're introducing a new attack vector: open up a bunch of shit PRs, burn a lot of cash for the target organization.

[zeeveener]

You're right. It doesn't solve for all scenarios and doesn't block malicious actors.

I do believe, however, that it would have a meaningful impact on the "drive-by" PRs that keep being used as examples; the thoughtless, throw-spaghetti-at-the-wall PRs that do not have malignant intent behind them.

Many large OSS projects would have the resources to eat that cost with Donors, Sponsors, and OSS hand-outs. That's why I clarified in my original post because I know this is not a general solution.

[schmichael]

The problem is you can get the LLM to iterate until it compiles and lints and even passes LLM review, but will that actually improve the quality of the contribution or just produce more line noise to mechanistically meet criteria?

To large complex projects often the kernel of an idea is the core value of a contribution, and it can take a lot of iteration to figure out how to structure it. Token bashing until CI is green does nothing to ensure the best approach is selected.

[solid_fuel]

> The problem is you can get the LLM to iterate until it compiles and lints and even passes LLM review

Worst of both worlds with this, if you're doing it in a github workflow. You wind up effectively paying for the testing/validation layer of someone else's irresponsible LLM use.

[zeeveener]

For sure, but that's not what I was referring to in my posts. I'm specifically referring to the callout that the contributions are so low quality they don't even pass linting or compile.

I could have been more explicit on that nuance, I suppose.

[10000truths]

That's why you sandbox. You can mitigate most low-hanging DoS fruits by running your server side hooks in a per-tenant cgroup that limits CPU and memory usage. One tenant per public key for trusted contributors, and one general-purpose tenant shared by all new/unknown contributors.

[all2]

Can't you prevent pushing from the client side with pre-commit hooks? I would expect a hook to fire on the developer's computer that prevents them from even committing/pushing (unless they nuke the hook in their local repo copy).

[0xffff2]

You have to manually install hooks in your local repository. They aren't propagated as part of the repo. Git has intentionally made hooks require a very explicit opt-in.

[all2]

Oh, good to know. I haven't used them much, so I'm a bit ignorant as to how they work in larger projects.

[pxc]

> Hooks (although there's no clean way to enforce they be "installed" on a clone), GHA Workflows (or their equivalents on other forges).

Git supports pre-receive hooks. But big multitenant forges like GitHub.com don't allow you to configure them because they're difficult to secure well. (Some of their commercial features are likely based on them, though.)

If you self-host a forge, though, you can configure arbitrary pre-receive hooks for it in order to do things like prevent pushes from succeeding if they contain verifiably working secrets, for example. You could extend that to do whatever you want (at your own risk).

[jmcqk6]

You're still talking about compute resources that need to be paid for and maintained for that. Spamming AI PR's is going to cost a lot of money.

[pxc]

At the end of the day, LLM slop PR spammers are essentially adversarial actors. Git hooks are ultimately a tool for good faith developers within a given community (your team, your company, your regular contributors) in maintaining good hygiene and avoiding lapses into preventable mistakes. That's true for all CI, too.

And the truth is, too, that it's super easy for an LLM agent to run a build and tests. Good faith contributors using LLMs will never open PRs that don't build not because they're willing to "go the extra mile" and do manual work, but because they give the slightest fuck and have any respect or consideration for the humans they're working with.

LLM spam presents a different problem than any of that stuff was meant to solve. It's a malicious act, and you're right that tooling that burns the defender's compute can't be a solution. :-\

[abustamam]

All of my personal projects, many of which will never be publicized, use hooks and GHA to ensure compilation of changes.

It is quite strange that a large project like Zig would not have such a thing. I'm sure it's not trivial but it seems important to invest time into.

[lexh]

But... this particular project does have such automation in place? It isn’t hard to find:

https://codeberg.org/ziglang/zig/src/branch/master/.forgejo/...

[papyrus9244]

One of my pet peeves with git (and systems both similar, and based on it) is that automated tests run after you've made the commit and push.

In my mind the commit (let alone the push to a publicly accesible server) should be done after, and only if, the automated tests are successfully executed. And there's no easy way to implement this, other than having a dirty branch that you discard after rebasing onto a more long lived one.

[sethammons]

There are lots of reasons to commit when things are yet working. How else would you share code that you need help with?

The solution is gated merges. No merging to main unless ci passes.

Every org I have worked at bemoaned a flaky release process and refused e2e black box acceptance tests because "they are too slow." And every org I have worked at has realized they were wrong. We got appropriate gates that run in 5 minutes and an ops person is the only person who can force past any gate in case of emergency.

Guardrails like this only become more important with the accelerant that is ai.

[10000truths]

You can use a pre-receive hook on a git server to reject pushes that fail compilation. Downside is that it requires admin access on git forges, so you're only able to do this if you self-host.

[jwolfe]

Pre commit hooks exist. People just don't like being prevented from committing for reasons such as this.

[sauercrowd]

I mean even having linters and everything still creates a whole bunch of noise in their PR section, not to mention that a lot of the changes I make to stuff that's written by codex is not stuff that's caught by linters.

It's just bad/wrong/context lacking decisions and mental models it introduces, that if not carefull will just create a massive mess of a codebase. (I know, because I've tried, and had to deal with it)

And if someone vibecodes a PR and it works, why dont they just share the prompt so a repo owner could vibecode it themselves?

[abustamam]

Vibe coding is often not a single prompt, it's an entire workflow (if you're doing it right).

[sauercrowd]

Don't disagree, but the "if you're doing it right" is a big asterisk for an open source project with people you have no idea what quality bar they're at.

And in my experience it's quite hard to figure that out by quickly looking at it.

Not to mention that contributions on github (almost?) never include the prompt chain anyway, so the status quo is even worse

[abustamam]

That's a fair point. I was just speaking generally.