[seriesf]

One thing that really squicked me out when I left Google is how other companies, even large and sophisticated ones, are using all kinds of garbage that comes from canonical or red hat or percona, and they have NO IDEA what's in there. Say what you want about google's NIH culture, but in regards to code provenance and verifiable builds they are doing the right thing and many others are not.

[jsty]

Whilst it would be nice if everyone had the time and resources to code review and build their entire source dependency tree, is this ever going to be a reality for the long tail of enterprises who struggle with even resourcing / recruiting for their current workload? I think the vast majority are going to continue outsourcing this responsibility onto enterprise distros / vendors for a long time to come.

[brown9-2]

I think things would be easier for the long tail with more investment from all in the tools space - better support for monolithic repos, unified CI/CD systems, etc.

[KaiserPro]

It depends on your threat model.

If you are a large company tech company (1-5k employees) there are far bigger risks than dodgy binary builds from upstream. (like leaked API keys to github...)

However, if you are a hyperscale, high value company (ie a place which has enough data or digital cash to be worth dicking with) then its a worthy problem.

[GuyPostington]

Can you give an example of this garbage?

[mayakacz]

I think 'garbage' is a strong word, but I believe what the original poster is trying to say is that there are a lot of binaries, packages, and libraries that most organizations will consume from upstream, and not verify directly. This requires either trust on a third party (often many third parties - in the case of open source), or more intense validation of those components and any changes to those components.

Binary Authorization for Borg performs verification for pieces that come out of Google's CI/CD pipeline. For third party code, see in the doc, "When importing changes from third party or open source code, we verify that the change is appropriate (for example, the latest version)."

Disclosure: I work at Google and helped write this whitepaper on Binary Authorization for Borg.

[seriesf]

Literally anything that comes from a vendor in a package? Percona server/toolkit? Every binary package in Ubuntu? The Linux kernel as built and distributed by Red Hat?

[pjc50]

Yeah, that's not a definition of garbage that's going to be taken seriously.

[seriesf]

You're right; Google is the only big tech that takes insider risk seriously.