[pixelbeat__]

It's worth noting that this discusses the centos based provisioning and hardware management platform.

The actual distribution used to _run_ all Meta backend services is completely separate and built from source (it does share the (non centos) kernel). This is done for flexibility, performance, service isolation reasons

[ot]

This is not accurate, the system packages are from the CentOS binary distribution. You might be thinking of C/C++ runtime libraries, which are distributed as separate packages used by the internal ("fbcode") binaries, but system binaries link to the standard distribution.

[pixelbeat__]

(Hi Giuseppe!)

Right, the standard centos system binaries are used to provision services and manage hardware.

The internal services linking against the runtime libs you mention, are actually linking against about 2000 built from source packages, and are essentially a separate distro (with a distro in this sense being an ABI compatible set of libs running on a kernel)

[ot]

I think that's not what people would normally call "the distribution" :) systemd, coreutils, ssh, shell, ... come from CentOS.

(Hi Pádraig!)

[loeg]

> The actual distribution used to _run_ all Meta backend services is completely separate and built from source (it does share the kernel).

That does not seem to be true. The hosts I'm looking at have systemd and glibc RPMs that were built on centos.org hosts, with coreutils from redhat.com. The kernel was built on a Facebook host, but that's it (of the handful of components I've spot-checked).

[pixelbeat__]

Use ldd on any service binary to see that it's linked against a separate distro. (It's best to use the ldd from the corresponding platform).

There are actually three distros on each host. Current runtime platform, previous runtime platform, centos platform.

[trws]

For cases where you might not know which ldd will actually work, I like to use patchelf or readelf to get the interpreter then use the —list arg directly. That way it always actually gives correct results, using a different loader can change the selected library paths even if it thinks it works. One-liner would be approximately $(patchelf —print-interpreter tgt) —list tgt My apologies if the dashes became an em-dash or similar, on a phone.

[codetrotter]

Maybe they just don’t trust you enough and all of the machines you think are physical hosts that you ssh into are in fact nothing but containers, and there is a whole other team of people at said company controlling the real servers :thinking_face: