[CamperBob2]

I don't get how this can work, and Moxie (or rather his LLM) never bothers to explain. How can an LLM possibly exchange encrypted text with the user without decrypting it?

The correct solution isn't yet another cloud service, but rather local models.

[FrasiertheLion]

The model is running in a secure enclave that spans the GPU using NVIDIA Confidential Computing: https://www.nvidia.com/en-us/data-center/solutions/confident.... The connection is encrypted with a key that is only accessible inside the enclave.

Within the enclave itself, DRAM and PCIe connections between the CPU and GPU are encrypted, but the CPU registers and the GPU onboard memory are plaintext. So the computation is happening on plaintext data, it’s just extremely difficult to access it from even the machine running the enclave.

[olejorgenb]

How is it then much different than trusting the policies of Anthropic etc? To be fair you need some enterprise deal to get the truly zero retention policy.

[FrasiertheLion]

Enclaves have a property that allows the hardware to compute a measurement (a cryptographic hash) of everything running inside it, such as the firmware, system software such as the operating system and drivers, the application code, the security configuration. This is signed by the hardware manufacturer (Intel/AMD + NVIDIA).

Then, verification involves a three part approach. Disclaimer: I'm the cofounder of Tinfoil: https://tinfoil.sh/, we also run inference inside secure enclaves. So I'll explain this as we do it.

First, you open source the code that's running in the enclave, and pin a commitment to it to a transparency log (in our case, Sigstore).

Then, when a client connects to the server (that's running in the enclave), the enclave computes the measurement of its current state and returns that to the client. This process is called remote attestation.

The client then fetches the pinned measurements from Sigstore and compares it against the fetched measurements from the enclave. This guarantees that the code running in the enclave is the same as the code that was committed to publicly.

So if someone claimed they were only analyzing aggregated metrics, they could not suddenly start analyzing individual request metrics because the code would change -> hash changes -> verification fails.

[olejorgenb]

Thanks for explaining :)

> First, you open source the code that's running in the enclave, and pin a commitment to it to a transparency log (in our case, Sigstore).

This means you have reproducible builds as well? (source+build-artifacts is signed)

Also - even if there are still some risk that the link is not 100% safe, maybe it's safe to assume vendors like yourself going through all that trouble are honorable? (alternatively - they are very curious of what "paranoid" people would send through LLMs :sweatsmile:)

[FrasiertheLion]

We don't have reproducible builds because we attest the full OS image that we run, which is the Ubuntu image. Unfortunately bit-by-bit reproducible binaries for OS images is kind of an unsolved problem, because it requires the hundreds of package maintainers across all dependencies to eliminate any sources of non-determinism in the compilation. Things like timestamps and file reordering are very common and even one of these changes the entire hash.

So we do the next best thing. We decide to trust Github and rely on Github Actions to faithfully execute the build pipeline. We also make sure to pin all images and dependencies.

[boramalper]

They explain it in Private inference [0] if you want to read about it.

[0] https://confer.to/blog/2026/01/private-inference/