[nostrademons]

What you're speaking of is basically the capability security model [1], where you must explicitly pass into your software agent the capabilities that they are allowed to access, and there is physically no mechanism for them to do anything not on that list.

Unfortunately, no mainstream OS actually implements the capability model, despite some prominent research attempts [2], some half-hearted attempts at commercializing the concept that have largely failed in the marketplace [3], and some attempts to bolt capability-based security on top of other OSes that have also largely failed in the marketplace [4]. So the closest thing to capability-based security that is actually widely available in the computing world is a virtual machine, where you place only the tools that provide the specific capabilities you want to offer in the VM. This is quite imperfect - many of these tools are a lot more general than true capabilities should be - but again, modern software is not built on the principle of least privilege because software that is tends to fail in the marketplace.

[1] https://en.wikipedia.org/wiki/Capability-based_security

[2] https://en.wikipedia.org/wiki/EROS_(microkernel)

[3] https://fuchsia.dev/

[4] https://sandstorm.io/

[codethief]

> modern software is not built on the principle of least privilege because software that is tends to fail in the marketplace.

Fingers crossed that this is going to change now that there is increased demand due to AI workflows.

[nostrademons]

I'm hoping, but not particularly optimistic.

The dynamic that led to the Principle of Least Privilege failing in the market is that new technological innovations tend to succeed only when they enter new virgin territory that isn't already computerized, not when they're an incremental improvement over existing computer systems. And which markets will be successful tends to be very unpredictable. When you have those conditions, where new markets exist but are hard to find, the easiest way to expand into them is to let your software platforms do the greatest variety of things, and then expose that functionality to the widest array of developers possible in hopes that some of them will see a use you didn't think of. In other words, the opposite of the Principle of Least Privilege.

This dynamic hasn't really changed with AI. If anything, it's accelerated. The AI boom kicked off when Sam Altman decided to just release ChatGPT to the general public without knowing exactly what it was for or building a fully-baked idea. There's going to be a lot of security misses in the process, some possibly catastrophic.

IMHO the best shot that any capability-based software system has for success is to build out simplified versions of the most common consumer use-cases, and then wait for society to collapse. Because there's a fairly high likelihood of that, where the security vulnerabilities in existing software just allow a catastrophic compromise of the institutions of modern life, and a wholly new infrastructure becomes needed, and at that point you can point out exactly how we got this point and how to ensure it never happens again. On a small scale, there's historical precedence for this: a lot of the reason webapps took off in the early 2000s was because there was just a huge proliferation of worms and viruses targeting MS OSes in the late 90s and early 2000s, and it got to the point where consumers would only use webapps because they couldn't be confident that random software downloaded off the Internet wouldn't steal their credit card numbers.

[black_knight]

Sometimes better ideas can be around for a really long time before they gain any mainstream traction. Some ideas which come to mind are anonymous functions and sum types with pattern matching, which are only recently finding their way into mainstream languages, despite having been around for ages.

What it might take is a dedicated effort over time by a group of believers, to keep the ideas alive and create new attempts, new projects regularly. So that when there is a mainstream opening, there is the knownhow to implement them.

I always include a lecture or two in my software security course (150 students per year), on capability based security. I am also on the lookout for projects which could use the ideas, but so far I have only vague ideas that they could be combined with algebraic effects in some way in functional programming.

[codethief]

> but so far I have only vague ideas that they could be combined with algebraic effects in some way in functional programming.

This. Algebraic effects seem very much destined for this purpose. You could safely run any code snippet (LLM generated or not) and know that it will only produce the effects you allowed.

[nostrademons]

Interesting. I hadn't heard of algebraic effects, but they remind me a lot of the Common Lisp condition system, or delimited continuations, or monads in Haskell. There's even a shoutout to monads in the top Google result for the context:

https://overreacted.io/algebraic-effects-for-the-rest-of-us/

I assume that the connection to capability security is that you use the effect handler to inject the capabilities you want to offer to the callee, and their access is limited to a particular dynamic scope, and is then revoked once it exits from that block? Handler types effectively provide for the callee and define what capabilities it may invoke, but the actual implementation is injected from a higher level?

[black_knight]

Your description sounds about right!

I learned to understand it [0][1] as a way of creating free monads by wishing for a list of effects at the type level. Then later you worry about how to implement the effects. Solves the same problem as monad transformers, but without commit to an order up front (and without all the boilerplate of mtl).

My idea is that you should be able to create and pass around typed capabilities for effects, and then transparently get them “effectuated at their place of creation”.

[0] : http://okmij.org/ftp/Haskell/extensible/more.pdf

[1] : https://hackage.haskell.org/package/freer-simple

[dbmikus]

I'm going to be pedantic and note that iOS and Android both have the capability security model for their apps.

And totally agree that instead of reinventing the wheel here, we should just lift from how operating systems work, for two reasons:

1. there's a bunch of work and proven systems there already

2. it uses tools that exist in training data, instead of net new tools

[nostrademons]

App permissions in iOS and Android are both too coarse-grained to really be considered capabilities. Capabilities (at least as they exist in something like Eros or Capsicum) are more "You have access to this specific file" or "You can make connections to this specific server" rather than "You have access to files" and "You have access to the network". The file descriptor is passed in externally from a privileged process where the user explicitly decides what rights to give the process; there is no open() or connect() syscall available to user programs.

[saagarjha]

This seems neat in theory but it is very difficult to actually do in practice. For example, let's say that you are allowed to make connections to a specific server. First, you have to get everyone onboard with isolating their code at that granularity, which requires a major rewrite that is easy to short-circuit by just being lazy and allowing overly broad permissions. But even then "a server" is hard to judge. Do you look at eTLD+1 (and ship PSL)? Do you look at IP (and find out that everything is actually Cloudflare)? Do you distinguish between an app that talks to the Gmail API, and one that is trying to reach Firebase for analytics? It's a hard problem. Most OSes do have some sort of capabilities as you've mentioned but the difficulty is not making them super fine-grained but actually designing them in a way that they have meaning.

[nostrademons]

Yes, exactly. The implementation difficulties are why this idea hasn't taken the world by storm yet. Incentives are also not there for app developers to think about programming this way - it's much easier to just request a general permission and then work out the details later.

For the server ID, it really should be based on the public key of the server. A particular service should keep its private key secret, broadcast a public key for talking to it, and then being able to encrypt traffic that the server can decrypt defines a valid connection capability. Then the physical location of the server can change as needed, and you can intersperse layers of DDoS protection and load balancing and caching while being secure in knowing that all the intervening routers do not have access to the actual communication.

[saagarjha]

Should Google and YouTube share a key? How about Google LLC and Google UK Limited?

[nostrademons]

No, and both Google and YouTube should be multiple keys. YouTube, for example, should have separate capabilities for watching videos (ideally on a video-by-video basis); autoplaying; viewing metadata; posting comments; uploading videos; viewing playlists; editing playlists; signing up for YouTube Premium; and a number of other operations. That way, somebody who has granted their software agent the capability to play YouTube instructional videos doesn't risk having their credit card charged for YouTube Premium or ruining their reputation by posting questionable spam on everybody else's videos.

Setting up the economic incentives to encourage this is a hard problem. Right now, the disincentives to this are: 1) it adds engineering overhead to secure everything behind a different private key 2) it adds PM overhead to determine what the right granularity is and what potential adversarial avenues exist 3) it creates user confusion as they're bombarded with a list of fine-grained permissions 4) it prevents tech companies from cross-selling new features and impedes new feature discovery. There's basically no reason to do it other than protecting the user, and the user probably won't know they've been compromised until long after they start using the service.

[Terretta]

One can sort of get there today combining something like attribute based access control, signed bearer tokens with attributes, and some sort of a degrees-of-delegability limiter that a bearer can pass along like a packet TTL.

Did you want it in rust?

- https://github.com/eclipse-biscuit/biscuit-rust

- https://www.biscuitsec.org/

[pdntspa]

How would that even work when the web is basically one big black box to the OS? Most of the stuff that matters to most consumers is on the web now anyway. I don't see how 'capabilities' would even work within the context of a user-agent LLM

[nostrademons]

You'd have to rewrite most of the software used in modern life. Most of it is conceptually not built with a capability security model in mind. Instead of providing the LLM with access to your banking app, you need a new banking app that is built to provide access to your account and only your account, and additionally also offers a bunch of new controls like being able to set a budget for an operation and restrict the set of allowable payees to an allowlist. Instead of the app being "Log into Wells Fargo and send a payment with Zelle", the app becomes "Pay my babysitter no more than $200", and then the LLM is allowed to access that as part of its overall task scheduling.

This is a major reason why capability security has failed in the marketplace.

[oleszhulyn]

I'm building this for AI devices [1]. It's my YC F2025 submission.

[1] https://uni-ai.com