[AnonHP]

> Any call the agent makes to an external connector is reviewed by an LLM judge to prevent dangerous operations.

A few basic questions: how reliable is this judge since it’s based on an LLM? What additional measures can an admin or someone with more technical knowledge take to tighten this further if needed? Are the rules in or used by this judge visible to the users or an admin?

[tcdent]

It's not reliable, and presenting it as-such shows a lack of understanding of basic security posture.

Don't take my word for it, watch the news in <6 months and we'll certainly see it get bypassed.

[dewey]

Maybe that's a hot take, but it's probably still safer than how things are often done in reality when employees want to "get something done" and sign up for random services, download their own software or other variants of shadow IT.

[tcdent]

Completely agree.

This posture is applicable while we ease out of what I'm calling the "golden era" (loose restrictions on access with limited consequence) but one of the reasons I am pedantic and bearish about solutions like this is that they too have a short lifespan that we can already estimate the expiration date of.

[ChrisMarshallNY]

Aren't a large number of massive data leaks caused by some Marketing person setting up an AWS instance, and dumping the corporate DB into it?

[dewey]

I wouldn't say that's a "Marketing person" failure mode, there's enough open buckets, public mongodb or Elastic instances with wrong credentials.

[ChrisMarshallNY]

Fair point.

[felixding]

It's not safe or reliable, no matter how hard you try. Fundamentally, agents will try whatever they can to achieve the goal, as we've seen in many recent security incidents.

If determinism is important, as it is in many business use cases, use agents to build the tool, but don't let them run it.

I built a similar product around this exact idea: AI builds the workflow. Fixed rules run it.

[hudsongr]

We assume LLMs can always be fooled, so the judge is just one step of the verification process. The deterministic layers, per-tool db roles, read-only-by-default integrations, query timeouts, are the ones an admin can configure. The judge's rules aren't exposed to admins today. I like your idea of letting admins add their own rules or knowledge to the judge, we should add that.

[Arshad-Talpur]

One of the core issue here is LLMs are probabilistic not deterministic, even with set of rules or context , llms are inherently built to make best guess, so my question is how you are ensuring that LLMs are following rules?

[marinoseliades]

You're right, and we don't. The judge is best-effort screening not enforcement. Enforcement is deterministic with per-tool Postgres roles, proxy-injected creds, host allow-list, human approval on destructive writes. Those hold whether the judge is right or wrong.