[xmprt]

> When a question touches restricted data — student PII, sensitive HR information — the agent doesn’t just refuse. It explains what it can’t access and proposes a safe reformulation. "I can’t show individual student names, but here’s the same analysis using anonymized IDs."

This part is scary. It implies that if I'm in a department that shouldn't have access to this data, the AI will still run the query for me and then do some post-processing to "anonymize" the data. This isn't how security is supposed to work... did we learn nothing from SQL injection?

[stephbook]

I see two vectors here

- The bot giving out PII by accident. You ignore it and report it.

- You trying to fool the bot into giving you PII you're not supposed to have. But you've created an audit trail of your 100 failed prompt injections. The company fires you.

This isn't public facing, open to anyone. This is more like a shared printer in the office.

[thunfischbrot]

In the strongest interpretation of that it would offer only data which the user is allowed to access. Why do you assume that them implementing a feature to prevent PII being accessed that they then turn around and return data which the user is not supposed to access?

[xmprt]

If it's PII data the best thing for them to do is not even allow the AI to have access to it. They're admitting to that so I doubt they've gone through the effort to forward the user's auth token to the downstream database.

And with security it's always best to assume the worst case (unless you're certain that something is safe) because that would lead you to add more safeguards rather than less.

[adambb]

To be fair to them, the architecture description said that each datasource had a unique agent, so the orchestrator AI didn't have direct data access, and that they specifically only allow access to data the user has permissions for.

Unclear if each datasource agent is ALSO AI based though, in which case it has just pushed the same concern down the line one hop.