[yrb]

I got the impression that SRE basically have low level access to the storage stack. So wouldn't be subject to most of the normal application level logging that I would assume would red flag this behaviour pretty fast.

The only way to get around this is to have someone audit all their actions constantly, which you need someone equally or more familiar with the systems they are working with.

I think that is pretty impossible to implement that level of overview with humans, the best way to go normally is the 'buddy system' so no one can access a system unless they have a 'buddy' with them. Like the military do in nuclear weapon silos.

[jacquesm]

Access to the low level storage stack would not allow you to query with so much detail and would likely not have an interface that would allow you to modify user settings at will. So he must have used some higher level tools.

[birken]

Well it depends.

For example if an application uses Bigtable, then the key + column names often gives a lot of information about what data is stored there, which if somebody had access to some basic application data they might be able to get at somebodies specific data.

However as you might expect there are many safeguards in place, including ensuring every action is fully and securely authenticated so even low level SREs cannot read application data without a paper trail. This story is pretty surprising to me, and if true this guy is an idiot.

[yrb]

Depends how subversive he was trying to be I guess. I was thinking more around the query layer for bigtable etc. He probably would have known the stack top to bottom.

I am not sure this is a 'solvable' problem. You can mitigate by always working in pairs. But even that just reduces the potential for privacy breaches.