Yep. Too many people design systems (both technical and otherwise) under the assumption 'their team' will always be in control, and it'll be their enemies who suffer.
Or that hey, this power will never be abused because the system won't allow it, and those in control will always have our best interests in mind.
Don't collect more data than you have to, limit the power of any systems you implement, and always design things under the assumption your enemies will take control of it in future.
> Don't collect more data than you have to, limit the power of any systems you implement, and always design things under the assumption your enemies will take control of it in future.
This is great advice if you want to design Good systems, but under barely-regulated capitalism, those who design Good systems will be out-competed and put out of business by those who design Profitable systems :(
I remember when Schmidt came out with this line thinking the same thing.
But back then it seemed like a distant and improbable scenario - I objected to his statement more on point of principle than out of any realistic sense that it was things were going to get that bad in the near future.
Turns out they did.
(before anyone says it - yeah. Back then there were probably already enough examples of law enforcement overreach - not to mention the decades-long injustice of the "war on drugs". I need to learn to be more cynical)
> But back then it seemed like a distant and improbable scenario
I am trying to understand this perspective. I was there and nothing seemed improbable or remote. The remoteness was merely a function of technical & economic conditions. Historic precedents, domestic and foreign, past or near present, all pointed the same direction, underlining high probabilities.
Pessimism is rarely the correct inclination, with the exception of questions concerning freedom, power, security, and control. It is appropriately rational to question and highlight worst case outcomes in such cases.
This same pattern is happening yet again ('surprise!') with generative AI. Maybe it is necessary to assure that 'yes, this technology is very cool' as red flags are raised.
It is a very simple thought, backed by unassailable historic evidence: Humans enjoy lording it over other human beings. We should never create systems that permit a tiny tiny subset to realize such base desires. A very simple idea, truly.
Discussions on surveillance and misinformation often involve people advocating for granting more power to the government to prosecute who they believe violate their value system, unless the value system somehow changes and now you become the criminal. As an example, this is why breaking E2EE with backdoors to stop pedophilia, revoking immunity to social platforms for users' speech and the like are bad ideas - some day your values will become abhorrent, and the same tools that you used against others will be used against you.
I have nothing to hide, beyond a bit of casual piracy, at least until laws change significantly, but people I care about might.
My distaste for corporate stalking is, if I'm perfectly honest, at least partly selfish feelings of discomfort in losing what privacy I still have, but it is mostly concern for what some will use information about others to enforce.
Or that hey, this power will never be abused because the system won't allow it, and those in control will always have our best interests in mind.
Don't collect more data than you have to, limit the power of any systems you implement, and always design things under the assumption your enemies will take control of it in future.