[srer]

Increasing we automate processes, have programs do the work humans once did.

It's extremely helpful and productive, but it has a darker side. The processes are rigid because machines are rigid, and the designers cater to the 99% cases.

But then the 1% happens, and you're left out in the cold.

In the old world of humans and paper, as wasteful as it was, it was easy for exceptions to be made if the clerk was willing, and if they weren't you'd find another clerk, or a clerks supervisor. The processes tended towards being flexible.

But today, you increasingly don't interact with any humans, or if you do don't be surprised if, in your unusual case, they say "the computer won't let me".

As governments move more and more towards digitization, and embrace machine learning, I expect similar stories might unfold - only it won't be with an opt in social media website.

[EnKopVand]

> As governments move more and more towards digitization, and embrace machine learning

I spent a decade in the public sector digitalisation of Denmark, a country that competes with Estonia about having the most digitalisation in the world.

I fully believe we should legislate against automated processes taking decisive actions.

It’s inefficient, but what I experienced in regards to laws is that they are way more messy than anyone working in digitalisation seem to realise. We build a system that let employees report their business-related driving, in Denmark you get a tax-reduction when you drive in your own car for work purposes, and the laws covering it is basically an A4 page of tax-law that seems sort of clear. You have 3 set of taxation rates that you get to deduct from, they are meant to be used for different types of work related driving. Simple, right?

Well, it turned out that in 9 different municipalities there was 9 different ways to interpret that A4 page of law text, and, more than a 100 different union agreements on how to extend or alter the tax law for certain groups of workers.

As hilarious as it was to sit through meetings with different sets of tax people from different municipalities getting into heated arguments about who was break the law, it was also sort of eye opening for me at the time. Because we made this as an OSS project where we bought the development that we project managed. My role was part of the project management team as a code-reviewer/specifier of sorts, and all our estimates simply went out the window when we realised we really had to build all those different ways of interpreting the law, as well as making room for future alterations. In the end, it didn’t extend the project that much, I think we still delivered it on schedule but it was a very different product with lots and lots of setup required, because the different municipalities needed to be capable of deciding which rules were turned on for which groups of workers, as well as control over how the approval system was handled by everything from tax lawyers going through every submission to secretaries to RPA robots simply clicking accept on everything.

The system wasn’t related to decision making automation that couldn’t be easily undone by humans, because it was still a relatively simple system. But if that sort of complexity is what you get from some of the simplest legislation we have, then imagine what it would look like for laws covering thousands of A4 pages of text.

[nonrandomstring]

Dehumanisation of essential civic processes is a step towards "cybernetic governance", and is a topic I explore in some detail in Digital Vegan [1]. This is distinct from what most of us still call "e-governance" in subtle ways. I am concerned that people do not yet understand the nuances between processes that can be automated to really improve life and where we cross the line into technofascist dystopias that will tear societies apart.

I share an attitude with Frank Zappa here. Zappa was rather oddly against "Love song lyrics". He said they led to poor mental health by propagating unrealistic expectations of intimate relations.

Similarly, I think that Science Fiction has a lot to answer for. I personally love most SciFi, but like Orwell the Cyberpunk genre was misinterpreted as a blueprint instead of a warning, and many people carry around disported, unrealistic and quite mentally damaged ideas of what a "good" technological society should look like.

[1] https://digitalvegan.net

[fauigerzigerk]

>I fully believe we should legislate against automated processes taking decisive actions.

There are certainly decisions that should not be fully automated. But this has very little to do with the account recovery issue we're talking about.

I believe that account recovery, and more generally proving your identity, can be done automatically with greater accuracy and far more securely than any process involving humans.

We have secure, electronic, government issued identity documents that are perfectly suitable for automation. Let's just use them! If we must legislate then let's introduce a right to prove our identity using our government issued ID.

There are other issues related to oligopoly accounts that are hard to solve. But proof of identity is not one of them.

[mellavora]

> We have secure, electronic, government issued identity documents that are perfectly suitable for automation.

And what do you propose as a solution if your government-provided identity gets lost or stolen or hacked?

Or for people who have a hard time getting such a doc? (note: Sweden currently has a crisis because it can take over 1 year to get a passport).

Or for people who live in countries which don't have these systems?

Are you really ok with uploading a video of you holding your passport every time you want to log onto a service (see "id.me" controversy)?

Now, what might be nice is if the government used a highly secure crypotgraphic system to allow identity verification, but drivers licenses and passports aren't that.

[fauigerzigerk]

>And what do you propose as a solution if your government-provided identity gets lost or stolen or hacked?

Report the old one stolen/compromised, get a new one, use it in the account recovery process.

>Or for people who have a hard time getting such a doc? Or for people who live in countries which don't have these systems?

This is a core responsibility of any government. It works well enough in many countries and we should not wait for the last government on earth to get its act together before using it. It can be gradually introduced country by country.

>Are you really ok with uploading a video of you holding your passport every time you want to log onto a service (see "id.me" controversy)?

Having a right to prove your identity using an official ID is not the same as having an obligation to do so. I would only use it with a few key accounts that I trust (and with financial institutions where ID checks are mandatory).

Also, I wouldn't have to hold up my passport at all, nor would I have to do it every time I log in. The platform would read the passport chip once upon registration or during account recovery and check if the picture on the chip matches my face.

>Now, what might be nice is if the government used a highly secure crypotgraphic system to allow identity verification, but drivers licenses and passports aren't that.

https://www.icao.int/Security/FAL/PKD/Pages/ePassport-Basics...

[michaelt]

> Having a right to prove your identity using an official ID is not the same as having an obligation to do so.

I'm sceptical as to whether you can avoid it becoming an obligation.

You sign up for $SOCIALNETWORK. Some opaque 'bot detection' process deems your account 'suspicious' and locks it. They offer to unlock your account if you prove your identity using an official ID.

That makes it obligatory in practice, if not in theory.

[fauigerzigerk]

I share your scepticism, but that's a political decision. Nothing protects us from bad political decisions besides participating in the democratic process.

What's happening right now is that we are sacrificing a lot for the financial benefit of corporations and for politicians' control obsession while we can't use some of the same technologies and capabilities for our own benefit.

We often have an obligation to prove our identity using a government issued ID, but we have no right to do so when we want to.

In my view, that's a bad deal.

[Perseids]

You can also read this story another way: Non-scaled manual processes have accumulated decades or generations of accidental complexities. I've seen this in the example of a central software my university was ordering to manage the records of all grades, achieved credits, registered exams and so forth. Most of this was already managed by a centralized agency (Zentrales Prüfungsamt) but every faculty had slightly different examination regulations and processes. It's not that most of these differences really provide any benefit to the students or the institution – electrical and mechanical engineering are so close to each other that there is no rational way to explain why they can't have the same length of time the registration window for practical courses is open – except that everybody is used to the way it is now and each faculty makes a stand for their right for the status quo.

And in my opinion the reason for most of the conflicts that arose is a failure of expectation management what the digitization effort can accomplish (in a reasonable budget): Software systems are cost efficient only with mostly homogeneous processes. Their development is such an expensive undertaking that it can only compete with individually trained humans when you can amortize the costs over large amount of use cases (c.f. https://xkcd.com/1319/ ).

Thus the first step should always be to get everybody on-board to give up some of their non-essential individuality. There is no need for car taxation to change from municipality to municipality. (Be aware of the reverse phenomenon as well, though: Individual needs getting thrown under the rug by systems that are too rigid or simplistic in the wrong places. See all the falsehoods programmers believe about {names, time, gender, ...} articles. TFA in my opinion is not an example of that phenomenon btw.: Facebook, like Google, is justifying cost cutting at places which obviously need trained human support, with a fetish for technological solutions.)

Of course this homogenization is not something that my parent poster would be in any position to accomplish, so this is not meant as a critique. Also I agree with EnKopVand that automated processes (or even overly rigid bureaucracies) should not take decisive actions on their own.

[EnKopVand]

You should absolutely read it that way, and, you should go even further and point fingers at the legislation itself. In my decade of public service we had five different ministers of “digitalisation” (they had other titles because IT doesn’t win votes) that all put effort into making our laws better suited for digitalisation. I think we even had a prime minister get into it, and every prime minister throughout my entire life has had an ambition of making laws less complicated.

Well, let’s just say that while you’re completely correct, I don’t think we should wait for our countries to become less Kafkan, which is why I’m a fan of simply banning the automated decision making. Maybe if you hurt the bureaucracy where it matters (cost) we might actually get some officials who deal with the root cause of the issues.

[at-fates-hands]

>> In the old world of humans and paper, as wasteful as it was, it was easy for exceptions to be made if the clerk was willing,

One of my first jobs out of college was an account manager at a big corporation. It was an easy job. Half customer service and half sales. The guy in the cube next to me was always chided for being a dinosaur because his desk (in his words) "looked like a tree just puked on his desk" because of all the paper copies he had floating around - but damn if he couldn't find a contract six months old or an email with some promise he had made a client a year earlier.

It was his file system and it worked magically. You'd ask him a question and he'd look around and then dive to a stack of copies of contracts, come up with the right contract and let the customer know the details so quickly. Customer's loved him because he could reference things so quickly and was so sharp with conversations and notes he had taken. No problem if his laptop crashed - he already had a paper copy. He always referenced himself as a go-between the paper world and the digital world that was quickly consuming his talents.

I heard he retired a few years back - but he was the guy you're talking about to a tee. He was around during the transition to email from everything being paper. Dude still made it work, even when he knew his time had come and gone.

[femto]

> As governments move more and more towards digitization, and embrace machine learning, I expect similar stories might unfold - only it won't be with an opt in social media website.

It's already arrived, in the form of the Australian government's "Robodebt" scheme: 20,000 automated debt notices per week with minimal oversight [1]. The government denies it killed people, but there are claims that it did [2]. After several years a court eventually stopped the scheme and awarded about $2 billion in compensation, but by then a lot of lives had been ruined.

[1] https://en.wikipedia.org/wiki/Robodebt_scheme

[2] https://www.theguardian.com/australia-news/2020/jul/31/not-c...

[treeman79]

My wife and most of her friends have all lost their Facebook accounts at least once. They all gave up getting them back. Many tears as most of them use it as their only photo backup for kid pictures.

At this point it’s just routine for them to have their account taken over and lost periodically.

[Syonyk]

> Many tears as most of them use it as their only photo backup for kid pictures.

Painful as that is, those of us who tend more technical should be helping people understand the nature of social media companies, and encouraging backups in whatever form is supported.

... and then helping push them over the edge to find other ways to communicate with friends, share photos, etc. "Do it all on Facebook!" was novel, 10-12 years ago. Now it's just a lack of creativity and a willingness to help add feet to the next yacht.

[mnahkies]

My girlfriend's mother had this happen to her recently. They got in and changed her password, profile picture and name. Account recovery options didn't work, and also reporting the profile as stolen / whatever option was most appropriate on the form didn't achieve anything.

I found it quite confusing as to the motivation, and seeming gap in the armour of automated bans.

As far as I know, many people get instant banned if they attempt to setup a second profile for their Oculus or similar, I assume the motivation is to get a fake account that has history to avoid this. What surprises me though is that changing password, profile picture and full name in quick succession + attempts to recover / report don't trigger this mechanism/some kind of review process.

[baby]

I feel like losing all your stuff is part of your digital journey. Then you learn the importance of backups.

Nowadays cloud backups are almost becoming default which is a good thing imo.

[black_puppydog]

Facebook was their cloud backup

[Ekaros]

Backup is a backup only if you have working copy. If not it is the working copy even not in use.

[baby]

Facebook is not a backup tool, that was their mistake.

[Schroedingersat]

It has nothing to do with automation and everything to do with unaccountable centralisation of power.

It does not matter what moderation scheme they use -- automated or beauraucratic, if you gift the town square to a private entity and allow access to it to be determined hy their whims you get this.

[artificialLimbs]

Reminds me of the rollout of Obamacare. I went to sign up on the website almost immediately. Ran into a host of errors. Called numerous times and every agent there told me they couldn't help and didn't know what was wrong. The solution: call us back in 3-4 weeks.