[anddt]

I tend to believe this is a valid approach for ML application in public institutions. Being ML a black box or a potential source of discrimination, setting up systems to ensure credibility within the public should be addressed early on in the design process (perhaps while designing the model itself).

I wouldn't be surprised to see a future where all ML applications affecting citizens and the general public interest (excluding defense, military, etc.) will be obliged to be open-source and subject to public scrutiny.

[read_if_gay_]

> I wouldn't be surprised to see a future where all ML applications affecting citizens and the general public interest (excluding defense, military, etc.) will be obliged to be open-source and subject to public scrutiny.

I would be positively surprised.

[anddt]

Sorry if it didn't come across in the post, it'd be no doubt a fair step towards more ethically sound governments.

[bettem]

In Australia, any Federal Government algorithm can be FOI’d by a member of the public. It’s one of the reasons the Aus Federal Government statisticians tend away from black box AI and towards more traditional methods.

[anddt]

> [...] Government statisticians tend away from black box AI and towards more traditional methods.

I was indeed wondering how constraints on model interpretation forces governments to stay away from NN and more complex approaches.

I have worked for a fraud-prevention company for a few years, and the financial nature of the decisions involved led to reward model interpretation (sometime at accuracy's expense) in order not to create friction with clients. Definitely not the best not being able to explain why your model performs poorly or what actions are needed to solve a problem.

[wjnc]

In theory for NL one could argue this is already the case. Government internals are public (WOB / transparent government). When I was a civil servant we once shared data and code with a citizen, although more regularly shared are internal documents. Our government has to make informed and documented decisions (AWB - general public law) and no decision that has legal ramifications can be made without manual intervention (an example: even automatical traffic fines have a name / number of an attested civil servant that finalized the fine in the paper trail). So all ingredients are there. Suppose they use external models or tools, how could they ever substantiate their decision in court without giving quite a lot transparency to the court?

Now in practice .... Our IRS has had a discriminatory way of working in place where no-one involved had access to their own files (GDPR?!) and even judges were reluctant to order transparency in the following proceedings giving the IRS the benefit of the doubt. Ultimately this became a politican scandal and will take years to resolve. And in the paragraph above I would have argued that the basics considering any case should be easily available in as much as 8 weeks for every citizen...

[anddt]

> no decision that has legal ramifications can be made without manual intervention

I think there's no other way around this: ultimately, everything that has legal implication will be potentially subject to dispute. We're not yet in the dystopian environment where disputes can be handled by a model :)