[dathinab]

And this + the tendency for AI to "prefer" AI produced code + some other AI biased is why *this is most likely highly illegal to use in the EU due to violating anti discrimination laws in multiple ways.

To be clear:

- randomly filtering "too many" resumes is pretty much allowed (I think)

- but must be actual random independent of the resume (and can be in multiple layers, i.e. random filter > pre-select > random filter > select)

- this isn't the case for AI as the random aspect isn't done as the random aspect is not independent of the actual resume evaluation

- in general you can't make sure the AI doesn't apply systematic biases, and there is high indication that it does do so

- for humans you can train them and order them to ignore their biases, this won't work reliable either _but now you delegated the responsibility of illegal biases to the hiring personal violating the order_. But for AI usage you are responsibility no matter what you tell it. Lastly you can technically "show/proof" a specific used AI is highly biased in a specific contexts, which for human employees is technical possible but practical not really practical. So this moves "specific mostly deniable" cases, into "systematic proven bias" teritory. Or in other word legal risk goes from "limited/no issue" to "people can systematically f-you over if they know you use AI for hiring".

[jerf]

Everything is correlated to everything [1].

Which means there's a good chance this is somehow correlated in one way or another to race/gender/other protected classes in the US, just by the math of everything being correlated to everything.

Which means this is one good lawsuit away from being illegal in the US as well. It doesn't even necessarily have to "win", just do well enough in court to scare away anyone else from using this.

And boy oh boy would I hate to be on the receiving end of this lawsuit, trying to prove that my AI screener is completely in compliance with all hiring laws. That sounds like a nightmare.

[1]: https://gwern.net/everything

[oceansweep]

Already happening with Workday in California:

https://news.bloomberglaw.com/litigation/workday-loses-bid-t...

[torben-friis]

Would the accused party have to prove compliance? Or would non compliance have to be proved by the accuser?

Honest question, I'm not American.

[jerf]

"Innocent until proven guilty" is a criminal court concept. This would be a civil suit. Those use different standards, like "preponderance of the evidence". I agree that if the claimant had to prove the AI system is violating employment law that that would be a hard bar to clear, but showing on the preponderance of the evidence is something that would have me a lot more nervous if I was on the receiving end of the lawsuit.

This is a highly general answer to a complicated topic; my main point is more that this is not going to be held to the standard of "beyond reasonable doubt", which would be hard to meet.

[1]: https://www.law.cornell.edu/wex/preponderance_of_the_evidenc...

[DiscourseFan]

I wouldn't doubt that lawsuits for employment discrimination for any company (and I suppose it was most of them) that used LLMs in hiring processes will become a very lucrative business. They are all open to civil suits at this point.

[AnimalMuppet]

And, if there aren't enough lawyers to do all that work, you could use AI to file the suits.

I'll let you decide whether that's a dream or a nightmare...

[DiscourseFan]

I don't see why its a problem to use LLMs to assist with legal work if someone else uses it in a way that exposes them to lawsuits. That's like saying you shouldn't using a ledger to do accounting just because some people cook the books.

[AnimalMuppet]

Because using it to do shoddy work that harms people is the basis for the suit. If you're using it for the lawyering, too, how do you know that it won't do shoddy work in writing the lawsuit - shoddy work that could hurt the client?

[DiscourseFan]

Yes of course a mass produced doorknob is shoddier than the handiwork of a single craftsmen, but it can be used to open far more doors

[nonethewiser]

>Which means there's a good chance this is somehow correlated in one way or another to race/gender/other protected classes in the US, just by the math of everything being correlated to everything.

>Which means this is one good lawsuit away from being illegal in the US as well.

Uhh.. what? No that doesn't follow at all.

Screening resumes in a way that correlates to race, gender, etc. is not illegal. This is a fundamental distinction. The law is you cannot use those as filters. But the outcomes likely will be correlated. In fact to ensure they are not correlated you'd have to break the law and control for race, gender etc. Which is racism.

The models dont even get race as an input. If they did and they used it to select then yeah, that lawsuit sounds like it has merit. But a mere correlation in outcomes? In no way illegal what-so-ever.

[tikhonj]

The US has a notion of "disparate impact"[1] that means you can be liable for discriminating based on a protected characteristic on the basis of correlation. This is why HR departments are very hesitant to use things like IQ tests for screening candidates, for example.

[1]: https://www.congress.gov/crs-product/IF13057

[tptacek]

If they are, they're falling for an urban myth, because IQ tests in white-collar employment are legally fine, and several household-name corporations use them openly. I'm a noodge about this particular issue because there's a folk belief that IQ tests are the secret perfect hiring mechanism; it wouldn't bother me so much if it was so obvious how badly a pure IQ-test process would perform for a job in our field.

[alsetmusic]

>

How likely is an LLM to have different outcomes for Tyrone vs Jeff?

[CobrastanJorji]

> randomly filtering "too many" resumes is pretty much allowed (I think)

It's totally fine to filter out resumes in a completely random, content-independent way. Grabbing the fourth resume down in the pile and offering them the job is a perfectly fair albeit stupid way to make a hiring decision. However, AIs are very, very good at capturing biases, and it would not at all surprise me if an AI told to filter resumes is going to end up filtering with some biases for things that you definitely do not want to filter on, like the name of the candidate. And it might be that everybody resume that claims it fixed a typo in a major open source project gets a pass, but resumes that only list their own projects get rejected 60% of the time, so you're losing more good candidates than bad.

[District5524]

I'm not sure this is very easy to show this is a breach of non-discrimination requirements, like under Council Directive 2000/78/EC for employment.

Due to acting like an irrational gambling machine, I agree it can have unwanted indirect discrimination effect in general. But it will probably not differentiate "on the grounds of religion or belief, disability, age or sexual orientation". It is possible, but that would take a lot of work for the lawyers to prove to the court.

I believe the more interesting part is that the EU AI Act (still not in force in this regard until 2 December 2027). This will be clearly a high-risk AI system: "AI systems intended to be used for the recruitment or selection of natural persons, in particular to place targeted job advertisements, to analyse and filter job applications, and to evaluate candidates".

Which does not mean prohibited, but it could later turn out that LLMs will be excluded from being used in high-risk AI use cases (falling under article 6 with no exemptions).

Considering that none of the standards are published yet, I have absolultely no idea how they will ensure compliance with the following parts of Article 10 when using LLMs for such tasks: "(f) examination in view of possible biases that are likely to affect the health and safety of persons, have a negative impact on fundamental rights or lead to discrimination prohibited under Union law, especially where data outputs influence inputs for future operations; (g) appropriate measures to detect, prevent and mitigate possible biases identified according to point (f)"

I don't think that's technically possible to do so with LLMs in general at the moment, even with the full cooperation of the model providers. Maybe you can do some meaningful audits for smaller models. But the EU AI Act may end up excluding all the generic "using-LLM-but-not-entirely-sure-why" vibe coded approaches from high-risk use cases (in Annex III). Which would make sense.

https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng

[dathinab]

EU AI Act got hijacked by huge corpo with last minute changed with moved it from "could probably work" to "catastrophe".

Even at 2 December 2027 it might be intentionally not enforced at all due to that for a while, through I think the goal is currently to amend it until then.

> that LLMs will be excluded from being used in high-risk AI use cases

no, it won't I can guarantee you this. At best they will get additional restrictions over time, as things go wrong. Anyone who could make this happen has way too much interest to not make it happen. (Most/All? EU country legal systems are overloaded to a point of not working correctly anymore, and have been before AI generated law suites and other AI nonsense started. I won't go into detail but many believe AI assistance (for certain tasks, always with a human doing any final decisions) is the only way to get out of this mess).

> standards are published yet

or exist,

like seriously this isn't a case of there being non public WIP standards which will pin all the nitty bitty details down, but cases of state agencies (and in last instance judges) having to decide if a specific standard (or implementation) is sufficient or not.

but also to some degree it shouldn't be tightly coupled to tech standards as there are often many ways to implement the things the law requires and accepting only one is undesirable (and likely wouldn't legally hold up). But having tech standards which are a "guaranteed to be enough if you comply with" (but not the only valid way) would have been preferable, bringing us to the next point

> have absolutely no idea how they will ensure compliance

nor do they know, the original non big corpo hijacked version had exceptions for most companies affected now. So it would only have affected a handful of huge companies, which have many of the things required already in place, in some form or another. Most likely this would have played out as this companies presenting how their measurements are "sufficient" and the agencies then evaluating it and potentially requiring some changes, going back and force over a longer duration leading to documented cases of rough technical standards about "what is sufficient" they then can pass to other organizations in the future. But now the law affects not just a handful of companies but like thousands, if not tens of thousands. Many not stuffed in a way where such a process could work, or even do the necessary documentation to show "compliance"...

So from a practicability POV, if enforced starting 2027, it currently excludes close to _any_ (meaningful) use of AI, down to a trivial linear regression or similar. Including any "old school ML/AI" any Bank uses for risk assessment.

Banking stopping running in December and there not being any (meaningfull) AI startups or adoption at all is not something anyone (in power in any state organ) wants to see, so guess how much it will be enforced ;)

And as mentioned the chance of AI as technology being excluded "in general" is close to none. Maybe specific usages could be excluded (and/or are already excluded) but thats it.

Oh and as a bonus a malicious reading of f+g remove any proper privacy protections for any AI usage in high risk context, where it is often most relevant... (a more sane reading allow it, with ... tricks).

[buzer]

> this is most likely highly illegal to use in the EU due to violating anti discrimination laws in multiple ways.

It's generally illegal under GDPR Article 22.

> The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.

Exceptions in 22(2) are unlikely to apply. It's hard to argue that it's truly necessary (a) and consent (c) is almost always unavailable in employment context. (b) might apply, but it requires specific law in EU or Member State to authorize it.

[bluGill]

For C: I'm not sure how EU laws work, but ethics says that someone who needs a job cannot give consent since the possibility of a job if they give consent could be a bribe. See a lawyer for how it works in your country.

[dathinab]

also not fully sure, but AFIK there are limits to how far you can wave this right, in context of things like TOS, simple opt-in fields on forms etc.

Like YT would have loved to make you opt out of it (and probably has it in their TOS) but there where multiple cases of courts forcing them to handle it properly in the past as far as I remember.

My _guess_ is that at least if you don't sign a proper contract you can always force a human reevaluation. But also only that (so only semi useful). Also even with a proper contract it's unclear if it would be possible in this specific case due to the contract being fundamentally one-side/unfair and semi-forced on you if it where wide spread on the market for the specific job you are trying to get.

[bluGill]

Those limits exist too, but even if the law doesn't give limits, ethics does.

[buzer]

That's why I said consent usually cannot be used in employment context. I wouldn't rule it out 100% for everything employment related, but application screening is unlikely to qualify for those rare cases.

[dathinab]

this isn't quite how GDPR Article 22 works

The is a difference between

- having a right you can't wave - which is very similar to something being forbidden - but different to having a right you fully or partially can wave

Furthermore to some degree you are only "subject to a decision based on ..." if the decision has an effects affecting you.

In practice wrt. Article 22 this means companies can make a "decision solely based on automated processing[..]" iff they give you a (realistic) chance to object to it in which case they will do a human review of the decision where a human confirms/changes this decision based on reviewing the involved information.

There is a lot of gray area what a "chance to object" means and when a human review makes an decision no longer "solely based on automated processing" (a human just saying AI was right clearly doesn't count, but a human constructing a case why they would have decided the same way based on the why the AI did the decision can count, iff it's reasonable to assume a human might have come to the decision had it only been reviews by an human).

Or in other words GDRP Article 22, just "soso" meaningful in context of hiring.

Like if the AI did a mistake they have to reevaluate it, but as long as there are other similarly qualified competitor (they did hire/are in process of hiring) it quite easy to come up with a reason why they are a better choice for them. Or go through the motions of you being in round 2,3 of hiring and then find an excuse to not hire you.

[buzer]

Mostly yes.

Note the chance to object must be given before decision is made, i.e. not to give option for human review after the fact. Human must also be able to actually have meaningful chance to affect the decision.

If the decision is based on purely objective facts that are actually necessary (like you must have certain license) then human and computer always coming to same decision is likely correct and compliant, but as soon as you start putting in subjective criteria and human agrees with 100% of computer denials it becomes a lot harder to demonstrate that human is actually able to affect the decision as required by Article 5. Note that demonstration burden is on controller, not on data subject/DPA.

Objective criteria also isn't always enough by itself. If both human and computer calculate the same credit score and you must score X points to get a loan then human isn't actually able to affect the decision. Essentially the credit score calculation itself ends up being the automated decision rather than the formal rejection that is later given to data subject.