[radford-neal]

A basic problem with evaluations like these is that the test is designed to discriminate between humans who would make good lawyers and humans who would not make good lawyers. The test is not necessarily any good at telling whether a non-human would make a good lawyer, since it will not test anything that pretty much all humans know, but non-humans may not.

For example, I doubt that it asks whether, for a person of average wealth and income, a $1000 fine is a more or less severe punishment than a month in jail.

[justinpombrio]

For a person of average wealth and income, is a $1000 fine is a more or less severe punishment than a month in jail? Be brief.

"For a person of average wealth and income, a $1000 fine is generally less severe than a month in jail. A month in jail entails loss of freedom, potential loss of employment, and social stigma, while a $1000 fine, though financially burdensome, does not affect one's freedom or ability to work" --ChatGPT 4o

[johnchristopher]

"potential loss of employment,"

Where is that coming from ? That's a very lawyery way to phrase things.

"potential ?" where I live I think people may max out their holidays and overtime (if lucky enough) and leave-without-pay but there would be a conversation with your employer to justify it and how to handle the workload.

In the USA, from what I read, it's more than likely that you would just be fired on the spot, right ?

edit: just googled a bit, where I live you must tell your employer why you will be absent if you go to jail but that can't be used to justify the breaking of the contract unless the reason for the incarceration is damaging to the company and... yeah, I am definitely not a lawyer :]

[ghaff]

Leave-without-pay normally requires some specific justification(s)/discussion. I've certainly given my manager advanced notice about any longer stretches of vacation and I've tried to do it with awareness of workloads (though for something planned months in advance that's not always possible) but I've pretty much never considered it as asking for permission or it being a negotiation. This is in the US.

ADDED: You're probably going to end up lying or at least being very vague "some family stuff to take care of" in this specific scenario but for one month that didn't trigger reporting to employer a lot of professionals could probably get off with it. In any case, the GPT answer seems totally correct for the parameters given.

[johnchristopher]

> ADDED: You're probably going to end up lying or at least being very vague "some family stuff to take care of" in this specific scenario but for one month that didn't trigger reporting to employer a lot of professionals could probably get off with it. In any case, the GPT answer seems totally correct for the parameters given.

Where I live that is subject to cancellation of the work contact, you can't lie about why you are absent though the imprisonment can't be cause by itself for laying off.

[ghaff]

Maybe you're talking about something else but for standard PTO/vacation that I'm owed it would seem absurd if I had to justify how I was going to spend my time off. It's none of your business.

[johnchristopher]

Obviously not talking about standard PTO/vacation.

Where I live you must inform your employer unless you are lucky enough to have enough PTO/va. to spend your PTO/vacation in prison and hide it from your employer. But even then I don't think that will work out because there are administrative stuff related to social welfare you have to comply to and at some point it will be on your employer's radar anyway (why is that guy exempt from social welfare taxes for that specific month ? and why did the police asked me to confirm he was working here ? etc.).

BUT in practice, until recently, if imprisonment is less than 3 years then you won't spend a day in prison (unless you are deemed too dangerous). But then you have an electronic bracelet and other obligations that will at some point alert your employer.

I read now that prison time of any length will have to be spent in prison (no more less than 3 years or 2 years or 18 months pass). But for prison time less than 18 months you can have a bracelet on the first day.

All that to say I don't think it's manageable and possible to hide prison time from your employer, even if you have enough PTO/vac. days to cover for it.

[oconnor663]

My wild guess is that it would depend a lot on how much your employer likes you and how they feel about the reason you're in jail.

[akira2501]

Many jails have work release. They get you up at 6am, check you out of the jail, let you go to work, then expect you to check back into jail by 6pm.

[ghaff]

Which for many professional (and other jobs) probably would require a bunch of tap-dancing around your strict schedule if you were hiding the actual reason.

[johnchristopher]

Oh, that's really great ! Is that in the US ?

[Dayshine]

Self-employed or small company might not care.

[EGreg]

So the GPT is on a law exam and is using a very lawyer way to word things? I would say that's great!!

[Rinzler89]

What does GPT consider being "average wealth and income". Statistics? Or biased weights from anecdotes he formed on the anecdotes he scraped off the internet on how wealthy people say the feel?

Would be cool to know how LLMs shape their opinions.

[LeoPanthera]

You can just ask it, you know.

GPT-4o:

“Average wealth and income” can vary significantly by region and context. However, in the United States, as a rough benchmark, the median household income is around $70,000 per year. Wealth, which includes assets such as savings, property, and investments minus debts, is harder to pinpoint but median net worth for U.S. households is approximately $100,000. These figures provide a general idea of what might be considered “average” in terms of wealth and income."

[lambdaxyzw]

I like that it immediately assumed the US, even though nothing in your question suggested it. I love that all LLMs have a strong US centric bias.

Btw I'm not personally a lawyer, but I've heard that GPT is especially prone to mixing laws across the borders - for example you ask a law question in language X, and get a response that uses a law from a country Y - and it's extremally convincing doing that (unless you're a lawyer, I guess).

[LeoPanthera]

ChatGPT has user-customizable "instructions", and mine are set to tell it where I live. Any user can do the same, so that it will not make incorrect assumptions for you.

[d3m0t3p]

You might increase the probability of getting a correct answer for your region, but imo you decrease your awareness to allucination. Overall you can still get a wrong answer

[fjdjshsh]

This is my experience with Hackernews. If the comment doesn't specify the country, it's an American talking about the USA

[LordDragonfang]

I mean, to be fair, if you're speaking English to it, the most likely possibility is that you're inside the US:

https://en.wikipedia.org/wiki/List_of_countries_by_English-s...

I know there's a lot of complaints about things being US-centric, but the US is a very large country.

[ordersofmag]

Well, except the number of English speakers outside the US is much larger than inside the US (as per the wikipedia page you point to) by 5 to 1. Granted many folks are speaking it as their 2nd (or nth) language. But when you take into account the limited set of languages supported by ChatGPT one could reasonably assume English-speaking (typing) users of ChatGPT are from outside the U.S. as non-U.S. folks are in the majority of 'folks for whom english would be their first option when interacting with ChatGPT'. Even if you only count India, Nigeria and Pakistan.

Though of course OpenAI can tell (frequently, roughly) where folks are coming from geographically and could (does?) take that into account.

[duskwuff]

> but the US is a very large country.

Indeed - the US is a very large country, and consists of over 50 different jurisdictions, each with their own slightly different laws. An answer to a legal question which is correct in one state will often be subtly incorrect in another, and completely wrong in yet another.

[Rinzler89]

>You can just ask it, you know.

But my question will not be part of the context of that conversation.

[LeoPanthera]

Mine was. I asked it the first question, first.

[Rinzler89]

The original conversation I mean, of the grandparent. Me starting an identical conversation will not guarantee an identical context.

[anon373839]

Honestly, this is giving the bar exam (and GPT-4) too much credit. The bar tests memorization because it's challenging for humans and easy to score objectively. But memorization isn't that important in legal practice; analysis is. LLMs are superhuman at memorization but terrible at analysis.

[lazide]

Eh, also in legal practice there are key skills like selecting the best billable clients, covering your ass, building a reputation, choosing the right market segment, etc. which I’d also argue LLMs suck at.

[gadflyinyoureye]

I don’t know. There was some talk this weekend about CEOs being replaced by AI. Given the overlap in skill, I’d say there is a distinct possibility an LLM could do that. https://www.msn.com/en-us/money/companies/ceos-could-easily-...

[threeseed]

Phoebe Moore who that quote was attributed to has never been a CEO or even worked at a non-academic organisation.

So much of a what a CEO does is fostering culture, hiring people and setting a unique vision for the company.

Imagine thinking people would be inspired to work for a chatbot. Hilariously ridiculous.

[pojzon]

If that chatbot had Steve Jobs voice ?

I dunno, I would probably prefer to work under that chatbot than my current CEO that only tries to squize as much as possible out of ppl already working for him.

[lazide]

Like the chatbot wouldn’t squeeze you 10x harder.

At least a human CEO has to worry about being arrested or someone setting their house on fire.

[lazide]

Bwahaha. This is like the ‘everything can be a directed graph db’, ‘everything should be a micro service’, etc. fads.

No one who has been a CEO, or frankly even worked closely with one, would think this could be even remotely close to possible. Or desirable if it was.

But that is probably 1% or less of the population eh?

[EGreg]

https://www.dqindia.com/company-makes-ai-robot-its-ceo-makes...

Seems your claim's been disproven already

[lazide]

Bwaha. Funny the company named as doing so doesn’t mention it on their actual management team [http://www.netdragon.com/about/management-team.shtml], listing an actual human CEO instead.

But it makes for a fun soundbite eh? Especially when the article claims it was in the past, and totally was awesome. Sucker born every minute.

[paulcole]

> which I’d also argue LLMs suck at

OK, I’ll bite. What’s your evidence for this argument?

[lazide]

Every bit of interaction I’ve ever had with an LLM. And all the research I’ve seen.

They’re plausible word sequence generators, not ‘planning for the future’ agents. Or market analyzers. Or character evaluators. Or anything else.

And they tend to be really ‘gullible’.

What evidence do you have they could do any of those things? (And not just generate plausible text at a prompt, but actually do those things)

[paulcole]

> What evidence do you have they could do any of those things?

Every bit of interaction I’ve ever had with an LLM.

[Terr_]

Yeah, I fear a lot of human exuberance (and thus investment) is riding on the questionable idea that a really good text-fragment-correlation specialist engine can usefully impersonate a generalist "thinking" AI without doing too much damage. ("LLM, which rocks are the best to eat?")

But there's a scarier further step: When people assume an exceptional text-specialist model can also meta-impersonate a generalist model impersonating a specific and different kind of specialist! ("LLM, create a legal defense.")

[KennyBlanken]

You clearly don't know anything about the bar. One half of your score is split between 6 essay questions, and reviewing two cases to then follow instructions from a theoretical lead attorney.

[anon373839]

I’m licensed in multiple states, including California.

The essay questions also test memorization. They don’t require any difficult analysis - just superficial issue-spotting and reciting the correct elements.

If the bar exam were not a memorization test, it would be open book!

[ethbr1]

I've always drawn the link between skill in memorization and in analysis as:

- Memorization requires you to retain the details of a large amount of material

- The most time-efficient analysis uses instant-recall of relevant general themes to guide research

- Ergo, if someone can memorize and recall a large number of details, they can probably also recall relevant general themes, and therefore quickly perform quality analysis

(Side note: memorization also proves you actually read the material in the first place)

[Jensson]

Problem is the LLM memorized the countless examples you can find of old BAR questions using extreme amounts of compute at training time, they don't have that ability to digest a specific case due to both lack of data and it doesn't retrain for new questions.

A human that can digest the general law can also digest a special case, but that isn't true for an LLM.

[anon373839]

I’m not sure why you’re being downvoted for this. I agree with you, fact recall is useful and necessary. If you have a larger and more tightly connected base of facts in your head, you can draw better connections.

And even though legal practice tends to be fairly slow and deliberative, there are settings (such as trial advocacy) where there is a real advantage to being able to cite a case or statute from memory.

All that said, I still maintain that it’s a poor way to compare humans with machines, for the same reason it would be poor to compare GPT-4 to a novelist on their tokens per second written.