[akiselev]

Use it to analyze the California & US Code, the California & Federal Codes of Regulation, and bills currently in the California legislation & Congress. It's far from useless but far more useful for creative writing than any kind of understanding or instruction following when it comes to complex topics.

Even performing a map-reduce over large documents to summarize or analyze them for a specific audience is largely beyond it. A 32K context size is a pittance when it comes to a single Title in the USC or CFR, which average into the millions of tokens each.

[p1esk]

Interesting - do you believe average humans (not professional lawyers) would do better on this task?

[akiselev]

Yes. I can parse them just fine after reading a single book called Introduction to Legal Reasoning [1]. I can also autonomously take notes and keep track of a large context using a combination of short and long term memory despite not having any kind of degree let alone experience or a license to practice law.

How do you think people become lawyers and how smart do you think the average lawyer actually is? The problem is that there's hundreds of thousands if not millions of pages, not that it requires superhuman intelligence to understand.

Even if it were capable of intelligence in the bottom quartile of humanity it would be SO MUCH more useful than it is now because I'd be able run and get something useful out of thousands of models in parallel. As it stands now GPT4 fails miserably at scaling up the kind of reasoning and understanding that even relatively stupid humans are capable of.

[1] https://www.amazon.com/Introduction-Legal-Reasoning-Edward-L...

[streakfix]

Did you try fine tuning gpt4 with that book as input?

[sjy]

Fine-tuning requires you to train the model with a set of prompts and desired completions. Building a suitable dataset is not trivial and it's not clear what it would mean to use a book for fine-tuning anyway – masking sentences and paragraphs and training the model to complete them in the book's style?

[streakfix]

> masking sentences and paragraphs and training the model to complete them in the book's style?

That would work.

[akiselev]

OpenAI doesn't support fine tuning of GPT4 and with context stuffing,the more of the book I include in the input the less of the bills I can include - which, again, are millions of tokens - and the less space there is for memory.

[p1esk]

I believe you. But at the same time they showed during the demo how it can do taxes, using a multi page document. An ability to process longer documents seems more like an engineering challenge rather than a fundamental limitation.

[akiselev]

Doing taxes using a few small forms designed together by the same agency is not as impressive as you think it is. The instructions are literally printed on the form in English for the kind of people who you consider dumber than ChatGPT.

It quickly breaks down even at 8k with legislation that is even remotely nontrivial.

[karpierz]

Probably not.

I'm not sure that pointing out that LLMs are as useful for parsing legal code as the average human is something to brag about though.

[p1esk]

You're not sure that having an AI which is (at least) as intelligent as an average human is something to brag about? Seriously?

[karpierz]

Where did you get that the LLM was as _intelligent_ as a human?

All we've shown is that LLMs are as useful for parsing legal text as the average human. Which is to say, not.

A dog is also as useful for parsing legal texts as the average human. So is a rock.

[p1esk]

Where did you get that the LLM was as _intelligent_ as a human?

First hand experience -I’ve been using it daily for the past two months.

[karpierz]

Ah, if that's what you mean then there are plenty of intelligent systems out there.

I've used Google search for decades and it's been able to answer questions better than humans ever could. Same for Google Maps, though arguably they're the same system at this point. My calculator is far more intelligent than any human I've met, at least when it comes to adding large numbers. My compiler can detect even the slightest syntax error with impeccable accuracy. Microsoft word has an incredible vocabulary. Wikipedia knows more historical events than any human dead or alive. And so on.

Shit, users thought Eliza was intelligent in the 60s.

If what you really mean is that LLMs are cool and useful, then sure. Just say that instead of couching it in some vague claim of intelligence.