[akiselev]

Yes. I can parse them just fine after reading a single book called Introduction to Legal Reasoning [1]. I can also autonomously take notes and keep track of a large context using a combination of short and long term memory despite not having any kind of degree let alone experience or a license to practice law.

How do you think people become lawyers and how smart do you think the average lawyer actually is? The problem is that there's hundreds of thousands if not millions of pages, not that it requires superhuman intelligence to understand.

Even if it were capable of intelligence in the bottom quartile of humanity it would be SO MUCH more useful than it is now because I'd be able run and get something useful out of thousands of models in parallel. As it stands now GPT4 fails miserably at scaling up the kind of reasoning and understanding that even relatively stupid humans are capable of.

[1] https://www.amazon.com/Introduction-Legal-Reasoning-Edward-L...

[streakfix]

Did you try fine tuning gpt4 with that book as input?

[sjy]

Fine-tuning requires you to train the model with a set of prompts and desired completions. Building a suitable dataset is not trivial and it's not clear what it would mean to use a book for fine-tuning anyway – masking sentences and paragraphs and training the model to complete them in the book's style?

[streakfix]

> masking sentences and paragraphs and training the model to complete them in the book's style?

That would work.

[akiselev]

OpenAI doesn't support fine tuning of GPT4 and with context stuffing,the more of the book I include in the input the less of the bills I can include - which, again, are millions of tokens - and the less space there is for memory.

[p1esk]

I believe you. But at the same time they showed during the demo how it can do taxes, using a multi page document. An ability to process longer documents seems more like an engineering challenge rather than a fundamental limitation.

[akiselev]

Doing taxes using a few small forms designed together by the same agency is not as impressive as you think it is. The instructions are literally printed on the form in English for the kind of people who you consider dumber than ChatGPT.

It quickly breaks down even at 8k with legislation that is even remotely nontrivial.

[p1esk]

The instructions are printed, yet I, and many other people, hire an accountant to do our taxes.

What if someone finds a good practical way to expand the context length to 10M tokens? Do you think such model won't be able to do your task?

It seems like you have an opportunity to compare 8k and 32k GPT-4 variants (I don't) - do you notice the difference?

[akiselev]

> The instructions are printed, yet I, and many other people, hire an accountant to do our taxes.

I can mow my lawn yet I still hire landscapers. That doesn't say anything about the difficulty of cutting grass or the intelligence of a DeWalt lawnmower but about specialization and economic tradeoffs - like the liability insurance accountants carry for their client work.

> What if someone finds a good practical way to expand the context length to 10M tokens? Do you think such model won't be able to do your task?

Not based on the current architecture (aka predict next token). It already fails at most of my use cases at 32K by default, unless I go to great lengths to tune the prompt.

> It seems like you have an opportunity to compare 8k and 32k GPT-4 variants (I don't) - do you notice the difference?

32K works better for my use case but requires much more careful prompt "engineering" to keep it from going off the rails. In practice, actually getting full 32K use out of it is a disaster since the connection will drop and I have to resend the entire context with a "continue" message, costing upwards of $10 for what should cost $2-4 per call. I haven't actually tried 32K on as much as a whole USC Title because that would costs thousands.