[pton_xd]

GPT4, as smart and impressive as it is, starts forgetting or confusing key instructions with as few as 500 tokens (in my experimentation). Practically speaking the advertised 32k context window could be a few orders of magnitude smaller depending on what you're asking it to do!

[tarruda]

Did you see the Gemini 1.5 1M token demo? They upload the three.js library and ask it to make changes, which it does successfully and pretty quickly.

I want LLMs to fail at my profession as much as everyone at risk of losing their jobs, but unless Google is lying, things are looking pretty grim.

https://youtu.be/SSnsmqIj1MI?si=N0zYY_Zbbfz3KRWK

[pton_xd]

Yeah, LLMs seem exceptionally good at summarizing large amounts of structured data with a prompt at the end, like that YT demonstrates.

If you have a back-and-forth conversation, with the previous conversation chunks prepended as context to the next interaction, it will rapidly lose track of where you instructed it to spend its attention.

The manner in which the context is used seems to make a huge difference.

[betaby]

I didn't see Gemini, I saw other demos only to learn later on that they were staged.

[Workaccount2]

Google seems to have learned from that demo this time and made very plain and upfront demos.

[tarruda]

I hope that you are right, and that they faked the demo to have a short term pump on stock price.