[jader201]

Am I the only one that feels like Claude is clearly winning code generation, and Gemini in general LLM?

I just don’t feel like OpenAI has a legitimate shot at winning any of the AI battles.

Therefore, I feel like “Sam Altman may control our future” is a far stretch.

[guelo]

Well I just canceled my Claude Pro subscription because of the mysterious limits that I don't experience with codex, even after paying for "extra usage". If Anthropic can't figure out their capacity problems they are in trouble.

[chrisjj]

I doubt Anthropic see this as their capacity problem. They like "extra usage", and users who don't, well its their capacity problem.

[dartharva]

The preference rankings keep fluctuating on every release for me. A year ago it was Gemini dominating coding tasks, then it was Claude, now it is the latest Codex again. With the next point release(s) the cycle will continue.

[gambiting]

>>and Gemini in general LLM?

You might be. Or at least I feel like Gemini is actually dumber than a house of bricks - I have multiple examples, just from last week, where following its advice would have lead to damage to equipment and could have hurt someone. That's just trying to work on an electronics project and askin Gemini for advice based on pictures and schematics - it just confidently states stuff that is 100000% bullshit, and I'm so glad that I have at least a basic understanding of how this stuff works or I would have easily hurt myself.

It's somewhat decent at putting together meal plans for me every week, but it just doesn't follow instructions and keeps repeating itself. It hardly feels worth any money right now, like it's some kind of giant joke that all these companies are playing on us, spending billions of these talking boxes that don't seem that intelligent.

I also use claude at work, and for C++ programming it behaves like someone who read a C++ book once and knows all the keywords, but has never actually written anything in C++ - the code it produces is barely usable, and only in very very small portions.

Edit: I just remembered another one that made me incredibly angry. I've been reading the Neuromancer on and off, and I got back into it, but to remind myself of the plot I asked Gemini to summarise the plot only up to chapter 14, and I specifically included the instruction that it should double check it's not spoiling anything from the rest of the book. Lo and behold, it just printed out the summary of the ending and how the characters actions up to chapter 14 relate to it. And that was in the "Pro" setting too. Absolute travesty. If a real life person did that I'd stop being friends with them, but somehow I'm paying money for this. Maybe I'm the clown here.

[staticman2]

I'm curious: did you give Gemini the entire text of Neuromancer or did you expect it to use search results for chapters 1 to 14?

I would have just fed it the text of chapters 1 to 14 from a non drm copy.

[gambiting]

I just asked like I said, give me plot summary until chapter 14, don't spoil the rest of the book. And of course when I told it what it just did it was like oh I'm sorry, here's a summary without the spoilers for the ending. So clearly it could do it without additional context.

[calf]

I wouldn't expect any LLM to be able to respect such a request. Do they even have direct access to published works to use as reference material?

Also, last time I played 20 questions with ChatGPT, it needed 97 turns and tons of my active hinting to get the answer.

[gambiting]

>>Do they even have direct access to published works to use as reference material?

I mean, clearly, given that it did answer my question eventually. Also wasn't it a whole thing that these models got trained on entire book libraries(without necessarily paying for that).

>>I wouldn't expect any LLM to be able to respect such a request

Why though? They seem to know everything about everything, why not this specifically. You can ask it to tell you the plot of pretty much any book/film/game made in the last 100 years and it will tell you. Maybe asking about specific chapters was too much, but Neuromancer exists in free copies all over the internet and it's been discussed to death, if it was a book that came out last year then ok, fair enough, but LLMs had 40 years of discussions about Neuromancer to train on.

But besides, regardless of everything else - if I say "don't spoil the rest of the book" and your response includes "in the last chapter character X dies" then you just failed at basic comprehension? Whether an LLM has any knowledge of the book or not, whether that is even true or not, that should be an unacceptable outcome.

[calf]

Why though? They seem to know everything about everything, why not this specifically.

The problem with this line of reasoning is that it is unscientific. "They seem to" is not good enough for an operational understanding of how LLMs work. The whole point of training is to forget details in order to form general capability, so it is not surprising if they forget things about books if the system deemed other properties as more important to remember.

[staticman2]

I wouldn't expect an AI to know exactly what happens in every chapter of a book.

Knowing the plot of Neuromancer isn't the same as being able to recite a chapter by chapter summary.

I tried this Neuromancer query a few times and results greatly vary with each regeneration but "do not include spoilers" seems to make Gemuni give more spoilers, not less.

[dominotw]

how is gemini winning in general llm. what is general llm .

[SwellJoe]

General LLM is what Apple is paying Google for.

[tartoran]

I noticed that Apple speech to text has gotten pretty good lately. Is that because they’re paying Google? Not sure I use other AI features from Apple as I have my Siri turned off.

[laserlight]

> Is that because they’re paying Google?

No, the Google deal hasn't shipped yet.