[nox101]

I have tried the same, and had ChatGPT utterly fail.

> me: what's the relatively recent movie with a pig on the movie poster

> gpt: The movie you're referring to is likely "Pig," a 2021 American drama film. It stars Nicolas Cage as a truffle hunter who lives alone in the Oregon wilderness. He embarks on a quest to find his beloved foraging pig after she is kidnapped. The film was directed by Michael Sarnoski .

> me: no, that movie does not have a pig on the poster

https://en.wikipedia.org/wiki/Pig_(2021_film)#/media/File:Pi...

> gpt: My mistake, let's correct that. Given your description, another film that fits the criteria of having a pig on its movie poster and is relatively recent is "The Menu" (2022). This film features a darkly comedic and thrilling story set against the backdrop of an exclusive restaurant on a remote island, where a young couple experiences a very unusual dining experience. The pig on the poster is part of the film's intriguing promotional material, hinting at the film's critique of culinary culture and excess.

https://en.wikipedia.org/wiki/The_Menu_(2022_film)#/media/Fi...

> me: No, that move does not have a pig poster

> gpt: It seems I misunderstood your initial request. After a more detailed search, I couldn't find a recent movie specifically featuring a pig prominently on its movie poster beyond the titles we've discussed. Could there be any other details about the movie you're thinking of that might help narrow it down? For instance, any specific elements of the plot, actors involved, or even the genre might provide more clues to identify the correct film.

movie I was looking for was "The Hunt": https://en.wikipedia.org/wiki/The_Hunt_%282020_film%29#/medi...

[jvanderbot]

Inb4 we all tell you you're using it wrong. There are certainly better ways to do this, but this is really the major thing that drives me nuts. No matter how many times I tell some LLM I want it to ask clarifying questions to provide better answers, it just won't. You end up doing exactly this, guessing what information might trigger the right recall.

[arcanemachiner]

I've found that there are often different trigger phrases I can use to get an LLM to "change its mind".

For Llama, I can just say "Are you sure?" and it will change its tune (unless it's quite "certain" about the results).

Qwen is more insistent, but will change course if I say "I looked it up and it says you're wrong".

[Avicebron]

if we formalize what might trigger the recall we could keep a mental model of it, maybe call it bangs, something like !wikipedia would only index data sourced from training on wikipedia..you might be on to something..

[BoorishBears]

> No matter how many times I tell some LLM I want it to ask clarifying questions to provide better answers, it just won't

You're telling me there's a modern frontier model that refused to ask clarifying questions after you told it to?

[jvanderbot]

I can't tell if you're being sarcastic, but yes, when it asks questions they are trivial disambiguation of mainly language, and never information seeking like you might expect when doing more than superficial investigation

[BoorishBears]

If you can't tell if I'm being sarcastic about an LLM asking clarifying questions, you are possibly not great at using LLMs.

Prompting isn't necessarily the career some people wanted was sold as, but it's not a bad idea to practice a bit and build a sense of what a clear and effective prompt looks like.

-

To be clear, I get telling people it's a "you" problem every time an issue with LLMs comes up isn't helpful... but sometimes the disconnect between someone's claimed experience, and what little most people actually can agree LLMs are capable of is so great that it must be PEBCAK.

I just tried the original checkpoint of GPT 3.5 Turbo and it was able to handle drilling up and down in specificity as appropriate with the prompt: "I need you to help me remember a movie. Ask clarifying questions as we go along."

[BehindBlueEyes]

I think it depends a lot on what llm and tooling you have access to.

I had great results prototyping agents at work for specific tasks and answering in a specific style including asking appropriate clarification questions.

But at home with just free/local options? I've nowhere near the same settings to play with and only had very mixed results. I couldn't get most models to follow simple instructions at all.

[jvanderbot]

Nobody has more PEBKAC problems than me. It might be my experience is colored by older models. I can give it another shot. But I do use LLMs quite a lot and got decent and surprising functionality out of them in the past. I was just consistently vexed by this one thing when doing information seeking search -like activity.

[BoorishBears]

The recent web search integration actually made ChatGPT much worse at this specific kind of search (I want a piece of media, but I only remember characteristics of it)

If you explicitly tell it not to search, it's less likely to anchor itself in trying to search the web and going down rabbit holes.

It also more likely to pause at times and ask the right questions to nudge.

[glenstein]

Agreed, it has been much more valuable to me for it to rely on it's training than it's web searches.

[JKCalhoun]

Ha ha, you should see what Google returns when you ask it for the recent movie with a pig on the movie poster.

(Hint: it is a film from 2021 starring Nicolas Cage.)