[antiraza]

I asked Claude to help me research the fictional world of an established IP, and it refused, saying it was a fact-based tool and wasn't made to work with fictional content.

¯ \ _ ( ツ ) _ / ¯

An established IP's world is fact from my point of view. I got very little traction there compared to ChatGPT which had a no friction response that I could use to build upon.

[ek750]

Do we need some way to grade these services based on vertical or use-case?

I actually tried the same tech questions to multiple services when I first started playing around with these commercial LLMs. I would copy and paste the same question to GPT4, MS Bing (I soon stopped using that since I already have a sub to gpt4), claude, bard, and recently You (https://you.com) and while Claude.ai was rarely as good as GPT4, it wasn't too far off for tech questions.

I'm not very creative, so maybe the use of it helping with writing fiction or roleplay might help me, I haven't tried it yet.

Did you try Claude with non-fictional tasks, and if so, how does that compare to GPT4?

[antiraza]

I did not try Claude for a research based task based on non-fictional content.

I think it's good that LLMs becomes specialized tools that can go deep into their expertise, I just think 'a fact engine' -- if that's what Claude is aiming to be -- needs to have correctly rigid controls on what defines fact. From that POV, I think I agree with the 'over-censored' label for Claude earlier in the thread... The intention may not be censorship, but if the LLM is so gunshy about what is fact vs. not, it's going to have a really narrow (and therefore potentially unreliable) lens.