[putlake]

Thanks for sharing the link. It is a testament to Google's failure marketing its cloud offerings that, despite wanting to try out this API, I had been unable to find it until you helpfully shared the link.

I went down the cloud console rabbit hole to play with it. Only the bison model was available (of the non-descriptively named gecko, otter, bison, unicorn models). For innocuous prompts I got this error: "The response is blocked because the input or response potentially violates our policies. Try rephrasing the prompt or adjusting the parameter settings. Currently, only English is supported."

Where it did respond, I liked the quality. It was comparable to the GPT-3.5 API. Perhaps a little better even.

[sanxiyn]

From personal experience, Bard's quality seems to be between GPT-3.5 and GPT-4, closer to GPT-4 if you have to bet. Except when fresh or live data matters, where Bard is clearly superior. (Bard's training data is up to Feb 2023 compared to ChatGPT's Sep 2021, and Bard also gets live data from Google search.)

MMLU benchmark score agrees with this estimate: GPT-3.5 (70.0%), PaLM 2 (81.2%), GPT-4 (86.4%).

[jerrygenser]

Does this take into account failure to consider context? I have a lot of trouble with bard not using context from previous messages.

[sanxiyn]

Bard's context length is equal to that of GPT-3.5.