[lhl]

I'm still on the waitlist for GPT-4 API access. Note, that text-davinci-003 cost about $90 to benchmark at $0.02/1K tokens, so if you're able to use a GPT-4 model (for completion and not just instruction) that'll probably be $270-$540 in credits to benchmark...

[hhh]

I have GPT-4 8k access and am willing to run the evals if someone wants to pay. Email in my acc info (the character is h)

Just a note, I get errors semi-frequently when running queries against GPT-4 often (timeouts mostly…) so any code would need to handle that well.

[MacsHeadroom]

You should benchmark GPT-3 Curie (7B) for comparison since it is the same size as llama-7B and StableLM-7B.

That will give us some indication of how much better these models are than GPT-3 at the same size.

[jimsimmons]

Just think about benchmarking 32K GPT4 haha