[jxy]

how odd! What exactly is lmsys using? Some hidden API that google give them so they can have a better ranking there?

[j-b]

Most likely through this platform: https://console.cloud.google.com/vertex-ai

[jxy]

Thanks. I managed to google and get two different API endpoints.

From the vertex ai:

    API_ENDPOINT="us-central1-aiplatform.googleapis.com"
    PROJECT_ID="test00"
    MODEL_ID="gemini-pro"
    LOCATION_ID="us-central1"
    
    curl \
    -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json" \
    "https://${API_ENDPOINT}/v1/projects/${PROJECT_ID}/locations/${LOCATION_ID}/publishers/google/models/${MODEL_ID}:streamGenerateContent" -d '@request.json'

and from the makersuite:

    curl \
      -X POST https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent?key=${API_KEY} \
      -H 'Content-Type: application/json' \
      -d '@request.json'

[j-b]

Created a simple app to test Gemini here:

https://github.com/dssjon/gemini/blob/main/app.py

[declaredapple]

> Some hidden API that google give them so they can have a better ranking there?

I don't know about that second part - but it would make sense that google (and others) may want to use lmsys's arena to benchmark their models.

After all, Human A/B tests are far better then the current automated benchmarks.

I would like more info from lmsys as to how they're accessing these though.