[skytrue]

I feel like people are forgetting that these models are non-deterministic. You can give it the exact same input and get a different response, so "testing" in this way doesn't work. That's also why (right now) automation is impossible. You aren't guaranteed an expected output.

[aix1]

I wonder what's the source of non-determinism here? (During inference, I mean.)