[seibelj]

The solution is so blindingly obvious: You are creating AI not to replace humans, but to assist humans. If your humans can manage 50 emails per hour, you judge the quality of improvements to the AI on how any more emails per hour they can handle. This is from improving the tools humans use to completely eliminating the human from the equation in certain cases. Then your customers need to rate the quality of the service to see if automatically handled emails had the side-effect of reducing quality, which then needs to be weighed against the cost savings.

[exolymph]

Agreed. I even think that we'll get closer and closer to general AI the more we use the technology that's ready now to help humans.