[PaulHoule]

You're right.

That's why you build another threshold-type learner and then apply logistic regression to convert the score produced by learner A into a probability score.

Then you can tune at the exact point of the precision-recall curve that maximizes business value.