Can't you use Captcha services? If the big tech captcha services are too costly, one could create a rudimentary captcha defense against these bots pretty easily. Am I missing something?
If you throw a captcha in actual users' faces they'll most likely just leave for a competitor that doesn't harass them with busywork, and you'll be left with only bots patiently trying to get past it.
Oh, I've tried captchas, and I can say that they're an awful experience for the humans you accidentally hit, at least for a service like mine that is relatively low value per session (a dictionary website). Within minutes of changing my WAF configuration to captcha users that it thinks are high-probability bots, I'll get angry feedback from users in Singapore (say) who don't want to have to solve a puzzle in order to look up a word in the dictionary. I don't blame them.
I like the Cloudflare challenge ideas suggested on this thread, though, I might try them again.