This has been tried and tested a few times in industry, notably at Netflix and Duolingo for artwork and notifications
https://towardsdatascience.com/multi-armed-bandits-part-1-b8...