If you fix the position of the 'thrower' and just throw them at ridiculous speeds so gravity doesn't matter you'd end up with a hyperbolic backboard.
So annoyingly for an 'optimal' solution you'd need to specify where people can throw from and how fast. Frankly you might as well just use a hyperbola with one foci on the hoop and the other on the middle of the court, or maybe a point slightly higher than the court itself as the balls will be coming in at a lower angle (or maybe even a downwards angle?).
I would guess that if there is enough spin for the Magnus effect to be significant, it would also be significant with respect to the rebound, as the surfaces in contact are not frictionless. On the other hand, I am not sure there is much spin in most basketball shots, and, given a backboard designed without taking it into consideration, there would be no incentive to add spin.
I do not think you can make definite claims about optimality without putting it in the context of some probability distribution over all possible incoming trajectories etc.
I was wondering how you could get an analytical solution, but I think you’d have to define the problem much more rigorously to do so. As he mentions in the video, different basketball paths can contact the same point on the backboard from any number of angles, so it seems fairly clear that there is no perfect solution.
So annoyingly for an 'optimal' solution you'd need to specify where people can throw from and how fast. Frankly you might as well just use a hyperbola with one foci on the hoop and the other on the middle of the court, or maybe a point slightly higher than the court itself as the balls will be coming in at a lower angle (or maybe even a downwards angle?).