[contravariant]

If you fix the position of the 'thrower' and just throw them at ridiculous speeds so gravity doesn't matter you'd end up with a hyperbolic backboard.

So annoyingly for an 'optimal' solution you'd need to specify where people can throw from and how fast. Frankly you might as well just use a hyperbola with one foci on the hoop and the other on the middle of the court, or maybe a point slightly higher than the court itself as the balls will be coming in at a lower angle (or maybe even a downwards angle?).

[CrazyStat]

Wouldn't you rather want an ellipsoid with the thrower at one focus and the hoop at the other?

[contravariant]

I may have gotten things the wrong way around. You definitely want whatever conic has the foci in the right spots.

[shawnz]

Don't forget about spin as well, which can significantly change the trajectory due to the Magnus effect

[mannykannot]

I would guess that if there is enough spin for the Magnus effect to be significant, it would also be significant with respect to the rebound, as the surfaces in contact are not frictionless. On the other hand, I am not sure there is much spin in most basketball shots, and, given a backboard designed without taking it into consideration, there would be no incentive to add spin.

[tomjakubowski]

Backspin is an essential part of a reliable jump shot, and topspin is employed on finger roll layups close to the rim.

[mturmon]

Better to account for gravity first, no?