Exactly - you want timeouts, exponential backoff, etc. You want to know what happens if a network failure happens - you don't want to retry ad nauseam.
Some of the most reliable programs are those without exponential backoff and which retry infinitely in network failure situations. `ping` is one example.
And yet I'd be completely fine with a ping that was capped to running for no more than a week. YMMV though.
And more to the point, you don't want a program that goes on forever in a setup like ethereum. I guess your rebuttal was called for since the initial point was a little hyperbolic.