Oh, that goes back a long ways, and goes across cultures too. It's a human thing, not an "American" thing. Maybe you can find some culture somewhere in the world that unironically hangs signs that idiomatically translate to "The Shitters ->", but it'll be the exception rather than the rule.
I don't think anyone really associate the the "rest" in "restroom" with resting. May as well call it infantile to call it a "bathroom" when only children take baths.
Right, I resist calling it a bathroom in places where it won't actually have a bath. Guess I should start asking at restaurants where the water closet is?