Hard disagree. Economic growth is what raises people's standard of living. We're not going to eliminate poverty without continued growth. That doesn't mean it doesn't also cause problems, but those problems are solvable.
But we know that in the medium term, growth is limited to cubic; and in the long term to quadratic. Building a society that depends on a few centuries or millenia of exponential growth is designing for failure when that asymptote is reached, and all the assumptions about future growth are violated.
You're talking about centuries from now, when technology and culture could be very different. We might spread out through the solar system. It's not something to worry about now.