Not necessarily. Black locust (Robinia psuedoacacia), e.g., is one of the faster growing hardwoods in North America, but produces very hard, dense, and strong wood.
Or maybe not quite that hard. Some woods are already hard enough to be inconvenient to work. Though maybe it's still all benefit if you're just trying to make industrial feedstock for cross laminated timber or something.