During Apollo the federal budget was also many times smaller. In real dollars NASA's budget is only about 20% smaller than its average in the 60's. The difference with respect to human space travel is risk aversion and lack of focus and political will.