Technically you don't go to prison. You go to prison for lying about what you owe or cheating. But if you don't pay you generally settle with the IRS and get put on a payment plan or something similar. I know people who have been through this process.
My list of big demand drivers isn’t exhaustive and applies more to the current system than to the gold standard system of the past you’re referring to. Under fiat, the top marginal tax rate was over 80-90% for much of the time.