[giantg2]

"he said that I’m pretty SOL in arguing this loss amount."

That's how I see it. The government has obscene resources and power to prosecute you. Even if you win, you'll likely be screwed with the cost to defend. The law generally favors the victim and in many cases judges seem to accept any amount that can explained, even if it's not fair.