If you aren't a permanent resident you are almost always tied to your current job, and can technically be required to leave within two weeks if you are laid off or fired. (Getting a new job that would accommodate your visa requirements on a two-week notice might not be trivial.) If you are a permanent resident, you are subject to many of the annoyances discussed (i.e., IRS) without the benefits of citizenship...
He was talking about not living in the USA specifically, so your not tied to your job like that, and you don't have any US tax requirements unless the income comes from the USA.