Not to mention that a lot of farming is done by undocumented workers, who don't necessarily appear on payrolls. It's hard to find people to do hard, dangerous, exhausting work for so little pay.
My guess is the biggest reason that non-farm labor is tracked is because someone decades ago decided to do so and now if we want to do longitudinal comparisons we need to track the same thing.
It's an interesting enough statistic since there are areas that are very heavily agricultural and areas with very few farms and probably not as much mixing between the two compared to (for instance) manufacturing and clerical jobs which may happen in the same building.