[jrumbut]

That is likely true, though there are some special, farm only ways to be employed such as the H-2A Visa of which there were 2.4 million in 2018 if I'm reading this right: https://www.ers.usda.gov/topics/farm-economy/farm-labor/#h2a.

My guess is the biggest reason that non-farm labor is tracked is because someone decades ago decided to do so and now if we want to do longitudinal comparisons we need to track the same thing.

It's an interesting enough statistic since there are areas that are very heavily agricultural and areas with very few farms and probably not as much mixing between the two compared to (for instance) manufacturing and clerical jobs which may happen in the same building.