I rather like having access to the "quaint farm stand" a couple miles down the road from me in the summer. Not everything needs to be about the quest for efficiency.
Let's suppose that these people are not, in fact, making a good faith effort to make their land agriculturally productive, and they shouldn't get the agriculture exemption.
Ok, fine. They don't get the benefit, and they pay more in taxes...or they sell, the houses are demolished, and the land is put to more "productive use" (i.e. sprawl/suburbs). The county or the city or what have you gains more in property taxes, and the evil rich people are no longer getting away with not paying more taxes.
But now instead of uncultivated land (remember, it was a fake farm), you have concrete, asphalt, hardy board, one or two species of grass, crepe myrtles...you get the picture.
If anything, there should be a national wildlife sanctuary tax exemption program, so that this is actually codified.