I can't speak for most places, but New Hampshire is almost all new-growth forest. If you roam the woods you'll find miles of ancient stone walls demarking the edges of long-gone farms. Of course, there's not much LAND in New Hampshire so it definitely can't take on California's role (even disregarding all the other problems), but I wanted to give an example of abandoned farmland.
How competitive is NH with a region like California weather-wise? I would imagine that the growing season is much shorter in NH, making it way less attractive. Farming in New England is either specific to the climate or exists mainly because that's where many Europeans first colonized America. Just because farming made economic sense there 50-100 years ago, doesn't mean that it makes economic sense there today.
My understanding is that the Northeast was primarily deforested for sheep farming. The stone walls delineate old paddocks for rotated grazing.
Most of the ground is really quite marginal; thin soil, rocky as sin, etc etc. It's possible it'd would be economically viable as pasture-land; it wouldn't surprise me if the only The West won as pasture land was that it was all essentially free out there. Maybe if the gov't charged competitive rates for grazing on public land (and was actually able to collect) you'd see a resurgence of Eastern pasture-land.