[_delirium]

I don't think we have a lot of data on these kinds of large-scale transitions, even though we have a lot of data in nominal terms. That's a common problem in machine learning and data mining as well (my area of research), where a large amount of nominal data can be misleading if what you're really trying to model isn't that common.

In this case the relevant event is sector-wide shifts in employment, of the farms->factories variety. We have very little data on those, because they occur once or perhaps twice in our data set. So we can't really generalize with any confidence about them: which features of that particular transition are general features of a sector-to-sector transition, and which ones depend on idiosyncracies of farms or factories or the particular time? Will factories->X look the same as farms->factories? We do have somewhat better data on which depend on idiosyncracies of place, since you can look at the industrial revolution in the U.S. versus in the UK or Germany or Japan (though those aren't independent data sets).

We have a lot of data in raw terms, but I think of that as just having data on a small number of macro-scale events, but at high resolution (monthly or better for many years). Now if we had observed 4 or 5 such macro-scale shifts (at least), that's the kind of data that would be needed to build a good model of how such transitions work. So I don't think we can say with much confidence, certainly not statistically valid confidence, how market mechanisms handle such shifts. What we have very good data on is how market mechanisms work in the "normal" case, within a largely stable paradigm. But for farm->factory type transitions, we have basically small-N case studies that people are trying to extrapolate from. The "market will sort it out—it always does" view is certainly one possible extrapolation, but I don't think the evidence confirming it is very strong.

[Variance]

>if what you're really trying to model isn't that common.

Technology increases are quite common. Again, look at that GDP per capita chart I linked; the logarithmic growth of technology as measured by GDP per capita benefit has been both continual, and, importantly, very steady. It's surprisingly steady, all the way up to the end of the 20th century. To echo some of the articles that have been posted on HN recently, we are in a time of unprecedented growth and social change; but we always have been. It's not like something is magically different this time around just because the HN demographic is the one participating in it, and everyone is making smartphone apps instead of web 2.0 pages, or laying intercontinental fiber before that, and so on. Technological growth is gradual and smooth in aggregate, even if individual markets can be disrupted more noticeably, which is one of the prime reasons why the job market is able to keep unemployment as low as it has been for so long, and will continue to do so in the future.

The "data set" you're thinking of seems to be something along the lines of one data point being "the industrial revolution happened, and long-term unemployment levels didn't rise." But that's not what I'm talking about. I'm talking about how we know that technology has been increasing for hundreds of years, as measured by things like the GDP per capita chart I linked, something including hundreds and thousands of data points. That all comes down to a single check: despite hundreds of years of technological growth, which is growth just like we are experiencing now and will be in the future, is unemployment higher? No. Assembly lines were a paradigm shift, cars were a paradigm shift, the modern western office environment was a paradigm shift, computers were a paradigm shift, and the internet was a paradigm shift. The entire country was always unprepared in terms of skills to capitalize on those shifts, but the unemployment rate stayed low. Apparently, if people want and need jobs, jobs will come into existence. The law of supply and demand.

I really want to stress that there's nothing substantially different in terms of innovation currently, contrary to your argument. If nanotechnology makes physical manufacturing of products functionally costless, or if the singularity makes all decision-making and programming jobs irrelevant, then sure. But social media? Smartphones and tablets? Hardly. The skills required to do these things have existed for decades; the underlying technologies are very iterative, as technological progress usually is.

Technology levels can be measured in terms of GDP per capita, which can be followed back hundreds of years. Same with unemployment rates. But in the end, is the unemployment rate 50% or 75% right now because half of the workforce is simply unneeded or unskilled, or unable? No. Non-recessionary unemployment baselines are still around 5%. To say that this particular, incremental paradigm shift that we're currently experiencing of increasing virtualization of our lives is any different is to be mistaken into thinking that our current era is a special snowflake apart from all the others before it; and that your judgement of technological change being too much for the job market to handle is different than that of Thomas Malthus's two centuries earlier, or all those thousands of voices in between.

I want to note that I'm very optimistic about the future of technology. I wouldn't be surprised to find that the human era will be over by the end of the century. But I also recognize that Malthusian-spectrum arguments never pan out, no matter how unique this current era is supposed to be in comparison to all the previous unprecedentedly unique eras.

edit: and can I add that I would be quite happy to see some technological change so rapid and sudden that it would put something like 5-10% of our workers out of a job pseudo-permanently. That would mean that some tremendous benefit to the economy has suddenly descended from the heavens of innovation. But those kinds of things just don't happen in eras like our own--unemployment shocks are instead currently temporary (usually being fixed in less than a decade) and due to very non-technology reasons, like the housing bubble and credit crunch this past decade.

[prodigal_erik]

People survived the industrial revolution, but horses largely didn't, because for the first time ever we had machines that could replace them in nearly all of their niches. People have never been threatened in this way, even as particular industries disappeared there was mostly enough menial work to retreat to that required a little too much perception and decision-making to automate. What do they do when all menial work can be done by some machine that doesn't eat or sleep or complain? We're finally starting to raise the bar on how clever and creative you must be to be relevant on the job market, and a growing number of people are inevitably going to fall below it.

[Variance]

That's a flagrantly false analogy. But see above--it's not time for humans to be outmoded quite yet. And when it is, it won't just be low-skill jobs that disappear.

[prodigal_erik]

Why not? Jobs don't require uniform levels of creativity, and we aren't waiting for a singularity before we start. We already have GPS-guided tractors replacing farmhands and self-checkout registers starting to replace retail clerks, while white-collar fields like engineering and software have merely shed some rote work (drafting and data entry) and medicine and law have changed even less.

I think the jobs most likely to disappear are terrible wastes of human minds; I just wish we weren't so vicious towards people we can't find a need for right now.

[Variance]

This was already answered in my big comment. There is no trend towards any unemployment at all; there never has been any unemployment associated with tech growth; tech growth is smoother and more iterative than people are conceptualizing, so it's not like there's any technical reason that this would change; and the current tech is no revolutionary exception to the trends of tech growth we've had in the past. Basically, you assume that a certain level of tech growth implies that people who can't keep up with it will be out of a job, but we empirically observe that there is no reason to suspect this. The sorts of events that could break these patterns are singularities, not mobile apps.