Most of the world's data analysis still happens on data sets that are at most a couple of thousand observations. It'd be neat if ggplot2 was faster for large datasets, but I can imagine that's not a priority.
The last time I measured the runtime, it was slow even for medium sized data sets -- in comparison to lattice or to R's "native" plot functions. It's probably not a problem for interactive use but it becomes annoying when generating reports with many visualizations.