[btilly]

You are right that in the presence of sufficient data, everything converges to the normal and we're fine.

However the difference between the t-test and the normal to which it is an approximation is that the t-test takes into account corrections the rate of convergence to the estimated normal. The details of that convergence is dependent upon the distribution of individual samples, and the t-test is predicated on the assumption that each and every sample is from a normal distribution.