[jampekka]

If by stochastic methods you mean something like MCMC, they are increasing in popularity, but still used a lot less than analytical or numerical methods. And almost exclusively only for more complicated models than basic linear regression. Sampling methods have major downsides, and approximation methods like ADVI are becoming more popular. Though sampling vs approximations is a bit off topic, as neither usually have closed form solutions.

Even the most popular more complicted models like multilevel (linear) regression make use of the mathematical convenience of the square error, even though the solutions aren't fully analytical.

Square error indeed gives estimates for normally distributed noise, but as I said, this assumption is quite often implicit, and not even really well understood by many practitioners.

Analytical solutions for squared errors have a long history for more or less all fields using regression and related models, and there's a lot of inertia for them. E.g. ANOVA is still the default method (although being replaced by multilevel regression) for many fields. This history is mainly due to the analytical convenience as they were computed on paper. That doesn't mean the normality assumption is not often justifiable. And when not directly, the traditional solution is to transform the variables to get (approximately) normally distributed ones for analytical solutions.

[xadhominemx]

It’s not because of analytical convenience, it’s because of the central limit theorem.

[jampekka]

Not everything is a linear combination of large number of (IID) samples, and thus not everything is gaussian distributed.

[kgwgk]

You’re implying that many things are though.

[jampekka]

Yes, and I was explicit about it in another comment in this post.

[kgwgk]

Ok, so we all agree that in most cases the reason to minimize square error is that it’s the appropriate thing to minimize - not that it has an analytical solution, convenience or tradition.