[nchagnet]

Regarding the "why the action is this object" part of the question, I find that the easiest way to think about it is from the Hamiltonian perspective. There you can think of it as minimising energy along a trajectory. From that point, a Lagrangian is just a mathematical trick to express the symplectic structure differently.

But if your question was more about "why minimizing something yields trajectories", I personally would argue this is beyond physics. As an empirical science, physicists have seen this kind of behaviour broadly (optics, classical mechanics, quantum mechanics) and just unified it as an overarching principle.

Finally regarding the proof to newtonian mechanics, I don't have anything handy from the pure Newtonian perspective beyond the usual "minimises the lagrangian and your equations of motions look the same". However, you might be interested in proofs which show newtonian gravity as low energy approximation of general relativity. And since general relativity has a nice action formulation, it all gets nicely tied in.

Hope this helps!

[in_a_hole]

But simply getting to the Lagrangian picture from the Hamiltonian picture would just leave me wondering why the Hamiltonian picture works!

My motivation for getting to the bottom of all this is to fill the gaps in my physics understanding at least up to quantum mechanics. I have a grasp of QM but I would like to have some insight into the conceptual leaps that brought us there from classical mechanics. QM works in the Hamiltonian picture and I recall from my undergrad days that you get there from a Legendre transformation on the Lagrangian (or something to that effect) so I'm trying to understand the justification of that approach before moving up the conceptual ladder.

Ideally I would like to be able to trace my way from simple postulates based on observation of the physical world all the way to QM, then maybe to QFT after that.

[Cleonis]

About transitioning from Classical Mechanics to QM, guided by observations.

There is a very interesting approach in the quantum physics book by Eisberg and Resnick, section 5.2

To arrive at the Schrödinger equation Eisberg and Resnick construct what they refer to as a plausibility argument.

The goal: to arrive at a wave equation that when solved for the Hydrogen atom will have the electron orbitals as set of solutions.

Eisberg and Resnick state 4 demands:

-1. Must be consistent with the de Broglie/Einstein postulates. wavelength=h/p, frequency=E/h

-2. Must be such that for a quantum entity followed over time the sum of potential energy and kinetic energy is a conserved quantity.

-3. Must be such that the equation is linear in \Psi(x,t): any linear combination of two solutions \Psi_1 and \Psi_2 must also be a solution of the equation. (Motivation: in experiments electron diffraction effects are observed. Interference effects can occur only if wave functions can be _added_.)

-4. In the absence of a potential gradient the equation must have as a solution a propagating sinusoidal wave of constant wavelength and frequency.

Eisberg and Resnick proceed to show that the above 4 demands narrow down the possibilities such that arriving at the Schrödinger equation is made inevitable.

To me the second demand is particularly interesting. The second demand is equivalent to demanding that the work-energy theorem holds good. The recurring theme: the work-energy theorem.

I have a (html)-transcript of the Eisberg & Resnick treatment that I can make available to you.

There is a youtube video with a presentation that is based on the Eisberg & Resnick plausibility argument.

https://youtu.be/2WPA1L9uJqo

In that video the presentation of the plausibility argument is in the first 18 minutes, the rest of the video is about application of the Schrödinger equation.

[aeonik]

You know about the Ultraviolet catastrophe?

Physics Explained Ultraviolet Catastrophe: https://youtu.be/rCfPQLVzus4

This Veritassium Video goes back further in time and talks about Action, and the Genesis of this idea. https://youtu.be/qJZ1Ez28C-A

Chemistorian, The history of Atomic Theory. https://youtu.be/SqYPrA7upiE

[nchagnet]

"QM works in the Hamiltonian picture and I recall from my undergrad days that you get there from a Legendre transformation on the Lagrangian (or something to that effect) so I'm trying to understand the justification of that approach before moving up the conceptual ladder."

That is only one approach to QM. As you rightly point out, Hamiltonian and Lagrangian approaches are always two sides of the same coin: one is only the Legendre transformation of the other and so they describe the same physics.

So to that end there is a neat QM Lagrangian representation: the path integral formulation. You can apply it to basic QM as well as to QFT or even QFT in curved spacetimes and string theory.

So if your goal is to "trace your way from simple postulates", that is a good way: assume your system can be described by an action, and go from there. It works in pretty much every scenario. In most research I've been involved, you always end up constructing generic actions whose coefficients eventually determine the behaviour of the theory.

And to get back to what I assume is your real conundrum (why do we extremize something to begin with), I just don't think there's any true answer as to why nature behaves this way.

What we can answer is: what is the action, what does it represent, what does it mean to extremize it? The short answer to this (provided by the path integral formulation I mentioned earlier) is that the action is essentially controlling a probability distribution of paths in a given geometry. In quantum mechanics, we interpret this distribution with that of actual particles. When you extremize the distribution, you essentially find the most likely trajectory, and if your distribution is peaked enough around that trajectory, then you can take this path as representative of your system when fluctuations are ignored.

So in the classical limit of QM, that trajectory is all that's left (and that would be the classical mechanics trajectory).

Interestingly a similar interpretation exists in statistical physics. If you "complexify" your time dimension, your action is again on a Euclidean (instead of lorentzian) spacetime and the time direction behaves like a circle whose radius sets a scale akin to a temperature. This might sound a bit complex but where I'm going with this is that once again, you can think of this euclidean path integral as a distribution of paths over fluctuations (this time thermal), and the extremum is this time the system's behaviour when at equilibrium.

[in_a_hole]

> And to get back to what I assume is your real conundrum (why do we extremize something to begin with), I just don't think there's any true answer as to why nature behaves this way.

Having read about some of the history of this idea, it seems to have been originally built on philosophical grounds, the idea that nature chooses the most harmonious path, as opposed to Newton's laws which seem to come from intuition based on observation of the world. If you keep asking "why?" in either framework you will eventually run up against an epistomological barrier which is unlikely to ever be crossed but in the case of Newton's laws, their basis in physical intuition makes them much easier (for me at least) to accept as given and take as a starting point for constructing a world model. With this being the case I think an acceptable result for me would be to find a proof of equivalence between the Newtonian and Lagrangian pictures. From my reading it seems like the derivation from D'Alambert's principle may be part of the journey.

[Cleonis]

About d'Alembert's principle. A modern name for it is 'd'Alembert's virtual work'.

The modern concept of 'work done' was formulated around 1850 (Eighteen-fifty). That is, we shouldn't assume that back in the days of Lagrange d'Alembert's principle was understood in the same way as it is today.

Joseph Louis Lagrange motivated his notion of potential energy in terms of d'Alembert's principle.

The recurring theme is the concept of 'work done'.

In case you hadn't noticed yet, I'm the contributor who notified you of a resource I created, with interactive diagrams.

There is this distinction: the work-energy theorem expresses physical motion, whereas d'Alembert's virtual work expresses, as the modern name indicates, virtual work.

My assessment is that using d'Alembert's virtual work is an unnecesarily elaborate approach. The same result can be arrived at in a more direct way.

[in_a_hole]

I haven't had a chance to really dig into your resource yet but I am definitely going to do so. Perhaps I'll wait a few days until the change you mentioned is implemented.