[rocqua]

The compiler tends to be limited in how it can change calling conventions by external visibility of the functions. Generally if you compile a function down to an object file the compiler will want to make that object file linkable with any other object files importing that symbol properly.

Whole program optimization gives the compiler some ways around that. I am not sure how much freedom it gives the compiler.

[account42]

With LTO the compiler should be free to fudge the calling convention for most calls even between translation units.

Here is GCC doing that optimization for a static noinline function: https://godbolt.org/z/cn6Wz9Kvn

Similarly, compilers can also clone functions if it makes sense to propagate constants from call sites into the function. Example: https://godbolt.org/z/59z6xT75n

I'm sure there is more room for improvement. A perfect compiler would always optimized programs as a whole and only regard function boundaries as hints at best. In practice, you have to keep complexity in check somehow.

[jcelerier]

On Linux with GCC and Clang you can use -fvisibility=internal to tell the compiler to not care at all about this and go wild with ABI. Of course it needs to be done carefully...

[gpderetta]

Compilers can clone functions (so there are two variants with different calling conventions) or even create alternate entry points.