That's a function of your runtime dynamic linker, which is most likely provided by glibc (on linux). I believe it supports a.out still; at the very least, it used to.
Nope. It's a function of the kernel, which needs to know the binary format in order to load the executable. The dynamic linker is invoked at a later phase of execution, and in the case of the ELF file it does that by reading the dynamic linker's name from the PT_INTERP entry in the Dynamic table stored in the PT_DYNAMIC segment. By default, if the kernel doesn't recognize the ELF magic (first 4 bytes in the file are `\0x7fELF`), or can't load the dynamic linker specified in the PT_INTERP entry (eg. wrong architecture), it will launch /bin/sh as the interpreter and feed the loaded file to it assuming it's a text file containing a shell script.
If the kernel has the bin_fmt mod loaded, it can also recognize other binary formats before falling back to /bin/sh.
If the kernel has the bin_fmt mod loaded, it can also recognize other binary formats before falling back to /bin/sh.