Guess: Intel changed the spec. There's quite a few generations between a 286 and a P4, and new BIOS code doesn't need to run on discontinued CPU types. And new execution contexts like https://en.wikipedia.org/wiki/System_Management_Mode might benefit from minimizing the setup needed to run in protected mode.