There are a limited number of scancodes; defined by the USB standard. The annoying thing for keyboards is that the OS controls what scancode maps to what character and that makes it impossible to have certain keys.
For example, you can't have an ! key. That is always Shift+1 (with a US layout, that is). QMK and other firmwares will let you make an ! key, but it's implemented by pressing shift, then pressing 1. So you could never make a keyboard shortcut that is Shift+! even though you can press that on your keyboard. The OS wouldn't be able to tell the difference between than and Shift+1. As a corollary, you couldn't rebind your keyboard such that Shift+1 outputs @ instead of !.
It's really just a classic case of bad abstraction. All the keyboard hardware does is scan rows and see which columns are also active. That gives you a grid position like (1,2) for "w". It is then forced to translate that to a scancode that it sends to the computer. The computer then sees that scancode and translates it to a letter. Obviously, things would be easier to program if the keyboard just emitted characters, or the OS just read key matrix grid positions. (In the first case, your X key would always be X, no matter what. Annoying for laptop users that want to use Dvorak, of course. In the second case, things like QMK wouldn't need to exist, you could just write a normal program running on your computer to add layers, shifting, tap dances, etc.)
The end result is that everything sucks. Isn't it always?
I know that was probably just meant as an illustration, but you can have an ‘!’ key — it's 07:00CF “Keypad !”¹.
Linux will ignore it² though, because being Linux they had to NIH their own key codes, and Windows will ignore it³ because it wasn't on the IBM PC keyboard in 1981.
Stepping back, why would you actually want the keyboard firmware to have configurable settings and runtime state? xkbcomp (Linux/Xorg) is buggy as hell, but I'd say it's still fundamentally the right abstraction to do this on the host. Configuring a layout on the keyboard is itself the hack, really only encouraged by the recent innovation happening there.
It's been a hack for years, not merely recently. Maltron keyboards used to switch layouts between the "PC" and "Maltron" layouts by the keyboard moving scancodes around, in response to a physical switch at the back of the keyboard, back in the days of PS/2 keyboards.
I've thought about this a lot, because I've bumped up against that limitation a few times when customizing my Erogodox layout.
My conclusion: QMK could do this, or at least some firmware could do this. It's hard to come up with an eloquent way to express it, however.
But there's no a priori reason that pressing shift-1 couldn't be special-cased to instead send shift-2, yielding @. It's all being precomposed by the firmware, after all.
It's too bad, because I have a custom key for delete-back-word, and I want shift-thatkey to send delete-forward-word, but as you point out, I can't. But that's a limitation of QMK, not a fact of nature.
You can write code to do that in your process_record_user hook, but it's probably going to be flaky. I would just use a modifier other than shift; people tend to call them "raise" and "lower" and just change the layer. That way everything is in control of QMK.
The disadvantage of using layers is that you can't press the layer-activate key on one keyboard and then press your key on another keyboard. With shift (and OS-level modifiers) you could technically do that. But it's a weird case that probably doesn't come up very often.
For example, you can't have an ! key. That is always Shift+1 (with a US layout, that is). QMK and other firmwares will let you make an ! key, but it's implemented by pressing shift, then pressing 1. So you could never make a keyboard shortcut that is Shift+! even though you can press that on your keyboard. The OS wouldn't be able to tell the difference between than and Shift+1. As a corollary, you couldn't rebind your keyboard such that Shift+1 outputs @ instead of !.
It's really just a classic case of bad abstraction. All the keyboard hardware does is scan rows and see which columns are also active. That gives you a grid position like (1,2) for "w". It is then forced to translate that to a scancode that it sends to the computer. The computer then sees that scancode and translates it to a letter. Obviously, things would be easier to program if the keyboard just emitted characters, or the OS just read key matrix grid positions. (In the first case, your X key would always be X, no matter what. Annoying for laptop users that want to use Dvorak, of course. In the second case, things like QMK wouldn't need to exist, you could just write a normal program running on your computer to add layers, shifting, tap dances, etc.)
The end result is that everything sucks. Isn't it always?