While this tool doesn't work with Windows11 it's pretty useful if you're still on Windows 10 (or earlier). Another tool I wholeheartedly recommend is the 7+ Taskbar Numberer [0]. This should be default functionality built into windows but I'm happy that someone took the time to build it.
The solution there is to use the existing W10 taskbar that wasn’t removed, we can safely assume they’ll remove the W10 taskbar at some point and then the 2nd half of that blog post is relevant.
[0] https://ramensoftware.com/7-taskbar-numberer