Ok that's messed up. If i tap it numerous times in succession it stops toggling, every now and then briefly toggling on. Holding it down at any point in the many-tap sequence activates it as you expect in the first place.
I genuinely don't see any perceivable delay on a Macbook without any third party keyboard customization software installed. Typing caps lock and the letter "a" in quick succession results in an "A" for me.