Your right. When you consider it there has been a few million years of iteration of the human ear to attune it to the sound of the human voice. A difficult problem to overcome to say the least.
It's not just that, it has to do with the way noise cancellation circuits work. They work great on constant white-noise-like hums within a certain frequency band; they don't work on transients that cross a wide range of frequencies (as does human voice). They especially don't work well at higher frequencies because of the shorter wavelengths and the distance between the drivers and your eardrums (which can vary), and human voice has a lot of higher-frequency components. If you could stick the drivers right next to your eardrums with custom-fitted in-ear plugs, you could get far better performance, but that's neither economical nor desirable to most people for comfort reasons.