Nitrogen is apparently pretty dangerous too because, from what I understand, your body doesn't feel like it's not "getting enough air" like it does with high CO2 concentrations? Something like that, I don't recall the specifics...
It's build up of co2 in your lungs/body that gives you the urge to breathe, yeah. So any messing around with the air composition could kill you without noticing the lack of o2, as long as you keep breathing thus getting rid of the co2.
Joking with some coworkers I once took a big breath of what should've been helium but had accidentally been misfilled with trimix welding shield gas (He/CO2/Ar).
I knew immediately what had happened as I coughed in duck noises.