Well, oxygen _is_ poison. It's eager to react (sometimes violently) with almost everything. It rusts and oxidates perfect shiny metals and silicon making everything an oxide!
"Poison" can also refer to a substance toxic to other animals. We say that chocolate is poisonous to dogs for instance. And a good fraction of Earth's biosphere was killed off by oxygen poisoning in the first of Earth's great mass extinctions.
Also, the dose makes the poison and excess oxygen actually can poison humans. Deep sea divers have to worry about excess oxygen inducing seizures if they mess up their breathing gasses enough. And even 100% oxygen at regular pressure will slowly damage the lungs, something ICUs have to worry about.
Nick Lane had a great book about oxygen, Oxygen, which maybe isn't as good as his book about mitochondria but is well worth reading.