I don't think anyone associates ether with a drug first.
From Thesaurus: "ether - the fifth and highest element after air and earth and fire and water; was believed to be the substance composing all heavenly bodies"
Though another spelling of that is "aether" (I think this is more common than "ether").
Ethers are also a group of rather common organical compounds in chemistry. This is the page Wikipedia gives you when searching "Ether", as gateway allures to. Apparently, a specific compound of those (diethyl ether) can be used as a drug. Seems rather unlikely to me that people would connect the name to the drug first.