Larger headphones like what? Impedance should matter there, not size, and it would surprise me to learn that there are high-Ω headphones that an iPhone can drive effectively, but the adapter can't.
A sysadmin I work who has had Apple-everything for years (iMac, MacBook Pro, MacBook Air, iPhone, Apple Watch...) with walked in last week without his Apple Watch.
Asking him about it, he pulled out a new Pixel and said it was for op's very reason. In particular he noted that the pixel had a much better DAC than any one iPhone he'd ever had - and in particular that it could drive all of his higher end headphones. He also mentioned that the new iThing adapter has it own DAC inside the adapter... it's apparently powered by the lightning port with its own drivers - and that he had tried it out anyway but found it incapable of properly driving any of his nicer headphones.