Yea, difficulty would drop. You'd end up producing a block more often than every 10 minutes. There's probably some lower bound on difficulty (and block frequency) that also maintains network consensus.
I meant drop xN by less baseline but increase xN by more difficult blocks, thus still 10min per block. But your sibling commenter now noted that the block size doesn’t affect difficulty. That’s confusing.