[Mr-Frog]

older vision LLMs chopped up images into patches which were projected into the same embedding token space as words. Newer ones use an encoder to more efficiently project an image into token space. Then it runs through the same attention layers as the text component.