|
|
|
|
|
|
by tbalsam
880 days ago
|
|
|
I enjoyed this paper (I share a discord with the author so I read it a bit earlier). It's not entirely clear from the comparison numbers at the end, but I think the big argument here is efficiency for the amount of performance achieved. One can get lower FID numbers, but also with a ton of compute. I can't really speak technically to it as I've not given it a super in depth look, but this seems like a nice set of motifs for going halfway between a standard attention network and a convnet in terms of compute cost (and maybe performance)? The large-resolution scaling seems to be a strong suit as a result. :) |
|
|
Yeah, the main motivation was trying to find a way to enable transformers to do high-resolution image synthesis: transformers are known to scale well to extreme, multi-billion parameter scales and typically offer superior coherency & composition in image generation, but current architectures are too expensive to train at scale for high-resolution inputs.
By using a hierarchical architecture and local attention at high-resolution scales (but retaining global attention at low-resolution scales), it becomes viable to apply transformers at these scales. Additionally, this architecture can now directly be trained on megapixel-scale inputs and generate high-quality results without having to progressively grow the resolution over the training or applying other "tricks" typically needed to make models at these resolutions work well.