[jellyksong]

This is basically what the field of neural architecture search tries to do. Here’s a good (somewhat technical) introduction: https://lilianweng.github.io/lil-log/2020/08/06/neural-archi...

[IdiocyInAction]

I think NAS is a bit higher level than what the OP had in mind - NAS isn't usually used to search for fundamental operations like self-attention or convolution. But I guess you could probably adapt it quite easily.

[jellyksong]

I believe some algorithms, like AutoML-Zero, search on the level of mathematical operations.