[IdiocyInAction]

I think NAS is a bit higher level than what the OP had in mind - NAS isn't usually used to search for fundamental operations like self-attention or convolution. But I guess you could probably adapt it quite easily.

[jellyksong]

I believe some algorithms, like AutoML-Zero, search on the level of mathematical operations.