[RC_ITR]

GP likely means training transformers on raw data (similar to protein folding transformers) to find patterns that humans cannot (due to lack of context, bias, or whatever).

Problem with the assumption though is that transformers are good at identifying and replicating patterns given a set of rules (i.e. how proteins fold and misfold depending on the environment).

Hubble data isn’t so much “we know the rules but not their interactions” as much as “we don’t really know the full set of rules,” so that particular example probably wouldn’t be that fruitful.

In general, biology (where we understand the basic rules but not the complex ways they are combined) is the most fertile ground for transformer driven research.