[calebkaiser]

Implementing models directly from papers is typically pretty doable (and is of course more straightforward when the full implementation is open sourced). Often there is some amount of specific knowledge, like particular hyperparameters, that is missing and has to be trial and errored by the community, but generally speaking, getting the core model architecture implemented is a reasonable task for most well documented models.

Reproducing the exact training run, however, is basically impossible without the original dataset and training pipeline (here meaning all of the code + infra involved in actually executing the pre and post training loops). Also, it would be exorbitantly expensive to do if you weren't also a lab trying to train a similar model.

But you can still scale the architecture down and experiment as a solo researcher using the published research. There are probably some open source implementations already on GitHub for any given big open model release.