[pama]

Does this method build assumptions about the distribution of the evaluation dataset and make the bit-level reproduction of an evaluation unlikely?

[renchuw]

Well, this method is based on the assumption that embeddings can accurately represent the texts and their structural relations are preserved.

So long as you have all the random seeds fixed, I think reproduction should be straight forward.