[p-e-w]

Sounds like they have it the wrong way round. Considering the uncertainty surrounding all of this, and the initial problems with replication, why would Nature publish this before it has been independently replicated?

Not sure what they're worried about here. They already have priority, and if this replicates, every journal on Earth will be fighting over the privilege to publish it.

[Maxion]

Because that’s the scientific method? It would be entirely infeasible for them to require someone else to test the sample.

I don’t think you’ve been in academia?

[p-e-w]

> Because that’s the scientific method?

The world's leading science journal publishing grand claims without verification, in the face of mounting evidence that the claims might be incorrect, is most certainly not "the scientific method".

> It would be entirely infeasible for them to require someone else to test the sample.

What nonsense. Why would that be infeasible?

[szundi]

Actually this is the scientific method. Anything else just makes publishing groundbraking results harder and depends on some people’s (faulty) judgements.

[mort96]

Publishing incorrect results should be hard, especially if they would be ground breaking if true.

[physicsguy]

Nature basically publishes the sensational claims papers. That's their model, anything "high impact" goes there. It's then on other scientists to publish work refuting it or proving it's not correct. It's an issue that the refutations tend to go in lower impact journals, but it's the way it's works.

[epups]

Extraordinary claims require extraordinary evidence.

But seriously, why can't they ship their sample somewhere?

[SiempreViernes]

They are busy trying to characterise it properly and making sure they understand what they have created. At least that's what they ought to be doing if this isn't a big scam.

[ethbr0]

> why would Nature publish this before it has been independently replicated?

Because it's Nature?

We're not talking about Organic Syntheses [0], which has replicated every paper before publishing since 1921.

[0] https://www.orgsyn.org/

[peyton]

I guess don’t hate the player. They’ve got a floating rock. What would materially affect conclusions?

[zarzavat]

Pride. They don’t want to be remembered for some incomplete Arxiv preprint. They want to be remembered for the finished paper in a top level journal.

[WillAdams]

Most likely the peer review process will include replication.

[tnecniv]

Normally (at least in my field), that’s beyond the scope of peer review in the traditional sense. Peer review typically just involves review of just the material submitted to the journal (manuscript, data, maybe code depending on its size and complexity). Depending on the experiment, reproducing a result may require resources, equipment, or skills unavailable to the reviewer or that cannot be acquired in a reasonable amount of time needed for the review.

Peer review serves basically as a smell test where capable experts certify that the paper is novel, interesting, properly justifies its claims, and contains enough detail to reproduce the results. It is not uncommon for only a few labs to contain all of the expensive, specialized equipment to reproduce an experiment, and getting that experiment working (if it is reproducible) can take a significant time commitment from multiple individuals in a lab. Also keep in mind that reviewers are normally not paid for their services, journals typically want at least two independent reviewers (who they pick, sometimes with input from the author to narrow down who has the background to perform the review), and going from submission to publication is typically a many-month process without the high standard of reproduction.

Reproduction happens in the manner that is going on right now. If your publication makes waves, people will want to build on your result and either reproduce it as a first step or conduct a related experiment that adds evidence to support the hypothesis. This way is far more practical, but the devious thing is that, if you lie and manipulate your data, some poor grad student might waste two years of their PhD trying to get your thing to work.

Having journals or some independent body fund reproduction themselves runs into both capital challenges and the problem of attracting a large enough number of capable scientists to cover every discipline who would only want to work on reproduction full time. As nice as that would be, I’m not sure such a venture would work in the real world.

[jacquesm]

That's true but if a reputable journal publishes this before that verification is in and it ends up being mistaken or even a hoax or fraud then their reputation will be dinged so with the stakes this high given the material and the visibility there is a fair chance that they will indeed wait for independent replication before committing to publishing it. But I don't see it happening during the peer review process itself, it will happen simply because another lab tries the same and gets lucky (or not, in which case the whole thing will blow over).

[phanimahesh]

The journals reputation will be dinged if it comes to light that they have not adequately performed their due diligence. Which, as has been explained above, does not include replication.

If you require replication for publication practically nothing will get published ever. Once published, others will attempt to replicate, and it is the authors' reputation on the line if nobody manages to replicate it following the paper and consulting with the authors.

[jacquesm]

They are not going to replicate this themselves, obviously. But extraordinary claims etc, and this one is about as extraordinary as they come. Papers that are non-controversial even if they might turn out to be wrong get published all the time. But papers where 95% of the scientists out there are going to be super skeptical will find it much harder to get published. Both the authors and the publishers have a reputation to protect. Keep in mind that the peer-reviewers will have to sign off on the publication as well.

[RF_Savage]

How can you replicate it before the paper with the stuff to replicate is published?

[jacquesm]

It would be a variation on what's already out there right now. And the whole problem is that the recipe is vague, in other words even if they have a working sample they don't know exactly what goes into making it.

[RF_Savage]

Which is why it in arxiv and not some journal.

Seems like somebody pulled the trigger on publishing too early.

[lovich]

Why would Nature have published the production method of a random material if they didn't have experimental evidence of novel behavior?

[cosmojg]

They do that every now and then. Peer review rarely, if ever, involves any sort of replication.

[lovich]

I’m asking on a legitimate basis. I am not involved in the science world and only know about Nature as a publication from high school and college. The way Nature was described to gave me the sense that you weren’t going to have a paper published unless you had some surprising results, and that didn’t include the null hypothesis.

If that’s not the case for this publication then I would appreciate if someone showed me how they changed in the past decade

[michaelt]

They publish surprising and high-profile results, yes.

However, there's also an informal crazy claims filter. If your paper is so surprising they think it might be made up or wrong, they might ask for extensive revisions prior to publication.

So they won't publish my paper saying the moon is made of cheese no matter how surprising such a finding might be.