[godelski]

  > Overall it seems pretty unscientific.

I'd agree with all your points and add some things to help people better "sniff-test" these kinds of papers.

  1) The paper is suggesting aliens... your suspicion hats should always go on
    - Carl Sagan said: "Extraordinary claims require extraordinary evidence". Is the evidence extra-ordinary?
  2) The authors aren't experts
    - Stephen Bruehl: A doctor of Anesthesiology
    - Brian Doherty: "Independent Researcher"[0]
    - Alina Streblyanska: Actually maybe a astrophysics researcher?[1]
    - Beatriz Villarroel: The top Google hit for her is for a UFO wikipedia[2]
  3) Authors don't share affiliations
    - Corresponding author has no domain expertize and no clear affiliation to others.
  4) Authors have hints of metric hacking
    - Villarroel has 8 citations in a paper with only 18[3]
  5) The GitHub repo is dead: https://github.com/dca-doherty/VASCO-ML

None of these things are enough to conclude that the paper is wrong, but they are red flags and don't require actually understanding any of the details of the paper.

If you do understand statistics there's clearly more red flags. The +/- windowing being a pretty big one, since there are much better tools for this (errors don't need to be symmetric! Nor do they need to be uniform!). There's also a pretty big assumption made that cshimmin didn't mention: the paper assumes all nuclear tests are in the public record. But I also assume if you have a strong statistics background then there's a high probability you didn't upvote the post.

[0] The man has effectively no online presence. Google searching his email yields effectively nothing except people posting about this paper in UFO groups (https://www.google.com/search?q=%22briandohertyresearch%40gm...). His linked GitHub also makes him anonymous (https://github.com/dca-doherty/) and his website linked is just about finding day care in Texas. He has one more paper on ArXiv, but it is from a few weeks prior

[1] Found their Linkedin (https://www.linkedin.com/in/alina-streblyanska-95b2375b/). Their most recent paper is also on UAPs, along with Villarroel. But also, they work for "Society of UAP Studies", which should be a big red flag. Also, they were working as a Post-doc for 12 years, which is a bit insane

[2] https://www.wikidisc.org/wiki/Beatriz_Villarroel and here's here Google Scholar https://scholar.google.com/citations?user=_Jc8gm0AAAAJ

[3] I looked at some other papers of hers and they show a similar pattern. This explains her citation count (which is rather low) and h-index (it's better to just click on the references and you'll see it's predominantly her referencing herself):

  - 2602.15171: 9 citations total, 8 are hers
  - "A cost-effective search for extraterrestrial probes in the Solar system" has many more, but still 6 to herself (and 3 to Loeb)
  - Transients in the Palomar Observatory Sky Survey (Yes, this is in "Nature"): 20 citations, 5 hers
  - Aligned, Multiple-transient Events in the First Palomar Sky Survey: 11/36
  - On the Image Profiles of Transients in the Palomar Sky Survey: 5/5
  - A Civilian Astronomer's Guide to UAP Research: 7/98 (actually not a red flag, but the title sure is...)
  - and so on

[cshimmin]

Not gonna lie, the first thing I noticed was that the first author was in an anesthesiology department. Your guidelines for sniff-testing are not unreasonable, and can definitely be helpful to people who are unfamiliar with the research area. But I quite intentionally did not appeal to any of those. As a (somewhat) subject matter expert, it's important to _ignore_ things like ad hominem judgement, and instead address the paper on its self-contained merits. And more importantly, to share my assessment of those with the lay public.

[godelski]

I'm glad you did it that way. I hope, my comment works well as an addendum to your type of comment. I don't think would have worked well on its own, nor prior to yours. Especially since nothing I said is an absolute rule that allows one to reject a work. But this paper sure does smell suspicious. I think it's good to have the stronger reasons to be suspicious and then understand some softer flags to navigate in unfamiliar territory.