| Depends on the field a bit, but my general process involves using a few questions to signal one way or the other. 1. What are the actual claims made by the authors? That is, what do they claim to have found in the study? If you cannot find these, there is a good chance the paper is not particularly useful. 2. For each claim, what were the experiments that led to that decision? Do those logically make sense? 3. Are the data available, either raw or processed? Try reproducing one of their more critical figures. Pay close attention for any jumps in series (e.g. time suddenly goes backwards -> did they splice multiple runs?), dropped data points, or fitting ranges. If the data are not available, consider asking for them. If the code exists, read through it. Does the code do anything weird which was not mentioned in the paper? 4. How do they validate their methods? Do they perform the appropriate control experiments, randomization, blinding, etc? If the methods are so common that validation is understood to be performed (e.g. blanking a UV-Vis spectrum), look at their data to find artifacts that would arise due to improper validation (e.g. a UV-Vis spectrum that is negative). 5. Do they have a clear separation of train and test / exploration and validation phases of their project? If there is no clear attempt to validate the hypothesis for new samples, there is a good chance the idea does not transfer. |