[pacbard]

Not all null results are created equal.

There are interesting null results that get published and are well known. For example, Card & Kruger (1994) was a null result paper showing that increasing the minimum wage has a null effect on employment rates. This result went against the common assumption that increasing wages will decrease employment at the time.

Other null results are either dirty (e.g., big standard errors) or due to process problems (e.g., experimental failure). These are more difficult to publish because it's difficult to learn anything new from these results.

The challenge is that researchers do not know if they are going to get a "good" null or a "bad" one. Most of the time, you have to invest significant effort and time into a project, only to get a null result at the end. These results are difficult to publish in most cases and can lead to the end of careers if someone is pre-tenure or lead to funding problems for anyone.

[MITSardine]

Is that actually a null result though? That sounds like a standard positive result: "We managed to show that minimum wage has no effect on employment rate".

A null result would have been: "We tried to apply Famous Theory to showing that minimum wage has no effect on employment rate but we failed because this and that".

[roenxi]

Looking at the definition in the article, it definitely is a null result. However the example does illustrate that 'a null result' probably isn't a very interesting thing to talk about because it covers too many types of result. I think what people on HN actually want to track is something more like 'a boring null result'. The real question is whether there a process that is reliably being followed and leading to research that matches reality (where a statistically significant result suggests something is real) or is scientific publishing highly biased towards odd results (where studies that muck up or get lucky with statistical noise are over-represented).

In this case we would expect some studies of the minimum wage to show it increases employment regardless of what the effect of wage rises is in the general case - eg, some official raised the minimum wage while a sector went into a boom for unrelated and coincidental reasons.

[iamjfu]

From a related nature article (https://www.nature.com/articles/d41586-024-02383-9), "null or negative results — those that fail to find a relationship between variables or groups, or that go against the preconceived hypothesis." According to this definition, I think both examples you provided are null results. Particularly here, where the context is the file drawer problem.

[Analemma_]

No, because in theory a minimum wage increase could decrease the unemployment rate. If it does neither, that’s a null result.

[mitthrowaway2]

That's sort of like saying "I measured acceleration due to gravity by dropping a bowling ball off the Tower of Pisa, and got a null result: the ball simply hovered in midair."

If the result is very surprising and contradicts established theory, I wouldn't consider it a null result, even if some parameter numerically measures as zero.

[Analemma_]

The question being asked is, "what is the correlation between these two variables": is it positive, negative, or zero (null). The null hypothesis is the baseline you start from because the overwhelming majority of variables from physical observations are uncorrelated (e.g. what is the correlation between "how many people in the treatment group of this clinical trial made a full recovery" and "the price of eggs in Slovakia").

Measurements of some physical quantity are a different kind of experiment, you cannot phrase it as a question about the correlation between two variables. Instead you take measurements and put error bars on them (unless what you're measuring is an indirect proxy for the actual quantity, in which case the null hypothesis and p-value testing does become relevant again).

[mitthrowaway2]

In fact, you can express measurement of g as a linear correlation between y-y0 and (t-t0)^0.5.

[im3w1l]

Null means nothing, zero. In the context of scientific articles a null result means that the difference is zero. What difference? Well it depends. It could be difference between doing something and not doing it. The difference between before and after some intervention. Or perhaps the difference between two different actions.

In this case the difference between before and after raising the minimum wage.

Furthermore, the thing with a null result is that it's always dependant on how sensitive your study is. A null result is always of the form "we can't rule out that it's 0". If the study is powerful then a null result will rule out a large difference, but there is always the possibility that there is a difference too small to detect.

[ktallett]

Null means something doesn't exist and proving something doesn't exist can at times especially in certain sciences be as valuable as proving something does exist. Especially if it stops the rest of the field chasing the pot of gold at the end of the rainbow.

[jruohonen]

> Is that actually a null result though?

The above is a good point but I would extend it further. I mean, philosophically, you get a positive result from a negative (null) result by merely changing your hypotheses (e.g., something should not cause something else).

[tempusalaria]

if you are p testing this isn’t the case. A positive result is a much stronger assertion

[jruohonen]

Sure and of course you shouldn't change your hypotheses. That's kind of the whole purpose of pre-registrations at a meta-level of things.

[Natsu]

> Card & Kruger (1994) was a null result paper showing that increasing the minimum wage has a null effect on employment rates. This result went against the common assumption that increasing wages will decrease employment at the time.

I had to look that up, because more precisely, it showed that a particular minimum wage increase in NJ from $4.25 to $5.05 didn't increase unemployment in 410 particular fast food joints in 1989-1990 - https://davidcard.berkeley.edu/papers/njmin-aer.pdf - not that "increasing the minimum wage has a null effect on employment rates" at all, ever, no matter what. It's not as if increasing the minimum wage to something impossible to afford like $100 trillion wouldn't force everyone to get laid off, but nobody generally cares about the limiting case like that, as that's relatively unlikely.

The interesting part is non-linearity in the response, seeing where and how much employment rates might change given the magnitude of a particular minimum wage increase, what sort of profit margins the affected industries have, elasticity of the demand for labor and other adaptations by businesses in response to increases, not whether it's a thing that can happen at all.

And we're seeing a lot more such adaptions these days. There are McDonalds around here that have computers in the lobby where you input your order yourself and I've gotten drones to deliver my food instead of DoorDash drivers. That kind of thing was not yet practical back in 1989, when I remember using an Apple ][ GS and it's not clear that findings like this should be relied upon too heavily given that some of the adaptations available to businesses now were not practical back then, especially when technology may change that even more in the future.

[kazinator]

What matters for publication is a surprising result, not whether it confirms the main hypothesis or the null one.

The "psychological null hypothesis" is that which follows the common assumption, whether that assumption states that there is a relationship between the variables or that there is not.

[SubiculumCode]

One of my most celebrated recent "null result" was published in Biological Psychiatry, a journal with an impact Factor about 13 last I checked. Even though it was a null result, it was decently powered longitudinal study that provided evidence inconsistent with the prevailing view in autism research. Even if there exists a difference that we failed to detect, it probably much smaller than what others had expected on the basis of second-grade evidence (cross-sectional group differences).

[aydyn]

I agree. More succinctly, it confounds two types of results: a null result due to statistical noise (big error bars, experiment failure) and a null result where the null model is more likely (actually, the effect doesnt exist).

Like many things in statistics, this is solved by Bayesian analysis: instead of asking if we can reject the null hypothesis, the question should be which model is more likely, the null model or the alternate model.

[hatthew]

Also, doesn't a null result mean "we weren't able to measure an effect" rather than "there is no effect"? That's a pretty big difference in how important something is.

[reliabilityguy]

Sure. But publishing this result with your experimental methodology may help others to refine the experiment and get a better answer.

It is absolutely shameful that negative results are almost never published. I sure that a lot of money and effort wasted by repeating the same dead-end experiments by many research groups just because there is no paper that said: “we tried it this way, it didn’t work”.

[9rx]

1. Papers are written to invite replication — the most important part of the scientific process. It is already difficult to compel replication even when you only put the most promising research in people's faces. Now you want them to have to also sift through thousands of entrants that have almost no reason for replication attempts?

2. It's easy to call it shameful when it isn't you who has to do the work. If you are like most other normally functioning people, you no doubt perform little experiments every day that end up going nowhere. How many have you written papers for? I can confidently say zero. I've got better things to do.

[jraph]

The lack of null result publication is a real issue for research and scientific knowledge in general, including for accurate positive results (what if once you see a positive result by statistical chance but the experiment was actually conducted before you by 10 other people but that wasn't published because the results were null?)

> Papers are written to invite replication

They are written to share knowledge, new discoveries. We hope that they are replicated.

> It's easy to call it shameful when it isn't you who has to do the work.

We are not judging anyone, we are qualifying the situation. And we are speaking of publishing results for experiments that have already conducted, not voluntarily making up null cases and doing the related experiments. It wouldn't make sense and would be harmful. But if you did an experiment that produced a null result, not publishing is a loss of knowledge. Again, we are not judging anybody, it's a failure of the publishing system. But this cannot change if nobody points it out.

I'd have troubles understanding a researcher not acknowledging today that the lack of null result publication is an issue. It would show a lack of perspective IMHO. And for someone acknowledging that there's an issue, pushing back is not the right stance.

[9rx]

> They are written to share knowledge, new discoveries. We hope that they are replicated.

If they aren't replicated, no new knowledge is gained. Not really.

> it's a failure of the publishing system.

What failure? Again, the "publishing system" is to submit the "best of the best" research to the world in order to invite replication. While nothing is perfect, we don't want people to have to sift through papers that have effectively no reason to be replicated in a quest to find something that is. That would make things infinitely worse.

The internet was created for the minor-leagues. If you are willing to put in the effect to document your "less than the best" research, put it up on a website for all to see already. There is nothing stopping you. But who wants to put in the effort?

> We are not judging anyone

How do you define "shameful" if it isn't related to judgement of someone?

[jraph]

You assume that null results are not worth it and are not the best of science. You seem to be ignoring the issue where researchers collectively waste time redoing "failing" studies again and again because the null results are not published, and other issues not publishing them causes.

We fundamentally disagree here and I'm not willing to put in the effort needed to try to convince you otherwise, many other comments are here are better than what I could write.

I do hope you take the time to take a step back (and you seem to be in defensive mode, if so, you need to go out of this) and reconsider.

> How do you define "shameful" if it isn't related to judgement of someone?

Do you know the expression "it's a shame"?

> used when you wish a situation was different, and you feel sad or disappointed

https://www.ldoceonline.com/dictionary/it-s-a-shame-what-a-s...

[reliabilityguy]

> It's easy to call it shameful when it isn't you who has to do the work.

What work? The work of writing the paper?

It seems to me that it’s better that research group A will spend 10 days to write the paper about their dead-end experiment, than 20 other research groups will do the same series of experiments, wasting a lot of time, money, and energy just because there was no paper that said “it doesn’t work”. Perhaps it would be better if those 20 research groups would instead try 20 different ways to fix the faulty method in hopes to get somewhere, than dosing the same thing.

I do not understand how it’s even a question for debate.