[Upvoter33]

It actually relates to many things, but mostly it's because all the matters is getting credit for an idea. Realize that the process is basically to create something and then write about it to advance knowledge, thus sharing it with all. What's left at the end of that process is to recognize those who have done a lot of positive sharing. Those who are so recognized get the best jobs, the most funding, the best students, etc. As such, if an idea slips out of your lab and into some other lab, you might not get credit, and hence, all the craziness. It's one of those cases where it's not a great system, but it's also hard to figure out a better one.

[hatmatrix]

On the other hand, physics seems to not have this issue. Experimentalists working on topics related to CERN publish jointly (with like 500 co-authors) and theoreticians regularly post pre-prints under review on arXiv. They are a much smaller community though.

[evgen]

This is a consequence of the required resources for the experiments. If it takes a ten billion dollar piece of equipment to produce your results then chances are you are going to be one little cog in a very big machine. There are not papers with 500 co-authors in any other field or even in sub-fields of particle physics that do not require massive accelerators to achieve results.

[inciampati]

There are many large-scale projects in biology that benefit from open collaboration. In general, these require lots of work and produce useful resources used by a whole field. Insights that could revolutionize a field can sometimes be described in a sentence or two, and reproduced in a few days or weeks, so scientists are more uneasy sharing them.