[eeeeeeehio]

Academics seem to have this fixation on "ideas":

> And it’s not just a pace thing, there’s a threshold of clarity that divides learned nothing from got at least one new idea.

But these days, ideas are quite cheap: in my experience, most researchers have more ideas than students to work on them. Many papers can fit their "core idea" in a tweet or two, and in many cases someone has already tweeted the idea in one form or another. Some ideas are better than others, but there's a lot of "reasonable" ideas out there.

Any of these ideas can be a paper, but what makes it science can't just be the fact that it was communicated clearly. It wouldn't be science unless you perform experiments (that accurately implement the "idea") and faithfully report the results. (Reviewers may add an additional constraint: that the results must look "good".)

So what does science have to do with reviewers' fixation on clarity and presentation? I claim: absolutely nothing. You can pretty much say whatever you want as long as it sounds reasonable and is communicated clearly (and of course the results look good). Even if the over-worked PhD student screws up the evaluation script a bit and the results are in their favor (oops!), the reviewers are not going to notice so long as the ideas are presented clearly.

Clear communication is important, but science cannot just be communicating ideas.

[Al-Khwarizmi]

Clarity is and should be absolutely crucial, though.

As an academic I need to be up to date in my discipline, which means skimming hundreds of titles, dozens of abstracts and papers, and thoroughly reading several papers a week, in the context of a job that needs many other things done.

Papers that require 5x the time to read because they're unnecessarily unclear and I need to jump around deciphering what the authors mean are wasting me and many others' time (as are those with misleading titles or abstracts), and probably won't be read unless absolutely needed. They are better caught at the peer review stage. And lack of clarity can also often cause lack of reproducibility when some minor but necessary detail is left ambiguous.

[auggierose]

Clarity is relative. You can be super clear, but if it goes against what the reviewer thinks they know, it will be perceived as unclear. You can also point to references that clear up any remaining doubt about how something is meant, but of course the reviewer will never check out these references.

In the end, getting a paper accepted is a purely social game, and has not much to do with how clear your science is described, especially for truly novel research.

[intended]

That is not clarity in a paper.

1) The whole opening segment is the literature review.

2) If you are coming up with a novel concept, then you would be explaining how it shows up in relation to known fact.

Then you would be providing evidence and experiment.

The entire structure is designed to ensure as many affordances for the author to make their case.

Being accepted as a social game is the cynical view that ignores that academia still works. It’s academia itself which recognizes these issues and is trying to rectify the situation.

[auggierose]

As you have seen in the post (did you read it, dear reviewer?), references on the first page where unhelpful, so 1) comes already with a caveat.

And so on.

I think the social game view is at this point entirely justified, and there is nothing cynical about it. And no, academia does not still work.

A given "structure" is also ridiculous, and part of the problem. Once you care more about the form than the content, form is what prevails.

The truth is: To understand a paper properly, you need to deal with it properly, not just the 5 minutes it takes to skim the first pages and make up your opinion there already. Fifteen pages is short enough, and if you cannot commit to properly review this, for a week or so of dedicated study, just don't review it. We would all be better off for it.

[ancillary]

> The truth is: To understand a paper properly, you need to deal with it properly, not just the 5 minutes it takes to skim the first pages and make up your opinion there already. Fifteen pages is short enough, and if you cannot commit to properly review this, for a week or so of dedicated study, just don't review it. We would all be better off for it.

Reviewing dynamics make this hard. There is little to no reward for reviewers, and it is much easier to write a long and bad paper than it is to review it carefully (and LLMs have upset this balance even further). To suggest that every submitted paper should occupy several weeks of expert attention is to fundamentally misunderstand how many crappy papers are getting submitted.

[auggierose]

Not several. One full week would be enough.

To suggest that peer review means anything when this is not the case is the true fundamental misunderstanding here.

In other words: Fuck peer review. You are no peer of mine.

[setopt]

> But these days, ideas are quite cheap: in my experience, most researchers have more ideas than students to work on them.

By “idea” researchers usually imply “idea for a high-impact project that I’m capable of executing”. It’s not just about having ideas, but about having ideas that will actually make an impact on your field. Those again come in two flavors: “obvious ideas” that are the logical next step in a chain of incremental improvements, but that no one yet had time or capability to implement; and “surprising ideas” that can really turn a research field upside down if it works, but is inherently a high-risk/high-reward scenario.

Speaking as a physicist, I find the truly “surprising ideas” to be quite rare but important. I get them from time to time but it can take years between. But the “obvious” ideas, sure, the more students I have the more of them I’d work on.

> Any of these ideas can be a paper, but what makes it science can't just be the fact that it was communicated clearly. It wouldn't be science unless you perform experiments (that accurately implement the "idea") and faithfully report the results. (Reviewers may add an additional constraint: that the results must look "good".)

I kinda agree with this. With the caveat that I’d consider e.g. solving theoretical problems to also count under “experiment” in this specific sentence, since science is arguably not just about gathering data but also developing a coherent understanding of it. Which is why theoretical and numerical physics count as “science”.

On the other hand, I think textbooks and review papers are crucial for science as a social process. We often have to try to consolidate the knowledge gathered from different research directions before we can move forward. That part is about clear communication more than new research.

[eeeeeeehio]

It's not too difficult to state any idea, even a surprising one. But often, papers with surprising ideas (or maybe the right thing to say is surprising results?) turn out to be wrong!

I think it's still the case that there's lots of ideas that (if they worked!) would be surprising. Anyone can state outlandish ideas in a paper -- imo the contribution is proving (e.g. with sound "experiments", interpreted broadly) that they actually work. Unfortunately, I think clarity of writing matters more to reviewers than the soundness of your experiments. I think in CS this could very well change if the reviewers willed it (i.e. require artifact submission with the paper, and allow papers to be rejected for faults in the artifact)

[mnky9800n]

I think in some ways science has been co-opted by careerists who try to minmax output to accelerate their careers. Being idea obsessed is part of this. It’s much easier to get a paper published that’s on the hype train as opposed to a paper that challenges some idea. Publications justify grant money, grant money justifies more people and more power, more power justifies promotions. And if you talk with early career scientists they all will say they are only doing it until they get a permanent position. Then they will become more curious. Maybe they do, maybe they don’t, I have many older colleagues who are quite curious compared to their younger counterparts. but I believe rewarding ambition at the expense of curiosity is somewhat anti intellectual. It’s sad because I think science should reorganise as the current structure of departments into disciplines may be dated and restructuring could help alleviate this a lot since interdisciplinary work may leverage curiousity over ambition as curiosity will be rewarded with high impact work. But who knows. I can arm chair my way into anything.

[smolder]

Tell me what the careeerists have done to hurt your nonexistant novel approach to any hard problems in science.

[rocqua]

The point of a paper isn't "I had this idea" nor is it "I have this evidence". It is "I had this idea, and it turned out to work! (btw here's the evidence I found that convinced me it works).

The value lies in getting true ideas in front of your eyeballs. So communicating the idea clearly is crucial to making the value available.

[eeeeeeehio]

I agree with you. But what part of the blog post, or the peer review process in general, do you think ensures that only true ideas get in front of eyeballs?

I can write anything I want in the paper, but at the end of the day my experiments could do something slightly (or completely) different. Where are reviewers going to catch this?

[rocqua]

As the Article said, the first page gets a paper accepted. The remaining pages serve to not get the paper rejected. That includes actually backing up the claims on the first page.

Wrong things definitely still make it through, both mistakes and fraud. But it is a pretty strong filter.

[eeeeeeehio]

I agree that peer review can be a strong filter, but it's a filter for claims and evidence that sound true. CS papers can and do hide important details in the code (details which, I argue, would get a paper rejected if they were stated in the paper).

Regardless of the strength of the filter, if the filter's inputs are just "the paper", but the claims depend on the details in another artifact (i.e. the code), how can we argue that peer review filters for the truth?

[smolder]

This person is a clown, probably with a paid agenda, and they should be disallowed from saying such dumb things where smart people with useful skills might read it.

[jhrmnn]

I have a theory that this focus on ideas vs solutions also divides individual researchers, in what drives them. Agreed that academia celebrates and rewards ideas, not solutions. And maybe that’s ok and how it should be, solutions can be done in industry? But the SNR of ideas feels too high at this point.

[QuadmasterXLII]

generating the ideas “planets move at constant per planet velocity” “planets move at a specific speed as a power law function of distance from the sun and we fit the paramets great” “each planet sweeps equal areas in equal time” is cheap, but evaluating which idea is good is expensive, and the whole value of that evaluation is captured in the final idea

[smolder]

Your comment is cheap in a way that's embarrassing. You wrote an anti-science rant while failing to meet the standards you complain about.

[smolder]

This whole take is embarrassingly ignorant and no one with the credentials has the time to check you. We need people to do real thinking and they need to ignore you.

[eeeeeeehio]

I don't understand why took the time to leave (three?) personal attacks, rather than just provide your perspective? I'm willing to acknowledge that my opinion has limitations (I think it mainly applies to CS-adjacent fields with empirical performance evaluations, and where experiments can easily be independently verified).

I would be interested to hear other perspectives.

[smolder]

It really upsets me when i see this kind of hollow rhetoric, which is specifically designed to confuse, and to stear people away from learning and instead make them sceptical of the most honest pursuers of truth.

[smolder]

None of this attack was personal, that is wrong. All was deserved. Complaining about academia without any specifics is one of the dumbest and most well funded and prominent memes in the pseudo intellectual podcast sphere today.

[tossandthrow]

I think the take is reasonable.

In particular, the lines between science and some industry is blurring.

Eg. Machine learning where universities appear almost lazy compared to their industrial counter parts.