[adminprof]

This entire thread is really frustrating for me to read. A lot of these ideas are one-liners from people who have had very little experience with different parts of academia, except maybe being a student. Every single idea here are debated to death by academics already (not just university academics, but funding agencies, academic societies, award panels, journal editors and conference organizers), with changes happening all the time.

If it was this simple that a random person here could come up with how to "solve" academia, we'd have already done it decades ago. The ideas also lack nuance and when you get into the definitions of things (for example, p-hacking), then things become a lot more grey; are you allowed to look at a dataset that you spent 2 years collecting if your first hypothesis does not pan out? The clear cut cases are obvious to everyone, it's the grey area that takes 99% of the time to figure out.

Imagine reading a thread where everyone is proposing "solutions" to software development. It'd go something like "software development is a cesspool and 80% of it fails (see voting systems, MySpace, electronic health records, Theranos. Here's what software companies need to do:" [yes I'm being intentionally stupid to demonstrate how annoying this is]

1) stop releasing before the bugs are fixed. Software and games are rushed out. Companies need to take their time to fix the bugs so the users don't have to encounter them.

2) no more technical debt. Programmers are sloppy and introduce technical debt because they are not incentivized to do high quality programs. [yes, see how triggering that is]

3) cap team sizes. Everyone knows that large teams fail more spectacularly. Gmail and Napster and the original version of Google were made by a group of 4 people. Software teams need to be 4-5 people max.

4) programmers must use a transparency scorecard. Software companies like Oracle and IBM charge ridiculous amounts for their work. They hide costs and cut corners. Programmers should be transparent about the work they are doing each day, what data they access, and which functions they are writing.

These changes need to happen. derp derp

[nl]

Not sure who downvoted this, but this is 100% on the money in my experience.

People here are writing comments like that funding should be tied to "how systematic, logical, and well-documented the research is" as though these things are correlated to the existing criteria at all.

Much of the criticism here is like someone who knows how to build roads criticising agile software development because it makes it look like no one knows what they are going to build in the end. It's frustrating, and wrong, but the errors are subtle and aggregative.

[adminprof]

Yeah I have no idea how it got to -2, but maybe it was my tone. I was even trying to not call out any specific post or debate any single issue.

The thing about academia is it's full of people who love talking about ways to make it better, and rotate into positions of power where they can change things after a few years.

The solutions are complex, require convincing many different stakeholders (even if they're amenable to the change), nailing lots of detail to make it work right. Because peoples lives and careers are on the line. Reputations of entire fields, the way medical discoveries happen, billions of dollars of taxpayer money, major institutions, etc. are not things you want to hack and discover that whoops, you just incentivized the wrong thing and set back cancer research for a decade.

[marcus_holmes]

So, what's the best proposal or line of thinking coming from all this deep thought about how to solve it?

Because you can't possibly be saying that the current system is the best we can do, or that the problem is intractable.

(I agree with most of your proposals about software engineering btw. We should do more of this).

[adminprof]

It'd be a process: participant in a community in a subfield, listen to ideas that have been tried and outcomes, come up with an idea using those lessons learned and success metrics, build consensus around a new idea, test in a single-instance (like one conference, grant review panel, tenure committee at one university), share lessons to the field, do this for a couple of years to show clear success, expand to multiple events in the field, become an exemplar field for that idea and "infect" other fields

Sounds slow but there's thousands of such experiments happening simultaneously right now. This is how a long of major field-sized changes have happened, like the transition to conferences from journals (which had many initial problems like during tenure review or a lack of quality in reviews), etc. Ideas will lose traction at various stages (for example, there was a movement some time ago to use alpha=0.001 instead of alpha=0.05 for null hypothesis testing, which has been limited to that field or subfield).

[marcus_holmes]

Interesting, thanks.

Is the move to pre-publishing servers (like arxiv) a part of this? How does SciHub (and similar) figure into it?

It does sound slow, and a bit trivial, if I'm honest. Are there any examples you can share of successful experiments that have travelled across field boundaries?

[mturmon]

You're right. Love your example.

HN does not have much actual collective expertise in this area (say: governance structures for technical research), but HN is solution-oriented, so ideas will be proposed. They just tend to be not very good. (See also: HN climate-studies threads.)

I've spent 10 or 15 minutes on this thread, and didn't read any comments actually building on what the OP said. OP's author does have expertise in this - he has been writing a series of good stories for Science magazine in this general area. Sigh.

[roel_v]

Thank you for writing this. Your examples were lethargic. I know I should stay away from the 'academia is broken duh duh' threads here but I get sucked into them anyway. They're always full of armchair experts who don't know anything about universities or research except from being students or at best junior researchers and then think this makes them experts on the matter.

But the article itself had that same vibe for me, especially with how cock sure it is that the 'questionable research practices' are 'fraud'. I mean, come on - someone making up the responses of a 500 people questionnaire; yes that I could call 'fraud'. But not being included as an author on a paper, or being included when you didn't contribute that much? I've been in both situations and in some cases, I was completely fine with it; in others I was a bit miffed (mostly because of the same interpersonal frictions that happen everywhere where people work together) - but in none of those I would call it anywhere near 'fraud' or even 'dishonest'. Yes, there exist people who pay 1 or 2 people to write papers for them and then publish those papers with themselves as the only author. Again, that I would call 'fraud'. But the 99.9% of other cases - not even close. Just like because there is one billionaire underage sex trafficer, doesn't mean all of them are and that 'the system' is 'broken'.

And this is how I, again, got sucked into a completely non-productive 'discussion' that is so far removed from reality so as to be completely irrelevant anyway...

[webmaven]

> But not being included as an author on a paper, or being included when you didn't contribute that much? I've been in both situations and in some cases, I was completely fine with it; in others I was a bit miffed (mostly because of the same interpersonal frictions that happen everywhere where people work together) - but in none of those I would call it anywhere near 'fraud' or even 'dishonest'.

As I understand it, ghosting becomes a more significant issue when it enables the omitted author to peer-review their co-authors' papers (and vice-versa) without disclosure of the conflict of interest.

[roel_v]

shrug Sure, theoretically, but someone who wants to ensure a positive review can just as easily collude with someone who didn't do any of the work. And then still - yes it's possible that a completely bogus paper gets signed off on by a dishonest conspirator, but the editor should see the discrepancy between that and the reviews of the other authors, and dig deeper. But the real problems start in the grey zone; like when you're on the fence between 'reject' and 'major revisions'. There is no 'objective' truth there and quite honestly as an author it's a crap shoot and always has been. It sucks the first few times it happens but none of leads to 'peer review being broken'.

[nl]

To add to this - there are plenty of issues with peer review, but someone deliberately avoiding an authorship so they can peer review seems low down the list of real world problems.

[Chris2048]

> If it was this simple that a random person here could come up with how to "solve" academia, we'd have already done it decades ago.

Just because an idea is simple to come up with, doesn't mean it's also easy to implement.

Also, can you explain to me how your programming analogy works? Because I agree with a lot of it, though it seems I'm not supposed to.

[adminprof]

Sure I guess I didn't make that very clear. Just going down the line:

1) stop releasing before the bugs are fixed. Software and games are rushed out. Companies need to take their time to fix the bugs so the users don't have to encounter them.

- This is a tradeoff between shipping time and bugs. You will never fix all bugs, so it's unrealistic. And it's a business decision in many cases. Even the definition of a bug is tricky, like is a usability issue a bug? So it's just a naive idea.

2) no more technical debt. Programmers are sloppy and introduce technical debt because they are not incentivized to do high quality programs.

- No one wants to create technical debt. Obviously it slows down development later on. Again this could be a business decision. Some programs like one-off data science scripts don't need to fix all their technical debt. Technical debt also accrues naturally (like just changing environment, platform, standards) so it's not possible to aim to not have debt in the very beginning. Hindsight is 20-20 and all that. Saying programmers are not incentivized to do high quality programs is just a blanket naive statement, and depends on the definition of high quality programs.

3) cap team sizes. Everyone knows that large teams fail more spectacularly. Gmail and Napster and the original version of Google were made by a group of 4 people. Software teams need to be 4-5 people max.

- Depends on the type of software. Can't just generalize given a few token examples. Expectations also change over the course of the product.

4) programmers must use a transparency scorecard. Software companies like Oracle and IBM charge ridiculous amounts for their work. They hide costs and cut corners. Programmers should be transparent about the work they are doing each day, what data they access, and which functions they are writing.

- This uses one subsection of the software economy to make a point (as a fallacy). But also some of these measures don't make sense, like some programmers read a lot of code or delete lines, and so the metrics are not generalizable.

In summary, these are ideas that someone who has not done long-term software development would say, or someone who has only had experience with one type of software or company would say. They're not well defined, not generalizable, and don't account for the complex and varied sociotechnical process that software development is.