[zerohp]

I have enough experience to know if this is true or not. Many times it was faster to buy more machine, but often it was not. We already had 10000 cores.

I proposed, implemented, and tested an 8 line change to our alignment tool that saved 6% cpu time. It took me two days, most of which was my spare time at home. This one program was using 15 cpu years every month. Nobody cared. It never went into production. I started interviewing for a new job and left shortly after that.

[ansgri]

How complicated was the bureocracy that you couldn't push the change into production yourself after verifying that it is a strict speed-up and doesn't break anything? I think such barriers are incompatible with the word 'research', where the first you need is freedom.

[ht_th]

Research is highly competitive business mixed with industry involvement (or government involvement). You have to publish and fast. You have to develop your discoveries into something that can be monetized. You have to collaborate with industry to get funded. You have to cut costs to keep doing what you want to do. And so on. The idea of freedom in (fundamental) research seems long dead. How I long for the freedom in the research labs in the first half of the 20th century. To really explore an idea without regard for cost, returns, (publishable) results. A researcher can dream :-(

[ansgri]

> The idea of freedom in (fundamental) research seems long dead

Is it so in the US? Or where? Here in Russia it is far from true, at least in the top institutes. As long as you produce publishable results, you may do virtually whatever you want, and nowadays pretty much anything is publishable. And this way you get funding, too, because the funding agency doesn't seem to want you to solve some particular problem, it just wants to be sure your science keeps up with the world.

The downside here is that the academy usually pays bad. Thus it seems most successful labs work like 70/30 on commercial projects and "pure science". Anyway, when you work on commercial projects you usually get much more interesting results than you'd care to publish.

[ht_th]

Here in the Netherlands it is. I assumed it to be the same in the Western world, but those kind of generalizations often turn around to bite me in the ass. We, researcher in the Netherlands, have to produce as funding depends on it. Furthermore, as the government funds less and less, we have to get more funding from industry. And finally we have to try to market our research more. This all means that we can not afford to just do whatever we think is best for the only purpose of extending our knowledge. We have to think about our career and the sustainability of our research (strand) in the long run.

That does not mean that we're just lapdogs for industry or Mammon, but it does mean that we're selective in what we do and how we do it.

[latj]

Some labs are conservative because they are worried that they will not be able to reproduce the same analysis. For example, imagine that a lab had been collecting samples and executing the current code as they came in. Now imagine, two years later, someone starts drawing some conclussions based on an aggregation of the results over a petabyte of that data. On one hand, you could just say- nope, we cant reproduce the same analysis, but we can use all of our computational power for a month and reanalyze all of the data using the current packages/code. On the other hand, a more conservative idea might be to try to record the entire state of the environment when particular samples are recorded, so that in theory you could replay all that analysis- fire up the vm from 2 years ago, install the same version of all the packages, install the code with the same tags, and analysis that data set, then do the same thing for every other data set. Smaller labs I think are just hoping that no one tries to replicate their studies or asks them if they can reproduce their results.