[Vaslo]

In my former life I was a graduate organic chemist, but I “quit” with a Masters. I loved Synthetic Chemistry as an undergrad and thought I wanted a PhD in. There were three lousy things (for me) about the career that made me bail.

1). So many syntheses have horrible yields just like this one. You’d start with grams of material to end up with micrograms. I loved solving these problems as an undergrad in books, but reality was far different. You don’t think much about side products until you start doing novel chemistry.

2). So much trial and error. There were happy go lucky chemists that fell into projects that were smooth as butter, while brilliant chemists would toil 12 hour days to try and something to write up as a thesis. I was neither brilliant nor lucky and took 4 different projects over two years before finally landing on something marginally MS worthy. They need a journal of failed chemistry because only the working stuff gets published. So many failures could be logged so I didn’t waste my time doing non-working or poor yielding reactions.

3). Suspicious results in journals. I would read about a reaction and someone would put a 75% yield as their result and I could barely get 20. I always thought I was just bad, but a really smart chemist challenged me one day and tried to do it himself and couldn’t do much better. He tried it 30 different ways over the year as he did other stuff. He never could get a good yield. We talked to our advisor and we wanted to challenge the result, but the advisor didn’t want to start trouble. It was past the time I decided to leave with a masters, but made me feel a little better about my lousy abilities. No one could ever possibly doublecheck every result from every publication anyways.

All this said, there are some brilliant and patient scientists out there that drive the field forward. Just a few rough around the edge items I’d love to see change.

[girvo]

> So many failures could be logged so I didn’t waste my time doing non-working or poor yielding reactions

In a different but related world, clandestine chemists share failures more often than they do their successes, at least in the communities I was a part of a very long time ago.

I'd wager this is because our substrates, reagents and solvents are such a pain in the arse to get compared to an actual lab, that wasting any of them is a no-go if it can be avoided.

Related to that, but we reused solvents and recycled material a lot more than I did doing my B.Sci in Chemistry, for the same reasons!

[orbifold]

I actually think it is the same in chemistry, knowledge of which approaches don't work are a competitive advantage that is passed around under the hand. I've definitely observed this is in theoretical physics and pure mathematics, it is often much more important to know about things that have never been published then it is to read close to incomprehensible publications. So for example it is much more valuable to have attended the right seminars and have private copies of lecture notes, because by the time something is published it probably is out of date 1-3 years.

[disillusioned]

Do they then go on to be script consultants for Breaking Bad?

[djaque]

That's kind of my experience with research, but in physics. Already have three projects under my belt w/ no publishable results. All of them involved much more work than the other grad. students in my lab including building like three different instruments from scratch, but the other students have published. Their projects were much more "safe" than mine and their papers are pretty small, but at least they have papers.

[joshvm]

Don't worry. Instrumentation is a pain for publishing because often it creeps into engineering territory. I spent a year of my PhD trying to get something to work, it didn't and I ended up building an entirely new instrument in the third year. That's how it goes. I know people who came out of their PhDs with five papers (in the UK that's a big deal), but they're in fields where its relatively easy to publish and/or got lucky with their topic. Or you work with someone with a funky new dataset that breeds publications.

I think most instrumentation scientists are sympathetic to this. There are also lots of instrumentation jobs that don't require stupid numbers of publications because it's not practical to find candidates. There are relatively few good hardware people in the sciences (especially fields where you can't get a mech/electrical engineer).

I suggest finding some conferences to start. They're a good venue for telling people what you did. There are also journals specifically for building stuff, SPIE has a lot for astrophysics, for example.

[djaque]

Thanks, that's good to hear. Fortunately I do have two papers in the works right now, one of them from an instrument I made and another from some simulation work.

The engineering really is the crux of the problem. I personally find it very interesting and I feel like an oddity in my department for that. Most other physicists kind of look down on the "e-word" to the point that my advisors made me change references to "engineering challenges" into "experimental challenges" for a conference talk I gave. I'm in a very equipment heavy field too which is why I find it so strange.

What's frustrating though is that sometimes it feels like I'm only doing engineering with no physics in the mix. Hopefully thay will change over the next year though.

[cushychicken]

What field would you go into if you were interested in making instruments for scientific research? Would that be engineering physics?

[joshvm]

Instrumentation is field-specific. It really depends how bespoke your requirements are. Some places buy stuff off the shelf, others fabricate in-house.

In astronomy it's called instrumentation. Astro has relatively sane nomenclature. You have observational (looking at stuff and gathering data), theoretical (theory and simulations) and instrumentation (building stuff). Everyone in the field understands those terms. That said, instrumentation spans everything from construction, design, calibration/characterisation, etc.

In other branches, as the OP pointed out it may be "experimental" physics. Problem with that term is it's extremely vague. That's essentially anything which doesn't involve theory or simulation - lab work. That doesn't necessarily mean you actually build anything though. I was surprised at how little we got taught about instrumentation during my physics degree. Of course we had labs, but it was more about using kit than how it was built. It's almost as if it's someone else's problem, even though that's not how research actually works.

The irony of physicists sneering at engineers is that most labs employ a bunch of them who do most of the actual design and build work. They tend to be less focused on publication, but they are absolutely critical employees. The sorts of people who've spent 20-30 years working on some uber niche detector tech and know more than the people writing the papers for sure. A good chunk have physics PhDs.

When you get to big money stuff like particle accelerators and telescopes, a lot of this is contracted out. Physicists and engineers are responsible for designing the spec, but it's less tinkering in a lab with hardware. In industry it's mostly called engineering (e.g. optical, mechanical, electrical), but I've seen engineering physics too. That tends to be in companies that build very niche equipment specifically for physics research.

[djaque]

Exactly, the instruments I was talking about are used to measure the properties of photocathodes. It's such a niche field that there's no off the shelf solution. Everybody makes their own device and ours is one of like two in the world with it's grade of sensitivity.

There is one professional engineer in the lab my group is a part of, but they primarily work on the bigger projects. For this, I have to do all of the CAD work and talking to the machinists. I also machined a lot of the parts myself due to time constraints.

[djaque]

I'm in accelerator physics. Basically we study the physics of charged particle beams with the goal of improving their properties/developing new applications. I thought it was the perfect combination of engineering, which I like, with pure physics.

[wodenokoto]

I never got published in the field I have a masters in, for what I'd like to think of as somewhat comparable reasons.

However, my take-away was that the successful researchers were the ones who could take any decent experiment and figure out what was publishable about it, or at least steer it into a publishable direction.

[dguest]

What branch of physics? There are hundreds of published null results looking for dark matter or Supersymmetry, for example. Instead of documenting a failed search, the papers focus on what versions of the theories the experiment excludes to 95% confidence.

[djaque]

I'm in acceleraror physics and study photocathodes. Unfortunately the state of those projects is that we thought a material should have great photoemission properties under some very specific circumstances.

When we measured it, the material was actually terrible and we don't understand why it's that way outside of some speculation. Unfortunately it also feels like our lab doesn't have the type of expertise to explain why it's so bad either at least not without years of learning.

My advisor's explanation on publishing was that if we had great results that followed our initial theory of what's going on then we could write a paper and nobody would question it. But, now that we contradict theory the threshold of evidence is much higher and he doesn't want to go ahead with it.

We trust our data, but he's concerned about getting it through peer review. For what it's worth we did publish it in a conference proceeding, but it doesn't have the same weight as a journal article.

[darawk]

> My advisor's explanation on publishing was that if we had great results that followed our initial theory of what's going on then we could write a paper and nobody would question it. But, now that we contradict theory the threshold of evidence is much higher and he doesn't want to go ahead with it.

Isn't that sort of the most important thing to publish? Something that contradicts theory means something new. Seems like all the more reason to pursue it, no?

[missosoup]

In a vacuum yes. In the world of academia and its perverse incentives, no.

Publishing something that goes against the grain means a lot more scrutiny on your work and bigger humiliation for any mistakes made. If you're publishing against the grain, you have to double triple check every single detail to make sure it's all iron-clad. This costs a lot of additional time and money. When publishing research that preaches to the choir, no such considerations are required.

In an ideal world, publishing against the grain should be encouraged and there should be no extra reputational penalty for getting caught with errors in that type of research (heaps of errors go unnoticed in mainstream papers, even popular ones). But that's not how it works sadly.

[jschwartzi]

Really academia needs a way to request comment on an experimental procedure without requiring results or conclusions. Then you could publish negative or questionable results and seek speculation from other researchers.

[ridaj]

It reminds me of the paper reporting that neutrinos had been detected going faster than the speed of light[0]. It turned out to be an instrumentation issue. Some of the folks involved had to step down.

Yes, it is important to publish results that violate established theories, but if there's already a lot of validation of those theories through experimentation, you're going to have legitimate pushback, especially if your method of experimentation isn't the most sophisticated.

If the result of your publication is a theoretical breakthrough, then that's great, but unless you're in the top tier within that discipline, the likely outcome is that your research turns out to have a big flaw which you missed, and you invite a bunch of people to point out to the world that you can't do science properly...

[0] https://www.sciencemag.org/news/2012/06/once-again-physicist...

[djaque]

Thanks, this is in line with what I feel is going on. I woke up this morning to see a lot of comments on how my experience is proof that academia is broken. Instead, it just feels like everybody is being very cautious about our results.

We don't understand why our data doesn't line up with theory and this isn't some super well funded experiment like the ones at the LHC with like a million grad. students. This is just me in the lab alone with the hundreds of subtle ways the experiment could have gone wrong or needed a slight modification to the theory. It would take us a lot of effort to distinguish an experiment gone wrong from an interesting new development and that's not always worth it. Especially for such a niche area of research like I'm in.

Also, we did publish our data. We just did it in a conference proceeding with a lot of caveats attached to it instead of in a big journal article. That's the aspect that I was complaining about, but I do understand my advisor's decision.

I have to strongly disagree that science is broken here.

[michaelt]

If the theory says it's possible to juggle five chainsaws but nobody's managed to demonstrate it, and I try it without success, have I proven it impossible? Or have I just proven I'm not good a enough juggler?

[djaque]

Exactly, that's the state of the experiment. There's just so many things that could have gone wrong without us noticing. Even though we trust the instrument is working, we don't know if the material under study is exactly as we expect.

[BurningFrog]

Yet another example of how 2020 institutional science is broken.

For the purpose of producing knowledge. It does other things.

[djaque]

Could you please explain why you think this means science is broken?

There are real issues in academia, but I disagree that what happened here is wrong. We did publish our data, we just didn't do it in a journal article. In my fields, this is actually fairly typical and journals are less of the norm unless you have a big result.

The whole problem is that extraordinary claims require extraordinary evidence. Sometimes when you don't have much funding and there's only a few grad. students in the lab, that evidence is no longer worth the time to collect.

See the comments below about the "neutrinos travel faster than the speed of light" paper. There is a real danger in publishing theory-breaking results when you don't have 100% confidence in them.

[BurningFrog]

> Could you please explain why you think this means science is broken?

To be precise, I already think science is broken, and your story seemed to confirm it.

Obviously, that "this is actually fairly typical" is not a counterargument to a systemic belief.

That said, it's entirely possible I jumped the gun here.

I take the reproducibility crisis to be a strong indicator that something is Very Wrong with science. I'm also convinced that the modern peer review system is fundamentally broken.

[101404]

Why would the results not be publishable?

[wenc]

> They need a journal of failed chemistry because only the working stuff gets published.

I think there's theoretical value in this, and many have tried, but the incentives/disincentives for doing so isn't favorable. Here's some reasons why I think it's difficult to motivate people to publish negative results:

1) Negative results, while important in advancing science, don't get you grants.

2) Negative results need to be peer-reviewed -- there are "good" negative results (good protocol, failed result) and "bad" negative results due to bad data collection, wrong conclusions (bad protocol, failed result).

3) Given that it's so much easier to get a negative result than a positive one (as in anything there are only few ways to be right, tons of ways to be wrong), the volume of papers to review is orders of magnitude higher. Reviewers have to really sift to find the needle in the haystack. Between teaching classes, sitting on mindless committees, writing grants, mentoring grad students, etc. academics don't have that kind of time.

4) It would incentivize poor/mediocre labs to publish a lot of negative results to get their pub count up (these are the ones that currently publish unsubstantiated positive results in fly-by-night journals).

5) Bad faith authors may publish fake negative results to throw others off a promising line of inquiry.

(Note: some of these disincentives also apply to positive results in journals today)

The current method I know for exchanging "good" negative results is word-of-mouth, usually during post-conference drinks at the bar. (works for tech too!) I'm not sure if it's possible to arrange incentives in such a way as to make publishing "good" negative results worthwhile.

EDIT: there are exceptions. If the space of solutions is known a priori and bounded (say only n ways to do something), then publishing n results if even all n are failures is worthwhile. This situation doesn't come up all the time (the solution space is often open), but when it does, it's worth publishing all n results.

[ggcdn]

Maybe instead of full-fledged journal model, negative results should be published in a short (few pages) failure report. This addresses at least a few of the points you highlight. Skip all the cruft: here’s what we tried to do, here’s our methodology, and here’s the result.

Then these could be classified by methodology/process/chemicals/etc for people to look up before starting their research.

[wenc]

Yes, sounds like that could work. Like a wikipedia of experiments tried, with contact details and a public discussion forum for anyone interested in going deeper into the results.

In many fields, you can generally tell if someone is faking it about from how they respond and what they say. It's not that easy to fake specialized knowledge. This will help increase the signal-to-noise ratio.

The trick is to bootstrap a high-quality community, where top people in the field will want to engage. MathOverflow managed to do this for pure math. Quora on the other hand used to be good in the early days (many SV names) but has struggled to maintain quality. There's also the problem of moderating professional rivalry and reputation-maintenance in academia -- these are non-trivial issues in some smaller fields.

[Vaslo]

Many really good points here I hadn’t thought much about.

[avian]

As someone who spent 4 years working on a PhD (wireless/RF technology) and then quit, your points 2 and 3 ring a lot of bells. I think "journal of failed experiments" is a wonderful idea.

[dguest]

Anyone can put a paper on arXiv. The bigger problem is that most journal editors will be skeptical of citing arXiv. If you're collaborating with more traditional academics it can be a challenge to even submit a paper with such citations.

I've had to push my colleagues to cite a number of non-traditional sources: arXiv, github, and zenodo for example. Fortunately most of them agree that citations are cheap and that giving more people credit is generally a good thing.

One thing that helps is publicly stating how you want your research cited. If you don't have a peer reviewed publication in the pipeline, tell people how to cite your work on a blog or your github page or somewhere. Most people default to peer-reviewed journals for citations and get confused when one doesn't exist, so an explicit statement really helps.

[mattkrause]

I think that's less true than it used to be: a lot of papers now cite bioarxiv or arxiv preprints. I've never gotten any pushback and most journals explicitly allow it.

Science: "We do allow citations to papers posted at arXiv or bioRxiv." https://www.sciencemag.org/authors/instructions-preparing-in...

Nature (and other NPG journals): "Preprints may be cited in the reference list of articles under consideration at Nature Research journals...." https://www.nature.com/nature-research/editorial-policies/pr...

Cell (and other Cell Press journals, including Neuron and Current Biology): "Posted preprints may also be included in the References list with appropriate identification information...." https://www.cell.com/cell/authors

PNAS: "Preprints are cited as follows with a DOI or preprint ID number, and the date of posting...."https://www.cell.com/cell/authors

[avian]

> Anyone can put a paper on arXiv.

Unfortunately as an academic it's really hard to justify spending time on writing up a paper that's only going to be published on arXiv. Regardless of citations, it doesn't count towards your publication quotas or PhD requirements.

[jcelerier]

already has existed for some time : https://blogs.plos.org/collections/positively-negative-new-p...

[nython]

> There were happy go lucky chemists that fell into projects that were smooth as butter

If it makes you feel any better (I 'quit' at the start of my bioinformatics PhD), there's a high chance that their projects went smooth as butter because they were also happy-go-lucky about double-checking their results. See for example that 75% yield paper you mentioned.

[z3t4]

I sometimes bake stuff. But had problems with one particular recipe. When changed the proportions I always failed. Tried it over and over. Only to find out one of the baking measurements that's over 20 years old showed the correct measure at 1dl but incorrect on 2 and 2.5 dl. And the recipe only worked when I used the incorrect measurement.

[fouc]

Sharing failures seems like the best way to make progress. I mean we're talking about SCIENCE here. If we don't have a comprehensive list of what does not work, that's just going to make progress so much slower.

[spockz]

I also wonder how many breakthroughs we would not have because of somebody saying that something didn’t work.

[fouc]

That's true. Failures need to be reproduced too.

[eru]

Thanks for the insights! Sounds very much like the replication crisis in the social sciences.

I got similar stories about how to (not..) grow mammalian cells in vats.

[wesleywt]

You are describing my life. Except I am in biotechnology.

[dguest]

In point 3: did you try to contact the authors of the paper you were trying to reproduce?

I don't know organic chemistry, but in my field the authors are often willing to discuss their results, especially if it might mean more citations for them.

[tralarpa]

May I ask what your field is? Is it CS? I am asking because we computer scientists are sometimes a little bit naive when it comes to research and publications. In other fields, like pharmaceutics, where future patents can be worth hundreds of millions of dollars, authors are very careful what they tell others.

[dguest]

> May I ask what your field is?

Collider physics. I'm guessing that a lot of the less applied sciences are the same way: we're looking for fundamental laws so it's more about convincing people that your result is valid than about trying to make money off of some application.

It's very hit or miss of course: there are still some groups that act very secretive and some that are more open.

[lazyjones]

As an IT guy, I am wondering why this is still a mostly manual and rather dangerous process. Surely programmable machines can be built to process reagents in a safe and flexible way?

[tikej]

People are working on it: https://www.gla.ac.uk/news/archiveofnews/2018/november/headl...

I am chemical engineer and in my studies there was no programming and algorithm training at all (we did tiny bit of Scilab to solve some systems of equations but that’s all!).

Chemists in general (beyond theoretical and some open-minded exeptions) don’t program and don’t want to program. Synthetic/organic chemist still perceive synthesis as form of “art” ;) Therefore it would require huge shift in the mentality.

It is going to happen but not easily and and later than it could for social reasons :(

[LeifCarrotson]

I'm a controls engineer. We're very good at making machines that will do a single process over and over. It's insanely difficult to do arbitrary processes with one machine.

[wuschel]

Hi,

That is a very good question. In my PhD I built, among other things, a very primitive parallel stirring system in order to speed up the synthesis of test batches. Although a very crude device, there was tons of stuff that I needed to optimize.

There are many steps in the practical work that is chemical synthesis. It might be removing air, adding reagents in different elemental states, cooling/heating/keeping and certain temperature, observing the reaction, taking aliquots, terminating reaction, and the many steps of purification.

I'd argue that while automation is possible to a certain degree (continuous reactor systems are the most the most interesting IMHO), the resulting mashines are always problem (reagent, reaction, etc) specific. And here lies the problem.

[marcosdumay]

Chemistry uses a very large set of "primitive steps" (from reactants and catalyzers to reactors and reaction conditions), so any such machine will have a narrow set of reactions it can produce. Unless of course, your automation machine has the capacity of assembling and dissembling equipment.

Besides, cleaning everything is very machine unfriendly.

[EL_Loco]

There are machines that do this. Prety expensive though. Maybe too expensive for a uni's department where there won't be enough work to justify the machine's price?

[lazyjones]

Not sure how flexible this one is, but it looks cheap enough for a small department to make their own sildenafil: https://science.sciencemag.org/content/363/6423/eaav2211.ful...

The yields are quite bad though AFAICT.

[londons_explore]

This is especially true since the number of possible "instructions" is pretty low. Heat it, cool it, mix it, filter it, centrifuge it, compress it, take a sample and put it through a test instrument.

Pretty much all chemistry is some sequence of the above.

[girvo]

But your number of inputs is huge, and cleaning steps in between others in a reusable/reconfigurable machine is a difficult challenge to overcome all on its own. It’s not impossible and smart people are working on it, but it is incredibly hard.

[foobarian]

Something that is not obvious from article or posts in this thread, but seems to be taken for granted: how exactly do you know how much of some target molecule you created? The article for example mentions 1g of remdesivir, but that seems like a complicated chemical so how do they tell it apart from something very similar but different? Some kind of GCM or electron microscope?

[Metacelsus]

NMR and mass spectrometry are two powerful analytical tools, both of which they used

[wuschel]

Former chemist here. What are you doing in this life? Feel free to contact me - see email in profile.

[rwmj]

To what extent can synthesis steps be automated? Do you use a computer to plan how to get to a chemical? Do you use robots to carry out some or all steps?

[simlan]

In a usual lab no automation whatsoever. When something goes commercial there are two routes. One is adapting to bigger batches where you will get some automation (dosing, temperatur controll, mixing). The last step is to have full blown process where you pipe stuf from reaction a to b. However, that last one works best with large scale basic chemicals. Think all kinds of polymer materials.

There is a trend to get pharmaceuticals to a stage where flow chemistry can be used (like a small version of the full blown basic chemicals processes). This is however still a research field because a lot of processes don't lend themselves to continue flow.

The most automation in chemistry can be found in the analytical side of things. A good example right now are the covid tests that are run on large automated liquid handling systems.

[CamperBob2]

I asked this question once. I couldn't figure out why there's no SPICE for chemistry (SPICE being any of several accurate, refined, and well-understood simulation programs for EE work).

You know the whole deal with the three-body problem? How closed-form solutions become intractable in a hurry once you go past two or three mutually-influential orbiting bodies? As it was explained to me, that's why there's no SPICE for chemistry. Modeling exactly what happens when complex orbitals with dozens or hundreds of electrons interact with each other is one of those things we just have no clue how to implement in a practical application.

[tikej]

(Quantum) Many body problem is indeed at the very heart of chemistry. If it was solved creation of new materials and chemicals would be less of trial and error.

In principle it comes down to the fact that problem space explodes tensorially with number of electrons e.g. in principle you need grid size of 3N dimensions per every electron so for helium a^6 where a is number of points in the grid (and you probably need at least thousands of tens of thousands to accurately solve differential equations). This can’t be done in practice so other methods (expansion of problem in basis usually) are used.

Unfortunately solving Schrödinger precisely requires incredible amounts of both computing power and theoretical expertise. Qualitative results are up there for most systems thanks to developments in theoretical chemistry but it is still quite manual process that requires expert computational scientist to make sure results are reliable.

[saalweachter]

I believe I've seen occasional reports on chemistry sims on HN; the last I saw, someone had managed to simulate an "interesting" number of atom at a rate of 1 ns / week.

[OscarCunningham]

Excitingly, this is one of the problems that quantum computers will be good at. We might see a time when chemists can run experiments faster on a computer than they can in the lab.

[godzillabrennus]

A great deal of published research is not reproducible regardless of field.

[twomoretime]

Your anecdote seems to confirm a number of problems which are ravaging academia at large.

Having gone through a master's myself, I think when we pushed two generations of children into college, we generally lowered the bar - across the board, effectively. And that's related to what's happening in the US today.