Chemical Irreproducibility and Where it Comes From

Before we decided what to focus CheMastery on, we carried out a market study asking chemists of different seniority, geographical location and chemical discipline:

“What are the top 3 chemistry problems?”

The question was purposely vague to avoid biasing or leading the question. And the results were:

In first place: irreproducibility,

second place: cost/time inefficiency

and third place: scale-up

No surprises here.

Unfortunately, irreproducibility shouldn’t come as a shock to anyone. It is a big problem and has been around for a while. Every scientist you talk to, chemistry or other industry, will have stories to share.

We were also of course, not the first to spot this. There were many people that talked about irreproducibility and many published studies that captured this before CheMastery even existed.

One of the most famous articles was published by M. Baker in Nature in 2016 which showed that ~86% of chemists experienced irreproducibility. It also looked at other STEM disciplines which also admitted to the problem but fared a little better. Another notable advocate for these issues is Derek Lowe, check out his “Thoughts on Reproducibility” article. But the list doesn’t stop here, there are many more.

Frankly, we’ve never met a chemist who hasn’t experienced this, and we’ve talked to many biologists who tried to argue that they have it even worse than chemistry.

What an embarrassing debate to have!

However, it was the rest of our conversations with these chemists that were particularly interesting.

While the vast majority of chemists agreed on irreproducibility being a problem, there were fundamental differences in where the blame is pointed and what they do about it.

These differences in thinking and action (or lack of thereof) are incredibly important not only for understanding the problem but also in how they shape the future of chemistry.

Some chemists we talked to blamed irreproducibility on the quality of materials supplied by vendors, others blamed their colleagues, some blamed the flawed academic peer-review publication system and many of them were honest enough to admit that regardless of the multiple flaws in the chemical supply and information chain, they also couldn’t reproduce their own work sometimes.

The reality is, all of the above are true. Such fundamental problems are often complex and multi-sourced. So, let’s dig a little deeper.

Suppliers

Supplier batch-to-batch reproducibility of chemicals is not always reliable and there are many contributing factors to that. The most straightforward case and arguably the main challenge is how these materials are made.

It is important to note first that we are referring to specialty chemicals here, and by specialty chemicals we mean materials that are made on the milligram, gram or low kilo scale. Basic chemicals that are made in larger quantities are often made in chemical plants which follow some procedures and regulations, and hence are often reliable.

Specialty chemicals however, which account to ~99% of unique molecules made, are often synthesised and purified by manual execution and this comes with its challenges. Not only in batch-to-batch quality but also cost, speed of delivery and other issues like traceability and transparency in how and where they are made.

Another noteworthy factor is that many vendors sell chemicals that are not made by them, and this leads to inconsistencies in information, quality and execution. Very few suppliers check all their materials via analytical methods to verify or even guarantee quality, even fewer then even share this information with their customers and similarly, very few have full knowledge of the chemical pathways and methods that created them.

However, the knowledge of how a material was made can be extremely powerful, especially in the reproducibility context, as there can be cases where it makes a difference. For example, traces of metals present from the synthesis of these materials (or other impurities that are hard to spot in analytical data) might be destroying or catalysing a future reaction or application. And for some customers, the method or location of manufacture might be part of environmental, sustainability or other policies too.

Irreproducibility in manual execution and between colleagues

Difficulties in reproducibility between colleagues, departments or companies are also very common and stem from many different factors including: differences in training and interpretation of procedures, differences in experience, skill and knowledge, attention to detail, unstandardised experimental descriptions, unmonitored day to day variations etc.

Most of these are quite straightforward to understand and notice but much harder to solve. Chemists coming from different Universities or other training programs will have vast differences in knowledge and even in practical execution. For example, two chemists will often setup a reaction differently: do you start the reaction time when you set it up or when it’s actually refluxing, do you clamp the reflux condenser or not, do you add via syringe, pipette, a flask or a dropping funnel etc.

Some decisions are more fundamental and impactful than others, but the tricky part is that we don’t always know which of these factors are important and which are not, in every single case.

For example, in some cases stir rate might make a huge difference and is especially important in scale-up and manufacture, but R&D chemists very rarely note or test this in their procedures. In other cases it might be the order of addition of certain chemicals that perhaps form an intermediate which then reacts with the third etc. The best bet for reproducibility and traceability (as well as future proofing manufacture) is to capture absolutely everything. But that is hard, some would even argue unfeasible, to document and execute manually.

If I gave you the perfect recipe that is 25 pages long and had every single piece of information you needed to reproduce something, would you read it? And would you follow it to the letter? Chances are you will skim read, interpret according to your training and execute some parts according to your experience. That is human nature.

Humans are great for intelligence but not for repetitive work, and chemistry is by its nature practically repetitive. Even when you do research.

Setting up another reaction, doing another filtration, then another crystallisation. The mental work might be new, but the practical work is the same techniques over and over again. The absolute majority of organic chemistry requires: heating or cooling, stirring, adding a reagent, an organic workup, a filtration, a crystallisation, column chromatography or distillation. These techniques are the workhorses of organic chemistry and with these you can make the majority of molecules.

How many more filtrations can you do before you master the skill. We get bored, we loose focus, we do things fast, we skim read papers. That’s what chemists do, that’s what humans do.

There are of course many other factors that contribute to irreproducibility, like for example the fact that environmental conditions are rarely taken into consideration.

Some reactions require a certain level of moisture even under inert gas. UV and how bright it is in the lab that day can make a huge difference as well as baseline parameters that are not that basic! A famous example that every chemist is guilty of is setting up a reaction at room temperature. Room temperature has become equivalent to “I haven’t used external heating or cooling” but how many chemists actually note if it was, by scientific definition, 20 degrees Celsius on that day? There are many other examples like this one and in some cases they can be very impactful.

A lot of these execution variations also explain the phenomenon of the “golden hands”. In case you haven’t heard of this, “golden hands” is this one chemist who is incredibly good at executing a particular reaction when everyone else fails at it. Scientifically, there is no reason why that should be the case other than they know or do something that the rest of the chemists don’t. Something about the way the procedure is executed is different and it is not captured in the experimental description, leading other chemists to be unable to reproduce.

It makes skill and knowledge person-specific, rather than organisation specific, and for the business minded out there, this costs a lot of money and time.

The lack of industry-wide standards on how we communicate and describe experimental procedures is a big part of reproducibility. It widens the already different ways of executing a reaction and allows room for interpretation. Science should be factual. “Science is show me, not trust me”, another great post on reproducibility, by Philip B.Stark at Berkeley.

Here's an example from our CEO: "I’ve previously encountered an experimental from the literature which did not specify a temperature and my colleagues failed to produce the material while following the procedure. When I read it I could immediately spot the problem, which was that it should have been executed at-78 degrees Celsius instead of what my colleagues assumed (which was a fair assumption): at the lack of temperature specification, try room temperature. This information was obvious to me because I had experience in that reaction type, but my colleagues didn’t. While experience is important and makes a huge difference, in a world where trillions of reactions exist in the chemical space, it is impossible for chemists to know it all."

Before you dismiss this example as a bad chemist, who after all doesn’t mention one of the most fundamental of things, temperature. We did a study of chemical databases and found that a significant amount of experiments do not specify temperature (% varies from database to database).It’s not just a single bad chemist or an attention to detail issue or an assumption, interpretation or experience problem.

It all comes down to lack of standardisation.

Documentation and communication have to be standardised even between specialised personnel.

These are some of the reasons why Good Manufacturing Practice (GMP), Chemistry Manufacturing Controls (CMC) and International Organization Standardisation (ISO) processes exist but unfortunately, these are rarely followed in R&D settings and many of them are generic and organisational rather than material specific.

Lastly, as this post is getting long, through our chemist interviews we also heard about another manifestation of irreproducibility, this time between organisations.

Contract Research Organisations (CROs) and Enterprises alike both told us that when they work with each other they often have difficulties in skill and knowledge transfer.

For example, a Pharma pays a CRO to develop some work for them. It usually takes months (sometimes longer) and when the work is delivered and the Pharma tries to continue that work internally, they can’t always reproduce it. Similarly, we’ve also heard about this problem from CROs who are upset with being blamed for irreproducibility when it might have been the Pharma’s chemist fault.

We also heard from Enterprises themselves.

It's very common for organisations to have difficulties reproducing work between their departments and between sites, sometimes in different countries.

It’s a combination of differences in batch of materials, chemist skill and knowledge, experience, assumptions, lack of communication and execution standards, the inevitable flaws that stem from manual execution, environmental differences etc.

Irreproducibility in peer-reviewed publications

The peer-review system for academic publications is flawed in many ways. It’s not a secret and yet this is another fundamental problem that very few in the industry are trying to change. However, this is a large and complex subject for another post.

If you want to read more about this we recommend the article “Is the staggeringly profitable business of scientific publishing bad for science” written by Stephen Buranyi and published in The Guardian in 2017.

Irreproducibility is a large and complex problem and as we explored, it can have many different manifestations and often multiple reasons at the same time.

However, although this is a problem that the industry generally agrees on, very few businesses actually track its impact on their cost, speed of discovery and retention of skill and knowledge.

The good news is that digitisation and automation of chemistry addresses all of the above. We experienced these problems first hand multiple times and with these in mind we created an end-to-end automated system which standardises the information of an experiment, both in content and language, and then executes the work reproducibly while using scale-up processes to ensure manufacture applicability.

Today’s reality requires chemists to do more. Discover more, faster, cheaper, better. This fast frenzy combined with manual execution is a recipe for disaster.

Industrial revolution created automation to lower costs, improve standards and meet the demand. Why isn’t science doing the same?

‍

‍