Nature reported yesterday a new paper by Russell Funk on research innovation or “disruptiveness”
The number of science and technology research papers published has skyrocketed over the past few decades — but the ‘disruptiveness’ of those papers has dropped, according to an analysis of how radically papers depart from the previous literature.
Data from millions of manuscripts show that, compared with the mid-twentieth century, research done in the 2000s was much more likely to incrementally push science forward than to veer off in a new direction and render previous work obsolete. Analysis of patents from 1976 to 2010 showed the same trend.
So has everything already discovered by getting most low hanging fruits (A)? Are scientists no more taking any risk (B)? Or is the “disruptive” science now hidden in the meaningless research (C)? Or did only change citation practices (D)? The answer is in the original paper
Specifically, despite large increases in scientific productivity, the number of papers and patents with CD5 values in the far right tail of the distribution remains nearly constant over time. This ‘conservation’ of the absolute number of highly disruptive papers and patents holds despite considerable churn in the underlying fields responsible for producing those works… These results suggest that the persistence of major breakthroughs—for example, measurement of gravity waves and COVID-19 vaccines—is not inconsistent with slowing innovative activity. In short, declining aggregate disruptiveness does not preclude individual highly disruptive works.
In my own words: Progress is found in the top percentiles just as many decades before. But most research publications are a waste of money and even harmful for cluttering up the research field.
There seem to be also some critical comments and of course some methodological issues need to be clarified for any further interpretation (eg exclusion of reviews, validity of the 5 year interval, …). 5 years may not be enough in some fields, medical practice doesn’t even change for a long time – see also the comment of Bruce Albert. In any case, the authors promised to give me the CD5 dataset which will be nice to look up my own work.
20 Feb 2023
Forgot to update this post as there is an option E – that the study is just describing an artefact,.. I received the dataset one week later but couldn’t verify the claims when analyzing my own “disruption score”. Upon inquiry RF said that PubMed doesn’t include citations for all papers. “So to drop these papers from the data, required that papers had at least 1 reference in their reference list, and had been cited at least 1 time”.
The numbers were however still confusing as are 2.3 million entries in the CD5 file while Pubmed had roughly 18 millions entries in 2010 according to https://www.nlm.nih.gov/archive/20110328/bsd/licensee/2010_stats/2010_LO.html.notice.html. So I asked if the discrepancy may be explained by an additional constraint? RF explained that “For the Nature paper, we only analyzed data up through 2010, for consistency with the other data sets used in the paper. But we computed the measure for more recent years” which may have led to the missing scores.
A colleague also wrote about the study later in a German magazine https://www.laborjournal.de/rubric/narr/narr/n_23_03.php basically saying that science is not disruptive, it builds nearly always on earlier ground work: “Disruptive is economic gobbledegook”.
Interestingly and only last week I learned about another much more extensive reanalysis that arrives at very similar conclusions “Dataset Artefacts are the Hidden Drivers of the Declining Disruptiveness in Science“. Holst et al. describe in this paper
Our reanalysis shows that the reported decline in disruptiveness can be attributed to a relative decline of these database entries with zero references. … Proper evaluation of the Monte-Carlo simulations reveals that, because of the preservation of the hidden outliers, even random citation behaviour replicates the observed decline in disruptiveness.
And well, there is now also a PubPeer entry but only from the last year.
