measuring scientific progress

What does fast scientific progress look like? An unsatisfying metric is the volume of results. By that measure, scientific progress is at an all-time high because there are about 15M papers published per year. Maybe scientific progress is measured by how much money is spent on science? That doesn't seem right, because the cost of major discoveries is trending upwards¹ and thus we have to keep spending more just to maintain the same pace of discovery. Scientific productivity has declined, rather than improved, over the last 25 years.

I'd like to propose that the lifetime of scientific claims is a measure of progress: how long a statement lasts in discourse before it is overturned or modified. This measures disruption, the "Kuhnian" view of how science progresses.² Specifically, it captures claims being revised or overturned, which means we have modified our understanding of the world.

methods

I've constructed Kaplan-Meier curves, which are commonly used to present human survival in diseases. They are the main figure presented in in clinical trial papers. The x-axis is duration of a claim (time since first reported) and the y-axis is probability it is still "survived."

The zero-time point for my analysis is when a claim rises to significance and is reported in a review paper. I sampled review papers from physical sciences and medicine, stratified by decade to get enough coverage. I ended up with 151 papers across chemistry, materials, medicine, biology, agriculture, environmental science, engineering, etc.. The claims were extracted with GPT-5.6-sol (max 25 per paper).

Once I had the claims, I used PaperQA3 to search the literature and determine whether each claim had been overturned or modified. If a claim was overturned or modified, I included a DOI and an explanation in the raw data. We reported some expert assessments of this method in Skarlinski et al.³ and found that subject experts agreed 70% of the time. However, those assessments were based on much older models (gpt-4-turbo-2024-04-09).

Once the claim is overturned or significantly modified, we mark that as "death" for purposes of the Kaplan-Meier curve. I had GPT-5.6-sol review these according to a simple rubric:

Let's look at some example claims to see if they match our intuition:

This all seems reasonable, and I did iterate a few times to get this level of quality by changing the prompts, the types of claims, and the burden of evidence.

results

Here is the main Kaplan-Meier curve:

This goes back to the 1950s. A typical scientific claim has a half-life of 115 years. This is interesting to me because we often assume that once something is in the scientific record, it is correct. But actually, after 10 years, 10% of claims are overturned or significantly modified. At 100 years, about 50% have survived. Neat.

The claims used for the analysis came from 151 review papers spanning each decade from the 1950s through the 2010s. Because I'm working with open-access articles, the sample is skewed toward the 2000s and later, but each decade still contains at least 10 papers. There are 3,444 claims in total, 529 of which were modified. You can find the raw data here.

Now, we can break it out by decade and see the change in the lifetime of scientific claims. This shows how the rate of change in claims is changing as a function of decade (i.e., acceleration).

So over the last 50 years, we've gone from claims lasting hundreds of years to a few decades. The 2010s have less coverage because we have fewer events, but the trend is clearly downward. This is our "acceleration" of science: a rate of change of a rate of change. As is tradition, we can then predict a singularity where scientific progress becomes so rapid that it exceeds human comprehension. Imagine, for example, that 50% of scientific claims are rewritten every year (a half-life of 1 year), or that scientific claims are refined every day!

Assuming an exponential fit, we can regress to find these events:

field of study

I showed this analysis to wanted to see it broken out by topic. But I didn't get even sampling, so the event count/uncertainty is much higher in some fields. Here are the fields that have had at least 100 claims in the analysis:

I categorized the claims using OpenAlex's service based on DOI of originating paper. I'm not sure what there is to learn from this. I guess that fundamental biology goes a little bit faster than medicine. I would probably need to sample more evenly and just more claims to say anything deeper.

limitations

There are some clear limitations from this method. We are relying intensely on extrapolation. We don't have a long enough observational data to predict the trends, because scientific claims are pretty durable! And the acceleration trend may not continue nor be an exponential process.

Another limitation is sampling bias. My measurements are conservative because we only look at claims that became significant enough to be written in a review paper. That filters out some of the early work in a field, where publications come faster than a review can be written. I think it's a reasonable compromise because we're interested in "significant" claims only. Another sampling bias is open-source papers. That creates weird effects, where the top journals in the 1990s and prior, were likely not open source. Reviewing the articles by-hand, it seems to have worked out fine though.

And of course, we're using LLMs. I found it hard to replicate human judgement, but mostly because I had a hard time of deciding what is a "claim." One failure mode is that the claim is so specific, that it likely was not studied further after the review paper. Here's one from a 1967 review paper in materials science:

PaperQA3 couldn't find a later paper that studied compact-specific ratios so it's not clear if this claim really "survived." I did say to only include claims that are about a scientific phenomenon, rather than experimental observations, and this one is boarderline.

Here's another claim that is boaderline usable:

Basically no one else wrote a paper about curing this material under these specific conditions. And the claim "thermal curing extends π-conjugation" is obviously material dependent so that it cannot be evaluated broadly.

Anyway, you can see the raw data here. I generally find that I agree with it and the claims are actually quite amazing to read. It's truly an impressive constellation of facts.

discussion

When I was born, the accelerating impact of technology was already common knowledge and unremarkable. When you read works from roughly 1880–1930, technology progress and acceleration was an emerging idea that was still debated. In my last post, I wrote about past predictions of a technology singularity as far back as the 1860s. There is also a delightful book, The Vertigo Years, that specifically covers the mania around the turn of the twentieth century. It's a fun read about social panic over bicycles, automobiles, factories, and the spread of technology throughout the world.

One historical argument that stuck with me was Henry Adams's prediction of a singularity based on the duration of scientific "epochs." He asserted that the mechanical era was 300 years, then the electrical era was 20 years, and we were entering the "ethereal era." He died before 1918, but there was such a flurry of activity in atomic theory, quantum mechanics, and then nuclear physics after he made these predictions that his argument that we would reach a singularity of progress in 1921 was not far off. Scientific turnover was so fast that the model of the atom went through about 8 versions from 1904 to 1926. It must have been tough to be a physics textbook publisher during that time period!

After that period, progress in atomic theory and nuclear physics began to slow.

I've often wondered if science is just an exercise in mining some finite resource. We sometimes strike a lode and have rapid progress. But then, things slow down once it's mined out. A famous (notorious?) recent paper showed via multiple measurements that papers are becoming less impactful and thus science is ossifying.⁴ Another analysis found that the gap from when work is done to when a Nobel Prize is awarded is increasing, because we're having more trouble ascertaining what is significant.⁵ Some even posit that the decline in impactful discoveries is because we're just running out of science left to do!⁶

I could have believed these arguments about a decline in breakthroughs in 2015, when it looked like there were only a few bright spots (e.g., quantum computing, metal-organic frameworks, genome-wide association studies, and deep learning for vision). Science in 2015 felt an order of magnitude less impactful than the Human Genome Project or the development of the Internet, which had begun roughly 25 years earlier.

Now, of course, large language models have emerged as a hugely disruptive technology. If they hadn't emerged at the same time as diffusion models, we'd be raving about diffusion models for their ability to generate images, videos, molecular structures, and more. If neither were around, we'd be very excited about CRISPR, Gaussian splats, hardware-automation breakthroughs driven by reinforcement learning, and many more things.

So, I do not think science is finite and we will continue to see an acceleration of discovery.