The progress of AI is starting to feel inevitable. Many watch the monthly release of the METR plot, which shows the progress of AI models in completing human tasks ordered by the difficulty of the tasks.¹ The progress is faster than exponential: it is super-linear on a log-plot.² At the time of writing this, it appears to be doubling in capability every 4 months.³ And that doubling time will decrease again.³ AI models can now do 16 hours of human work today and in 4 months it will be 32 hours and maybe 3 months after that it will be 64 hours.¹

This month, an internal OpenAI model was announced to have disproven a famous and well-studied math conjecture in discrete geometry with one model output.⁴⁵ Just a long stream of tokens leading to a counter-example.⁶ Demis Hassabis, who won the Nobel Prize for AlphaFold,⁷ said this month we are entering the foothills of the singularity.⁸ Andrej Karpathy, the closest thing we have to a universally-loved AI celebrity, unexpectedly joined Anthropic to work on recursive improvements for AI models.⁹ It appears we are on the brink of runaway intelligence with recursive improvement of AI research.

I would like to present a different view on where the world is headed. We have arrived at the invention of LLMs because we have been already on an exponential ride of progress in science and technology. The last 100 years has been a stream of scientific and industrial revolutions. During this era, human intelligence has not increased to any significant degree.¹⁰ Instead, we have been accumulating scientific discoveries over time. And built upon them. We have heavily relied on the scientific method. Scientists do experiments and reason about their outcomes.

Human history is direct evidence that massive progress in science and technology isn't necessarily coupled to increasing intelligence.

I believe there are diminishing returns to intelligence. The best hypotheses do not survive first contact with experimental results. I know Nobel Prize winners and have talked to them about discoveries. They are brilliant, but not in some weird inscrutable way depicted in media. They have a regular stream of good ideas and have smart people test them, and the ideas often don't work.

A reason intelligence will have diminishing returns is how complex the physical world is. A single cell contains so much complexity that, ignoring quantum effects, it would take 10^38 FLOPs to simulate it for one day.¹¹ That would require about 20x the current energy generation capacity of Earth, assuming an infinite supply of GPUs.¹² A future artificial super-intelligences might be able to use various approximations to simulate a cell and predict its properties from the genome, but a high school student can just grow a few and measure them. The scaling of intelligence will never exceed the information from empirical measurements.

I'm not arguing that models will not continue to grow in intelligence. My argument is that intelligence is no longer the rate-limiting step in making discoveries, and thus not solely increasing the rate of scientific progress. Scientific discoveries loosely require intelligence, logistics, and capital. You need a good hypothesis and experimental design. You need to buy the ingredients for an experiment. Then you need to do the experiment and analyze the results.

I believe we are now unblocked in the intelligence part of this process. That math conjecture I mentioned above? It was reproduced by someone with their ChatGPT subscription for $1 of tokens with a few hints (i.e., assume the conjecture is false).¹³⁶ You don't need some secret model internal to a frontier lab. In biology, two papers came out in Nature this month that showed models from one year ago are able to make novel scientific discoveries.¹⁴¹⁵¹⁶ We have expert-level AI that can do weeks of frontier research for $1 in 10 minutes.¹³⁶

We have all the pieces to launch the greatest scientific revolution around us. The world should be moving capital and talent to building the machine that makes scientific discoveries. The moat and durable advantage of the future will be scientific discoveries, not increasingly intelligent models. Consider now that the moat of frontier labs is not compute or models, but the internal knowledge for building models.

The group that does the experiments and accumulates knowledge will make the next big throughs. They will cure Alzheimer's Disease. The group that tests tons of hypotheses around superconductors will build the next superconductor. Many have had their perspective warped over the last year as intelligence has exploded. However, I think we're entering the era of scientific discovery and not the era of superintelligence.