
Deep research and AlphaEvolve are recent developments in AI that point to the promise of automating scientific research. AI Pioneer Yoshua Bengio suggests that constructing “Scientist AI” is the most promising solution for AI Safety. In this post I will explain why an AI scientist misses the point of science, is not solution for AI safety, and may even introduce more AI risk rather than less.
An AI scientist is a misnomer. While I am not a scholar of “science” itself, as a practicing researcher there are clear boundaries of what science is and is not. Science is a process by which we come to understand the world. Science isn’t simply the production of true facts about the world written down on paper, for there to be knowledge there must be a knower. It isn’t clear at all that current AI systems understand or know anything as a “subject” which disqualifies these systems as a scientist.
The natural counter argument is to say well what we mean by an “AI scientist” is a system that automates the production of knowledge which human scientists can use to understand. This is another type of fundamental misunderstanding of how science is done and what it’s for. David Deutsch explains in his book, The Beginning of Infinity, that science is fundamentally a creative act of producing explanations that survive the scrutiny of criticism, an epistemology he borrows from Karl Popper. Another tradition of epistemology due to Thomas Kuhn points out that science is fundamentally a social institution where humans further understand together as subjects. While AI tools can certainly accelerate answering specific questions formulated by human scientists (AlphaFold, for example empowers human scientists to ask and answer questions about protein structure), these tools will not ask creative or socially meaningful research questions.
At best AI-accelerated science won’t be more (or less) beneficial than any previous technology-empowered science developments such as through the introduction of the microscope. In fact, it’s best not to think of these developments as “AI” but think about each development as a specific technology development, for example AlphaFold and AlphaEvolve as better search algorithms for protein structure or code optimization. At worst, if the public buys into “AI-accelerated science” tools that are generic ChatGPT like text generators then we will see the emergence of never-before-seen vast amounts of “science spam,” which is neither meaningful nor useful to humans in anyway. Suppose there were some gems in this generated science spam, which isn’t likely due to the fundamental lack of creativity of these systems, then who are these gems for? who consumes them? who sorts through them? These are now familiar questions being asked by writers when questioning what the purpose of generated text serves with a consensus building that generative writing is mostly only useful because of the time pressure that our society puts us under and doesn’t serve humanity. However, science spam likely will waste time instead of saving it
Let’s return to why AI scientists are specifically bad for AI safety. As we saw above, attempting to automate AI safety research is fundamentally doomed. Despite this, several organizations dedicated to building AI scientists like Bengio’s LawZero are attracting millions of dollars of funding and press attention that misleads the public into believing that this is a worthwhile endeavour. This form of “safety washing” is unfortunately common now that AI has had a resurgence and results in a fundamental distraction and misallocation of research away from tractable research agendas. Finally, this research direction also implies a relinquishing of control (the AI now does the science) and assignment of trust to AI systems which is unwarranted and as I argued in previous posts, if we know one thing about AI safety is that giving AI systems decision making power results in fundamentally bad outcomes for humanity.
Image generated by Imagen-3 using the prompt: “SNES 16-bit style of a Robot with a lab coat that is typing spam on a computer screen”