As AI systems become more capable and more autonomous, they could move faster than our institutions, concentrate power in too few hands, or make decisions that people no longer understand or control.

That concern becomes especially important as we approach systems that can improve themselves and future versions of themselves, often called recursive self-improvement. As the process of building AI becomes more automated, the pace of AI progress could accelerate rapidly. This moment calls for extreme care.

I believe there are two ways that AI progress could go very badly that we have to avoid. First, we could lose control of the future to AI. The risk is that it moves so fast that people can no longer follow what’s happening or intervene when needed. This would obviously be terrible.

We do not want to build systems—we think it’s important that everyone avoids building systems—where we cannot make the alignment, monitorability, and safety guarantees that we must. We need to understand what these systems are doing and have strong evidence that they will do what people intend, even as they get very, very smart. Actually especially as they get very very smart. It doesn’t matter whether people put the risk of catastrophe at 10%, or 1%, or 12%, or .1%. None of these levels are remotely acceptable. And we should not train models that we cannot make an extremely strong case that we will be able to keep under human control.

So that’s one way things can go wrong. In the other direction, these systems could concentrate too much power in too few hands. No one person or company or country should be able to use the most powerful AI models to impose their worldview on everyone else. A company or country that believes only it can be trusted with this technology can use that belief to justify almost anything else. We have to reject that logic, even when it’s convenient for any one actor.

We now have models capable of discovering knowledge that we have never had access to before. Think about the progress we’ve seen in mathematics. Three summers ago, our models were doing reasonably well at elementary-school math. Two summers ago, they were doing quite well in high-school mathematics competitions. Last summer, we reached the level of a gold medal at the most prestigious international mathematics competition. And just a few weeks ago, this summer, one of our models solved one of the Millennium Prize Problems: the Navier–Stokes equations.

These equations are used in aircraft design, weather forecasting, and the study of blood flow. We are not solving math problems simply for the sake of mathematics, but to enable people to discover new knowledge and use it to improve healthcare, raise living standards, and expand human potential everywhere in the world. There is still so much left to discover.

But it is very important that companies do not substitute themselves for the democratic process. If AI is to be democratic, the most important decisions cannot be made solely by laboratories in San Francisco. They need to be shaped through democratic processes and by governments that are accountable to the people they serve. At the international level, this will require cooperation.

What do we need

. a mechanism for complementary national and international standards for frontier AI: standards for measuring capabilities, assessing risks, determining whether safeguards are sufficient, and maintaining meaningful human oversight as systems become more autonomous.

. common standards so that countries can compare evidence, verify compliance, and develop a shared language and understanding of what is happening.

. rapid and accurate incident reporting, along with classification and reporting protocols, so that the world can learn from failures before they become catastrophes.

. secure channels through which governments, critical infrastructure operators, and technical experts can communicate emerging vulnerabilities and new threats.

More details here: