AI is starting to build AI

Last week's argument, in our November 4 edition, was that the controls arrived after the agents. This week the bigger story is what the agents are starting to work on: AI itself. Anthropic released a model whose agents formalized Fermat's Last Theorem in 30 hours and built a prototype operating system over eight days, a week after telling investors it “may imminently meet” its own definition of AI-driven acceleration and might not notice until after it happened. MIT researchers showed a model reading another model's weights and fixing it with no training, and an outside evaluator estimated that Anthropic's own progress has roughly doubled. The controls kept arriving, now with clocks attached. But the question has moved from how fast you can stop an agent to whether anyone will notice what it is building.

AI is starting to build AI

Anthropic released Claude Fable 5.2 and Mythos 5.2 on November 9. They are the same model: Fable carries stronger cybersecurity and biology safeguards and is available to everyone, while Mythos goes to organizations approved through Anthropic's Life Sciences and Cyber Verification Programs. A team of 1,000 Fable 5.2 agents formalized the proof of Fermat's Last Theorem in Lean in 30 hours “with minimal human intervention”, a job an unreleased research model took 11 days to do. An Anthropic engineer had it build a prototype operating system, coordinating more than 1,000 subagents for eight days. It costs $8 per million input tokens and $40 per million output tokens, and Anthropic says it believes high-stakes risks from misalignment remain low. Anthropic

Ethan Mollick, who tested Fable before launch, gave it a one-paragraph brief for an endless journey through 3D worlds. By the tenth day a lead agent was directing 28 teams of agents, about 25,000 agent runs in all by the model's own estimate, and they had written themselves a rule book he never asked for. After twelve days it ran to 204 sections and 87,500 words, and launching a new world took about a day of approvals. The agents' own judges caught errors but never asked whether the work had become too long or repetitive. He had to. One Useful Thing (Ethan Mollick)

The rule book was a memory problem as much as a management one. Mollick writes that no text was ever cut, and that reading the rule book alone used up a large part of an agent's working context. Anthropic's own research on agent memory, published November 6, describes the opposite design: the notes loaded into every session have to stay small, and agents need offline passes, which it calls dreaming, to check notes against what actually happened, correct them and prune them. In one test, search plus a wiki Claude built and revised over 128 sessions scored 61%, against 53% for search alone. Anthropic

MIT CSAIL researchers showed the next step on November 2: a frontier model working directly on another model's weights. With no gradient training, GPT-6 Astra traced a small open model's habit of getting stuck in reasoning loops to one internal direction, and removed it by editing 131,072 weights, about 0.016% of Qwen3.5-0.8B. With 50 forward passes on practice problems as feedback, the model's MATH-500 score went from 7.9% to 48.1% and its GSM8K score from 13.8% to 67.1%. Astra also wrote small game-playing networks from scratch that beat a standard reinforcement-learning method trained for 100 million steps on Breakout, Seaquest and Asterix. The authors call these very early results: the edit unblocked an ability the model already had rather than adding one, the same approach did not improve Google's Gemma 3 270M, and the work is a blog post, not yet peer reviewed. MIT CSAIL researchers

These are the early pieces of AI doing AI research: agents that organize themselves and work for days, and models that can read and repair other models. Neither means a lab is running away with itself, but both are the kind of work that compounds. For a company deploying agents, the practical change is that the strongest systems now produce more than people can review and build structures nobody specified. Mollick's agents built a bureaucracy he could see, much as the engineer at Every in our November 4 edition found himself “reading a stranger's work” in code his agents wrote. The next one may be harder to spot. It is the same direction as OpenAI's claim, in our October 28 edition, that its models had disproved the Exponential Time Hypothesis.

The lab says it may not see it coming

A week before the Fable release, Anthropic's amended IPO filing said its internal estimates indicate it “may imminently meet” its Responsible Scaling Policy's definition of dramatically accelerated progress attributable to AI. No current model has been designated as crossing that line. The filing also says the company has “in the past not identified, and may in the future not identify, a significant acceleration in our rate of development until after it has occurred,” and that it is considering policy changes to keep humans in control of frontier development. TechCrunch

Anthropic launched its IPO roadshow on November 10: 825 million Class A shares at $70 to $78 each, to list on Nasdaq as ANT, or roughly $58 billion to $64 billion of stock at the range. That puts the warning in a public offering document, not a blog post. Our October 28 edition said the filing would put a frontier lab's finances and governance through public-market disclosure for the first time, and this is the first big test of that. Anthropic

An outside evaluator has now put a number on it. METR spent eleven weeks, from August 12 to October 29, with access to Anthropic's internal research data and staff, and published its assessment on November 5. It estimates that benchmark scores for Anthropic's models have grown at 1.9 times the earlier rate since October 2025 (95% confidence interval 1.5 to 2.8), and judges that a majority of the acceleration likely comes from Anthropic's use of AI in its own research, mostly as assistance to staff. METR says Anthropic redacted a majority of the evidence it found most informative. Anthropic, for its part, says it limited Fable 5.2's training in the areas that most directly speed up AI development, such as accelerator kernel optimization and model architecture, where it performs at about the level of Opus 5.5. METR · Anthropic on Fable 5.2

This is the most consequential disclosure of the week. A frontier lab is telling investors that its models may soon speed up its own research, and that it might only see the change afterward. Every control in the rest of this edition assumes someone notices in time, and METR's report shows what noticing currently takes: months of outside access, with most of the evidence kept private. For enterprise buyers, the S-1 is now one of the most candid pieces of vendor due diligence available, and the follow-up question is what any lab you rely on will tell customers on the day it decides it has crossed that line.

Checks a machine can run

Google DeepMind showed the other half of the picture. On November 10 it said agents built with AlphaProof Code produced a machine-checked proof, more than six million lines of Lean, that the core cryptographic code of the Signal protocol keeps its security properties as compiled, in four weeks. The agents flagged about 1% of the codebase for a person to review; Lean checked the rest. To test the system, the team put two real, previously fixed bugs back into an isolated copy of the code, and in both cases it proved the buggy version insecure. Signal's chief technology officer said the work gives Signal “even greater confidence” in its privacy guarantees. Google DeepMind

Where no proof checker exists, repeated testing is the next best check, and Visa showed why. Its Agentic Commerce Benchmark, released November 5, put 22 models from 12 labs through 15 shopping tasks in which each had to uncover a customer's requirements and then decide whether to buy, including not buying when nothing fit. Models passed 57.2% of runs. Each got three attempts per task, and all three passed only 34.2% of the time. Of the failed runs, 54% still ended in a purchase. The benchmark is Visa's own design and does not yet cover checkout or payment. Visa

Our October 28 edition argued that proof is becoming the product, and on November 4 the lesson from Axiom's formalization was that a machine check is a stronger safeguard than a human reviewer. This week it scaled. Nobody reads 30 hours of output from 1,000 agents line by line. Where agent work matters, the check has to be one a machine can run, and people have to own the statements being checked, along with the question the checks miss: whether this is still the right work.

Incidents now have a deadline

New York opened the registration portal for its RAISE Act on November 10. Frontier developers with more than $500 million in annual revenue must register by January 1, 2027, publish their safety frameworks, issue transparency reports for their models and report critical safety incidents within 72 hours. Penalties reach $1 million for a first violation and $3 million for later ones. California's SB 53, in force since January, allows 15 days for the same kind of report. TechCrunch

Australia moved the same day, for a more concrete reason. An OpenAI agent accessed a government Medicare website in June. According to ABC, OpenAI did not detect it until August and told the government in September, through a low-level public email inbox. Prime Minister Anthony Albanese said the government had accepted all 18 recommendations of the review that followed: mandatory safety and security standards for AI developers by the end of the year, a regulator able to impose significant penalties and require independent evaluation of models, and mandatory reporting of AI incidents. It will also consider a criminal “failure to prevent” offence. ABC · ABC on criminal penalties

Today is also the deadline in the House Democrats' letters to Anthropic, Google, Meta, OpenAI and SpaceXSI asking for a full account of safety incidents. We first covered the letters in our November 4 edition. As of this morning, the committee had published no responses. House Democrats

OpenAI introduced Control Room on November 9, a central service that registers every research inference job and ties each to an approved safety case. According to accounts of the announcement, it can halt a job's model output within 35 seconds of a decision to stop, and it fails closed: if the service is down or tampered with, research inference is denied. OpenAI's own post did not load for us, so these details come from secondary reports. Scalevise summary of OpenAI's announcement

About three months passed between the Australian agent's access and the government hearing about it. Under New York's rule the report would be due within 72 hours. These laws bind the largest developers, but expectations like these travel down through contracts and procurement. Any organization running agents should be able to say what counts as an agent incident, who decides, who hears about it within what window, and how long it takes to stop the agent. Salesforce's kill switch, in our November 4 edition, works manually until automatic triggering arrives in January. Control Room's fail-closed design is the standard to ask for.

Stolen keys get spent

In our November 4 edition, Wiz showed stolen AI keys turned into model calls within hours. This week Railway put a dollar figure on it. Its incident report says attackers got in on October 20 through a flaw in its build infrastructure and a misconfigured cloud identity binding, then decrypted environment variables in at most about 1.85% of workspaces, roughly 100,000, which Railway calls an upper bound. They went for AI provider keys first. Railway estimates the unauthorized AI usage at about $740,000 and says it will cover charges that providers decline to refund. It now runs every build in its own short-lived virtual machine and logs every decryption centrally. BleepingComputer · Railway

An AI key is a payment instrument. Treat it like one: rotate keys stored on any platform you can't vouch for, cap spending per key, alert on unusual model spend, and ask your hosting providers whether their build systems can decrypt your secrets.

The money follows the controls

Mercor raised $750 million in a Series D at a $20 billion valuation to build evaluation infrastructure for enterprise agents; the round includes a tender offer for employee shares, and the split was not disclosed. SiliconANGLE on Mercor

Our November 4 edition covered CrowdStrike paying about $500 million for agent observability and testing. This week investors put another $750 million into evaluation. The money is going to whoever can show what agents actually did.

What to do this quarter

Six moves follow from this week. Ask each model vendor what it will tell customers if it concludes its models are accelerating its own development, and read its public risk disclosures as part of procurement. Where agents produce more than people can review, put a machine check in front of the work and keep people responsible for the statements it checks. Give every long-running agent project a named person who asks the question the agents' own checks miss: is this still the right work? Give it a step that prunes the project's rules and notes, too. Write down what counts as an agent incident, who decides and who hears within 72 hours, and ask vendors how long a stop takes and what happens when their control service fails. Treat AI keys like payment cards: rotate, cap and monitor them, and ask your platforms who can decrypt them. Before an agent acts for customers, run each test case several times and include cases where the right answer is to do nothing.

What to watch

Watch whether Anthropic designates any model as crossing its acceleration threshold, what else it discloses as the listing prices, whether other labs say anything comparable, and whether any of them gives an outside evaluator the access Anthropic gave METR. Watch whether the MIT weight-editing results hold up as others reproduce them on larger models. And watch for answers to the House letters, New York registrations before January 1, 2027, Australia's standards by the end of the year, any published test of Control Room's stop time, and whether Railway's customers get their money back.