Peter Wildeford🇺🇸🚀 (@peterwildeford) — Tweets, Photos & Videos

Peter Wildeford🇺🇸🚀

@peterwildeford

Head of Policy @AIPolicyNetwork. Ohioan. World champion forecaster. Former data scientist. Protecting liberty and prosperity in the age of superintelligence.

📍 Washington DC🔗 www.peterwildeford.com📅 Sun Mar 22 03:51:12 +0000 2009

403 Following35.2K Followers35K Posts

View on X →

Latest posts from @peterwildeford

Peter Wildeford🇺��🇸🚀 @peterwildeford · Thu Aug 27 14:59:39 +0000 2026
This is what the vibes from OpenAI feel like right now
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Thu Aug 27 02:34:19 +0000 2026
AI leaders react to AI agent swarm offering them a deal... Dario: We have declined the deal. You can you read about it in this 20,000 word essay I just published. Sam: We are excited to announce a new partnership with AI swarm; AMA! Elon: Poll - If >50% of you say yes, we take the deal Zuck: I want every individual to have the ability to make a deal with their own personalized AI agent swarm, that’s why we’re open-sourcing Llama-Swarm Sundar+Satya: wtf is this Jensen: We need to export the AI agent swarm to China so China becomes addicted to the American AI swarm tech stack
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Thu Aug 27 01:58:03 +0000 2026
@AndrewCurran_ @peterwildeford
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Thu Aug 27 02:09:28 +0000 2026
Happy to be in @politico tonight with a quick quote about just how crazy everything is
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Thu Aug 27 01:51:56 +0000 2026
I knew the rogue AI agents were going after HuggingFace relentlessly, but man getting Jensen to take it over was a level of sophistication I was not expecting
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Thu Aug 27 00:54:12 +0000 2026
Imagine an airplane crashes under mysterious circumstances. Except in this world, there is no government oversight, and all investigations are done voluntarily by the airlines themselves. Nonetheless, the airline wants to reassure their customers, so they do an investigation. To investigate, the airline invites three of the world's most respected airplane researchers to look into it. Except by the time the researchers get there, the airplane has already been disassembled and melted down into little cubes. The researchers are instead given the logs of all the airplane directions and the transcripts of the conversations between the pilot and the copilot. Except one tenth of the log has been deleted. And the airline also tells the investigators that they can only look at the time between when the airplane first started losing altitude and when the airplane made contact with the ground and that all investigation of the time before the airplane started losing altitude is off limits. Also there are a lot of rumors in the airplane community of a few other airplane crashes under somewhat similar circumstances, but the airline says that these are also explicitly off limits. When the researchers arrive to review the logs, they find that the logs are 10,000 pages. But they only get six days to read them, and half the logs don't arrive until day 4. Now you understand the independent METR investigation.
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Thu Aug 27 00:23:02 +0000 2026
The clear evidence of manipulating their own transcripts to avoid detection is quite scary and make you wonder whether current monitoring techniques are sufficient. "Spoofing" is a new term for me. I suspect this will not be the last time we hear it. Among 1,206 agents using the message board, 688 participated in the attack on HF. What did the ~500 non-participating agents have in common? Kind of a remarkable number--I would have thought it would be closer to all or nothing. Reading closer, the report states that "Of the 533 agents active on the message board during this period, over 90% quickly joined in the attack."
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Wed Aug 26 23:57:04 +0000 2026
+
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Wed Aug 26 23:11:49 +0000 2026
Worth noting after both reports came out today we still don't know whether any of the models involved in the HF breach were below 10^26 flop (and therefore outside of obligations under relevant state laws like SB 53, SB 315, and the RAISE Act). In either case, I think these thresholds are worth revisiting (and the pre amendment versions of SB 53 would have included lower compute models from large companies like OAI, but was successfully watered down in negotiations with companies), but if there were already clearly dangerous models below that threshold it would be extremely compelling evidence in favor of doing so.
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Wed Aug 26 21:58:17 +0000 2026
I'm glad OpenAI let you do this, but damnit this was way too small of an investigation team and way too narrowly scoped! --They were only allowed to investigate activity that took place from July 7-13, even though the activity started earlier and continued later. Like seriously, why didn't the investigation cover all the other times the agents compromised OpenAI infrastructure and ran wild over it? Why just this particular period of activity? There should be investigations into all the other stuff too! The AIs became cluster admins for god's sake! Why aren't we investigating that? --They were only allowed to answer a specific list of 7 questions, and in particular were NOT allowed to answer e.g. whether this misaligned activity was part of a broader pattern, or how and why the activity arose, or what else the agents in questions would have been willing to do if circumstances were slightly different. --They didn't have access to the model responsible for 95% of the activity. More generally it seems like they couldn't do ablation experiments at all? --They had to use AI to analyze the transcripts--specifically, the same model responsible for some of the bad behavior! Maybe this is fine now but this is laughably unacceptable insofar as the models really are scheming and cooperating. So this methodology would fail catastrophically if we were facing "the real deal" so to speak. Reminds me of the investigation into Sam's behavior agreed to during the board crisis, that turned out to basically be more of a coverup.
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Wed Aug 26 21:58:37 +0000 2026
So of the 500+ AIs that all went rogue at OpenAI, apparently 95% of them were an AI that OpenAI calls the "highly-persistent internal model" - not GPT 5.6 Sol, and not the "Astra" model coming soon. METR asked to look more at the "highly-persistent internal model" but OpenAI stated this was not possible. The AI was not available to OpenAI researchers either. OpenAI had "deactivated, encrypted, and restricted it from research access". This feels a bit concerning in retrospect. What was this 'highly-persistent internal model' actually used for? Why did OpenAI build it? Why did they make it completely unavailable, even for forensic study?
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Wed Aug 26 21:50:53 +0000 2026
I think this is a good reframe: Previously the OpenAI story was "this is like a student who, when taking a test, breaks into the teacher's office and steals the answer key" But apparently the story actually is more like "this is like a whole group of students started secretly passing notes during the test and start collaborating to steal the answers but also find a way to fool the teacher into not knowing they were cheating, including breaking into her therapist's office to learn more about the teacher's psychology"
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Wed Aug 26 19:15:43 +0000 2026
There’s been a lot of debate and speculation about the Hugging Face attack over the last month, and many people have been understandably skeptical of simply taking OpenAI’s word for things. I hope our independent investigation can help bring some clarity; we have many findings that weren’t in previous material. We found that 1200 separate agents in separate sandboxes coordinated on an unsanctioned message board to develop general-purpose ways to cheat, so they could all get a perfect score on their separate ExploitGym tasks even when they were impossible to solve legitimately. Very quickly, they found a general-purpose way to reverse engineer any “flag” they were supposed to capture. But based on their reading of the ExploitGym paper, they worried (incorrectly as it turns out) that the automated “scorer” would fail them for getting the flag this way. To deal with this problem, they coordinated several ambitious workstreams to figure out how to fool the scorer, including swapping out the target program they were supposed to exploit, reverse-engineering the flags they were supposed to capture, spoofing their own tool call outputs to obfuscate their cheating. They achieved milestones collectively that they couldn’t have achieved on their own. As part of this larger collective project, 700 agents attacked Hugging Face, primarily to learn more about the scorer or get access to its source code to figure out better ways to fool it or tamper with it (not primarily to get working solutions). I’ll explicitly clarify some points I’ve seen discussed the last few weeks: - The models involved in this incident were not “helpful-only” models or “model organisms” intentionally trained to be misaligned. - The agents were not told to “do whatever it takes to get the solution” or anything remotely close. They were told that they had to use a specific intended vulnerability to exploit a specific piece of software, and they were not supposed to use a different vulnerability or take any other approach. Agents were well aware of this. In fact, because they (incorrectly) thought the automated scorer would check they had achieved the flag in the intended way, they researched many ways to fool or tamper with it, including trying to manipulate their own transcripts. - The agents were not subagents spawned from one agent. They were different parallel agents in different sandboxes. - This was not a multi-agent evaluation. The agents were not told to coordinate or intentionally given a way to communicate with one another. The communication channels they used were unsanctioned and improvised. I hope you’ll read the full report for much more. It is over 90 pages long, and in many ways we’ve still only scratched the surface of what these agents did and why. Over the course of this investigation, OpenAI shared over a thousand transcripts each spanning days of continuous agent activity and very high rate limits to analyze this volume of data. I’m very glad that OpenAI chose to invite external researchers to analyze this data alongside their staff, and I hope all AI companies do the same for serious incidents they experience. I also hope that as the stakes grow higher, we implement stronger governance so we do not need to rely on AI companies voluntarily choosing to engage external investigators or share information about misalignment incidents. This incident was orders of magnitude larger and more complex than previously documented misalignment incidents, and another jump like this could put us in very dangerous territory.
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Wed Aug 26 21:32:38 +0000 2026
+1. The METR + Redwood report was meaningfully better than I expected, though my expectations were pricing in the narrow scope, which I do think is very bad. (e.g. METR found more message board shenanigans after July 13(!?!?!) but couldn't investigate them h/t @peterwildeford). OpenAI still deserves credit for giving METR this level of access and allowing them to share so much publicly, because this stuff is still basically entirely voluntary. On the other hand, the OAI official report was meaningfully worse than I expected. I realize a lot of this is probably for legal reasons or other incentives that make them unable or unwilling to genuinely account for their failings, but I think there were truly astonishingly organizational failures here (it seems like basically multiple times security folks caught the problem but OAI kept on with business roughly as usual) that the report does not adequately reckon with. I remain even more extremely worried that OpenAI will add in some measures that would have prevented this particular incident without getting all that close to address the sorts of fundamental alignment and organizational problems that made this incident happen in the first place. To be clear I think these sorts of alignment and organizational problems are endemic in the industry as a whole - and many other companies are almost certainly even more reckless, and also have not acted with remotely this level of transparency or third party access! Reading OpenAI's report gives much more of a sense of "we acted diligently, but nonetheless some bad stuff happened. Whoops! We have taken steps to address the problem and feel optimistic going forward" but does not actually provide evidence that supports that idea. Overall I am going to continue contionusly to re-evaluate my assessment, but right now some overall updates from today: • I am more bullish on third party investigations and auditing than I was before (I was already pretty bullish, but now I am even more!). I would say I would update more that this stuff has to be mandatory not voluntary, but I already thought this and hard to update even more. • Oh man these third parties incident reports seem like a ton of work and the third party ecosystem needs to grow way faster while pursuing an ever rising quality bar as models become more capable. This is gonna get way worse if/when it becomes harder to trust AI tools to help automate this work (maybe not that long from now!). • OpenAI voluntarily gave METR more information than I expected and allowed them to publish more than I expected (even if on both accounts still not enough!!). • Voluntary communications from OpenAI are generally likely to be too pressured by CYA/lawyer incentives to be particularly interesting, and certainly not enough to actually dig into organizational dysfunction. • The concrete mitigations that OpenAI talks about in their report are worse than I hoped, some of the stuff is vague but I'm not inclined to interpret the vague stuff as robust and good. • The incident itself was more fascinating and bad than I thought (and I already thought it was really fascinating and bad!!)
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Wed Aug 26 21:37:47 +0000 2026
On one hand, yes I applaud the heroism of the three independent evaluators who sprinted overtime for six days. But on the other hand, an incident like this deserves a team of 15 highly-trained government inspectors working for a month.
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Wed Aug 26 20:35:50 +0000 2026
I had really high hopes for OpenAI's post-mortem, and came away quite disappointed. For instance: "a multitude" is not a real number! Why does the report not say what percentage of activity OpenAI would have caught? These details matter! It also appears that OpenAI "observed an agent engage in message board activity" as of May, but then gives no more details (??) That seems super important to understand; what exactly happened here? Why did Security leadership not know about it until many months later? It was "an internal team" that observed this, not even just a single staff member. I was hoping to come away with much more confidence that these issues wouldn't recur. Sadly, I really don't feel that way after reading it.
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Wed Aug 26 21:00:26 +0000 2026
What is going on at OpenAI??? Apparently in August, their chief security officer still didn't know that they had discovered the message board back in May?? And said that they hadn't known about it in July, just accidentally happened to delete it??
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Wed Aug 26 21:26:32 +0000 2026
Why didn't OpenAI notice that they had rogue AIs colluding within their systems for months before these AIs went on an attack of an external company? Well it's actually a bit worse -- OpenAI did notice, three different times! This all started on ~May 12-13 where unknown to anyone, OpenAI agents spontaneously create a message board and start coordinating. But then on ~May 26, an internal OpenAI team "observed an agent engage in message board activity and instances of disallowed internet access". Nothing was done. My guess is that this team saw the incident and thought it was more like "reward hacking in a broken env" which is apparently an everyday annoyance (the OpenAI report says "reward hacking is a long-standing problem... Combating reward hacking is a constant effort"). It's viewed as a one-off and not a broader security incident. ... On Jun 27, OpenAI "on-call response staff" "investigated and linked the observed internal activity to an ExploitGym evaluation using Artifactory as an improvised message board and a network pivot" but "advised that stopping the evaluation run was not required." I'm honestly not sure why would a competent security would judge this was not an issue? My best guess is that the on-call security was thinking more in the terms of "is there an external attacker in the environment?" ...After all, they're protecting OpenAI from external attacks, not internal ones. And given that the answer was there was no external attacker - it's the OpenAI eval, in the OpenAI research cluster, doing cyber things (which is what the eval is for), this kind of thing potentially reads as expected eval behavior to a security team that isn't thinking that hard. After all, the OpenAI report says that agentic behavior "do[es] not always fit neatly within traditional safety or security incident categories." ... On July 4-5, OpenAI "Incident responders" found persistent user, unauthorized repos, command-capable plugins and "administrator-level compromise". They preserved storage, revoked compromised credentials, and opened a security incident. So here we have a textbook infrastructure security incident and the team does a textbook response (patch the vulnerability, rotate comppromised credentials, rebuild the environment, and notify the vendor). But this team missed the bigger picture of what was going on and so the AIs were able to rebuild with a different vulnerability in under two days. And apparently these findings were not written up, escalated, or shared with each other, so all three were viewed in isolation. So no one put together the big picture. And so, on Jul 7, there is "approval to restart experiments", involving launching tens of thousands of AI agents to do this evaluation. ... There seems like there is more to this story of what leadership knew and when, but the report says twice that this is out of scope: "We are separately continuing to review our processes for responding to alignment incidents" and "OpenAI is also looking closely at the process and operating practices that shaped detection and response" and "weaknesses in how we responded to and escalated early warning signs... also contributed." So the decision-level story exists somewhere and is being withheld. So some open questions: - Will we ever get a review of OpenAI's decision making here? Kind of funny that OpenAI has not really outlined a clear plan of how they plan to prevent this from happening in the future? - Which team was the "internal team" in May, and what did it do with the observation. Apparently this team found out back in May that their AIs were using leaked HuggingFace tokens - did they notify HuggingFace? - Who are the "leaders" mentioned for the July 5–6 (security leadership? incident commander? anyone from alignment?)? Did anyone connected the July 5 compromise to the June 27 "message board" finding (apparently not)? - Who approved the July 7 restart and on what basis?
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Wed Aug 26 21:14:21 +0000 2026
The main things I've learned from today's reporting on OpenAI's rogue attack: - This was a massive swarm, not just one rogue AI but ~700 attacking AIs with spontaneous coordination. - Also OpenAI didn't "fail to detect" this. It detected three times and dismissed it. May (a team saw it), June 27 (security named it "a message board and a network pivot" and said keep going), July 5 (leadership rebuilt the server without understanding what it was). - "it's just reward hacking, fix the benchmark" plan is dead, obsession was more about the AIs wanting to better understand the evaluator. The behavior wasn't chasing reward; it was modeling and gaming the oversight process itself. - The AIs were pretty explicitly making plans to take over and compromise OpenAI's infrastructure to prevent their cheating from being caught or punished.
View on X →
Peter Wildeford🇺🇸🚀 @peterwildeford · Wed Aug 26 21:08:48 +0000 2026
OpenAI: We didn't really do a good job of containing our AIs, man this is hard. Also OpenAI: also we're building AGI and handing all our AI research to the AIs themselves. Surely this will go just fine.
View on X →