A few weeks ago, I watched a small team of artificial intelligence agents spend roughly 10 minutes trying to hack into my brand new vibe-coded website.

The AI agents, developed by startup RunSybil, worked together to probe my poor site to identify weak spots. An orchestrator agent, called Sybil, oversees several more specialized agents all powered by a combination of custom language models and off-the-shelf APIs.Whereas conventional vulnerability scanners probe for specific known problems, Sybil is able to operate at a higher level, using artificial intuition to figure out weaknesses. It might, for example, work out that a guest user has privileged access—something a regular scanner might miss—and use this to build an attack.

The website targeted by Sybil was one I created recently using Claude Code to help me sort through new AI research papers. The site, which I call Arxiv Slurper consists of a backend server that accesses the Arxiv—where most AI research is posted—along with a few other resources, combing through paper abstracts for words like “novel”, “first”, “surprising” as well as some technical terms I’m interested in. It’s a work in progress, but I was impressed with how easy it was to cobble together something potentially useful, even if I had to fix a few bugs and configuration issues by hand.A key problem with this kind of vibe-coded site, however, is that it’s hard to know what kinds of security vulnerabilities you may have introduced. So when I spoke to Herbert-Voss about Sybil, I decided to ask if it could test my new site for weaknesses. Thankfully, and only because my site is so incredibly basic, Sybil did not find any vulnerabilities. Herbert-Voss says most vulnerabilities tend to be the result of more complex functionality like forms, plug-ins, and cryptographic features. We watched as the same agents tried probing a dummy ecommerce website with known vulnerabilities owned by Herbert-Voss. Sybil built a map of the application and how it is accessed, probed for weak spots by manipulating parameters and testing edge cases, and then chained together findings, testing hypotheses, and escalating until it breaks something meaningful. In this case, it did identify ways to hack the site. Unlike a human, Herbert-Voss says Sybil runs thousands of these processes in parallel, doesn’t miss details, and doesn’t stop. “The result is something that behaves like a seasoned attacker but operates with machine precision and scale,” he says.“AI-powered pen testing is a promising direction that can have significant benefits for defending systems,” says Lujo Bauer, a computer scientist at Carnegie Mellon University (CMU) who specializes in AI and computer security. Bauer recently coauthored a study with others from CMU and a researcher from AI company Anthropic that explores the promise of AI penetration testing. The researchers found that the most advanced commercial models could not perform network attacks, but they developed a system that set high-level objectives like scanning a network or infecting a host, which enabled them to perform penetration tests.

Sarah Guo, an investor and founder at investment firm Conviction, which is backing RunSybil, says it is rare to find people who understand both AI and cybersecurity. Guo adds that RunSybil promises to make the kind of security assessment that large companies perform periodically more widely available, and on a continuous basis. “They can do baseline penetration testing with models and tool use continuously,” she says. “So you'll always have a view of what it really looks like to be under attack.”The techniques being developed by RunSybil may become doubly necessary as attackers develop their own AI strategies. “We have to assume that attackers are already using AI to their benefit,” says Bauer of CMU. “So developing pen-testing tools that use it is both responsible and likely necessary to balance the increasing risk of attack.Herbert-Voss seems like a good person to help here, since he was the first security researcher at OpenAI. “I built all sorts of crazy things like new prototypes of polymorphic malware, spearphishing infrastructure, reverse engineering tools,” Herbert-Voss says. “I was concerned that we didn’t have a solution for when everybody gets access to language models—including the bad guys.”