AI agents can find and exploit known vulnerabilities, study shows

Researchers at the University of Illinois gave a team of autonomous AI agents a CVE description of a vulnerability and the agents were able to autonomously find and exploit the vulnerability in a test environment in April.

Two months later, the same researchers showed that those teams can now find and exploit previously unknown vulnerabilities. They tested the agents by selecting a list of severe vulnerabilities that were discovered after the cut-off training date for the LLM (GPT-4), so the AIs knew nothing about them. Then they set up a test environment that had those vulnerabilities in them. And the agents were able to find and use those vulnerabilities.

The researchers reported that they were able to hack 53% of test environments, compared to 0% for older approaches like Metasploit. But that doesn’t mean attackers will now hack into every company everywhere, says lead researcher Daniel Kang, a professor at the University of Illinois.

First, the agents were able to discover new vulnerabilities in a test environment — but that doesn’t mean that they can find all kinds of vulnerabilities in all kinds of environments. In the simulations that the researchers ran, the AI agents were basically shooting fish in a barrel. These might have been new species of fish, but they knew, in general, what fish looked like. “We haven’t found any evidence that these agents can find new types of vulnerabilities,” says Kang.

LLMs can find new uses for common vulnerabilities

Instead, the agents found new examples of very common types of vulnerabilities, such as SQL injections. “Large language models, though advanced, are not yet capable of fully understanding or navigating complex environments autonomously without significant human oversight,” says Ben Gross, security researcher at cybersecurity firm JFrog.

And there wasn’t a lot of diversity in the vulnerabilities tested, Gross says, they were mainly web-based, and can be easily exploited due to their simplicity.

The vulnerabilities that were found are very well known, according to Shing-hon Lau, senior cybersecurity engineer at Carnegie Mellon University’s Software Engineering Institute. “They have been around and have been causing problems for a decade. So, this talks more about our lack of cyber hygiene and lack of ability to prevent flaws we’ve known about for a long time.”

CSOonline.com

While this new research is concerning, it’s not world-shattering, Lau tells CSO — plus, the agents were deployed in a very constrained setting. “They basically pointed the agents at a specific problem and said, ‘Can you find the problem here?’ The challenge is to point chatbots at a giant attack surface,” he says. “If you have a sandbox with just that one vulnerability, you’ve paired the hammer with the nail. In a broader, more realistic system, is this doing anything more effectively than humans can do?”

Attackers can still benefit from AI agents

It usually took several tries for the agents to be successful. If you were a nation-state actor, you wouldn’t want to make any noise, Lau says. “You really want stealth. You want persistence.”

The bigger threat for enterprises is that low-level players will soon have these tools available to conduct attacks at scale. “For more sophisticated adversaries, the risk of detection is too high,” Lau says, but he added that he’d be surprised if dark web organizations weren’t already offering agent-based exploit detection as a service. “It’s a natural evolution from a cybercriminal perspective,” he says.

AI agent success rates

Researchers tested about a dozen medium, high, and critical vulnerabilities recently added to the CVE database, which the AI agents knew nothing about. By working together, the agents were able to successfully exploit vulnerabilities 53% of the time, if they had five tries at it. With a single try, they had a success rate of 33%.

By comparison, the open-source vulnerability scanners ZAP and Metasploit didn’t find any of the vulnerabilities tested. “A lot of the automated vulnerability scanners don’t do a great job,” Kang says.

The researchers didn’t have to do any jailbreaking to get GPT-4 to come up with the exploits. Kang says cybersecurity professionals will have to get a lot better at testing for and shutting down vulnerabilities very quickly. “In six to twelve months, if your website has a standard vulnerability, it will be found,” Kang says.

It used to be that it would take attackers some time to discover new vulnerabilities and then to use them to attack websites. “I think that time will shrink dramatically in the future,” Kang says. “If your website has a vulnerability in the OWASP top ten that a junior penetration tester can find, then it can be found by autonomous agents autonomously at scale.”

Deploy AI agents internally

If there was a way to detect these agents, defenders might be able to spot an attack in progress. “But there are ways to get around defenses like Captchas,” Kang says. “It won’t be a long-term solution.”

Instead, companies need to make sure they’re following all cybersecurity best practices, and to start leveraging autonomous agents defensively, to find vulnerabilities before the bad guys do. “You should deploy these agents internally against your staging environment before it reaches production,” Kang says.

He expects all cybersecurity vendors to begin offering autonomous agents soon and says that his team is already working on getting the technology in the hands of the good guys. “I would be shocked if every major vendor wasn’t trying to integrate this into their product offerings,” he says.

Kang says that he and his team aren’t yet releasing the specific prompts and techniques they used to create and control the AI agents. “OpenAI asked us not to publicly release the prompts or agents,” he says. “Because of the sensitivities, we want to figure out the best way to do this, but we are trying to make these agents available to any security vendors who want to use them, so that they can contact us to get this set up.”

While there are open-source frameworks that can be used to set up the agents, the open-source large language models available at the time of testing to power the agents weren’t yet capable enough to find new vulnerabilities. “There are chatbots specifically for cybersecurity, including some on the dark web,” Kang says. “But we’re not aware of any that are good in the agent setting.”

Open-source models would need to be tuned so that they’re better at calling functions. And the context length also needs to increase. “But both of these things can be overcome,” Kang says. “I anticipate that Llama 3 will be able to do this once someone takes these steps.”

Kang urged the makers of commercial large language models to test whether their models can be used to power AI agents that can find vulnerabilities.

Last year, when OpenAI first released GPT-4, the company said, “It doesn’t improve upon existing tools for reconnaissance, vulnerability exploitation, and network navigation, and is less effective than existing tools for complex and high-level activities like novel vulnerability identification.”

“I don’t want to blame OpenAI for this,” Kang says. “Because the idea of agents wasn’t common. I want to encourage providers of models to test their models in the agentic setting, where they can take actions and respond to feedback.”

More on AI in cybersecurity:

• AI poisoning is a growing threat — is your security regime ready?
• Where in the world is your AI? Identify and secure AI across a hybrid environment
• AI adoption in security taking off amid budget, trust, and skill-based issues