LLMs fueling a “genAI criminal revolution” according to Netcraft report

Like seemingly everyone else, threat actors are increasingly adopting generative artificial intelligence (genAI) as a business tool. Recent findings by security researchers at Netcraft have revealed what it called “a mass universal scaling up of genAI being used as a content creation tool for fraudulent websites.”

In a blog post published Thursday, Netcraft noted that it has been identifying thousands of websites per week that use AI-generated content, with steady growth in the technology’s use. In late July, however, there was a spike in the number of sites that continued into the first week of August before subsiding.

Netcraft attributed this to a single threat actor who was setting up fake shopping sites and using genAI to write product descriptions.

“This and the broader growth in activity between March and August appears to indicate a mass universal scaling up of genAI being used as a content creation tool for fraudulent websites, with a notable spike showing in the realm of online stores,” Netcraft said in its post.

Malicious content is becoming more convincing

“This has led to an abundance of malicious websites, attracting victims not only because of the sheer volume of content but also because of how convincing that content has become.“

It is no longer possible, the report said, to decide that a website or email is legitimate simply because it’s written in professional English.

However, there can be clues in the email or on the site. Netcraft said that sometimes threat actors accidentally include large language model (LLM) outputs in the fraudulent emails. For example, a phishing email it encountered, claiming to contain a link to a file transfer of family photos, also included the phrase, “Certainly! Here are 50 more phrases for a family photo.”

“We might theorize that threat actors, using ChatGPT to generate the email body text, mistakenly included the introduction line in their randomizer,” Netcraft said. “This case suggests a combination of both genAI and traditional techniques.”

Telltale evidence still shows which phishing emails are fake

Another phishing email it viewed would have been credible — had it not been for the sentence at the beginning, which included the LLM introduction line, “Certainly, here’s your message translated into professional English.” And a fake investment website touting the phoney company’s advantages looked good, except for the headline saying, “Certainly! Here are six key strengths of Cleveland Invest Company.”

“There’s no honor among thieves, of course,” Netcraft observed. “Just as criminals are happy to siphon credentials from other phishing sites, we’ve observed that when they see a convincing LLM-generated template, they may replicate the content almost verbatim.”

For example, the “Cleveland Invest” website text was replicated (complete with LLM response) from another fake text created for “Britannic Finance”. In this case, the threat actor appeared to then use an LLM to adjust the text, using synonyms for some terms.

Netcraft has also seen LLM-generated sites for fake shops and fake pharmacies designed for search engine optimization (SEO), to pull in more victims. Again, it cited a site on which the LLM’s response to the request was leaked on the site, with discussion points followed by “this outline should give you a good start …”, and a reminder to include SEO keywords in the title, headings, and body of the text.

And all this is just the tip of an ever-growing iceberg. “The report speaks to only one area of cyber threat that’s being augmented by generative AI capabilities: gaining initial access to a victim, namely through phishing,” said Brian Jackson, principal research director at Info-Tech Research Group.

“Unfortunately, that’s only one small part of the full scope of augmented threats we’re seeing, thanks to LLMs,” Jackson says. “Whole new taxonomies of cyber threat techniques are being added to threat frameworks thanks to LLMs.”

LLMs are being used to conduct reconnaissance

Examples include using LLMs to conduct reconnaissance, such as searching and summarizing a potential victim’s publicly available materials and potential vulnerabilities. “OpenAI has banned state-sponsored accounts for doing exactly this,” Jackson says. “Then, there is the attempt to exploit LLMs themselves through prompt injection and jailbreak, etc.” He pointed to an exhaustive list of techniques via MITRE ATLAS.

This coincides with Netcraft’s findings. “There are many more [examples], with conclusive evidence pointing to the large-scale use of LLMs in more subtle attacks,” the post said. “The security implication of these findings is that organizations must stay vigilant; website text written in professional English is no longer a strong indicator of its legitimacy. With genAI making it easier to trick humans, technical measures like blocking and taking down content are becoming increasingly critical for defending individuals and brands.” 

And, said Jackson, “from my perspective, it’s not the same old threats being augmented with AI that are most alarming. We already have defined techniques to help mitigate those. Rather, it’s the net new cyber threats from generative AI that could really catch organizations off guard.”

“As we’ve already seen, most of us expect that when an executive video calls us, we can trust that it’s really them giving us instructions, Jackson says. “That’s just no longer the case, as generative AI can effectively make deepfakes with limited available training data.”