Scam Sites at Scale: LLMs Fueling a GenAI Criminal Revolution

Cybercrime groups, like other businesses, can create more content in less time using GenAI tools. Over the last 6 months, Netcraft identified threat actors using these technologies across a range of attacks, from innovating advance fee-fraud to spamming out the crypto space. In total, our observations show LLM-generated text being used across a variety of the 100+ attack types we cover, with tens of thousands of sites showing these indicators.

Netcraft’s first party research into the use of generative artificial intelligence (GenAI) to create text for fraudulent websites in 2024 include:

  • A 3.95x increase in websites with AI-generated text observed between March and August 2024, with a 5.2x increase1 over a 30-day period starting July 6, and a 2.75x increase in July alone—a trend which we expect to continue over the coming months
  • A correlation between the July spike in activity and one specific threat actor
  • Thousands of malicious websites across the 100+ attack types we support
  • AI text is being used to generate text in phishing emails as well as copy on fake online shopping websites, unlicensed pharmacies, and investment platforms
  • How AI is improving search engine optimization (SEO) rankings for malicious content
  • July 2024 saw a surge in large language models (LLMs) being used to generate content for phishing websites and fake shops. Netcraft was routinely identifying thousands of websites each week using AI-generated content. However, in that month alone we saw a 2.75x increase (165 per day on the week centered January 1 vs 450 domains per day on the week centered July 31) with no influencing changes to detection. This spike can be attributed to one specific threat actor setting up fake shops, whose extensive use of LLMs to rewrite product descriptions contributed to a 30% uplift in the month’s activity.

    These numbers offer insight into the exponential volume and speed with which fraudulent online content could grow in the coming year; if more threat actors adopt the same GenAI-driven tactics, we can expect to see more of these spikes in activity and a greater upward trend overall.

    “As an AI language model, I can make scam emails more believable”

    Threat actors in the most traditional forms of cybercrime—like phishing and advance fee fraud emails—are enhancing their craft with GenAI.

    Netcraft observed signs of threat actors’ prompts being leaked in responses, providing insight into how they are now employing LLMs. In our Conversational Scam Intelligence service—which uses proprietary AI personas to interact with criminals in real-time—Netcraft has observed scammers using LLMs to rewrite emails in professional English to make them more convincing.

    Fake investment platforms are particularly well positioned for LLM enhancement, because the templates we’ve typically seen for these scams are often generic and poorly written, lacking credibility. With the help of GenAI, threat actors can now tailor their text more closely to the brand they are imitating and invent compelling claims at scale. By using an LLM to generate text that has a professional tone, cadence, and grammar, the website instantly becomes more professional, mimicking legitimate marketing content. That is, if they remember to remove any artifacts the LLM leaves behind.

    There’s no honor among thieves of course. Just as criminals are happy to siphon credentials from other phishing sites, Netcraft observed that when they see a convincing LLM-generated template, they may replicate the content almost verbatim. To evade detection and correct errors in the original template, some threat actors appear to be using LLMs to rewrite existing LLM-drafted text.

    Threat actors are becoming more effective at using GenAI tools in a highly automated fashion. This enables them to deploy attacks at scale in domains where they don’t speak the target language and thus overlook LLM-produced errors in the content. By example, Netcraft came across numerous websites where page content itself warns against the very fraud it’s enabling.

    It’s no surprise that threat actors are beginning to utilize GenAI to both create efficiencies and improve the effectiveness of their malicious activities. Netcraft has been observing this trend for some time and developing suitable countermeasures in response. Netcraft’s platform flags attacks with indicators of LLM-generated content quickly and accurately, ensuring customers get visibility of the tactics being used against them.

    For the complete research report visit here.

    Featured

    • Live from GSX 2024: Post-Show Recap

      Another great edition of GSX is in the books! We’d like to thank our great partners for this years event, NAPCO, LVT, Eagle Eye Networks and Hirsch, for working with us and allowing us to highlight some of the great solutions the companies were showcasing during the crowded show. Read Now

      • Industry Events
      • GSX
    • Research: Cybersecurity Success Hinges on Full Organizational Support

      Cybersecurity is the top technology priority for the vast majority of organizations, but moving from aspiration to reality requires a top-to-bottom commitment that many companies have yet to make, according to new research released today by CompTIA, the nonprofit association for the technology industry and workforce. Read Now

    • Live from GSX 2024: Day 3 Recap

      And GSX 2024 in Orlando, is officially in the books! I’d like to extend a hearty congratulations and a sincere thank-you to our partners in this year’s Live From program—NAPCO, Eagle Eye Networks, Hirsch, and LVT. Even though the show’s over, keep an eye on our GSX 2024 Live landing page for continued news and developments related to this year’s vast array of exhibitors and products. And if you’d like to learn more about our Live From program, please drop us a line—we’d love to work with you in Las Vegas at ISC West 2025. Read Now

      • Industry Events
      • GSX
    • Bringing New Goods to Market

      The 2024 version of GSX brought with it a race to outrun incoming hurricane Helene. With it’s eye on Orlando, it seems to have shifted and those security professionals still in Orlando now have a fighting chance to get out town. Read Now

      • Industry Events
      • GSX

    Featured Cybersecurity

    Webinars

    New Products

    • Luma x20

      Luma x20

      Snap One has announced its popular Luma x20 family of surveillance products now offers even greater security and privacy for home and business owners across the globe by giving them full control over integrators’ system access to view live and recorded video. According to Snap One Product Manager Derek Webb, the new “customer handoff” feature provides enhanced user control after initial installation, allowing the owners to have total privacy while also making it easy to reinstate integrator access when maintenance or assistance is required. This new feature is now available to all Luma x20 users globally. “The Luma x20 family of surveillance solutions provides excellent image and audio capture, and with the new customer handoff feature, it now offers absolute privacy for camera feeds and recordings,” Webb said. “With notifications and integrator access controlled through the powerful OvrC remote system management platform, it’s easy for integrators to give their clients full control of their footage and then to get temporary access from the client for any troubleshooting needs.” 3

    • 4K Video Decoder

      3xLOGIC’s VH-DECODER-4K is perfect for use in organizations of all sizes in diverse vertical sectors such as retail, leisure and hospitality, education and commercial premises. 3

    • Mobile Safe Shield

      Mobile Safe Shield

      SafeWood Designs, Inc., a manufacturer of patented bullet resistant products, is excited to announce the launch of the Mobile Safe Shield. The Mobile Safe Shield is a moveable bullet resistant shield that provides protection in the event of an assailant and supplies cover in the event of an active shooter. With a heavy-duty steel frame, quality castor wheels, and bullet resistant core, the Mobile Safe Shield is a perfect addition to any guard station, security desks, courthouses, police stations, schools, office spaces and more. The Mobile Safe Shield is incredibly customizable. Bullet resistant materials are available in UL 752 Levels 1 through 8 and include glass, white board, tack board, veneer, and plastic laminate. Flexibility in bullet resistant materials allows for the Mobile Safe Shield to blend more with current interior décor for a seamless design aesthetic. Optional custom paint colors are also available for the steel frame. 3