Clustering Attacks on Web Apps to Find the Real Story Behind the Headlines

Clustering Attacks on Web Apps to Find the Real Story Behind the Headlines

Our goal of clustering attacks on web applications is two-fold

Security products which aim to block attacks may do their job perfectly while also reporting the attacks that are blocked. However, one of the biggest problems in the cyber security arena today is alert fatigue, where there are too many alerts to manually process. The largest data breach in history  affecting more than 41 million customer payment card accounts could have been prevented if the right action from the visible security alerts were taken. A web site protected by a web application firewall may be targeted by anywhere from hundreds of thousands of attacks to millions of attacks in a single day.

The amount of manpower given for processing and analyzing these alerts is always not enough, and the result is a flood of important data which is not handled and analyzed due to the alert fatigue. However, by leveraging artificial intelligence, we can develop sophisticated machine learning algorithms to cluster alerts to automate and consolidate those alerts, condensing days or weeks of work into minutes.

Our goal of clustering attacks on web applications is two-fold:

  1. Highlight interesting patterns inside the attacks
  2. Distill the massive amount of attacks to a few actionable incidents

Clustering can help us create a “story” out of the attacks (naming them based on behavior), making them more easily understood to a human observer and easier to analyze. For example, when seeing a cluster called, “SQL injection attack from several IPs in China using a Havij scanner,” the story behind it is much clearer than analyzing the thousands of attacks this cluster contains and trying to find the common pattern between them.

A simple “group-by” algorithm that takes the alerts and groups them by a specific attribute is not good enough. The reason is that there is not a single structure for attacks, and there is no single attribute which can define all the attacks. Thus, a more sophisticated algorithm, which considers a general distance function between attacks on web applications, is needed.

The algorithm has three main stages:

  1. Feature extraction
  2. Distance calculation
  3. Clustering of the attacks

Feature extraction

The raw data that enters the algorithm is an HTTP request that contains an attack stopped by the firewall, with some additional fields containing more data about the attack, like the source IP and the type of attack.

By leveraging our web application security domain knowledge, we extract additional meaningful features from the raw data that can help us describe the attack.

For example, the source of an attack is not defined solely by the IP. We also use geolocation services to extract more the about the origin of the IP, like source country, ISP, coordinates, ASN etc. It is also useful to know whether this IP comes from some kind of anonymity framework like TOR or an anonymous proxy.

Distance Calculations

The next task is to determine a way to calculate the distance between two attacks. This is a core stage of the algorithm as it determines when two attacks are similar, which in general is what the algorithm is trying to achieve. Calculating a distance between two points in the plane is easy – there is a precise formula to do it – but how can we calculate the distance between two URLs or two IPs?

We need to find a method to calculate the distance for every meaningful feature we have in our data, and then combine all these distances to find a single measure between two attacks.

Clustering of the Attacks

The final step is to take the data with all the extracted features and the distance measure between attacks to construct clusters of the attacks. In our case, we used a streaming clustering algorithm. This algorithm creates the clusters over time by receiving a stream of data as more and more attacks enter the system.

The importance of clustering in streaming mode is that the attacks are being delivered in real time. This method of stream clustering helps the performance of the algorithms in both time and memory, as not all the attack data is stored in memory all the time, only the current clusters with their unique features.

To conclude, clustering attacks on web applications help to understand the hidden patterns behind the attacks and to make huge amounts of data comprehendible to the human security expert. Constructing such a clustering algorithm requires more than just machine learning knowledge, it requires a high level of domain knowledge in cyber security to understand and construct the various parts of the algorithm.

To read more about clustering of attack on web applications see Imperva’s blog series.

Featured

  • It's Show Time

    I am one of those people that likes to see things get bigger and better. As advertised, ISC West is going to be bigger (more exhibitors) and better (more attendees). It’s show time in Las Vegas. Read Now

    • Industry Events
    • ISC West
  • SIA Releases New Report on Operational Security Technology

    The Security Industry Association (SIA) has released an impactful new resource – Operational Security Technology: Principles, Challenges and Achieving Mission-Critical Outcomes Leveraging OST. Read Now

  • Cyber Overconfidence Is Leaving Your Organization Vulnerable

    The increased sophistication of cyber threats pumped by the relentless use of AI and machine learning brings forth record-breaking statistics. Cyberattacks grew 44% YoY in 2024, with a weekly average of 1,673 cyberattacks per organization. While organizations up their security game to help thwart these attacks, a critical question remains: Can employees identify a threat when they come across one? A Confidence Gap survey reveals that 86% of employees feel confident in their ability to identify phishing attempts. But things are not as rosy as they appear; the more significant part of the report finds this confidence misplaced. Read Now

  • Mission 500 Debuts Refreshed Identity Ahead of Security 5K/2K at ISC West

    Mission 500, the security industry’s nonprofit charity dedicated to supporting children in need across the US, Canada, and Puerto Rico, has unveiled a refreshed brand identity ahead of ISC West. The charity’s new look includes a modernized logo with refined messaging to reinforce Mission 500’s nearly decade-long commitment to serving the needs of children and families in crisis. Read Now

    • Industry Events

New Products

  • Connect ONE’s powerful cloud-hosted management platform provides the means to tailor lockdowns and emergency mass notifications throughout a facility – while simultaneously alerting occupants to hazards or next steps, like evacuation.

    Connect ONE®

    Connect ONE’s powerful cloud-hosted management platform provides the means to tailor lockdowns and emergency mass notifications throughout a facility – while simultaneously alerting occupants to hazards or next steps, like evacuation.

  • PE80 Series

    PE80 Series by SARGENT / ED4000/PED5000 Series by Corbin Russwin

    ASSA ABLOY, a global leader in access solutions, has announced the launch of two next generation exit devices from long-standing leaders in the premium exit device market: the PE80 Series by SARGENT and the PED4000/PED5000 Series by Corbin Russwin. These new exit devices boast industry-first features that are specifically designed to provide enhanced safety, security and convenience, setting new standards for exit solutions. The SARGENT PE80 and Corbin Russwin PED4000/PED5000 Series exit devices are engineered to meet the ever-evolving needs of modern buildings. Featuring the high strength, security and durability that ASSA ABLOY is known for, the new exit devices deliver several innovative, industry-first features in addition to elegant design finishes for every opening.

  • QCS7230 System-on-Chip (SoC)

    QCS7230 System-on-Chip (SoC)

    The latest Qualcomm® Vision Intelligence Platform offers next-generation smart camera IoT solutions to improve safety and security across enterprises, cities and spaces. The Vision Intelligence Platform was expanded in March 2022 with the introduction of the QCS7230 System-on-Chip (SoC), which delivers superior artificial intelligence (AI) inferencing at the edge.