Artificial Intelligence in Cyber Security

The hype around artificial intelligence and its terminology

Today, it seems like nearly all products in cyber security are now using artificial intelligence (AI). You will probably notice that some experts are talking about a breakthrough idea while others are stoking the hype still further. The topic is already a hot topic in public discourse, but the fact remains that almost all security experts see a great deal of potential for AI to solve current challenges in the cyber security space. On the other hand, hackers are also taking advantage of new approaches that AI affords and at this point, the key question is how significant the potential and impact of this new technology is.

DL has become feasible on a large enough scale within the last few years – enabled through new storage technologies to store and access more significant amounts of data and to process this massive amount of information using enforced computing power.

Today’s challenges – Technological restrictions & detection gaps

First, it is necessary that we look at today’s challenges. When you analyze the security architecture of many companies, you see that most of them have invested a lot to get better value out of their security architecture and Security Operations Centres (SOCs). Meanwhile, companies are still struggling to protect their businesses and their data. These problems arise because internal processes and security teams are limited through technological restrictions, which are compounded by the sophistication of the attacks – and the attackers – that they face. 

Consequently, we need to overcome technological restrictions and automate processes to enable SOC teams to stay ahead of even the most sophisticated hackers. Disparate tools need to be integrated into a common security architecture, and the benefits of AI with improved performance, better accuracy, and the handling of large datasets will enable automated and intelligent investigation and response processes. 

On top of that, due to drastic increases in the amount of new unknown malware each day, the total number of malware actors, and the size of the Darknet, we have reached the limitations of signatures and heuristics. Signatures represent the fingerprint of the malicious code and help to detect and identify malware. One weakness of this approach is that simple changes in the malware’s characteristics often bypasses signatures. Some experts say that anti-virus signatures now catch no more than 30-40% of malware, though others consider that this level may be as high as 65%.

Another classic approach to the detection and identification of malware is the use of heuristics, but this also requires that the attacking methods, code, and functions are known and predefined. Consequently, the effectiveness of these reactive approaches is relatively low regarding their accuracy and the number of false positives.

Additionally, data communications are increasingly encrypted, making it even more difficult and expensive to detect the latest threats using Deep Packet Inspection (DPI) based approaches. The main drawbacks of DPI with signatures and heuristics include its reactive nature and the need to analyze each packet header and payload, which in turn requires lots of resources and makes it inefficient and expensive at scale. 

Identifying Malware, suspicious HTTP traffic, or abnormal behavior with Machine Learning 

A well-known example of ML and supervised learning – this means that we have to train the engine with predefined, labeled data – is a sandboxing solution to identify malware or malicious domains. The algorithm extracts relevant features, assigns importance to each and can then predict if the unknown input file is malware or benign based on a percentage value as shown in the table.

Another example is to analyze characteristics in HTTP headers to identify unique patterns of command-and-control behavior that usually don’t exist in normal data traffic. As a prerequisite, data scientists and security engineers need to analyze a wide range of command-and-control traffic and focus on characteristics that are common across many types of malware. 

This information is fed into the learning algorithm, and a model is generated which can then predict if HTTP based command-and-control communication occurs. Instead of reactively trying to keep up with attackers when they change domains and IP addresses, this model quickly detects command-and-control communication without using signatures. 

The classic use cases of ML with unsupervised learning – which means that you don’t have to feed the algorithm with labeled data directly – is based on the principle of finding logical groupings to identify outliers from local norms. After a period of baselining, abnormal network traffic from a host can be an indicator of malicious activity. Second, the identification of access to resources a user or host does not typically access could also show outliers from local norms. A third example is the behavior pattern which is too regular for a human. Here it is essential that outliers do not necessarily mean security incidents. It instead says that they need to be investigated and the baseline updated. 

The use of Deep Learning approaches will improve the Security Operation Centers

The most significant use case, which is expected to be a fundamental approach for the future in the cybersecurity space, is the use of DL for the threat detection and investigation model in the Security Operation Center (SOC). Today’s security engineers are overwhelmed with incoming security events without context. This problem is exacerbated by the skills shortage in the security market, and manual processes. Furthermore, detection models need to have long-term memory to retain the context of related activities over time, in order to identify slow-moving attacks. 

DL brings the opportunity to solve all these problems and to increase the value of SOCs. With statistical linkage methods, DL can understand the relation between multiple events and can then provide automatic threat scoring for compromised hosts. DL models can also assign importance to behaviors that reflect strategic phases of the kill chain process. The use of DL considerably relieves the security team, as the number of relevant events is reduced, correlation produced, and events assigned to respective steps of a kill chain process. 

A radical thesis is that security models where metadata is extracted by network sensors supported by other system logs and enrichments like IOCs (Threat Intelligence) will substitute legacy approaches based on DPI. Based on this, DL engines will identify malicious activity accordingly. There are lots of advantages, especially for high scale environments, because only a smaller amount of data needs to be analyzed. These new security models are more cost-effective than the scaling of DPI solutions, and will also solve the problem of identifying threats in the increasing amount of encrypted communication.

Nils Kammann is a Solutions Architect at Xantaro with a focus on security, and is responsible for business development and innovative, future-proof, and sustainable security solutions. He has 15 years of experience, a deep understanding of security and network technologies, and works very closely with customers and market leaders.