What is the difference between signature-based and anomaly-based intrusion detection system IDS technologies?

With the profusion of e-commerce web sites, online banking and other high profile applications, it is understandable that organizations should want to avail themselves of the best possible protection against unauthorized entry.

But, the threat of network intrusion is much wider than those heavily publicized incidents of web site defacement would have us believe. In fact, it would be a misnomer to imply that this is just an Internet-only problem. In reality, the threat of network intrusion hangs over any organization that possesses a network that is open to the outside world.

Because the byword of every modern organization is connectivity, even those companies that have no direct Internet presence remain vulnerable to hacker attack and intrusion. Just because you don't have a web site or, equally, because your site doesn't feature any e-commerce capabilities, doesn't make you immune to the possibility of someone gaining unauthorized access to your network. Most organizations running a network have the capability to allow members of staff and even outside contractors to connect to their systems remotely. This makes it easier for workers to connect from home, or while on the move. It also renders the network susceptible to unauthorized entry by third parties.

So, what's the answer? Well, one of the most prevalent solutions is the installation of a sophisticated firewall system. Undoubtedly, this can help 'hide' major parts of your system from unwanted attention. But, the problem remains that we still need to provide external connectivity, data communications, Internet access and maybe even voice-over IP (VoIP) for the organization.

This is where intrusion detection comes into the equation. Think of it as a well-trained guard dog, and you'll get the general idea. Now imagine that the rooms in your home represent your network, and the perimeter fence represents your firewall. Being the prudent householder, you'll realize that some visitors onto your premises may not be welcome. Now, because you have a gate to allow you to mingle with the outside world, and vice versa, this leaves you vulnerable to the attentions of these undesirable individuals. And this is where your trusty guard dog makes its presence heard.

Because your guard dog has been trained to sniff out unwanted guests, it sounds a warning whenever it detects the presence of any unauthorized third party coming through the gate. And this is the basis of intrusion detection. Just as firewalls need open gates in them to enable communication, intrusion detection either sits behind the firewall to warn of unauthorized entry into the network, or in front of the firewall to see who is approaching the gate. But, while there are many intrusion detection solutions on the market, some are more efficient than others in the elimination of what we term 'false positives,' as well as in the correct identification of unauthorized traffic.

Most intrusion detection systems (IDS) are what is known as signature-based. This means that they operate in much the same way as a virus scanner, by searching for a known identity - or signature - for each specific intrusion event. And, while signature-based IDS is very efficient at sniffing out known s of attack, it does, like anti-virus software, depend on receiving regular signature updates, to keep in touch with variations in hacker technique. In other words, signature-based IDS is only as good as its database of stored signatures.

Because signature based IDS can only ever be as good as the extent of the signature database, two further problems immediately arise. Firstly, it is easy to fool signature-based solutions by changing the ways in which an attack is made. This technique simply skirts around the signature database stored in the IDS, giving the hacker an ideal opportunity to gain access to the network.

Secondly, the more advanced the signature database, the higher the CPU load for the system charged with analyzing each signature. Inevitably, this means that beyond the maximum bandwidth packets may be dropped. So, feeds may have to be split and then recombined after analysis, increasing complexity and cost. In addition, it means that the greater the number of signatures searched for, the higher the probability of identifying more false positives.

Also, because an attacker knows that the IDS will trigger an alarm when it detects certain attack signatures, that hacker will tend to evade the IDS by disguising the attack. For example, hackers are aware that signature-based IDS traditionally has a problem with the complexities of application interactions. This is compounded by the fact that application protocols have become increasingly complex as they expand to provide support for features like Unicode.

Briefly, Unicode allows uniform computer representation of every character in every language, by providing a unique code point or identifier for each character. Unicode is a standard requirement of well-known computer languages such as Java and XML, making it a feature of many modern operating systems. Because signature-based IDS can miss characters written in Unicode transformation format, it is easy for an attacker to submit a URL containing an exploit that would allow other programs to be run and files accessed on the host computer.

And, because of the hackers' tendency to continually test and probe, it is only a matter of time before they discover a way around even the most sophisticated signature-based intrusion detection systems.

Any organization wanting to implement a more thorough - and hence safer - solution, should consider what we call anomaly-based IDS. By its nature, anomaly-based IDS is a rather more complex creature. In fact, to use our earlier analogy, it's like our guard dog personally interviewing everyone at the gate before they are let down the drive. In network traffic terms, it captures all the headers of the IP packets running towards the network. From this, it filters out all known and legal traffic, including web traffic to the organization's web server, mail traffic to and from its mail server, outgoing web traffic from company employees and DNS traffic to and from its DNS server.

There are other equally obvious advantages to using anomaly-based IDS. For example, because it detects any traffic that is new or unusual, the anomaly method is particularly good at identifying sweeps and probes towards network hardware. It can, therefore, give early warnings of potential intrusions, because probes and scans are the predecessors of all attacks. And this applies equally to any new service installed on any item of hardware - for example, Telnet deployed on a network router for maintenance purposes and forgotten about when the maintenance was finished. This makes anomaly-based IDS perfect for detecting anything from port anomalies and web anomalies to mis-formed attacks, where the URL is deliberately mis-typed.

Anomaly testing requires more hardware spread further across the network than is required with signature based IDS. This is especially true for larger networks and, with high bandwidth connections, it is therefore necessary to install the anomaly sensors closer to the servers and network that are being monitored. The rationale here is that the amount of data is lessened the closer the sensors are to the application, than if they were located close to or at the network backbone. Placing them too close to the main backbone simply results in too much data being detected.

Anomaly-based detection certainly isn't the straight-from-the-box solution that signature testing purports to be. Once properly installed, any anomalies detected need to be analyzed by trained human operatives. Some may argue that this makes an anomaly-based solution much more of a 'hands on' service than signature IDS. But, looking at the amount of labor involved in nursing a normal signature-based IDS, I would argue that this is not the case.

Even the largest enterprises frequently lack the necessary experience for analyzing signature and especially anomaly-based IDS. This type of security monitoring often requires a connection to a security operation center. So, because IDS can only operate as a process, these IP security centers of excellence have a constant eye towards to the Internet for new and emerging types of attacks. In fact, returning to our analogy, the guard dog has to be constantly retrained, as visitors to the gate may carry different packages or simply dress differently to avoid detection.

All in all therefore, signature-based IDS only scratches the surface of what most organizations need to protect against, because it relies on spotting a duplication of events or types of attack that have happened before. Anomaly testing requires trained and skilled personnel, but then so does signature-based IDS. And, anomaly testing methods can be guaranteed to provide far more effective protection against hacker incidents. It also means that, because of the involvement of the human element, there is a valuable additional tier of defense between your organization and the evils of the outside world.

Arnt Brox () is CEO of Proseq (www.proseq.net) and chair of the Norwegian Network Security Research group.

Signature intrusion detection systems (SIDS) are based on pattern matching techniques to find a known attack; these are also known as Knowledge-based Detection or Misuse Detection (Khraisat et al., 2018). In SIDS, matching methods are used to find a previous intrusion. In other words, when an intrusion signature matches with the signature of a previous intrusion that already exists in the signature database, an alarm signal is triggered. For SIDS, host's logs are inspected to find sequences of commands or actions which have previously been identified as malware. SIDS have also been labelled in the literature as Knowledge-Based Detection or Misuse Detection (Modi et al., 2013).

Figure 1 demonstrates the conceptual working of SIDS approaches. The main idea is to build a database of intrusion signatures and to compare the current set of activities against the existing signatures and raise an alarm if a match is found. For example, a rule in the form of "if: antecedent -then: consequent" may lead to "if (source IP address=destination IP address) then label as an attack ".

SIDS usually gives an excellent detection accuracy for previously known intrusions (Kreibich & Crowcroft, 2004). However, SIDS has difficulty in detecting zero-day attacks for the reason that no matching signature exists in the database until the signature of the new attack is extracted and stored. SIDS are employed in numerous common tools, for instance, Snort (Roesch, 1999) and NetSTAT (Vigna & Kemmerer, 1999).

Traditional approaches to SIDS examine network packets and try matching against a database of signatures. But these techniques are unable to identify attacks that span several packets. As modern malware is more sophisticated it may be necessary to extract signature information over multiple packets. This requires the IDS to recall the contents of earlier packets. With regards to creating a signature for SIDS, generally, there have been a number of methods where signatures are created as state machines (Meiners et al., 2010), formal language string patterns or semantic conditions (Lin et al., 2011).

The increasing rate of zero-day attacks (Symantec, 2017) has rendered SIDS techniques progressively less effective because no prior signature exists for any such attacks. Polymorphic variants of the malware and the rising amount of targeted attacks can further undermine the adequacy of this traditional paradigm. A potential solution to this problem would be to use AIDS techniques, which operate by profiling what is an acceptable behavior rather than what is anomalous, as described in the next section.

Page 2