Intrusion Detection and Prevention Systems

The process of intrusion detection involves the monitoring of systems or networks for any signs and occurrences of events or incidents deemed either suspicious or in violation of to an organization’s security policy or rule set. An “Intrusion Detection System (IDS)” is a hardware appliance or software that has the ability to detect signs of a threat through its database of broad rule sets and, accordingly, alerts security administrators, typically via an e-mail alert. An extension of intrusion detection is the process of intrusion prevention. In this instance, an “Intrusion Prevention System (IPS) not only detects these threats, but also prevents them, usually by modifying ACLs.

---

Featured Title image comes from Raj, N., (2013), Intrusion Detection and Intrusion Prevention

---

Detection Methods

An IDS or IPS usually comes with one of three detection methods; however, some come with more than one. They are signature-based, behavior/anomaly-based, and heuristics-based. Each method has its distinct advantages and disadvantages, so which method you choose to apply to your IDS/IPS architecture is ultimately up to management and the security team.

Signature-based Detection

A signature-based IDS or IPS works identical to anti-malware programs in that the system identifies a threat based on a matching “pattern” of the known threat. For example, detecting missing flags in a TCP three-way handshake that violates RFC 793 would be an example of a signature.

The advantage of using a signature-based detection method is that they are simple to use. A signature-based IDS, for instance, will compare a packet or log entry to a list of signatures using string comparison operations. The disadvantage is that they won’t detect any unknown threats. We can somewhat mitigate this drawback to a small extent by continuously updating for more signatures. Signature-based detection systems tend to be stateless, meaning they can’t keep track or understand the state of communications between two systems.

Behavior/Anomaly-based Detection

The anomaly-based approach monitors traffic and compares it to a database of unacceptable traffic patterns by analyzing traffic flows. They accomplish this task by first creating a performance baseline of acceptable traffic flows. Then the system compares the captured traffic to the baseline to identity any significant deviations from the mean. For example, users typically take 1 or 2 attempts to login to their account. However, if Debbie from accounting has 10 failed login attempts, the system will alert to this as unusual activity. Another example would be an unusual spike in bandwidth coming from one particular system. Or, if one account unusually sends out a large number of email-messages.

The advantage of using the anomaly-based detection method is that we can now detect previously unknown threats because we are no longer relying on known signatures. The disadvantage comes with the initial implementation of an anomaly-based system in creating these initial profiles. It takes time to build a performance-base line of normal network and system activity. And, during this “training period, the system is especially susceptible to a lot of false positives. The security administrators must therefore take up time and resources analyzing each false positive to ensure it really is just a false positive. Once the profile is complete, attackers can evade alerting the system by occasionally performing malicious activity in small doses, effectively evading the organization’s detection and prevention systems.

Heuristics-based Detection

The heuristics-based detection method is a kind of stateful protocol analysis technique in which the system is using  vendor-developed best practices and characteristics (protocol models) and compares them to observed conditions in order to identify and fix issues within the network. Heuristic means “rule of thumb” and some systems even rely on  uses artificial intelligence.

Unlike the signature-based method, the heuristics-based method is able to keep track and understand the state of communications, transport, and application protocols on the network, similar to third and new generation firewalls. This allows the system to keep track of requests and their responses. The biggest disadvantage to using a heuristics-based IDS or IPS is that they are very resource-intensive due to the overhead involved during stateful protocol analysis.

IDS and IPS Technologies

An IDS and IPS can be further divided into 4 groups based on the specific events they monitor. They can be network-based, host-based, or wireless.

Network-based

A Network-based IDS (NIDS) and a Network-based IPS (NIPS) monitor particular network segments and analyze the traffic for suspicious activity. They are usually deployed on the network perimeter or at the perimeter of a server cluster.

This slideshow requires JavaScript.

Host-based

A Host-based IDS (HIDS) or a Host-based IPS (HIPS) monitor a specific system. They monitor traffic going into and out of the host, system logs (log-based detection), running processes, application activity, configuration changes, file modification, and so forth. Security administrators often deploy a HIDS or HIPS on their critical systems only.  HIDS can be proactive or reactive. A “reactive” HIDS informs the user of suspicious activity after is has happened whereas a “proactive” HIDS sniffs the incoming traffic and alerts the user in real time.

Wireless-based

A Wireless-based IDS (WIDS) is like a NIDS, except it is monitoring wireless network traffic. Wireless-based detection systems are great at discovering malicious or benign rogue access points.

What An IDS/IPS Can Do For You

An IDS/IPS possesses a lot of information gathering capabilities, so pay attention to what capabilities your product can provide for you. A good IDS/IPS should be able to identify all the hosts on your network by hostname, IP address, and MAC address; that way, when an unauthorized device connects to your network, you’ll be promptly alerted. Some IDS/IPS will even identify the applications in use on your network. Through application whitelisting and the proper configuration of your IDS/IPS, you can be alerted when vulnerable or unauthorized applications are in use on your network.

Another neat capability about an IDS/IPS is its ability to detect unauthorized network configuration changes. By keeping track of the hop count between devices, an IDS/IPS can detect when a device has been re-configured or moved on the network. The awesome power of an IDS/IPS is in their extensive logging capabilities, including organizational details, such as timestamps, session IDs, layer 3/4/7 protocols, source/destination IP addresses, source/destination port numbers, the number of bytes transmitted, and state-related information. They even provide you with an alert type, rating (e.g., priority, severity, impact, and confidence), and any prevention action performed.

As for detecting suspicious network activity, an IDS/IPS can detect application layer reconnaissance and attacks, such as banner grabbing, buffer overflows, injection attacks, brute force attacks, malware, and so forth. An IDS/IPS is monitoring multiple TCP/IP protocols and applications as well. At layer 4, they will detect any types of authorized or unauthorized port scanning. At layer 3, they will detect IP address spoofing, IP header manipulation, and ICMP error codes.

In some IDS/IPS, they will alert to unauthorized application services or any security policy violations pertaining to forbidden Web site visits, tunneled protocols, and much more. Ultimately, it comes down to the IDS/IPS administrator to specify what characteristics of activity should not be permitted and what to alert to. Fortunately, since there are many thing to be on guard for, most of these IDS/IPS already come with a rich set of rules that you can apply. And, of course, you can craft your own rules as you see fit! IDS/IPS rules are much more robust than firewall rules.

An IDS/IPS can also use “thresholds” in which there are set limits on normal and malicious behavior. For example, after a certain number of failed login attempts in a specified time range, the system will alert to a brute force attack.

IDS and IPS Components

Host-based detection and prevention systems are usually just software that you install on the system you wish to monitor; however, network-based detection and prevention systems are a little different.

NIDS and NIPS, for example, rely on sensors. These sensors, or “agents” are either software or hardware appliances. The sensors send information about network traffic to a centralized device called a “management server.” The management server performs additional analysis of the network traffic. An IDS/IPS architecture can use one or more management servers or they may not have a management server at all. Most IDS/IPS architectures, however, are implementing at least one management server or a database server. To manage this architecture, security administrators can login to a console to configure sensors, apply updates, analyze network traffic, customize and print reports, and monitor their status.

Network-based IDS and IPS

Network sensors can be placed in either inline mode or passive mode. In “inline” mode, the sensor itself is placed directly in-the-line of network traffic. Like a firewall, the traffic is passing through the sensor. The primary motivation for placing a sensor inline is so that it can stop or “prevent” malicious traffic. For that reason, the inline mode is used for NIPS rather than NIDS.

Inline sensors can prevent malicious activity by performing inline firewalling. These sensors offer firewall capabilities that drop malicious packets or terminate connections. Or, it could disrupt an attack by reconfiguring the ACL on another network device, such as a firewall or router. They can also throttle bandwidth usage. For example, if an inline sensor notices a DoS attack for a specific protocol, such as ICMP, the sensor can limit the percentage of ICMP. These sensors, however, are much more powerful than a standard firewall.

In “passive” mode, the sensor is placed out-of-line so that no traffic travels through the sensor. Instead, the sensor receives a copy of the traffic and sends it to a management server for analysis. These sensors are usually placed at important network segments, such as on a DMZ subnet. To deploy a passive sensor, you can install it on a switch’s spanning port or on the physical cable medium itself through a tap device. If you are using multiple passive sensors to monitor a high-traffic area, you can install a load balancer on the network tap to evenly distribute copies of network traffic to the sensors. This way, no sensor is easily overwhelmed with the amount of traffic or corresponding analysis. Just keep in mind that splitting the network traffic among multiple sensors can reduce detection accuracy. Of course, with the passive sensor no longer inline with traffic, the system cannot stop or prevent malicious traffic, but it can detect and alert security administrators to suspicious traffic.

 

Actually, a passive sensor does somewhat possess prevention capabilities. A passive sensor can perform “session sniping” by sending TCP RST (Reset) packets to each endpoint observed in a malicious connection. This is useful for TCP connections, but won’t work for ICMP or UDP connections.

 Limitations and Other Concerns

An IDS/IPS architecture sounds great, but they are not without their flaws. An IDS or IPS cannot detect or prevent attacks within encrypted network traffic. HTTPS, SSH, and encrypted VPN connections are therefore hard to analyze. One way to sidestep this limitation is to place sensors in areas where they can analyze the traffic either before the traffic is encrypted or after the traffic is decrypted. For example, traffic coming in from a VPN concentrator is encrypted, but traffic leaving the VPN concentrator and has since been decrypted can be monitored.

Furthermore, a NIDS/NIPS cannot perform under high loads. Both inline and passive sensors will start dropping packets under high loads. Some ways to avoid this are deploying sensors that can perform only a partial analysis during high loads or drop low-priority traffic. Some vendors are installing high-bandwidth NICs on their sensors too.

Also, keep in mind that though an IDS/IPS is there to detect and prevent attacks, they are not immune to attacks themselves. For example, an IDS can detect a DoS attack, but the sensor (whether a dedicated hardware appliance or software installed on a network node) can fall prey to a DoS attack too. Some security administrators prevent this by putting the IDS/IPS architecture on their own separate network, called a “management network.” In this scenario, the management servers, database servers, and sensors are all connected to the management network only, not the organization’s standard network. This helps conceal the IDS/IPS components from attackers, but it’s also a more pricey option due to additional costs in network equipment.

To keep IDS and IPS components safe, administrators should have separate accounts, each assigned with the least amount of privileges required to perform their job. IDS and IPS components should also be configured in a way that it limits direct access to other network devices that do not require direct access. This helps contain an attack, should it occur.

Conclusion

An IDS/IPS may take some tuning and customization, but they are definitely worth the management. My next post will probably be about writing Snort Rules. Snort is an open source NIDS and NIPS.

 

References

Raj, N. (2013). Intrusion Detection and Intrusion Prevention [Featured Title Image]. LinkedIn Corporation. Retrieved from https://www.slideshare.net/nikhilraj149/intrusion-detection-and-prevention-system

Scarfone, K., & Mell, P. (2007). Guide to Intrusion Detection and Prevention Systems (IDPS)). National Institute of Standards and Technology. Retrieved from http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-94.pdf

University of Maryland University College. (2012). Intrusion Detection and Prevention Systems. [Online Module]. Retrieved from the University of Maryland University Website