Why a controversial cybersecurity prediction about IDS from 2003 is still relevant

In 2003 technology analyst Richard Stiennon made a highly controversial prediction: intrusion detection systems (IDS) would be obsolete in few years.

Why?  IDS systems of that era were overwhelming IT shops with a deluge of alerts and failed to deliver on protection promises.  As an article in SC Media quoted his prediction, intrusion detection systems would be subsumed into the firewall:

“IDSs are a market failure and vendors are now hyping intrusion prevention systems (IPSs), which have also stalled. Functionality is moving into firewalls, which will perform deep packet inspection for content and malicious traffic blocking, as well as anti-virus activities.”

Bold predictions draw criticism

The bold prediction drew criticism from many sides and particularly from the vendor community.  In a contributed article to eSecurity Planet – “IDS is Dead – Long Live IDS” – one critic argued this merely moved the functionality of deep packet inspection to the network edge but didn’t change its purpose.

That critic proposed the problem with IDS wasn’t its location, but the absence of context about alerts and the inability to share data with other security tools: 

“The problem is that most packet sniffing solutions – whether an IDS, IPS or ‘deep packet inspection firewall’ are context-free. They have no idea whether an attack is relevant, and the volume of events that they produce tend to hide the dangerous attacks in low-risk noise.

Like most security operations with a layered security philosophy, each layer (and each device within a layer) is managed and run independently. There’s no intra-layer or inter-layer cooperation, communication or correlation. Simply moving the packet inspection out to the firewall doesn’t help this issue at all; the volume of false alarms will still be enormous, and the sensors will still be unaware of the larger IT ecosystem that they exist to protect.”

The solution, Mr. Hollows then argued, was to integrate and correlate heterogenous data from the vast array of tools and sensors deployed in the enterprise. Correlation of security alert data would provide the context to prioritize alerts, and avoid the fruitless and time-intensive effort of investigating false positive and trivial true positive alerts.

A prediction is proved true

Today, Mr. Stiennon stands by his prediction, though he points out the language of his prediction was more precise and nuanced. For example, he did not write “IDS is dead” in a press release – the PR team did.

More importantly, all these years later, Mr. Stiennon’s prediction was largely right. Much of the initial IDS functionality was subsumed by the firewall and many pivoted to IPS.

“All the IDS products of that day are gone, you can’t buy them,” he said in a recent phone interview.

However, there is “still need to detect intrusions” he noted. Modern methods of intrusion detection have become more intelligent and include “behavior-based” models.  

Indeed, many technology analysts have observed that the IDS market has continued to grow during the intervening fourteen years.

As for signature matching methods of detection, many of those are “just as bad today as it was then” when made his original prediction. I believe signatures on the IDS have proven to have the same fundamental issues as signatures on virus scanning software. Ultimately, signatures can be highly effective at detecting known threats but are often simple to evade.

Network-based detection is still highly necessary, as evident by the continued growth in the space, and new perspectives on the problem have given rise to new categories of products. network traffic analysis, flow analysis, and metadata extraction tools are now all trying to address the shortcomings identified fourteen years ago.

To maintain relevance, it is crucial for the industry to address the following timeless challenges.

1. Network-based detection needs new innovation.

The early functionality of IDS did get subsumed into the firewall, and as a result innovation generally stalled. Still, intrusion detection persists in most large enterprises because it still solves some problems better than the other alternatives. Tools today must extend past what the traditional IDS and firewall offer in order to address the detection needs of tomorrow.  

2. Aim to reduce alert noise

Security professionals are still struggling with cybersecurity alert fatigue.  For example, a large enterprise can receive thousands of alerts that quickly overwhelm resources. Sixty percent of large banks report receiving 100,000 alerts per day or more, and still, others say the volume can be as high as 200,000 per day. 

The fix for this is to examine threats from multiple perspectives.  This enables the collection of better information to correlate and distinguish between real threats and noise.

3. Context is critical for prioritization

Many security tools still lack context – the ability to understand what a device was doing at the moment of infection – and what it then touched next. This is harvested from network metadata and is useful in triage, incident response, and investigation.

4. Cybersecurity technology still needs integration

A recent study by the Poneman Institute found that enterprises have on average seven different endpoint security tools.  Some of the big banks have as many as 25 different tools overall. The problem here isn’t the volume of tools, it’s that none of them talk to each other. 

Data collected by security tools within an environment should be the intellectual property of the user, not the vendor.  The vendor community needs to provide the means to share security data around as their customers see fit – even to other tools.

5. The SEIM is only as good as the data source

Security analytics didn’t resolve the burden of alert deluge either. Enterprises, in essence, hooked up every data hose they had to the SEIM and waited for insight. It didn’t happen. This is because analytics are only as good as the data source. More data isn’t the answer – security needs the right data.

Redefining a market reputation

IDS has obtained a bad reputation, and deservedly so. Products in this category have been guilty of producing a high degree of noise, which at best can be distracting and at worst obfuscate critical issues in a sea of data. 

The path to reputational redemption lies in greater innovation, not less. Noise can be addressed with better context and relevance around alert data. Signatures must be balanced with behavioral detection that is harder to evade and may offer a measure of protection against zero-day threats. This is the path forward for improving the quality of across all detection products. 

While the rumors of the death of traditional IDS have been exaggerated, its relevance wanes even while the market continues to grow. In its place must be a new generation that provides the ability to detect modern threats without leaving behind the functionality that made it popular in the first place.

During our phone interview, we asked Mr. Stiennon if he had another prediction for the New Year.  He pointed to one from his book – There Will Be Cyberwar: How The Move To Network-Centric War Fighting Has Set The Stage For Cyberwar. Though the book was published in 2015, he says he predicted a battle for information dominance between the U.S. and China in 2018.