Understanding Gartner, and its limitations

Clint Eastwood as San Francisco Police Inspector Harry Callahan in the movie Magnum Force made famous the line “A man’s got to know his limitations”. In the world of IT, the word of Gartner is often used as the gospel. But when it comes to Gartner, everyone needs to know their limitations.

Gartner is a powerful and huge research and advisory firm. With over 15,000 employees and nearly $3B in annual revenue, it is a force to be reckoned with. Its research covers the gamut of IT; its events and conferences are attended by industry elite; its reports are often viewed as gospel. With Gartner’s close relationships with Global 2000 executives, it means that what Gartner says matters and when Gartner speaks, CIOs listen.

One of the most powerful Gartner tools is their Magic Quadrant (MQ). Gartner notes that their MQs offer visual snapshots, in-depth analyses and actionable advice that provide insight into a market’s direction, maturity and participants. An MQ compares vendors based on Gartner’s standard criteria and methodology. Each report comes with a graphic that depicts a market using a two-dimensional matrix that evaluates vendors based on their Completeness of Vision and Ability to Execute.

Vendors love the MQ as it is a powerful marketing and PR tool. For example, the Magic Quadrant for Security Information and Event Management was released on December 4, 2017 and within days, I received email from McAfee and Rapid7 touting their inclusion in the report. While the report was available for purchase from Gartner for $1,995.00, McAfee, Rapid7, SecureWorks, IBM, Splunk and LogRhythm, amongst others, had the MQ available for free.

Similarly, the Magic Quadrant for Endpoint Protection Platforms was released on January 24, 2018. By January 26, Crowdstrike and Malwarebytes were also offering complimentary versions of the $1,995.00 report.

For the CISO, CxO or any executive that is considering an enterprise security software solution, the MQ is often their go-to guide to help them determine which vendors should be on their short list. Yet as valuable as the MQs may seem, they suffer from an inherent flaw, which is that Gartner does not actually test or use the software under evaluation.

And this is the point that is lost on many people – the MQ is meant as a market analysis report, not a recommendation list. In fact, Gartner analysts will readily tell you that any of the players in typically all but the niche category are strong options that warrant consideration.

It should be noted that Gartner is quite transparent in all this. They clearly state in the aforementioned MQ (on page 36 of the 39-page report) that sources of information to support their analysis include feedback from Gartner customers gathered through inquiry calls, face-to-face meetings and survey/polling tools; vendor information supplied in response to a survey, product demonstration and briefings; and vendor reference opinions gathered via polling tool.

The value of sources such as Consumer Reports, Car and Driver and the like is that real-world testing is done. Yet the inherent Achilles’ heel of the MQ (and other market reports like Forrester Wave), is that it is based far too much on schmoozing and trust, and not empirical real-world testing.

It’s also important to understand the audience for the MQ. They are made for the IT 10%, the Fortune 1000. These organizations have deep pockets and often require the most cutting-edge products available. For the 90% who don’t have such requirements, an MQ should be seen as nothing more than a wish list for the IT rich and famous.

#ThinkBeyond the MQ

The current #ThinkBeyond effort is a campaign aimed at informing buyers of security tools about the importance of looking beyond analyst recommendations in favor of forming their own educated opinion about purchasing and deploying technology that will actually benefit their organizations.

To that, those considering any enterprise software or hardware tool need to #ThinkBeyond the MQ. Any organization that’s considering a tool needs to perform their own evaluation and not rely on Gartner; or any other research firm for that matter. Firms needs to perform their own evaluations, rather than rely on Gartner, given the importance of these tools.

As former Gartner Research Director  Ben Tomhave notes: choosing a product should be part of an architectural process that first defines and understands a problem-space, and then progresses to identifying and evaluating possible solutions for that problem-space (assuming the problem-space is even worth solving!).

With that, thinking beyond the MQ means starting with these high-levels tasks:

1. What is your security problem and how do you expect this security product to solve it?

• If the person considering the product can’t effectively and articulately answer this question, stop and do not buy the security product.
• If you don’t know what problem you are trying to solve, you will waste time, effort and money.
• The single biggest mistake in security product procurement is that people buy security products without knowing specifically why they are buying them.

2. Security strategy

• Have you taken significant time for research, planning, and designing a strategy for the product implementation?
• Did you get all divisions involved and high level (CEO, CFO) support?
• Are you able to sell this to management without using technical jargon?
• When considering a specific product, don’t look at the micro level of a security product, look at the macro level of the security of the system you want to secure.

3. Risk analysis and assessment

• Without performing a comprehensive risk analysis, products operate in a vacuum.
• An effective risk assessment and analysis ensures that you are worrying about the right things.
• Don’t forget that many significant threats come from the inside.
• The ultimate outcome of a risk analysis should be to see if you really can benefit from the product. Don’t worry about missing the bus.

4. Don’t put too much faith in customer lists and product reviews

• Vendors love to show off their customer lists. They want you to think “if company X is using our product, shouldn’t you?
• Ask the vendor how recent and involved the customers are.
• Customer lists are often inconsequential, since many products end up as shelfware. Fortune 500 companies own 1 of nearly everything.
• As to the question “which product is best?”; it depends on what your requirements are. Make your goal to outsmart Gartner, marketing and sales people.

5. Define your specific requirements via details business need and requirements definition

• Ensure an RFP defines your specific needs.
• Don’t let the vendor determine your business requirements.
• Sales people know how to sell, but don’t know about your organizations culture, budgets, operations, etc.
• You must drive your
• Remember what Ben Tomhave says: just because a firm is a leader, doesn’t mean they’re the best choice for your organization. There are very strong options in the Visionaries and Challengers quadrants who absolutely should be considered, too.

Gartner unlimited

While one should use the MQ with caution, that’s not to say that Gartner, IDC, Forrester, 451 Group, and other analyst firms are simply pushing products.  Some of the smartest people I know are at these firms. And if one is a client, their information security expertise can mean the difference between security success and failure.

There are too many to name, but a few of the smartest people in the room that come to mind are:

Anton Chuvakin – while his title is Gartner Research VP and Distinguished Analyst (are there analysts that are not distinguished?), he has a firm grip on the security space. With a PhD in physics, he understands things at both the quantum and macro level. He spent a few years at a number of vendors (we actually worked together at netForensics), and truly knows how to make security work.

Also at Gartner are smart apples such as Erik Heidt, Jeffrey Wheatman and Jay Heiser. Those needing tactical information on security metrics, security and risk management programs, or a cloud security primer for 2018; will find their research invaluable.

Over at IDC, my very clever friend Pete Lindstrom and his astute colleague Robert Westervelt wrote a report last week on how the Meltdown and Spectre attacks require careful risk analysis and thorough patch testing. Their sage advice in this and other reports, without any products being pushed, is truly a valuable information asset for any information security professional.

Conclusion

Failure does not have to be an option when purchasing security products. It’s important to understand that Gartner is a great organization if you know how to use their research, with its limitations.

Using a Gartner MQ is a great way to get an initial list of some of the products in the particular space. But it is not the definitive list by any stretch of the imagination. It’s simply a catalog of some of the more prominent commercial vendors that have raised their awareness within Gartner. There is a lot of information that can be leveraged from an MQ, but there is a big difference between leverage and blind faith.

Firms that are serious about implementing security will focus on ensuring they use the right tools for the job. It’s a custom project that most often can’t be met by simply using a generic tool like a MQ.  There are a thousand points of light that go into IT product decisions. And the MQ is but one data point of light.