The Problem With The Industry Analyst

Ed. Note: This piece has been in process for the better part of 2 years. It started following the 2020 RSA conference where several startups I work with were given extremely poor advice, costing them tens, if not hundreds of thousands of dollars. The more I’ve spoken to my industry peers, the more I’ve come to realize that my concerns were not only justified but shared among many of you. Hopefully, this is will come across as helpful as I intend.

If you’re a long-time reader, you are well acquainted with my passion and love for this industry. I truly feel blessed to be able to wake up every day and do what I do, and, after doing this for more than 25 years, I have met countless people along the way that feel the same. Like myself, many of us not only do this as a profession, but most have a somewhat altruistic feeling about the career we have chosen.

I guess this is, in part, one of the reasons for my deeply passionate dislike of those so-called “experts” who make up “Research Advisory” firms. I know that may seem a bit harsh, but when I step back and take a hard look at the issues these research organizations have caused in the industry I so dearly love, I am frankly just furious.

Now, before you start flaming me with emails, I readily acknowledge that there are a handful of individual analysts that I DO respect, follow, and love speaking with, but unfortunately, this is a very small lot. This piece is not around those individuals, but rather, about the firms collectively, their self-regard, their seemingly endless self-importance, and the damage they are doing to our industry.

If you’ve ever attended a webinar I’ve participated in, read a piece I’ve written, or even just been part of a casual conversation, you’ll know that this topic is near and dear to my heart. After decades of witnessing blatantly misleading or inaccurate information being used as gospel, and seeing the disasters that follow, one has to honestly wonder how these companies have (multi)billion-dollar market values.

Here’s the thing – I know I’m not alone. Many of you have also voiced your displeasure with how things are but also feel there aren’t many alternatives. While much of this article needs to be said, something truly needs to be done; and that is why I’ve decided to join TAG Cyber.

Let’s rip off the band-aid and call it what it is. Every single one of these firms is, by definition, pay-to-play. If you subscribe to their service or hire them for advice, you will be included in their ‘research’. If you don’t – or more to the point – if you can’t afford to pay the exorbitant fees, then you are on the outside looking in. You may have the best product on the market, the most amazing service anyone has ever seen, it just doesn’t matter because, you won’t pay the extortion, so you can’t play in their research.

This is never more apparent than when we start discussing the infamous super-duper 4-boxes or the extra-curvy parabolas that analysts seem to think drive purchasing decisions and big checks. Guess what? They don’t. Want to know what I (and virtually every one of my peers I’ve ever spoken to) use as the key deterministic factor in actually buying a product? It’s quite simple really – Your. Stuff. Works. As. Expected. Yep, that’s it. Not a dot on some pretty chart, or some self-important adjective-describing quadrant. Frankly, I don’t care where you are in the box or where you’re located on the curves, if your solution works as promised in a proof-of-concept, you have a good chance of being included in the evaluation process. 

What a concept, right? 

Here’s the thing. Most of us on the buying side understand that the box/curve thing is all around pay-to-play. The more a vendor pays, and the longer they pay it, generally gets you a better spot in the rankings. Now, again, there are a few analyst firms that may install and test the products, but even the independence of that process has been called into question recently.

For the most part, we are all aware that most of these analyst firms do nothing other than interview the executives the vendors provide as references, so we know the deck is already stacked in the wrong direction. How? Well, a vendor would never give an analyst a problematic customer to interview about the product. Logically, vendors will always supply their best, most referenceable customers for the analysts to interview, so the analyst only gets to hear about the benefits and successes and never about the disasters, false promises, and broken functionality.

For example, not that long ago, I was running an RFP for a global solution that was supercritical for the entire company, not just security. While going through the RFP process when we started digging into a ‘quadrant leading’ vendor’s solution where, much to our surprise, many of the responses received were ‘We’ll have to customize the application to add that functionality.’ While I would typically understand some of that from a young startup, I would never expect that from a ‘leader’ on one of these analysts’ graphs. It was painfully obvious that whoever did the research had no idea of what questions to ask or had any experience in implementing such a solution. It quickly became obvious why they got ranked at the top of the chart – and no, it wasn’t their features/functions.

The truth of the matter is, what makes a solution a ‘leader’ for one enterprise could be a detriment for another. I know this will be of no surprise to anyone, but in all of the years working in this industry, I have never come across two organizations that even resembled one another, much less being close enough to blindly use the same product evaluation. Positioning a solution or vendor as a ‘leader’ insinuates that it is the be-all-end-all for the vast majority of enterprises, which is simply an invalid assumption.

Over the years, I have volunteered on numerous Executive and Customer Advisory Boards. Time after time, I have seen countless, young, eager, startups wasting their entire year’s marketing budget on a tiny 8×8 booth in the middle-of-nowhere back corner of the RSA Exhibition Hall because ‘Such and such analyst said we had to be on the show floor if we wanted to be considered a real vendor.’ 

I’m sorry, but what?? 

Perhaps a little perspective is in order… In 2020, over 700 vendors packed the RSA Exhibition Hall floors. The Hall was open for a total of 23 hours over 4 days. This equates to an attendee spending an absolute maximum of 2 minutes per vendor if they want to try to see every booth on the show floor. which we all know is complete nonsense when it comes to the value proposition. Even if someone did find your space on the outskirts of the main walkways, do you think there would be any interest in chatting with “Yet Another Security Startup” who, thanks to the expense of the booth space and the cost of the analyst firm, can barely afford proper signage or to have the right technical folks available to run the demos?

While we could probably have a lengthy discussion about the RSA Conference itself, this is more about why these tiny start-ups are even on the show floor, to begin with. Most small firms struggle to cover the booth with adequate personnel (since they are not flying their entire company of 10 people to the conference), so you end up with well-intentioned marketing/sales reps who are sadly unprepared to have the technical discussions that generally occur. Between the cost of the booth space, marketing & signage, and travel expenses, you have to truly question the value proposition to the startup and – just as importantly – the amount of money these analyst firms are making on the outdated, useless, cookie-cutter advice of “You must have a booth at RSA”.

The way I see it, with all of the money being spent on these analyst firms, they should be able to connect the start-ups with interested CISO’s and facilitate meetings at the conference. A popular approach that’s come to light over the last few years is the off-site hotel suite meeting/demo space at a hotel close by. Wouldnt it be a great service if the firms brokered introductions between the interested CISOs and vendors, and facilitated their offsite meetings while at RSA? This way, both sides of the equation get value – the CISO gets a demo and a quiet place for a discussion, and the vendor gets an honest, valuable lead for a potential customer – both of which are hard to come by on the show floor. In this scenario, the vendor has a clear calendar of meetings and can be sure the right team is in place, and the CISO can work their schedule around those vendors who are critical to the upcoming year’s initiatives and critical directives. Win. Win.

Another interesting function industry analysts claim to provide is the definition of market segments, which conveniently, also allows them to further define and sub-segment their fancy graphs, which again, directly drives company revenue. The fact is, the more fancy-4 box charts or parabolic graphs that can be defined and sold, the more revenue the firms generate, irrespective of the impact it may have on the industry.

One such example, which is a major pet peeve of mine, is the whole “Next-Gen SEIM” segment that was devised a few years back. While I will be the first to complain about how the SEIM industry has done little to advance over the past decade, the reality is, we all still need some way to aggregate and analyze the network, application, and user activity within our infrastructures. While the legacy SEIM’s did a fairly decent job on network and application traffic, they have historically fallen short on monitoring user behavior.

Then, several years ago, tools to begin integrating and analyzing “User Behavior Analytics” (UBA) finally hit the market, allowing us to complete the triad of monitoring that was needed. These tools did not replace the legacy SEIM’s, but rather, augmented them to provide visibility into a major risk area. But then, seemingly overnight, we began to see the “Next-Gen SEIM” market segment, which was included all of these UBA tools in the same segment as the legacy tools, treating the UBA tools as if they could alone replace the legacy solutions already running.

Now, as a beta-tester of some of these UBA solutions, as well as a formal early adopter, I was struck by the complete lack of comprehension of the operational capabilities between the legacy solutions and the UBA’s of the world. Did these analysts not understand that this was an apple-to-oranges comparison? Did they fathom that, no matter how amazing the UBA solution was, I still needed to monitor network and application events? Was their vision so myopic that they were pushing an approach that would ultimately lead to more security breaches?

The hard truth is, they didn’t care as long as they sold more graphs, driving more revenue into the machine. Is it any wonder that these multi-billion dollar companies are completely out of touch with the markets they supposedly serve?

The one thing that is oft-overlooked when considering industry research is what experience the analyst brings to the table. Do they know what it’s like to operationalize a SEIM? Have they ever dealt with incidents like the SolarWinds Orion breach or the Log4j issue? Have they ever had to put their career on the line when spending millions to secure a corporate infrastructure with 20-year old devices that have been end-of-life for the last 5?

Probably not.

Ultimately, after three tours as a CISO, and 18 years of ‘sitting behind the desk’, I concluded that something needed to change. Something needed to be done to address the pay-to-play business model that is, in my humble opinion, so damaging to our industry. We need practitioners writing research, not theorists. We need business acumen when advising start-ups, not dated, cookie-cutter recommendations. We need trusted allies who can assist security teams with examples of failures, as well as successes. We need advisors who can help CISO’s navigate executive boards when the boards themselves don’t know what to ask for.

When I look back on my life, both personally and professionally, the one rule I have always tried to live by was that I shouldn’t complain about something if I wasn’t willing to change it. From starting youth recreational sports programs and helping start an education foundation in my town, to continue giving back to the security industry, donating time and experience to helping those new to the field, I have always had a certain amount of pride in how I’ve given back.

While I realize this may come off as being a bit self-serving, if I am being completely transparent, there has been quite a bit of introspection over the past 18 months, especially when it comes to my career. When evaluating all of the prospects out there, I realized that I wanted to have a bigger impact on this wonderful community that I so love, and what better way than to try to fix one of the most egregiously broken functions of our space. TAG is hyper-focused on fixing the problem that has been plaguing us far longer than COVID – and I couldn’t be prouder in joining them to fight the good fight.

So, if you’re a CISO, CIO, or a Vendor, reach out, let’s chat, and together, perhaps we can right this ship.