When Security Metrics Miss The Point

Before we begin, I’m going to ask your indulgence for a moment while I share something a bit personal. I know it may seem a bit odd at first, but I promise it will all come together quickly, as will its tie in with security metrics.

If you’ve ever met me in person, you would know that I’m a ‘Big Guy’ – I’m 6’1” and I go about 240. Now, if we have never had the pleasure of meeting in person, you likely have an image of a fairly round and portly guy in mind, and frankly, I don’t blame you. My Body Mass Index (BMI) is about 31% and by every medical definition ever published, I am somewhere between obese and morbidly obese.

The idea behind BMI is that a ‘healthy’ person of a given height should be within a range of upper and lower weights. A well-intentioned effort to give the general population an understanding about what their ‘optimal’ weight should be. But when we look at it closely, BMI is nothing more than a metric used by the medical profession to put some type of measurement on a person’s weight/height ratio. Unfortunately, the BMI calculation doesn’t consider the type of weight a person carries – whether its fat, muscle, or water – only that they have it. Because of the lack of context behind the BMI, it can be misleading as a person’s true health status. For example, every world class bodybuilder, all who average 3%-5% body fat, are all morbidly obese according to the BMI. Kind of strange in my opinion.

Why is this important?  Well, over the past year, I have worked incredibly hard to shed a lot of the unhealthy weight but in doing so, I’ve packed on a bit of muscle. Since muscle if far more dense than fat, only a little muscle weighs the same as a lot of fat, so by looking at my BMI, you wouldn’t know that I’ve dropped almost three pant sizes and while I’m not quite in a large, my XL shirts have plenty of room. I am arguable in the best shape I’ve been in decades, yet my BMI hasn’t changed in a year.

Now, I’m sharing all of this to prove an important point that every security executive needs to come to terms with: even though they are well intentioned, just like the BMI, security metrics can be horribly misleading.

Don’t get me wrong. I am a huge advocate of measuring your security program and leveraging those metrics to communicate risk with all of your stakeholders.  That said, all too often those metrics are used for shock and awe rather than communicating important messages around risk. I have lost count of the number of meetings I’ve been in over the years that talk about how many thousands or millions of SPAM messages that were blocked or how many open vulnerabilities there are, but never once mentioned the single phish that got in which caused a department’s worth of people headaches for more than a few days. After all, how many times have we seen the fancy PowerPoint deck talking about firewall blocks or packets analyzed, but never anything that speaks to the reduction of risk in the environment.  

After countless years of presenting to boards, executives, and colleagues, I’ve found that I’ve developed almost a split-personality when I’m asked about what metrics to track.  There are metrics that I need to manage risk across my enterprise, and there are different metrics that my executives are interested in. Sometimes they are the same, but most times they are not.

Operational vs. Risk Metrics

Whether we like to admit it or not, many of us run the operational side of security as well as the policy or strategic side. When running an operation whose sole focus is on defending us against attacks, the kinds of metrics I want collected are of very little interest to my board.  Do I care about the number of packets analyzed or the number of SPAM messages blocked? Of course I do – but it’s far more about ensuring I have enough headroom with my solution rather than the amount of risk I mitigate; and besides – I am certainly not about to ‘scare’ my board with the shock and awe numbers they generate.  Here’s an analogy I use a lot – the NTSB doesn’t report on how many miles Teslas drive every year, but they certainly report on how many catch fire. The same logic applies to metrics – we don’t need to report when our solutions are doing what they are supposed to – only when they don’t.

If you feel compelled to talk about the shock and awe numbers, do yourself (and your board) a favor and talk about efficacy – not volume. Telling your board that your anti-SPAM solution is 99.9735% effective means far more to them than saying you blocked a gazillion SPAM emails, and as a side benefit, you get to open up a dialog with around how no solution is 100% perfect. There you go – a win-win.

When we get down to it, the board doesn’t really care about how you run your SecOps – you’re the expert they have hired, so they expect you to manage what you do. That said, communicating risk to the board is also a critical function of your job and they expect you to be able to do that effectively. Understanding how your board thinks is critical to your success, but even more important is understanding that they are not security geeks, so by developing your metrics program around technical risks is not the best approach.

Your goal is not to use metrics to scare your executives, but to find metrics that they can relate to. To quote one of the most influential psychiatrists of the 20th century, Milton Erickson once said:

“Every person’s map of the world is as unique as their thumbprint. There are no two people alike. No two people who understand the same sentence the same way… So in dealing with people, you try not to fit them to your concept of what they should be.”

Milton Erickson

Ponder that for a moment. Most of us deal with boards and management teams which number in dozens of participants. Your metrics need to make sense not to the one person you are speaking to, but the dozen or so board members who all come from diverse backgrounds and experiences. You don’t have one different map of the world to deal with, but dozens – who all heard the exact same words you spoke, and who all interpreted those words slightly differently. Well planned metrics bridge the communications gap that comes with having multiple world maps in your boardrooms.

So, after all that, what are some of the metrics I rely on most? Well, I’m glad you asked. However, rather than share what specific metrics I like, I think it’s more useful to share themes I’ve found to be highly successful.

Operational Metrics:

So even after all of this, I admit I do share certain operational metrics with my executives and board.  

• SOC Efficacy: Metrics like Mean Time to Close (MTTC)/Mean Time to Resolve (MTTR) reflect the efficiency of the SOC team in resolving events and closing incidents. This is a key indicator of staffing challenges in the SOC and highlights the potential need for hiring or training existing staff. There are numerous other SOC-related measurements you can identify, so pick the ones that not only measure risk reduction, but also demonstrate value and effectiveness.
• Compound Annual Growth Rate of events and incidents: In the financial world, CAGR is a common term with a well-defined meaning. By using this metric to represent the growth of events, incidents, and attacks, the executives understand the resonating for the budgetary investments in the security infrastructure and SOC. Used hand and hand with the MTTC metric.
• Solution Efficacy: The overall effectiveness of the existing solutions.  This is where we measure SPAM, NIDS/NIPS, Anti-Virus and any other solution we have deployed.  This is also used to show the adoption rate of new measures like multi-factor authentication, privileged access management, and user certification hygiene.
• Solution Life Expectancy: This metric shows any security solutions that have less than 20% headroom or beginning to show a decreased efficiency due to changes in infrastructure, attack vectors, or business functions. Primarily used to set the stage for budgets or capital expenses.

Risk Metrics:

Ultimately, this is bread and butter of any metrics program. Each of the categories below can leverage the same data collection for both functional risk mitigation strategies, as well as communicating those risks to executives.

• Attack Metrics: Attack metrics are arguable the easiest to obtain and the hardest to use effectively and the most susceptible to succumb to the pitfall of shock and awe. Here’s the thing about attack metrics – while the month-over-month volumetrics are important, most of the rest of it is useless noise. Are we really at a point where we need to highlight the same port-scanner that hits you every month? No – we’re better than that. We will talk about the new attack(s) we’re seeing that we are susceptible to and what we’re doing about them, but let’s not waste everyone time talking about the attacks that are dropped on the floor because our firewall/IPS is doing its job.
• Vulnerability Metrics: The stalwart of the metrics world has to undoubtedly be reporting vulnerabilities. The key to effective vulnerability metric reporting is to relate them to potential financial impact to the company. Do not report a count of generic 5-tier risks (none through Critical) to the board without any insight to the financial impact of your critical systems. Again – avoid the shock and awe, but rather, put these findings into context by associating them with the revenue that could be impacted by attacks impacting those systems.  
• Availability Metrics: It seems all too often, the availability of a system is prioritized well behind the confidentiality or integrity of a system, rather than giving it an equal footing. Have you done a business impact analysis on that 30-year-old system that runs that old Cobol-68 program which just happens to drive 75% of your revenue? Well guess what?  The board wants to know you’re on it and there’s a plan to ensure its upgraded, migrated, or backed up even though there isn’t a published exploit anywhere in the world. If you’ve forgotten with the C.I.A. is, perhaps it’s time for a refresher.
• Regulatory Metrics: We all have them – whether its PCI, HIPAA, SOX, or any other government/industry related acronym, regulatory requirements are something we all have to deal with. When discussing these risks with your board, do not just talk about the gaps you have. Make sure you also articulate the potential fines, especially in this GDPR world, and how those gaps could directly impact the levels of fines faced. It’s easy to again fall into the shock-and-awe situation with this, but try to avoid it. Use as much realistic data as possible, especially when dealing with publicly disclosed fines.

So, is your next board meeting going to be filled with fear-inducing, shock and awe, BMI-type metrics, or are you going to focus on communicating those risks that the board wants to hear in a way they want to hear it?

Remember, every person in that room interprets your words in their context – not yours. Make sure your metrics bridge the maps of all the world’s before you.

An abbreviated version of this article was subsequently published on the KLogix blog and can be read here.