Hotwash metrics


Estimated Reading Time: 2 minutes

The Insidious Nature of Security Metrics

As anyone who has built a successful information security program can tell you, metrics are a fundamental key to any long-term success. Personally, I am a firm believer in the adage ‘That we cannot measure, we cannot improve’, so while reading a COVID-19-related article this past week around infection and death rates, the article’s relationship to the information security space became readily apparent. The message was clear – how the data is presented is as important as what data is presented.

The article reviewed how the media uses a logarithmic scale to present the critical statistics around the COVID-19 pandemic, and, more importantly, how the general population struggles to interpret the criticality of such scales. Recently, a team of scientists conducted an experiment where they took the same, identical data points around COVID-19 infections and plotted them separately, one on a linear scale and one on a logarithmic scale. They then took the two graphs and asked around 2,000 people the same basic questions of both of them. Over 83% of the respondents deduced the correct answers while looking at the linear graphs, but only 40% were correct while using the logarithmic graph. Anecdotally, when the respondents were asked which chart represented a worse infection scenario, many picked the linear scale.

Image

As I’ve written about before, the metrics we use to communicate to our boards and executives are a double-edged sword. When chosen thoughtfully, they can communicate an evolution of security maturity without instilling Fear, Uncertainty, and Doubt. However, as many of us have operational aspects of our responsibilities as well, so we also must manage and communicate the sheer volume of events and attacks that occur daily. For many enterprises, this equates to hundreds-of-millions to billions of events a day, of which, only a very small subset of those are investigated as potential incidents. From a statistical reporting perspective, this is eerily similar to the infection vs death rate statistics that the researchers used in the above experiment.

I think it’s an interesting thought experiment to consider why we presume that something we understand is comprehensible by others just as easily. Does your board truly grasp the messages you’re trying to convey with your statistical reporting, or are they just blindly nodding in agreement because they are familiar with the slides you present month over month? After reading this research piece, I will be completely revamping my board metrics to use different graphs and charts in an effort to reinvigorate the message. If this works well, perhaps I’ll make this a regular practice. Same KPI’s, same data, just different visualizations.

Is it time for you to do the same?


The Chronicles of a CISO Weekly Hotwash is a recap of critical stories and news that occurred over the past week. The goal is to hopefully share some insight and perspective on how these stories may impact our industry, our careers, or our companies.


Copyright © 2002-2020 John Masserini. All rights reserved.


Share!

Leave a comment

Your email address will not be published. Required fields are marked *