An Open Letter to Security Vendors – Part III

The Good.. The Bad.. The Tech..

In Part I, we discussed the issues around vendors marketing approach and the failed messaging that’s resulted. In Part II, we reviewed the need to be an honest partner rather than ‘just’ vendors, and in this third and final installment of my Open Letter to Vendors, we’re going to take a look at the technology challenges that many vendors overlook. This isn’t about the bits and bytes, but rather around the considerations we face when looking for a solution – and the challenges we face when it comes to implementation.

So, now let’s talk tech…

An Open Letter to Security Vendors – Part III

There are three fundamental points that most startups – and many vendors in general – overlook when developing solutions and tool suites. You may have come up with a great solution to a problem, or a new, cutting edge way to analyze network traffic or some other way to address the risk in my environment. However, what you really need to remember when you think about your solution set is this:

It’s disruptive, it’s complex, and it’s operationally intensive.

I’ve seen a lot of potentially amazing products fall by the wayside because the product teams had forgotten about one of these three facts. There is an old AppDev adage that goes something like “We can do it fast, we can do it well, and we can do it cheap…. pick two.” Unfortunately, too many vendors fail to realize this adage doesn’t suffice in the security realm. While most of us hardly expect to drop a solution into our infrastructure and expect perfection (regardless of what the marketing types say), we do expect some consideration to be given with how we are going to live with your solution for years to come. I’m not buying a pen test that I can find someone else to do next year; I’m putting in a solution that will likely hit end-of-life in my data center.

Let’s take each point one by one.

Your solution is disruptive:
The vast majority of security solutions are, by their very nature, disruptive. While there are certain aspects you cannot avoid, finding ways to ensure your solution causes as little disruption as possible is a key ‘win’ for you. At the absolute minimum, standing up an appliance or server takes time and effort, but when you layer on a ‘blocking’ mode, people start having cold sweats. Invariably, there is someone on the team that’s had a bad experience with an IPS/Proxy/Firewall deployment that supposedly caused a business outage, which you will be compared to. While very few of us would entertain the idea of tossing a new solution in front of a revenue-generating application without running it non-intrusively first, there are things that product teams need to consider about how potentially disruptive any solution is. IPS/IDS, Proxies, Database Access Controls, Anti-malware sandboxing – all of these have the potential of disrupting the business flow in some way, but let’s not forget about other more benign solutions like SIEM and Big Data solutions which want to ingest terabytes of data. If you think pushing all those logs to a server isn’t disruptive to the network, I have some network engineers I’d love for you to meet.

One last consideration for the disruptive argument. Agents. Are. Bad. Please re-read that again – and again if need be. I have no doubt that my infrastructure teams would chase me down the hall with pitchforks and fire if I were to suggest we push yet another agent onto the endpoints. So when I look for a solution to a problem, anything that doesn’t include an agent gets points above those that do. While there are unquestionably times when some type of agent is needed, please make sure you absolutely need it. Most operating systems these days allow you to interrogate activity over the network, so unless it’s absolutely necessary, try to find a way around the endpoint agent.

Your solution is complex:
With the possible exception of a password vaulting app on my phone, every security solution out there has a level of complexity that was not considered during the marathon, late-night, Mountain Dew driven coding sessions. You are a trusted solution that will be potentially buried deep within my infrastructure. What part of that model makes you believe it would be ok to expect to HTTP (or worse, FTP) data from the internet? How about being proxy-aware and acting like a normal client? Is that too much to ask? What about high availability and redundancy? If you’re deployed in a HA environment, do the two devices need to be hardwired together or can they be geographically dispersed? And let’s not forget my favorite “gotcha” of all… If your customer support crew cannot articulate all of the ports and services that are required across the infrastructure for your solution to function, then You. Messed. Something. Up.

It’s also important to understand how your integration with the rest of the infrastructure is going to happen. LEEF/CEF log formats, JSON/REST API’s, or STIX/TAXII data interchange formats are solid wins that should be considered from the beginning – not something tacked on at the end. Don’t ever forget you’re part of a larger ecosystem and you must play well with everyone – or you’ll be banished from the playground… And this brings me to my final point …

Your solution is operationally intensive:

In the words of the immortal Rod Serling…

Imagine for a moment, if you will, being surrounded by a cornucopia of LCD screens, the missile-command like complexity of persistent and never-ending assaults against your network. To your left, the screaming stream of log events traversing the screen at blinding speed. To your right, green, yellow, and red boxes flashing angrily as your hosts are poked, prodded and tickled. Tickled into giving up that tiny bit of data that would provide the assailant with their next foothold. You, the lone defender in this never-ending war, absorbing the mountains of data before you as you attempt to deduce your opponent’s next move as if you were Bobby Fischer studying a chessboard…

Your next move could have you home on time for dinner, or forever stuck in..

The SOC Zone..

While admittedly a bit dramatic, all too many vendors fail to understand the operational complexity they introduce into the enterprise. Think about how the folks in today’s security operations center are inundated with information. Think about how the vast majority of InfoSec teams are understaffed. Think about how critical time is when dealing with an incident. These are all things you need to look at when designing, enhancing, or selecting features to develop. Because if the end-users choose not to use your product due to the complexity, then you lose – period.

Remember, you are supposed to be a trusted partner working to achieve mutual success, which ties directly to how much of an impact you are to my team. How much overhead does it take to keep your solution up-to-date? Do we need to do ‘fresh installs’ to do a version upgrade? Do we need to keep an old version of a browser or java around because you haven’t certified on the most recent version? Does it involve a thick client on the desktop to use? These are all considerations that go into the decision process when selecting a solution and a partner. The more operationally intensive your solution is, the less likely you are to gain points.

So, all this said, how do you avoid the tech pitfalls that burn all-too-many solutions? Perhaps some of the tried-and-true methods below will give you some ideas that you can apply to your own development process. In no particular order:

• Implement a Beta program: While many of your customers can’t run beta code in production, some will be able to and others will have labs that they can use. Be selective and keep it limited to only those customers who truly want to partner with you by providing feedback, logs, sample reports, etc. Here’s the caveat – you need to take your beta sites seriously and listen to their feedback. Typically, they will require a little extra TLC due to the instability of the codebase, but they will also help you weed out bugs and usability challenges before the official release ever hits the street.
• Institute a Customer Advisory Board: A Customer Advisory Board is incredibly useful in garnering honest feedback not only about a solution but also about your marketing efforts and sales positioning, as well as providing industry insight. Putting together a CAB requires support from the entire company – all the way up to the CEO – in order to be successful. A well thought out CAB will include great customers, unhappy ones, and non-customers alike. Consider the value you get from hearing about why someone did not choose you or why a long time customer is unhappy.
• Designate one release a year for operational improvements: Make one release a year focused on operational improvements rather than what you perceive as product enhancements. Can you make a frequently used feature more readily accessible? Can you automate some repetitive tasks so the users don’t need to go through the effort? Having one release each year focused on improving the usability of your solution goes a long way in retaining happy customers.
• Post-implementation Satisfaction Surveys: I’m still at a loss as to why more vendors don’t do this. It is so incredibly easy to set up a web-based survey that there is really no excuse as to why you don’t ask for feedback about how the implementation went. Where there any challenges with the install? How did the support team do? Any suggested changes to the process? All valuable questions.

So there you have it. Hopefully, this provided some insights into what we see from this side of the desk and you gained valuable takeaways from this series. Perhaps your next marketing deck won’t be about your CEO’s experience with startups, and your sales rep will earn trust when he looks a potential customer in the eye and says, “Sorry, we don’t do that yet,” or maybe your next release will make my team’s job a bit easier.

I guess only time will tell.

Originally posted on SecurityCurrent.com.