forming questions to ask endpoint security vendors

I wonder how often a vendor calls competing vendors to try and get sales pitches, calls, and demos out of The Other Team? Probably less often than I’d like to think. I imagine they have enough work to do without resorting to filling time with some casual spying.

Anyway, DarkReading has posted this article, “20 Endpoint Security Questions You Never Thought to Ask” (I read that headline over a good 5 times in my best movie trailer hype voice). Even though I’m a bit snarky in my response to these questions, this article does make a good foundation for any sort of endpoint security requirements gathering might be needed when evaluating new products.

Do note that I don’t really trust sales people. I trust being hands-on with products, or at least eyes-on with an in-depth demo from an engineer. So any questions that are easy for sales to just fib about, I tend to reword into “Show me…” types of questions. I’m also crazy wary of articles written about a product segment by someone whose business lies in that segment. I get that they’re knowledgeable, but they’re also happy to slant the discussion to favor their own products, even if it’s subconsciously done.

A few key questions are missing here. Asking about licensing models is always a necessity. Asking about central management tool requirements is another. Plus, these are endpoints we’re talking about. Does this include server class systems? What about when a mobile device is off-prem? What port allowances on my network are necessary to be opened? Will the endpoints be listening and do I need to protect that opening? Are updates and central callback communication encrypted or protected somehow? To be fair, this list was about questions I never thought to ask…though, let’s also be super fair and say that most of these questions are baseline questions everyone should be asking already.

“1. How easy is your solution to deploy?” This is a fair question, but I’d reword this as, “Show me the process to deploy your solution.” I’ll make the determination on whether that is easy or not. Do I need to burn a domain admin account for this? Do I need to sit and wait to do them one by one? Do I need privileged staff? Does the tool run as local system/root or do I need service accounts? Will this discover endpoints or do I have to populate with a list or one by one? And so on.

“2. How easy is your solution to manage?” Again, I’d reword this to, “Show me how to manage the endpoints from a central management tool.” I’ll decide if it’s easy or not.

“3. How easy is it to configure rulesets and tune the solution once deployed? Aside from the fact that threats are continually evolving, if there are activities that appear malicious elsewhere but are benign in my environment, I need a way to filter those out.” For the first part, yet again, “Show me how to configure the rulesets and give me an install that I can play with directly.” I’ll decide if it’s easy or meets my requirements. For the second part, I’m not sure what examples there may be, but I might ask whether any given rule or protection can be fully turned off if I want them off, or if I can make exceptions by running multiple policies. I’ve run into tools with pieces that just can’t be turned off (Sophos!), and it can be very frustrating.

“4. How easy is it to update your solution’s knowledge base or take advantage of the latest knowledge around attacker activity? If you can’t make it easy for me to operationalize what you’re selling me on, then your solution isn’t going to work for me.” Yet again, show me how to configure updates and what gets updated.

“5. What additional load on the endpoint does your agent introduce?” I honestly don’t think this question has been relevant for many years (virtualization concerns notwithstanding), and even if so, a sales call won’t produce a negative answer. More than likely, an extensive proof of concept roll-out will be necessary to answer this. One does have to think about whether virtualized endpoints will be included. Do they all scan at the same time and overload my hosts?

“6. You want me to install yet another agent? I would be willing to do that if you articulate how you can consolidate functionality that I currently get from multiple different agents into one agent.” I don’t think this is relevant, either, unless I am looking at a tool to replace several others. Otherwise, when we’re talking endpoint security, we’re going to be talking agent-like footprint. The exception? Mainfram…I mean, fully virtualized environments where the security is abstracted out into the host/hypervisor layer.

“7. How does your solution integrate with my existing security infrastructure? I have a complex ecosystem of products deployed and yours needs to play nice with it.” I doubt this is very relevant. I mean, let’s say it’s the best product but it doesn’t “integrate” well with my infrastructure. Sorry, but that’s my problem to deal with. It might be better to ask questions about how notifications/alarms are raised, logged, sent, and handled. And to ask to see the dashboards or status checks or audit reports. However, this is definitely an internal question to raise, and there may be complex environments where this and other tools will stomp on each other for a while.

“8. Not all intrusions involve malware. What is your strategy to detect intrusions that use no malware at all?” I actually am not sure what this question is asking about.

“9. Is your solution part of an overall platform, or is it just another point product that I need to figure out how to integrate into my operational workflow?” A good question. Basically, what else do I need to get value out of your tool(s)?

“10. Does your solution leverage and facilitate correlation with other data? I have a lot of great data elsewhere in my enterprise. Do you know how to take full advantage of it to improve your efficacy?” This seems like a question specific to a flavor of solution…

“11. Is your solution based on knowledge of attacker tactics, techniques, and procedures (TTPs)? If not, how do you identify that type of activity?” At this point in the game, I don’t much care or expect visibility into the inner workings of the major players in this field. Real world roll out will tell me if updates and signatures and behaviors are stopping what I actually see coming in daily.

“12. How does all the knowledge you’re selling me on make its way into the product to help me mitigate risk?” This goes back into the previous question: I doubt I’ll ever have this visibility beyond a slick sales talking point or two that I just have to accept until I see the product in use for a year.

“13. Do you really have behavioral analysis and machine learning built into your solution, or is it just signatures and rulesets behind the scenes?” Fair enough, but a bit of a softball question that sales people hope you ask so they can hit a major talking point they’ve rehearsed in their sleep.

“14. Do you provide ability to remotely contain and remediate endpoints?” I actually really like this question, but it needs to lead into a demonstration of remote management and remediation. I’ve seen tools that do a decent job and yet are useless when it comes to proper enterprise level management.

“15. How efficient and powerful is your enterprisewide search? If I have an incident, or even if I don’t, I need to be able to slice and dice the data collected by my endpoint solution in an instant.” A good question to lead into viewing the reports, dashboards, or other logging and display capabilities. Before asking this, however, I make a point to have an idea about what sort of data will be relevant to me. Are there any metrics I want to see? Or as an analyst, what might I need to know to investigate an event (or non-event)?

“16. How effective is your solution in a real enterprise against binaries you’ve never seen before?” Another softball question. If you’re big enough and good enough to ask about this question, you’re big enough to have your internal malware team throw some things at the tool during a POC or small demo. Otherwise, this really doesn’t apply to you.

“17. What is your true positive detection rate in the wild? Results from your lab don’t interest me here.” I don’t even know what the author expects this answer to be, but it’s leading and a softball.

“18. What percentage of events and alerts that you fire are false positives? Again, results from your lab don’t interest me here.” Again, never going to get a real answer here. This is going to tie into how the tool’s logging, alerts, reports, and dashboard all work, in conjunction with how granular and complex you can tune and configure the solution. All of this working together allows analysts to tune down the false positives.

“19. What is the upgrade path for your solution? It should be a smooth and straightforward transition from one version to the next.” I’d also ask how often major versions come out, and I’d try to find someone who has used the solution to tell me about any issues with upgrades or if there are any ramifications for falling behind on patches (like one feature becoming broken over time, orphaned endpoints that haven’t checked in, or something). Walk through the upgrade process and see if it’s just about running an installer and pushing out the new version, or if it’s ugly like database upgrade scripts and other complex steps.

“20. How does your solution facilitate my information sharing initiatives?” Basically, does it do the right reports that I want? Or, at least that’s how I see this question going.

Leave a Reply

Your email address will not be published. Required fields are marked *