late night thoughts on security metrics

I have recently begun reading Andrew Jaquith’s recent book called Security Metrics on, predictably, security metrics. Andrew runs the site and mailing list. So far I have been very intrigued by his approach from my standpoint of a technical guy who likely will one day be in IT/security management. Security metrics are an inevitability, so I might as well start thinking about it in my roles.

Early on I was pleased to see Andrew tackle the problem of data sharing. It’s one of those things I firmly believe is holding us back, and illustrates our problems (and stigmas) with sharing useful information with each other. If you know where I work, I certainly can’t be very open about a damaging incident at work, especially if people at work may read my writings. And so on.

I was also pleased to see him quickly tackle the problems with ALE (Annualized Loss Expectancy) and expose it for the guesswork that it really is. Many people I’ve talked to have insinuated their disdain at something like trying to predict ALE, although few go far enough to outright challenge the general (read: CISSP) acceptance of it as gospel. Likewise, he put good solid wording to my own intuitions about scorecards, grades, and health colors, namely that they’re ambiguous and don’t mean anything. They’re really meant to start discussions, not quickly show value.

I was surprised Andrew didn’t use “pen-test” or “vuln assessment” terms when introducing his discussion on diagnostic measurements and hypotheses/subhypotheses. The method of answering diagnostic questions to prove or disprove a subhypothesis seems to be a vuln assessment to me.

One part that rubbed me slightly wrong was in the Perimeter Security and Threats section, under Attacks (pg 51-52). Andrew says, “You’ll note that [this]…leaves out such common statistics as the most commonly attacked ports and the most ‘dangerous’ external URLs. I have omitted them deliberately, because they don’t pass the ‘So what?’ test.” I’m a bit in Bejtlich’s camp when it comes to measuring and knowing your threats. Some of these measures such as top 10 ports, top 10 attacking addresses, and top 10 URLs help an organization know their threats (attackers) better. Granted, I also buy that Andrew is looking into organizational effectiveness and efficiency, and that view can still survive without looking to the external threats. Metrics paint a good picture of the past, but some measures like top 10 ports may indicate something happening right this moment that may be of some concern. Still, a minor point and not worth arguing about at all, as I accept both him and my stances as just a matter of opinion.