If you’re in infosec and you blinked, you’ll notice today that “threat hunting” is a thing. It’s more than a thing, it’s quite the movement (though probably ranking below blockchain, machine learning, and AI as far as infosec marketing buzzwords). Wikipedia’s page on the subject was created mid-2016. I spent about an hour doing Google searches trying to find earlier mentions of the term/process, but really found nothing. I think Sqrrl/David Bianco [pdf] was one of the earlier describers of threat hunting.

So, where did threat hunting come from?

First, threat hunting is about looking for threats already in your environment. Underlying this process is the assumption that attackers can and have gotten in, and that your current protections are not enough (with an optionally-implied “advanced threat” assumed). This process identifies weaknesses, compromises, and informs defenses. This stands in contrast to reactive technologies like IPS and SIEM alerts, or preventative processes like Antimalware defenses. Threat hunting by skilled analysts has them creatively hypothesizing about attackers, examining evidence and logs for confirmation of a weakness or presence of an attacker, and then informing defenses to protect against them.

Threat intelligence is often pulled into this mix, which is just a fancy way to saying, “increasingly detailed, technical information about attackers and the things they do when attacking.” If you have a mature threat hunting process, you’re helping create that internal intelligence, rather than just consuming outside sources.

Cool, but where did this come from? Why weren’t we doing this 10 years ago?

Ten years ago, we had blue teams (defenders) against red teams, pen testers, and real attackers. The blue team would wield various security tools and try to prevent, detect, and mitigate against those attackers with somewhat standard playbooks and relatively rigid signatures and alerts.

We’ve had various enlightened members on both sides who have contributed to threat intelligence over the years, and plenty of blue team members who have tried (probably largey in vain) to squeeze more out of their tools and high level access in their environments to sleep better at night. So, many of the tasks that a threat hunter may go through are known, but certainly had never before been truly organized, planned, or laid down in writing; they’re tasks many of us have tried to go through when work is light (hah!). One difference today is the ability to gather and parse massive amounts of data collected in an environment.

It has been long found that environments are becoming more complex (i.e. the attack surface keeps increasing, networks are becoming less efficient to defend, and no external consulting source can quickly given guidance beyond broad strokes and “well, it depends…” conversations) and relatively static signature-based tools don’t find even mediocre incidents. Blue teams have traditionally been poor about internalizing red team findings and creating effective mitigations. And red teams, particularly vulnerability assessment and pen testing teams have become a crowded market with several of their tasks somewhat commoditized through automation and point-in-time limitations.

Threat hunting teams can shore up plenty of these weaknesses. Rather than a point-in-time external pen test, threat hunting teams can act as internal, always-on pen testers; they think like attackers, study attacker methods, determine IOC, and inform and test defenses.

Threat hunting helps turn an infosec team from believing they are not compromised, to at least knowing they are not compromised via X, Y, or Z. And in today’s landscape, the X, Y, and Z incidents hitting the news are what board members ask about.

Threat hunting also acts as an outlet to some otherwise mundane tasks in infosec: analyzing logs and alerts and directing the rollout of a new patch every week. A way to exercise some creativity, learn more about attacker methods and tools (poppin’ root shells, my dudes!), and actually test internal controls and visibility.

They also fill a small gap in between security tools, blue teams, and red teams. Sure, red teams can detect weaknesses and security tools can act as a locked door or tripwire, but what about things the red team misses, or what about attackers who have already leveraged those weaknesses to get into an organization and evade security tools?

Ten years ago, threat hunting probably only went as far as a blue team admin finding something “weird” happening on a server or workstation, and investigating it to find an existing compromise. They fix the compromise, wonder how someone/something got in, wonder if it impacted anything else, and then likely moved on when those answers were not going to be efficiently forthcoming.

Sounds like a no-brainer to start doing this, yeah? Well, not so fast.

To my mind, the main challenge with threat hunting is being able to log, capture, and know everything (or enough) in an environment such that testing a hypothesis is efficient. An organization that is poorly logging things, has poor standards, and struggles to control the environment, will have a very uphill battle to do any real threat hunting. At that point, you’re walking into a forest inferno and looking for dumpsters that are on fire.

I missed this getting announced, but a few weeks ago a new CIS Top 20 Critical Security Controls doc came out, version 7. There is a registration wall to get past (and one of the few times I had mailinator denied ever), but that shouldn’t be a problem for anyone in security.

This new version chops the list of 20 slightly differently, and actually moves the previous #3 item about secure configurations down to #5. Instead of typically advising to tackle the top 5 items as a priority, the top 6 items are now considered “Basic” controls. Items 7 through 16 are “Foundational” controls, and 17 through 20 are “Organizational.” While adding a different visualization/chunking to the list, this really doesn’t change anything. I do like that this results in vulnerability management appearing at #3 right below the two inventory controls. I think this is appropriate. Secure configurations are hard (“yuck, documentation!”), and so many people lose steam at that control.

I do like the change to 17 away from the awkward skills assessment and gaps wording to being more standard as a “Security Awareness and Training Program.” Previously, this always sounded like a way to train internal security staff (and was probably worded that way to promote training from the previous custodians of the list, SANS), which then left questions about security awareness programs in general.

There are also many more individual sub-controls under each control, which I really like. In the past, I could usually add one or two extra bullets under each control just to fill it out a bit, but I feel like these are fairly solid so far.

There are still some minor gaps, however. For instance, traditional physical security isn’t present, though that usually falls into a facilities sort of department. Cloud security isn’t really a thing on its own, though clearly every control on-prem will have analogous controls in the cloud, depending on what sort of cloud presence is maintained. I’d love to see Threat Hunting functions get rolled into Pen Test/Red Team Exercises. I always have to think twice about control #7: Web and Email Security. It just seems like it should be included in other items, but it’s a large enough attack surface that I get pulling it out. And I also always wince that an entire appsecdev section gets shoved into a single control down at #18.

There are also very few call-outs to documentation and diagrams. They’re so valuable for insiders, outsiders, new employees, new vendors, and so on to get a quick handle on how a network is laid out, critical data flows, attack surface, and high level posture. No space ship general lacks for a holographic display of the important pieces, and I wish most items called out more artifacts like policies, diagrams, and such in the sub-controls.

Lastly, #10 Data Recovery Capabilities is always a tough one for me. In my mind, this is control #0 that every organization should have: Backups! And it’s also probably the one control that has the smallest infosec scope in the list. Do you do it? Is recovery proven and ready? Retention is set in policy? Cool, move on! Eliminating this as a top 20 control would free up a slot for something else. Some inflate this item into BCP/DR processes or otherwise blow it up into Availability or Resiliency in general. I get it, but the sub-controls don’t reflect it.

Overall, are these huge changes? No, but they do reflect incremental changes in our landscape that bring the list up to modern standards. And this list remains one of my primary and initial roadmaps for infosec in organizations.

Panera is going through a bit of a debacle last night and today. The original security researcher posted about it in detail after shit hit the fan. Near the end, he poses a few reflective questions. I figured I would poke at them!

“1. We could collectively afford to be more critical of companies when they issue reactionary statements to do damage control. We need to hold them to a higher standard of accountability. I honestly don’t know what that looks like for the media, but there has to be a better way to do thorough, comprehensive reporting on this.”

I think everyone in media and public relations would say this is about understanding the short attention span of media and their audience. Plus, every media outlet wants to get out there first, at least amongst their peers. (This is probably why, as a fan of CNN, I continue to be appalled at their lack of spelling and grammar over the past couple years…) But, yes, I’ve been sick of the vague, cookie-cutter statements for 15 years now. “We take security seriously…” and, “affected by a sophisticated, advanced attack…” and, “no further signs of abuse/disclosure (within only 60 minutes of discovery)…” The problem is one of transparency. The company usually has no reason to be very detailed, which means someone in the know (either inside or the researcher or someone who pieces it together and rediscovers it independently) needs to reveal the details responsibly. And I usually fall on the side of full disclosure as opposed to no disclosure or “responsible disclosure, which really just means stifling it with a smile.” It’s the best way for all of us to learn and get better, and also make educated choices about where to do business.

As far as holding accountable overall, that’s a rough one. While security companies and other Business-to-Business (B2B) firms can struggle after a breach, I don’t know of any retailers or food service companies that have been terribly impacted by a breach. Food quality scares can threaten Chipotle, but breaches seem to get ignored. This Panera one is a little different as the platform affected was a mobile ordering app, which is probably used by slightly more connected and savvy users; those that may never use it again due to this, but will probably will eat at Panera.

“2. We need to collectively examine what the incentives are that enabled this to happen. I do not believe it was a singular failure with any particular employee. It’s easy to point to certain individuals, but they do not end up in those positions unless that behavior is fundamentally compatible with the broader corporate culture and priorities.”

This is a meaty issue, for sure. I think many security issues don’t get exposed or talked about, because it is always (always!) easier to consciously or unconsciously ignore it. It’s hard finding security issues; sometimes you have to really try. And many people are just trying to complete their tasks and get through their days. The security team (if there is one) has this responsibility, but in a world where development wants to go at the speed of agile, no one can slow them down. Security has to move at that speed as well, which is difficult since security inherently is always behind the curve a little bit, and often perceived as adding no value.

This starts with a mandate or at the very least interest in security from a high level. It’s fine enough to say, “We don’t want to be the next Equifax/Panera/Boeing.” From there, security management needs to be based on fact, and not belief. It also needs to constantly be questioned and improved. None of us know how to secure everything; we learn incrementally or seek assistance/ideas elsewhere, and then weave that into our internal security fabric over and over and over. When that improvement process stops, gaps will appear. (To me, this is where threat hunting is becoming a thing; you can get pretty good with your security posture and do the basic things well to go, but to keep improving, you need that unit that keeps hunting, poking, learning, and injecting further information into the whole.)

“3. If you are a security professional, please, I implore you, set up a basic page describing a non-threatening process for submitting security vulnerability disclosures. Make this process obviously distinct from the, “Hi I think my account is hacked” customer support process. Make sure this is immediately read by someone qualified and engaged to investigate those reports, both technically and practically speaking. You do not need to offer a bug bounty or a reward. Just offering a way to allow people to easily contact you with confidence would go a long way.”

I basically agree with this, but also make sure that investigations are done properly (carefully) by your IR team, and make sure someone has a quick line to PR/legal if questions arise or things escalate. Lack of response can be appropriate, but that should be stated and at least give the correspondent some indication an email was successfully delivered. Basically, security@ email address should suffice, and optionally a quick mention on an appropriate Contact Us page.

Here’s a rant that makes me look pretty stable. 🙂 Nick Selby’s post, “Do You Make Users Rotate Passwords? Well, Cut It Out.” I agree with the general sentiment, and I get the annoyance, but not so much the general way this is presented without making some qualifications.

Just to get the elephant in the room out of the way: All of this discussion is somewhat moot once we throw in the requirement of multi-factor authentication. Which makes sense, especially as biometrics (slowly) continues to make headway, which is like a password we won’t ever be able to change.

I’m also not making any assumptions on password strength, either chosen or forced. I can’t expect every user to practice good password hygiene, so I can’t really add that to my arguments. I’m also not going to make assumptions on complexity requirements forced on users by the system.

OK, let’s first make some distinctions: corporate/employee* vs consumer sites. There are two major types of accounts I have in mind with this discussion. First, accounts that an enterprise uses to identify its employees, usually set up and managed by IAM/Help Desk folks and rotated every 90 days or so, and removed upon termination. These employees typically come to an office and sit at a workstation to log into. Second, there are accounts used on consumer sites such as Gmail, Amazon, DreamHost, FaceBook, etc. These are usually set up via self-serve and probably don’t force changes except when compromise is suspected to some degree.

In the former, there are arguable times when these account passwords may get divulged or known, such as a Help Desk worker doing “something” on your system to troubleshoot an error over a lunch break and wants to log in after the screensaver kicks in (hey, another rant-worthy piece of bait!). There are too often small one-offs where an account password is shared. I hate it, but I understand it happens. There are still two use-cases for enterprises to use proactive password rotations. If a user has a shared password and needs that friendly reminder to change it, or if an unknown compromise of a password has occurred, the forced rotation of a password will close both of these gaps.

For consumer site accounts, users are left to shoulder the responsibility of the confidentiality of their passwords. If they share it with someone else, it’s on them to change the password after usage expectation has ended (ever “borrow” someone’s Hulu account longer than intended?). For consumer sites, the onus for keeping your password to yourself is on the user, with the only exception being a failure in the security of the site, which can have two outcomes. First, a compromise is suspected/known and all affected users are asked to change their password. Second, the compromise and exposure of a database that contains user passwords in reversible format, but where this compromise may not be known for months or years.

In the past, this conversation has sort of been about rotating passwords faster than most attackers can crack accounts, but I’d argue that’s less the case, and the real way it should be worded is to limit the window of opportunity for an actor to possess something that is still valid that they should not possess. Whether that means cracking times or exposure to an unknown compromise, I don’t really care.

Has password rotation ever “increased security?” I’d argue not really, but it helps deal with *decreased security* scenarios, namely someone has your password and you didn’t know it, or you did, and failed to change your password after the need to know it has passed. In the past, this also includes the scenario of password cracking. On the other hand, perhaps rotating passwords at varying levels has helped prevent situations where users use the same password for many accounts. Rotating passwords at varying times may decrease password re-use across accounts and actually be slightly better for security, but that’s just strange to think about.

Users create predictable variations from existing passwords, though! I suppose, if you know the base form of that password. For some, it’s easy to guess. New account in April? Try Spring18 or Spring2018. Then things get a bit more predictable, sure. For corporate accounts, at that point we’re already starting to get close to account lockouts or other alarms. For consumer sites, it’s harder to guess that base form, in my estimation. That said, I would say this argument has some weight. I’d bet some old passwords for users, if cracked, probably would inform their future choices (Bulls2015 may today be Bulls2016 or Bulls2017 or Bulls2018). But I suppose a cracked password that requires guesses to get the current password is better than a cracked password still being valid?

Users hate it and are inconvenienced! This is less a substantive argument and more a commentary that security and convenience are always at odds, and finding the sweet spot between them is a dance between objective and subjective discourse. Ask 20 infosec pros any scenario and you’ll get 22 different answers, all of which are varying shades of correct.

Why 90 days? Why not 30 minutes? I don’t know, I don’t think that’s the point? I think this is just acknowledging that security is not about finding *THE* right answer, but finding what works between the goals and people. Ok, fine, I think 30/60/90 days requirements back in 2003 were about cracking times for typical Windows account hashes. Roughly. Very roughly…

At any rate, all of this is pretty anecdotal, as is probably 99% of the discussion on this topic. Even places that try to say there is “lots of evidence” one way or another never really seems scientific or defensible from twisting the statistics around to form an opposing hypothesis or just have too small a sample. (Yeah, this is my way of dismissing stats and not wasting my time pouring over academic papers to support whatever my position is or will be. If paid to do that, I’ll be happy to!)

(Now, all of this said…I’m playing a little bit o’ Devil’s Advocate here. I won’t defend either position on password rotations terribly hard or to the death. But I think defaulting to password rotation in corporate account cases is the better approach and defaulting to unchanging passwords for consumer sites is also the better approach, with MFA swinging things away from rotations.)

* There is also the small slice of the puzzle with actual shared enterprise accounts or service accounts. The former being maybe a shared login to something low value (for instance, each concurrent user is a cost or something). The latter being your normal service accounts where various admins may be able to retrieve the password or set it up on a new server. I won’t deal with this, since it should be obvious these are rotated regularly, especially as employees leave the enterprise or lose their need-to-know.