It couldn’t be more timely to see a couple blog posts on the topic of threat hunting, one from Robert M. Lee and another by Richard Bejtlich. (Updated to also add some Twitter comments links, as I think I agree with this position.) Reason? The past couple weeks I’ve been reading papers and other posts and job descriptions of “threat hunting,” as I try to figure out what that means and what it does in a security organization.

See, I’ve been part of the infosec community since around 2001 in some for or other. But around 2014-2015 or so, I fell a bit out of touch; I didn’t read much from twitter or other blogs and feeds of mine and didn’t really do any cons or other learning. As such, I turn around in 2016 and plug back in, and things like “kill chain” and “threat hunting” take me a bit by surprise at how suddenly they’ve popped up. And in the case of the latter term, I’ve been trying to figure out where it came from and what it really means. I mean, I like the idea of the task, but it doesn’t seem like a full time position to me; it seems like an amalgamation of other duties, or maybe just a way to save money on external pen/red team tests by getting internal offense members, and concocting some additional things for them to do so you don’t lose them to boredom.

(Side note, it’s an interesting time, where organizations want to do more than just strictly blue things, but it’s hard to make sure your offense-minded folks don’t get bored or jump for those flashy full-time pentesting gigs. Likewise, blue tools and signatures are reaching their limits of usefulness, but other techniques and detection and analysis require more effort, intelligence, and experience to wield for proper value; enter offensive minds.)

My confusion stems from hearing about the tasks, and not being sure if the hunter is looking for latent, active compromises in dark corners of the enterprise, or if they are testing for weak points in an organization’s posture and providing fixes. Or maybe they are like architects for new tools like UBA/UEBA as they attempt to emulate attackers and define how detection tools may help identify them better, especially when you have to rely on behavior and anomalies rather than hard signatures or IoCs (hey, there’s another surprise term that is new!). To me, security is not just about watching alerts in a SOC, but about constantly improving the position of the security functions. In a sense, I’m always looking for ways to have more complete visibility, or at least know where my blind spots may be. Role-playing and tabletop exercises help stimulate that thinking as well. Things like analyzing a new vulnerability announced and how to tell if that affects or has already affected me, or a new minor incident and what pieces of information are annoying to procure. Every blue question has a chance to improve the environment.

Also, is a threat hunter part of the blue team, part of the IR team, or part of an internal red team? Or maybe some combination of two or more of those areas; a sort of way to fill in some time between other tasks. Is it a way to get your SOC members something a little more mentally stimulating than watching alerts all day? Does it complement or replace role-playing exercises?

Anyway, I don’t have these answers yet, but these were timely articles on a topic I’ve been currently wrestling with.

Maybe it’s because summer has given up the fight and it’s diving colder today for the weekend, but I feel ranty.

My other rant this morning is about security through obscurity. I hate seeing people say that this is bad. I mean, passwords fit into this category! The proper frame of mind is to say, “security through *only* security” is bad. I can move my SSH port to tcp 32154. Does that make SSH more secure? Not in itself. Does it make it harder to find and thus adjust my risk factor? Yes, somewhat. All those port 22 scans on the Internet will pass me up. Obscurity can certainly, and almost always is, part of one’s security posture.

Also, I hate when people say, “I could care less.” Well, that means you could in fact care less, which means you care. You mean to say, “I couldn’t care less.”

*curmudgeonly sounds*

A thread recently came up on TechExams.net forum about the order one got their certifications and which ones helped their career or were unnecessary. I typically don’t try to regurgitate my life story in random places, but I liked the question enough to ruminate on it a little bit. My certification path is a “stop and start” type and requires a few extra timeline points to explain some things.

~1998/1999 – Started blogging – Just a personal milestone for me.

2001 – MIS 4-year degree – A career milestone. I may have felt a little obligated to get this; I mean, after high school, you go to college, right?

2001 – Found a deep interest in information security – Several factors came together to this realization (writing a gaming site post about finding a career by using your PC gaming skills and getting into Linux distros, for instance), but the singular event that really informed me was picking up a random thick book from Barnes & Noble in my earliest efforts to keep learning after school: Hack Attacks Revealed by John Chirillo. The knowledge and attack/defense tools stoked a fire that will not be going away.

~2001/2002 – Started blogging about tech/security & first vanity domain – At some point, I started blogging regularly about tech stuff and security topics, mostly just interesting links and tools. Many of these posts are either lost or buried in another data file backup somewhere along with personal blog postings. After leaving school and leaving their nice hosting, I also picked up my first vanity domain.

2002 – First “real” job – Basically, the real start of my career!

2002-2006 – Lull #1 – My 4 years at this job marked two things. First, I learned an absolute ton and had an absolute blast doing it. I grew by leaps and bounds during this period. Which probably is the reason this period is also marked by no formal learning of certification. I was underpaid by a company that also wouldn’t pay for training, but I didn’t much mind it since I was learning so much on the job. So, this was my first lull in learning, but it didn’t really feel like it. I have a strong nostalgia factor from these years of my career, work- and enthusiasm-wise.

2006 – Security+ – A job change later and I wanted to demonstrate my interest in security better. Before LinkedIn, you really only had in-person networking and your resume to demonstrate security acumen. If your job title was generic, but you managed security devices, it was difficult to show it. Also, I wanted to learn more. Security+ worked great for this. Spent personal time studying books and passed the exam. At the time, this was also a lifetime cert, which was a bonus I wouldn’t understand at the time.

2006 – terminal23.net domain – At this point, my technical blogging eclipsed my personal stuff in both effort and frequency, so I separated them with a second vanity domain. I took some effort to pull old technical posts into this blog, but some much older stuff I wouldn’t bother with.

2009 – CISSP – I also pursued this one on my personal time using self-study books. I would happily pass on my first attempt. Even at this time, there were threads of CISSPs being derided for not actually knowing anything (one guy at my testing center that I talked to was a sales guy on his third attempt, because it was required for his sales position…), but there was and probably still is no better certification to demonstrate interest in and at least some wisdom about security. In fact, this cert probably opened the most doors and got me the most recruiter attention of anything else I’ve picked up, by far. It’s definitely a gateway cert, and I think everyone in security should at least have this on their roadmap. Sure, you can skip it if you have good demonstrable security work and/or good networking, but for most of us, this makes a statement itself. Even now, almost 10 years later, I’m not sure when I will burn it or let it lapse… On the down side, I didn’t get a raise for this, paid for it myself, and didn’t use it to springboard into another job. Maybe a wasted opportunity for me, but I like where I am today for it.

2009-2017 – Lull #2 – During this period, I grew quite a bit with my skills during work hours, but for the most part, I did not pursue any formal education. I signed up for the PWK/OSCP (PWB at the time) cert, but work threw me, well…work, and I didn’t have the time to devote to it, so I let it slide. It didn’t help that my company did not really budget for training nor encourage it; in fact, I had a manager whose teams always seemed to stagnate and work behind the times with old tech/code/habits. I wouldn’t say I coasted during this period, but I was very comfortable and my days were filled with work at a manageable pace.

2017 – OSCP – Finally started getting that bug to get better jobs and re-find my enthusiasm and learning passion that I had in my first “lull” and early years. I decided to pursue the OSCP again on my own time and dime, and achieved it after about 4 grueling months. Of all of my certs so far, this one gave me the most street cred, and for hiring managers who know it, it definitely gets their attention. Particularly the 24-hour exam.

2017 – OSWP – I knew I wanted to keep learning, and I remember the hey-days of war-driving and backtrack wireless cracking, so I wanted to revisit those activities with what I knew was a much lighter cert in the OSWP. Took about 2-3 fairly casual weeks from start to finish. Really enjoyed it, and left me hungry for more.

2018 – CCNA Cyber Ops – I don’t remember how I learned about this, but Cisco basically gave out free training and certification exams for lots of people who already had various industry certs, so I got this certification for free, though I did have to devote plenty of personal time to get it. This didn’t improve my resume at all, but I did like the experience. And I have to be honest, while I kept up with security blogging over many years, from about 2015-2016 I got a little out of touch with the security industry. And taking the Cyber Ops course filled in some gaps of new ideas and things like “threat hunting” and the “cyber kill chain” and “diamond models” which had been basically introduced at the time. Ultimately, this course pursuit got me back up to speed of the buzzwords of a SOC. Unless Cisco builds something compelling around it, I don’t plan to renew this one.

2018 – GCFA – For me, this is the first time in my career I’ve had corporate backing for education, and also marks a culmination of the next part of my career where I have strong, specific goals for growth. Also, a point where I stepped just slightly outside my comfort zone to formally learn something new that I identified as a weak spot.

Footnote
One thing I will notice in my timeline compared to some other postings in that thread is how some people earn certs and are immediately rewarded with it leading to a new job or a raise of some sort. But, my timeline has almost none of that; many of my certs were earned and I would not say they directly led to a future job. Maybe a few interviews, but certainly not in the same calendar year as the cert was earned. I’ve also had the luxury of not having jobs that required extra study.

I also noticed that I never had (until this year) company or managerial backing for growth like this, and I also never had peers or colleagues who pursued certs or further formal education. That certainly makes a difference, as I do become influence by those around me, as most do. I had to find the effort to self-start, most of the time.

There’s really no way to say it without sounding conceited, but all of my certs came from my own motivation and my own desire to learn and/or demonstrate knowledge in security. That’s not any less or more than other reasons to get certs, but I found that enlightening for me. It also helps illustrate what makes me happy, what drives my passion for this industry, and informs my plans for the future.

I find it crazy that I’ve not seen this before, but I got linked today over to the MITRE Ten Strategies of a World-Class Cybersecurity Operations Center free book (pdf). Holy crap this is awesome. The rather large first section talks about building a SOC and the various considerations that go into it. And then the top 10 strategies build on that foundation to further guide the growth of the SOC.

Every section has wonderful nuggets of truth like this one in strategy #5 (Favor staff qualify over quantity):

Analysts must be free to analyze. It is indeed true that Tier 1 analysts have more structure in their daily routine for how they find and escalate potential intrusions. However, those in upper tiers must spend a lot of their time finding activities that just “don’t look right” and figuring out what they really are and what to do about them. Overburdening analysts with process and procedure will extinguish their ability to identify and evaluate the most damaging intrusions.

Honestly, this might be my second favorite technical book, up there with The Practice of System and Network Administration (Limoncelli).

A week ago I flew down to Orlando, Florida to attend Splunk .conf18. In thinking back on this, I have to say this is the very first vendor-specific conference I think I’ve ever attended in my 15 years in IT. Based on who you ask, the con itself had 7500-9500 attendees in its largest event to date. That’s pretty impressive! I attended as many talks as I could, and I left pretty happy with the content I consumed. The talks and slides are all available online for consumption.

Day 0 – Sunday
My goals for this day were just to get to Orlando and settled into the hotel and do some recon of the grounds and environment. On the plane, listened to some Darknet Diaries; finally finding some time to do some podcasts! Took some time to hit the Boardwalk on the ground and already get sick of the heat and humidity.

Day 00 – Monday
Goal today was to get registered for the con! The line was super quick, even at 10:30am with the masses to get checked in, get a badge, pick up the backpack/water bottle freebie, and then pick up the freebie hoodie. Beyond that, this day was pretty casual until the evening.

First Timer Orientation talk – This was a nice intro to the con, even though the room was moved and I didn’t hear about it until a co-worker texted me. I guess I need to click update notices in the event app! (Come on, I’m in security, I don’t click accept/download buttons unless I have to.) Also, this was the only talk that I attended with a drink-in-hand speaker. (I’m not a huge drinker or want others to drink, but to me, this still sets a tone and statement for the sort of partially or fully informal a venue may be. This is why I like smaller cons over larger vendor ones.)

Welcome Soiree – This was a neat way to get people to the vendor floor: an evening event with free food and alcohol stations throughout the vendor floor. Scoped out vendors, splunk experts, projects, and plenty of swag. And I will admit, I evaluate vendor booths on three things: 1) whether I know and like them as a product/company and want to say hi, 2) whether I want some of their swag or not (either for me or to give away to others), and 3) whether I want to buy them (and I’m not a purchasing approver, so that’s pretty much no one). I had fun down here, though someone kept turning on music every now and then and it was ridiculously loud.

Day 1 – Tuesday
Visionary and Roadmap Keynote + Breakfast – For the morning keynotes, buses took us to ESPN Arena where we picked up breakfast bags before taking seats. After the talk, I don’t think the bus crews were ready for the flood of people, and organization broke down pretty hard on one side of the venue, but we all got back in decent time (albeit later than intended due to the overlong keynote).
Security Super Session: Splunk Security Vision and Roadmap

Security Super Session: Splunk Security Vision and Roadmap – A strong, high-level look at Splunk and using it for security operations. Not much to say on this one. The diagrams are wonderful (and would be used in several talks I’d see over the course of the con) for designing your security operations around.

Find and Seek – Real-time Asset Discovery and Identity Attribution Using Splunk – I didn’t actually see this talk. Tuesday was the one day where I was all over the grounds for various talks, and required buses to get me places in time, and the buses were still a little chaotic. I was on time getting to this talk, but after about 15 minutes after the start time, we were all still waiting outside the room. Thankfully, it was right next to a sandwich distribution station, so I just left with my lunch to eat elsewhere. I’ll have to catch this recording later.

Let’s Get Hands-On with Splunk Enterprise Security, Splunk Phantom, and Real Boss of the SOC Data – This was the one “laptop required” talk I attended, and honestly one could have been just fine sitting back and watching along. This session had several hundred people in it, and as such you have to expect them to move on and not wait for anyone, and move on they did! Thankfully, this is the introduction talk for a broader and slower workshop for security people to get from Splunk throughout 2019. As it was, I really enjoyed getting hands-on a bit with some practice data for finding attacks. The data itself was used in the BOTS competition the previous evening. While I’m new with Splunk, it’s these hands-on demos and doing actual things with the data that get me excited, rather than high-level, perfect-situation statements.

Threat Hunting and Anomaly Detection with Splunk UBA – I really liked this talk and speaker. While nothing about Splunk and anomalies and hunting were new to me, I really loved the best/worst practices examples. That’s the sort of detailed, technical stuff that I eat up, rather than non-filling high-level statements.

Pub Crawl – Similar to the soiree from the previous night, only with craft beer stations and less food overall. Other than the alcohol and snacks, I didn’t really need a second round through the vendor hall.

House of Blues – We also got invites to a party at the House of Blues. The music was just passable, but it was an excellent buffet, and I got a chance to sample the infamous Voodoo Shrimp (which was basically forgettable, to me). The best part was just getting another evening without a food bill!

Day 2 – Wednesday
Product and Technology Keynote – I’m not a huge breakfast person, and I found out you can watch the keynotes online, so I didn’t even bother heading out to see this one live. I opted to stay near the hotels and not fight lines for a latte.

Hacking Your SOEL: SOC Automation and Orchestration – I love technical talks, less so high level ones. But if there is one talk that I’d recommend that is high level about SOEL, and SOAR, and SOC automation, I’d point people to this one. The speaker just plain made sense of all of this. Sure, it was high level, but also detailed enough to formulate a roadmap for the future on the topic. One of the more solid talks I attended.

Attack Surface Reduction: Using Splunk to Spot the Security Flaws in your Network – The description for this was probably reflective of a longer talk that got cut down. This talk ended up being basically a firewall review 101 session, but using Splunk to view your logs for activity on firewall rules under review. I did learn just one thing from this: monitor for sessions that hang, i.e. no endpoint listens on the target port anymore. I probably would have done that, but I think it’s important to keep that situation in mind. The rest was really pretty newbie material.

Which brings me to one of my main challenges: Finding the right level of talk for the topic. For instance, I’m a newbie with Splunk, but security concepts I’m very deep with, both defense and offense. I would love to have known this talk would be at a newer level of security, as I would have avoided it. This would apply to some of the threat hunting and SOC automation talks, which sometimes felt like they were just saying the same high level things over and over without a ton of deeper substance (i.e. for people less senior than I). This might not be a con issue, as it might just be my inaccuracy with using the con properly, i.e. less talks, more 1-on-1 and breakout discussions.

Cops and Robbers: Simulating the Adversary to Test your Splunk Security Analytics – Came into this very interested, but also skeptical on why the heck I’d want to spend time automating attacks like I’m some QA team. But this talk made a great case for why you do this, and how you approach it, particularly with Phantom and some other tools. Looks very cool for use on an internal testing team that evaluates not only internal response and controls, but also can test security products and even do some training exercises with your Splunk teams.

WMI – The Hacker’s Chocolate to their Powershell Peanut Butter – Probably the deepest technical talk I saw at the con dealing with attackers using WMI, WinRM, and Powershell in modern attacks, often going fileless, and how you could use Splunk and general logging to hunt these compromises down. I really enjoyed it, and was a great reflection on the Splunk security research arm.

Monitoring and Mitigating Insider Threat Risk with Splunk Enterprise and Splunk UBA – As a Splunk newbie, I wanted a mix of talks on some of their products and how I can wrap my security team around them and my own priorities and goals. This was a good talk about implementing insider threat detection using Splunk UBA. I’ll likely revisit this again as we start our own projects on this in the coming quarters.

Search Party! At Universal Islands of Adventure – Such an absolutely fun time having the park to ourselves to avoid lines and endless children in order to ride Hogwarts Express, Harry Potter’s Forbidden Journey, and the Jurassic Park river ride. The Express was super fun, and Forbidden Journey ride absolutely awesome, and the Jurassic Park ride a fun mess that stopped 3 times and ended up taking about 30 minutes to get through. The walk around the park was fun, though the back half through Marvel and the Comic Book zones were plenty unexciting compared to the other areas. Really wish we had more than 2-3 hours, but fun and free nonetheless!

Day 3 – Thursday
Guest Keynote: Steve Wozniak – I don’t really have a huge desire to listen to Woz; smart dude with lots of money and the ability to opine about technology. Fine. To make sure people made it to this talk, it was not broadcast like the other keynotes, so I just opted to skip.

Overall on this day, the food stations and snacks were far skimpier on this day. I still never had to visit the main food tents, but I definitely had to look for food myself otherwise.

“MAKE IT RAIN!” How to Save Money Monitoring, Managing, and Securing Your Cloud Using the Splunk App for AWS – By now, I know that I should expect high-level statements when I see CEO, CTO, or other high-level manager titles in the speaker list for a talk. And then a talk like this comes around to prove me wrong. (I’ve honed my stance on this to apply only to Splunk as a company itself when its higher-level managers speak.) This talk was an actionable demonstration of tying some important AWS logs into Splunk and showing how that is valuable for operations and even security. A slightly short talk, but really nice to sit through as someone new to Splunk, new to AWS, and subsequently new to doing them at one time.

From Threat Modeling to Automated Response – Identifying the Adversary and Dynamically Moving to Incident Response – Yet another talk about threat hunting and TTPs and adversary profiling. A good talk, but I don’t think it included anything that I didn’t already know.

If there’s anything in my year that will define it, it’ll be the prevalence of Kill Chains, Threat Profiling, and Threat Hunting. I can’t escape the same ol’ statements about them. I had it throughout the Cisco CCNA Cyber Ops course, the SANS FOR508 course, multiple talks at Splunk .conf, and beyond. I’ve long had a post waiting about how and why threat hunting is such a deal these days (it comes down to getting internal value and blending offense into the internal blue teams, plus trying to make sense of the new breeds of security tools that don’t just alarm on bad, but require human decision-making to piece together multiple things…).

Blueprints for Actionable Alerts – This apparently is a version of a talk done for several years, and it kinda feels like it. For some strange reason, I didn’t get much out of this, though on the surface I should have. It’s really a discussion in figuring out how to tackle an environment with 4000 alerts in a day, and reducing that piece by piece to be manageable and useful. I think everyone sort of does this their own way, which all sort of dance around the same gameplan.

Splunk P30X: Become a Lean, Mean, Splunkin’ Machine in 30 Days – Probably the best and most useful talk I attended at the con. The point is to have an actionable, lunch-hour plan to tackle and do various Splunk activities to culminate in being able to pass the Fundamentals exam at the end. I loved the actionable approach to this, as well as the follow-up activities the authors are releasing to support it that I can directly consume. Not only the 30-day plan, but also additional materials for newbies. Wonderful talk!

Day 4 – Friday
Nothing much exciting here, just a full day of getting back home.

Overall Thoughts
I loved the overall experience and benefits to going; it was fun, got to visit a fun park, and so on. This could double as a family vacation if you brings the family along. Next year, the con is in Vegas, and I’ll admit that has less appeal to me as a venue/area.

If I go again, and have others with me, I’ll lobby somewhat hard to get signed up for one of the competitions they hold, either Boss the of NOC or Boss of the SOC, where teams pour over and parse out data to answer questions about operations or security incidents, respectively.

I don’t often find fairly general articles to have enough interesting nuggets and quotes to bother saving, but sometimes they just flow so well and include plenty of head-nodding things to agree with, all with wording that I appreciate. One such article came across from Dark Reading, Think Like an Attacker, How a Red Team Operates. Dark Reading seems to like limiting the ability to read articles, so I don’t mind being a bit liberable in pulling out quotes I like.

“The whole idea is, the red team is designed to make the blue team better,” explains John Sawyer, associate director of services and red team leader at IOActive. It’s the devil’s advocate within an organization; the group responsible for finding gaps the business may not notice. I just love that sound byte. I want that to be my elevator job description.

“The main function of red teaming is adversary simulation,” says Schwartz. “You are simulating, as realistically as possible, a dedicated adversary that would be trying to accomplish some goal. It’s always going to be unique to the target. If you’re going to get the maximum value out of having a red teaming function, you probably want to go for maximum impact.” The early part of the article goes a great job of succinctly comparing pen testing and red teaming while also illustrating how these have changed as time has moved on. Old school pen testing has shifted to be called red teaming as a way to further differentiate as pen testing has become commoditized.

The team ends up chaining together a small series of attacks – low-level vulnerabilities, misconfigurations – and use those to own the entire domain without the business knowing they were there, he says. Typically, few employees know when a red team is live.

Red and blue teams may work together in some engagements to provide visibility into the red team’s actions. For example, if the red team launches a phishing attack, the blue team could view whether someone opened a malicious attachment, and whether it was blocked. After a test, the two can discuss which actions led to which consequences. Beyond actually enjoying it, this is my whole value proposition for my interest in offense and red teams: It makes my defense better. Which makes me get better on offense. Which makes my defense get even better… Getting a root shell or DA credential is the addiction, the satisfaction is passing on the information to make improvements.

More and more companies are starting to realize if they limit themselves to the core fundamentals of security, they’re waiting for something bad to happen in order to know whether their steps are effective, says Schwartz. Red teaming can help them get ahead of that… Many companies are building red teams in-house to improve security; some hire outside help.

The main reason behind building a red team internally is because as it grows and improves along with defenses. As security improves, so do the skills of red teamers. Offensive experts and defenders can attack one another, playing a cat-and-mouse game that improves enterprise security, he continues. Internal teams are also easier to justify from a privacy perspective.

Overall, the pros argue a full red team can help prepare for modern attackers who will scour your business for vulnerabilities and exploit them – but they’ll help you stop real adversaries.

“The difference between a red team and an adversary is, the red team tells you what they did after they did it,” Schwartz says.

That’s such a strong ending to this article, that I had to pull a bunch out right there. Wonderful!

Part of the reason it took so long to take my GCFA exam was the splitting of my study time with Hack The Box progress. Earlier this year I bought into VIP access to HTB, and I wanted to keep practiced up with, and learning new, offensive skills. I did more than I was expecting, and after making some friends smarter than I, was actually able to far surpass my goals and expectations to achieve 100% completion and top Omniscient ranking sometime in mid August. I still have to go back and properly learn some of the things I found way too difficult to do alone (let’s face it, the best people are the best due to learning through teams, and the best red teams have multiple people): namely binary exploitation and reversing. I get how they work, but I need my hand held way too hard right now! I still also have the “optional” sections to complete (Fortress and Endgame), and I’d like to dive into RastaLabs and Offshore, probably in 2019.

Really, my main goal was to keep the skills and processes I developed in the PWK fresh, while also learning new and more advanced tricks and tools and techniques. And I feel like I’ve succeeded in that aspect. These skills get dusty and rusty if not practiced regularly, either on one’s own or while in the course of work duties.