Years ago, there weren’t really good listings of various vulnerable virtual machines, vulnerable web apps, shady packet captures, or other challenges that you could learn from, so I kept my own list here on my site. These days, however, Vulnhub.com offers a far better listing than I could keep. So I might reorganize my “learn” section of links, but for now, these will stick around and I’ll refer to Vulnhub for my damn vulnerable VM itch.

I wrote this post as a draft several years ago. I don’t plan to finish it, and thought I’d just publish it out now. This is not really complete, but I did fill in a few gaps. Also taking the time to hate on formatting things in WordPress…nested bullets are making me swear more…

I’ve spent some time tuning a SIEM as part of my day job. I thought I’d share the general steps to do this. Sadly, I can’t be too technical without becoming specific to the tool(s) I use, but I can at least outline the general concepts.

1. Gather logs from your devices/apps into your SIEM.
   This is typically an easy process, and every SIEM vendor will tout how easy it is. Don’t be fooled, this is easy by design, and not because their tool is particularly easy. It’s easy to suck in Windows logs, Linux logs, and even easier to catch the flood of syslogs you can send to the SIEM. Vendors will always tout how you can be up and running (often they will say, “gathering logs” without letting that sink in that a SIEM does more than just “gather logs”) in 10 minutes, a half hour, an hour, or a few hours. The hardest part is just finding all the logs you want to send to the SIEM.Assume you’ll be adding new logs as you go for devices you either overlooked early on, forgot about, or didn’t even know you had (high-availability pairs, dev environments…). And make sure to devise a plan to ensure that new devices get discovered or configured per a build process to get into your SIEM.
2. Make sure your SIEM understands and adequately parses the common logs you send it. Just because a SIEM can suck in the logs and archive them away, doesn’t mean the SIEM is configured to parse out the contents and make decisions on them. Most common devices and sources are fine, but some of your more uncommon devices aren’t going to be set up without special effort. You can get assistance from the vendor, but often they will tell you, “You tell me what you want, and I’ll set it up for you.” So…make sure you know what you want! Your time is usually better spent looking at the logs you get, demonstrating the activity you want to alarm on, catch the log, figure out what is unique about it, and then making your own alarm based on that uniqueness. Repeat ad nauseam.It should also be mentioned that you need a process in place to make sure your SIEM is still working properly and that logs are still showing up. It’s fine if you event on incoming logs that indicate badness, but are you looking for a total absence of logs as well? Hopefully your SIEM can raise these issues to you in more than a dashboard you have to eyeball manually.
3. Tune out the noise as much as you can and tune in to what you really want to know about.This part really forces you to think a little bit like an attacker and what she will do against your devices and what it will look like in your log files for various devices. This often includes two different levels of monitoring (maybe more if you decide that you get operational value out of the SIEM, rather than just security value):

   • events of interest in the logs
   • actual generated alerts/alarms that are sent to someone or raised for someone to look at

   This is also a very subjective piece of the puzzle, and somewhat artful as well, as you creatively ignore what you don’t care about, without accidentally ignoring too much. Probably will help to talk to your SIEM vendor on any tips they have to avoid performance problems with filters or log rules. Maybe their device will bog down when you give it too many filters, or maybe it will bog down when you don’t give it enough! For instance, if you log your VMWare ESX server syslogs, you’ll find that you literally do not need 99% of what you get sent.

   For every event you generate, ask yourself:

   • does this give me value towards alerting/detecting an incident in progress?
   • does this give me value towards guessing that there might be something strange happening?
   • does this give me value in finding more about events around an alert or incident?

   If an event is coming in all the time, and you know you’ll never consider it an event of interest, try to configure a filter for it. But be careful!!! For every filter you make, you create a blind spot. For instance, I have an account that leverages CatTools to log into various network devices and back up the config files, and this runs multiple times daily. I don’t ever want to know about these valid logins, so I ignore such log entries. But this means if an attacker gets into that account, she can use that account to get into the same network devices and I’ll be blind to those log entries. A better method might be to put a threshold on the frequency of logins per hour. Hopefully, though, any further manual things she does with that account on the devices will get flagged, since I really want to have an event for every actual command run on some devices (we’re not THAT big that I can’t just turn around and ask someone if they just did something…).

4. Close the gaps in your logs by getting those non-standard and custom logs understood by the SIEM. This is difficult to do, and probably dives deeply into one of two options:

   • Send logs to your vendor and wait for them to properly figure out the parsing.
   • Dive deep into the guts of the tool’s rules, and figure out the parsing on your own.

   The first option leaves you at the mercy of someone else, but they will certainly be better at it than you alone. The second option is a deeply technical and possibly very difficult task that will takes lots of trial and error and time. The reward is great, however, if you can master your chosen tool’s rules and how to cover stuff that isn’t covered by default.

5. Manage everything that is left in the alarms/alerts or events.If you have a SIEM in place and it is chomping on logs and generating alarms, but no one is watching it, you’ve effectively just made a complicated log management process that is only useful for forensic purposes, and does *not* raise awareness of security events or satisfy any goals for detection and response. If an alert comes in and it is an attacker, but no one looks at the alarms, then what was the point? I understand checklist security, but that doesn’t mean I will be satisfied by it.

   At a minimum, you should see positive, but non-incident hits from:

   • patches, assuming you also have a file integrity monitor in place, but at the very least a reboot should be noticed or maybe events for installed software
   • vulnerability scans should light up your SIEM (similarly, any pen-tests or other security reviews)
   • failed logins on various devices. No matter how much you tune, you can’t tune out every accidental series of failed logins and yet still be alerted when someone malicious is doing it. Sometimes it just takes one strange attempt to log into a network device as “root” that is important to see, yet you clearly don’t want to be told every time Sales Guy Bob mistypes his password three times in a row. You want to know when an account is locked out, but can’t tell from a SIEM whether that was again Sales Guy Bob doing it.
   • essentially, anything with a threshold value will be tripped someday. If you only care when a single device pings out to 50 different IP addresses, someday something will do that.
   • if you want to know that someone got into your border router and made config changes, you’ll also pick up authorized changes from your own admins.
   • user rights changes and account management are an every-day occurrence in a business, but it is common for an attacker to try to elevate their rights either on a local box or the domain. You should see hits for both. This includes new user creation.

6. Lastly, test your alerts to make sure you’re getting what you need, and learn from the computer forensics disciplines.Generate some events that you would want to make sure you pick up if an attacker either attacked your systems or actually got in. Someone added an account to your Domain Admins group in Windows Directory? You better be watching that. Verify that you’ll see what you want to see. Be creative, and be informed about the forensics artifacts left behind in the logs that you can and should alert on in your SIEM.