Lenny Zeltser goes over 8 tips for detecting website compromise. There are far too few security writers who have technical chops enough to not stand out amongst a pack of geeks at a security conference, but Lenny is one of the better ones, so any criticism I have for his writing is done very lovingly. Snuggly-like, even. I figured I could make mention of this nice article and give my own reactions.

Lenny starts out with 3 overly obtuse best practices. Sort of like saying, “If you want to get into shape, you should run more,” which *sounds* easy. I think he takes these items like I do: you’d be remiss if you left them out, but can’t do them justice by being less obtuse about it.

1. Deploy a host-level intrusion detection and/or a file integrity monitoring utility on the servers. – This biggest problem with this (collective) item is how much activity this is going to generate for an analyst. And if you do tune it down to a quiet level, I’d argue that you’re not going to see what you want. But, as said already, a necessary evil to a security posture, whether you like it or not. At a bare minimum, you should know every time a file is changed and a new file appears in your web root (with exceptions for bad apps that need to write temp files and other crap to itself- the bane of web root FIM…).

2. Pay attention to network traffic anomalies in activities originating from the Internet as well as in Internet-bound connections. – This part, “Internet-bound connections,” really needs to be implemented as soon as possible before an organization has so much going on that you can’t ever close it down without preparing everyone for the inevitable breaking of things no one knew were needed. Watching traffic coming in? Well, not so easy, and you’ll probably just end up looking for stupidly large amounts of traffic (which may be normal if you service a large client sitting all behind 2 proxy IPs) or the most obvious Slapper/Slammer IDS alerts. But you absolutely want to know that you just had 500 MB of traffic exfiltrate from your web/database server to some destination in the Ukraine, and it contained a tripwire entry in an otherwise unused database table. I would drop the buzzword, “app firewall,” in this item as well. You should also know what is normal coming out of your web farm (DMZ), and anything hitting strange deny rules on the network should be checked into.

3. Centrally collect and examine security logs from systems, network devices and applications. – Collect: Yes. Examine: Yes, with managed expectations. I really want to say yes, but having done some of this, 99% of the examined stuff that bubbles up to the top is still not valuable. There’s a reason I think gut feelings from admins catch lots of incidents and strangeness on a system/network, and it’s not because it shows up clearly in logs. Especially with application logs, if you want them to be trim and tidy, you’re looking at a custom solution which includes custom manhours, overhead, and future support resources.

Side note on custom app logs: If you can, log/alert any time a security mechanism routine gets used, for instance if someone attempts to put a ‘ into a search field (WAF).

4. Use local tools to scan web server’s contents for risky contents. – Certainly should do this as much as you can. You can scan for even deeper things like any new file at all (depending on the level of activity normally on your web root) or files created by/owned by processes you don’t expect to be writing files.

5. Pay attention to the web server’s configuration to identify directives that adversely affect the site’s visitors. – This can be a bit easier in Apache or even IIS 7.0+, but some web servers like IIS 6 are hideous for watching configurations. Yet, definitely keep this in mind. Thankfully, attackers have extra hurdles than just writing a file in the web root, when it comes to such servers.

6. Use remote scanners to identify the presence of malicious code on your website. – I agree with the spirit of this, but I think internal scanners need to be done as well. If a bad file is present but not discoverable from an existing link on your site or easily-guessed name, then an external scan will totally miss it.

I should take a moment to stress that many of these items include signatures or looking for known badness, but none of that should replace actually looking, at some point, at every page you have for things that are clearly bad to a human, such as a defacement calling card or something.

7. Keep an eye on blacklists of known malicious or compromised hosts in case your website appears there. – It’s hard to say this shouldn’t be done, but it certainly offers little return on the investment of time. If you *do* happen to show up before any other alarms sound, then clearly it is nice.

8. Pay attention to reports submitted to you by your users and visitors. – I’d personally overlook this item 9 times out of 10, but it is really oh so necessary.

If I wanted to add an item or two:

9. Scorched earth. – While not a detection mechanism, you do run certain risks if your web code sits out on the Internet for a long period of time and grows stale. You should refresh from a known good source as often as you think you need to. This can include server configs and the like. (For instance, I roll out web root files every x minutes on my web servers at work, and I regularly wipe out web server configs and rebuild them via automation.) Don’t be that company that had a backdoor gaping open to the world for 2 years, when a simple code refresh would have closed the hole. A diff comparison mechanism may satisfy the ‘detection’ criteria of this list.

Swa Frantzen (ISC) has a great discussion of recent DigiNotar drama going on. I do take minor exception to this statement:

I for one would love to know who that external auditor was that missed defaced pages on a CA’s portal, that missed at least one issued fraudulent certificate to an entity that’s not a customer, and what other CAs and/or RAs they audit as those would all loose my trust to some varying degree. This is not intended to publicly humiliate the auditor, but much more a matter of getting confidence back into the system. So a compromise that an unnamed auditor working for well known audit company X is now not an auditor anymore due to this incident is maybe a good start.

I totally understand this sentiment, and actually do agree with it. But we do have to be careful that we don’t set every single security auditor/expert up for failure, where one mistake causes the hammer to drop. (Speaking of elephants in rooms, the seeking or assumption of perfection is a ‘subtle’ one…)

Granted, repeatedly missing defaced pages hits the facepalm category, but I think this oversight (from tripwires on attacks to page inventory reviews to edit/ownership times to web app sec checks, etc) can happen to literally every organization if they’re not rigorous in their testing, though it still comes down to knowing what is valuable in the eyes of a threat, and being extra careful around those processes (i.e. issuing a trusted certificate!).

Sitting back and pondering this scenario while nursing some scotch illustrates all sorts of things that are wrong with the security mindset in our world, ya know? Maybe “wrong” is a bad word for it, but rather the challenges we face and will eternally face, as a function of reality.

McKeay wrote a great post about blogging this past weekend, and I think any security blogger should check it out. I really like his subpoints about blogging and working and balancing both:

I’ve learned a number of lessons about blogging the hard way. I’ve learned that no matter what I think I’m writing, what’s important is how other people are reading it… I’ve realized that people are reading and judging what I write, for good and for ill. And when I write something people read, it can get back to my employer.

I also like this:

More often than not, my employers have maintained an air of benevolent ignorance towards my blog, but every so often I’ve gotten the “we’ve read your blog and are not happy” conversation. Not often, but it has happened and it’s never comfortable talk. I’ve actually told at least one manager that my blog and podcast are more important to me than my job.

For me certainly, blogging is a personal thing, a way to organize my own thoughts, record something for the future, or vent a little bit. It’s also a way to dive deeper into what would always be a hobby for me, even if not a job. Even if I didn’t have a single reader my blogging habits wouldn’t change a bit.

Anyway, here are some points of my own that I try to follow.

1. Separate work from personal if you need to. This is a big deal in the past 5 years, where work and play time are blending together, largely because something you “say” (digitally) during your personal time can now easily persist for years for people at work to discover. Things you could say with buddies or at a bar don’t just stay with buddies or at the bar in a single point in time. Therefore, with blogging especially, I try to keep work separate. I don’t hide my identity on here, but likewise I don’t advertise my blog to work colleagues (they can easily find it on their own if they want) and I don’t mention my employer anywhere on here. I also leave deep personal things aside, though some incidents/anecdotes if read by the right people, would recognize themselves in them, but I also try to make sure they’re generic enough and have enough of a point to not be uncomfortable. Besides, if I piss someone off, I hope they have my own viewpoint and just move on with life. It’s a big deal to be able to agree to disagree; a very useful skill. I like dark grey cars and don’t like white cars. You might not agree. And it would be silly to get pissed about that. Same goes for what I post on my blog or elsewhere on the Internet with my screenname.

I admit, my hard divide between work and personal is slowly going away, in part to my next point, but also partly because security work and play is a career goal.

2. Don’t present false faces. I don’t like when people “front,” or present themselves in a way that isn’t in line with who they really are. Life is way too short and precious to not be yourself in anything you do, and if being yourself gets in the way, make changes to be someone better. In that regard, I don’t typically pick my words carefully on my blog; if I have an opinion, I’ll be out with it. (Though it does help that I’m an easy-going kind of guy anyway…and this is also easy to say for someone who thinks of himself as a very decent guy who is sympathetic to objectivist beliefs…)

3. It’s easy to apologize or admit to being wrong. I don’t mean this to sound like a copout for bad behavior, but it is easy to apologize for or admit to being wrong. I find it’s more important to put your opinions out there and be contritely wrong, than to bottle everything up and stew. And this is a tough thing for an INFP to say! (And it’s something my risk-averse nature will always fight with me about.) Granted, that doesn’t mean you can be an asshole and then be contrite about it and things are fine…be reasonable!

4. Remember the important things in security: integrity and privacy. This also applies to IT work in general. Typically we are in positions to know very deep secrets and have access (or get access) to very sensitive things. The same principles that prevent me from perusing my CEOs mailbox are the same that dictate what I divulge on a blog, or anywhere on the Internet. Hopefully most people in white hat security are at least aware of these principles in every facet of their lives.