Lenny Zeltser goes over 8 tips for detecting website compromise. There are far too few security writers who have technical chops enough to not stand out amongst a pack of geeks at a security conference, but Lenny is one of the better ones, so any criticism I have for his writing is done very lovingly. Snuggly-like, even. I figured I could make mention of this nice article and give my own reactions.

Lenny starts out with 3 overly obtuse best practices. Sort of like saying, “If you want to get into shape, you should run more,” which *sounds* easy. I think he takes these items like I do: you’d be remiss if you left them out, but can’t do them justice by being less obtuse about it.

1. Deploy a host-level intrusion detection and/or a file integrity monitoring utility on the servers. – This biggest problem with this (collective) item is how much activity this is going to generate for an analyst. And if you do tune it down to a quiet level, I’d argue that you’re not going to see what you want. But, as said already, a necessary evil to a security posture, whether you like it or not. At a bare minimum, you should know every time a file is changed and a new file appears in your web root (with exceptions for bad apps that need to write temp files and other crap to itself- the bane of web root FIM…).

2. Pay attention to network traffic anomalies in activities originating from the Internet as well as in Internet-bound connections. – This part, “Internet-bound connections,” really needs to be implemented as soon as possible before an organization has so much going on that you can’t ever close it down without preparing everyone for the inevitable breaking of things no one knew were needed. Watching traffic coming in? Well, not so easy, and you’ll probably just end up looking for stupidly large amounts of traffic (which may be normal if you service a large client sitting all behind 2 proxy IPs) or the most obvious Slapper/Slammer IDS alerts. But you absolutely want to know that you just had 500 MB of traffic exfiltrate from your web/database server to some destination in the Ukraine, and it contained a tripwire entry in an otherwise unused database table. I would drop the buzzword, “app firewall,” in this item as well. You should also know what is normal coming out of your web farm (DMZ), and anything hitting strange deny rules on the network should be checked into.

3. Centrally collect and examine security logs from systems, network devices and applications. – Collect: Yes. Examine: Yes, with managed expectations. I really want to say yes, but having done some of this, 99% of the examined stuff that bubbles up to the top is still not valuable. There’s a reason I think gut feelings from admins catch lots of incidents and strangeness on a system/network, and it’s not because it shows up clearly in logs. Especially with application logs, if you want them to be trim and tidy, you’re looking at a custom solution which includes custom manhours, overhead, and future support resources.

Side note on custom app logs: If you can, log/alert any time a security mechanism routine gets used, for instance if someone attempts to put a ‘ into a search field (WAF).

4. Use local tools to scan web server’s contents for risky contents. – Certainly should do this as much as you can. You can scan for even deeper things like any new file at all (depending on the level of activity normally on your web root) or files created by/owned by processes you don’t expect to be writing files.

5. Pay attention to the web server’s configuration to identify directives that adversely affect the site’s visitors. – This can be a bit easier in Apache or even IIS 7.0+, but some web servers like IIS 6 are hideous for watching configurations. Yet, definitely keep this in mind. Thankfully, attackers have extra hurdles than just writing a file in the web root, when it comes to such servers.

6. Use remote scanners to identify the presence of malicious code on your website. – I agree with the spirit of this, but I think internal scanners need to be done as well. If a bad file is present but not discoverable from an existing link on your site or easily-guessed name, then an external scan will totally miss it.

I should take a moment to stress that many of these items include signatures or looking for known badness, but none of that should replace actually looking, at some point, at every page you have for things that are clearly bad to a human, such as a defacement calling card or something.

7. Keep an eye on blacklists of known malicious or compromised hosts in case your website appears there. – It’s hard to say this shouldn’t be done, but it certainly offers little return on the investment of time. If you *do* happen to show up before any other alarms sound, then clearly it is nice.

8. Pay attention to reports submitted to you by your users and visitors. – I’d personally overlook this item 9 times out of 10, but it is really oh so necessary.

If I wanted to add an item or two:

9. Scorched earth. – While not a detection mechanism, you do run certain risks if your web code sits out on the Internet for a long period of time and grows stale. You should refresh from a known good source as often as you think you need to. This can include server configs and the like. (For instance, I roll out web root files every x minutes on my web servers at work, and I regularly wipe out web server configs and rebuild them via automation.) Don’t be that company that had a backdoor gaping open to the world for 2 years, when a simple code refresh would have closed the hole. A diff comparison mechanism may satisfy the ‘detection’ criteria of this list.

McKeay wrote a great post about blogging this past weekend, and I think any security blogger should check it out. I really like his subpoints about blogging and working and balancing both:

I’ve learned a number of lessons about blogging the hard way. I’ve learned that no matter what I think I’m writing, what’s important is how other people are reading it… I’ve realized that people are reading and judging what I write, for good and for ill. And when I write something people read, it can get back to my employer.

I also like this:

More often than not, my employers have maintained an air of benevolent ignorance towards my blog, but every so often I’ve gotten the “we’ve read your blog and are not happy” conversation. Not often, but it has happened and it’s never comfortable talk. I’ve actually told at least one manager that my blog and podcast are more important to me than my job.

For me certainly, blogging is a personal thing, a way to organize my own thoughts, record something for the future, or vent a little bit. It’s also a way to dive deeper into what would always be a hobby for me, even if not a job. Even if I didn’t have a single reader my blogging habits wouldn’t change a bit.

Anyway, here are some points of my own that I try to follow.

1. Separate work from personal if you need to. This is a big deal in the past 5 years, where work and play time are blending together, largely because something you “say” (digitally) during your personal time can now easily persist for years for people at work to discover. Things you could say with buddies or at a bar don’t just stay with buddies or at the bar in a single point in time. Therefore, with blogging especially, I try to keep work separate. I don’t hide my identity on here, but likewise I don’t advertise my blog to work colleagues (they can easily find it on their own if they want) and I don’t mention my employer anywhere on here. I also leave deep personal things aside, though some incidents/anecdotes if read by the right people, would recognize themselves in them, but I also try to make sure they’re generic enough and have enough of a point to not be uncomfortable. Besides, if I piss someone off, I hope they have my own viewpoint and just move on with life. It’s a big deal to be able to agree to disagree; a very useful skill. I like dark grey cars and don’t like white cars. You might not agree. And it would be silly to get pissed about that. Same goes for what I post on my blog or elsewhere on the Internet with my screenname.

I admit, my hard divide between work and personal is slowly going away, in part to my next point, but also partly because security work and play is a career goal.

2. Don’t present false faces. I don’t like when people “front,” or present themselves in a way that isn’t in line with who they really are. Life is way too short and precious to not be yourself in anything you do, and if being yourself gets in the way, make changes to be someone better. In that regard, I don’t typically pick my words carefully on my blog; if I have an opinion, I’ll be out with it. (Though it does help that I’m an easy-going kind of guy anyway…and this is also easy to say for someone who thinks of himself as a very decent guy who is sympathetic to objectivist beliefs…)

3. It’s easy to apologize or admit to being wrong. I don’t mean this to sound like a copout for bad behavior, but it is easy to apologize for or admit to being wrong. I find it’s more important to put your opinions out there and be contritely wrong, than to bottle everything up and stew. And this is a tough thing for an INFP to say! (And it’s something my risk-averse nature will always fight with me about.) Granted, that doesn’t mean you can be an asshole and then be contrite about it and things are fine…be reasonable!

4. Remember the important things in security: integrity and privacy. This also applies to IT work in general. Typically we are in positions to know very deep secrets and have access (or get access) to very sensitive things. The same principles that prevent me from perusing my CEOs mailbox are the same that dictate what I divulge on a blog, or anywhere on the Internet. Hopefully most people in white hat security are at least aware of these principles in every facet of their lives.

More information about the RSA hack has been uncovered. In the article, I especially liked this:

The email attack is not particularly complex, F-Secure says. “In fact, it’s very simple. However, the exploit inside Excel was a zero-day at the time, and RSA could not have protected against it by patching their systems.”

This should be a classic scenario for role-playing in any security operation. The first question from any manager: “What do we do to prevent, detect, or mitigate this?”

Visa has a slidedeck posted Identifying and Detecting Security Breaches. Sounds fun! If you’ve been around security for a while, nothing will be new in this deck, but it’s a nice and short to breeze through for ideas if something is missing in your enterprise security posture. Every bullet point also makes for a decent item to review or ask your team (if you have one) to describe how it is handled. (I do believe in role-playing!)

Of course, the danger in a slide deck like this is how deceptively easy it makes all of this sound! 🙂

In a new bit of detail that I hadn’t read previously, Dave Lewis posted about the recent IT admin “hacking” incident that occurred via free wifi at a McDonald’s: “An information-technology administrator has pleaded guilty to crippling his former employer’s network after FBI agents traced the attack to the Wi-Fi network at a McDonald’s restaurant in Georgia. The administrator was caught after he used his credit card to make a $5 purchase at the restaurant about five minutes before the hacks occurred.” Yeah, brilliant.

So, what should this guy have done? I have ideas, and I’ll assume we’ll stick to a McDonald’s.

location
– don’t go to any store you’ve been to before or will ever go to again.
– don’t do this in your own city; go to some other large city; day trip!
– legally park blocks away from the McDonald’s
– or park districts away and take public transporation (paid for in cash)
– do this at normal, busy hours and especially if you see other wifi users present
– en route, don’t speed, don’t do anything to get your location logged
– don’t go through tollbooths (if possible) and try to avoid cameras
– if you can discreetly do it, maybe rent a car

equipment
– use completely generic laptop and gear; nothing you can’t part with
– don’t name your computer anything that reflects you
– change the mac address (just because you can)
– don’t install customized stuff on the laptop; reduce the amount you may leak on the wire
– hopefully it is cool but sunny so you can go with a hat, sunglasses, popped collar…
– truly lose or “lose” your computer after (wipe it, sticker it up, etc)
– leave your cell phone at home (or turned off)

you
– don’t draw any attention to yourself; be invisible
– don’t wear your favorite clothes; be generic or even disposable
– buy a small meal or drink to go (no trays)
– for the love of god, pay in cash; pay for everything en route in cash (no ATM stops!)
– take your trash with you and dispose later
– don’t hide in a corner, but don’t let cameras or employees see your screen without you knowing it

– don’t browse the internet or check your email; do your business and leave
– remove jewelry or cover any tattoos or recognizable marks/traits you have

I’m sure there are more ideas if I spent more time, and I normally don’t think about how to stay off the grid like this, but this is a decent start for being mischievious at open wifi.

I’ve been mentally writing and rewriting a post about SIEM and IPS and spending time on tuning alarms, but just don’t really have a ton to say that’s new. Then I posted (minutes ago) about how we can’t have nice things…. It got my wheels turning…

One point the author makes is, “[solutions] tend to require a bunch of integration work…” Well, that’s sadly true; every enterprise vendor customer wants something different, some checkbox or some strange integration. The problem is how the vendors will often satisfy the need, but then insist on using that as an excuse to include the feature in the base product for everyone. This bloats products, making them difficult and confusing to use. The age old, “we’ll get customer Y to fund this new idea which we’ll then resell over and over after.”

I also believe it leads to dumber products and large blindspots, especially in security products that lose sight of answering the core security questions, “What actually gives me security value?” “What value does X give me?” It’s hard for a vendor to globally answer those, so it’s nice to let customers actually put in their own work on the tool, rather than automate everything and make it ramen-noodle-bland. Instead, vendors seem to be answering, “What would you like in the tool,” without referencing back to the core questions.

Getting back to SIEM and a concrete example, it’s a frustrating time trying to tune alarms down to a level where I’m not inundated by thousands of “usually nothing” alarms and not cutting such large swaths of blindness that a truck can drive on into my network. All while working within the sometimes awful boundaries of the tools at hand. I’m often mentally lamenting not being able to parse the logs myself!

Spend enough time with a SIEM, and you start to realize it’s not very good from a security perspective except in hindsight (investigation and forensics) and centralized log gathering. Kinda like DLP, it takes hands-on time to get past marketing positioning and actually figure out for yourself what the real value is going to be. There are better detection mechanisms than SIEM alone. (If your SIEM alerts on an event your better detection tools shovel to it, why aren’t you alerting from the first tool? The tuning will be better.) [Assertions like these are why this is incomplete…]

I’m sure there’s marketing in there, and maybe this is a long-term vs short-term marketing problem where you want a tool to sellsellsell rather than be a narrow-focus, useful, and long-term successful tool like an nmap or nessus or something; your tool just *is* useful rather than superficially forcing it.

This might be one of the underlying and subtle problems of a compliance-driven industry, unfortunately. Certainly not a nail in the coffin of compliance, but definitely a problem.

I can see why Twitter challenges and even betters blogs, as I see far more interesting and new stuff than I normally would with just an RSS reader as people I barely know retweet links from people I’d never know. This short article flew by this morning: “To write good code, you sometimes have to write bad code”.

I don’t even need to quote anything there, and if I had to make a change, I would remove, “sometimes.” This applies not only to code, and performance, and security, but to life in general. Taking some risks and being wrong is one of those weaknesses I struggle with regularly. Just have to keep saying: doing and being wrong is better than not doing at all. And that’s true pretty much every time I make a plunge. Sure I might get my hand slapped and I might even get egg on my face or skin a knee, but (and I have this up on my board at work): “A calm sea does not make a skilled sailor.”

There are so many little idioms I’ve stuck to me like velcro balls in a Double Dare physical challenge, like how we learn the most when shit hits the fan, growth through adversity, and so on.

For the article, I don’t think you *can* write good (and secure) code without first writing and learning from bad code. The problem is so many people in [web/mobile] development jobs only have homegrown knowledge and end up learning on the fly with production-level apps. We’re still in a relative infancy with computer programming (or higher level languages which change every 5-8 years like tech fads).

Since I saw this site for the first time, I glanced at a few articles on the MyInfoSecJob.com site, particular the security challenges. Reading the comments (i.e. solutions) for the first challenge, this pretty succinctly illustrates why infosec is so frustrating for business and IT persons! The range of answers is phenomenal, from simple to complex to flat-out suggesting complex setups with specific hardwired vendors and various other things.

I don’t think the answer to any, “help me secure this,” challenge should be to grab your favorite 600 page IT security book and thump it on the desk like you’re some pimp on Exotic Liability flopping your meat on the table. Keep it simple, and keep it on task with the information presented. Nothing in a network/data diagram really begs for a sermon about file permissions, and OS patching, and extraneous complexity for what is obviously a small shop. If you want to get further down that road, you can’t do so intelligently without more information. You’re just going to lose your audience (or demonstrate your lack of experience when suggesting over-the-top recommendations or flatly inappropriate ones…).

Anyway, based on that security challenge, these would be my simple recommendations:

– Replace the hub with a managed switch, assuming that is the basis of the underlying network connecting the users, the servers, and the router together. That’s the one real question the diagram makes me ask, “Is the hub separate, or is that what the blue ethernet network bar is supposed to be? You can pick up a soho one if you want for $100, or drop a grand or two for an enterprise level one.

– Drop in a firewall/VPN hardware device behind the router (i.e. between the internal network and the router). Configure this to position the web server into a one-armed DMZ, and set up necessary firewall rules to allow the access shown as needed on the diagram. Configure the VPN so external people can log into it and get to the fileserver as needed. Get a decent enough one that you can budget for; the features and support will be worth it. As a bonus, make sure VPN users are in their own subnet and even segment off the fileserver to its own, and configure firewall rules as necessary for everyone’s access. In the absence of other technologies, at least losing one part to an incident won’t caused the rest to be suspect; at least not by default. At worst, grab an old PC and figure out a tool like Untangle or IPCop…

This leaves open questions, but they’re questions that require further dialogue with the client.