Richard’s post about monitoring and “management by fact” got me thinking about security for the real world admin. What is the best sort of server to monitor? That’s easy, the server that requires the least changes. If you stand up a server and don’t need to do anything beyond patches and application-level updates (for a DNS server, adding DNS records…), monitoring that box becomes amazingly easy and informative.

You can quickly tell when something is wrong. Besides, typically in troubleshooting (and it is part of Cisco’s troubleshooting methodology) is to ask pretty early on, “What changed?” This is something really near and dear to my heart, since I used to be pretty heavy into sciences back in college: observable changes causing observable results. If something weird happens, figure out what the one-off is that caused it.

There are really two problems in business that fight a never-ending battle against the unchanging server.

First, the technical ability of the admin is crucial. Take a new DNS admin tasked with standing up a DNS server. It might not take long to get the DNS server up and running, but to get it tuned for performance and security may take weeks, months, even years of small changes, mistakes, and troubleshooting. For an expert, experienced DNS admin, this “time to stable” is far shorter and much more ensured. This is partly why we need more experts (training) in the back rooms of IT, the luxury of making mistakes to become experts, and time to do proper research so we can be empowered to do more initiatives outside of our comfort zones (otherwise we just say, “no”).

Second, business sometimes likes to cut corners, especially with money and especially with IT infrastructure. If a server isn’t choking, it must have room to put more on it, right? This defeats trying to efficiently “manage by fact” in the IT back rooms. If you have an SBS box that does basically everything that can be crammed into it, the constant flux of use and changes can make creating a baseline and monitoring for oddities frustrating.

I love the idea of managing by fact, and I think for the most part of security, that should be the goal to someday reach.