I like stories of things that work and don’t work in security. The SANS Internet Storm Center reported this story of a router hack.

Three concepts stand up pretty loudly here, and are echoed in the lessons learned part of the story.

1. Monitor for changes! Having a script pull configs and compare them for changes, then raise an alarm is really small effort for huge gains. This can also work as an internal change management control as well.

2. Logs are vital.

3. We make mistakes as humans, and we need to assume they will be made and those mistakes will be found by an attacker eventually. Always review devices, configs, settings, logs, scripts, etc. Reviewing this stuff is boring and often reveals nothing, but that one time it does reveal something like an unremoved test account or access, will save bundles. If that attacker had more time and had simple done more, he may have already captured some data or dug in deeper into your network, past the config-protected routers. At least the Rancid script cut this off, but there was still a window of time where the attacker was in control and could have done more.