when security goes too far

An article just ran across my desk about a bank whose legitimate (albeit poorly implemented) email announcement to customers was mistaken for a phishing attempt. This is an example of a false positive. But just how damaging can a simple false positive be?
What we do now:
– automatic spam filters that “learn” what spam is
– manually populated spam filters
– spam blacklisting which can blacklist sources or content across a wide swath of customers
– heuristic and behavior-based virus scanning
– phishing site blacklisting
– blacklisting of DNS, domain, or IPs based on complaints or automatic alerts
– network and system shunning via IDS/IPS linked to firewalls
That’s a lot of stuff reacting to security incidents. What might have happened to this company? Someone may have reported them to a phishing blacklister or alerts may have automatically done this, blocking perhaps the domain, emails, website IP, or even DNS for this bank. This could cost tons of money in lost business, public relations, and direct costs to fix or workaround the issue.
In a previous job, we sometimes were blocked from emailing AOL members because, after a complaint or two, AOL would block our email servers for 24 hours. The sad thing is, we never spammed people unless their own employer or they requested it or agreed to it. Also, one of our clients, a major financial institution at one point had their domain blacklisted for spamming. Now, they may have really been spamming, but due to that disruption in service by being placed on a blacklister, they had to change their domain name and all the infrastructure that it used. Wow!
And as much as people like this stuff, mistakes will still be made. People will make bad judgements, misconfigurations, or poor decisions like the bank email security campaign linked above. To make a mistake and cause your company millions is just a bad situation waiting to happen.
Dan Kaminsky was correct in his talks last year (BlackOps of TCP 2005) decribing how scary it is to have IDS/IPS automatically making firewall rules and shunning networks. This means that attackers can actually write your firewall rules and can do some things as disastrous as having your own network shun its own name servers and be subjected to DNS poisoning.