InfoWorld has posted some more information which casts a different light on the situation in San Francisco where a “rogue” network admin is currently in jail for, well, something to do with holding the keys to the city’s FiberWAN and not letting them go. I think this article lends a hell of a lot of food for thought to both managers and engineers in enterprises, even if it is not fully accurate.

Picked this up from HardOCP of all places.

IT Stress. Every techie knows the feeling, or at least is subconsciously familiar with it if they don’t explore it directly. That feeling of impending stress or anxiety that is so very common in an IT job. More specifically on the network/systems side as opposed to programmers, managers, and analysts.

There is a pretty common cause for this sort of stress. In fact, the cause can help explain several differences between the systems/network side of IT versus the programming side. It explains why project management for programmers can be easy compared to trying to project manage a group of administrators, for instance.

USERS! Users are the leading cause of str…err…that’s cliche now. Let’s try something new…

New Technology Stress/Syndrome!

New Technology Stress/Syndrome (NTS) is the stress and anxiety felt by admins who are implementing or supporting new technologies that they are not otherwise familiar with, often with impending deadlines or expectations on ultimate correctness.

NTS appears in two forms. First, when an admin has to implement something new in his or her environment (replacing analog phones with VOIP; setting up a new IDS…). Second, when an admin has to support or troubleshoot or learn an already-implemented technology they don’t know about (John is on vacation and the system he is the expert on is broken and needs up now; John just quit leaving the other admins to cover his work duties…).

Now, it is really not enough to say that unknown technology causes the stress. It is more accurate to say that new technology coupled with impending deadlines is the real combination that causes stress. Most techies like learning new things, but typically in their own time. Being forced to learn something new, and have it learned and correct in *VAR_PROJECT_LENGTH* is really where the pressure sets in.

You know you have an admin lined up to fall victim to NTS when a project manager-type person asks them how long task 23B will take, and they respond with what amounts to, “I really don’t know.” That means they’ve never done it before, this is new technology, and estimating time is about as fun and accurate as predicting how many clouds are in the sky right now. That’s an admin feeling NTS! Regularly hearing your admin muttering to himself, “This is never gonna work…” is another good sign of NTS.

Consultants are not immune to NTS either! In fact, consultants likely constantly have to deal with learning new environments built by Jim-Bob and Cochese in the back room using bailing wire, 25 flashlights, and a box of extra-large condoms. Thankfully, consultants have an easy escape from the effects of long-term NTS exposure: contract expirations! Survive the gauntlet, then walk away and let the next sap live with it!

Ways to combat NTS:
– peering admins together. NTS can really set in when an admin feels like they are on an island with no help or support or colleagues to bring in and receive help from. Sometimes it’s enough to just have someone else to talk to. An admin alone is an admin that has to fight with his own thoughts or despair the longer NTS sets in. However, the lone admin who prevails in this fashion will emerge a battle-worn veteran, stepping out of the bonds of adolescence and into the confident world of adulthood; he might even find his power animal in between the monitor-glow-and-red-bull-induced hallucinations. Such touched admins should be revered in the workplace!

– befriend your support contracts – NTS is rooted in an admin tackling something they don’t know already. Having available and friendly support from a vendor or consultant can not only save time, but also dramatically reduce NTS. Admins should be readily encouraged to begin interfacing with vendor support early on. Most often, it just takes someone at hand who has done the task before and knows the end result is not impossible, to comfort the admin and massage away NTS!

– training – Again, nothing like giving the admin knowledge so that NTS doesn’t actually become an issue.

– proper time allocation – While NTS is about not knowing a technology, it is coupled with deadline expectations. Trying to learn a VOIP solution in 3 weeks when never having dealt with it before can be a sure way to lose your admin to the deep depravity that NTS can conjure! While an admin may not be able to estimate the time needed, give him or her plenty of leeway to help reduce NTS.

– less rigid time tracking requirements – Some companies or managers request that admins track their time spent on various tasks or projects through the day. While this has some value to managers in supporting their stafffing decisions, when taken too rigidly, this can add to the effects of NTS. It acts as a constant time-based reminder to admins, and can become an unnecessary guilt-trip if a task that should have taken 30 minutes takes 3 hours because of a misjudgement.

– be less rigid with project deadlines – Some projects simply have to be done by a certain date, but many are not quite so rigid. Yeah, project managers are notorious for trying to hit rigid deadlines, excuses bedamned! But they do have to realize that the admin side of IT are typically strong-willed enough to push back, or if not, know that the failed infrastructure will do the pushing for them once deadlines hit. Keep deadlines in mind to make sure the project will complete, but be considerate of NTS when dealing with lapsing deadlines that won’t necessarily kill any babies or cause gas price hikes.

– allow alcohol in the workplace, and LAN parties – Few things help an NTS-afflicted admin unwind more than being able to drink beer and LAN in the workplace, even if just after-hours. Besides, too often NTS-laden admins are spending after-hours time in the workspace anyway, so one may as well provide some accomodations to make the time less stressful and more comfortable.

– MAME cabinet in the server room – Again, a nice way to unwind for the NTS-afflicted. Keep it in the server room where it will be less used by unauthorized persons. The blips and beeps can be explained away as server noises. And admins won’t work up an unsightly sweat when conquering arcade-level game challenges.

Thankfully, rest assured that good admins will eventually learn their new technology challenges and become old hands at supporting it, answering questions about it, and even exhibit accuracy in time-estimates for various tasks! This might not be without several painful lessons, but the end result should be worth the adventure with as little NTS affliction as possible. Hoo-ray!

Farnum posted this the other day and I can’t stop watching it and laughing, giggling, like a little girl. Head over to www.thewebsiteisdown.com. This is not safe for work without headphones, and possibly visually later on.

Over time I’ve been putting together a list or laws or rules that govern our industry, or affect us. I’m adding a new one:
Security is the new darling of mass media. Since security is not absolutely and will always be broken and security at some point has to trust something else without assurances, then security will always be potentially broken (no matter how insane or movie-script-like the scenario is). Likewise, security will always ultimately depend on fallible people. Thus, the media will forever have something to wave around when it comes to security, or insecurity incidents. And thus, the media should not be our guideline or focus when it comes to evaluating our security stances. The media only provides for measurements of security theater (which itself is important to keep in mind, but does itself not convey much real security value).*

* If I were to play devil’s advocate here, I would say that many things like even our police department does not convey as much security value as it does value through security theater. What prevents us from looting and pillaging and ravaging in the streets? Most often our collective moral compass; our knowledge of right and wrong and not wanting to be seen doing the wrong thing. This is why once a criminal steps over that moral line, it is forever easy to continue to step over it. The internets don’t and never will (and never would have!) that same moral inhibition, no matter how much we try to strip away anonymity (we can’t)…

With all this DNS stuff going around, obviously Dan Kaminsky has found something interesting, and the fix is to use random source ports. Now, that might simply mask the real vulnerability by upping the effort needed to leverage it. Or it might simply prevent some other avenue to be popped (someone on FD threw out ICMP responses..). I really don’t know, and am lookig forward to the outing at Black Hat (I won’t be there, but I’ll be waiting and watching from afar).

Halvar Flake has a blog post that can help put this issue into a bit of perspective, at least to the net geeks. He essentially says we shouldn’t have been trusting DNS anyway, so this isn’t a huge thing to worry about. To the rest of the world, unfortunately, that doesn’t necessarily apply quite as nicely…

1. Halvar will tell us we shouldn’t be trusting DNS anyway. The rest of the world does not understand that and will be asking either why we use it, or why we don’t use a secure implementation of it. Of course, at some point somewhere we have to deal with something we can’t trust if we are to interact…

2. C-levels wouldn’t understand it if this bug became weaponized and used to mass-poison servers, preventing them from trading their stocks (or their company’s stocks!). Untrusted or not, they’re affected and that will slide downhill and become our major headache.

Yesterday, Dan Kaminsky posted a long post about his latest DNS find. In it, he give some incentive to find the bug before his talk!

Now, if you do figure it out, and tell me privately, you’re coming on stage with me at Defcon. So I can at least offer that

And Dan also has a tool on his site to test your DNS server (it appears to go after the DNS server authorizative to your IP address, i.e. the ISP DNS server. When I run the test regularly, it always tells me “All requests came from the following source port: 35353.”

If I were an ISP, I would do this against my own DNS and then watch the wire to see exactly what tests Dan is probing my server with. 🙂 Some further reading by Joe Stewart on DNS cache poisoning and Dan J. Bernstein on various DNS challenges. And yet more DNS challenges. And another nice paper which includes non-spoofing poisoning attacks (btw, papers with no dates in them fail).

Some questions: Is the bug in the servers or actually in the cache gets poisoned on the clients because of a predictable responses? If on servers, I wonder if it is more vulnerable to local networks as opposed to external attackers? Just some thoughts…

In some random browsing (and ranting!), I ran across a post by Ron Newby which talked about a recent quote from Matasano. Ron reacted to Matasano’s: “Firewalls are underrated, but only by an industry which is perpetually looking at selling you the next new thing.”

I think the point here is simply a paradigm difference between how people get things done. It might also be reflected in that Matasano guys likely use the tools whereas Ron has been on the sales side (or has that background).

If you have to dig a hole, do you expend some energy and use a shovel, or do you rent a backhoe (with covered canopy, internal air venting, a scoop on the other end, new paint job…)?

On one hand, you spend time and effort to do what is maybe a more surgical job with a tool that will almost certainly not fail you. (And you better have the back for it!)

On the other hand, you save time but may have to wait in line to acquire the equipment, maintain it, operate it, and probably lose some surgical ability with a large scoop and machinery between you and the ground being broken.

And of course both choices still require measuring where the hole should be, dealing with the excised dirt, tracking the progress, making sure the direction is clear, etc.

There is merit to either situation such that I would never dispose of one or the other. However, I will never say, “…they are promoting firewalls, which suck, and will always suck, and should be shot…”

What should I have instead of my firewalls?* Some UTM that runs on clunky Java and tries to do 39 different things, none of which is does exceedingly well and only 3 functions I need for a consulting gig at a ma and pa shop (because their marketing teams place more emphasis on number of features rather than useful/valuable features)? Sure, there’s the age-old “security religion” issue where one side will denounce firewalls because they can’t stop everything, but that’s, again, a paradigm and a situational difference. Not a universal right or wrong, value or non-value.

I agree with Matasano that it sucks we keep having vendors push new things on us over and over and end up driving a lot of the security we see in the press today (yay marketing and sales cycles!). I mean, they have reasons to innovate for their own economic gain; not necessarily because the security industry has new needs.** And I will say that just because something is new, does not mean it adds value to me beyond tried and true tools from the past.

* Fine, yes firewalls should be better defined before denouncing or defending them. And yes, firewalls that have no context into application layers 1-7 have less utility.

** There are new things, don’t get me wrong. Old tools won’t protect new stuff like virtualization or newer web 2.0 coding languages or practices, for instance.

A report posted by Brian Krebs at the Washington Post (one of the few major publications whose security reporter I actually enjoy reading most of the time!) further illustrates why the assholes in IT and Infosecurity exert control and policy over end user systems.

Sometime late last year, an employee of a McLean investment firm decided to trade some music, or maybe a movie, with like-minded users of the online file-sharing network LimeWire while using a company computer. In doing so, he inadvertently opened the private files of his firm, Wagner Resource Group, to the public.

The breach was not discovered for nearly six months. A reader of washingtonpost.com’s Security Fix blog found the information while searching LimeWire in June.

It would be nice to allow employees full use of the web and their systems if it weren’t so risky, eh?

Updated DNS cache poisoning information. It’s nice to have a recent posting, eh?

I got pointed (via elamb) to an article discussing net neutrality: five facts everyone should know, hosted by the folks at 10gea.org (10gigE Alliance). Net neutrality on Wikipedia.

The concept of net neutrality is an interesting one, especially when you look at the economics of it. It makes sense to limit traffic if you’re a carrier with limited bandwidth/resources (or will someday be limited). But it makes sense to have unlimited traffic if you’re a consumer. Business wants to cut costs; consumers want their freedom of choice.

To emphasize points in the article:

2. Net packaging. Yeah, I think we should really never talk about net packaging ever again. AOL tried this approach with their wall-garden business model. It doesn’t work or suffice for too many users. Likewise, for every site that wants to charge even small prices for content, there are 3 other sites with nearly the same content for free. Or if it is new and has no peers, it will in a year or two when the business model proves unprofitable or too many alternatives appear. Cable companies still tout packages of channels, but this is slowly going away (as slowly as they can make it).

3. Networks are “protecting” consumers. Fine, this is a great marketing point, but the article is correct: any protection is simply a coincidental by-product. And even then, it can’t be all that secure for everyone. Any protections an ISP will provide will be like swatting flies with a sledgehammer. Even non-ISP services like DNS providers or site advisors or email server blacklists are clumsy and end up swatting legit sites in their wide swings.

4. Speed Throttling. I don’t feel this has as much to do with net neurality because it is more a function of speed as opposed to open access or blocked traffic. It’s also something I won’t get into much. I’ll pay what I have to for satisfactory service and move on. Then again, I am weird and couldn’t tell you the price of gas on any given day or week (I don’t check prices, I just fill when I need to, pay it, move on; it’s not important enough to care about until it impacts me such that my habits have to change and I drive less…). As long as I can pay the bills and do what I do on the net, I don’t much care.

Thankfully, despite all the passionate talk about net neutrality, this is a geek’s realm: internet access. There will always be alternative providers that understand geeks and offer good bandwidth without restrictions or delusions of making money off weird implementations (like Mediacom, my cable provider, which hijacks every bad DNS lookup I make in a browser). This is still an economic consumer system that is ultimately going to be ruled by demand, not supply. Sure, there are plenty of consumers who will just do whatever, but they’ll just end up in the next AOL walled-garden of disappointment.

Over on DarkReading I just read up on a finding by Dan Kaminsky that is resulting in a rather huge rollout of DNS server patches from a crazy number of vendors. Seems like someone either hit on a critical issue or, as Ptacek is quoted in the article, an exploit has been developed.

It sounds logical that the issue is related to old issues with spoofing query responses fast enough (and when leveraging recent well-known PRNG issues) and today’s ability to send lots of packets really fast. Bombard a server with specific DNS queries while at the same time spoofing a bombardment of responses to the server that look like they are from an authoritative server, and you might just hit upon a good combination which can poison the DNS cache of that server for a short time. Anyone else making the same DNS request from a poisoned server will be given the bad IP address and get sent to the bad server.

Being able to actually weaponize this would be pretty valuable as users would really never know they were on a bad site unless their browser queries several DNS servers to compare the results or the bad server IP is blacklisted somehow. Calling in to tech support when the site doesn’t work (for instance when the login isn’t accepted) will result in a lot of testing before even possibly hitting upon the problem. Then again, attackers can just make a fake front page and pass the users on to the real site after farming out the login info. Until the accounts are hijacked, no one is going to be the wiser.

Follow the links in the article for more information on the older issues. More info on PRNG vulnerabilities can be found from your local Google site.

Deepsec 2007 talks are up on Google Video. Just in case I find some free time. 🙂

Richard Bejtlich has posted recently a comparison of current information security practices to the times of Galileo. Rather than listen to the same old rhetoric and belief, Galileo centered his claims on empirical (measured) evidence. This sounds similar to the concept of “management by fact” (which Bejtlich has posted on previously as well). I think there is a lot of merit in measuring what we do in infosec and then managing by fact.

I do, however, have one minor criticism of this approach, while not actually disagreeing at all with Bejtlich.

Galileo used measurements to shatter beliefs, but many things that seem like beliefs in infosec may well have been at one time or still are based on measurements (the validity of the measurements may be suspect, however!).

Would it be management by belief if 50 companies reported measured success with a password policy, and I simply accepted that conclusion and implemented it? Or that patching within 30 days didn’t help 500 incidents so why bother? Holding too firmly to the Galileo example (management by fact) may end up insinuating that unless you personally have made the measurements, then everything else is belief. But not everyone has a big telescope.

This might be a discussion on the validity of statistics versus facts versus belief versus best practices versus risk…

Galileo benefitted from two things that we do not have. A) Nothing he nor anyone else did would change the a priori truth that the Earth revolved around the Sun. People just had to measure it correctly. B) No one had provided the proper measurements before. At all. We don’t have the assurances of A in infosec, nor are we forging absolutely new ground like B.

Now, while I offer up the above, I don’t say that companies should get away with not measuring their own implementations, not at all! I just don’t want to too stubbornly go down a road that leads to an egocentric security stance that may or may not be right.

Maybe because of A this is a discussion that needs to branch into two directions and not mix the two: macroscopic infosec and microscopic infosec. Macroscopic infosec would deal with large entities and their interactions (ISPs, global security, standards, compliance, or universal practices that everyone should pay attention to). Microscopic infosec may be dealing with what one company implements within its virtual walls, how it measures it, and manages by fact.

He says this was written 10 years ago, but it could have been written today. Thomas Ptacek has an excellent (and painful!) essay on how to hide security problems in your products/services.

Bruce Schneier has a new Security Matters article up on Wired. He talks about the growing trend of “kill switches” on various electronic devices.
Definitely not a good idea, but I think at least in the consumer markets, economic forces will keep such products from getting too out of control. For instance, I am still in the casual market for a new portable digital music player, and I won’t be getting an iPod. Basically I don’t trust Apple in conjunction with iTunes and my digital media (not all of which is legal). I want to manage my device out of band, and really never have to worry about DRM or the firmware suddenly making decisions for me.

Bruce is correct in worrying about the chains of authority when you start giving one device power over another. The wider the more dangerous. Maybe Windows should have a killswitch that is remotely accessible? We can bring back teardrop/Ping of Death!