Securosis pointed me over to a really cool post by Michael Howard as he discusses SDL and the SMBv2 bug that was patched this month.
The takeaway I get is you really can only do so much to scan code, do code analysis, and even code reviews. There will still be bugs like this that make their way through. Automatic analysis just can’t find things like this. And humans make mistakes when reviewing things. (I suppose even code variables could have metadata in them to be marked as “untrusted inputs” and thus highlighted for more scrutiny? It’s like writing code to vet code…which is just odd to me since I’m not into comp sci…but maybe that’s what he’s talking about with their “analysis tools.”)
The only current way to find a bug like this is fuzzing.
But that should bring up the point of how much is enough fuzz testing? For instance, you won’t know if there *is* a problem in some code, so how long and deep should you fuzz? How do you prove it is secure? At some point, you really just have to release code and hope that what is essentially real-world fuzzing by millions of people will eventually reveal any missed issues, at which point your response teams can patch it promptly. Although, hopefully you’ve done enough fuzzing to match just how critical your software is to others (Windows? Pretty critical!).
Funny, that sounds a lot like the mantra, “security eventually fails, so make sure your detection and response is tight.” I’m glad we already look past raw numbers of security bugs, and focus in on how quickly they’re fixed by vendors, and how transparent/honest their process may be. Microsoft has really come a long way down this road.