Microsoft Third-Party Patch Testing

I just read an article titled Microsoft Turns to External Patch Testers. The goal is "is to provide a small number of dedicated external evaluation teams with access to the [beta] patches to test for application compatibility, stability and reliability in simulated production environments." This article cites a Microsoft rep saying "'This is a very controlled program... We have only invited participants with whom we have a close relationship, where we are sure that confidentiality will be maintained.'"

This comment makes me question if Microsoft understands what it is doing: Stephen Toulouse, program manager at the Microsoft Security Response Center, "made it clear that the outside testers had no access to information on the vulnerability addressed by the patch. 'They're evaluating the updates in a private, closed-lab environment. They are required to sign an NDA [nondisclosure agreement] and they don't ever know what the patch is correcting. They're simply simulating a real-world deployment in a lab environment and looking for potential problems,' Toulouse said."

At the very least, patch recipients will be able to see what files were changed on the target system if they use file integrity verification software. The testers may not know exactly what problem is being corrected, but any competent tester will know that XYZ.dll and ABC.dll have been replaced by Microsoft's beta versions.

Any program involving greater testing of patches is probably a good idea. However, Microsoft should have realistic expectations concerning the sharing of information on replacement of .dlls and other Windows components.