Thursday, April 27, 2006

Thoughts on Patching

As I continue through my list of security notes, I thought I would share a few ideas here. I recorded these while seeing Ron Gula discuss vulnerability management at RMISC.

Many people recommend automated patching, at least for desktops. In the enterprise, some people believe patches should be tested prior to rollout. This sounds like automated patching must be disabled. I'm wondering if anyoen has implemented delayed automated patching. In other words, automatic updates are enabled, but with a two or three day delay.

Those two or three days give the enterprise security group time to test the patch. If everything is ok, they let the automated patch proceed. If the patch breaks something critical, they instruct the desktops to not install the patch until further orders. I think this approach strikes a good balance since I would prefer to have automated patch installation be the default tactic, not manual installation.

Determining which systems are vulnerable results in imagining a continuum of assessment tactics.

  • At the most unobtrusive level we have a "paper review" of an inventory of systems and their reported patch levels.

  • Next comes passive assessment of traffic to and from clients and servers.

  • Traditional vulnerability scanning, without logging in to the target, is the next least obtrusive way to assess hosts.

  • Logging in to a host with credentials is another option.

  • Installing an agent on the host is a medium-impact approach.

  • Exploiting the host is the final way to definitively see if a host is vulnerable.

On a related note, Ron mentioned that the costs of demonstrating compliance far exceed those of maintaining compliance. This is sad. Ron also noted he believes auditors should work for the CFO and not the CIO. I agree.


foQ said...

This "Delayed Patching" scheme is exactly the kind of scenario that Windows Update Server addresses. All patches are downloaded to the server, then they are "published" to different groups. So you could have the latest patch published to the finance group, but not the PR group if it breaks one of their apps. It's really a great tool for anyone who wants to test patches before putting them in production environments.

LanDesk has a similar feature in their Security Suite which will put the patch on the PC, but not install it until you give the command. That way you don't kill your network if you need to patch right now.

Richard Bejtlich said...

foQ: that is cool, but I mean patching happens automatically unless someone intervenes -- not patches are downloaded and then told to be applied. I want the default to be patch application, albeit with a small window to stop automatic application if necessary.

foQ said...

I believe that if an organization takes a systematic approach to testing patches, they will also take a systematic approach to applying them, as well. Conversely, if an organization does not test patches, they are likely to either apply everything blindly or apply nothing. If an organization applies everything, then your idea for delayed patching might work with enough delay so that other organizations and vendors could test and fix things.

This could partially be accomplished by having Automatic Updates check for new patches on a certain day of the month. If AU were set to check on the 1st of every month, this would give the patches at least 2 weeks of maturity before being applied. I guess this would be the kind of strategy you are invisioning?

Ryan Russell said...

Note: I work for BigFix (who sells a patch management product) and co-moderate the mailing list. So, I've thought about the topic a bit.

What you're proposing a deadman's switch, which I'm a fan of, for unmanaged (usually home) computers.

There are plenty of good reasons for not forcing patches on people, and 2-3 days is WAY too short for the majority of enterprises. Some of the biggest patches, such as XP SP2, are often held for a year. You can get away with that, since you have hotfixes to SP1 still, mitigating methods, etc..

Any decent patch management product lets you flip a switch and deploy to all clients on-command, so there's really no need for a drop-dead date. There are many admins as most companies, so it's not like it will just get forgotten. Again, this is obviously for managed computers.

Remember, we just had a MS Patch Tuesday where the patches broke many things. That sort of thing is why businesses wait and do testing.

jbmoore said...

TI has a group which tests Microsoft patches and decides whether to include them in SMS rollouts and internal Windows Update Servers. They omit certain patches if the patch breaks particular apps. I know this because a good friend is a testing administrator. TI has its own version of SP2 with some MS hotfixes omitted because of software incompatabilities with in-house applications and such. Many of their helpdesk calls are from TI employees who go to the MS Windows Update Website and patch their laptops and then wonder why their applications suddenly don't work.

Anonymous said...

Yes. But unfortunately, not in an automated way.

For quite some time, my patch strategy has been, essentially: 'Apply the patches 3-5 days after release, unless there's a big uproar in the mailinglists and blogs I subscribe to'.

Basically, I'm letting everyone else do my testing for me. Why reinvent the wheel?