Tuesday, July 14, 2009

White Hat Budgeting

After publishing Black Hat Budgeting last month, several readers asked me how to spend the same $1 million on defense. This is a more difficult question. As I wrote in the previous post, for $1 million per year an adversary could fund a Western-salaried black hat team that could penetrate and persist in roughly any target it chose to attack. That does not hold true for defense, i.e., for $1 million per year a defender could not fund a Western-salaried white hat team that could plan, resist, detect, and respond to any $1 million black hat team.

So, if you had $1 million to spend on defense, how could you spend it? I turned to my 2008 post Defensible Network Architecture 2.0 as a guide. One interesting aspect of the eight DNA 2.0 tenets is that half of them are IT responsibilities (or at least I would strongly argue they are): inventoried, claimed, minimized, current. All of that is just "good IT." Security can provide inputs, but IT should own those aspects. That leaves monitored, controlled, assessed, and measured.

With that's, let's allocate the funding. With such a small team we would expect people to move among the roles so they don't burn out, and so they can grow their capabilities.

  • Staff. Without people, this operation goes nowhere. We allocate $850,000 of our budget to salaries and benefits to hire the following people.

    • The team leader should have experience as an enterprise defender as a minimum. The leader can be very skilled in at least one speciality but should be familiar with all of the team's roles. The team leader needs a vision for the team while preserving business value. Because this team is so small the leader has to do strategic thinking and overall management, including the "measured" aspect of DNA 2.0. $120,000.

    • The incident response team is responsible for detecting and responding to intrusions. They perform the "monitor" aspect of DNA 2.0. We hire three people, one with Windows expertise, one with Unix expertise, and one with infrastructure expertise. $330,000.

    • The security operator is responsible for the "controlled" aspect of DNA 2.0. He or she seeks to minimize intrusions by deploying and operating countermeasures. This person is also a utility player who can learn other roles and consult as necessary. $80,000.

    • The threat operator performs an advanced security intelligence and analysis role. He or she should be able to reverse engineer malware while also paying attention to underground activities and applying that knowledge to all aspects of the team's work. $120,000.

    • The Red-Blue Team performs adversary simulation/penetration testing (red) and collaborative vulnerability assessment (blue) activities. With a team this size there is only room for two technicians. Red-Blue handles the "assessed" aspect of DNA 2.0. $80,000 for the blue, $120,000 for the red.

  • Technology. At this point we only have $150,000 left. We can spend $100,000 on technology. It should be clear that $100,000 isn't going to buy much of any commercial tools. In fact, the $1 million security operation is going to have to rely on several realities.

    • Built-in capabilities. This team is going to have to rely on capabilities built into the products deployed by other IT teams, like the computer and networking groups. This actually makes a good amount of sense. Is it really necessary to deploy another host firewall on Windows if you can use IPsec policies and/or Windows firewall? With a budget that small, these are the uncomfortable choices to be made.

    • Open source software. The $1 million security team should deploy a lot of open source software. Sguil could be the NSM suite of choice, for example. By spending money on staff who know their way around open source tools, you can go very far using what can be downloaded for free. Let the staff contribute back to the community and it's a win-win situation.

    • Commodity hardware. You can't buy hardware for free, and those NSM sensors and other open source packages need to run on something. A decent amount of the budget will be spent on hardware.

    • Cloud hosting. The Cloud becomes an attractive place to store logs, do processing, and other activities that don't scale well or work well on commodity hardware. Security concerns are lessened when the alternative is no security services.

  • Miscellaneous. The last $50,000 could be spent on incidentals, training, team awards, travel, or whatever else the group might require to attract and retain talent.

Note I did not advocate outsourcing here. You spend too much money and probably won't receive value for it.

With such a small team, there is no concept of 24x7 support. 8x5 is the best you can get. The ability of the team to detect and respond to intrusions in a timely manner is going to decrease as the enterprise grows. A team of 8 security defenders will be strained once the company size exceeds 10,000 people, at the largest.

I am much less comfortable building out this team, compared to the Black Hat Budgeting exercise. There are way too many variables involved in defending any enterprise. Most companies really are unique. However, this is a good point to stop to see if anyone has comments on this approach.

Richard Bejtlich is teaching new classes in Las Vegas in 2009. Late Las Vegas registration ends 22 July.


Anonymous said...

Do the salary numbers reflect full FTE cost w/ benefits or just the salary? If those are full FTE costs I am somewhat alarmed at the low value of skilled security personnel.

Richard Bejtlich said...

There's a couple ways to look at this. You hire staff in a very cheap place in the country so you're not paying the premium for people in Silicon Valley, NYC, Boston, northern VA, etc.

You can't afford the best money can buy. Period.

You chop one of the staff and distribute the money to the other people.

Your benefits really stink.


hogfly said...

You've left out all forms of equipment. You have an IR team with no IR/forensics gear. Outfitting the team will cost you atleast $30k if you expect them to be effective. There's also a spending discrepancy..the blackhats have $200k for tech and the whitehats have $150.

Why have a threat operator and three IR team members? Every IR team member should be a threat operator by trade. If the IR team isn't paying attention to threats, the threat landscape, and the underground, then I'd question their salary and their reason for even being there. I'd cut the threat operator and add 10k to each IR team member's salary.

In addition, not every member of the team needs to be highly paid or even highly trained. 1 Sr. member for 2 junior members is a good breakdown. You can drop salaries and expand the team or have fewer, but more highly skilled members.

Richard Bejtlich said...

I told everyone this was going to be difficult. :)

What do you need to buy for $30k? The only commercial tool I could justify within such a tight budget would be F-Response. Otherwise you could do well or get by with open source. For example I was using Sleuthkit to do drive forensics when I first arrived at this company because I didn't have alternatives, and I tend to like Sleuthkit for the sort of drive forensics I need to do anyway.

I opted to spend less on tech and more on staff for the white hats. The black hats have 7 people, the white hats 8.

Richard Bejtlich said...

Regarding pay, I allocated $330k for the IR team. You could have high-medium-low there.

I really think you need a threat operator. The IR team is going to be busy on so many daily tasks that I think it helps having that guy in the back room who does nothing but figure stuff out and tear code apart. Of course the IR guys need to be threat-aware.

Anonymous said...

How are the salaries justified for each person on the team? I have a hard time understanding how much a particular skill-set is worth.

One example is the Blue team member getting 40k less then the red team member.

Richard Bejtlich said...

For this post and the black hat post, I used rough numbers based on my experience. I used a lower number for the blue team person because I believed a more junior person could do this role. The red team role has more responsibility because it involves exploiting production systems while blue team is vulnerability assessment.

Don Gray said...


I work for an MSSP so your approach and comment regarding outsourcing is very interesting to me.

One of the biggest drivers we see for engaging our services is precisely the fact that 24x7 coverage is not possible without greatly expanding staff.

In addition, attracting the talent you are indicating and keeping them current and trained is no small feat for a "regular" company either.

I would suggest that a productive twist on your scenario is to spend the $150K on an MSSP that is going to provide the 24x7 coverage for monitoring, an SIEM platform (security portal) for the in-house analysts to use, and the reporting tools to communicate security posture to management.

Rob Floodeen said...


Thanks for starting a conversation on this topic.

I would like to see the roles expanded to include skill sets. Like I showed you last year at FIRST, the break down of roll to function.

For example (not conclusive just to give an example)

Jr. Blue Team member,
Scripting / Intro Programming Logic
Operating Systems

Sans Track 3 (Judy's class)
TCP/IP Weapons

abc's of 123s

Online Challenges:


Richard Bejtlich said...

Rob, you're killing me. I'll save that for a book chapter or presentation. :)

Mariusz Ziułek said...


I've read posts about black/white hat budgeting and find it very unrealistic.

Do you really think that person with these skills:
...should have experience as a vulnerability researcher, exploit developer, penetration tester, enterprise defender, and preferably an intelligence operative... would take "job" for 120k$/year ?

If I were such person I would take red team's member position in defender's team, making the same money for LEGAL and (in my opinion) easier job.

In my opinion blackhat's salaries should be doubled because of a risk (in the end it's criminal activity) they face.

nr said...

Don, my feeling is that an employee of a MSSP does not have the same investment or motivations in protecting a company compared to someone working directly for the company. I also would be surprised if $150,000 would get anything beyond scope dopes who don't really understand the business they're monitoring.

You can see some of Richard's previous posts on MSSP versus internal security staff for his viewpoints.

Mariusz, what would be illegal about a red team's job functions? I think you may be misunderstanding how a red team works.

Mariusz Ziułek said...


I wrote that working as a blackhat would be illegal not as a red team's member.

Phil Agcaoili said...

Here's a novel approach...

Break everything and fix it and detect nothing.

Spend your entire $1M finding, exploiting, and fixing your own holes.

Without detection, you won't have to alert that there was a breach.

Assumption here is that SOX, PCI, etc. goes out the door since you do not have detect controls.

What if you outsourced credit processing? No more PCI.
Went private?

I think this model works.

Funny part is apply this to the US government.

Could you have hardened the Navy infrastructure without anyone knowing or caring? You would just spend the resources finding and fixing, keep moving, and repeat without ever stopping since you cycle through once you hit your last target (of course add discovery).

Thoughts on this?