Follow-Up to Donn Parker Story

My earlier post is being debated on the private Security Metrics mailing list. I posted the following tonight:

Chris Walsh wrote:

> Alrighty.
>
> It's time for a Marines vs. Air Force slapdown!

I should have anticipated that someone on this list would read my blog!

I do not agree with all of Donn's points, and I state in my post some
of his ideas are weak. I would prefer Donn defend himself in person.

However, I am going to stand by this statement:

"As security professionals I agree we are trying to reduce risk, but
trying to measure it is a waste of time."

I agree with Donn that a risk measurement approach has not made us
more secure. That does not mean nothing can be measured. It also
does not mean that measurements are worthless.

Removing the double negatives, I am saying that some things can be
measured, and measurements can be worthwhile.

Rather than spending resources measuring risk, I would prefer to see
measurements like the following:

1. Time for a pen testing team of [low/high] skill with
[external/internal] access to obtain unauthorized access to a
specified asset using [public/custom] tools and [zero/complete] target
knowledge.

Note this measurement contains variables affecting the time to
successfully compromise the asset.

2. Time for a target's intrusion detection team to identify said
intruder (pen tester), and escalate incident details to the incident
response team.

3. Time for a target's incident response team to contain and remove
said intruder, and reconstitute the asset.

These are the operational sorts of problems that matter in the real
world. These are only three small ideas -- not a comprehensive
approach to the problem set.

Sincerely,

Richard

PS: Go Air Force. :)

Comments

Anonymous said…
The problem with the first measurement is that there is no objective way to grade pen testers as having high or low skill. You can't grade them by years of experience or college degrees.

The second measurement runs against the first. A successful penetration will occur more quickly if the attacker(s) simply disregard detection--of course, there is the caveat that they could get discovered before they get access. If the attacker's goal is to not be detected, he should proceed very slowly and carefully, generating only a minimum amount of traffic and spreading his activities over a longer period of time to avoid triggering various alert thresholds.

The time taken to contain and remove an intruder will depend on what was compromised which doesn't make this measurement very useful for comparison.
12:26 PM
Richard Bejtlich said…
Steven,

1. I can differentiate among pen testers after talking to them for 30 mins or less. More formally, as part of your selection criteria, have candidates complete one or more exercises to vet their skill levels. I have participated in such exercises.

2. Re detection -- easy: Time for a pen testing team of [low/high] skill with
[external/internal] access to obtain [stealthy/semi-stealthy/unstealthy] unauthorized access to a
specified asset using [public/custom] tools and [zero/complete] target
knowledge.

3. The term "specified asset" answers your "what was compromised" comment.
1:59 PM
Post a Comment

Popular posts from this blog

Zeek in Action Videos

Image
This is a quick note to point blog readers to my Zeek in Action YouTube video series for the Zeek network security monitoring project .  Each video addresses a topic that I think might be of interest to people trying to understand their network using Zeek and adjacent tools and approaches, like Suricata, Wireshark, and so on.  I am especially pleased with Video 6 on monitoring wireless networks . It took me several weeks to research material for this video. I had to buy new hardware and experiment with a Linux distro that I had not used before -- Parrot .  Please like and subscribe, and let me know if there is a topic you think might make a good video.
Read more

MITRE ATT&CK Tactics Are Not Tactics

Image
Just what are "tactics"? Introduction MITRE ATT&CK  is a great resource, but something about it has bothered me since I first heard about it several years ago. It's a minor point, but I wanted to document it in case it confuses anyone else. The MITRE ATT&CK Design and Philosophy document from March 2020 says the following: At a high-level, ATT&CK is a behavioral model that consists of the following core components: • Tactics, denoting short-term, tactical adversary goals during an attack; • Techniques, describing the means by which adversaries achieve tactical goals; • Sub-techniques, describing more specific means by which adversaries achieve tactical goals at a lower level than techniques; and • Documented adversary usage of techniques, their procedures, and other metadata. My concern is with MITRE's definition of "tactics" as "short-term, tactical adversary goals during an attack," which is oddly recursive. The key word in the tacti
Read more

New Book! The Best of TaoSecurity Blog, Volume 4

Image
  I've completed the TaoSecurity Blog book series . The new book is  The Best of TaoSecurity Blog, Volume 4: Beyond the Blog with Articles, Testimony, and Scholarship .  It's available now for Kindle , and I'm working on the print edition.  I'm running a 50% off promo on Volumes 1-3 on Kindle through midnight 20 April. Take advantage before the prices go back up. I described the new title thus: Go beyond TaoSecurity Blog with this new volume from author Richard Bejtlich. In the first three volumes of the series, Mr. Bejtlich selected and republished the very best entries from 18 years of writing and over 18 million blog views, along with commentaries and additional material.  In this title, Mr. Bejtlich collects material that has not been published elsewhere, including articles that are no longer available or are stored in assorted digital or physical archives. Volume 4 offers early white papers that Mr. Bejtlich wrote as a network defender, either for technical or pol
Read more