Tuesday, February 21, 2006

Brief Thoughts on MJR Pen Testing Post

I learned of this post by Marcus Ranum through commentary by Dave Goldsmith. In brief, I agree with much of what MJR says. However, I think pen testers perform a valuable service. I do not think that it is possible for some modern enterprise code to be fully comprehended by any individual or team of developers or security engineers.

If the code cannot be fully understood statically, it must be tested dynamically. A live test will reveal how the system acts when working, and may reveal unanticipated interactions or vulnerabilities. In light of this fact, I think pen testers who unearth these flaws perform a valuable service. If it's not tested, it's not a service.

Update: Thanks to Tom's comment below, I changed the attribution to fellow Matasano poster Dave Goldsmith.


tqbf said...

Fair warning: the chargen is a group blog. Window, Dave, Jeremy, and Dino post too. In this case, the post is Dave's.

My thoughts on Ranum's post are more involved and less forgiving. I like MJR, but he frustrates the hell out of me. I agree with Dave though: the original Bugtraq poster is clearly a Jerry Springer guest in waiting, and MJR is not working hard to avoid falling into the "Springer Show Surprise Guest: Your Jilted Ex-Husband!" mold himself.

I speak for the team when I say that we are, as a group, tired of hearing that it takes less talent to break something than to build it. We do both. But I encourage MJR to tell that to Paul Kocher, the late Hans Dobbertin, or Xiaoyun Wang. It's a lot easier to design something crappy than it is to break something good.

Anonymous said...

Oh now I get it - pentesters - pens that test real euro currency. That's bad..real bad...

David Belle-Isle said...

Actually that "mjr" has the perfect engineer mentality: stubborn and i-know-everything attitude. Those people are a real pain to work with.


Chris_B said...

sad: OK the guys who are professional breakers of things of course deserve credit, credit which is limited since so far we dont have a way to do something like a UL/materials sheet for things on a network.

sadder: the professional breakers of things have to get so ruffled when someone points out (quite correctly IMO) that much of the work they do could (and should) in fact be part of an automated QA process.

Joshua said...

I'm having a slight challenge understanding the value system...

Is "a bucket of warm spit" worth more, or less, than, say, a bucket of cold spit? The former is at least fresh (or recently re-heated); the latter is somewhat aged. :)

tqbf said...

MJR didn't "point out" that security testing could be built in to QA process (and what part of security testing do you believe should be automated? how much benefit do you believe would accrue relative to the size of the security problem overall?).

MJR said "penetration testers" aren't "worth a bucket of warm spit".

I'm not sure it's within bounds of reasonable discussion to suggest that it's the pentesters who are getting "ruffled" here.

It should also be made clear that the original thread to which MJR responded isn't about the value of pentesting. It's about something far dumber than that.

Richard Bejtlich said...

I should have pointed out that I think the "spit" comment is unfair. I would reserve terms like that for people like "0x80" and not other security professionals. I also agree that it is not trivial to perform pen testing and I respect those that do.

Chris_B said...

tqbf: What MJR undiplomatically said isnt the interesting point unless you are in a position to have your feathers ruffled. The actual words are of far less interest to me than the idea that some of what is now mostly a post shipping "process" of discovery by whomever could or should be a pre-ship QA process.

If we assume that certain parts of what gets lumped into the process of "pen testing" can be automated, most especially the parts of banging on the network connections to see how the product reacts, it would seem to me that would become a very valuable QA process. Lets say that for now, this is best done by a trained human who can interpret the results. However the same thing could once be said of most of what can now be done by people with access to automated scanners, etc.

If such a QA tool is produced within the framework of a commonly used set of QA tools then businesses which offer more advanced services in the way of pentesting could differentiate themselves.

tqbf said...

I think it's obvious to most people in the industry that there is an unfilled niche for automated security testing in product development. There's a discussion on DailyDave swirling around fuzzers in test-driven development in the wake of the LDAP findings.

I'm responding to you because I think you're making a hasty generalization. The fact that there are uses for automated tools, run by generalist software developers, has nothing to do with MJR's argument or the argument over third-party pen-testing in general.

I'll allow that you're not trying to wade into the argument about whether pen-testers are immoral or "failed security analysts", point out that your original comment was not germane to MJR's point or my own, and agree that QA is valuable.

Then, turning my attention directly to your argument: what portion of security testing do you believe is automatable? How much benefit do you believe would accrue from industry-wide adoption of such tools relative to the overall security problem? Do you know, or do you just have an intuition that many of our problems are simplistic and should be easy to catch?

I ask because the majority of our findings are not fuzzer results or overflows; looking over the queue of advisories we've got lined up, I see very few problems that could be caught by any automated tool I could reasonably conceive of. So while I want to encourage better vendor exit testing and have a lot of respect for QA, I'm uncomfortable with the argument that better QA would make a dent in the problem.

For the record I want to point out that I'm a professional builder, contracting as a professional breaker while we bootstrap a first rev of a product that has nothing to do with software testing or individual vulnerabilities. But you can see which side of the pentester vs. developer debate has my sympathy.

David said...

It's interesting that mjr is described as an engineer that "knows it all", and then he goes on in this post to deride pen testers.

What's interesting about this? What do you think engineers do? They pen test with iron, steel, concrete, etc. When a foundation to a building is poured they do tests on the concrete to ensure that it's the right consistency and can withstand the load properly. Bridges are tested in wind tunnels, loads are calcualted, etc. Voila, pen testers!

Very funny if you ask me, very very funny!

GM said...

I also disagree that pentesters aren't valuable. The best ones don't just check for code bugs: they try to find weaknesses in combinations of code, business processes, and human fallibility. A software engineer is, in many cases, not going to be able to cover all those viewpoints with the creativity and knowledge necessary to test a complex system. I don't believe security testing is ever going to be a pure QA matter -- not as long as security also rests in the hands of the people who make the rules and then break them.

I speak as someone who hires pentesters, and respects the ones who are 100% ethical and understand the concept of trespassing.

Chris_B said...

tqbf: thanks for your reasonable response. I should have made it more clear that I did previously understand the position of your business and your history. No disrespect intended to you at all.

I am of the opinion though that a majority of the people in the profession loosely described as pentesting do not have the background, experience, etc. to justify their work not being automated. In the 10 or so years I've been in and around itsec in various positions, I've seen alot of work done under the auspices of pentesting that ammounted to little more than button clicking. IMO this includes many "specialized" applications intended to break other bits of software.

It seems we agree on the value of improving the state of the industry and building more security testing into QA. We may not see eye to eye on how much of the work is automatable, but thats another matter.