Brief Thoughts on MJR Pen Testing Post

I learned of this post by Marcus Ranum through commentary by Dave Goldsmith. In brief, I agree with much of what MJR says. However, I think pen testers perform a valuable service. I do not think that it is possible for some modern enterprise code to be fully comprehended by any individual or team of developers or security engineers.

If the code cannot be fully understood statically, it must be tested dynamically. A live test will reveal how the system acts when working, and may reveal unanticipated interactions or vulnerabilities. In light of this fact, I think pen testers who unearth these flaws perform a valuable service. If it's not tested, it's not a service.

Update: Thanks to Tom's comment below, I changed the attribution to fellow Matasano poster Dave Goldsmith.

Comments

Anonymous said…
Oh now I get it - pentesters - pens that test real euro currency. That's bad..real bad...
Anonymous said…
sad: OK the guys who are professional breakers of things of course deserve credit, credit which is limited since so far we dont have a way to do something like a UL/materials sheet for things on a network.

sadder: the professional breakers of things have to get so ruffled when someone points out (quite correctly IMO) that much of the work they do could (and should) in fact be part of an automated QA process.
Anonymous said…
I'm having a slight challenge understanding the value system...

Is "a bucket of warm spit" worth more, or less, than, say, a bucket of cold spit? The former is at least fresh (or recently re-heated); the latter is somewhat aged. :)
I should have pointed out that I think the "spit" comment is unfair. I would reserve terms like that for people like "0x80" and not other security professionals. I also agree that it is not trivial to perform pen testing and I respect those that do.
Anonymous said…
tqbf: What MJR undiplomatically said isnt the interesting point unless you are in a position to have your feathers ruffled. The actual words are of far less interest to me than the idea that some of what is now mostly a post shipping "process" of discovery by whomever could or should be a pre-ship QA process.

If we assume that certain parts of what gets lumped into the process of "pen testing" can be automated, most especially the parts of banging on the network connections to see how the product reacts, it would seem to me that would become a very valuable QA process. Lets say that for now, this is best done by a trained human who can interpret the results. However the same thing could once be said of most of what can now be done by people with access to automated scanners, etc.

If such a QA tool is produced within the framework of a commonly used set of QA tools then businesses which offer more advanced services in the way of pentesting could differentiate themselves.
Anonymous said…
It's interesting that mjr is described as an engineer that "knows it all", and then he goes on in this post to deride pen testers.

What's interesting about this? What do you think engineers do? They pen test with iron, steel, concrete, etc. When a foundation to a building is poured they do tests on the concrete to ensure that it's the right consistency and can withstand the load properly. Bridges are tested in wind tunnels, loads are calcualted, etc. Voila, pen testers!

Very funny if you ask me, very very funny!
Anonymous said…
I also disagree that pentesters aren't valuable. The best ones don't just check for code bugs: they try to find weaknesses in combinations of code, business processes, and human fallibility. A software engineer is, in many cases, not going to be able to cover all those viewpoints with the creativity and knowledge necessary to test a complex system. I don't believe security testing is ever going to be a pure QA matter -- not as long as security also rests in the hands of the people who make the rules and then break them.

I speak as someone who hires pentesters, and respects the ones who are 100% ethical and understand the concept of trespassing.
Anonymous said…
tqbf: thanks for your reasonable response. I should have made it more clear that I did previously understand the position of your business and your history. No disrespect intended to you at all.

I am of the opinion though that a majority of the people in the profession loosely described as pentesting do not have the background, experience, etc. to justify their work not being automated. In the 10 or so years I've been in and around itsec in various positions, I've seen alot of work done under the auspices of pentesting that ammounted to little more than button clicking. IMO this includes many "specialized" applications intended to break other bits of software.

It seems we agree on the value of improving the state of the industry and building more security testing into QA. We may not see eye to eye on how much of the work is automatable, but thats another matter.

Popular posts from this blog

Zeek in Action Videos

New Book! The Best of TaoSecurity Blog, Volume 4

MITRE ATT&CK Tactics Are Not Tactics