Brief Thoughts on MJR Pen Testing Post
I learned of this post by Marcus Ranum through commentary by Dave Goldsmith. In brief, I agree with much of what MJR says. However, I think pen testers perform a valuable service. I do not think that it is possible for some modern enterprise code to be fully comprehended by any individual or team of developers or security engineers.
If the code cannot be fully understood statically, it must be tested dynamically. A live test will reveal how the system acts when working, and may reveal unanticipated interactions or vulnerabilities. In light of this fact, I think pen testers who unearth these flaws perform a valuable service. If it's not tested, it's not a service.
Update: Thanks to Tom's comment below, I changed the attribution to fellow Matasano poster Dave Goldsmith.
If the code cannot be fully understood statically, it must be tested dynamically. A live test will reveal how the system acts when working, and may reveal unanticipated interactions or vulnerabilities. In light of this fact, I think pen testers who unearth these flaws perform a valuable service. If it's not tested, it's not a service.
Update: Thanks to Tom's comment below, I changed the attribution to fellow Matasano poster Dave Goldsmith.
Comments
sadder: the professional breakers of things have to get so ruffled when someone points out (quite correctly IMO) that much of the work they do could (and should) in fact be part of an automated QA process.
Is "a bucket of warm spit" worth more, or less, than, say, a bucket of cold spit? The former is at least fresh (or recently re-heated); the latter is somewhat aged. :)
If we assume that certain parts of what gets lumped into the process of "pen testing" can be automated, most especially the parts of banging on the network connections to see how the product reacts, it would seem to me that would become a very valuable QA process. Lets say that for now, this is best done by a trained human who can interpret the results. However the same thing could once be said of most of what can now be done by people with access to automated scanners, etc.
If such a QA tool is produced within the framework of a commonly used set of QA tools then businesses which offer more advanced services in the way of pentesting could differentiate themselves.
What's interesting about this? What do you think engineers do? They pen test with iron, steel, concrete, etc. When a foundation to a building is poured they do tests on the concrete to ensure that it's the right consistency and can withstand the load properly. Bridges are tested in wind tunnels, loads are calcualted, etc. Voila, pen testers!
Very funny if you ask me, very very funny!
I speak as someone who hires pentesters, and respects the ones who are 100% ethical and understand the concept of trespassing.
I am of the opinion though that a majority of the people in the profession loosely described as pentesting do not have the background, experience, etc. to justify their work not being automated. In the 10 or so years I've been in and around itsec in various positions, I've seen alot of work done under the auspices of pentesting that ammounted to little more than button clicking. IMO this includes many "specialized" applications intended to break other bits of software.
It seems we agree on the value of improving the state of the industry and building more security testing into QA. We may not see eye to eye on how much of the work is automatable, but thats another matter.