Thursday, June 16, 2005

(ISC)2 Conducting CISSP Exam Survey

Last month I reported a friend's experiences with the CISSP exam. This week I received an email from (ISC)2 regarding a survey of the CISSP exam. It reads in part:

"(ISC)2 would like to extend to you the opportunity to provide key input into the content of the CISSP® examination. With assistance from Schroeder Measurement Technologies, Inc., (ISC)2’s services entity,(ISC)2 is conducting a CISSP job analysis study through an online survey. The purpose of the job analysis study is to ensure the currency of future CISSP examinations.

As a CISSP certificate holder, we are asking you to participate in the survey. *Your responses are valued and essential*. We ask that you set aside 20 to 30 minutes of your time no later than Thursday, July 14, 2005 to complete the online survey."

Once I started taking the survey, I saw these guidelines.

"A comprehensive list of important job tasks performed by an Information Systems Security Professional is presented on the following pages. Please provide your ratings to the tasks in relation to the practice of Information Systems Security Professionals at your work site."

I was initially excited by the prospect of ISC2 using survey results to revamp the terrible CISSP exam... until I started looking at the survey. Here are a few screen captures. To the right of each item are radio buttons saying "Not Performed, Of No Importance, Of Little Importance, Moderately Important, Very Important, Extremely Important."



This first section presumably asks if these technologies are important. Is this the way an exam should be written? The next screen shot is even worse.



What am I supposed to do here, say a Value Added Network (VAN?) is "Moderately Important" while a hub is "Of Little Importance"?

I looked at one more section, shown below, before giving up.



This survey is a disaster. The CISSP certification should be about security principles. ISC2 should take a look at a wonderful book like Ross Anderson's Security Engineering to figure out what matters. Asking me about hubs or CHAP or the PSTN is foolish. Whatever results ISC2 thinks it gets from this survey will not improve the certification. Again, the only value CISSP retains is its Code of Ethics.

11 comments:

Martin said...

Richard,

You ever listen in on the CISSP mailing list? From the rumblings there and the limited response from ISC^2 management, it looks like things are probably going to get worse before they get better. The Powers that Be at ISC^2 just don't get it. And I don't know why. I can only guess that they've become more concerned with making money than they have with supporting the 'Gold Standard' of security.

Martin

Marcus Ranum said...

Worse, yet, anyone who's studied even introductory statistics or taken an undergrad-level psychology course on testing knows that basing decisions on the input from a survey is extremely dangerous unless you're very careful to avoid sampling bias. In this example, the survey is going out to - what? "some bunch of people." And the responses will be a mixture of "responses from those who had the time to respond" or "responses from those who had an agenda they wanted to push" etc. One wonders if they track who they sent surveys to, or if they're just going to collect results and tabulate them and rely on them. Wonder what'd happen if a ferocious lobby of UNIX weenies decided to all reply and completely bias the survey?

This is a very unreliable way of getting crucial input and it really makes me wonder whether the people building a test from this survey are qualified to build any tests at all - apparently stats 101 or intro psychology aren't useful to a CISSP. ;)

mjr.

Ronaldo C Vasconcellos said...

Good points from both Richard Bejtlich and Marcus Ranum - and I couldn't imagine something different from these guys :-)

Day by day I'm running away from this CISSP thing...

-Ronaldo

Anonymous said...

It seems they are now driven by making as much money as possible from their training business instead of doing what they should have been doing in the first place: run a certification body that is fully opened about their practices and totally neutral.

When the training side of your business has taken over the management side, it is too late.

Stop fighting in the training market and start doing your job in the psychometric market. Start listening to your constituents because what they are telling you today is what will kill you in the future if you do not listen and act now.

It is about time the old boys club that calls themselves "The Board" gets dismantled and some people who really care about the value of the certifications gets in place.

This is not about to change considering that nomination for the board has to be approved by the board.

Mr. Disappointed

Anderson Ramos said...

Unfortunately, you don't know what is a job analysis, so you don't have any idea about how this survey will influence the CISSP certification exam. Get informed before publishing things on your website. Read ISO 17024 to understand exactly how this is used. Don't talk about the procedures used by the (ISC)² if you don't know then.
By the way, if you really believe that an Infosec Pro doesn't need to know what's CHAP, to understand the security implications behind this particular protocol and how this influences the security that it provides, I think you're really mistaken. That's not the only sort of thing that the professional must know, but obviously is something important.

Anonymous said...

This job task analysis is just another excuse from ISC2 to change the content of the CBK which will make anyone else who provides books and training outdated.

If they would be transparent, they would publicly open their CBK and they would not distribute what they call a study guide????

This study guide of theirs is totally inadequate. It is a series of high level bullets that does not help a potential candidate to the exam.

Anonymous said...

Please feel free to contact (ISC)2 with any comments you may have. The comments that started this discussion about the survey are completely misunderstood. The survey was asking what are YOUR tasks - sure hubs and routers are important - so are mainframes - which ones do you work on? Not everyone works on everything - we need a sense of what to include and what not to include in the CBK. It is noteworthy that the CISSP exam process is the only one in our market today that has received ANSI accreditation - which indicates the integrity and skill used in preparing the examination and making it meaningful for all parties.
I invite your replies as well. khenry@isc2.org

Richard Bejtlich said...

khenry@isc2.org,

Your idea of how to construct a useful survey is "completely misunderstood." Your goal may be noble but your implementation will not yield the results you want.

Anderson Ramos,

I know full well what is involved with job analysis. I am part of the BSD Certification Group which published a Task Analysis Survey and the results of that survey.

You should "get informed before publishing things" on my Web site.

Anonymous said...

i don't know who you guys bribbed to get the CISSP ANSI accreddited. it's a poorly written and implemented exam.

$500 US for a dinky scantron test. thank's for the memories.

Anonymous said...
This comment has been removed by a blog administrator.
Manikandan said...
This comment has been removed by a blog administrator.