Thursday, January 10, 2008

How can a blog reader find competent operations personnel?

I received the following question from a blog reader. I am interested in hearing what you think.

I'm team lead for a small private-sector security operations team. We are fortunate that we have a reasonably interesting and attractive work environment, readily available financial resources, and a relatively manageable event load.

We've been trying to hire a mid to senior level analyst position for at least a year now, and have been having absolutely no luck whatsoever.

The job responsibilities mainly consist of analyzing events from the SEM and NSM stacks, documenting and resolving incidents, and conducting regular vulnerability management operations.

A majority of the applications we get seem to come from security "architects" who may have some product deployment experience, but little to no applicative analysis skills necessary to un-haystack the needles, or pursue an incident to closure.

Very few of the interviewees can even get past the technical phone screen, which consists of the following three questions:

  1. You see an IDS/IPS event in your event console called "some kind of IDS event name here".

    • What would you do to investigate the event, and how would you validate that the event was a real attack and not a false positive?

    • How would you determine if this was a one-off event, or part of an overall pattern?

    • What other kinds of information would you seek out to build a more complete picture of the context around this event?


  2. After having investigated the event, you have gathered enough positive indicators that the actual traffic consisted of a legitimate attack against a server you suspect may be vulnerable to an an attack of that kind.


    • How do you determine what may have happened to the server? (This question is usually geared towards whatever platform the candidate might have actual technical experience with.)

    • What would you do if you saw a subsequent event that indicated the target system had downloaded a file from the internet soon after the original IDS event?

    • How could you recover the file? What would you do to analyze it? (This question usually evolves into some platform-specific live forensics, network forensics, and incident response.)


  3. You conduct a vulnerability scan that produces output that indicates that a server X (operating system Y) may be vulnerable to issue Z.


    • What would you do to validate the finding?

    • How would you validate the finding if the report indicated the issue was present on 100 machines? (This again is usually geared towards a platform that the candidate has the most experience with).

    • What would you do to address the issue?



These three topic areas seem to cut to the core of what raw analysis tasks an operations analyst must be able to perform well. The kinds of answers I expect are specific, detailed, and accurate given the scenarios supplied (i.e. application-level attack against a 3-tier windows-based web application merits one kind of response vs. a client-side buffer overflow attack against a web browser, etc.).

Maybe one or two of our candidates out of several dozen have even been able to answer them competently enough for a second round (and they eventually accepted more lucrative offers). I'd even be happy if the candidates could get two out of three.

Am I setting the bar too high? Are there some magic keywords in the job req that I'm missing? Am I going to have hire juniors and train them up? Is there even such a thing as a senior operations analyst?


My initial response is that the number of people who can independently and competently answer these questions is remarkably small. Furthermore, the number of shops that are collecting the data necessary to answer these questions is also small.

What do blog readers think?

30 comments:

Anonymous said...

As Richard said, that skill set is rare and commands a premium in todays marketplace.

"The job responsibilities mainly consist of analyzing events from the SEM and NSM stacks, documenting and resolving incidents, and conducting regular vulnerability management operations."

During my life as a consultant (which I left a year ago) I only saw such positions filled at very large companies and government or military agencies.

Why it is rare and what that says about the state of security in businesses of all sizes is left as an exercise to the reader.

"Maybe one or two of our candidates out of several dozen have even been able to answer them competently enough for a second round (and they eventually accepted more lucrative offers)."

Can you afford to fill this position? Here in the Upper Great Plains of the US someone who can do what you ask either commands >$100,000/yr or they are running (or starting up) their own security management service.

I think your best option would be to groom junior analysts. Of course, that is a Catch-22 situation since you probably don't have anyone to groom them. Probably the biggest benefit of grooming a junior into a senior is you can better manage their salary. Promoting a junior analyst making $50,000 to a senior making $75,000 saves you $25,000 over hiring at $100,000.

shadow said...

I could answer all of the questions. I don't really think it is all too complicated. I've had a few years experience with penetration testing and intrusion analysis. I don't think you need a "senior" analyst. You need someone who is technically adept and can adapt to new situations. I'd be happy to interview with you as I think those are really good questions. Feel free to contact me at onebitcipher at g mail dot com.

David Bianco said...

It's hard to recruit well-trained and experienced technical security talent, especially if you can't afford salaries closer to the top of the range. I don't really have a solution, but I'd like to offer props to your reader for such a great set of questions. The mere ability to ask those is a good sign.

Anonymous said...

Excellent post Richard. I think you hit the nail on-the-head with your short response. The amount of candidates that can answer these questions is few and far between. Most of the people who can answer them are already making good money in nice positions. On the other hand some people with these skills aren't always very articulate and might feel better being sent a pcap and told to answer some questions about it in order to prove some knowledge. I have interviewed people like this before. Its not uncommon. Ill spare you my "this is because most candidates are security paper pushers" rant for today.

Anonymous said...

I am in the middle of interviewing process at the moment and here are a couple of my observations:

Most companies are looking at bringing in junior people and grooming them. I think that is both a good and bad thing. Its great that additional people are being brought into the industry and getting experience. Its bad that many of these organizations are weighed down by too many junior level people and there are not enough mentors to go around. What I have also observed is that many companies are advertising for very senior people but the pay rate is so low its laughable. This could be lack of experience with looking for these types of positions or it could be that they are trying to take advantage of an uncertain market right now.

My second comment is I didn't view the questions posed as overly technical. They are more process related questions (with an exception or two.) And how you answer them is as important as what the answers you give are.

John Ward said...

Not to be too snide, but the questions are a little on the vague side. They need to be more specific with the capabilities of their system. Does the IDS/IPS setup they have do full content capture, or just statistical capture. In they do a full content capture, then answering the investigative and forensics questions would be simple. What are the corporate policies? Can the machine be allowed downtime, are there backups, etc. Is there any sensitive data, is drive cloning allowed? Are there any clean backups or images of the machine to restore during the investigative proess to minimize downtime. While not necessarily technical questions, these are policy things that come up in any environment where downtime is more of a risk to PHB types than the damage from an intrusion. Maybe that is the point of the interview, looking for someone who will assess the capabilities of the existing system. Otherwise, I would just say if their candidates cannot answer these, then it may be a sign that they aren't offering the right salary for the skill set or that there is just not enough qualified candidates out there. This is definitely the case in the programming world, where "programmers" are a dime a dozen, but competent programmers are a little in short supply.

Anonymous said...

I agree with John Ward: the questions are too vague. When trying to answer them in my head I only came up with a dozen more for each, which would end up in a 3-hr phone conversation (which might be fine...). Maybe the questions should be more exploratory in testing the candidate's agility and ability to assess situations correctly (which in my mind is what a good sec manager must have)

Steve said...

Could I get in contact with them? I could pass the phone screen and am kind of bored where I'm at. Interesting opportunity... Maybe just a link to their monster/dice/craigslist/whatever ad...

Anonymous said...

As the anonymous blog reader in question, the three questions merely represent a framework in which to assess the candidates skills. We usually fill in the 'signature name here' and 'server'/'platform' stuff with specific items and scenarios lifted from the candidate's resume. It serves the dual purpose of measuring their own analytical inquisitiveness, as well as their competency within their stated fields of expertise. The vagueness is intentional because it is typical of what an analyst will actually see on an alarm/event browser - the who, what, and where are not immediately available, and it is up to the analyst to determine which subsystem picked up the event, what part of the network (perimeter/core) it was on, whether full content and session data is available, etc. The response portion of the questions is also similarly vague for the same reasons - will the candidate immediately disconnect the system (as at least 50% of the candidates have stated as their first recourse), or will they refer to documented IR policies and attempt minimally invasive live forensics? Both kinds of answers are very indicative of the overall mindset and operational experience of the candidate within an enterprise. Likewise, someone who focuses exclusively on policies and procedures may not have any technical investigative abilities whatsoever, and may be similarly unqualified. Such is the conundrum.

MarvinK said...

It seems pretty straightforward:

Persons who could answer the questions took better-paying jobs.

You're going to either need to come up with questions that are more specific--if you're more interested in technical skills than communication skills, or you're going to have to pay more. Technical security people are expensive. Semi-technical people with excellent communication skills often demand even higher salaries. Excellent communication and technical skills are difficult to find--and cost plenty.

Anonymous said...

If it's taking you more than a year to fill a position, then you're not being realistic with the salary you're offering. Cough up the extra dough or go with a more junior candidate and pay for his training.

LaRoach said...

As others have said, it looks like a rather specialized skill set. They probably need to look for a very competent *trainable* candidate and home grow what they need. Word of warning though, don't become a farming operation. Once they are trained in make sure they want to stay with pay and benefits to match the position. All too often people are brought in at a lower pay grade, trained in, and then kept at the lower skill pay grade. Once they realize they can go elsewhere (and if they are smart cookie you are looking for they will) for a lot more cash your investment is toast.

LonerVamp said...

Blogger comments seem to be wonky today, so I put my thoughts on my own blog (http://www.terminal23.net). The short version is that I like the questions. They go after what you're looking for; don't change them. But are you (or the company) properly valuing the position and looking for candidates in the right places? Also, evaluate the need for "senior" in the title, or if you'd be willing to groom a person or two in lieu of a senior one.

Anonymous said...

As a senior IDS analyst working at a large MSSP we ask these sort of hiring questions all the time. It _is_ difficult to find competent people but I think it may be even harder to find a good security analyst who can also conduct a proper vulnerability scan and server side forensics. You may need to split up the job duties in order to fill the positions.

Anonymous said...

this position would not happen to be in the utility sector would it?

Anonymous said...

Since I do this sort of thing for a living, I'll chime in. The questions don't seem vague at all to me. In fact they're the framework around which any investigation begins, proceeds and completes. I think the problem is that there are not a lot of people who have the kind of experience you're looking for. Not everyone gets the keys to the kingdom so they can poke around in all the right places and get the kind of experience that can answer those questions.

I'm not going to assume you're not offering enough for the position, because you said "mid to senior". This indicates you have a range in mind based upon the candidate's qualifications, experience and the interview. Perhaps part of the problem might be location? Or the benefits package you have to offer? Stability of the company? Hours? (If it's a 24/7 operation, that could certainly be an issue for some.)

I think, in general, you have two problems. A small pool of potential candidates, most of whom are quite happy where they are, and something about the job offer itself that isn't overwhelmingly attractive. Without knowing more, it's hard to say for sure.

I wonder if you've fished around in the edu space? There are some wonderfully talented people there who don't generally go looking for work but might be tempted by the right situation.

Wayne Anderson said...

I have to agree that the questions provided here are NOT vague at all, so long as the initial mode of answers are intended to look for framework responses.

Once you have established the framework of the response, you can always narrow to specific actions within the framework. For example, in the case of number 2, you might start with something as general as "Open security response policy". I guarantee you that the document referred to is different for every single enterprise but I also guarantee you that most interviewees wont think of that most basic step. You can always move into things like a step like "if the potentially compromised node is part of a high availibility array, drop the node from the cluster/NLB stack and isolate it from the production network".

Those kinds of more specific stuff may show your technical brownie points but dont show the interviewer that you understand WHERE your technical action is fitting into the response stack.

Recently, I did some test cleanup for an online role assessment company. Honestly, I came away from the experience rather disappointed in that the beta testing pool whose comments and responses I was using to assist in item cleanup really were offbase probably 70% of the time or more. Just straight up, factually wrong. It makes you wonder how many "90 day wonders" (as my grandfather calls them) are really out there passing themselves off as security professionals.

In this specific situation, I am not at all surprised. As a trainer in the industry with a consulting background in messaging and security, I can tell you that my experience with other trainers and organizations in the industry is that there hasnt been the real demand in the private sector for security training. Its expensive. Its time consuming. Managers have limited funds and lets face it, getting your engineer trained on a security course or that nice shiny email course... well.... the email course includes something on security.... right?

FInding qualified professionals that understand something at a higher level than the technical implementation of the concepts is difficult. Finding someone who can do both the conceptual and the implementation? Even more so.

Make sure your salary and benefits offerings are realistic in this most compettiive sector of the IT industry.

Dont despair, however. DoD 8570 means that a huge workforce in the defense space will soon be forced to train both formally and on the job in security concepts.

marklar said...

Watching an IDS console is a pretty crap job for a senior tech/analyst, although they're exactly what you need to do incident response properly. A good senior will need to be rewarded (not just salary but conditions too) and encouraged to stay long enough to develop your juniors. Burn-out rate can be high if you have 'difficult' customers.

I've been interviewed for this type of role by several companies that are in this space, most of them went overboard with 2nd and 3rd interviews, behavioral interviews, etc. and simply took too long (months) to get back to me with ANY feedback. I suspect they were waiting to sign new contracts with their customers, but it certainly makes applicants realise what might happen if the customers start dropping off.

shadow said...

I've noticed a lot of companies out there that are hiring for security positions and pass up good candidates are stuck in the paradigm of believing years of experience equals competence. This is usually not the case. In fact, I would say that if you're doing the same job for 10 years you are most likely not competent enough to move up to a higher position. This kind of mind set could possibly be attributing to the inability to fill this position.

Just my 2 cents.

Anonymous said...

"shadow", if you're technically proficient and uninterested in a management position, then where do you go after you've "mastered" (as if anyone ever really does) the basics of security?

I handle IDS/IPS, investigations (on all platforms/OSes) and forensics (among other things), and I *really* enjoy what I do. (I love the thrill of the hunt.) I'm completely uninterested in getting enmeshed in the politics and bs and endless meetings and reports of management and I have no desire to move out of the security profession.

So am I incompetent to move up? And where *is* up, exactly? I'm the senior analyst. My pay rate is higher than anyone else in the department. My opinions are highly regarded by my ISO. I am constantly consulted on security issues, quoted in the press and interviewed on tv.

Yet, according to you, I'm chopped liver.

That's an interesting perspective, to say the least.

Anonymous said...

After reading the second paragraph I commented out loud, "You are not paying enough."

The person confirmed this at the end of their query when they mentioned that those who pass to the second round of interviews eventually accept more lucrative offers.

Offer more money. Period. If you can't do that, you will have to settle for a less skilled individual that you could "train up". Just be ready for them to jump ship if you can't keep up with raises and bonuses. Hate to say it, but specialized skills = higher pay.

As far as the questions, I found them to be much too vague. I've been asked question like this before and they usually require a dozen or more clarifying questions from the interviewee. When I have answered questions like this, I often have to make a lot of assumptions about the environment so that usually leads to an answer that fails to satisfy either the interviewer or interviewee.

David Bianco said...

I'm a little amazed at all the comments saying that the questions are too vague, and require too many additional follow-up questions by the candidate in order to figure out the answer.

Firstly, these questions are not vague. They're almost the exact same questions I ask myself every time I view a Sguil alert. Not that I start over from 0% every time, but every alert is unique and depending on the alert, the host, the network or other factors, I may have a different set of resources available to me.

On top of that, I wonder why people think it's bad to elicit a bunch of follow-up questions from the candidate? I don't need someone who can spew the correct answer. I want someone who can think, reason and show some understanding of the incident investigation process. The best way to know if he or she fits the bill is to listen to the questions they ask, and then see if they arrive at a (not the) correct answer.

John Ward said...

David,

Your comment hits on the point. The fact that you mention a Sguil alert narrows down what you have available. You know what tool you have, and what data is available, none of this was given int he sample interview questions. In Sguil, you at the very least session data. If you have the disk space, you probably also have the full content of the session. So the investigative process from a network standpoint is fairly straight forward. The interview questions don't address this, and from what I gathered, are simply platform based, leaving host based monitoring. Maybe the point of the interview is to get the potential candidate's to ask these kinds of questions to get an idea of their sphere of knowledge. I still believe that if the folks who can answer these kinds of questions aren't sticking around, then its a money thing.

Anonymous said...

It's not always about money. There was a time when I absolutely refused to take a job more than fifteen minutes drive from my house. Even the direction mattered - I refused to have to drive in the same direction as the bulk of traffic.

It all depends on the individual.

I thought it was interesting that you seized on David's post to buttress the argument about the vagueness of the answers. David's post was based upon the question. I suspect that's exactly what the questions are designed to do - elicit knowledge from the interviewee - what are you familiar with - what tools have you used - what's your approach to solving these problems.

Roman said...

In regards to the questions being too vague, I don't really see this as a flaw. If I were being asked these questions, I would ask my own questions to gain the specifics of the system. They are looking for analysis after all, not 'how do I patch/exploit this vulnerability on Windows/Linux/etc?'. It's up to the candidate analyst to narrow down the problem set.

Michael Dundas said...

"Anonymous said .... Since I do this sort of thing for a living, I'll chime in. .."

I agree. I have done this job for several years, then managed the team that did this work, and recently switched positions to handle our more strategic clients. I have been at my current company for 4 years. They are a great company, they pay very well compared to others, they offer excellent benefits package, are very understanding of personal time, I work with great people, ... I could go on. If someone was going to try and entice myself to change companies, there would have to be a really great upside beyond what I currently have. I actually worry that if something ever happened that required me to leave I would have difficult time finding something comparable.

The three questions you listed above you stated that most do not get past these questions on the telephone interview. Maybe it is where you are advertising the position? They are basic questions anyone with experience in this type of work should be able to answer easily.

-mike.

John Ward said...

Not targeting David specifically. I just named him he was the last one to comment, and since I've worked with him on something in the past I know he won't take offense to anything I say.

Anonymous said...

Makes for an interesting conversation, there would be a lot of mind reading going on here. While I would like to find people who can do good security work, most of the questions are not hard, but are hard and are going to be somewhat specific to the operating systems and environments that someone comes from. I know I would also be asking a lot of questions about the questions, trying to nail down more specifics.

There is also the "interview burden" here, if the questions wander when they are being asked, then there are going to be more troubles than accuracy. If the person gets bored then you have the wrong candidate.

Many people in many organizations hve not seen industrial scope information security, you are hitting against culture, and performance, not all will pass. what is interesting is that there are no people questions, the focus on technology alone is generally a bad thing, you want to see how they think as well, and if they have the skills to deal with people.

PaulM said...

I'm definitely late to the party, but this topic is near and dear to my heart.

The bottom line is that SIM is new and it changes the way you do security ops. And some of the process and procedure paradigms that come out of a SecOps program are going to be driven by the product you use.

The bottom line is that you hire intelligent, mentally flexible people with an appetite for security and train them up. It's faster and cheaper than trying to snipe experienced security ops people.

Xenophon Fenderson, the Carbon(d)ated said...

This particular blog reader is very excited to discover three very good questions to add to his Technical Interview of Doom. (DOOOOOM!!) ;) At this point, I'd be happy with even partial responses to any of the questions, but I'm finding that "analysts" are generally only capable of running canned reports and that they cannot easily interpret the results of those reports as a result. (Hell, that's as much the fault of the tools as of the staff, but I'll rant and rave on my own blog.)