How can a blog reader find competent operations personnel?
I received the following question from a blog reader. I am interested in hearing what you think.I'm team lead for a small private-sector security operations team. We are fortunate that we have a reasonably interesting and attractive work environment, readily available financial resources, and a relatively manageable event load.
We've been trying to hire a mid to senior level analyst position for at least a year now, and have been having absolutely no luck whatsoever.
The job responsibilities mainly consist of analyzing events from the SEM and NSM stacks, documenting and resolving incidents, and conducting regular vulnerability management operations.
A majority of the applications we get seem to come from security "architects" who may have some product deployment experience, but little to no applicative analysis skills necessary to un-haystack the needles, or pursue an incident to closure.
Very few of the interviewees can even get past the technical phone screen, which consists of the following three questions:
- You see an IDS/IPS event in your event console called "some kind of IDS event name here".
- What would you do to investigate the event, and how would you validate that the event was a real attack and not a false positive?
- How would you determine if this was a one-off event, or part of an overall pattern?
- What other kinds of information would you seek out to build a more complete picture of the context around this event?
- After having investigated the event, you have gathered enough positive indicators that the actual traffic consisted of a legitimate attack against a server you suspect may be vulnerable to an an attack of that kind.
- How do you determine what may have happened to the server? (This question is usually geared towards whatever platform the candidate might have actual technical experience with.)
- What would you do if you saw a subsequent event that indicated the target system had downloaded a file from the internet soon after the original IDS event?
- How could you recover the file? What would you do to analyze it? (This question usually evolves into some platform-specific live forensics, network forensics, and incident response.)
- You conduct a vulnerability scan that produces output that indicates that a server X (operating system Y) may be vulnerable to issue Z.
- What would you do to validate the finding?
- How would you validate the finding if the report indicated the issue was present on 100 machines? (This again is usually geared towards a platform that the candidate has the most experience with).
- What would you do to address the issue?
These three topic areas seem to cut to the core of what raw analysis tasks an operations analyst must be able to perform well. The kinds of answers I expect are specific, detailed, and accurate given the scenarios supplied (i.e. application-level attack against a 3-tier windows-based web application merits one kind of response vs. a client-side buffer overflow attack against a web browser, etc.).
Maybe one or two of our candidates out of several dozen have even been able to answer them competently enough for a second round (and they eventually accepted more lucrative offers). I'd even be happy if the candidates could get two out of three.
Am I setting the bar too high? Are there some magic keywords in the job req that I'm missing? Am I going to have hire juniors and train them up? Is there even such a thing as a senior operations analyst?
My initial response is that the number of people who can independently and competently answer these questions is remarkably small. Furthermore, the number of shops that are collecting the data necessary to answer these questions is also small.
What do blog readers think?
Comments
"The job responsibilities mainly consist of analyzing events from the SEM and NSM stacks, documenting and resolving incidents, and conducting regular vulnerability management operations."
During my life as a consultant (which I left a year ago) I only saw such positions filled at very large companies and government or military agencies.
Why it is rare and what that says about the state of security in businesses of all sizes is left as an exercise to the reader.
"Maybe one or two of our candidates out of several dozen have even been able to answer them competently enough for a second round (and they eventually accepted more lucrative offers)."
Can you afford to fill this position? Here in the Upper Great Plains of the US someone who can do what you ask either commands >$100,000/yr or they are running (or starting up) their own security management service.
I think your best option would be to groom junior analysts. Of course, that is a Catch-22 situation since you probably don't have anyone to groom them. Probably the biggest benefit of grooming a junior into a senior is you can better manage their salary. Promoting a junior analyst making $50,000 to a senior making $75,000 saves you $25,000 over hiring at $100,000.
Most companies are looking at bringing in junior people and grooming them. I think that is both a good and bad thing. Its great that additional people are being brought into the industry and getting experience. Its bad that many of these organizations are weighed down by too many junior level people and there are not enough mentors to go around. What I have also observed is that many companies are advertising for very senior people but the pay rate is so low its laughable. This could be lack of experience with looking for these types of positions or it could be that they are trying to take advantage of an uncertain market right now.
My second comment is I didn't view the questions posed as overly technical. They are more process related questions (with an exception or two.) And how you answer them is as important as what the answers you give are.
Persons who could answer the questions took better-paying jobs.
You're going to either need to come up with questions that are more specific--if you're more interested in technical skills than communication skills, or you're going to have to pay more. Technical security people are expensive. Semi-technical people with excellent communication skills often demand even higher salaries. Excellent communication and technical skills are difficult to find--and cost plenty.
I'm not going to assume you're not offering enough for the position, because you said "mid to senior". This indicates you have a range in mind based upon the candidate's qualifications, experience and the interview. Perhaps part of the problem might be location? Or the benefits package you have to offer? Stability of the company? Hours? (If it's a 24/7 operation, that could certainly be an issue for some.)
I think, in general, you have two problems. A small pool of potential candidates, most of whom are quite happy where they are, and something about the job offer itself that isn't overwhelmingly attractive. Without knowing more, it's hard to say for sure.
I wonder if you've fished around in the edu space? There are some wonderfully talented people there who don't generally go looking for work but might be tempted by the right situation.
Once you have established the framework of the response, you can always narrow to specific actions within the framework. For example, in the case of number 2, you might start with something as general as "Open security response policy". I guarantee you that the document referred to is different for every single enterprise but I also guarantee you that most interviewees wont think of that most basic step. You can always move into things like a step like "if the potentially compromised node is part of a high availibility array, drop the node from the cluster/NLB stack and isolate it from the production network".
Those kinds of more specific stuff may show your technical brownie points but dont show the interviewer that you understand WHERE your technical action is fitting into the response stack.
Recently, I did some test cleanup for an online role assessment company. Honestly, I came away from the experience rather disappointed in that the beta testing pool whose comments and responses I was using to assist in item cleanup really were offbase probably 70% of the time or more. Just straight up, factually wrong. It makes you wonder how many "90 day wonders" (as my grandfather calls them) are really out there passing themselves off as security professionals.
In this specific situation, I am not at all surprised. As a trainer in the industry with a consulting background in messaging and security, I can tell you that my experience with other trainers and organizations in the industry is that there hasnt been the real demand in the private sector for security training. Its expensive. Its time consuming. Managers have limited funds and lets face it, getting your engineer trained on a security course or that nice shiny email course... well.... the email course includes something on security.... right?
FInding qualified professionals that understand something at a higher level than the technical implementation of the concepts is difficult. Finding someone who can do both the conceptual and the implementation? Even more so.
Make sure your salary and benefits offerings are realistic in this most compettiive sector of the IT industry.
Dont despair, however. DoD 8570 means that a huge workforce in the defense space will soon be forced to train both formally and on the job in security concepts.
I've been interviewed for this type of role by several companies that are in this space, most of them went overboard with 2nd and 3rd interviews, behavioral interviews, etc. and simply took too long (months) to get back to me with ANY feedback. I suspect they were waiting to sign new contracts with their customers, but it certainly makes applicants realise what might happen if the customers start dropping off.
Just my 2 cents.
I handle IDS/IPS, investigations (on all platforms/OSes) and forensics (among other things), and I *really* enjoy what I do. (I love the thrill of the hunt.) I'm completely uninterested in getting enmeshed in the politics and bs and endless meetings and reports of management and I have no desire to move out of the security profession.
So am I incompetent to move up? And where *is* up, exactly? I'm the senior analyst. My pay rate is higher than anyone else in the department. My opinions are highly regarded by my ISO. I am constantly consulted on security issues, quoted in the press and interviewed on tv.
Yet, according to you, I'm chopped liver.
That's an interesting perspective, to say the least.
The person confirmed this at the end of their query when they mentioned that those who pass to the second round of interviews eventually accept more lucrative offers.
Offer more money. Period. If you can't do that, you will have to settle for a less skilled individual that you could "train up". Just be ready for them to jump ship if you can't keep up with raises and bonuses. Hate to say it, but specialized skills = higher pay.
As far as the questions, I found them to be much too vague. I've been asked question like this before and they usually require a dozen or more clarifying questions from the interviewee. When I have answered questions like this, I often have to make a lot of assumptions about the environment so that usually leads to an answer that fails to satisfy either the interviewer or interviewee.
Firstly, these questions are not vague. They're almost the exact same questions I ask myself every time I view a Sguil alert. Not that I start over from 0% every time, but every alert is unique and depending on the alert, the host, the network or other factors, I may have a different set of resources available to me.
On top of that, I wonder why people think it's bad to elicit a bunch of follow-up questions from the candidate? I don't need someone who can spew the correct answer. I want someone who can think, reason and show some understanding of the incident investigation process. The best way to know if he or she fits the bill is to listen to the questions they ask, and then see if they arrive at a (not the) correct answer.
Your comment hits on the point. The fact that you mention a Sguil alert narrows down what you have available. You know what tool you have, and what data is available, none of this was given int he sample interview questions. In Sguil, you at the very least session data. If you have the disk space, you probably also have the full content of the session. So the investigative process from a network standpoint is fairly straight forward. The interview questions don't address this, and from what I gathered, are simply platform based, leaving host based monitoring. Maybe the point of the interview is to get the potential candidate's to ask these kinds of questions to get an idea of their sphere of knowledge. I still believe that if the folks who can answer these kinds of questions aren't sticking around, then its a money thing.
It all depends on the individual.
I thought it was interesting that you seized on David's post to buttress the argument about the vagueness of the answers. David's post was based upon the question. I suspect that's exactly what the questions are designed to do - elicit knowledge from the interviewee - what are you familiar with - what tools have you used - what's your approach to solving these problems.
I agree. I have done this job for several years, then managed the team that did this work, and recently switched positions to handle our more strategic clients. I have been at my current company for 4 years. They are a great company, they pay very well compared to others, they offer excellent benefits package, are very understanding of personal time, I work with great people, ... I could go on. If someone was going to try and entice myself to change companies, there would have to be a really great upside beyond what I currently have. I actually worry that if something ever happened that required me to leave I would have difficult time finding something comparable.
The three questions you listed above you stated that most do not get past these questions on the telephone interview. Maybe it is where you are advertising the position? They are basic questions anyone with experience in this type of work should be able to answer easily.
-mike.
There is also the "interview burden" here, if the questions wander when they are being asked, then there are going to be more troubles than accuracy. If the person gets bored then you have the wrong candidate.
Many people in many organizations hve not seen industrial scope information security, you are hitting against culture, and performance, not all will pass. what is interesting is that there are no people questions, the focus on technology alone is generally a bad thing, you want to see how they think as well, and if they have the skills to deal with people.
The bottom line is that SIM is new and it changes the way you do security ops. And some of the process and procedure paradigms that come out of a SecOps program are going to be driven by the product you use.
The bottom line is that you hire intelligent, mentally flexible people with an appetite for security and train them up. It's faster and cheaper than trying to snipe experienced security ops people.