Friday, November 03, 2006

Real Insider Threats

Just the other day I read the following in Cliff Berg's book High-Assurance Design:

Roles should be narrowly defined so that a single role does not have permission for many different functions, at least not without secure traceability.

The CTO of a Fortune 100 financial services company once bragged to me over dinner that if he wanted to, he had the ability to secretly divert a billion dollars from his firm, erase all traces of his actions, and disappear before it was discovered.

Clearly, the principles of separation of duties and compartmentalization were not being practiced within his organization.

Now I read the following in VARBusiness:

Federal law enforcement officials Tuesday arrested the well-known CEO of White Plains, N.Y.-based MSP provider Compulinx on charges of stealing the identities of his employees in order to secure fraudulent loans, lines of credit and credit cards, according to an eight-count indictment unsealed by the U.S. Attorney's office in White Plains.

Terrence D. Chalk, 44, of White Plains was arraigned in federal court in White Plains, along with his nephew, Damon T. Chalk, 35, after an FBI investigation turned up the curious lending and spending habits. The pair are charged with submitting some $1 million worth of credit applications using the names and personal information -- names, addresses and social security numbers -- of some of Compulinx's 50 employees...

Terrence Chalk is also charged with racking up more than $100,000 in unauthorized credit card charges. If convicted, he faces 165 years in prison and $5.5 million in fines, prosecutors say. Damon faces a maximum sentence of 35 years imprisonment and $1.25 million in fines.

These are exactly the problems I mentioned earlier. Both cases make me sick. In the former, the Fortune 100 CEO knew his organization was broken but he thought it was a joke. In the latter, someone in a position of authority abused his access and ruined the financials lives of his employees.

This is a great example of the need to implement proper corporate governance by not centralizing the roles of CEO, President, and Chairman of the Board in a single person. Furthermore, none of those people should have access to the data abused by Mr Chalk. That level of access should stop at the VP for Human Resources.

Obviously the smallest of companies (mine included) can't separate certain duties because there are too many roles for too few people! However, organizations with 100 or more employees should certainly be taking steps to limit the access all employees have -- including the CEO.

This includes system and security administrators. According to surveys like those conducted by Dark Reading, a certain percentage of those with privileged access are abusing their power.

I often hear that system administrators should be responsible for securing their systems. I believe that sys admins should configure their systems as securely as possible, but outside parties (auditors, independent security staffs) should be responsible for auditing system activity and ensuring operation in compliance with security policies.

At some point we will also be able to remove the ability of system administrators to access sensitive data, perhaps using role-based access control (RBAC). There is no need for a sys admin who maintains a platform housing Social Security numbers and the like to be able to read those records. It will not be popular for current sys admins to relinquish their "godlike" powers, but it will result in more secure operations. Sys admins are data custodians; they are not data owners.


John Salomon said...


I don't usually post to other peoples' blogs, although I couldn't pass this one up:

At some point we will also be able to remove the ability of system administrators to access sensitive data...

Won't happen. Your rationale behind this is correct, but I believe it's a utopian fantasy where the exact access parameters of those who Make Things Work (tm) are so strictly defined as to defy any potential for abuse of access rights. There are a few problems in this argument, though.

I hope this doesn't sound like a strawman, but I'm very worried about any approach that goes out from the idea of systems administrators as inherent weak points in the security of a system. The user-as-weakness argument has been proven, but the moment you begin to approach those who need near-unfettered access as a reality of their daily work as a liability, you run into trouble.

My experience in several large banks has shown me that this leads to over-proceduralization and excessive control mechanisms, which stymie innovation and often substantially reduce the ability of IT to respond to the business' needs. With the result that, instead of waiting for the approved change window, a trader will often go ahead and roll his own.

This leads to the problem of people who need a certain functionality simply ignoring security policies and restrictions. A bond trader bringing in $20 million per day is rarely subject to the same sort of security limitations as a line employee, and I don't see this being the case anytime soon.

Likewise with banal things like senior managers and password resets. At some point, there is a gray zone between what is permissible security-wise and what is necessary to ensure profitability. I suppose a sensible middle ground of "permit within reason, but control" is the way to ensure things get done. Sorry for the diatribe.. :-)

Richard Bejtlich said...

Hi John,

Your bond trader example made me think of the many scandals involving bond traders.

Anonymous said...

someday someone will write a simple little app that logs sysadm activity in a way that can't be tampered with (hash values?) and present a synopsis of the actvity to management - by fax or something. Some easy way that sysadmins will begin to become accountable...

Or not. I probably don't know what I am talking about anyway.

John Ward said...

I believe I have commented on this before. I believe that Sysadmins have entirely too much power and too little capacity to handle it. The often inflated egos become such that they believe they are above accountability. Example: I have a friend in law enforcement. He is constantly fighting to keep the TI folks OUT of their systems since it contains sensitive, fedarly protected information. It gets even hazier since there is information about minors in their systems. The admins believe it is their business and their right to gain access and control these systems. They bitch, they threaten, and they try sneaky stuff to try to force their way into these systems, some of which has landed one or two of them on the wrong end of the law enforcement agencies attention. Problem is, this is the attitude I come across in just about every organization I've been in. It's scarry to think how easily they can get information like SSN's, bank routing information, performance reviews, and in the case of the law enforcement agency mentioned above, the names and "places where the bad man touched" of juvinile victims.

Marc Spitzer said...


From what little I know about it, its been done and by in large the market did not want it. B class systems or better. The problem is you need to get rid of the G_d bit, for real. By "for real" I mean that you need to make sure not only that the os supports this but that the bussiness does not give any one person all the access needed to do SA work. This means things that used to take 15-20 min to do could now take days. It also makes normal personel issues like vacation really painful to small shops.


Don't scare the SA's, as a SA many of the bad ones are too arogant and imature to believe that it can happen to them, scare their bosses and their legal department(even if it is yours). Make a list of the laws that the SA's will break just by looking at the info with out a valid legal reason and the bad consiquences that will happen to the managenet involved, along with the SA of course. Explaine how you guys are dead serious about this and how win loose or draw you pay for your own lawyer etc. Have this all written up for every manager who will supervise the SA's, up to VP/Director level, to sign. Let them read it and ask them are you sure you want access to these systems?

John Salomon said...

To be honest, my beef isn't usually with the SAs--in my humble experience, including several years as a mail, web, unix, storage, network, firewall, god-knows-what-administrator, normally you're just plain not interested in what's going through your hands. Mr. Ward hits the nail on the head when he fears what info these guys _can_ get at; however, I always snicker at managers who have the unspeakable arrogance to think that their mails are interesting enough for someone who's probably working a 14 hour day anyway to waste time on... :-) So Mr. Spitzer is pretty bang-on. There are things you can do technically and things you just need to do as a matter of policy and management (good example here being web blocking--if your users are spending 5 hours a day looking at porn, that's not a technical problem, that's a leadership problem, and to say otherwise is cowardly, but I digress...)

The bond trader example touches on a very good point--those of you who've worked in or consulted for financial institutions know how frequently traders, especially those dealing in more exotic financial products, write and manage their own software. These are generally smart guys bringing in massive amounts of cash. I'm always interested to watch the sparks fly whenever anyone suggests systematic governance for these often kludged-up applications, and find it interesting to watch the attempts to get senior management backing develop. You see, we'd love to stand behind this policy, but alas, we have this bond issue coming up..

Mr. iamnowonmai is probably the most spot-on; 'sulog' and its ilk, i.e. consistent and secure logging and change control, have always proven to be the best way to keep track of whats going on and still let people get work done.

Anonymous said...

Thank you for the kind words, Mr Salomon!

I have a part-time position opening next year, and I am trying to get administration to allow a full background, psychological, and polygraph exam on the lucky candidate. Don't think it will get far, buit will still try.

John Salomon said...

Well, polygraphs are pretty inaccurate (they're not admissible as court evidence, except, I believe, as supporting evidence), and as for psychological exams, most that I'm familiar with are notoriously unreliable. My girlfriend has a masters in psych, and does enormous amounts of interviewing for her company (large international strategy consulting outfit.) She's got some hilarious stories in that regard...

I think you're best off with references and clustered interviews. Nothing goes above experience. You should have an HR drone who takes care of all the administrata like background checks and, IMHO, no more. :-)

John Ward said...


The road goes both ways. SA's are part of the issue, managers are the other part. I've been in organizations where managers have access to the same info as SA's. And anyone who has been in corporate management knows it doesn't take long for petty pissing matches to start between managers. I always thought it was a curious trend in my current job that managers make "good friends" with the TI folks. While I am sure the large part of that is so that they can pull favors, the biggest downside is when they "pull favors". I have yet to see it happen however. Rich's argument about the seperation of powers makes sense, but I always refer to the AT&T breakup. From a consumer perspective, it just made things ridiculous as you got bounced back and forth with the "Its not our problem, go to your long distance/local provider" runaround. And look what happened in the end. On one hand, it keeps the power spread among the different titles, on the other hand it just increases bueracracy.

Rob Lewis said...

The past emphasis on network security rather than information-centric security has led to the problem of IT staffers who are unaccountable for snooping. They do need to run the network; they don't need to access sensitive data to do their jobs.

An improved MLS/TOS implementation which governs business data flow can assure that anyone who needs certain access privileges to maintain systems can do so without abusing access rights. In smaller shops where there are less options for separation of duties, non-negotiable audit trails that are forensically defensible will help act as a deterrent where certain persons are forced to wear 2 hats. Behaviors tend to straighten out quickly when staff know they are being monitored anyway.

Security policies that are unenforceable are useless against the revenge motivated staffer or one who goes off the beam. Technology that can provide internal controls post-authentication will remove opportunities for abuse and remove temptation for authorized users.

To Mr. Salomon who thinks that to "be able to remove the ability of system administrators to access sensitive data.....won't happen", we are doing it already with no interference with the SA's ability to do his job.

The difference is that it must be done at the host level, not the network.

John Salomon said...

To Mr. Salomon who thinks that to "be able to remove the ability of system administrators to access sensitive data.....won't happen", we are doing it already with no interference with the SA's ability to do his job.

This is fine and good. Maybe my absolute statement was poorly worded; even a basic set of role-based privilege definitions on a system provides a good start. However, it all relies pretty heavily on a willingness to implement not just technical privilege restrictions and auditability for SAs. I'm more concerned about creating an enforceable bullet-proof, no-exceptions, always-enforced Chinese wall preventing senior management from essentially cajoling or browbeating SAs into going beyond what they're supposed to be doing, even if it's (technically) within their rights to do so.

Data protection legislation and audit trail requirements (SOX, anyone?) provide some safeguard. However, I've seen a staffer ordered by the head of HR of a Fortune 500 company to brazenly break a data privacy law, as the ca. $35,000 penalty in case of discovery was far outweighed by the potential financial damage to the company were the law to be respected. Kind of difficult to resist when the CIO, CFO and CEO are tacitly behind such an order, especially in countries with weak whistleblower protection.

My point? I believe that this is an area lacking control and audit far too frequently neglected, in favor of what often seems to border on paranoia about what evil, uncontrolled systems administators might do left to their own devices.

Chris_B said...

First of all Richard was right to use the terms data custodians and data owners. This is an important distinction.
Second of all, my experience is that propper data classification does not get in the way of routine SA/IT work. As far as traders go, well as said before, all you can do is present a statement of risk and potential compliance violations to management and let them decide. This is my 2cents based on years of doing itsec for financials.

Rob Lewis said...

There is obviously a need for addressing concerns at both SA and C-levels. Non-negotiable auditing addresses both levels and would give SEC auditors trails to determine inappropriate actions. The CSO should be implementing that Chinese wall, if necessary, through domain and role separation, or whatever, because that is his job. The CSO or any SA should also make sure they protect their own backsides in cases where a superior orders them to violate a regulation.

The same forensically defensible audit logs give C-level execs proof that they have performed their duty properly (due diligence) as well as assist in legal cases against a peer or a staffer.

There may come a time when, if one chooses to do the easy thing instead of the right thing, there will be an eventual cost to be paid for that decision and hopefully there will be less and less tolerance for unethical behaviors and criminal acts, expressed by large fines and jail terms. In the mean time, I am of the belief that the company that protects trade secrets and private data will hold a competitive advantage.