Consider This Scenario

The other day I posted I Am Not Anti-Log. I alluded to the fact that I am not a big log fan but I do see the value of logs. This post will give you an indication as to why I prefer network data to logs.

Yesterday morning I installed OSSEC on the one system I expose to the Internet. OSSEC is really amazing in the sense that you can install it and immediately it starts parsing system logs for interesting activity.

The system on which I installed OSSEC only offers OpenSSH to the world. Therefore, you could say I was surprised when the following appeared in my Gmail inbox this morning:

OSSEC HIDS Notification.
2007 Feb 02 06:25:01

Received From: macmini->/var/log/auth.log
Rule: 40101 fired (level 12) -> "System user sucessfully logged to the system."
Portion of the log(s):

Feb  2 06:25:01 macmini su[14861]:
 (pam_unix) session opened for user nobody by (uid=0)

I don't know what that means, but I don't feel good about it. At this point I know what everyone is thinking -- SSH to host macmini and look around! Don't do it. If my macmini box is owned, the last thing I want is for the intruder to know that I know. The only defense when you suspect a box is compromised is to play dead and stupid. So where do I go to learn more about what's happened?

I do have one log-centric option. If I am running Syslog on macmini, and sending the logs elsewhere via Syslog to a central collector, I should check the collector for unusual logs from macmini. However, if I check the Syslog server, and see nothing unusual, does that mean anything? Not really! The lack of log messages may indicate whatever the intruder did was not logged by the victim. Maybe the first act involved killing remote logging! This is one of the reasons I am not a big fan of logs. The absence of logs does not confirm integrity. What can I do now?

My best option is to hop onto my NSM sensor and look for suspicious connections to or from macmini around the time I got this OSSEC alert. For example, the following capture shows part of the screen output showing sessions from 0800 GMT to about 1600 GMT. My OSSEC event occurred at 1125 GMT. This search for inbound sessions shows only ICMP at the time of the OSSEC event. A similar check for outbound sessions did not reveal anything interesting. Therefore, the OSSEC event was probably caused by some sort of daemon running on macmini, not by a login from a remote unauthorized user.

Notice the difference between the data presented in the network sessions vs the host logs. The absence of suspicious session data means no remote intruder interacted with the system during the time in question. In other words, a lack of a record showing an inbound OpenSSH session does mean no OpenSSH sessions occurred at the time in question. If you wonder if some sort of advanced covert channel ICMP-fu is happening, I have the full content validating each of the ICMP "sessions" and could investigate them if I so desired.

Also keep in mind the value of collection session and full content independent of alerts. If I waited for an alert before starting to log sessions and/or full content, I'd be in the same boat with the host-centric loggers. By doing content-neutral logging (i.e., grab everything, continuously) I can look for suspicious activity regardless of the presence or absence of alerts.