Wednesday, July 28, 2010

Time Issues in Libpcap Traces

Time is an important aspect of Network Security Monitoring. If you don't pay close attention to the time shown in your evidence, and recognize what it means, it's possible you could misinterpret the values you see.

My students and I encountered this issue in TCP/IP Weapons School at Black Hat this week. Let's look at the first ICMP packet in one of our labs.

I'm going to show the output using the Hd tool and then identify and decode the field that depicts time.

In the following output, 2d 0c 65 49 occupies the part of the packet where Libpcap has added a timestamp.

Hd output:

$ hd icmp.sample.pcap
00000000 d4 c3 b2 a1 02 00 04 00 00 00 00 00 00 00 00 00 |................|
00000010 ea 05 00 00 01 00 00 00 2d 0c 65 49 5f bf 0c 00 |........-.eI_...|
00000020 4a 00 00 00 4a 00 00 00 00 0c 29 82 11 33 00 50 |J...J.....)..3.P|
00000030 56 c0 00 01 08 00 45 00 00 3c 02 77 00 00 80 01 |V.....E..<.w....|
00000040 ea f1 c0 a8 e6 01 c0 a8 e6 05 08 00 43 5c 07 00 |............C\..|
00000050 03 00 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e |..abcdefghijklmn|
00000060 6f 70 71 72 73 74 75 76 77 61 62 63 64 65 66 67 |opqrstuvwabcdefg|
00000070 68 69 |hi|

Tpo convert 2d 0c 65 49 to a time, we have to swap the bytes, so becomes 0x49650c2d, or 1231359021 in decimal. 1231359021 is a Unix timestamp that we can convert with the -r option found in the FreeBSD version of the date command.

First let me show the date on this system so you can see the timezone of the FreeBSD system, and then I'll convert the seconds into a human readable time.

$ date
Wed Jul 28 00:11:23 EDT 2010

$ date -r 1231359021
Wed Jan 7 15:10:21 EST 2009

So, this ICMP packet has a timestamp of Wed Jan 7 15:10:21 EST 2009. Note that the date command produces a time in EST and not EDT. 15:10:21 EST becomes 16:10:21 EDT. I would have preferred seeing the date output show EDT since that is the time zone on the system in question, but I can understand the output. That seems simple enough, right?

Let's see what Tcpdump says about this packet. First I run the date command to remind us where we are running Tcpdump.

FreeBSD Tcpdump:

$ date
Wed Jul 28 00:11:23 EDT 2010

$ tcpdump -h
tcpdump version 3.9.8
libpcap version 0.9.8

$ tcpdump -n -tttt -r icmp.sample.pcap
2009-01-07 16:10:21.835423 IP >
ICMP echo request, id 1792, seq 768, length 40

As we expected, this packet has a timestamp of 16:10:21 (ignore the fractions of a second), and since the time zone is EDT it matches what we expect.

Let's see what a tool like Tshark says.

FreeBSD Tshark:

$ tshark -v
TShark 1.0.7
Compiled with GLib 1.2.10, with libpcap 0.9.8, with libz 1.2.3, without POSIX
capabilities, without libpcre, without SMI, without ADNS, without Lua, without
GnuTLS, without Gcrypt, with Heimdal Kerberos.
NOTE: this build doesn't support the "matches" operator for Wireshark filter

Running on FreeBSD 7.2-RELEASE-p7, with libpcap version 0.9.8.

Built using gcc 4.2.1 20070719 [FreeBSD].

$ tshark -n -t ad -r icmp.sample.pcap
1 2009-01-07 15:10:21.835423 ->
ICMP Echo (ping) request

What? Why does it show 15:10:21? That's the result for EST, not EDT, which is the time zone of the FreeBSD system.

Let's see if a Linux system in EDT behaves the same way.

$ date
Wed Jul 28 00:44:54 EDT 2010

$ tcpdump -V
tcpdump version 4.0.0
libpcap version 1.0.0

$ tcpdump -n -tttt -r icmp.sample.pcap
2009-01-07 16:10:21.835423 IP >
ICMP echo request, id 1792, seq 768, length 40

$ tshark -v
TShark 1.2.7
Compiled with GLib 2.24.0, with libpcap 1.0.0, with libz, with POSIX
capabilities (Linux), with libpcre 7.8, with SMI 0.4.8, with c-ares 1.7.0, with
Lua 5.1, with GnuTLS 2.8.5, with Gcrypt 1.4.4, with MIT Kerberos, with GeoIP.

Running on Linux 2.6.32-23-generic, with libpcap version 1.0.0, GnuTLS 2.8.5,
Gcrypt 1.4.4.

Built using gcc 4.4.3.

$ tshark -n -t ad -r icmp.sample.pcap
1 2009-01-07 15:10:21.835423
ICMP Echo (ping) request

Again, we see Tcpdump correctly honor the local time zone (EDT) and display the timestamp as 16:10:21, whereas Tshark shows the timestamp as EST or 15:10:21.

I am really disappointed by this Tshark behavior. Incidentally, you get the same results from any tool in the Wireshark suite, such as Wireshark itself, capinfos, etc.

Does anyone know why Tshark and the like don't really honor the local time zone, and instead use Standard Time instead of recognizing Daylight Savings Time?


Scott said...

Work entirely in GMT. Come up with tools to convert logs to GMT. Move all the timeline entries to GMT.

As an added benefit when (sadly more like if) affected companies and/or groups interoperate they can quickly share timelines and immediately compare them rather than have knowledge of local daylight savings time rules, etc.

Argent said...

I'm wondering if the EST/EDT bit is based on the date of the timestamp, rather than using the date which is currently in effect. e.g. if you have a packet capture from 18 July and you look at it in December, I'm guessing you'll see the opposite behaviour.

Working in GMT avoids any interesting time-zone related problems, (for example, in a capture crossing a DST boundary, time can move backwards or leap forwards) but loses some of the context.

For example, in this case, the packet arrived at 3:10PM in the eastern time zone. Using local time zone, it may be easier to correspond with some other events (e.g. going out for afternoon coffee or the like). For certain types of attacks, these sorts of correlations can be quite interesting.

Anyway, I'd argue that tshark, date, etc are doing the correct timezone parsing, as EDT does not exist anywhere in the world during the winter months.

Anonymous said...

I agree with Argent. The timestamp of your sample ICMP packet was created during standard time, not daylight saving time. Tshark is doing the right thing.

Anonymous said...

Like anon, I agree with Argent. If you are doing time correlation across multiple logs, correlating with physical events, I would prefer the actual time of the event rather than one adjusted for current time of the current OS

Gerald Combs said...

If I understand the tcpdump sources correctly, the "-tttt" flag shifts the time zone offset according to the current time. That is, if you run "tcpdump -n -tttt -r icmp.sample.pcap" on November 6 and November 7 this year you'll get different results. We can certainly add this functionality to Wireshark but I have a feeling it wouldn't be received very well.

Anonymous said...

Thinking about the Clifford Stoll review recently, and watching the doc on youtube, Stoll was perplexed why the activity was right around lunchtime (12 noon). Why would the hacker choose the busiest time, when people might observe the user? Then he realized the hacker was from Germany and it made sense, because it was evening in Germany.

I'd agree with the comments above, I'd want the raw date, perhaps that's not a bug, it's a feature. Although it does make me think, because this is a daylight savings issue, that this is an unpatched system possibly, and it's not adjusting? Just a thought - there was a Java patch for dst some years back..

Richard Bejtlich said...

All I'm asking for here is a little transparency. My students were very frustrated to see on result in a Wireshark-related tool and another in Tcpdump.

Richard Rowlandson said...

First of all I want to say that by no means am I an expert in this topic, and I am still learning the capabilities of these tools and how they can be applied in various ways during an analysis.

However, I agree with Richard in that there needs to be a standard way of displaying time across these tools. I attended his class this week and it was a little distracting to have to keep in mind how the time format was displayed by each tool, and consequently, have to accommodate for the differences. Not only is it distracting to have to accommodate for the differences, but you also have to know that the problem exists in the first place. Adding a manual, human component like this into the equation can lead to conversion errors which, in turn, could cause the analyst to miss potentially critical information during the analysis.

Aside from the obvious consistency issue, IMO, I think that having the tools set to report the time in GMT by default is a good start and would allow for a standard correlation across time zones. In addition to this, I think that the incorporation of a switch that would allow the analyst to specify a particular time zone offset would be advantageous. This would give more flexibility in how we are able to visualize the data. For example, if you had an analysis that included multiple pcap files from multiple time zones, by default they would all show in GMT. However, the analyst would have the option to convert them to the time zone corresponding to the site where the initial event happened, or look at each pcap in the time zone from where it originated. Everyone has their own way of making sense of this type of data, but personally, having the ability to view the timelines it in different ways would be advantageous.

Anonymous said...

@Gerald Combs - you do understand that in this case TShark properly reported the actual "time" of the event in EST when TCPDump failed to do that?? TCPDump did not take EST/EDT changes into account and reported the wrong "time". Richard says that Tcpdump honored the local time zone, but that was actually the wrong thing to do. He said that all tools in the Wireshark suite had the same results - the correct results. So why include this Tcpdump functionality/code into Wireshark?

caf said...

I tend to agree with Argent and Anonymous x 2 - Wireshark showed 15:10:21 because that's the time that the clock on the wall showed when that packet came in.

If you want consistent times, switch to UTC. Woe bedtide the analyst who is attempting to make sense of a packet log annotated in local time that covers the hour when the clocks are actually put back - when the same wallclock time happens twice an hour apart...