One Weird Trick for Reviewing Zeek Logs on the Command Line!

Are you a network security monitoring dinosaur like me? Do you prefer to inspect your Zeek logs using the command line instead of a Web-based SIEM?

If yes, try this one weird trick!

I store my Zeek logs in JSON format. Sometimes I like to view the output using jq.

If I need to search directories of logs for a string, like a UID, I might* use something like zgrep with the following syntax:

$ zgrep "CLkXf2CMo11hD8FQ5" 2020-08-16/*

2020-08-16/conn_20200816_06:00:00-07:00:00+0000.log.gz:{"_path":"conn","_system_name":"ds61","_write_ts":"2020-08-16T06:26:10.266225Z","_node":"worker-01","ts":"2020-08-16T06:26:01.485394Z","uid":"CLkXf2CMo11hD8FQ5","id.orig_h":"192.168.2.76","id.orig_p":53380,"id.resp_h":"196.216.2.24","id.resp_p":21,"proto":"tcp","service":"ftp","duration":3.780829906463623,"orig_bytes":184,"resp_bytes":451,"conn_state":"SF","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"ShAdDafF","orig_pkts":20,"orig_ip_bytes":1232,"resp_pkts":17,"resp_ip_bytes":1343,"community_id":"1:lEESxqaSVYqFZvWNb4OccTa9sTs="}
2020-08-16/ftp_20200816_06:26:04-07:00:00+0000.log.gz:{"_path":"ftp","_system_name":"ds61","_write_ts":"2020-08-16T06:26:04.077276Z","_node":"worker-01","ts":"2020-08-16T06:26:03.553287Z","uid":"CLkXf2CMo11hD8FQ5","id.orig_h":"192.168.2.76","id.orig_p":53380,"id.resp_h":"196.216.2.24","id.resp_p":21,"user":"anonymous","password":"ftp@example.com","command":"EPSV","reply_code":229,"reply_msg":"Entering Extended Passive Mode (|||31746|).","data_channel.passive":true,"data_channel.orig_h":"192.168.2.76","data_channel.resp_h":"196.216.2.24","data_channel.resp_p":31746}
2020-08-16/ftp_20200816_06:26:04-07:00:00+0000.log.gz:{"_path":"ftp","_system_name":"ds61","_write_ts":"2020-08-16T06:26:05.117287Z","_node":"worker-01","ts":"2020-08-16T06:26:04.597290Z","uid":"CLkXf2CMo11hD8FQ5","id.orig_h":"192.168.2.76","id.orig_p":53380,"id.resp_h":"196.216.2.24","id.resp_p":21,"user":"anonymous","password":"ftp@example.com","command":"RETR","arg":"ftp://196.216.2.24/pub/stats/afrinic/delegated-afrinic-extended-latest.md5","file_size":74,"reply_code":226,"reply_msg":"Transfer complete.","fuid":"FueF95uKPrUuDnMc4"}

That is tough on the eyes. I cannot simply pipe that output to Jq however:

$ zgrep "CLkXf2CMo11hD8FQ5" 2020-08-16/* | jq .
parse error: Invalid numeric literal at line 1, column 28

What I need to do is strip out the filename and colon before the JSON. I learned how to use sed to do this thanks to this post. 

$ zgrep "CLkXf2CMo11hD8FQ5" 2020-08-16/* | sed 's/.*gz://' | jq .

{
  "_path": "conn",
  "_system_name": "ds61",
  "_write_ts": "2020-08-16T06:26:10.266225Z",
  "_node": "worker-01",
  "ts": "2020-08-16T06:26:01.485394Z",
  "uid": "CLkXf2CMo11hD8FQ5",
  "id.orig_h": "192.168.2.76",
  "id.orig_p": 53380,
  "id.resp_h": "196.216.2.24",
  "id.resp_p": 21,
  "proto": "tcp",
  "service": "ftp",
  "duration": 3.780829906463623,
  "orig_bytes": 184,
  "resp_bytes": 451,
  "conn_state": "SF",
  "local_orig": true,
  "local_resp": false,
  "missed_bytes": 0,
  "history": "ShAdDafF",
  "orig_pkts": 20,
  "orig_ip_bytes": 1232,
  "resp_pkts": 17,
  "resp_ip_bytes": 1343,
  "community_id": "1:lEESxqaSVYqFZvWNb4OccTa9sTs="
}
{
  "_path": "ftp",
  "_system_name": "ds61",
  "_write_ts": "2020-08-16T06:26:04.077276Z",
  "_node": "worker-01",
  "ts": "2020-08-16T06:26:03.553287Z",
  "uid": "CLkXf2CMo11hD8FQ5",
  "id.orig_h": "192.168.2.76",
  "id.orig_p": 53380,
  "id.resp_h": "196.216.2.24",
  "id.resp_p": 21,
  "user": "anonymous",
  "password": "ftp@example.com",
  "command": "EPSV",
  "reply_code": 229,
  "reply_msg": "Entering Extended Passive Mode (|||31746|).",
  "data_channel.passive": true,
  "data_channel.orig_h": "192.168.2.76",
  "data_channel.resp_h": "196.216.2.24",
  "data_channel.resp_p": 31746
}
{
  "_path": "ftp",
  "_system_name": "ds61",
  "_write_ts": "2020-08-16T06:26:05.117287Z",
  "_node": "worker-01",
  "ts": "2020-08-16T06:26:04.597290Z",
  "uid": "CLkXf2CMo11hD8FQ5",
  "id.orig_h": "192.168.2.76",
  "id.orig_p": 53380,
  "id.resp_h": "196.216.2.24",
  "id.resp_p": 21,
  "user": "anonymous",
  "password": "ftp@example.com",
  "command": "RETR",
  "arg": "ftp://196.216.2.24/pub/stats/afrinic/delegated-afrinic-extended-latest.md5",
  "file_size": 74,
  "reply_code": 226,
  "reply_msg": "Transfer complete.",
  "fuid": "FueF95uKPrUuDnMc4"
}

Maybe this will help you too.

*I use the find command in other circumstances.

Update: Twitter user @captainGeech42 noted that I could use grep -h and omit the sed pipe, e.g.:

$ zgrep -h "CLkXf2CMo11hD8FQ5" 2020-08-16/* | jq .

Thanks for the tip!

Comments

Post a Comment

Popular posts from this blog

Zeek in Action Videos

Image
This is a quick note to point blog readers to my Zeek in Action YouTube video series for the Zeek network security monitoring project .  Each video addresses a topic that I think might be of interest to people trying to understand their network using Zeek and adjacent tools and approaches, like Suricata, Wireshark, and so on.  I am especially pleased with Video 6 on monitoring wireless networks . It took me several weeks to research material for this video. I had to buy new hardware and experiment with a Linux distro that I had not used before -- Parrot .  Please like and subscribe, and let me know if there is a topic you think might make a good video.
Read more

MITRE ATT&CK Tactics Are Not Tactics

Image
Just what are "tactics"? Introduction MITRE ATT&CK  is a great resource, but something about it has bothered me since I first heard about it several years ago. It's a minor point, but I wanted to document it in case it confuses anyone else. The MITRE ATT&CK Design and Philosophy document from March 2020 says the following: At a high-level, ATT&CK is a behavioral model that consists of the following core components: • Tactics, denoting short-term, tactical adversary goals during an attack; • Techniques, describing the means by which adversaries achieve tactical goals; • Sub-techniques, describing more specific means by which adversaries achieve tactical goals at a lower level than techniques; and • Documented adversary usage of techniques, their procedures, and other metadata. My concern is with MITRE's definition of "tactics" as "short-term, tactical adversary goals during an attack," which is oddly recursive. The key word in the tacti
Read more

New Book! The Best of TaoSecurity Blog, Volume 4

Image
  I've completed the TaoSecurity Blog book series . The new book is  The Best of TaoSecurity Blog, Volume 4: Beyond the Blog with Articles, Testimony, and Scholarship .  It's available now for Kindle , and I'm working on the print edition.  I'm running a 50% off promo on Volumes 1-3 on Kindle through midnight 20 April. Take advantage before the prices go back up. I described the new title thus: Go beyond TaoSecurity Blog with this new volume from author Richard Bejtlich. In the first three volumes of the series, Mr. Bejtlich selected and republished the very best entries from 18 years of writing and over 18 million blog views, along with commentaries and additional material.  In this title, Mr. Bejtlich collects material that has not been published elsewhere, including articles that are no longer available or are stored in assorted digital or physical archives. Volume 4 offers early white papers that Mr. Bejtlich wrote as a network defender, either for technical or pol
Read more