Wednesday, August 19, 2020

One Weird Trick for Reviewing Zeek Logs on the Command Line!

Are you a network security monitoring dinosaur like me? Do you prefer to inspect your Zeek logs using the command line instead of a Web-based SIEM?

If yes, try this one weird trick!

I store my Zeek logs in JSON format. Sometimes I like to view the output using jq.

If I need to search directories of logs for a string, like a UID, I might* use something like zgrep with the following syntax:

$ zgrep "CLkXf2CMo11hD8FQ5" 2020-08-16/*

2020-08-16/conn_20200816_06:00:00-07:00:00+0000.log.gz:{"_path":"conn","_system_name":"ds61","_write_ts":"2020-08-16T06:26:10.266225Z","_node":"worker-01","ts":"2020-08-16T06:26:01.485394Z","uid":"CLkXf2CMo11hD8FQ5","id.orig_h":"192.168.2.76","id.orig_p":53380,"id.resp_h":"196.216.2.24","id.resp_p":21,"proto":"tcp","service":"ftp","duration":3.780829906463623,"orig_bytes":184,"resp_bytes":451,"conn_state":"SF","local_orig":true,"local_resp":false,"missed_bytes":0,"history":"ShAdDafF","orig_pkts":20,"orig_ip_bytes":1232,"resp_pkts":17,"resp_ip_bytes":1343,"community_id":"1:lEESxqaSVYqFZvWNb4OccTa9sTs="}
2020-08-16/ftp_20200816_06:26:04-07:00:00+0000.log.gz:{"_path":"ftp","_system_name":"ds61","_write_ts":"2020-08-16T06:26:04.077276Z","_node":"worker-01","ts":"2020-08-16T06:26:03.553287Z","uid":"CLkXf2CMo11hD8FQ5","id.orig_h":"192.168.2.76","id.orig_p":53380,"id.resp_h":"196.216.2.24","id.resp_p":21,"user":"anonymous","password":"ftp@example.com","command":"EPSV","reply_code":229,"reply_msg":"Entering Extended Passive Mode (|||31746|).","data_channel.passive":true,"data_channel.orig_h":"192.168.2.76","data_channel.resp_h":"196.216.2.24","data_channel.resp_p":31746}
2020-08-16/ftp_20200816_06:26:04-07:00:00+0000.log.gz:{"_path":"ftp","_system_name":"ds61","_write_ts":"2020-08-16T06:26:05.117287Z","_node":"worker-01","ts":"2020-08-16T06:26:04.597290Z","uid":"CLkXf2CMo11hD8FQ5","id.orig_h":"192.168.2.76","id.orig_p":53380,"id.resp_h":"196.216.2.24","id.resp_p":21,"user":"anonymous","password":"ftp@example.com","command":"RETR","arg":"ftp://196.216.2.24/pub/stats/afrinic/delegated-afrinic-extended-latest.md5","file_size":74,"reply_code":226,"reply_msg":"Transfer complete.","fuid":"FueF95uKPrUuDnMc4"}

That is tough on the eyes. I cannot simply pipe that output to Jq however:

$ zgrep "CLkXf2CMo11hD8FQ5" 2020-08-16/* | jq .
parse error: Invalid numeric literal at line 1, column 28

What I need to do is strip out the filename and colon before the JSON. I learned how to use sed to do this thanks to this post

$ zgrep "CLkXf2CMo11hD8FQ5" 2020-08-16/* | sed 's/.*gz://' | jq .

{
  "_path": "conn",
  "_system_name": "ds61",
  "_write_ts": "2020-08-16T06:26:10.266225Z",
  "_node": "worker-01",
  "ts": "2020-08-16T06:26:01.485394Z",
  "uid": "CLkXf2CMo11hD8FQ5",
  "id.orig_h": "192.168.2.76",
  "id.orig_p": 53380,
  "id.resp_h": "196.216.2.24",
  "id.resp_p": 21,
  "proto": "tcp",
  "service": "ftp",
  "duration": 3.780829906463623,
  "orig_bytes": 184,
  "resp_bytes": 451,
  "conn_state": "SF",
  "local_orig": true,
  "local_resp": false,
  "missed_bytes": 0,
  "history": "ShAdDafF",
  "orig_pkts": 20,
  "orig_ip_bytes": 1232,
  "resp_pkts": 17,
  "resp_ip_bytes": 1343,
  "community_id": "1:lEESxqaSVYqFZvWNb4OccTa9sTs="
}
{
  "_path": "ftp",
  "_system_name": "ds61",
  "_write_ts": "2020-08-16T06:26:04.077276Z",
  "_node": "worker-01",
  "ts": "2020-08-16T06:26:03.553287Z",
  "uid": "CLkXf2CMo11hD8FQ5",
  "id.orig_h": "192.168.2.76",
  "id.orig_p": 53380,
  "id.resp_h": "196.216.2.24",
  "id.resp_p": 21,
  "user": "anonymous",
  "password": "ftp@example.com",
  "command": "EPSV",
  "reply_code": 229,
  "reply_msg": "Entering Extended Passive Mode (|||31746|).",
  "data_channel.passive": true,
  "data_channel.orig_h": "192.168.2.76",
  "data_channel.resp_h": "196.216.2.24",
  "data_channel.resp_p": 31746
}
{
  "_path": "ftp",
  "_system_name": "ds61",
  "_write_ts": "2020-08-16T06:26:05.117287Z",
  "_node": "worker-01",
  "ts": "2020-08-16T06:26:04.597290Z",
  "uid": "CLkXf2CMo11hD8FQ5",
  "id.orig_h": "192.168.2.76",
  "id.orig_p": 53380,
  "id.resp_h": "196.216.2.24",
  "id.resp_p": 21,
  "user": "anonymous",
  "password": "ftp@example.com",
  "command": "RETR",
  "arg": "ftp://196.216.2.24/pub/stats/afrinic/delegated-afrinic-extended-latest.md5",
  "file_size": 74,
  "reply_code": 226,
  "reply_msg": "Transfer complete.",
  "fuid": "FueF95uKPrUuDnMc4"
}

Maybe this will help you too.

*I use the find command in other circumstances.

Update: Twitter user @captainGeech42 noted that I could use grep -h and omit the sed pipe, e.g.:

$ zgrep -h "CLkXf2CMo11hD8FQ5" 2020-08-16/* | jq .

Thanks for the tip!

No comments: