Network Monitoring for Threat Hunting

What’s happening in our networks?

If we don’t feel we have a handle on that question, we should enhance our network monitoring capabilities; or as we like to say “No Vision, No Security”.

In this post, we will briefly discuss how we configure and use Zeek with the Elastic stack for network monitoring and threat detection.

Before getting started it is worth noting that Zeek has some extra features compared to other open-source IDS that lead us to choose it. Here is a taster of what you can expect from Zeek:

• Flexible network security monitoring with event correlation
• Traffic inspection
• Attack detection
• Log recording
• Distributed analysis
• Full programmability
• Relatively easy to install

Keep in mind that Zeek is not about trying to tell you what is bad but rather what is happening within your network.

The Elastic Stack is designed to allow users to take data from any source, in any format, and to search, analyze, and visualize that data in real-time using an open-source framework, that is easy and quick to both install and maintain.

How it works

With Zeek is up and running, it will need to obtain a copy of all network packets, without being inline as a typical network sniffer would. As a result we first need to generate a copy of the live traffic using a packet broker, after which we will add the necessary configuration to allow the conversion of all data into JSON format.

Unfortunately by default Zeek logs are TSVs (Tabbed Separator Values) which need to be converted into JSON for ingestion into ElasticSearch. Easy peasy! This script will also configure it to use ISO8601 timestamps instead of UNIX epoch time.

Note: use your preferred directory — in my example, it is: nsm/bro/…

Also just change the value in “ascii.bro” file in this directory

/nsm/bro/share/bro/base/frameworks/logging/writers/ascii.bro

From this

To this

Restart Zeek and here we go!

And check if the logs are in JSON format.

The logs look like this
{
    "ts": 1570522374.209008,
    "uid": "C75Nxy4MNOcV3UZoma",
    "id.orig_h": "10.0.2.15",
    "id.orig_p": 44762,
    "id.resp_h": "148.251.233.2",
    "id.resp_p": 443,
    "proto": "tcp",
    "conn_state": "OTH",
    "local_orig": true,
    "local_resp": false,
    "missed_bytes": 0,
    "history": "C",
    "orig_pkts": 0,
    "orig_ip_bytes": 0,
    "resp_pkts": 0,
    "resp_ip_bytes": 0
} {
    "ts": 1570522126.778665,
    "uid": "CFU3TG1mVtNU2EsMK2",
    "id.orig_h": "10.0.2.15",
    "id.orig_p": 44734,
    "id.resp_h": "148.251.233.2",
    "id.resp_p": 443,
    "proto": "tcp",
    "duration": 0.000343,
    "orig_bytes": 0,
    "resp_bytes": 0,
    "conn_state": "OTH",
    "local_orig": true,
    "local_resp": false,
    "missed_bytes": 0,
    "history": "Ca",
    "orig_pkts": 0,
    "orig_ip_bytes": 0,
    "resp_pkts": 2,
    "resp_ip_bytes": 80
}

Add Filebeat conf

Until now we finished with Zeek, let’s move to the Elastic stack’s part. We are going to start with Filebeat which is responsible for reading the logs from Zeek and sending them to Logstash. We need to enable the Zeek module, with this command:

Next, we add a configuration option to enable the Zeek module to read from /nsm/bro/logs/current/… which is the directory where Zeek put its logs in real time

Now that the modules are configured, we also need to configure the Filebeat output for sending all the logs to Logstash, as previously mentioned.

In this part, we will create Logstash Inputs, Filters, and Outputs.

Input Section: Since the Zeek logs would be forwarded to Logstash by Filebeat, the input section of the pipeline uses the beats input plugin. Here we configure the port on which to listen for Filebeat data.

Filter section: The filter section is where the real work happens. The filter section is where available filter plugins are used to parse through each message Logstash receives. This is where fields are created and populated. We get our configuration based on the RockNSM project.

Note: since the filter has more than 1300 lines, you can find all the Logstash configurations in our Marketplace.

Output section: The output section is pretty straight-forward. Once Logstash is done parsing the event, it will send its output to Elasticsearch using the Elasticsearch output plugin. Here we configure the address of the Elasticsearch node and a few other settings.

Once successfully start Zeek, Filebeat, and Logstash we are able to see the logs on Kibana:

Let’s have a real use case, first of all, we are going to start a scan on our network and see how Zeek is going to detect it and how we are going to use a Kibana dashboard to show it!

Scanning our network using Nmap:

Open the current repository and open the notice file:

As you can see in the file above, the message is so clear and we can do some modifications for more detail.

Next, we will see the Notice’s log in Kibana:

This is just a drop from the huge ocean of network monitoring for threat hunting, where we have shown the required configuration for both the Elastic Stack and Zeek along with a simple use case to make it more clear.