![It all starts with an [SPAM] email](https://imtiazrahman.com/wp-content/uploads/2017/01/cropped-wall1.jpg)
SPAM (or more specifically phishing) email is the most popular and effective weapon of cyber attack now a day. What is the first thing people do after receiving a spam email? Some got caught in the spammer net, some ignore it, some deletes the email, but as a security enthusiast what will you do? Do some analysis! and how to do the analysis? Well to be honest manually cause there are some cool stuff in the market but those are expensive. Why don’t you setup your own analysis platform with open source tools? In this post i’ll show you how you can setup a platform from where you can perform some investigation and security analysis using thehive and cortex two open source system.
thehive: A scalable, open source and free Security Incident Response Platform, designed to make life easier for SOCs, CSIRTs, CERTs and any information security practitioner dealing with security incidents that need to be investigated and acted upon swiftly.
cortex: Powerful observable analysis and active response engine.
imap2thehive: A python script for collecting SPAM email from an IMAP server. Thanks to Xavier Mertens aka “xme“ for writing the wonderful script.
What I’ll show you in this post is to get a SPAM email from a predefine mailbox using a python script and create a case in thehive automatically. Based on the email observables perform some investigation, analysis using Cortex and response accordingly. In other words, Let’s see how you can build a Security Orchestration Automation and Response (SOAR) platform from scratch for free. I’ll use an Ubuntu 18.04 LTS server with 8GB of ram and 200GB of HDD.
Step 1: elasticsearch installation
thehive use elasticsearch for storing its data. So elasticsearch is mandatory for building thehive system. Follow the below procedure for install the elasticsearch.
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key D88E42B4
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
sudo apt install apt-transport-https
sudo apt update && sudo apt install elasticsearch
Do not start the service now, we will do it later.
Step 2: elasticsearch Configuration
After installation lets do some configuration on elasticsearch. Open the file elasticsearch.yml and paste the below lines at the end of the file.
sudo vim /etc/elasticsearch/elasticsearch.yml
network.host: 127.0.0.1
script.inline: true
cluster.name: hive
thread_pool.index.queue_size: 100000
thread_pool.search.queue_size: 100000
thread_pool.bulk.queue_size: 100000
Step 3: thehive Installtion
It’s time to install thehive project. Perform the following for installing thehive in your server.
echo 'deb https://dl.bintray.com/thehive-project/debian-stable any main' | sudo tee -a /etc/apt/sources.list.d/thehive-project.list
curl https://raw.githubusercontent.com/TheHive-Project/TheHive/master/PGP-PUBLIC-KEY | sudo apt-key add -
sudo apt-get update
sudo apt-get install thehive
thehive will not start after the installation. We will start it later.
Step 4: Cortex Installation
Like thehive perform the below steps for installing Cortex in your server.
echo 'deb https://dl.bintray.com/thehive-project/debian-stable any main' | sudo tee -a /etc/apt/sources.list.d/thehive-project.list
curl https://raw.githubusercontent.com/TheHive-Project/Cortex/master/PGP-PUBLIC-KEY | sudo apt-key add -
sudo apt-get update
sudo apt install cortex
Like thehive, Cortex will not start after the installation. We will do it later.
Step 5: Configure thehive
The only required parameter in order to start thehive is the key of the server (play.http.secret.key). This key is used to authenticate cookies that contain data. If thehive runs in cluster mode, all instances must share the same key. Type the below command to generate the key
cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -n 1
A random number will be printed in your terminal. Copy that key and open the /etc/thehive/application.conf file with your favourite editor and add the line at the bottom of the file.
play.http.secret.key="YOUR_SECRET_KEY"
Step 6: Configure Cortex
The only required parameter in order to start Cortex is the key of the server (play.http.secret.key). This key is used to authenticate cookies that contain data. If Cortex runs in cluster mode, all instances must share the same key.
Type the below command to generate the key
cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -n 1
A random number will be printed in your terminal. Copy that key and open the /etc/cortex/application.conf file with your favourite editor and add the line at the bottom of the file.
play.http.secret.key="YOUR_SECRET_KEY"
Step 7: service start
The thehive and cortex installation and configuration is done. Now its time to start the service. Perform the below steps to enable and start the service of Elasticsearch, thehive and cortex.
sudo systemctl enable elasticsearch.service
sudo systemctl enable thehive.service
sudo systemctl enable cortex.service
sudo systemctl start elasticsearch.service
sudo systemctl start thehive.service
sudo systemctl start cortex.service
sudo systemctl status elasticsearch.service
sudo systemctl status thehive.service
sudo systemctl status cortex.service
It may take some time to start the service. By default thehive will use port 9000 and cortex will use port 9001. Wait until the ports comes up.
Step 8: Login to thehive
Now its time to login to the thehive. Open a browser and got to http://<YOUR_SERVER>:9000 to access the thehive interface. For the first time you will see an image like below.
Click on “Update Database“. It will redirect you to the admin user creation page which is looks like the below image.
Create an admin user which we will use to login to thehive. After creating the user you will be redirect to the thehive login page which is similar to the below image.
Give your username and password and click “Sign In“. You will see a dashboard like below image which means you have successfully install and configure thehive.
Step 9: Login to Cortex
Like thehive the process is similar except the port. Open a browser and got to http://<YOUR_SERVER>:9001 to access the cortex interface. For the first time you will see an image like below.
Click on “Update Database“. It will redirect you to the admin user creation page which is looks like the below image.
Create an admin user which we will use to login to cortex. After creating the user you will be redirect to the cortex login page which is similar to the below image.
Give your username and password and click “Sign In“. You will see a dashboard like below image which means you have successfully install and configure cortex.
Step 10: Add Organization to Cortex
The cortex system is tenant based means you can create organization. The default organization is cortex but we will create our own organization. Click Add Organization and give an organization name and description. In my case i gave the organization name “training” and description is “Training Organization” like the below image.
After creating the organization it will shows up in the organization window. Now let’s create user under this newly created organization.
Step 11: User management in Cortex
Cortex has four different rules for its user. they are read, analyze, orgAdmin and superAdmin. Users who had read role can only read the report. The users who had analyze role can read report and can run jobs. Those who have orgadmin role can run jobs, can view report, can enable/disable analyzer etc but they cannot create another organization or neither can delete the organization. superAdmin can perform all activity except run jobs, view report, enable/disable analyzer and configure analyzer.
Let’s create two users in cortex with different roles. We will create one user with read, analyze, orgadmin role enable, which we will use for configuring the analyzer and the responders. Other user with read, analyze role will be used for the integration with thehive.
Click the training organization. In the left side of the screen you will see “Add user” button. Click on that button for creating our first user with read, analyze and orgadmin role selected like the below image.
After giving all information click the “Save user” button. Like the first on lets create the second user but this time we will only select read, analyze role like the image below:
Let’s logout from the system and login again with the orgadmin role user. In my case the user name is imtiaz.
Step 12: Login cortex with orgadmin role user
After login with the orgadmin role user you will notice that the dashboard is different like the below image:
It has different tab like Job history, Analyzer, Responders, Organization. Click on Organization tab. Under this tab click on Analyzers and Responders tab. The list will be empty now. We will load the analyzer and the respondesr.
Step 13: Analyzer and responders
There are some tools required for running the analyzer. Perform the below for installing the required software.
sudo apt-get install -y --no-install-recommends python-pip python2.7-dev python3-pip python3-dev ssdeep libfuzzy-dev libfuzzy2 libimage-exiftool-perl libmagic1 build-essential git libssl-dev
sudo pip install -U pip setuptools && sudo pip3 install -U pip setuptools
download the below repo for Cortex analyzer and responders.
cd /opt
sudo git clone https://github.com/TheHive-Project/Cortex-Analyzers.git
Each analyzer and responders have some dependency which we can install from “requirements.txt” file. Instead of installing them one by one you can use the below command to install them all.
for I in $(find Cortex-Analyzers -name 'requirements.txt'); do sudo -H pip2 install -r $I; done && \
for I in $(find Cortex-Analyzers -name 'requirements.txt'); do sudo -H pip3 install -r $I || true; done
It will take some time. Wait for the installation to be finished.
After the installation lets edit the /etc/cortex/application.conf file. Open the file and search for the word “analyzer“. You need to update the location of the analyzer. Edit the file and change the path value under analyzer like below:
analyzer {
# Absolute path where you have pulled the Cortex-Analyzers repository.
path = ["/opt/Cortex-Analyzers/analyzers"]
fork-join-executor {
# Min number of threads available for analysis.
parallelism-min = 2
# Parallelism (threads) ... ceil(available processors * factor).
parallelism-factor = 2.0
# Max number of threads available for analysis.
parallelism-max = 4
}
}
restart the cortex service.
sudo systemctl restart cortex.service
Be patient and wait. Cortex will take some time to restart the service. After the service comes up login with the orgadmin role user and goto Organization->Analyzers tab. You will see all your analyzer in the list like the image below
Go back to your terminal open the /etc/cortex/application.conf file. Paste the below lines after the analyzer section.
responder {
# Absolute path where you have pulled the Cortex-Analyzers repository.
path = ["/opt/Cortex-Analyzers/responders"]
fork-join-executor {
# Min number of threads available for analysis.
parallelism-min = 2
# Parallelism (threads) ... ceil(available processors * factor).
parallelism-factor = 2.0
# Max number of threads available for analysis.
parallelism-max = 4
}
}
restart the cortex service.
sudo systemctl restart cortex.service
Be patient and wait. Cortex will take some time to restart the service. After the service comes up login with the orgadmin role user and goto Organization->Responders tab. You will see all your responders in the list like the image below
Step 14: The integration
Login to cortex with a orgadmin role user and go to organization tab. you will see a page like below image.
Remember we crate a user called thehive earlier. We will use this user for doing the integration. Click on “Create API Key” button of thehive user. Your API key will be generated. You will now have three button beside your user name like the image below.
Click on the “Reveal” button. Your API key will be displayed. Copy the API key.
Go back to your server terminal and open /etc/thehive/application.conf file. Search for the text “Cortex” and uncomment the below line.
play.modules.enabled += connectors.cortex.CortexConnector
change your cortex configuration like below.
cortex {
"CORTEX-SERVER-1" {
url = "http://YOUR_CORTEX_SERVER_IP:9001"
key = "PASTE_YOUR_API_KEY_HERE"
# # HTTP client configuration (SSL and proxy)
ws {}
}
}
after making the change restart thehive service. Type the below command to restart the service,
sudo systemctl restart thehive.service
Login to thehive. Click on your user name showing at the right top of your screen, in my case it is “theHive Admin“, click on about. A box will appear and if everything ok then you will see the version of the cortex along with server name you just configure with a green “OK” is displayed in the box like the below image
Step 15: Enable analyzer in Cortex
After integration it’s time to test the full system. but before that we need to configure and enable analyzer in Cortex. Login to Cortex with orgadmin role user. Goto “Organization” and click on “Analyzer“. You will see all installed/loaded “Analyzer” available in your system. Some of them are paid some of them are free, some of them has both. For this lab we will use AbuseIPDB and VirusTotal analyzer. To enable the analyzer we need API key. Im not going to cover how to create the key but if you create account on VirusTotal or AbuseIPDB you will get your API key for free. Let’s enable the AbuseIPDB analyzer first. Click on “Analyzers config” and find the AbuseIPDB analyzer. By default it will be disabled.
Click on edit. A window will appear like the below image
Insert your AbuseIPDB API key here and leave the days field as it is or you can choose your own value. Click on save. Goto “Analyzers” tab find the AbuseIPDB analyzer and click on “Enable“. A window will appear where you can change the value or you can leave it with default value. Click on Save.
Your AbuseIPDB analyzer is now ready for analysis. Same way you can enable other analyzer VirusTotal. Provide the VirusTotal API key form “Analyzer config” and “Enable” the analyzer from “Analyzers” tab. Enable both “VirusTotal_GetReport_3_0” and “VirusTotal_Scan_3_0“
Step 16: Analysis from Cortex
You can now perform some analysis from cortex. Click on “New Analysis” from the left corner of your screen. A window will appear. Leave the TLP and PAP value as “AMBER” Choose the datatype as “ip“. Provide an ip address for example 8.8.8.8 (but you can choose other address also). The purpose of this analysis is to check whether the system is working or not. So I leave the TLP and PAP value as AMBER which is the default one. You can change it if you like. Check the “Analyzers” which you want to do the analysis.
Click on “Start” to start the analysis. You will see the analysis will comes up in the Job History tab.
After completing the analysis the Status column will show Success message. For failure it will show you Fail status. Click on view to see the detail report of the analysis.
Step 17: Testing the integration
So far we have our thehive and the cortex system is ready. Lets test the integration by creating a case in thehive and do the analysis. Before doing that create an user with read, write permission in thehive. After creating the user logout from your current user and login again with the newly created user. Lets do the same analysis but this time from thehive by creating a case.
Click on “New Case” and filled-up the form like below.
Click on “Create case”. The newly created case will be showed up like below.
Click on “Observables” and “add obsetvable(s)“. Here you can select multiple observables which you want to analyze. Click on the “Type” field and select your observables. For this case I want to check the status/reputation of the ip address, so i choose “ip” as Type, value is 8.8.8.8 and tag is “ip_address“. You can choose different address here if you like.
Click on “Create Observables“. Your observables will come up like the image below. Select the observables and click click on Action and chose “Run analyzers”.
After that the analyzer will comes up. Choose the analyzer which you want to use for this analysis. I choose booth analyzer.
Click on “Run selected analyzers“. After finishing the analysis you will see the status of the analysis like below image.
You can see your analysis report regarding the IP address here. Click on the red link to see the report. It will show you the raw JSON output. For better view you can download the report template. First download the report template from this link. Logout from your user and login to thehive with admin user. After login goto “Admin->Report templates” and click on “import templates“. After importing the templates logout from the admin and log back with your user again. Check the report again from the same location.
The thehive has a twitter like feed where you can see all you activity. You can find it in the right side of your screen. If you have result in thehive then the integration is working.
You can now go for the real fun part, analysis from a SPAM email.
Step 18: case from SPAM email
This is the main fun part of this tutorial. The idea is creating a case in thehive from a SPAM email and If the subject of the mail is “[ALERT]”, an alert will be created in thehive. Download the repo from this link. Thanks to the Author for writing this amazing python script.
After downloading the repo open the file imap2thehive.conf and change the [imap] and [thehive] section according to your environment. You need to generate an API key from your thhive interface. Login with admin and go to “Admin->Users” from here you can create an API key for a specific user which will be used in this file.
[imap]
host: <YOUR_IMAP_SERVER_IP>
port: <PORT>
user: <MAIL_USER>
password: <PASSWORD>
folder: thehive
expunge: true
spam: (X-Spam-Flag: YES)
[thehive]
url: http://<YOUR_THEHIVE_SERVER>:9000
apikey: <YOUR_API_KEY>
observables: true
whitelists: imap2thehive.whitelists
You can also change the tlp value from [alert] and [case] section if you like. The configuration is done. Now Login to your imap email account and create a directory called “thehive” (or you can choose any other name) like the below image. I use roundqube as my email client.
Now whenever you got a SPAM email reported from other users of your domain, put that email in thehive folder and make it unread.
Now got to your terminal and install the require dependency to run the python script. Go to your imap2thehive directory and run the below command
sudo pip install -r requirements.txt
This will download the necessary package for running the scripts. After that make the imap2thehive.py file executable. Run the script like below
sudo ./imap2thehive.py -c imap2thehive.conf
If its connect to your imap server then it will show you that “Connected to IMAP server” and will start processing the unread mail in thehive folder. It will create a case from your email and create the observables like domain, ip, url (if any). Login to your thehive server and you will see a new case with the email subject has been created. Click on the newly created case and goto observables. check which one you want to analyse. Click on “Action” and “Run analyzers“. Follow the twitter like feed for the progress of your analysis. If its success you will get the result like the below image.
And that’s how your manual analysis become automated.
Step 19: Response
Now it’s time to response. Like analyzers you can enable responders in cortex. Responders are very similar to analyzer though they have different purpose. You can also write your own responders. We will use the “Mailer” responder which, when customized for your environment, should allow you to send emails to specific user from thehive interface.
Login to Cortex with orgadmin role user. Go to “Organizations ->Responders Config” click on “Mailer“. A window will appear like the below image.
According to your environment provide your smtp_host, smtp_port and from (sender) address. Click save. Go to Responders tab and find Mailer_1_0. Click “Enable“. A window will appear like below image
Like before filled up the fields according to your environment and click Save. Click on “Refresh responders” button. Our responders is configured. Lets see the responder action from thehive.
Login to thehive. I have a case here which generate from a spam email with a subject line “Awating for your confirmation“. This is a real life email which I received couple of year ago. Using the python script I pull the mail from a predefine mailbox and make a case in the thehive.
When you click on a caes in thehive at the bottom you will see a section called Description. Inside description you will see the original mail message. In my case its looks like the below image.
Now the message is reported by a user who has received the mail. The user forward the mail to our pre-configure mailbox and we pull that message and start our investigation. From this Description window you can draft your reply email. See the below image for an example. You will find a pen icon below the Description. Click on that icon and it will allow you to make the draft. After making the changes click on the green tick mark to save the change.
Now the real trick. Go to Tags and add tags like below. You will have the email tag already their just add the mail:YOUR_RECIPIENT_EMAIL_ADDRESS. Replace YOUR_RECIPIENT_EMAIL_ADDRESS with the address you want to reply.
Go to “Responders”and click “Mailer_1_0″and click “Yes, run it”.
Go to “Responders Jobs” at the bottom of your case. If you configure the responders correctly then you will see an image like below with a status of cool green “Success” message.
Check with your recipient, he/she should get a message from your thehive system.
The End
The goal of this post is to show you how to build a system for doing security analysis and investigation in an automated way. You can do multiple analysis with your observables like url, domain, file from a single place and can response accordingly. A powerful Security Orchestration Automation and Response (SOAR) tool with lots of feature which cannot be discuss in just one blog post. All of the above you can build the system for free. Build your system in secure place first before going to the production and be careful abut the observables specially url, file etc. Happy hunting.
Thanks for reading the post. If you enjoyed the post, please share it with your network and let me know your thoughts in the comments.
About the Author: Imtiaz is working in a financial organization in Bangladesh and having experience in system, network and security administration. Feel free to contact with him on LinkedIn or Twitter.