Dynamic Bad IP Watchlist Automation with Powershell and SIEM

[Former Member]

The following sites display a text like list of known Bad IP's, which is easy to automate, parse and import into SIEM as a Dynamic Watchlist.

Sites:

http://malc0de.com/bl/IP_Blacklist.txt

https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt

https://zeustracker.abuse.ch/blocklist.php?download=badips

https://spyeyetracker.abuse.ch/blocklist.php?download=ipblocklist

https://palevotracker.abuse.ch/blocklists.php?download=ipblocklist

https://feodotracker.abuse.ch/blocklist/?download=ipblocklist

I have built a powershell script that goes to a saved file of the sites above, and scrapes those sites, for the IP's and saves them to a file location.

After the powershell script executes daily and it saves an individual file per site above. I then created a dynamic watchlist in SIEM that updates daily based on the new files written.

Below is a generic script I have built for everyone to use, every line has been commented at the end with a "###" to display what that line is doing.

$sitelist=Get-Content 'c:\locationoffile\filename.txt' #Change Location of the File Containing Sites providing 1 single column of Bad IP's

$web=New-Object Net.WebClient

$ipregex="(\b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b)"###RegEx for IP Matching to Filter out characters###

$folderpath='c:\destinationfolderlocation\' #Change location to the folder path you wish to store the IP Files

Remove-Item 'c:\destinationfolderlocation\*' -Force ##Removes old files to build new ones##

foreach ($site in $sitelist){ #imports Sitelist

  $filename=$site.Split("///") #building filename based on domain name of site

  $filename=$filename -replace('\.','-') #building filename based on domain name of site

  $filename=$filename[2] #building filename based on domain name of site

  $filename=$filename+"-raw.txt" #adding "raw" to the end of the filename for the initial Site download

  $fullfilepath=$folderpath+$filename #full file path of the first "raw" output

  $web.DownloadFile($site,$fullfilepath) #downloads site, outputs file to the full path declared on line 11

  $cleanoutputfile=$fullfilepath -replace("-raw","") #declares the new file name it will output to, containing the clean IP information

  Get-Content -Path $fullfilepath | #imports the raw file

  Select-String -Pattern $ipregex | #will select strings only containing IP address, will filter out commented out sections, and other non-IP information

  Set-Content $cleanoutputfile #outputs the data to the new clean file, this will be the file you will import into the SIEM dynamic watchlist

  Remove-Item $fullfilepath

}

All lines above highlighted in GREEN should be changed to reflect your environment:

               Line 1: Update ‘c:\locationoffile\filename.txt’ to the location they have imported the text file of the Sites in item number 1. Example: c:\users\username\desktop\SIEM\cleanIPsitelist.txt

               Line 4: Update ‘c:\destinationfolderlocation\’ to the folder as to which you want your files to export to once complete

               Line 5: Update ‘c:\destinationfolderlocation\*’ to the same as line 4 (Keep the * at the end, as this is a cleanup function)

I recommend after editing the parameters to test run this, look in the destination folder to make sure everything has worked properly.

There will be a file per site with the format of “domainname.txt” Example: “rules-emergingthreats-net.txt”

When choosing the output folder location, it’s key to remember the SIEM needs to have access to that share in order to actively get the updated information, whatever schedule is set:

In order to run this script you need to change your powershell settings Execution Policy. If this is locked down in your environment, you will need to sign this script with a certificate of your own in order to execute. If you have access to change the Execution Policy, you can change it by entering into your powershell profile “Set-ExecutionPolicy RemoteSigned”, keep in mind, this is not a completely secure manner to allow scripts to run, but if this acceptable in your environment, this is how you will get it to execute.

More information on Execution Policies can be read about here:

http://technet.microsoft.com/en-us/library/ee176961.aspx

I have also automated this via task scheduler built into Windows.

Once scheduled and the files are output to the preferred directory, you can build the integration with SIEM.

Once logged in to your SIEM appliance, click on System Properties>Watchlists>Add

I recommend creating a watchlist per site that contains the Bad IP information, with the Name to the site providing the information like below:

Choose "Dynamic", and a Time you wish it to update, and occurrence.

In the script provided it saves to a CIFS share called SIEM (\\servername\SIEM)

Enter the Valid IP, Share Name, Path Name will be the file name of the text file containing IP Information, and a Valid User ID and Password

In my implementation I chose the "Type:" as IP Address, so I could apply this whitelist in my dashboards as a source or destination IP filter.

After you choose the "Type:", you can click "Run Now" and it should populate a list similar to the one below.

You can now apply this to a dashboard, and it can show you alerts for all activity going from a specific device or all devices going to those malicious IP's.

[Former Member]

Awesome! Thanks for sharing. Would you be willing to share the PS script you're using to pull down the IP addresses?

[Former Member]

Thank your for this document.

When you add the Path you muss set a \ for the Name of the txt File.

One of this BAD IP Adresses dont work:

https://spyeyetracker.abuse.ch/blocklist.php?download=ipblocklist

[Former Member]

I haven't had any time to address the output, but here's what I have in PS to download the content:

$source = "http://malc0de.com/bl/IP_Blacklist.txt"

$destination = "c:\fakepath\dest.txt"

$WebClient = New-Object System.Net.WebClient

$WebClient.DownloadFile( $source, $destination )

The output is not in single column text, so I'm guessing grabbing the page header information may be needed. If anyone else gets the desired output, I'd appreciate the assistance.