Using Subscribed Lists for MCP Bypass

[jebeling]

Many customers have asked if it is possible to use subscribed lists for creating MCP bypasses. It is possible and can be done currently. New! Updating the Common Catalog (tied to the MCP policy) can now be automated with Common Catalog 2.0.3. See reply to this article below.

Getting the subscribed list you want and updating the source:

Configure a list to reference the subscribed lists in the MWG GUI under Policy > Lists. Select Subscribed Lists  and right-click and select Add

Name your list, add comments if desired, then select List Content is managed remotely, then choose McAfee Supplied list, then click on Choose. Choose your list from the drop down list. Then click on OK in the Choose List Content dialog and then OK in the Add List dialog.

Save your changes in the MWG GUI if you haven't already done so.

Your new list will be added to the appropriate type group. Select the list and click on export

 

Save the exported list and then open it with a file editor. The export will not be the contents of the list but will give you the filename/id that you will need in the next step.

Save your changes in the MWG GUI if you haven't already done so.

Converting the subscribed list:

Two types of subscribed lists are supported by Common Catalog and the conversion program. IP Range and String/Domain. The attached java program can be put on your MWG and executed via cron job on a periodic basis. It is recommended that you create the output files in \opt\mwg\files so that they can easily be retrieved via URL (without authentication) or from the GUI. New version of ConvertSList now posted below. If original list is string type or regex type it will remove "*." from any entries to make it compatible with a domain list type in MCP bypass. (e.g. "*.mcafee.com" is converted to "mcafee.com"). Also since I discovered that ePO common catalog does not like duplicate entries, duplicate entries are removed during conversion.

Move the java program to \usr\bin\ConvertSList.jar

Set up a cron job to execute the following command on a periodic basis (You will have a cron entry for each list you want to have available for Common Catalog import)

java -jar \usr\bin\ConvertSList.jar /opt/mwg/storage/subscribed_lists/update_server/com.scur.type.<list type>.<list ref #>.xml> > /opt/mwg/files/<dest filename>

Examples:

java -jar ConvertSList.jar /opt/mwg/storage/subscribed_lists/update_server/com.scur.type.string.166.xml > /opt/mwg/files/string166.xml would convert subscribed string list com.scur.type.string.166.xml to a file named string166.xml

java -jar ConvertSList.jar /opt/mwg/storage/subscribed_lists/update_server/com.scur.type.iprange.4148.xml > /opt/mwg/files/iprange4148.xml would convert subscribed string list com.scur.type.iprange.4148.xml to a file named iprange4148.xml

Getting the converted subscribed list from MWG:

If you've put the files in /opt/mwg/files you can get them from the MWG GUI by going to Troubleshooting > <ApplianceName> > Files:

Or you can enable the file server for HTTP or HTTPS through the MWG administrative GUI under Configuration > File Server. Default ports are 4713 for HTTP and 4714 for HTTPS.

Converted files can then be retrieved via URL in browser.

https://<mwgaddress>:<fileserverport>/files/<filename>

Examples:

http://192.168.1.222:4713/files/string166.xml

https://192.168.1.222:4714/files/iprange4148.xml

Importing the subscribed list into the Common Catalog: 

Log into your ePO server and select Common Catalog under Common Catalog in the Main Menu.

Select the Common Catalog that matches the MCP Policy you want to add the subscribed list to, then select Actions > Import From > File.

Choose your downloaded converted subscribed list file then find it and select it in the Import Catalog Dialog and click OK

That's it. If you've set up the cron job the list on MWG will automatically update as the subscribed list changes. 

List updates in ePO (and by extension MCP Policy) can now be fully automated see reply below

Comments and suggestions welcome as always. Please note that this is not an officially supported McAfee solution. The java code isn't pretty but has been successfully implemented in several environments. Please post here if you encounter any issues and I will attempt to assist.

[jebeling · Accepted solution]

Automation is now possible! You need the 2.0.3 updates to Common Catalog in ePO. 

Update Common Catalog in ePO Software Manager to 2.0.3. There are two extensions that need to be updated. The two extensions are Catalog Framework and Core Catalog. You may need to refresh software manager to see them (button in top left).

Create cron jobs on MWG to periodically convert the new list and update it in ePO. 

Identify name of MWG subscribed list to synch by process provided in original article above.

Example crontab to convert the list on the hour every hour

0 * * * * java -jar /usr/bin/ConvertSList.jar /opt/mwg/storage/subscribed_lists/update_server/com.scur.type.string.166.xml > /opt/mwg/files/string166.xml

Example crontab to push the list to epo at 192.168.11.136 5 minutes after every hour

5 * * * * curl -X POST -n -k -F overwrite=true -F catalogId=155886d4-d616-485b-9f71-af6254841240 -F data=@/opt/mwg/files/string166.xml  https://192.168.11.136:8443/remote/catalogFramework.importCatalog.do

Credentials for epo are stored in a netrc file on mwg (see -n option for curl) so that they aren’t visible in the crontab entry itself.

Overwrite=true was the feature not available in previous common catalog.

Only other trick is identifying the ePO catalog ID. This is the catalog ID associated with the MCP policy to which you will add the bypass list (MCP Demo SaaS -MCP in example above). You need to do that one time through the ePO Common Catalog API.

curl -X POST -n -k https://192.168.11.136:8443/remote/catalogFramework.getCatalogList.do this can be run from MWG itself and again I’ve used -n to get creds from netrc file

Example output:

[root@mwg70 ~]# curl -X POST -n -k https://192.168.11.136:8443/remote/catalogFramework.getCatalogList.do
OK:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<catalogs>
<catalog id="d3ab4ed4-efab-48d2-840d-714cbf76b801">
<name>McAfee Default</name>
<description>McAfee Default</description>
</catalog>
<catalog id="3844bbf5-fc18-45d1-9bca-ff8c221cbbfd">
<name>My Default - MCP</name>
<description/>
</catalog>

If I have time later, I will provide more detail. Reply to this post if you have questions, corrections or enhancements. Here are some useful references if you are unfamiliar with cron and curl

Using netrc with curl: https://ec.haxx.se/usingcurl-netrc.html

Note for netrc the username and password should be the login credentials used for epo and those credentials must have ability to edit common catalog.

Creating crontab: https://www.lifewire.com/crontab-linux-command-4095300

[Former Member]

Thanks a lot for this guide.

When I run the 'catalogFramework.getCatalogList.do' curl command, the output only shows two entries, where the GUI has 117.

As a consequence, I cannot update the catalog I'm interested in... Any idea why there is a limitation? How can I find the correct catalog ID other than by this method?

Some issues with Office 365 lists, that needs to be adressed in the Java file, like '.*' entries:

http://ocsp2.globalsign.com/* (from 'Office 365 CRLs URLs')

*view.officeapps.live.com (from 'Office 365 URLs')

[jebeling]

I would have to modify the convert list program to address the noted entries, but actually you cannot use those list entries for MCP bypass anyway. As noted in the original article only string/domain and IP Range lists are currently supported for MCP bypass. Wildcards are not allowed, and bypass is by domain or IP only. Best I could do is convert *view.officeapps.live.com to view.officeapps.live.com and http://ocsp2.globalsign.com/* to ocsp2.globalsign.com which aren't the same thing although the second conversion is probably good enough.

As for the catalog do output, are you running that against ePO?  The Catalog list in epo should have catalogs associated with your MCP policies. You probably only have two. The converted lists go into those ePO catalogs. You can put as many lists as you want into the single ePO catalog that is associated with a specific MCP policy.

Could you please specify the lists you saw those entries in? I'm not seeing them on my system, maybe they've already been corrected? 

Update. The MS list of with entries of the form http://... has been deprecated and so the conversion program was not updated or changed to handle that form also the other entries of the form .../* and *abc... are no longer present in the MS list. Entries of the form *.abc.... will convert properly using the updated program. If a list includes any other characters not found in proper domain or hostnames the conversion program will not result in common catalog consumable output. In other words, the program can convert a regex type list but only if the entries are straight domain or hostnames or have *. prepended. The resulting list will only be accepted by common catalog if it has straight domain or hostnames and there are no duplicates in the list.

---

[Former Member]

Thanks for your quick answers.

I ended up bypassing 'officeapps.live.com' and 'ocsp2.globalsign.com '.

As for the catalog list output, yes, I am running that against ePO. Here is the output:

<catalogs>
<catalog id="xxx">
<name>McAfee Default</name>
<description>McAfee Default</description>
</catalog>
<catalog id="xxx">
<name>One of the catalog names</name>
<description/>
</catalog>
</catalogs>

That is all, when there are 171 different common catalogs in ePO.

[jebeling]

The ePO API call should be a supported function. I would suggest opening a call with support. Maybe something with your catalog names, maybe your ePO login used doesn't have the appropriate permissions? If you are currently using a login with restricted permissions, I would suggest running as super admin to get the IDs, you only need to do it once unless you add additional catalogs. In the meantime, you should be able to import the list to your catalog through the manual process in the original article.

[Former Member]

OK, I'll open a Service Request then. I though that it was not a permission problem because I could see 2 valid entries (and not nothing / permission denied).

Thanks.

[jebeling]

Was support able to identify the issue?

[jebeling]

If you don't want to use netrc for your authentication to ePO in your curl command, its just -u username:password instead of -n

[aashish.khurana]

Hi,

I am able to use the ConvertList.jar program to import ipranges. But with Domains and regex I am still getting an error:

curl -X POST -n -k -F overwrite=true -F catalogId=33ba8c62-1526-47bb-b646-83f89ef338c2 -F data=@/opt/mwg/files/string175.xml https://192.168.1.201:8443/remote/catalogFramework.importCatalog.do

Error 0 :
editors.domainname.invalidValueMsg

Can you please guide me, if I am missing something while converting or using the script? Here is the command I am using for conversion:

java -jar /usr/bin/ConvertSList.jar /opt/mwg/storage/subscribed_lists/update_server/com.scur.type.regex.175.xml > /opt/mwg/files/string175.xml

Regards,

Aashish Khurana