Web Gateway max file size before redirecting to download page

[Former Member]

Hello Everyone,

We are developing a serious game using Unity3d  engine for an enterprise.
Our client uses McAfee Web Gateway.
Our game downloads successfully many files without any problem. However, when it comes to download a large file like 70MB, MWG redirects it to download page and our app cannot access the file.

From forum posts I've learned that they can re-configure their system to use "data trickling". However, changing their entire config just for us may not be a reasonable solution to offer to them.

Therefore, we are seeking a workaround. First approach is to divide the big files to smaller sized files. As files to download can reach 150-200MB each we need to find the maximum file size that MWG can handle without redirecting it to download page.

I will also appreciate any other possible solutions you may advice.

Many thanks in advance

[mkutrieba]

Hello,

there are only the 2 options, progress page and data trickling.

There is no need to fully use data trickling for all traffic. You can use different criteria to build this rule. Best would be if you know all important IP addresses or URL hosts used for these downloads. Then you could build a rule like:
URL.Host is in list <list with download URL hosts>, Action: Stop Rule Set, Event: Enable Data Trickling

Example rule set would look like:
Rule 1: URL.Host is in list <list with download URL hosts>, Action: Stop Rule Set, Event: Enable Data Trickling
or alternatively:
URL.Destination.IP is in list <list with download destination IPs>, Action: Stop Rule Set, Event: Enable Data Trickling
Rule 2: <default progress page rule for all browser related things>
Rule 3: <default data trickling rule for the rest>

Something like that is a common setup since applications cannot display a HTML (progress) pages normally. They have to use data trickling instead. Therefore, use a known simple criteria and configure a rule like above to separate this traffic in policy.

Let us know if you have further questions.

Regards,
Marcel

[Former Member]

Thank you mkutrieba for your prompt reply.

In their LMS system, implementation of these packages may be done several times in time. Therefore, the training department will need to request a specific rule from security team for every implementation.

Although, we will share your comment with them, this is still doesn't seems an appropriate way of dealing  with large living implementations, hence it requires exception configuration each time.

Therefore, we still need to find an alternative.
As you mentioned that there is no other options, the best solution seems reducing the file sizes.

What is the maximum file size that MWG accepts before redirecting it to "progress page"?

Thank you

[swilkens1 · Accepted solution]

Hi @Former Member ,

There is no "max file size" before redirecting the the Progress Page.

The default Progress Page rules specify that if a transaction takes longer than 5 seconds to return data to the client, redirect to the progress page.

So, while a larger size can make it more likely to download/scan for longer than 5 seconds, it is possible for a small, but complex archive with many child elements to also take a longer time to scan. It is a function of both the size, and the complexity of the file being scanned that can cause a Progress Page to be triggered under "normal/default" circumstances.

[Former Member]

@swilkens1 thanks for clarifying the way MWG works. This gives us a good start point.

[fw_mon · Accepted solution]

it seems the OP seeks a way to avoid triggering the progress page from the server side. My understanding that the progress page is triggered when the the web server is slow to start download. I think time-to-first-byte and time-to-first-X-KBs are important. The other way to avoid the progress page is using a different user-agent string - by default the progress page is triggered only for user-agents which contains *mozilla* string in it. My suggestions:

1. use fast web server and fast pipes, try use AWS or some fast CDN for hosting. The download should start immediatelly without delay and the server/pipe should provide high bandwidth download speed.

2. ask the client to try wget/curl for download - if it successful, use a custom downloader.

You can provide a direct link to your file so I can check it.

[swilkens1]

What @fw_mon  mentioned regarding user-agent strings is true; the default Progress Page rules only apply if it is assumed to be a browser based on the UA string matching *mozilla*.

While slow server-side downloads can also be a factor, I have more often seen the bigger contributor to be scan times on the MWG itself.

[Former Member]

Thanks fw_mon,

1- is obvious
2- It may be a longer way but if we can not solve it by decreasing the file sizes we will definitely try your solution

Thank you