Weird load balancing issue with web gateways

[clath13]

We have 2 web gateways configured to load balance.  I check the config using mfend and all looks right.  I monitor the messages log and there are no errors.  We have several pcs at a branch locations that connect back to our HDQ and then go to the Internet via our proxies.  We have all clients pointing to the proxies via a DNS name associated with a VIP - let's call it the .10.  The .10 consists of 2 proxies - let's call them the .1 and .2.  So if these PCs at the branches connect to the .10 and gets sent to the .1 for its session all is fine - traffic flows in and out.  If however, the pcs connect to the .10 and gets sent to the .2 the browser sits and spins and nothing is ever returned to the browser.  A pcap shows that the client has communicated out the .2 but that the communication back from the .2 never reaches the PC.  If we send traffic from the PCs directly to the .2, bypassing the .10, traffic flows correctly.  I have no idea how to troubleshoot this and am afraid to call support because it sounds so wacky - no other PCs in our environment are having this issue other than these odd balls at our branches - other PCs at the branches who connect via Citrix are fine - just the ones that bypass Citrix are the ones having the issue.  I've engaged our network guys and they believe, after testing, that the problem somehow lies with the proxies.  Can anyone think of anything to look at, test, change, etc.?

Thank you

[jacek]

Try to use wireshark on endpoint and tcpdump on MWG to figure out if packets are sent correctly and they reach endpoint.

Verify if configuration on both nodes are correct (especially Static Routes, Network Protection and default gateway in Network Interfaces).

If a firewall is enabled (Network Protection), try to disable it.

In "Proxies (HTTP(S)...)" section verify "Proxy HA" is enabled and that Management IP, Virtual IPs, Virtual router id and VRRP interface are configured correctly.

Are both nodes synchronize theirs policies with Central Management settings?

If not, are policies blocking traffic to node2?

[clath13]

Hi,

Thanks for your reply.

Try to use wireshark on endpoint and tcpdump on MWG to figure out if packets are sent correctly and they reach endpoint.  I believe the packets are sent correctly.  If we try to go to a blocked site on the PC the proxy tries to send the block page back but the PC never sees it - pcap on the proxy shows retransmit after retransmit after retransmit - see attached pcap snippet.

Verify if configuration on both nodes are correct (especially Static Routes, Network Protection and default gateway in Network Interfaces).  All is correct.  Not using Network Protection

If a firewall is enabled (Network Protection), try to disable it.  Not using it

In "Proxies (HTTP(S)...)" section verify "Proxy HA" is enabled and that Management IP, Virtual IPs, Virtual router id and VRRP interface are configured correctly.  All configured correctly.  Have also checked the logs and there are no errors.

Are both nodes synchronize theirs policies with Central Management settings?  All nodes sync via Central Management

If not, are policies blocking traffic to node2?  NA

Can't figure out what is so different about these branch PCs.  You would think if it were an issue with the proxies it would happen to everyone and not just these one-offs.

[smasnizk]

Unfortunately a picture of Wireshark don't help to see the response...When I could correctly identify WebGateway is sending 403, which should be most likely"authentication required". This means the client has to authenticate before web gateway will pass those requests. When its not happens you will get 403 again and again. This is a client issue ignoring 403 messages to authenticate him self. As workaround you can create bypass rule based on IP of those 2 clients somewhere on the top of your authentication rule set and check if this will fix the issue.

-Sergej

[clath13]

If I do that and put a bypass rule at the top of the authentication rule set it will apply to all proxies.  Since this is only happening when the client attempts a session to one gateway while traversing the HA VIP but is fine when going directly to the gateway I'm not sure how this would fix the problem.

[smasnizk]

This is what I can see from TCPdump picture its maybe not the full picture of this issue. If you're saying this happens only when traffic is redirected to VIP, you will need to trace the request on client and all Proxies you have in HA cluster to understand what happens underneath.

-Sergej