SOLVED: HOW TO TROUBLESHOOT CLIENT UPDATE/DEPLOYMENT FAILURES

[cdinet]

HOW TO TROUBLESHOOT CLIENT UPDATE/DEPLOYMENT FAILURES

The ePO 5.10 install guide has a section in it for Troubleshooting and log file reference information.

NOTE: Always be sure the product versions you are installing are supported versions for the specific build of OS/platform you are installing on.  KB51569 is the KB for supported environments for ePO.

Server (ePO or agent handler) logs can be located in ePO/AH install directory in the \db\logs folder.  The server log will show push agent failures or other possible communication issues.

On the client side, the logs will be located in c:\programdata\mcafee\agent\logs folder.

Masvc log will show it getting the client task and invoking it (or failing to invoke at the scheduled time).

Macompatsvc log will show agent to point product communication failures.

Mcscript log will show the update process.  For deployments, depending on agent version, there may be an mcscript_deploy log.  Those are for product deployment tasks only, where the mcscript log will be for updates.  If there is no mcscript_deploy log, then all updates and deployments will be in the mcscript log.  Mcscript will show where the breakdown occurs and whether it is a repository issue, point product lpc communication failure with the agent, or a point product issue.  Here are the steps it will go through.

1. Can it reach the repository and pull files from it? The log will show it downloading or failing to download files from a repository for one reason or another.  A “not up to date site” means that it hasn’t been replicated to yet since new content was added to the master repository.
2. Once it gets the files from the repository, can the agent communicate with the point product to send the updates to? You may see “point product is not running” or a failure to find a qualifying product (or similar error).  You may want to reinstall or upgrade the agent and/or point product in that case.
3. Once it gets past that point, in the case of deployments, you will see it running the setup for the point product. When the agent executes the setup files, then the agent part is done and successful.  The failure then will be on the OS or point product.
4. C:\windows\temp\mcafeelogs (or trellixlogs) folder will then contain the install logs for the point product to look at for troubleshooting those failures. At that point, you would go to that point product team for assistance.

Agent Deployment (Push Agent)

• Review Server Task Log result and most importantly, server_servername.log (DB\Logs)
• Keyword “push” in server.log – don’t forget that if multiple handlers exist in an environment, the push could be in a different server.log (when deploying, you can select the handler to use)
• Relies heavily on access to \\machinename\admin$ of endpoint. For all requirements and testing, see KB56386.

Injection

• Injection can occur when third-party DLLs which either have untrusted certs or no certs at all load up with Trellix processes, like McScript_InUse.exe. In that scenario, updates will end up failing with curl error 28 (meaning a timeout) and will be seen in the McScript.log or the McScript_deploy.log.
• This is due to McAfee Agent’s Self-Protection functionality – the self-protection rules are working as designed in this scenario – we WANT to prevent the process (McScript_InUse.exe, in this example) from successfully making network connections because it could be compromised by a potentially malicious file.
• Example from McScript.log:
  
  

o    network    URL(https://172.24.208.16:443/Software/SiteStat.xml?hash={0e773b7e-9786-11e7-3115-73e51b00cce7}) request, failed with curl error 28, Response 0, Connect code 0,

o    downloader                 Downloading file from https://172.24.208.16:443/Software/SiteStat.xml?hash={0e773b7e-9786-11e7-3115-73e51b00cce7} to C:\Windows\TEMP\SiteStat.xml failed.

• The log that contains the rest of the data is called mfemactl.log. It will show entries like:

o    C:\PROGRAM FILES\MCAFEE\AGENT\X86\MCSCRIPT_INUSE.EXE>(7208) was blocked from accessing('CREATE' (1)) <aac_object_section:c:\windows\syswow64\bmnet.dll <br="">
</aac_object_section:c:\windows\syswow64\bmnet.dll>

• Run the sysprep tool first to see if there are any DLLs that it finds and trusts.

Updates

A common problem, “my DATs/AMCore isn’t updating” can have many unique causes but is generally troubleshot in the same three-step manner:

1. Reproduce the issue

• It’s best to create and assign a new task (use an easily searchable name, like TestTask123). Remember that after assigning a new client task, the machine will have to communicate to receive the task (so send an Agent WakeUp or hit Collect & Send Props)

2. Confirm the task invoked and note status

• To see where and when a task started, review the masvc_machinename.log. Search the task name from the bottom up – the first thing you find should be the result of the task (if it has completed). For example:

2019-01-09 17:00:14.426 masvc(444.4768) Updater.Info: Updater engine exited with exit status as 0 and  term signal 0.

2019-01-09 17:00:14.497 masvc(444.4768) compatservice.Info: is_compat_running: 1, is_compat_required: 1

2019-01-09 17:00:15.428 masvc(444.4768) scheduler.Info: The task Daily Update Task is successful

3. Review logging

Non-Windows Agent Guide

Keep in mind:

• The McAfee Agent has separate packages for the different platforms. For example – a Windows package, a Linux package, etc. These packages must be checked in to the ePO Master Repository separately.
• The McAfee Agent can still be deployed (Push Agents) to non-Windows platforms, however it works entirely differently. Since a Windows deployment utilizes Windows file sharing, obviously that’s impossible for non-Windows clients. Instead, SSH protocol is utilized (port 22 by default). Red Hat/centOS have specific requirements to enable deployment and is a common source of push failures on those platforms. See the McAfee Agent Installation Guide for details.
• The Agent still has three services on non-Windows platforms: masvc, macmnsvc and macompatsvc.
• Non-Windows platforms are case-sensitive when working in the terminal/command line. Be wary to make sure your cases match, otherwise it will appear that the locations you’re attempting to access do not exist.

Log locations and data collection and service information

MacOS

MAC MER tool: KB86785

/Library/McAfee/agent/ (install files)

/var/log/install.log (to view install logs)

/var/McAfee/agent/ (data directory: includes logs, db files, etc.  Equivalent on Windows is ProgramData)

/etc/ma.d/ (product plugins)

To view the status of a service:

Sudo /Library/McAfee/agent/scripts/ma status

Stopping and starting services:

Sudo /Library/McAfee/agent/scripts/ma start

Sudo /Library/McAfee/agent/scripts/ma stop

Sudo /Library/McAfee/agent/scripts/ma restart

All other non-Windows platforms (Linux, UNIX, etc.)

Linux MER tool: KB83005

/opt/McAfee/agent (install files)

/var/McAfee/agent (data directory)

/etc/ma.d/ (product plugins)

[Sourcefire]

Amazing and thanks!  Not only will this help me but I'm sure this will help others; it should be a sticky.

Thanks again!

[cdinet]

Trying to make it a sticky, but there are some issues with that at the moment they are working on.  Thanks!

[JKBH1]

Thanks for the documentation!

Is there a way to send a notification (email or via Splunk) to the ePO admins if the amcore is not updating?

[cdinet]

You can set up automatic responses for sending notifications for failed updates (event id 2402) that can send an email.  If you have a registered syslog server in epo, you can enable specific events for being sent to that syslog server under server settings, event filtering.  

[cdinet]

I can't edit the original post, so I am re-posting some updated info - added more detail to assist better.

TROUBLESHOOTING UPDATE/DEPLOYMENT FAILURES

General Points to Consider

The first thing is to know which logs to look at when you see failures, no matter what type of issue you are troubleshooting.  The ePO 5.9 and 5.10 install guides have a section in them for Troubleshooting and log file reference information. 

https://docs.mcafee.com/bundle/epolicy-orchestrator-5.10.0-installation-guide/page/GUID-1841846D-8BC...

The second thing to look at is how the task is configured and whether the client is receiving the task or not.  It is important to know how it is assigned and whether the task itself is configured properly to understand how it can impact failures.

• Task settings
  • If deployment, are the values populated and correct items shown
  • If update, are the correct items selected to update
  • Are the update items in the branch that the agent policy has enabled for that particular update?
• Product deployment task
  • What groups is the task assigned to
• Run client task now
  • Initiated by user
  • Initiated by server task
  • Initiated by automatic response
• Assigned client task
  • What groups is the task assigned to
  • Is there any broken inheritance
  • Is it assigned based on tags
• Update now from client
• Did the client receive the task?
  • Review of masvc log on the client will show whether it received the task or not, if it was able to invoke it, and whether it ran on the defined schedule or failed for any reason to run
  • example: 

2019-01-09 17:00:14.426 masvc(444.4768) Updater.Info: Updater engine exited with exit status as 0 and  term signal 0. 

2019-01-09 17:00:14.497 masvc(444.4768) compatservice.Info: is_compat_running: 1, is_compat_required: 1 

2019-01-09 17:00:15.428 masvc(444.4768) scheduler.Info: The task Daily Update Task is successful 

NOTE: Always be sure the product versions you are installing are supported versions for the specific build of OS/platform you are installing on.  Search the kb at agent.mcafee.com for “supported environments” for the product you are researching on. 

• Server (ePO or agent handler) logs are in the ePO/AH install directory\db\logs folder.  The server log will show push agent failures or other possible communication issues. 
• On the client side, the logs are in c:\programdata\mcafee\agent\logs folder. 
• Masvc log will show it getting the client task and invoking it (or failing to invoke at the scheduled time). 
• Macompatsvc log will show agent to point product communication failures. 
• Mcscript log will show the update process.  For deployments, depending on agent version, there may be an mcscript_deploy log.  Those are for product deployment tasks only, where the mcscript log will be for updates. 

If there is no mcscript_deploy log, then all updates and deployments will be in the mcscript log.  Mcscript will show where the breakdown occurs and whether it is a repository issue, point product lpc or msgbus communication failure with the agent, or a point product issue.  Here are the steps to go through. 

• Can it reach the repository and pull files from it? The log will show it downloading or failing to download files from a repository for one reason or another.  A “Repository is not upto-date site” means that it hasn’t been replicated to yet since new content was added to the master repository.  
  • If superagent repository is using lazy caching, that might be failing to pull files from the server.
  • Is replication failing
  • If agent is using an agent handler, is agent handler able to pull files from the epo server or other agent handler?  Server log would show errors if that fails.
  • Agent is failing to connect to the repository
    • Can it resolve the server name?
    • Is there a proxy involved?
    • Is the repository path accessible via browser? (https://server:port/software/sitestat.xml)
  • Agent connects, but it fails to download some files
    • If only select files, can you download those select files from the browser?
    • If no files can be downloaded (example error 500 or similar), are there errors in the server logs of epo or agent handler, or if superagent, macmnsvc log?
  • Data collection would include mer from repository server it is trying to connect to as well as client failing to get files
    • In the event only some files fail, wireshark might also be required with corresponding logs
  • Once it gets the files from the repository, can the agent communicate with the point product to send the updates to? You may see “point product is not running” or a failure to find a qualifying product (or similar error).  You may want to reinstall or upgrade the agent and/or point product in that case. 
    • Most point products use msgbus to communicate with the agent, so look for any type errors. 
    • One example is in masvc log you might see “limited access on msgbus” error.  That is indicative of root certificate issue where either client does not have latest global root certificates and/or does not have latest msgbus cert updater package.
  • Assuming all files were downloaded properly and point product to ma is communicating successfully, in the case of deployments, you will see it running the setup for the point product. When the agent executes the setup files, then the agent part is done and successful.  The failure then will be on the OS or point product. 
  • C:\windows\temp\mcafeelogs folder will then contain the install logs for the point product to look at for troubleshooting those failures. At that point, you would go to that point product team for assistance. 

Agent Deployment (Push Agent) 

• Review Server Task Log result and most importantly, server_servername.log (DB\Logs) 
• Keyword “push” in server log – don’t forget that if multiple handlers exist in an environment, the push could be in a different server log (when deploying, you can select the handler to use) 
• Relies heavily on access to \\machinename\admin$ of the endpoint. For all requirements and testing, see KB56386. 
  • You can test access to the client by going to Windows Explorer on the epo server, Computer tab in menu bar, map network drive to \\machinename\admin$.  For credentials, use the credentials you will be using to push agent with.  A successful connection would map to the client’s Windows directory.

Injection 

• Injection can occur when a third-party DLL that either has untrusted certs or no certs at all attempts to inject into McAfee processes, like McScript_InUse.exe. In that scenario, updates will end up failing with curl error 28 (meaning a timeout) and will be seen in the McScript.log or the McScript_deploy.log. 
• This is due to McAfee Agent’s Self-Protection functionality – the self-protection rules are working as designed in this scenario – we WANT to prevent the process (McScript_InUse.exe, in this example) from successfully making network connections because it could be compromised by a potentially malicious file. 
• Example from McScript.log: 
  • o    network    URL(https://172.x.x.x:443/Software/SiteStat.xml?hash={0e773b7e-9786-11e7-3115-73e51b00cce7}) request, failed with curl error 28, Response 0, Connect code 0, 
  • o    downloader                 Downloading file from https://172.x.x.x:443/Software/SiteStat.xml?hash={0e773b7e-9786-11e7-3115-73e51b00cce7} to C:\Windows\TEMP\SiteStat.xml failed.  
• The log that contains the rest of the data is called mfemactl.log. It will show entries like: 
  • C:\PROGRAM FILES\MCAFEE\AGENT\X86\MCSCRIPT_INUSE.EXE>(7208) was blocked from accessing('CREATE' (1)) <aac_object_section:c:\windows\syswow64\bmnet.dll <br=""> 
    </aac_object_section:c:\windows\syswow64\bmnet.dll> 
  • NOTE:  When looking at injection issues, it is important to note which process is doing the blocking and which process is blocked.  If we are doing the blocking, that is not injection.  If our process is the one blocked, then that is an injection problem.
• Run the sysprep tool first to see if there are any DLLs that it finds and trusts. 
• In the event sysprep is not able to resolve the injection, then review KB88085 to get a full understanding of injection issues and resolution options. 

Non-Windows Agent Guide 

Keep in mind: 

• The McAfee Agent has separate packages for the different platforms. For example – a Windows package, a Linux package, etc. These packages must be checked in to the ePO Master Repository separately. 
• The McAfee Agent can still be deployed (Push Agents) to non-Windows platforms, however it works entirely differently. Since a Windows deployment utilizes Windows file sharing, obviously that’s impossible for non-Windows clients. Instead, SSH protocol is utilized (port 22 by default). Red Hat/centOS have specific requirements to enable deployment and is a common source of push failures on those platforms. See the McAfee Agent Installation Guide for details. 
• The Agent still has three services on non-Windows platforms: masvc, macmnsvc and macompatsvc. 
• Non-Windows platforms are case-sensitive when working in the terminal/command line. Make sure your cases match, otherwise it will appear that the locations you’re attempting to access do not exist. 

Log locations and data collection and service information 

MacOS 

MAC MER tool: KB86785 

/Library/McAfee/agent/ (install files) 

/var/log/install.log (to view install logs) 

/var/McAfee/agent/ (data directory: includes logs, db files, etc.  Equivalent on Windows is ProgramData) 

/etc/ma.d/ (product plugins) 

To view the status of a service: 

Sudo /Library/McAfee/agent/scripts/ma status 

Stopping and starting services: 

Sudo /Library/McAfee/agent/scripts/ma start 

Sudo /Library/McAfee/agent/scripts/ma stop 

Sudo /Library/McAfee/agent/scripts/ma restart 

All other non-Windows platforms (Linux, UNIX, etc.) 

Linux MER tool: KB83005 

/opt/McAfee/agent (install files) 

/var/McAfee/agent (data directory) 

/etc/ma.d/ (product plugins) 

[Joshua]

The McAfee Agent General policy, deselect the option Accept connections only from the ePO server

DGCustomerFirst.