Community Help Hub

GJGVA · ‎07-23-2019

Hello,

I've recently upgraded epo from 5.9 to 5.10. On 5.9, we were in the process of migrating the SHA1 to SHA2 with more than 85% of the agents updated but we didn't finish prior going to 5.10.

I've went back yesterday on this migration (5.10) and I had to restart the process over again. At this moment, 10% only replicated the new certificate and on a lot of stations, I've got communication issues between the server and the agent:

"Agent failed to communicate with the ePO server"
"Agent failed to send events"

I've tried the following KB87017 https://kc.mcafee.com/corporate/index?page=content&id=KB87017

The config works but the communication issue persists.

I've checked the logs on the server and here is what I can see:

orion.log (only full of this message):

2019-07-23 09:39:49,057 ERROR [http-nio-8444-exec-21] server.OrionLoginModule - Error occured while updating last logon time in the database
2019-07-23 09:39:49,151 ERROR [http-nio-8444-exec-7] server.OrionLoginModule - Error occured while updating last logon time in the database
2019-07-23 09:39:49,322 ERROR [http-nio-8444-exec-13] server.OrionLoginModule - Error occured while updating last logon time in the database
2019-07-23 09:39:49,385 ERROR [http-nio-8444-exec-1] server.OrionLoginModule - Error occured while updating last logon time in the database
2019-07-23 09:39:52,520 ERROR [http-nio-8444-exec-25] server.OrionLoginModule - Error occured while updating last logon time in the database
2019-07-23 09:39:52,583 ERROR [http-nio-8444-exec-14] server.OrionLoginModule - Error occured while updating last logon time in the database
2019-07-23 09:39:52,754 ERROR [http-nio-8444-exec-4] server.OrionLoginModule - Error occured while updating last logon time in the database
2019-07-23 09:39:52,817 ERROR [http-nio-8444-exec-6] server.OrionLoginModule - Error occured while updating last logon time in the database
2019-07-23 09:39:53,098 ERROR [http-nio-8444-exec-10] server.OrionLoginModule - Error occured while updating last logon time in the database
2019-07-23 09:39:53,238 ERROR [http-nio-8444-exec-21] server.OrionLoginModule - Error occured while updating last logon time in the database
2019-07-23 09:39:53,472 ERROR [http-nio-8444-exec-2] server.OrionLoginModule - Error occured while updating last logon time in the database
2019-07-23 09:39:53,566 ERROR [http-nio-8444-exec-13] server.OrionLoginModule - Error occured while updating last logon time in the database

stderr.log

Jul 23, 2019 9:34:46 AM org.apache.catalina.realm.JAASRealm authenticate
WARNING: Login exception authenticating username ""
javax.security.auth.login.LoginException: Login Failure: all modules ignored
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:906)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:588)
at org.apache.catalina.realm.JAASRealm.authenticate(JAASRealm.java:427)
at org.apache.catalina.realm.JAASRealm.authenticate(JAASRealm.java:348)
at org.apache.catalina.authenticator.BasicAuthenticator.authenticate(BasicAuthenticator.java:127)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:566)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
at com.mcafee.orion.core.server.mfsvalve.ValveContext.invokeNextInChain(ValveContext.java:27)
at com.mcafee.orion.core.server.LoginValidationValveHook.invoke(LoginValidationValveHook.java:89)
at com.mcafee.orion.core.server.mfsvalve.ValveContext.invokeNextInChain(ValveContext.java:25)
at com.mcafee.orion.core.server.AccessControlValveHook.invoke(AccessControlValveHook.java:81)
at com.mcafee.orion.core.server.mfsvalve.ValveContext.invokeNextInChain(ValveContext.java:25)
at com.mcafee.orion.core.server.mfsvalve.MfsValve.invoke(MfsValve.java:39)
at com.mcafee.orion.core.server.AjaxValve.invoke(AjaxValve.java:84)
at com.mcafee.orion.core.server.OrionUserSetupValve.invoke(OrionUserSetupValve.java:34)
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:270)
at com.mcafee.orion.core.server.OrionSingleSignOn.invoke(OrionSingleSignOn.java:203)
at com.mcafee.orion.core.server.ClientCertValve.invoke(ClientCertValve.java:60)
at com.mcafee.orion.core.server.ExternalAuthenticationStrategyExtPointValve.invoke(ExternalAuthenticationStrategyExtPointValve.java:140)
at com.mcafee.orion.core.server.ParameterEncodingValve.invoke(ParameterEncodingValve.java:34)
at com.mcafee.orion.core.server.ThreadLocalInfoCleanupValve.invoke(ThreadLocalInfoCleanupValve.java:25)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1137)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

FYI, I can login on ePO server without any issue.

all services are running properly besides that.

on epo agent client, I can see this on masvc_***.log

2019-07-23 09:42:53.633 masvc(7916.248) network.Debug: curl perform cb called with event 1
2019-07-23 09:42:53.633 masvc(7916.248) curl.Trace: text 47 TLSv1.2 (IN), TLS handshake, Server hello (2):
2019-07-23 09:42:53.633 masvc(7916.248) curl.Trace: text 47 TLSv1.2 (IN), TLS handshake, Certificate (11):
2019-07-23 09:42:53.633 masvc(7916.248) curl.Trace: text 44 TLSv1.2 (OUT), TLS alert, Server hello (2):
2019-07-23 09:42:53.633 masvc(7916.248) curl.Trace: text 64 SSL certificate problem: unable to get local issuer certificate
2019-07-23 09:42:53.633 masvc(7916.248) curl.Trace: text 22 Closing connection 23
2019-07-23 09:42:53.633 masvc(7916.248) network.Debug: URL(https://GC***:443/spipe/pkg?AgentGuid={3f67ae16-ad15-11e9-0dc7-c85b7604630a}&Source=Agent_3.0.0) request, socket(1132) cb with action REMOVE [curl handle = 0000000002832EC0].
2019-07-23 09:42:53.633 masvc(7916.248) network.Debug: URL(https://GC***:443/spipe/pkg?AgentGuid={3f67ae16-ad15-11e9-0dc7-c85b7604630a}&Source=Agent_3.0.0) request, stoping watcher
2019-07-23 09:42:53.633 masvc(7916.248) network.Trace: curl close socket callback, socket 1132
2019-07-23 09:42:53.633 masvc(7916.248) network.Notice: URL(https://GC***:443/spipe/pkg?AgentGuid={3f67ae16-ad15-11e9-0dc7-c85b7604630a}&Source=Agent_3.0.0) request failed with curl error <60>, response code <0>, http connect code 0
2019-07-23 09:42:53.633 masvc(7916.248) network.Debug: URL(https://GC***:443/spipe/pkg?AgentGuid={3f67ae16-ad15-11e9-0dc7-c85b7604630a}&Source=Agent_3.0.0) request, completed with Response 0 calling final callback
2019-07-23 09:42:53.633 masvc(7916.248) ahclient.Info: Network library rc = <1007>, Agent handler reports response code <0>.
2019-07-23 09:42:53.633 masvc(7916.248) ahclient.Debug: Proxy configuration setting "Use System Proxy"

ahclient.debug: no relay found in the network, calling final cb.
ahclient.error: agent failed to communicate with ePO server

Any idea how I can tackle/fix this?

Looking forward to hearing from you folks.

D

cdinet · ‎07-23-2019

With apache still stopped, run this command, replacing servername:port with your correct servername and port you are using.

http://servername:8443/remote/epo.command.sareRootAHCertToDb

Then start the server service and see if you get the same server certificate check failed error or not. You don't need to do anything on agent handlers, but after doing that, if apache starts working, then check the agent handler server logs for any errors. They should be located under program files (x86)/mcafee/agent handler/db/logs.

Was my reply helpful?
If this information was helpful in any way or answered your question, will you please select Accept as Solution in my reply and together we can help other members?

View solution in original post

cdinet · ‎07-24-2019

so sorry, typo - should be epo.command.saveRootAHCertToDb

Was my reply helpful?
If this information was helpful in any way or answered your question, will you please select Accept as Solution in my reply and together we can help other members?

View solution in original post

cdinet · ‎07-23-2019

Since they are failing to communicate, you can try one thing first. The main error is this: unable to get local issuer certificate.

How is your update task configured that they currently have - update all products, or just selected items? Are the agent key updater and msgbus cert updater packages enabled for updates and checked into the current branch? If not, then test on a few systems by sending a run client task now update task including only those 2 packages and see if it makes any difference in communication. If it does, then run that on small groups at a time to get them updated. If not, you may need to redeploy agents.

It is not a good idea to upgrade epo in the middle of cert migration, but maybe this will help them update the certs.

Was my reply helpful?
If this information was helpful in any way or answered your question, will you please select Accept as Solution in my reply and together we can help other members?

GJGVA · ‎07-23-2019

Hi,

thanks for your reply

Currently, the task scheduler is supposed to run 5 minutes after computer started and updates everything.

I've now 16% of the stations resynced.

I've tried your suggestion without success. I've tried to redeploy on 2 computers without success.

I've tried to redeploy using the new framepkg file generated right after the new certificate generation and I'm having the same results - unable to communicate with ePO server.

I've ran on those 2 computers the McAfeeEndpointProductRemoval_19.5.0.17 and ran again the latest framepkg.exe available on the source folder of the ePO server and same result: can't communicate with ePO server.

Any ideas?

cdinet · ‎07-23-2019

It might be a server side issue. If you look at the server logs on the epo server and agent handlers, do you see certificate or other communication errors? They are located in the epo/ah install directory, db\logs.

Was my reply helpful?
If this information was helpful in any way or answered your question, will you please select Accept as Solution in my reply and together we can help other members?

GJGVA · ‎07-23-2019

Hi

Thanks for guidance.

here is the server log file
20190722154205 I #01528 NAIMSERV Database initialization: Starting.
20190722154205 I #01528 NAIMSERV Database initialization: Succeeded.
20190722154205 I #01528 NAIMSERV Policy Manager initialization: Starting.
20190722154206 I #01528 EPODAL Succesfully initialized database access for [GC***\EPOSERVER,49715].[ePO_GC***]
20190722154206 I #01528 NAIMSERV Policy Manager initialization: Succeeded.
20190722154206 I #01528 NAIMSERV Server state at startup: Enabled
20190722154206 I #01528 NAIMSERV Generating temporary Apache user
20190722154206 I #01528 NAIMSERV Checking to see if the ePO server (GC***:8443) is available. We will try 12 times.
20190722154206 I #01528 NAIMSERV The Agent Handler successfully connected to the ePO server.
20190722154208 E #01528 NAIMSERV servinit.cpp(606): The server certificate check failed. This means that there is a discrpency between the certificates stored
20190722154208 E #01528 NAIMSERV in the server keystore and the certificates that the agent machines will use. To protect the server from getting
20190722154208 E #01528 NAIMSERV into a worse state we are shutting down the Agent Handler.
20190722154208 I #01528 NAIMSERV Shutting down server...
20190722154208 I #01528 NAIMSERV Releasing File Locks...
20190722154208 I #01528 NAIMSERV Cleaning up temp directory...
20190722154208 I #01528 NAIMSERV ePolicy Orchestrator server stopped.

for the epo/ah install directory logs, I can't find them sorry.

D

cdinet · ‎07-23-2019

With apache still stopped, run this command, replacing servername:port with your correct servername and port you are using.

http://servername:8443/remote/epo.command.sareRootAHCertToDb

Then start the server service and see if you get the same server certificate check failed error or not. You don't need to do anything on agent handlers, but after doing that, if apache starts working, then check the agent handler server logs for any errors. They should be located under program files (x86)/mcafee/agent handler/db/logs.

Was my reply helpful?
If this information was helpful in any way or answered your question, will you please select Accept as Solution in my reply and together we can help other members?

GJGVA · ‎07-24-2019

Hi,

important information, the given logs stopped right after the SHA migration started.

Since then, no more logs were generated on the server less for the two following files :

EpoApSvc_GC***.log (full of 20190724113118 I #07432 RMANJNI Getting license data...)
Replication.log

About the Agent Handler, one of the Agent Handler remote server has the following logs (no agent handler folder on ePO server):
server_GC***.log
20190724124824 I #03208 NAIMSERV Initializing server...
20190724124824 I #03208 NAIMSERV McAfee ePO 5.10.0.2428
20190724124824 I #03208 NAIMSERV Server name: GC***B (GC***B)
20190724124824 I #03208 NAIMSERV Platform: Server 6.3 McAfee\Agent Handler\
20190724124824 I #03208 NAIMSERV Processors: 1
20190724124824 I #03208 NAIMSERV Architecture: 64-bit
20190724124824 I #03208 NAIMSERV Physical memory: 4095 MB
20190724124824 I #03208 NAIMSERV Database initialization: Starting.
20190724124824 I #03208 NAISIGN Found master install key, decoding
20190724124824 I #03208 MFEFIPS Loading: "C:\Program Files (x86)\McAfee\Agent Handler", Role = Officer, Mode = Normal
20190724124824 I #03208 MFEFIPS Module Initialized.
20190724124824 I #03208 MFEFIPS MFEFIPS_Status() returned 1
20190724124824 I #03208 MFEFIPS Loading: "C:\Program Files (x86)\McAfee\Agent Handler", Role = Officer, Mode = Normal
20190724124824 I #03208 MFEFIPS Module Initialized.
20190724124824 I #03208 MFEFIPS MFEFIPS_Status() returned 1
20190724124824 I #03208 EPODAL Using SQL Authentication for [GC***\EPOSERVER,49715].[ePO_GC***]
20190724124824 I #03208 EPODAL Succesfully initialized database access for [GC***\EPOSERVER,49715].[ePO_GC***]
20190724124824 I #03208 NAIMSERV Database initialization: Succeeded.
20190724124824 I #03208 NAIMSERV Policy Manager initialization: Starting.
20190724124824 I #03208 EPODAL Succesfully initialized database access for [GC***\EPOSERVER,49715].[ePO_GC***]
20190724124824 I #03208 NAIMSERV Policy Manager initialization: Succeeded.
20190724124824 I #03208 NAIMSERV Server state at startup: Enabled
20190724124824 I #03208 NAIMSERV Generating temporary Apache user
20190724124824 E #03208 MFEFIPS PKCS12CertStore.cpp(237): R_PKCS12_decode() failure: 10039
20190724124824 E #03208 NAIMSERV common\AgentHandlerKeyService.cpp(130): Failed to open the certificate store.
20190724124824 I #03208 NAIMSERV Checking to see if the ePO server (GC***:8443) is available. We will try 12 times.
20190724124825 I #03208 NAIMSERV The Agent Handler successfully connected to the ePO server.
20190724124825 E #03208 MFEFIPS PKCS12CertStore.cpp(237): R_PKCS12_decode() failure: 10039
20190724124825 E #03208 NAIMSERV common\AgentHandlerKeyService.cpp(130): Failed to open the certificate store.
20190724124826 E #03208 NAIMSERV servinit.cpp(606): The server certificate check failed. This means that there is a discrpency between the certificates stored
20190724124826 E #03208 NAIMSERV in the server keystore and the certificates that the agent machines will use. To protect the server from getting
20190724124826 E #03208 NAIMSERV into a worse state we are shutting down the Agent Handler.
20190724124826 I #03208 NAIMSERV Shutting down server...
20190724124826 I #03208 NAIMSERV Releasing File Locks...
20190724124826 I #03208 NAIMSERV Cleaning up temp directory...
20190724124826 I #03208 NAIMSERV ePolicy Orchestrator server stopped.
20190724124826 I #03208 MOD_EPOREPO Database initialization: Starting.
20190724124826 I #03208 NAISIGN Found master install key, decoding
20190724124826 I #03208 MFEFIPS Loading: "C:\Program Files (x86)\McAfee\Agent Handler", Role = Officer, Mode = Normal
20190724124826 I #03208 MFEFIPS Module Initialized.
20190724124826 I #03208 MFEFIPS MFEFIPS_Status() returned 1
20190724124826 I #03208 MOD_EPOREPO Database initialization: Succeeded.
20190724124826 I #03208 MOD_EPOREPO Master Repository is considered local. No impersonation will be used.

This is it :

20190724124824 E #03208 MFEFIPS PKCS12CertStore.cpp(237): R_PKCS12_decode() failure: 10039
20190724124824 E #03208 NAIMSERV common\AgentHandlerKeyService.cpp(130): Failed to open the certificate store.

shall I proceed with your suggestion?

Cheers,

D

cdinet · ‎07-24-2019

Yes, please do.

Was my reply helpful?
If this information was helpful in any way or answered your question, will you please select Accept as Solution in my reply and together we can help other members?

GJGVA · ‎07-24-2019

I got this error :

Error 1 :
No such command: epo.command.sareRootAHCertToDb

Any thougts?

D

cdinet · ‎07-24-2019

so sorry, typo - should be epo.command.saveRootAHCertToDb

Was my reply helpful?
If this information was helpful in any way or answered your question, will you please select Accept as Solution in my reply and together we can help other members?

ePO 5.10 SHA1 to SHA2 migration - agent-server communication fails

Re: ePO 5.10 SHA1 to SHA2 migration - agent-server communication fails

Re: ePO 5.10 SHA1 to SHA2 migration - agent-server communication fails

Re: ePO 5.10 SHA1 to SHA2 migration - agent-server communication fails

Re: ePO 5.10 SHA1 to SHA2 migration - agent-server communication fails

Re: ePO 5.10 SHA1 to SHA2 migration - agent-server communication fails

Re: ePO 5.10 SHA1 to SHA2 migration - agent-server communication fails

Re: ePO 5.10 SHA1 to SHA2 migration - agent-server communication fails

Re: ePO 5.10 SHA1 to SHA2 migration - agent-server communication fails

Re: ePO 5.10 SHA1 to SHA2 migration - agent-server communication fails

Re: ePO 5.10 SHA1 to SHA2 migration - agent-server communication fails

Re: ePO 5.10 SHA1 to SHA2 migration - agent-server communication fails

Community Help Hub

Join the Community