Error updating server certificate

[BKozubski]

I am having trouble updating the server certificate for ePO.  I have followed How to generate a custom SSL certificate for use with ePO using the OpenSSL toolkit (trellix.com) without any luck.  I followed the entire document and generated a new CSR and certificate, then converted the private key using the openssl rsa command for OpenSSL 1.0.  This never creates a PKCS1 formatted file, as it looks the same as a file in PKCS8 format.

I have had luck in changing the header and footer of the converted private key to include "RSA".  This gets me passed the error "server certificate could not be updated since the uploaded private key file did not contain a valid private key", however I get a new error of "Error occurred while decoding the private key's PEM".

[cdinet]

You can't just manually edit the pem file to change headers and footers.  Are you sure you are using openssl version 1?  This will tell you whether you have pkcs1 or pkcs8 format in an unedited pem file.

You can find the difference between the PKCS#8 and PKCS#1 private key files from the first line of text.

• PKCS#1 files specify the below algorithm:

-----BEGIN RSA PRIVATE KEY-----

• PKCS#8 files don't show the algorithm, and might also be encrypted:

-----BEGIN PRIVATE KEY-----

or

-----BEGIN ENCRYPTED PRIVATE KEY-----

• If you still get an error stating 'private key doesn't match the certificate' after following the above steps, you must make sure that both the hashes match. Validate the match by executing an MD5 check on the private key and certificate. 
  
  NOTE: If they don't match, the private key doesn't belong to that certificate.
   
  • ​openssl rsa -noout -modulus -in C:\ssl\keys\mcafee.key | openssl md5
  • openssl x509 -noout -modulus -in c:\ssl\keys\mcafee.crt | openssl md5

[BKozubski · Accepted solution]

I manually edited it to see if it made a difference.  I was running OpenSSL 1.0.X off a RHEL 7 box while following the documentation I referenced.  For whatever reason OpenSSL 1 was not changing the format of the PKCS8 pem file.  We were able to install OpenSSL 3.0.8 on a Windows machine and follow the documentation for the OpenSSL 3.0 commands and this resolved the issue.

[cdinet]

Openssl version 1 does not create the pem file in pkcs8 format.  Only 3.x and above.  So that command to convert the file would not have done anything for that.  There are instructions in the kb to remove the password from the pkcs1 file that openssl 1 would have created.  Either way, glad you got it working.