RHEV/oVirt Environment is Non Responsive with Errors as: Received fatal alert: certificate_expired

Assalamualaikum.

Issue
1. Certificates have expired and all hosts are non responsive
2. Getting error as below in the engine logs:
ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler_Worker-64) [6d32f092]
Command 'UploadStreamVDSCommand(HostName = xyz, UploadStreamVDSCommandParameters:{runAsync='true', hostId='12345dbe-1db1-11e1-ade1-00215e97f418'})' execution failed: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired

ERROR [org.ovirt.engine.core.bll.UploadStreamCommand] (DefaultQuartzScheduler_Worker-64) [6d32f092] Command'org.ovirt.engine.core.bll.UploadStreamCommand' failed: EngineException:org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: javax.net.ssl.SSLHandshakeException:
Received fatal alert: certificate_expired (Failed with error VDS_NETWORK_ERROR and code 5022)

Resolution
Starting with RHEV 3.5.4 and beyond, the engine-setup utility updates all the certificates automatically.

For details please see the following solution.
1. Take backup of RHEVM database
2. Then run the # engine-setup to renew the certificates.

Example, below will be asked when the engine setup is run, provide the input as `Yes` to renew the certificates

--== PKI CONFIGURATION ==--

One or more of the certificates should be renewed, because they expire soon or include an invalid expiry date, which is rejected by recent browsers.
If you choose "No", you will be asked again the next time you run Setup.
See https://access.redhat.com/solutions/1572983 for more details.
Renew certificates? (Yes, No) [No]: Yes

3. It will also upgrade to the latest minor version available for RHEV
4. If the hosts certificates are expired then one by one put the host in Maintenance mode, then remove it and re-add the same so the new certificates are generated.

Root Cause
1. PKI certificates expired since initial RHEV installation and should be renewed.
2. RHEV-M upgrade asks to renew certificate. What does it mean?

Diagnostic Steps
1. Collect the following information from affected environment to check the issue further:
2. On the RHEV-M machine:
# cd /etc/pki/ovirt-engine; ls -lh database.txt database.txt.attr serial.txt cacert.conf cert.conf
# openssl x509 -in /etc/pki/ovirt-engine/ca.pem -noout -text
# openssl x509 -in /etc/pki/ovirt-engine/apache-ca.pem -noout -text
# openssl x509 -in /etc/pki/ovirt-engine/engine.cer -noout -text

3. SSH into the affected host:
# ls -al /etc/pki/vdsm/certs/
# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -text

wassalam..

Advertisements
This entry was posted in Linux, oVirt, RedHat, RHEV, RHV, Virtualization and tagged , , , , . Bookmark the permalink.