Oracle Real Application Cluster (RAC) provides a redundant solution to oracle database instances by combining multiple oracle instances in a cluster. The Cluster Ready Service (CRS) and Cluster Synchronizing Service (CSS) are among the major functions of the RAC solution which plays a major role in monitoring the high availability functions , status of the peer member nodes , monitoring the listener and DB instances.
When a server running Oracle RAC is booted , the first service started is the CRS cluster which brings up the required program functions such as CSS , ASM , Listener , HA functions , Event management functions and finally the DB instance.
How to resolve error CRS-2674 , CRS-4000
When trying to start a cluster , sometimes below error may be encountered
CRS-2672: Attempting to start 'ora.crf' on 'sharedb2'
CRS-2672: Attempting to start 'ora.cssd' on 'sharedb2'
CRS-2672: Attempting to start 'ora.diskmon' on 'sharedb2'
CRS-2676: Start of 'ora.diskmon' on 'sharedb2' succeeded
CRS-2676: Start of 'ora.crf' on 'sharedb2' succeeded
CRS-2674: Start of 'ora.cssd' on 'sharedb2' failed
CRS-2679: Attempting to clean 'ora.cssd' on 'sharedb2'
CRS-2681: Clean of 'ora.cssd' on 'sharedb2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'sharedb2'
CRS-2677: Stop of 'ora.crf' on 'sharedb2' succeeded
CRS-4000: Command Start failed, or completed with errors.
Possible Solution
First step is to analyze the CRS alert.log which may give a hint about the issue
In our setup , we have oracle files installed under /usr/oracle/
/usr/oracle/app/diag/crs/sharedb1/crs/trace/alert.log
In our case , we were hit with below errors in the log file , which gave us a hint about network connectivity issue between the DB servers.
16:54:07.539 [OHASD(23075)]CRS-8011: reboot advisory message from host: sharedb1, component: cssmonit, with time stamp: L-2022-03-02-10:29:36.409
16:54:07.539 [OHASD(23075)]CRS-8013: reboot advisory message text: oracssdmonitor is rebooting this node due to network timeout (no network activity for 28500 milliseconds).
16:54:07.539 [OHASD(23075)]CRS-8011: reboot advisory message from host: sharedb1, component: cssagent, with time stamp: L-2022-03-02-10:29:36.408
16:54:07.539 [OHASD(23075)]CRS-8013: reboot advisory message text: oracssdagent is rebooting this node due to network timeout (no network activity for 28500 milliseconds).
16:54:07.539 [OHASD(23075)]CRS-8017: location: /etc/oracle/lastgasp has 2 reboot advisory log files, 2 were announced and 0 errors occurred
Oracle RAC uses the private IP configured for internode communication. On trying to ping the peer member node peer IP , there was no response which indicated a clear communication issue.
On resolving the network connectivity and restarting the oracle cluster services , CRS module and related services gets started
Login as root user to oracle cluster ware directory
# cd /opt/grid/bin
Stop the cluster services using crsctl command
# ./crsctl stop cluster
Start the cluster services using crsctl command
# ./crsctl start cluster
CRS-2672: Attempting to start 'ora.evmd' on 'sharedb2'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'sharedb2'
CRS-2676: Start of 'ora.cssdmonitor' on 'sharedb2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'sharedb2'
CRS-2672: Attempting to start 'ora.diskmon' on 'sharedb2'
CRS-2676: Start of 'ora.diskmon' on 'sharedb2' succeeded
CRS-2676: Start of 'ora.evmd' on 'sharedb2' succeeded
CRS-2676: Start of 'ora.cssd' on 'sharedb2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'sharedb2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'sharedb2'
CRS-2676: Start of 'ora.ctssd' on 'sharedb2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'sharedb2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'sharedb2'
CRS-2676: Start of 'ora.asm' on 'sharedb2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'sharedb2'
CRS-2676: Start of 'ora.storage' on 'sharedb2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'sharedb2'
CRS-2676: Start of 'ora.crsd' on 'sharedb2' succeeded
Hope this is useful to few members debugging Oracle RAC CRS issues