联系：手机/微信(+86 17813235971) QQ(107644445)
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
客户反馈有一套19c 2节点rac，断电之后，一个节点数据库无法正常启动,通过crsctl命令查看发现crs进程没有正常启动
[root@xifenf1 ~]# /u01/app/19.0/grid/bin/crsctl status res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE xifenf1 STABLE
ora.crf
1 ONLINE ONLINE xifenf1 STABLE
ora.crsd
1 ONLINE OFFLINE STABLE
ora.cssd
1 ONLINE ONLINE xifenf1 STABLE
ora.cssdmonitor
1 ONLINE ONLINE xifenf1 STABLE
ora.ctssd
1 ONLINE OFFLINE STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE xifenf1 STABLE
ora.gipcd
1 ONLINE ONLINE xifenf1 STABLE
ora.gpnpd
1 ONLINE ONLINE xifenf1 STABLE
ora.mdnsd
1 ONLINE ONLINE xifenf1 STABLE
ora.storage
1 ONLINE ONLINE xifenf1 STABLE
--------------------------------------------------------------------------------
查看crs的alert日志发现集群时间间隔超过600s，无法启动csst进程
2024-06-11 17:33:09.953 [OCSSD(5020)]CRS-1605: CSSD voting file is online: /dev/asm_ocr5; details in /u01/app/grid/diag/crs/xifenf1/crs/trace/ocssd.trc. 2024-06-11 17:33:09.956 [OCSSD(5020)]CRS-1605: CSSD voting file is online: /dev/asm_ocr1; details in /u01/app/grid/diag/crs/xifenf1/crs/trace/ocssd.trc. 2024-06-11 17:33:10.024 [OCSSD(5020)]CRS-1605: CSSD voting file is online: /dev/asm_ocr2; details in /u01/app/grid/diag/crs/xifenf1/crs/trace/ocssd.trc. 2024-06-11 17:33:10.031 [OCSSD(5020)]CRS-1605: CSSD voting file is online: /dev/asm_ocr4; details in /u01/app/grid/diag/crs/xifenf1/crs/trace/ocssd.trc. 2024-06-11 17:33:10.040 [OCSSD(5020)]CRS-1605: CSSD voting file is online: /dev/asm_ocr3; details in /u01/app/grid/diag/crs/xifenf1/crs/trace/ocssd.trc. 2024-06-11 17:33:11.900 [OCSSD(5020)]CRS-1601: CSSD Reconfiguration complete. Active nodes are xifenf1 xifenf2 . 2024-06-11 17:33:13.344 [OCSSD(5020)]CRS-1720: Cluster Synchronization Services daemon (CSSD) is ready for operation. 2024-06-11 17:33:13.809 [OCTSSD(5488)]CRS-8500: Oracle Clusterware OCTSSD process is starting with operating system process ID 5488 2024-06-11 17:33:16.017 [OCTSSD(5488)]CRS-2407: The new Cluster Time Synchronization Service reference node is host xifenf2. 2024-06-11 17:33:16.018 [OCTSSD(5488)]CRS-2401: The Cluster Time Synchronization Service started on host xifenf1. 2024-06-11 17:33:16.105 [OCTSSD(5488)]CRS-2419: The clock on host xifenf1 differs from mean cluster time by 1031504618 microseconds. The Cluster Time Synchronization Service will not perform time synchronization because the time difference is beyond the permissible offset of 600 seconds. Details in /u01/app/grid/diag/crs/xifenf1/crs/trace/octssd.trc. 2024-06-11 17:33:16.579 [OCTSSD(5488)]CRS-2402: The Cluster Time Synchronization Service aborted on host xifenf1. Details at (:ctsselect_mstm4:) in /u01/app/grid/diag/crs/xifenf1/crs/trace/octssd.trc.
查看主机时间
[grid@xifenf1 ~]$ date ;ssh xifenf2 date Tue Jun 11 17:54:09 CST 2024 Tue Jun 11 18:04:34 CST 2024
修改主机时间
[root@xifenf1 ~]# date -s "20240611 18:06:00" Tue Jun 11 18:06:00 CST 2024 [root@xifenf1 ~]# su - grid Last login: Tue Jun 11 17:37:53 CST 2024 on pts/0 [grid@xifenf1 ~]$ date ;ssh xifenf2 date Tue Jun 11 18:06:09 CST 2024 Tue Jun 11 18:05:34 CST 2024
重启crs
[root@xifenf1 ~]# /u01/app/19.0/grid/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'xifenf1'
CRS-2673: Attempting to stop 'ora.storage' on 'xifenf1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'xifenf1'
CRS-2673: Attempting to stop 'ora.crf' on 'xifenf1'
CRS-2677: Stop of 'ora.storage' on 'xifenf1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'xifenf1'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'xifenf1'
CRS-2677: Stop of 'ora.mdnsd' on 'xifenf1' succeeded
CRS-2677: Stop of 'ora.crf' on 'xifenf1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'xifenf1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'xifenf1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'xifenf1'
CRS-2677: Stop of 'ora.cssd' on 'xifenf1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'xifenf1'
CRS-2673: Attempting to stop 'ora.gipcd' on 'xifenf1'
CRS-2677: Stop of 'ora.gpnpd' on 'xifenf1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'xifenf1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'xifenf1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@xifenf1 ~]# /u01/app/19.0/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@xifenf1 ~]# /u01/app/19.0/grid/bin/crsctl status res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE xifenf1 STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE xifenf1 STABLE
ora.crf
1 ONLINE ONLINE xifenf1 STABLE
ora.crsd
1 ONLINE ONLINE xifenf1 STABLE
ora.cssd
1 ONLINE ONLINE xifenf1 STABLE
ora.cssdmonitor
1 ONLINE ONLINE xifenf1 STABLE
ora.ctssd
1 ONLINE ONLINE xifenf1 ACTIVE:35600,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE xifenf1 STABLE
ora.gipcd
1 ONLINE ONLINE xifenf1 STABLE
ora.gpnpd
1 ONLINE ONLINE xifenf1 STABLE
ora.mdnsd
1 ONLINE ONLINE xifenf1 STABLE
ora.storage
1 ONLINE ONLINE xifenf1 STABLE
--------------------------------------------------------------------------------