KCLCHKBLK_4 – Database SOS

又一例:ORA-600 kclchkblk_4和2662故障

有客户恢复请求:由于未知原因导致aix环境的rac两台主机同时重启之后数据库无法正常启动,初步判断是由于写丢失导致故障（ORA-00742 ORA-00353）

Wed Feb 21 09:23:06 2024
ALTER DATABASE OPEN
This instance was first to open
Abort recovery for domain 0
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_5767246.trc:
ORA-01113: file 32 needs media recovery
ORA-01110: data file 32: '+DATA/xff/datafile/x5sys_cs.dbf'
ORA-1113 signalled during: ALTER DATABASE OPEN...
Wed Feb 21 09:23:27 2024
ALTER DATABASE RECOVER  datafile '+DATA/xff/datafile/x5sys_cs.dbf'  
Media Recovery Start
Serial Media Recovery started
WARNING! Recovering data file 32 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
Recovery of Online Redo Log: Thread 2 Group 14 Seq 48490 Reading mem 0
  Mem# 0: +DATA/xff/onlinelog/group_14.313.1060528521
Recovery of Online Redo Log: Thread 1 Group 7 Seq 64195 Reading mem 0
  Mem# 0: +DATA/xff/onlinelog/group_7.306.1060527979
Wed Feb 21 09:24:10 2024
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_5767246.trc:
ORA-00742: Log read detects lost write in thread %d sequence %d block %d
ORA-00334: archived log: '+DATA/xff/onlinelog/group_14.313.1060528521'
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_5767246.trc  (incident=336478):
ORA-00353: log corruption near block 139727 change 26346459680 time 02/20/2024 20:13:50
ORA-00312: online log 14 thread 2: '+DATA/xff/onlinelog/group_14.313.1060528521'

尝试屏蔽一致性强制拉库后数据库报ORA-600 kclchkblk_4
参考:ora-600 2662和ora-600 kclchkblk_4恢复、redo异常 ORA-600 kclchkblk_4 故障恢复

Wed Feb 21 09:55:26 2024
SMON: enabling cache recovery
Wed Feb 21 09:55:26 2024
Redo thread 2 internally disabled at seq 5 (CKPT)
Archived Log entry 112707 added for thread 2 sequence 4 ID 0xffffffffe144183b dest 1:
ARC0: Archiving disabled thread 2 sequence 5
Archived Log entry 112708 added for thread 2 sequence 5 ID 0xffffffffe144183b dest 1:
Wed Feb 21 09:55:28 2024
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6423264.trc  (incident=360479):
ORA-00600: internal error code, arguments: [kclchkblk_4], [6], [576721660], [6], [576702892]
Incident details in: /oracle/db/diag/rdbms/xff/xff1/incident/incdir_360479/xff1_ora_6423264_i360479.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Thread 1 advanced to log sequence 3 (LGWR switch)
  Current log# 7 seq# 3 mem# 0: +DATA/xff/onlinelog/group_7.306.1161510375
Archived Log entry 112709 added for thread 1 sequence 2 ID 0xffffffffe144183b dest 1:
Wed Feb 21 09:55:31 2024
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6423264.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kclchkblk_4], [6], [576721660], [6], [576702892]
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6423264.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kclchkblk_4], [6], [576721660], [6], [576702892]

后续处理中出现和这个错误类似的ORA-600 2662错误

Wed Feb 21 15:37:35 2024
SMON: enabling cache recovery
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6357664.trc  (incident=432423):
ORA-00600: internal error code, arguments: [2662], [6], [576742938], [6], [576834283], [12583104]
Incident details in: /oracle/db/diag/rdbms/xff/xff1/incident/incdir_432423/xff1_ora_6357664_i432423.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6357664.trc:
ORA-00600: internal error code, arguments: [2662], [6], [576742938], [6], [576834283], [12583104], [], [], [], [], [], []
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6357664.trc:
ORA-00600: internal error code, arguments: [2662], [6], [576742938], [6], [576834283], [12583104], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 6357664): terminating the instance due to error 600
Instance terminated by USER, pid = 6357664

通过对oracle scn进行修改,数据库open成功

SQL> recover database;
Media recovery complete.
SQL> 
SQL> 
SQL> 
SQL> oradebug setmypid
Statement processed.
SQL> oradebug DUMPvar SGA kcsgscn_
kcslf kcsgscn_ [700000000019B70, 700000000019BA0) = 00000000 00000000 00000000 00000000 
SQL> oradebug poke 
BEFORE: [700000000019B70, 700000000019B78) = 00000000 00000000
AFTER:  [700000000019B70, 700000000019B78) = 00000006 22710D2B
SQL> oradebug DUMPvar SGA kcsgscn_ 
kcslf kcsgscn_ [700000000019B70, 700000000019BA0) = 00000006 22710D2B 00000000 00000000 
SQL> alter database open;

Database altered.

后续检查发现obj$中的index异常（ORA-08102: index key not found, obj# 39）
类似文章:通过bbed修改obj$中dataobj$重现I_OBJ4索引报ORA-08102错误

SQL> truncate CLUSTER "SYS"."SMON_SCN_TO_TIME_AUX";

truncate CLUSTER "SYS"."SMON_SCN_TO_TIME_AUX"
                       *
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-08102: index key not found, obj# 39, file 1, block 967206 (2)

SQL> SQL> select object_name,object_type from dba_objects where object_id=39;

OBJECT_NAME                    OBJECT_TYPE
------------------------------ -------------------
I_OBJ4                         INDEX

对于此类问题使用非常规方法把obj$字典表进行重建（需要注意undo需要为自动管理方式,temp不能为空）,参考:
bootstrap$核心index(I_OBJ1,I_USER1,I_FILE#_BLOCK#,I_IND1,I_TS#,I_CDEF1等)异常恢复—ORA-00701错误解决

ora-600 2662和ora-600 kclchkblk_4恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ora-600 2662和ora-600 kclchkblk_4恢复

这两天连续处理两个case,一个是12.1.0.2版本数据库屏蔽一致性,强制open之后,报ORA-600 2662故障

这个错误本身是一个非常常见的错误,直接推scn即可解决,但是问题是12.1.0.2版本,oracle不允许以前常规的操作方法,就连oradebug都报错oradebug poke ORA-32521/ORA-32519故障解决,而且可以是rac环境,bbed修改文件头也相当麻烦,最后我们使用patch方法轻松解决

另外一例是11.2.0.4版本,强制open库报ora-600 kclchkblk_4

Wed Apr 28 21:25:38 2021
SMON: enabling cache recovery
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Errors in file /u01/app/oracle/diag/rdbms/dc/dc1/trace/dc1_ora_27832.trc  (incident=564430):
ORA-00600: internal error code, arguments: [kclchkblk_4], [2959], [904341694], [2959], [904131717], []
Incident details in: /u01/app/oracle/diag/rdbms/dc/dc1/incident/incdir_564430/dc1_ora_27832_i564430.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/dc/dc1/trace/dc1_ora_27832.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kclchkblk_4], [2959], [904341694], [2959], [904131717], []
Errors in file /u01/app/oracle/diag/rdbms/dc/dc1/trace/dc1_ora_27832.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kclchkblk_4], [2959], [904341694], [2959], [904131717], []
Error 704 happened during db open, shutting down database
USER (ospid: 27832): terminating the instance due to error 704
Instance terminated by USER, pid = 27832
ORA-1092 signalled during: alter database open resetlogs...

这个比较简单,参考redo异常 ORA-600 kclchkblk_4 故障恢复.

redo异常 ORA-600 kclchkblk_4 故障恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：redo异常 ORA-600 kclchkblk_4 故障恢复

朋友和我说,他们数据库由于存储控制器异常,导致数据库无法正常启动.
数据库recover database 提示需要已经覆盖的redo

-bash-3.2$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.5.0 - Production on Mon Nov 3 17:23:35 2014
Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, Real Application Clusters, Data Mining and Real Application Testing options
SQL> recover database;
ORA-00279: change 1639063379 generated at 11/01/2014 12:06:33 needed for thread
2
ORA-00289: suggestion :
/export/home/oracle/product/10.2.0/db_1/dbs/arch2_29919_790965041.dbf
ORA-00280: change 1639063379 for thread 2 is in sequence #29919
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.
SQL> select thread#,group#,sequence# from v$log;
   THREAD#     GROUP#  SEQUENCE#
---------- ---------- ----------
         1          1      30261
         1          2      30262
         1          3      30258
         1          4      30259
         1          5      30260
         2          6      29966
         2          7      29967
         2          8      29968
         2          9      29969
         2         10      29970
10 rows selected.
SQL> select group#,member from v$logfile;
    GROUP# MEMBER
---------- --------------------------------------------------
         1 /dev/md/oradg/rdsk/d115
         2 /dev/md/oradg/rdsk/d116
         3 /dev/md/oradg/rdsk/d117
         4 /dev/md/oradg/rdsk/d118
         5 /dev/md/oradg/rdsk/d119
         6 /dev/md/oradg/rdsk/d120
         7 /dev/md/oradg/rdsk/d121
         8 /dev/md/oradg/rdsk/d122
         9 /dev/md/oradg/rdsk/d123
        10 /dev/md/oradg/rdsk/d124
10 rows selected.
SQL> recover database;
ORA-00279: change 1639063379 generated at 11/01/2014 12:06:33 needed for thread 2
ORA-00289: suggestion : /export/home/oracle/product/10.2.0/db_1/dbs/arch2_29919_790965041.dbf
ORA-00280: change 1639063379 for thread 2 is in sequence #29919
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.

数据库没归档，redo覆盖，无法继续恢复,使用_allow_resetlogs_corruption屏蔽前滚,继续恢复

SQL> startup mount pfile='/tmp/pfile.txt';
ORACLE instance started.
Total System Global Area 2097152000 bytes
Fixed Size                  2053120 bytes
Variable Size            1090522112 bytes
Database Buffers          855638016 bytes
Redo Buffers              148938752 bytes
Database mounted.
SQL> recover database until cancel;
ORA-00279: change 1639063379 generated at 11/01/2014 12:58:20 needed for thread
1
ORA-00289: suggestion :
/export/home/oracle/product/10.2.0/db_1/dbs/arch1_30262_790965041.dbf
ORA-00280: change 1639063379 for thread 1 is in sequence #30262
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/dev/md/oradg/rdsk/d105'
ORA-01112: media recovery not started
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced

查看alert日志

Mon Nov 03 17:37:11 CST 2014
Completed crash recovery at
 Thread 2: logseq 1, block 3, scn 1639083385
 0 data blocks read, 0 data blocks written, 1 redo blocks read
Picked broadcast on commit scheme to generate SCNs
Mon Nov 03 17:37:12 CST 2014
Thread 2 advanced to log sequence 2 (thread open)
Thread 2 opened at log sequence 2
  Current log# 7 seq# 2 mem# 0: /dev/md/oradg/rdsk/d121
Successful open of redo thread 2
Mon Nov 03 17:37:12 CST 2014
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Nov 03 17:37:13 CST 2014
SMON: enabling cache recovery
Mon Nov 03 17:37:13 CST 2014
Errors in file /export/home/oracle/admin/iesdb/udump/iesdb2_ora_15700.trc:
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [1640589405], [0], [1639117006], [], [], []
Mon Nov 03 17:37:14 CST 2014
Errors in file /export/home/oracle/admin/iesdb/udump/iesdb2_ora_15700.trc:
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [1640589405], [0], [1639117006], [], [], []
Mon Nov 03 17:37:14 CST 2014
Error 600 happened during db open, shutting down database
USER: terminating instance due to error 600
Instance terminated by USER, pid = 15700
ORA-1092 signalled during: alter database open...

参考:In 10.1.0.2: ORA-600 [kclchkblk_4] and ORA-600 [2662] After Recovery of Database (Doc ID 275902.1),删除tempfile继续打开数据库

SQL> alter database tempfile '/dev/md/oradg/rdsk/d109' drop;
Database altered.
SQL> c/109/110
  1* alter database tempfile '/dev/md/oradg/rdsk/d110' drop
SQL> /
Database altered.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced

数据库依然报ORA-00600kclchkblk_4]错误,因此解决该问题选择使用bbed修改文件头scn来完成,具体参考类似文章:使用bbed修复损坏datafile header

-bash-3.2$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.5.0 - Production on Mon Nov 3 17:41:17 2014
Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.
Connected to an idle instance.
SQL> startup pfile='/tmp/pfile.txt' mount;
ORACLE instance started.
Total System Global Area 2097152000 bytes
Fixed Size                  2053120 bytes
Variable Size            1090522112 bytes
Database Buffers          855638016 bytes
Redo Buffers              148938752 bytes
Database mounted.
SQL> alter database open;
Database altered.
SQL> alter tablespace temp add tempfile '/dev/md/oradg/rdsk/d109' size 8388608000 autoextend off;
Tablespace altered.
SQL> alter tablespace temp add tempfile '/dev/md/oradg/rdsk/d110' size 8388608000 autoextend off;
Tablespace altered.
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area 2097152000 bytes
Fixed Size                  2053120 bytes
Variable Size            1090522112 bytes
Database Buffers          855638016 bytes
Redo Buffers              148938752 bytes
Database mounted.
Database opened.

至此数据库恢复完成,建议逻辑方式重建数据库

标签归档：KCLCHKBLK_4