Database SOS – Professional Database Recovery Support@+8617813235971

ORA-600 kcratr_nab_less_than_odr和ORA-600 2662故障处理

标题：ORA-600 kcratr_nab_less_than_odr和ORA-600 2662故障处理

异常断电之后,oracle启动报ORA-600 kcratr_nab_less_than_odr错误

Sun Aug 17 11:06:09 2025
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 11 processes
Started redo scan
Completed redo scan
 read 0 KB redo, 0 data blocks need recovery
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [40785], [33267], [40630]
Sun Aug 17 11:06:20 2025
Aborting crash recovery due to error 600
Sun Aug 17 11:06:20 2025
Trace dumping is performing id=[cdmp_20250817110620]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_2920.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [40785], [33267], [40630]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_2920.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [40785], [33267], [40630]
ORA-600 signalled during: ALTER DATABASE OPEN...

这个错误处理有多次处理经验
12c启动报kcratr_nab_less_than_odr
又一例ORA-600 kcratr_nab_less_than_odr
ORA-600 kcratr_nab_less_than_odr故障解决
 差点被误操作的ORA-600 kcratr_nab_less_than_odr故障
 11.2.0.4库中遇到ORA-600 kcratr_nab_less_than_odr报错
 ORA-600 kcratr_nab_less_than_odr和ORA-600 4194故障处理
一般重建ctl或者using backup ctl方式恢复即可实现0丢失打开库,但是这个库尝试打开报ORA-600 2662错误

SQL> startup nomount;
ORACLE 例程已经启动。

Total System Global Area 6847938560 bytes
Fixed Size                  2188768 bytes
Variable Size            4680845856 bytes
Database Buffers         2147483648 bytes
Redo Buffers               17420288 bytes
SQL> @rectl.sql

控制文件已创建。

SQL>
SQL>
SQL> recover database;
完成介质恢复。
SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [4413], [1200914792],
[4413], [1201104184], [12583040], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [2662], [4413], [1200914791],
[4413], [1201104184], [12583040], [], [], [], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [4413], [1200914789],
[4413], [1201104184], [12583040], [], [], [], [], [], []
进程 ID: 3132
会话 ID: 3018 序列号: 1

对于这样的ORA-600 2662错误是比较常见的问题,直接通过Patch_SCN工具修改scn即可正常打开库

Mon Aug 18 03:30:41 2025
ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 12 slaves
Mon Aug 18 03:30:41 2025
Recovery of Online Redo Log: Thread 1 Group 2 Seq 40787 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Completed: ALTER DATABASE RECOVER  database  
alter database open 
Beginning crash recovery of 1 threads
 parallel recovery started with 11 processes
Started redo scan
Completed redo scan
 read 1 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 40787, block 2, scn 18954891612040
Recovery of Online Redo Log: Thread 1 Group 2 Seq 40787 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 40787, block 4, scn 18954891632047
 0 data blocks read, 0 data blocks written, 1 redo k-bytes read
Mon Aug 18 03:30:47 2025
Thread 1 advanced to log sequence 40788 (thread open)
Thread 1 opened at log sequence 40788
  Current log# 3 seq# 40788 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Aug 18 03:30:47 2025
SMON: enabling cache recovery
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Completed: alter database open

后续增加tempfile,导出数据完成本次恢复任务

joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin)

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin)

expdp导出数据报ORA-39127 ORA-29548错误

ORA-39127: 调用 BEGIN :1 := SYS.DBMS_JAVA.START_EXPORT(:2, :3, :4, :5, :6, :7, :8, :9 ,:10); END; 时发生意外错误
ORA-29548: Java 系统类报告: release of Java system classes in the database (19.0.0.0.0 1.8) does not match that of the o
racle executable (19.3.0.0.0 1.8)
ORA-06512: 在 "SYS.DBMS_JAVA", line 520
ORA-06512: 在 line 1
ORA-06512: 在 "SYS.DBMS_JAVA", line 520
ORA-06512: 在 line 1
ORA-06512: 在 "SYS.DBMS_METADATA", line 11827
ORA-06512: 在 "SYS.DBMS_SYS_ERROR", line 95

对应的alert日志报错

2025-08-14T11:37:10.886745+08:00
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0 
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0 
2025-08-14T11:37:11.379211+08:00
Resize operation completed for file# 68, old size 83776K, new size 84800K
Resize operation completed for file# 68, old size 84800K, new size 85824K
2025-08-14T11:37:12.333138+08:00
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0 
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0 
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0 
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0

数据库版本（19.3版本,没有打任何其他patch）

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on 星期四 8月 14 11:17:08 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


连接到:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

出现这个错误一般是由于Oracle JVM组件和Oracle JVM库不同步导致,对于这个初始版本（没有打任何patch）,理论上不应该发生该问题,既然已经发生了,那就解决问题

SQL> startup upgrade;
ORACLE 例程已经启动。

Total System Global Area 1.5435E+10 bytes
Fixed Size                 16124312 bytes
Variable Size            4563402752 bytes
Database Buffers         1.0838E+10 bytes
Redo Buffers               17428480 bytes
数据库装载完毕。
数据库已经打开。
SQL> alter system set java_jit_enabled = FALSE;

系统已更改。

SQL> alter system set "_system_trig_enabled"=FALSE;

系统已更改。

SQL> alter system set JOB_QUEUE_PROCESSES=0;

系统已更改。

SQL> create or replace java system
  2          /

Java 已创建。

SQL> alter system set java_jit_enabled = true;

系统已更改。

SQL> alter system set "_system_trig_enabled"=TRUE;

系统已更改。

SQL> alter system set JOB_QUEUE_PROCESSES=100;

系统已更改。

SQL>  @?/rdbms/admin/utlrp.sql

会话已更改。

TIMESTAMP
--------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_BGN              2025-08-14 12:42:59

……………………

PL/SQL 过程已成功完成。

SQL> SELECT dbms_java.longname('TEST') long_name FROM dual;

LONG_NAME
--------------------------------------------------------------------------------
TEST

SQL> shutdown immediate;
数据库已经关闭。
已经卸载数据库。
ORACLE 例程已经关闭。
SQL> startup
ORACLE 例程已经启动。

Total System Global Area 1.5435E+10 bytes
Fixed Size                 16124312 bytes
Variable Size            4563402752 bytes
Database Buffers         1.0838E+10 bytes
Redo Buffers               17428480 bytes
数据库装载完毕。
数据库已经打开。

如果有其他情况,具体参考:ORA-29548 Error While Running Java Code or Datapatch In Database (Doc ID 2934546.1)文档

win环境断电强制拉库报ORA-600 kcbzib_kcrsds_1故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：win环境断电强制拉库报ORA-600 kcbzib_kcrsds_1故障处理

客户环境异常断电,导致数据库无法正常启动,执行recover提示缺少归档（ORA-00279 ORA-00289 ORA-00280错误)

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on 星期四 8月 14 11:17:08 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


连接到:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0


SQL> select open_mode from v$database;

OPEN_MODE
----------------------------------------
MOUNTED

SQL> select group#，sequence#,status from v$log;

    GROUP#  SEQUENCE# STATUS
---------- ---------- --------------------------------
         1      15082 ACTIVE
         3      15084 CURRENT
         2      15083 ACTIVE

SQL> recover database;
ORA-00279: 更改 885427209 (在 08/11/2025 22:14:56 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000015080_1051367880.0001
ORA-00280: 更改 885427209 (用于线程 1) 在序列 #15080 中


指定日志: {<RET>=suggested | filename | AUTO | CANCEL}

ORA-00308: 无法打开归档日志
'D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000015080_1051367880.0001'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件。

z这个库需要15080归档,但是由于非归档环境,redo最小为15082,所以这种情况,只能先恢复可以正常recover的数据文件，然后强制拉库

SQL> recover datafile 1;
完成介质恢复。
SQL> recover datafile 3,4,7;
完成介质恢复。
SQL> recover datafile 62;
完成介质恢复。

SQL> recover database until cancel;
ORA-00279: 更改 885427209 (在 08/11/2025 22:14:56 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000015080_1051367880.0001
ORA-00280: 更改 885427209 (用于线程 1) 在序列 #15080 中


指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: 警告: RECOVER 成功但 OPEN RESETLOGS 将出现如下错误
ORA-01152: 文件 1 没有从过旧的备份中还原
ORA-01110: 数据文件 1: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSTEM01.DBF'


ORA-01112: 未启动介质恢复


SQL> alter database open resetlogs ;
alter database open resetlogs
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [],
[], [], [], [], [], [], []
进程 ID: 6272
会话 ID: 3873 序列号: 46847

alert日志报错

2025-08-14T11:24:08.246798+08:00
alter database open resetlogs
2025-08-14T11:24:08.425971+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 28: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\PRESCRIPTION_P.DBF'
2025-08-14T11:24:08.643178+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 29: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\PRESCRIPTION_DETAIL_P.DBF'
2025-08-14T11:24:08.958486+08:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 885427209 time 
Resetting resetlogs activation ID 3572089731 (0xd4e9c383)
2025-08-14T11:24:08.988512+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 30: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\ORDER_LIST_P.DBF'
2025-08-14T11:24:12.029432+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 31: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\CHARGE_STATIS_P.DBF'
2025-08-14T11:24:13.687023+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 32: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\ECASETAB01.DBF'
2025-08-14T11:24:13.725060+08:00
Setting recovery target incarnation to 2
Checker run found 31 new persistent data failures
2025-08-14T11:24:14.701994+08:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
Endian type of dictionary set to little
2025-08-14T11:24:14.793084+08:00
Assigning activation ID 3729861866 (0xde512cea)
2025-08-14T11:24:14.848137+08:00
TT00 (PID:8068): Gap Manager starting
2025-08-14T11:24:14.870158+08:00
Redo log for group 1, sequence 1 is not located on DAX storage
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG
Successful open of redo thread 1
2025-08-14T11:24:14.912199+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2025-08-14T11:24:15.849098+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc(incident=717057):
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Please look for redo dump in pinned buffers history in incident trace file, 
if not dumped for what so ever reason, use the following command to dump it at the earliest.
ALTER SYSTEM DUMP REDO DBA MIN 4 128 DBA MAX 4 128 SCN MIN 1;
2025-08-14T11:24:17.402582+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Undo initialization recovery: err:600 start: 2392187 end: 2394046 diff: 1859 ms (1.9 seconds)
2025-08-14T11:24:17.528703+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2025-08-14T11:24:17.529703+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc(incident=717058):
ORA-00603: ORACLE 服务器会话因致命错误而终止
ORA-01092: ORACLE 实例终止。强制断开连接
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2025-08-14T11:24:18.995110+08:00
opiodr aborting process unknown ospid (6272) as a result of ORA-603
2025-08-14T11:24:18.997112+08:00
ORA-603 : opitsk aborting process
License high water mark = 42
USER (ospid: (prelim)): terminating the instance due to ORA error 
2025-08-14T11:24:23.749679+08:00
Instance terminated by USER(prelim), pid = 6272

遭遇到比较明显的ORA-600 kcbzib_kcrsds_1错误,通过自研的Patch_SCN小工具,以前有过类似恢复案例：
Patch SCN工具一键恢复ORA-600 kcbzib_kcrsds_1

然后直接open数据库

SQL> recover database;
完成介质恢复。
SQL> oradebug setmypid
已处理的语句
SQL> oradebug DUMPvar SGA kcsgscn_
kscn8 kcsgscn_ [7FF6623FF310, 7FF6623FF318) = 40B272E2 00000000
SQL>
SQL> alter database open;

数据库已更改。

然后安排导出数据,完成本次恢复任务.

log_archive_dest_1配置不当,当时standby redo无法归档引起adg不能实时同步

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：log_archive_dest_1配置不当,当时standby redo无法归档引起adg不能实时同步

oracle database dataguard无法利用standby redo实时同步

2025-08-12T18:41:57.474031+08:00
PR00 (PID:245113): Media Recovery Log /u01/app/oracle/fast_recovery_area/……/2025_08_12/o1_mf_1_34206_n9p6mmkb_.arc
PR00 (PID:245113): Media Recovery Waiting for T-1.S-34207 (in transit)

clear standby redo之后，实时同步一会儿，又无法实时同步

2025-08-12T18:43:28.183908+08:00
PR00 (PID:245113): MRP0: Background Media Recovery cancelled with status 16037
2025-08-12T18:43:28.184131+08:00
Errors in file /u01/app/oracle/diag/rdbms/dclhrdg2/DCLHR/trace/DCLHR_pr00_245113.trc:
ORA-16037: user requested cancel of managed recovery operation
PR00 (PID:245113): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Stopping change tracking
2025-08-12T18:43:28.356862+08:00
Errors in file /u01/app/oracle/diag/rdbms/dclhrdg2/DCLHR/trace/DCLHR_pr00_245113.trc:
ORA-16037: user requested cancel of managed recovery operation
2025-08-12T18:43:28.482234+08:00
Background Media Recovery process shutdown (DCLHR)
2025-08-12T18:43:29.178708+08:00
Managed Standby Recovery Canceled (DCLHR)
2025-08-12T18:43:44.724806+08:00
Clearing online log 11 of thread 1 sequence number 34111
2025-08-12T18:44:00.545234+08:00
Clearing online log 12 of thread 1 sequence number 34112
2025-08-12T18:44:10.770626+08:00
Clearing online log 13 of thread 1 sequence number 34113
2025-08-12T18:44:31.911565+08:00
Clearing online log 14 of thread 1 sequence number 34114
2025-08-12T18:44:42.111444+08:00
Clearing online log 15 of thread 1 sequence number 34108
2025-08-12T18:45:06.031673+08:00
Warning: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE has been deprecated.
Warning: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE has been deprecated.
2025-08-12T18:45:06.047722+08:00
Attempt to start background Managed Standby Recovery process (DCLHR)
Starting background process MRP0
2025-08-12T18:45:06.062512+08:00
MRP0 started with pid=219, OS id=245503
2025-08-12T18:45:06.064300+08:00
Background Managed Standby Recovery process started (DCLHR)
2025-08-12T18:45:11.086271+08:00
 Started logmerger process
2025-08-12T18:45:11.100434+08:00

IM on ADG: Start of Empty Journal

IM on ADG: End of Empty Journal
PR00 (PID:245509): Managed Standby Recovery starting Real Time Apply
max_pdb is 3
2025-08-12T18:45:13.275444+08:00
Parallel Media Recovery started with 80 slaves
2025-08-12T18:45:13.392342+08:00
Stopping change tracking
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34207 (in transit)
2025-08-12T18:46:22.717320+08:00
 rfs (PID:245728): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is Foreground (PID:15566)
2025-08-12T18:46:31.950359+08:00
 rfs (PID:244324): Archived Log entry 99 added for B-1149239150.T-1.S-34207 ID 0x57544f44 LAD:32
2025-08-12T18:46:32.185391+08:00
 rfs (PID:245735): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is ASYNC (PID:3334853)
 rfs (PID:245735): Primary database is in MAXIMUM PERFORMANCE mode
2025-08-12T18:46:32.331928+08:00
 rfs (PID:245735): Selected LNO:11 for T-1.S-34208 dbid 1299764411 branch 1149239150
2025-08-12T18:46:32.885400+08:00
PR00 (PID:245509): Media Recovery Log /u01/app/oracle/fast_recovery_area/……/2025_08_12/o1_mf_1_34207_n9p6mo8f_.arc
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34208 (in transit)
2025-08-12T18:46:33.669769+08:00
Recovery of Online Redo Log: Thread 1 Group 11 Seq 34208 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/DCLHR/std_redo11.log
2025-08-12T18:50:03.011923+08:00
Expanded controlfile section 28 from 141 to 282 records
Requested to grow by 141 records; added 2 blocks of records
2025-08-12T19:16:29.515866+08:00
 rfs (PID:245735): Selected LNO:12 for T-1.S-34209 dbid 1299764411 branch 1149239150
2025-08-12T19:16:29.515943+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34209 (in transit)
2025-08-12T19:16:30.522048+08:00
Recovery of Online Redo Log: Thread 1 Group 12 Seq 34209 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/DCLHR/std_redo12.log
2025-08-12T19:46:29.852369+08:00
 rfs (PID:245735): Selected LNO:13 for T-1.S-34210 dbid 1299764411 branch 1149239150
2025-08-12T19:46:29.852443+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34210 (in transit)
2025-08-12T19:46:30.858423+08:00
Recovery of Online Redo Log: Thread 1 Group 13 Seq 34210 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/DCLHR/std_redo13.log
2025-08-12T20:16:29.978563+08:00
 rfs (PID:245735): Selected LNO:14 for T-1.S-34211 dbid 1299764411 branch 1149239150
2025-08-12T20:16:30.012723+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34211 (in transit)
2025-08-12T20:16:30.013045+08:00
Recovery of Online Redo Log: Thread 1 Group 14 Seq 34211 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/DCLHR/std_redo14.log
2025-08-12T20:46:30.307116+08:00
 rfs (PID:245735): Selected LNO:15 for T-1.S-34212 dbid 1299764411 branch 1149239150
2025-08-12T20:46:30.339601+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34212 (in transit)
2025-08-12T20:46:30.339893+08:00
Recovery of Online Redo Log: Thread 1 Group 15 Seq 34212 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/DCLHR/std_redo15.log
2025-08-12T21:16:27.517565+08:00
 rfs (PID:245735): No SRLs available for T-1
2025-08-12T21:16:27.560282+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34213 (in transit)
2025-08-12T21:16:27.615513+08:00
 rfs (PID:245735): Opened log for T-1.S-34213 dbid 1299764411 branch 1149239150
2025-08-12T21:46:27.633777+08:00
 rfs (PID:245735): Archived Log entry 100 added for B-1149239150.T-1.S-34213 ID 0x57544f44 LAD:32
 rfs (PID:245735): No SRLs available for T-1
2025-08-12T21:46:27.787746+08:00
 rfs (PID:245735): Opened log for T-1.S-34214 dbid 1299764411 branch 1149239150
2025-08-12T21:46:28.439910+08:00
PR00 (PID:245509): Media Recovery Log /u01/app/oracle/fast_recovery_area/……/2025_08_12/o1_mf_1_34213_n9phockd_.arc
2025-08-12T21:46:32.804540+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34214 (in transit)
2025-08-12T22:16:27.968668+08:00
 rfs (PID:245735): Archived Log entry 101 added for B-1149239150.T-1.S-34214 ID 0x57544f44 LAD:32
 rfs (PID:245735): No SRLs available for T-1

从报错信息看是由于无可用的standby redo导致（No SRLs available for T-1）,查询standby redo状态

[oracle@db03 trace]$ ss

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Aug 13 15:58:46 2025
Version 19.20.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

s
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.20.0.0.0

SQL> elect group#,status from v$standby_log;

    GROUP# STATUS
---------- ----------
        11 ACTIVE
        12 ACTIVE
        13 ACTIVE
        14 ACTIVE
        15 ACTIVE

所有的standby redo都是active状态,导致无可用standby redo,出现该问题的原因,初步怀疑是standby redo无法归档导致,引起该问题的原因怀疑是由于该备库的log_archive_dest_N参数设置不当导致,检查相关参数

*.db_unique_name='DCLHRDG2'
*.fal_client='DCLHRDG2'
*.fal_server='DCLHRDG','DCLHR'
*.log_archive_config='dg_config=(DCLHR,DCLHRDG,DCLHRDG2)'
*.log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST valid_for=(all_logfiles,all_roles) db_unique_name=DCLHRDG'
*.log_archive_dest_2='service=DCLHR lgwr async valid_for=(online_logfiles,primary_role) db_unique_name=DCLHR'
*.log_archive_dest_3='service=DCLHRDG2 lgwr async valid_for=(online_logfiles,primary_role) db_unique_name=DCLHRDG2'

比较明显由于log_archive_dest_1和log_archive_dest_3的相关参数配置不对导致standby redo无法通过log_archive_dest_1进行归档导致,正确的配置方式为

*.log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST valid_for=(all_logfiles,all_roles) db_unique_name=DCLHRDG'
*.log_archive_dest_3='service=DCLHRDG lgwr async valid_for=(online_logfiles,primary_role) db_unique_name=DCLHRDG'

这样配置之后,该dg库同步日志正常.

RAC环境redo在各节点本地导致数据库故障恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：RAC环境redo在各节点本地导致数据库故障恢复

最近一个运行在win平台的rac,由于断电之后,集群两个节点均无法正常启动,客户进行了一系列尝试,结果到了ora-600 kclchkblk_4错误无法继续.
通过对数据库日志分析,回溯了故障大概的原因,启动的时候报错为:
节点1启动报错

Sun Aug 03 15:21:22 2025
alter database open
This instance was first to open
Beginning crash recovery of 2 threads
 parallel recovery started with 32 processes
Started redo scan
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl1\trace\orcl1_ora_7108.trc:
ORA-00314: 日志 11 (用于线程 2) 要求的 sequence# 147717 与 147541 不匹配
ORA-00312: 联机日志 11 线程 2: 'D:\REDOLOG\REDO011.LOG'
Abort recovery for domain 0
Aborting crash recovery due to error 314
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl1\trace\orcl1_ora_7108.trc:
ORA-00314: 日志 11 (用于线程 2) 要求的 sequence# 147717 与 147541 不匹配
ORA-00312: 联机日志 11 线程 2: 'D:\REDOLOG\REDO011.LOG'
Abort recovery for domain 0
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl1\trace\orcl1_ora_7108.trc:
ORA-00314: 日志 11 (用于线程 2) 要求的 sequence# 147717 与 147541 不匹配
ORA-00312: 联机日志 11 线程 2: 'D:\REDOLOG\REDO011.LOG'
ORA-314 signalled during: alter database open...

节点2启动报错

Sat Aug 02 15:45:43 2025
Successful mount of redo thread 2, with mount id 1735887907
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Lost write protection disabled
Completed: ALTER DATABASE MOUNT /* db agent *//* {1:47460:124} */
ALTER DATABASE OPEN /* db agent *//* {1:47460:124} */
This instance was first to open
Beginning crash recovery of 2 threads
Sat Aug 02 15:45:49 2025
 parallel recovery started with 32 processes
Started redo scan
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl2\trace\orcl2_ora_3444.trc:
ORA-00314: ?? 1 (???? 1) ??? sequence# 67782 ? 60818 ???
ORA-00312: ???? 1 ?? 1: 'D:\REDOLOG\REDO01.LOG'
Abort recovery for domain 0
Aborting crash recovery due to error 314
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl2\trace\orcl2_ora_3444.trc:
ORA-00314: ?? 1 (???? 1) ??? sequence# 67782 ? 60818 ???
ORA-00312: ???? 1 ?? 1: 'D:\REDOLOG\REDO01.LOG'
Abort recovery for domain 0
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl2\trace\orcl2_ora_3444.trc:
ORA-00314: ?? 1 (???? 1) ??? sequence# 67782 ? 60818 ???
ORA-00312: ???? 1 ?? 1: 'D:\REDOLOG\REDO01.LOG'
ORA-314 signalled during: ALTER DATABASE OPEN /* db agent *//* {1:47460:124} */...

看到这两个报错信息得出两个结论：
1）比较明显节点1需要thead 2的 group 11 sequence为147717,但是实际group 11文件的sequence为147541；而节点2启动需要thread 1的group 1 sequence为67782，但是实际中group 1文件的sequnece为60818,这两个都相差比较多,属于非正常的情况,很可能是文件本身有问题
2）这是一套win的rac架构,理论上redo应该在共享文件系统（一般是asm中)，而这个第一感觉很可能是本地文件系统中
客户当时恢复之时查询信息截图

查看了两个节点的最后redo切换信息

--节点1
Sat Aug 02 10:49:31 2025
Thread 1 advanced to log sequence 67782 (LGWR switch)
  Current log# 1 seq# 67782 mem# 0: D:\REDOLOG\REDO01.LOG

--节点2（redo每组2G,节点2长时间没跑业务,之时做数据库导出操作,所以切换时间比较久远）
Sat Jul 26 16:56:42 2025
Thread 2 advanced to log sequence 147717 (LGWR switch)
  Current log# 11 seq# 147717 mem# 0: D:\REDOLOG\REDO011.LOG

并查看两个机器d:/redolog信息(客户自行resetlogs之后的,非第一现场,但是可以确认两个节点各自有一份redo文件

本来这个是一个比较小的故障,只要把节点2的thread 1的redo拷贝到到节点1或者节点1的thread 2的redo拷贝到节点2,然后正常open库即可,现场恢复对rac不太熟悉,直接按照互联网上检索的处理方法,加上_allow_resetlogs_corruption然后强制拉库,结果不太幸运,拉库失败报ORA-600 kclchkblk_4错误

Sun Aug 03 18:59:24 2025
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 21497084214
Resetting resetlogs activation ID 1543012633 (0x5bf88119)
Sun Aug 03 18:59:46 2025
Setting recovery target incarnation to 3
Sun Aug 03 18:59:46 2025
This instance was first to open
Picked broadcast on commit scheme to generate SCNs
Sun Aug 03 18:59:48 2025
Assigning activation ID 1735960667 (0x6778a85b)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\REDOLOG\REDO01.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sun Aug 03 18:59:49 2025
SMON: enabling cache recovery
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl1\trace\orcl1_ora_8508.trc  (incident=728324):
ORA-00600: 内部错误代码, 参数: [kclchkblk_4], [5], [200595988], [5], [22247740], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl1\incident\incdir_728324\orcl1_ora_8508_i728324.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Sun Aug 03 18:59:53 2025
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl1\trace\orcl1_ora_8508.trc:
ORA-00704: 引导程序进程失败
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [kclchkblk_4], [5], [200595988], [5], [22247740], [], [], [], [], [], [], []
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl1\trace\orcl1_ora_8508.trc:
ORA-00704: 引导程序进程失败
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [kclchkblk_4], [5], [200595988], [5], [22247740], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 8508): terminating the instance due to error 704
Sun Aug 03 18:59:54 2025
opiodr aborting process unknown ospid (9480) as a result of ORA-1092
Sun Aug 03 19:00:09 2025
Instance terminated by USER, pid = 8508
ORA-1092 signalled during: alter database open resetlogs...
opiodr aborting process unknown ospid (8508) as a result of ORA-1092

这个故障之后,客户那边无法自行恢复,让我这边介入处理,对于这个错误以前处理比较多,一般就是scn问题,通过Patch SCN小工具快速解决

数据库open成功之后主要报一些ORA-600 4137,ORA-600 6006等错误

Database Characterset is ZHS16GBK
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl1\trace\orcl1_smon_8300.trc  (incident=1176205):
ORA-00600: 内部错误代码, 参数: [4137], [1.14.2713957], [0], [0], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl1\incident\incdir_1176205\orcl1_smon_8300_i1176205.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Fri Aug 08 19:03:14 2025
ORACLE Instance orcl1 (pid = 25) - Error 600 encountered while recovering transaction (1, 14).
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl1\trace\orcl1_smon_8300.trc:
ORA-00600: 内部错误代码, 参数: [4137], [1.14.2713957], [0], [0], [], [], [], [], [], [], [], []
Fri Aug 08 19:03:15 2025
ORACLE Instance orcl1 (pid = 25) - Error 600 encountered while recovering transaction (5, 19).
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl1\trace\orcl1_smon_8300.trc:
ORA-00600: 内部错误代码, 参数: [4137], [5.19.2318502], [0], [0], [], [], [], [], [], [], [], []
Starting background process MMON
Fri Aug 08 19:03:18 2025
MMON started with pid=29, OS id=4624
Fri Aug 08 19:03:19 2025
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl1\trace\orcl1_smon_8300.trc  (incident=1176207):
ORA-00600: 内部错误代码, 参数: [6006], [1], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl1\incident\incdir_1176207\orcl1_smon_8300_i1176207.trc
Starting background process MMNL
Fri Aug 08 19:03:19 2025
MMNL started with pid=30, OS id=8344
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ORACLE Instance orcl1 (pid = 25) - Error 600 encountered while recovering transaction (46, 28) on object 197344.
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl1\trace\orcl1_smon_8300.trc:
ORA-00600: 内部错误代码, 参数: [6006], [1], [], [], [], [], [], [], [], [], [], []

通过重建undo解决该错误,数据库稳定运行,没有再crash和报明显错误,导出核心数据,完成本次恢复任务.

Oracle 21c 202507补丁-21.19

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle 21c 202507补丁-21.19

Description	Database Update	GI Update	Windows Bundle Patch
JUL2025 (21.19.0.0.0)	38068980	38068887	37962964
APR2025 (21.18.0.0.0)	37655430	37642955	37532378
JAN2025 (21.17.0.0.0)	37350281	37349593	37500148
OCT2024 (21.16.0.0.0)	36991631	36990664	36878842
JUL2024 (21.15.0.0.0)	36696242	36696109	36521898
APR2024 (21.14.0.0.0)	36352352	36352207	36219877
JAN2024 (21.13.0.0.0)	36041222	36031790	35962857
OCT2023 (21.12.0.0.0)	35740258	35738010	35681617
JUL2023 (21.11.0.0.0)	35428978	35427907	35347974
APR2023 (21.10.0.0.0)	35134934	35132566	35046488
JAN2023 (21.9.0.0.0)	34839741	34838415	34750812
Oct2022 (21.8.0.0.0)	34527084	34526142	34468137
JUL2022 (21.7.0.0.0)	34160444	34155589	34110698
APR2022 (21.6.0.0.0)	33843745	33859395	33829143
JAN2022 (21.5.0.0.0)	33516412	33531909	33589769
OCT2021 (21.4.0.0.0)	33239276	33250101	NA

参考:Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), Bundle Patches, Patchsets and Base Releases (Doc ID 2118136.2)

ORA-600 kcratr_nab_less_than_odr和ORA-600 4194故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 kcratr_nab_less_than_odr和ORA-600 4194故障处理

客户一个11.2.0.1的库由于断电导致启动的时候报ORA-600 kcratr_nab_less_than_odr错误

Wed Aug 06 15:55:07 2025
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 27 processes
Started redo scan
Completed redo scan
 read 11787 KB redo, 4129 data blocks need recovery
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_3520.trc  (incident=2624321):
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [111988], [42537], [43285], [], [], [], [], [], [], []
Incident details in: d:\app\xff\diag\rdbms\primary\orcl\incident\incdir_2624321\orcl_ora_3520_i2624321.trc
Aborting crash recovery due to error 600
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_3520.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [111988], [42537], [43285], [], [], [], [], [], [], []
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_3520.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [111988], [42537], [43285], [], [], [], [], [], [], []
ORA-600 signalled during: ALTER DATABASE OPEN...

类似的ORA-600 kcratr_nab_less_than_odr故障处理不少,但是应用redo完成之后,再open报错的不多:
12c启动报kcratr_nab_less_than_odr
又一例ORA-600 kcratr_nab_less_than_odr
ORA-600 kcratr_nab_less_than_odr故障解决
 差点被误操作的ORA-600 kcratr_nab_less_than_odr故障
 11.2.0.4库中遇到ORA-600 kcratr_nab_less_than_odr报错
但是这个客户比较不幸,经过一系列尝试打开数据库,但是报ORA-600 4194错误

Wed Aug 06 17:09:33 2025
ALTER DATABASE RECOVER  database until cancel using backup controlfile  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 28 slaves
ORA-279 signalled during: ALTER DATABASE RECOVER  database until cancel using backup controlfile  ...
Wed Aug 06 17:10:17 2025
ALTER DATABASE RECOVER    LOGFILE 'D:\app\xff\oradata\orcl\REDO05.LOG'  
Media Recovery Log D:\app\xff\oradata\orcl\REDO05.LOG
Wed Aug 06 17:10:17 2025
Incomplete recovery applied all redo ever generated.
Recovery completed through change 2118813100 time 08/06/2025 15:02:15
Completed: ALTER DATABASE RECOVER    LOGFILE 'D:\app\xff\oradata\orcl\REDO05.LOG'  
Wed Aug 06 17:14:05 2025
alter database open resetlogs
Archived Log entry 188143 added for thread 1 sequence 111985 ID 0x634438e6 dest 1:
Archived Log entry 188144 added for thread 1 sequence 111986 ID 0x634438e6 dest 1:
Archived Log entry 188145 added for thread 1 sequence 111987 ID 0x634438e6 dest 1:
Archived Log entry 188146 added for thread 1 sequence 111984 ID 0x634438e6 dest 1:
Archived Log entry 188147 added for thread 1 sequence 111988 ID 0x634438e6 dest 1:
RESETLOGS after complete recovery through change 2118813100
Resetting resetlogs activation ID 1665415398 (0x634438e6)
Wed Aug 06 17:15:03 2025
Archived Log entry 188148 added for thread 1 sequence 1 ID 0x677cfac1 dest 1:
Successfully onlined Undo Tablespace 2.
Dictionary check beginning
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_smon_2428.trc  (incident=3264274):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\app\xff\diag\rdbms\primary\orcl\incident\incdir_3264274\orcl_smon_2428_i3264274.trc
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Wed Aug 06 17:15:06 2025
QMNC started with pid=34, OS id=3052 
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Completed: alter database open resetlogs

后续由于该undo异常报ORA-01595和ORA-600 4194等错误导致数据库crash

Wed Aug 06 17:15:49 2025
Doing block recovery for file 3 block 888
Resuming block recovery (PMON) for file 3 block 888
Block recovery from logseq 2, block 316 to scn 2118813732
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: D:\APP\xff\ORADATA\ORCL\REDO02.LOG
Block recovery stopped at EOT rba 2.447.16
Block recovery completed at rba 2.447.16, scn 0.2118813732
Doing block recovery for file 3 block 1960
Resuming block recovery (PMON) for file 3 block 1960
Block recovery from logseq 2, block 316 to scn 2118813622
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: D:\APP\xff\ORADATA\ORCL\REDO02.LOG
Block recovery completed at rba 2.317.16, scn 0.2118813626
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_smon_2428.trc:
ORA-01595: error freeing extent (2) of rollback segment (27))
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []

接手这个库之后,处理工作相对比较简单,直接对异常undo进行重建即可

Thu Aug 07 09:35:08 2025
create undo tablespace undotbs2 datafile 'D:\APP\xff\ORADATA\ORCL\undotbs02.dbf' size 128M autoextend on
Completed: create undo tablespace undotbs2 datafile 'D:\APP\xff\ORADATA\ORCL\undotbs02.dbf' size 128M autoextend on
Thu Aug 07 09:37:53 2025
drop tablespace UNDOTBS1 including contents and datafiles
Deleted file D:\APP\xff\ORADATA\ORCL\UNDOTBS01.DBF
Completed: drop tablespace UNDOTBS1 including contents and datafiles

然后数据库报大量的ORA-600 kdsgrp1错误

Thu Aug 07 10:03:53 2025
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_6156.trc  (incident=5121829):
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Thu Aug 07 10:04:00 2025
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_4564.trc  (incident=5121501):
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Thu Aug 07 10:04:06 2025
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_9160.trc  (incident=5121837):
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Thu Aug 07 10:04:16 2025
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_5868.trc  (incident=5121821):
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Thu Aug 07 10:04:28 2025
Errors in file d:\app\xff\diag\rdbms\primary\orcl\trace\orcl_ora_4448.trc  (incident=5122077):
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []

* kdsgrp1-1: *************************************************
            row 0x013a9ae9.0 continuation at
            0x013a9ae9.0 file# 4 block# 3840745 slot 0 not found
KDSTABN_GET: 0 ..... ntab: 1
curSlot: 0 ..... nrows: 2
kdsgrp - dump CR block dba=0x013a9ae9
Block header dump:  0x013a9ae9
 Object id on Block? Y
 seg/obj: 0x31dd9  csc: 0x00.7dc0f246  itc: 2  flg: E  typ: 1 - DATA
     brn: 0  bdba: 0x13a9082 ver: 0x01 opc: 0
     inc: 2  exflg: 0
 
 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x0000.000.00000000  0x00000000.0000.00  ----    0  fsc 0x0000.00000000
0x02   0x0000.000.00000000  0x00000000.0000.00  ----    0  fsc 0x0000.00000000

通过分析是由于index和表记录不匹配导致,对index进行重新rebuild完成本次修复任务.

pg_wal中文件的名称中的logseq和实际文件中的logseq不匹配

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：pg_wal中文件的名称中的logseq和实际文件中的logseq不匹配

一个朋友由于某种原因给我发了一个pg_wal里面的wal文件,让我帮忙分析下故障原因,我打开文件之后发现文件编号小的修改时间比编号大的文件修改时间要新

wal日志文件命名规则：
我们看到的wal日志是这样的：0000000100004D6E000000CF
其中前8位:00000001表示timeline
中间8位：00004D6E表示logid
最后8位：000000CF表示logseg
在上述截图中,可以看到logseq为CF的文件比D0的要新很多,这个初步给人感觉不太正常.使用pg_waldump分别对其进行dump操作

[postgres@xifenfei bin]$ pg_waldump /data/wal/0000000100004D6E000000D0|head -10
pg_waldump: fatal: could not find a valid record after 4D6E/D0000000

这样可以看出来CF的wal文件可以正常dump出来,但是D0的文件dump报pg_waldump: fatal: could not find a valid record after类似异常.通过od命令分别对两个文件进行分析

[postgres@xifenfei bin]$ od -x /data/wal/0000000100004D6E000000CF|head -1
0000000 d101 0006 0001 0000 0000 cf00 4d6e 0000
[postgres@xifenfei bin]$ od -x /data/wal/0000000100004D6E000000D0|head -1
0000000 d101 0007 0001 0000 0000 9400 4d6e 0000

从第8个字节到第12个字节结束为logseq的值,这里明显可以看出来D0文件的logseq值和实际文件中的不一致.尝试把D0修改为94之后即可正常的pg_waldump进行分析

通过这里可以的出来一个结论,pg_wal中文件的名称中的logseq和实际文件中的logseq不匹配.出现这种问题的本质是由于pg_wal中的wal日志是相当oracle的redo,是通过类似重命名机制(看到有文档介绍是说硬链接指向旧文件然后删除旧文件)引起的问题.

由于空间满导致PostgreSQL数据库异常处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：由于空间满导致PostgreSQL数据库异常处理

朋友和我反馈pg库异常,通过查看日志确认最初故障是由于磁盘空间满，导出出现类似:无法扩展文件 “pg_tblspc/16394/PG_13_202007201/16395/5055912.143″: No space left on device错误

2025-07-28 09:06:02.703 HKT [15352] 错误:  无法扩展文件 "pg_tblspc/16394/PG_13_202007201/16395/5055912.143": No space left on device
2025-07-28 09:06:02.703 HKT [15352] 提示:  检查空闲磁盘控件.
2025-07-28 09:06:02.703 HKT [15352] 语句:  insert into 语句
2025-07-28 09:06:02.703 HKT [576] 错误:  无法扩展文件 "pg_tblspc/16394/PG_13_202007201/16395/4477723.73": No space left on device
2025-07-28 09:06:02.703 HKT [576] 提示:  检查空闲磁盘控件.
2025-07-28 09:06:02.703 HKT [576] 语句:  update 语句
2025-07-28 09:06:02.706 HKT [11940] 错误:  无法扩展文件 "pg_tblspc/16394/PG_13_202007201/16395/5055912.143": No space left on device
2025-07-28 09:06:02.706 HKT [11940] 提示:  检查空闲磁盘控件.
2025-07-28 09:06:02.706 HKT [11940] 语句:  insert into 语句

后续对d盘空间进行了清理,pg报无法打开文件”pg_tblspc/16394/PG_13_202007201/16395/4477706.16″(目标数据块5544906): Permission denied

2025-07-28 10:53:10.435 HKT [11920] 语句:  insert into语句
2025-07-28 10:53:10.435 HKT [18300] 错误:  无法打开文件"pg_tblspc/16394/PG_13_202007201/16395/4477706.16"(目标数据块5544906): Permission denied

和报:警告: 无法写入pg_tblspc/16394/PG_13_202007201/16395/4477706的块5545387错误

2025-07-28 11:05:11.008 HKT [5380] 警告:  无法写入pg_tblspc/16394/PG_13_202007201/16395/4477706的块5545387
2025-07-28 11:05:11.008 HKT [5380] 详细信息:  多次失败 --- 写错误可能是永久性的
2025-07-28 11:05:11.015 HKT [12072] 错误:  无法打开文件"pg_tblspc/16394/PG_13_202007201/16395/4477706.17"（目标块5545398）：上一段只有952个块
2025-07-28 11:05:11.015 HKT [12072] 上下文:  写入关系pg_tblspc/16394/PG_13_202007201/16395/4477706的块5545398
2025-07-28 11:05:11.015 HKT [12072] 语句:  select 语句
2025-07-28 11:05:11.016 HKT [12072] 警告:  无法写入pg_tblspc/16394/PG_13_202007201/16395/4477706的块5545398
2025-07-28 11:05:11.016 HKT [12072] 详细信息:  多次失败 --- 写错误可能是永久性的

查看4477706相关文件情况,初步看部分文件大小不对,结合上述报错,可以断定该表异常

通过查询pg字典确认该表具体名称

xifenfei=# select oid,relname,relnamespace from pg_class where  relfilenode=4477706;
  oid  |     relname      | relnamespace
-------+------------------+--------------
 16880 | xifenfei01_log   |         2200
(1 行记录)

尝试pg_dump导入数据报错：多次失败 — 写错误可能是永久性的

这次运气比较好,损坏的对象是一个日志表,可以直接清理掉,对该日志表进行清理,然后正常导出数据,如果遇到的损坏表是业务需要的表,可以使用pdu(PostgreSQL表文件损坏恢复—pdu恢复损坏的表文件)进行恢复

一次非常幸运的ORA-600 16703(tab$被清空)故障恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：一次非常幸运的ORA-600 16703(tab$被清空)故障恢复

这次的ORA-600 16703的故障比较特殊,客户的一套rac运行了5年多没有重启,这次由于异常导致其中一个节点重启,然后触发了tab$被清空,异常节点启动报ORA-600 16703错误.朋友在故障之后,第一时间没有对在运行的节点进行重启(虽然也无法对外提供业务服务),使得恢复工作相对简单一些,恢复效果也是最完美的.这个是我在对于软件安装介质注入恶意脚本,300天之后重启触发tab$被清空的相关恢复case中,最完美的一次(以前遇到过一次客户是虚拟化环境通过cdp回退然后类似方法处理ORA-600 16703直接把orachk备份表插入到tab$恢复),凸显了这位朋友在故障发生之后对于问题的准确判断和果断的应对能力.
有朋友和我反馈,他们数据库突然报大量ORA-600错误,业务无法正常操作,我分析相关日志确认:节点2重启之后节点1开始报大量ORA-600错误,但是节点一直处于open状态

Fri Jul 25 15:28:53 2025
Decreasing number of real time LMS from 3 to 0
Fri Jul 25 15:29:18 2025
Reconfiguration started (old inc 13, new inc 15)
List of instances:
 1 2 (myinst: 1) 
 Global Resource Directory frozen
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Fri Jul 25 15:29:18 2025
 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Fri Jul 25 15:29:18 2025
 LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Fri Jul 25 15:29:18 2025
 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info 
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Submitted all GCS remote-cache requests
 Fix write in gcs resources
Reconfiguration complete
Fri Jul 25 15:29:20 2025
minact-scn: Master returning as live inst:2 has inc# mismatch instinc:0 cur:15 errcnt:0
Fri Jul 25 15:30:07 2025
Errors in file /u01/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_252634.trc  (incident=77234):
ORA-00600: internal error code, arguments: [ktsircinfo_num1], [0], [0], [0], [],[],[],[],[],[],[],[]
Incident details in: /u01/oracle/diag/rdbms/orcl/orcl1/incident/incdir_77234/orcl1_ora_252634_i77234.trc
Fri Jul 25 15:30:18 2025
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Fri Jul 25 15:30:19 2025
Sweep [inc][77234]: completed
Sweep [inc2][77234]: completed
Fri Jul 25 15:30:27 2025
Errors in file /u01/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_261587.trc  (incident=76487):
ORA-00600: internal error code, arguments: [ktsircinfo_num1],[0],[0],[0], [], [], [], [], [], [], [], []

通过grep筛选报错信息

[root@iZbp11c0qyuuo1gr7j98upZ tmp]# egrep "ORA-00600|ORA-07445" alert_1.txt |sort -u
ORA-00600: internal error code, arguments: [25027], [0], [0], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kkpo_rcinfo_defstg:delseg], [28941391], [], [], [], []
ORA-00600: internal error code, arguments: [ktsircinfo_num1], [0], [0], [0], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [16659], [kqldtu], [DEL], [0], [35038924], [], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [16659], [kqldtu], [INS], [0], [277736], [], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [16659], [kqldtu], [INS], [0], [28829570], [], [], [], [], [], [], []
ORA-07445: exception encountered: core dump [qknSetParent()+9] [SIGSEGV] [ADDR:0x10354] 
   [PC:0x1A48B9B] [Address not mapped to object] []
ORA-07445: exception encountered: core dump [qksxaMoveQbAnnotations()+168] [SIGSEGV]
   [ADDR:0x20304] [PC:0x1594954] [Address not mapped to object] []
ORA-07445: 出现异常错误: 核心转储 [qknExpRegIni_int()+87] [SIGSEGV] [ADDR:0x8C] 
   [PC:0x1A4D729] [Address not mapped to object] []
ORA-07445: 出现异常错误: 核心转储 [qksxaMoveQbAnnotations()+168] [SIGSEGV] [ADDR:0x0] 
   [PC:0x1594954] [SI_KERNEL(general_protection)] []

既然是由于节点2重启导致节点1报错,那分析节点2重启相关情况,第一次重启成功之后,数据库开始报ORA-600错误

Fri Jul 25 15:29:29 2025
QMNC started with pid=46, OS id=363757 
Fri Jul 25 15:29:31 2025
minact-scn: Inst 2 is a slave inc#:15 mmon proc-id:363622 status:0x2
minact-scn status: grec-scn:0x0000.00000000 gmin-scn:0x0000.00000000 gcalc-scn:0x0000.00000000
Fri Jul 25 15:29:33 2025
Errors in file /u01/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_m003_363779.trc  (incident=248519):
ORA-00600: internal error code, arguments: [kgmfvmi#3], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle/diag/rdbms/orcl/orcl2/incident/incdir_248519/orcl2_m003_363779_i248519.trc
Starting background process SMCO
Fri Jul 25 15:29:35 2025
SMCO started with pid=57, OS id=363802 
Fri Jul 25 15:29:35 2025
Completed: ALTER DATABASE OPEN /* db agent *//* {2:23784:2} */
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_m003_363779.trc  (incident=248520):
ORA-00600: internal error code, arguments: [kgmfvmi#3], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle/diag/rdbms/orcl/orcl2/incident/incdir_248520/orcl2_m003_363779_i248520.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Exception [type: SIGSEGV, Address not mapped to object][ADDR:0x10] [PC:0x2FDA4BB,kgmdelsis()+219][flags:0x0,count:1]
Errors in file /u01/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_m003_363779.trc  (incident=248521):
ORA-07445: exception encountered: core dump [kgmdelsis()+219] [SIGSEGV] 
  [ADDR:0x10] [PC:0x2FDA4BB] [Address not mapped to object] []
ORA-00600: internal error code, arguments: [kgmfvmi#3], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle/diag/rdbms/orcl/orcl2/incident/incdir_248521/orcl2_m003_363779_i248521.trc
Use ADRCI or Support Workbench to package the incident.
Fri Jul 25 15:29:47 2025
Errors in file /u01/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_363808.trc  (incident=248559):
ORA-00600: internal error code, arguments: [kkposds2], [18446744073709551615], [18446744073709551615], 
  [18446744073709551615], [], [], [], [], [], [], [], []

然后第二次重启数据库无法open成功,而是报ORA-600 16703错误

ALTER DATABASE OPEN /* db agent *//* {2:21799:2} */
Picked broadcast on commit scheme to generate SCNs
ARCH: STARTING ARCH PROCESSES
Fri Jul 25 15:41:23 2025
ARC0 started with pid=39, OS id=369231 
ARC0: Archival started
ARCH: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Fri Jul 25 15:41:24 2025
ARC1 started with pid=40, OS id=369242 
Fri Jul 25 15:41:24 2025
ARC2 started with pid=41, OS id=369244 
Fri Jul 25 15:41:24 2025
ARC3 started with pid=42, OS id=369246 
ARC1: Archival started
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
Thread 2 opened at log sequence 33585
  Current log# 7 seq# 33585 mem# 0: +DATA/orcl/onlinelog/group_7.269.1011373611
Successful open of redo thread 2
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file /u01/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_369210.trc  (incident=260494):
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle/diag/rdbms/orcl/orcl2/incident/incdir_260494/orcl2_ora_369210_i260494.trc
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
SUCCESS: diskgroup FRA was mounted
Fri Jul 25 15:41:30 2025
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_369210.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Errors in file /u01/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_369210.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 369210): terminating the instance due to error 704
Instance terminated by USER, pid = 369210
ORA-1092 signalled during: ALTER DATABASE OPEN /* db agent *//* {2:21799:2} */...
opiodr aborting process unknown ospid (369210) as a result of ORA-1092
Fri Jul 25 15:41:31 2025
ORA-1092 : opitsk aborting process

到这一步基本上就清晰了,大概率是遭遇到以前恢复的类似case,tab$数据被清空导致,类似案例
ORA-600 16703故障解析—tab$表被清空
 警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703
通过在故障主机上找到安装介质,验证md5确认该程序是被注入恶意代码程序

这个库由于还有一个节点处于open状态,相对处理比较简单,直接把备份的表数据反向插入回去即可

SYS@orcl1> select count(1) from ORACHK3C08C86E063530510ACD937;

  COUNT(1)
----------
     20696

SYS@orcl1> insert into tab$ select * from ORACHK3C08C86E063530510ACD937;

20696 rows created.

SYS@orcl1> commit;

Commit complete.

SYS@orcl1> select object_name,to_char(CREATED,'yyyy-mm-dd hh24:mi:ss') from dba_objects 
          2 where object_name in('DBMS_SUPPORT_DBMONITOR','DBMS_SUPPORT_DBMONITORP');

OBJECT_NAME                                  TO_CHAR(CREATED,'YY
-------------------------------------------- -------------------
DBMS_SUPPORT_DBMONITORP                      2019-06-19 17:06:46
DBMS_SUPPORT_DBMONITOR                       2019-06-19 17:06:46

然后清理掉恶意脚本,分别重启两个节点,完成数据恢复任务

这次故障能够快速顺利的恢复,和客户发现故障之后保留第一现场,没有把一个open的节点也重启有很大关系,open的节点也重启了,那后续恢复工作会麻烦很多,效果可能也没有这样的完美.