Oracle备份恢复 – Database SOS

差点被误操作的ORA-600 kcratr_nab_less_than_odr故障

标题：差点被误操作的ORA-600 kcratr_nab_less_than_odr故障

晚上接到一个客户电话,数据库无法启动,咨询我的Oracle Recovery Tools工具的授权问题,我远程登录客户的环境进行查看故障,发现客户进行了一下操作导致最后open报ORA-01152错误

C:\Users\Administrator>sqlplus sys/sys as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Tue Mar 18 22:43:24 2025

Copyright (c) 1982, 2010, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> startup mount
ORACLE instance started.

Total System Global Area 1603411968 bytes
Fixed Size                  2176168 bytes
Variable Size             973081432 bytes
Database Buffers          620756992 bytes
Redo Buffers                7397376 bytes
Database mounted.
SQL> recover database;
Media recovery complete.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1],
[27890], [1589], [1649], [], [], [], [], [], [], []

SQL> recover database;
ORA-00283: 恢复会话因错误而取消
ORA-00264: 不要求恢复


SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1],
[27890], [1589], [1649], [], [], [], [], [], [], []


SQL> recover database until cancel;
ORA-10879: error signaled in parallel recovery slave
ORA-01547: 警告: RECOVER 成功但 OPEN RESETLOGS 将出现如下错误
ORA-01152: 文件 1 没有从过旧的备份中还原
ORA-01110: 数据文件 1: 'D:\APP\ADMINISTRATOR\ORADATA\orcl\SYSTEM01.DBF'

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\orcl\SYSTEM01.DBF'


SQL> RECOVER DATABASE USING BACKUP CONTROLFILE;
ORA-00279: change 17384974762395 generated at 03/18/2025 18:30:34 needed for
thread 1
ORA-00289: suggestion :
D:\APP\ADMINISTRATOR\FLASH_RECOVERY_AREA\orcl\ARCHIVELOG\2025_03_19\O1_MF_1_27
890_%U_.ARC
ORA-00280: change 17384974762395 for thread 1 is in sequence #27890


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
D\APP\ADMINISTRATOR\ORADATA\orcl\RED001.LOG
ORA-00308: 无法打开归档日志 'D\APP\ADMINISTRATOR\ORADATA\orcl\RED001.LOG'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 3) 系统找不到指定的路径。


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
D:\app\Administrator\oradata\orcl\RED001.LOG
ORA-00308: 无法打开归档日志 'D:\app\Administrator\oradata\orcl\RED001.LOG'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件。

最初数据库启动报ORA-600 kcratr_nab_less_than_odr错误(这个是一个非常典型的错误,早期写过处理方法:ORA-600 kcratr_nab_less_than_odr故障解决)，客户处理故障思路不太清晰,使用了recover database until cancel和alter database open resetlogs等不当操作,导致数据库没有open成功.然后希望使用我的Oracle Recovery Tools小工具进行修复,但是根据我的判断,这个故障还用不上该工具,直接可以open库.对其进行ctl重建并open库

SQL> alter database backup controlfile to trace as 'd:/ctl.txt';

Database altered.

SQL> create pfile='d:/pfile.txt' from spfile;

File created.

SQL> shutdown immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> startup nomount;
ORACLE instance started.

Total System Global Area 1603411968 bytes
Fixed Size                  2176168 bytes
Variable Size             973081432 bytes
Database Buffers          620756992 bytes
Redo Buffers                7397376 bytes
SQL> @d:/xifenfei/rectl.sql

Control file created.

SQL>
SQL>
SQL> recover database;
Media recovery complete.

SQL> alter database open;

Database altered.

在open之后,数据库报ORA-600 4194错误

Wed Mar 19 01:51:12 2025
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 23 processes
Started redo scan
Completed redo scan
 read 793 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 27890, block 2, scn 17384974741793
Recovery of Online Redo Log: Thread 1 Group 4 Seq 27890 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\orcl\REDO04.LOG
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 27890, block 1589, scn 17384974762395
 0 data blocks read, 0 data blocks written, 793 redo k-bytes read
Wed Mar 19 01:51:14 2025
Thread 1 advanced to log sequence 27891 (thread open)
Thread 1 opened at log sequence 27891
  Current log# 2 seq# 27891 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\orcl\REDO02.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Mar 19 01:51:14 2025
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Tablespace 'TEMP1' #11 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
           Empty temporary tablespace: TEMP1
*********************************************************************
Database Characterset is ZHS16GBK
No Resource Manager plan active
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_smon_4084.trc  (incident=158542):
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_158542\orcl_smon_4084_i158542.trc
Wed Mar 19 01:51:26 2025
Trace dumping is performing id=[cdmp_20250319015126]
Wed Mar 19 01:51:31 2025
Sweep [inc][158542]: completed
Sweep [inc2][158542]: completed
Wed Mar 19 01:51:35 2025
Doing block recovery for file 3 block 1415
Resuming block recovery (PMON) for file 3 block 1415
Block recovery from logseq 27891, block 86 to scn 17384974782720
Recovery of Online Redo Log: Thread 1 Group 2 Seq 27891 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\orcl\REDO02.LOG
Block recovery stopped at EOT rba 27891.87.16
Block recovery completed at rba 27891.87.16, scn 4047.3242135803
Doing block recovery for file 3 block 264
Resuming block recovery (PMON) for file 3 block 264
Block recovery from logseq 27891, block 86 to scn 17384974782714
Recovery of Online Redo Log: Thread 1 Group 2 Seq 27891 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\orcl\REDO02.LOG
Block recovery completed at rba 27891.87.16, scn 4047.3242135803
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_smon_4084.trc:
ORA-01595: 释放区 (2) 回退段 (12) 时出错
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
Wed Mar 19 01:51:36 2025
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_5680.trc  (incident=158590):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_158590\orcl_ora_5680_i158590.trc

数据库open成功,但是后台报ORA-01595/ORA-600 4194错误,这个问题比较常见,直接处理异常undo即可恢复.

21c数据库恢复遭遇ora-600 ktugct: corruption detected

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：21c数据库恢复遭遇ora-600 ktugct: corruption detected

接手一个oracle 21c的库恢复请求,通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)脚本检测之后,发现undo文件offline之后,做了resetlogs操作,导致该文件目前处于WRONG RESETLOGS状态
21c
wrong-resetlogs

尝试恢复数据库ORA-16433错误

SQL> recover datafile 1;
ORA-00283: recovery session canceled due to errors
ORA-16433: The database must be opened in read/write mode.

2025-03-11T12:07:43.381296+08:00
ALTER DATABASE RECOVER  datafile 1  
2025-03-11T12:07:43.381296+08:00
Media Recovery Start
2025-03-11T12:07:43.381296+08:00
Media Recovery failed with error 16433
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 1  ...

官方对于该错误的解释

[oracle@iZbp11c0qyuuo1gr7j98upZ ~]$ oerr ora 16433
16433, 00000, "The database must be opened in read/write mode."
// *Cause:  An attempt was made to open the database in read-only mode after an
//          operation that requires that the database be opened in read/write
//          mode.
// *Action: Open the database in read/write mode. The database can then be
//          opened in read-only mode.

对于该错误,可以通过rectl进行解决,但是由于undo文件的resetlog 信息不对,无法直接重建ctl,这种情况下,通过Oracle Recovery Tools小工具对undo文件的resetlogs信息今星期修改

然后重建ctl成功

C:\Users\XFF>sqlplus / as sysdba

SQL*Plus: Release 21.0.0.0.0 - Production on 星期二 3月 11 12:15:00 2025
Version 21.3.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.


连接到:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0

SQL> CREATE CONTROLFILE REUSE DATABASE "ORCLCDB" NORESETLOGS  NOARCHIVELOG
  2         MAXLOGFILES 50
  3         MAXLOGMEMBERS 5
  4         MAXDATAFILES 5000
  5         MAXINSTANCES 8
  6         MAXLOGHISTORY 2920
  7  LOGFILE
  8         group 1   'H:\TEMP\ORCLCDB\REDO01.LOG' size 200M,
  9         group 3   'H:\TEMP\ORCLCDB\REDO03.LOG' size 200M,
 10         group 2   'H:\TEMP\ORCLCDB\REDO02.LOG' size 200M
 11  DATAFILE
 12          'H:\TEMP\ORCLCDB\SYSTEM01.DBF',
 13          'H:\TEMP\ORCLCDB\SYSAUX01.DBF',
 14          'H:\TEMP\ORCLCDB\UNDOTBS01.DBF',
 15          'H:\TEMP\ORCLCDB\PDBSEED\SYSTEM01.DBF',
 16          'H:\TEMP\ORCLCDB\PDBSEED\SYSAUX01.DBF',
 17          'H:\TEMP\ORCLCDB\USERS01.DBF',
 18          'H:\TEMP\ORCLCDB\PDBSEED\UNDOTBS01.DBF',
 19          'H:\TEMP\ORCLCDB\ORCLPDB1\SYSTEM01.DBF',
 20          'H:\TEMP\ORCLCDB\ORCLPDB1\UNDOTBS01.DBF',
 21          'H:\TEMP\ORCLCDB\ORCLPDB1\USERS01.DBF'
 22  CHARACTER SET  AL32UTF8 ;

控制文件已创建。

尝试open库报ORA-600 ktugct: corruption detected错误

SQL> recover database;
完成介质恢复。
SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [ktugct: corruption detected], [],
[], [], [], [], [], [], [], [], [], []
进程 ID: 8372
会话 ID: 978 序列号: 21772

通过对启动过程进行分析,确认是在seq$表访问的时候报错

PARSING IN CURSOR #2238117495280 len=102 dep=1 uid=0 oct=3 lid=0 tim=821317050254 
 hv=3967354608 ad='7ff9048c41b8' sqlid='axmdf8vq7k1rh'
select increment$,minvalue,maxvalue,cycle#,order$,cache,highwater,audit$,flags from seq$ where obj#=:1
END OF STMT
PARSE #2238117495280:c=6253,e=6253,p=5,cr=90,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=821317050254
BINDS #2238117156016:

 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=48 off=0
  kxsbbbfp=1a33f610  bln=22  avl=02  flg=05
  value=100
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=0 off=24
  kxsbbbfp=1a33f628  bln=22  avl=02  flg=01
  value=1
EXEC #2238117156016:c=95,e=95,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=3,plh=2239883476,tim=821317050580
FETCH #2238117156016:c=8,e=8,p=0,cr=3,cu=0,mis=0,r=1,dep=2,og=3,plh=2239883476,tim=821317050600
CLOSE #2238117156016:c=3,e=3,dep=2,type=3,tim=821317050619
BINDS #2238117156016:

 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=48 off=0
  kxsbbbfp=1a33f610  bln=22  avl=02  flg=05
  value=100
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=0 off=24
  kxsbbbfp=1a33f628  bln=22  avl=02  flg=01
  value=2
EXEC #2238117156016:c=87,e=87,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=3,plh=2239883476,tim=821317050746
FETCH #2238117156016:c=7,e=7,p=0,cr=3,cu=0,mis=0,r=1,dep=2,og=3,plh=2239883476,tim=821317050764
CLOSE #2238117156016:c=3,e=3,dep=2,type=3,tim=821317050781
BINDS #2238117156016:

 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=48 off=0
  kxsbbbfp=1a33f610  bln=22  avl=02  flg=05
  value=100
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=0 off=24
  kxsbbbfp=1a33f628  bln=22  avl=02  flg=01
  value=3
EXEC #2238117156016:c=84,e=84,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=3,plh=2239883476,tim=821317050885
FETCH #2238117156016:c=6,e=6,p=0,cr=3,cu=0,mis=0,r=1,dep=2,og=3,plh=2239883476,tim=821317050902
CLOSE #2238117156016:c=2,e=2,dep=2,type=3,tim=821317050917
BINDS #2238117156016:

 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=48 off=0
  kxsbbbfp=1a33f610  bln=22  avl=02  flg=05
  value=100
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=0 off=24
  kxsbbbfp=1a33f628  bln=22  avl=02  flg=01
  value=4
EXEC #2238117156016:c=83,e=83,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=3,plh=2239883476,tim=821317051019
FETCH #2238117156016:c=5,e=5,p=0,cr=3,cu=0,mis=0,r=1,dep=2,og=3,plh=2239883476,tim=821317051035
CLOSE #2238117156016:c=3,e=2,dep=2,type=3,tim=821317051050
BINDS #2238117156016:

 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=48 off=0
  kxsbbbfp=1a33f610  bln=22  avl=02  flg=05
  value=100
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=0 off=24
  kxsbbbfp=1a33f628  bln=22  avl=02  flg=01
  value=5
EXEC #2238117156016:c=88,e=88,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=3,plh=2239883476,tim=821317051158
FETCH #2238117156016:c=5,e=5,p=0,cr=3,cu=0,mis=0,r=1,dep=2,og=3,plh=2239883476,tim=821317051174
CLOSE #2238117156016:c=2,e=2,dep=2,type=3,tim=821317051190
BINDS #2238117156016:

 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=48 off=0
  kxsbbbfp=1a33f610  bln=22  avl=02  flg=05
  value=100
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=0 off=24
  kxsbbbfp=1a33f628  bln=22  avl=02  flg=01
  value=6
EXEC #2238117156016:c=83,e=83,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=3,plh=2239883476,tim=821317051292
FETCH #2238117156016:c=5,e=6,p=0,cr=3,cu=0,mis=0,r=1,dep=2,og=3,plh=2239883476,tim=821317051308
CLOSE #2238117156016:c=2,e=2,dep=2,type=3,tim=821317051323
BINDS #2238117156016:

 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=48 off=0
  kxsbbbfp=1a33f610  bln=22  avl=02  flg=05
  value=100
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=0 off=24
  kxsbbbfp=1a33f628  bln=22  avl=02  flg=01
  value=7
EXEC #2238117156016:c=83,e=83,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=3,plh=2239883476,tim=821317051425
FETCH #2238117156016:c=5,e=5,p=0,cr=3,cu=0,mis=0,r=1,dep=2,og=3,plh=2239883476,tim=821317051440
CLOSE #2238117156016:c=3,e=3,dep=2,type=3,tim=821317051456
BINDS #2238117156016:

 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=48 off=0
  kxsbbbfp=1a33f610  bln=22  avl=02  flg=05
  value=100
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=0 off=24
  kxsbbbfp=1a33f628  bln=22  avl=02  flg=01
  value=8
EXEC #2238117156016:c=83,e=83,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=3,plh=2239883476,tim=821317051558
FETCH #2238117156016:c=5,e=5,p=0,cr=3,cu=0,mis=0,r=1,dep=2,og=3,plh=2239883476,tim=821317051573
CLOSE #2238117156016:c=2,e=2,dep=2,type=3,tim=821317051588
BINDS #2238117156016:

 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=48 off=0
  kxsbbbfp=1a33f610  bln=22  avl=02  flg=05
  value=100
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=0 off=24
  kxsbbbfp=1a33f628  bln=22  avl=02  flg=01
  value=9
EXEC #2238117156016:c=82,e=82,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=3,plh=2239883476,tim=821317051688
FETCH #2238117156016:c=5,e=5,p=0,cr=3,cu=0,mis=0,r=1,dep=2,og=3,plh=2239883476,tim=821317051704
CLOSE #2238117156016:c=2,e=2,dep=2,type=3,tim=821317051719
BINDS #2238117156016:

 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=48 off=0
  kxsbbbfp=1a33f610  bln=22  avl=02  flg=05
  value=100
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=0 off=24
  kxsbbbfp=1a33f628  bln=22  avl=02  flg=01
  value=10
EXEC #2238117156016:c=82,e=82,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=3,plh=2239883476,tim=821317051821
FETCH #2238117156016:c=5,e=5,p=0,cr=3,cu=0,mis=0,r=1,dep=2,og=3,plh=2239883476,tim=821317051837
CLOSE #2238117156016:c=2,e=2,dep=2,type=3,tim=821317051852
BINDS #2238117508144:

 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=72 off=0
  kxsbbbfp=1a33f5f8  bln=22  avl=02  flg=05
  value=100
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=0 off=24
  kxsbbbfp=1a33f610  bln=22  avl=02  flg=01
  value=10
 Bind#2
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=0 off=48
  kxsbbbfp=1a33f628  bln=22  avl=01  flg=01
  value=0
EXEC #2238117508144:c=144,e=144,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=3,plh=3312420081,tim=821317052013
FETCH #2238117508144:c=164,e=164,p=1,cr=3,cu=0,mis=0,r=3,dep=2,og=3,plh=3312420081,tim=821317052189
CLOSE #2238117508144:c=4,e=4,dep=2,type=3,tim=821317052208
BINDS #2238117495280:

 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=1000001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=1a33f628  bln=22  avl=03  flg=05
  value=1749
EXEC #2238117495280:c=2211,e=2211,p=1,cr=33,cu=0,mis=1,r=0,dep=1,og=4,plh=2203911306,tim=821317052521
WAIT #2238117495280: nam='db file sequential read' ela= 82 file#=1 block#=1201 blocks=1 obj#=105 tim=821317052620
WAIT #2238117495280: nam='db file sequential read' ela= 78 file#=1 block#=1202 blocks=1 obj#=105 tim=821317052723
WAIT #2238117495280: nam='db file sequential read' ela= 77 file#=1 block#=1161 blocks=1 obj#=100 tim=821317052824
ktuisc detected lost changes to transaction table:
 xid of interest:  xid: 0x0003.00c.0006e788 xid from block :  xid: 0x0003.00c.0006e785
2025-03-11T12:18:43.210707+08:00
Incident 25673 created, dump file: C:\APP\XFF\diag\rdbms\orclcdb\o21c\incident\incdir_25673\o21c_ora_12976_i25673.trc
ORA-00600: 内部错误代码, 参数: [ktugct: corruption detected], [], [], [], [], [], [], [], [], [], [], []

FETCH #2238117495280:c=1350284,e=1359602,p=3,cr=4,cu=0,mis=0,r=0,dep=1,og=4,plh=2203911306,tim=821318412135
STAT #2238117495280 id=1 cnt=0 pid=0 pos=1 obj=100 
  op='TABLE ACCESS BY INDEX ROWID SEQ$ (cr=0 pr=0 pw=0 str=1 time=1 us cost=2 size=73 card=1)'
STAT #2238117495280 id=2 cnt=1 pid=1 pos=1 obj=105 
  op='INDEX UNIQUE SCAN I_SEQ1 (cr=2 pr=2 pw=0 str=1 time=209 us cost=1 size=0 card=1)'
<error barrier> at 0x0000003197FF21D8 placed dbsdrv.c@5035
ORA-00600: 内部错误代码, 参数: [ktugct: corruption detected], [], [], [], [], [], [], [], [], [], [], []
<error barrier> at 0x0000003197FF21D8 placed dbsdrv.c@5035
ORA-00600: 内部错误代码, 参数: [ktugct: corruption detected], [], [], [], [], [], [], [], [], [], [], []
2025-03-11T12:18:44.572180+08:00
Incident 25674 created, dump file: C:\APP\XFF\diag\rdbms\orclcdb\o21c\incident\incdir_25674\o21c_ora_12976_i25674.trc
ORA-00603: ORACLE 服务器会话因致命错误而终止
ORA-01092: ORACLE 实例终止。强制断开连接
ORA-00600: 内部错误代码, 参数: [ktugct: corruption detected], [], [], [], [], [], [], [], [], [], [], []

进一步分析trace信息

[TOC00001]
ORA-00600: 内部错误代码, 参数: [ktugct: corruption detected], [], [], [], [], [], [], [], [], [], [], []

[TOC00001-END]
[TOC00002]
========= Dump for incident 25673 (ORA 600 [ktugct: corruption detected]) ========
[TOC00003]
----- Beginning of Customized Incident Dump(s) -----
Corruption or lost changes detected in transaction table
Cleanout struct - flg: 0x00000000 ne = 1 cu: 0 cv: 1 cec:0 actcnt:0
xids being resolved:
 xid: 0x0003.00c.0006e788 sit: 2
inverse itl idx to cleanout entry mapping:
Dump of memory from 0x000002091036B32A to 0x000002091036B429

Block cleanout record, scn: 0xffffffffffffffff ver: 0x01 opt: 0x01
, entries follow...
-- Start of buffer-cache dump for undo segment header --
Dump of buffer cache at level 10 for pdb=1 tsn=2 rdba=16777376
BH (0x7ffaccf5e9e0) file#: 4 rdba: 0x010000a0 (4/160) class: 21 ba: 0x7ffacc17c000
  set: 57 pool: 3 bsz: 8192 bsi: 0 sflg: 2 pwc: 0,0
  dbwrid: 0 obj: -1 objn: 0 tsn: [1/2] afn: 4 hint: f
  hash: [0x7ff91759e230,0x7ff91759e230] lru: [0x7ffaccf5ec30,0x7ffaccf5e960]
  ckptq: [NULL] fileq: [NULL]
  objq: [0x7ff9041e1708,0x7ff9041e1708] objaq: [0x7ff9041e16f8,0x7ff9041e16f8] qhead: 0x7ff9041e16e8
  st: XCURRENT md: NULL fpin: 'ktuwh117: ktugus:kturec' fscn: 0x1a321837 tch: 1
  flags:
  LRBA: [0x0.0.0] LSCN: [0x0] HSCN: [0x0] HSUB: [65535] 
  Printing buffer operation history (latest change first): (cnt 10)
  01. sid:14 L353:gcur:set:MEXCL      02. sid:14 L145:zib:mk:EXCL       
  03. sid:14 L212:zib:bic:FSQ         04. sid:14 L122:zgb:set:st        
  05. sid:14 L830:olq1:clr:WRT+CKT    06. sid:14 L951:zgb:lnk:objq      
  07. sid:14 L372:zgb:set:MEXCL       08. sid:14 L123:zgb:no:FEN        
  09. sid:14 L083:zgb:ent:fn          10. sid:00 L203:w_ini_dc:bic:FVB  
  buffer tsn: 2 rdba: 0x010000a0 (4/160)
  scn: 0x192ea609 seq: 0x01 flg: 0x04 tail: 0xa6092601
  frmt: 0x02 chkval: 0x3395 type: 0x26=KTU SMU HEADER BLOCK
Hex dump of block: st=0, typ_found=1
Dump of memory from 0x00007FFACC17C000 to 0x00007FFACC17E000
7FFACC17C000 0000A226 010000A0 192EA609 04010000  [&...............]
7FFACC17C010 00003395 00000000 00000000 00000000  [.3..............]
7FFACC17C020 00000000 00000004 0000010F 00000FF0  [................]
7FFACC17C030 00000000 00000000 00000007 010000A1  [................]
7FFACC17C040 00000000 00000000 00000000 00000000  [................]
7FFACC17C050 00000000 00000000 00000000 00000004  [................]
7FFACC17C060 00000000 00000000 40000000 010000A1  [...........@....]
7FFACC17C070 00000007 010000B8 00000008 01000780  [................]
7FFACC17C080 00000080 01000180 00000080 00000000  [................]
7FFACC17C090 00000000 00000000 00000000 00000000  [................]
        Repeat 246 times
7FFACC17D000 00000000 67770DDD 6776D1BB 6776EEC1  [......wg..vg..vg]
7FFACC17D010 67770DDD 6775F337 6775F337 6775F34B  [..wg7.ug7.ugK.ug]
7FFACC17D020 67734C0B 676F7532 676CB861 6734082B  [.Lsg2uoga.lg+.4g]
7FFACC17D030 6734082B 671AD100 671AD100 671AD100  [+.4g...g...g...g]
7FFACC17D040 671AD100 671AA6DC 67014F31 67014F31  [...g...g1O.g1O.g]
7FFACC17D050 67014F31 67014F31 67014F31 62BB38EE  [1O.g1O.g1O.g.8.b]
7FFACC17D060 62BB38EE 62BB38EE 62BB38EE 62BB38EE  [.8.b.8.b.8.b.8.b]
        Repeat 1 times
7FFACC17D080 62BB38EE 62BB38EE 62BB0EE3 62BB0EE3  [.8.b.8.b...b...b]
7FFACC17D090 62BB0EE3 62BB0EE3 62BB0EE3 62BB0EE3  [...b...b...b...b]
        Repeat 14 times
7FFACC17D180 62BB0EE3 00000000 00000000 00000000  [...b............]
7FFACC17D190 00000000 00000000 00000000 00000000  [................]
        Repeat 99 times
7FFACC17D7D0 192EA31E 00000000 010000A1 001E3150  [............P1..]
7FFACC17D7E0 31500001 00000001 00000000 00080004  [..P1............]
7FFACC17D7F0 0001B000 00000068 7FFFFFFE 010000A1  [....h...........]
7FFACC17D800 001E3150 055E0000 00000000 0007314E  [P1....^.....N1..]
7FFACC17D810 0CEC0002 00000000 000B3146 1A180002  [........F1......]
7FFACC17D820 00000000 00173103 12B40002 00000000  [.....1..........]
7FFACC17D830 00000000 00000000 0006E7A0 010000A1  [................]
7FFACC17D840 192EA53E 00000000 00100009 00000000  [>...............]
7FFACC17D850 00000000 00000000 00000001 67770DEE  [..............wg]
7FFACC17D860 0006E7AF 010000A1 192EA4E4 00000000  [................]
7FFACC17D870 000F0009 00000000 00000000 00000000  [................]
7FFACC17D880 00000001 67770DE8 0006E790 010000A1  [......wg........]
7FFACC17D890 192EA58D 00000000 001F0009 00000000  [................]
7FFACC17D8A0 00000000 00000000 00000001 67770DF3  [..............wg]
7FFACC17D8B0 0006E797 010001FF 192EA431 00000000  [........1.......]
7FFACC17D8C0 000B0009 00000000 00000000 00000000  [................]
7FFACC17D8D0 00000001 67770DDC 0006E795 010001FF  [......wg........]
7FFACC17D8E0 192EA33D 00000000 000D0009 00000000  [=...............]
7FFACC17D8F0 00000000 00000000 00000001 67770DCC  [..............wg]
7FFACC17D900 0006E79A 010000A1 192EA4D1 00000000  [................]
7FFACC17D910 00160009 00000000 00000000 00000000  [................]
7FFACC17D920 00000001 67770DE6 0006E792 010000A1  [......wg........]
7FFACC17D930 192EA472 00000000 000A0009 00000000  [r...............]
7FFACC17D940 00000000 00000000 00000001 67770DE0  [..............wg]
7FFACC17D950 0006E7A6 010001FF 192EA3FB 00000000  [................]
7FFACC17D960 00090009 00000000 00000000 00000000  [................]
7FFACC17D970 00000001 67770DD8 0006E7A7 010000A1  [......wg........]
7FFACC17D980 192EA609 00000000 FFFF0009 00000000  [................]
7FFACC17D990 00000000 00000000 00000001 67770DFB  [..............wg]
7FFACC17D9A0 0006E79C 010001FF 192EA41A 00000000  [................]
7FFACC17D9B0 00030009 00000000 00000000 00000000  [................]
7FFACC17D9C0 00000001 67770DDA 0006E79E 010000A1  [......wg........]
7FFACC17D9D0 192EA481 00000000 00150009 00000000  [................]
7FFACC17D9E0 00000000 00000000 00000001 67770DE1  [..............wg]
7FFACC17D9F0 0006E7A3 010000A1 192EA44C 00000000  [........L.......]
7FFACC17DA00 00190009 00000000 00000000 00000000  [................]
7FFACC17DA10 00000002 67770DDD 0006E785 010001FF  [......wg........]
7FFACC17DA20 192EA391 00000000 00180009 00000000  [................]
7FFACC17DA30 00000000 00000000 00000001 67770DD1  [..............wg]
7FFACC17DA40 0006E7AB 010001FF 192EA355 00000000  [........U.......]
7FFACC17DA50 001A0009 00000000 00000000 00000000  [................]
7FFACC17DA60 00000001 67770DCD 0006E793 010000A1  [......wg........]
7FFACC17DA70 192EA51F 00000000 00000009 00000000  [................]
7FFACC17DA80 00000000 00000000 00000001 67770DEC  [..............wg]
7FFACC17DA90 0006E7AD 010000A1 192EA4FA 00000000  [................]
7FFACC17DAA0 000E0009 00000000 00000000 00000000  [................]
7FFACC17DAB0 00000001 67770DE9 0006E794 010000A1  [......wg........]
7FFACC17DAC0 192EA550 00000000 00130009 00000000  [P...............]
7FFACC17DAD0 00000000 00000000 00000001 67770DEF  [..............wg]
7FFACC17DAE0 0006E792 010000A1 192EA5DF 00000000  [................]
7FFACC17DAF0 00210009 00000000 00000000 00000000  [..!.............]
7FFACC17DB00 00000001 67770DF9 0006E766 010000A1  [......wgf.......]
7FFACC17DB10 192EA5C2 00000000 001E0009 00000000  [................]
7FFACC17DB20 00000000 00000000 00000001 67770DF7  [..............wg]
7FFACC17DB30 0006E787 010000A1 192EA561 00000000  [........a.......]
7FFACC17DB40 001C0009 00000000 00000000 00000000  [................]
7FFACC17DB50 00000001 67770DF0 0006E7A3 010001FF  [......wg........]
7FFACC17DB60 192EA3D2 00000000 00070009 00000000  [................]
7FFACC17DB70 00000000 00000000 00000001 67770DD6  [..............wg]
7FFACC17DB80 0006E7A5 010000A1 192EA498 00000000  [................]
7FFACC17DB90 00200009 00000000 00000000 00000000  [.. .............]
7FFACC17DBA0 00000001 67770DE2 0006E791 010000A1  [......wg........]
7FFACC17DBB0 192EA4DD 00000000 00010009 00000000  [................]
7FFACC17DBC0 00000000 00000000 00000001 67770DE7  [..............wg]
7FFACC17DBD0 0006E79F 010000A1 192EA5F6 00000000  [................]
7FFACC17DBE0 00080009 00000000 00000000 00000000  [................]
7FFACC17DBF0 00000001 67770DFA 0006E78A 010001FF  [......wg........]
7FFACC17DC00 192EA3A7 00000000 001D0009 00000000  [................]
7FFACC17DC10 00000000 00000000 00000001 67770DD3  [..............wg]
7FFACC17DC20 0006E797 010000A1 192EA45D 00000000  [........].......]
7FFACC17DC30 00060009 00000000 00000000 00000000  [................]
7FFACC17DC40 00000001 67770DDF 0006E796 010001FF  [......wg........]
7FFACC17DC50 192EA368 00000000 001B0009 00000000  [h...............]
7FFACC17DC60 00000000 00000000 00000001 67770DCF  [..............wg]
7FFACC17DC70 0006E78E 010001FF 192EA37D 00000000  [........}.......]
7FFACC17DC80 000C0009 00000000 00000000 00000000  [................]
7FFACC17DC90 00000001 67770DD0 0006E798 010000A1  [......wg........]
7FFACC17DCA0 192EA56F 00000000 00020009 00000000  [o...............]
7FFACC17DCB0 00000000 00000000 00000001 67770DF1  [..............wg]
7FFACC17DCC0 0006E7A1 010001FF 192EA3B5 00000000  [................]
7FFACC17DCD0 00140009 00000000 00000000 00000000  [................]
7FFACC17DCE0 00000001 67770DD4 0006E7A2 010000A1  [......wg........]
7FFACC17DCF0 192EA5CE 00000000 00110009 00000000  [................]
7FFACC17DD00 00000000 00000000 00000001 67770DF7  [..............wg]
7FFACC17DD10 0006E794 010000A1 192EA5A0 00000000  [................]
7FFACC17DD20 00120009 00000000 00000000 00000000  [................]
7FFACC17DD30 00000001 67770DF4 0006E7A2 010000A1  [......wg........]
7FFACC17DD40 192EA4A5 00000000 00050009 00000000  [................]
7FFACC17DD50 00000000 00000000 00000001 67770DE3  [..............wg]
7FFACC17DD60 0006E7A5 010000A1 192EA5E9 00000000  [................]
7FFACC17DD70 00170009 00000000 00000000 00000000  [................]
7FFACC17DD80 00000001 67770DF9 00000003 00000000  [......wg........]
7FFACC17DD90 00000000 00000000 00000000 00000000  [................]
        Repeat 1 times
7FFACC17DDB0 00000001 00000000 00000000 00000000  [................]
        Repeat 33 times
7FFACC17DFD0 00000000 00000000 00000000 00000000  [................]
        Repeat 1 times
7FFACC17DFF0 00000000 00000000 00000000 A6092601  [.............&..]
  Extent Control Header
  -----------------------------------------------------------------
  Extent Header:: spare1: 0      spare2: 0      #extents: 4      #blocks: 271   
                  last map  0x00000000  #maps: 0      offset: 4080  
      Highwater::  0x010000a1  ext#: 0      blk#: 0      ext size: 7     
  #blocks in seg. hdr's freelists: 0     
  #blocks below: 0     
  mapblk  0x00000000  offset: 0     
                   Unlocked
     Map Header:: next  0x00000000  #extents: 4    obj#: 0      flag: 0x40000000
  Extent Map
  -----------------------------------------------------------------
   0x010000a1  length: 7     
   0x010000b8  length: 8     
   0x01000780  length: 128   
   0x01000180  length: 128   
  
 Retention Table 
  -----------------------------------------------------------
 Extent Number:0  Commit Time: 1735855581
 Extent Number:1  Commit Time: 1735840187
 Extent Number:2  Commit Time: 1735847617
 Extent Number:3  Commit Time: 1735855581
  
  TRN CTL:: seq: 0x3150 chd: 0x0004 ctl: 0x0008 inc: 0x00000000 nfb: 0x0001
            mgc: 0xb000 xts: 0x0068 flg: 0x0001 opt: 2147483646 (0x7ffffffe)
            uba: 0x010000a1.3150.1e scn: 0x00000000192ea31e
Version: 0x01
  FREE BLOCK POOL::
    uba: 0x010000a1.3150.1e ext: 0x0  spc: 0x55e   
    uba: 0x00000000.314e.07 ext: 0x2  spc: 0xcec   
    uba: 0x00000000.3146.0b ext: 0x2  spc: 0x1a18  
    uba: 0x00000000.3103.17 ext: 0x2  spc: 0x12b4  
    uba: 0x00000000.0000.00 ext: 0x0  spc: 0x0     
  TRN TBL::
 
  index  state cflags  wrap#    uel         scn            dba            parent-xid    nub     stmt_num    cmt
  -----------------------------------------------------------------------------------------
   0x00  9 0x00  0x6e7a0  0x0010  0x00000000192ea53e  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855598
   0x01  9 0x00  0x6e7af  0x000f  0x00000000192ea4e4  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855592
   0x02  9 0x00  0x6e790  0x001f  0x00000000192ea58d  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855603
   0x03  9 0x00  0x6e797  0x000b  0x00000000192ea431  0x010001ff 0x0000.000.00000000  0x00000001 0x00000000 1735855580
   0x04  9 0x00  0x6e795  0x000d  0x00000000192ea33d  0x010001ff 0x0000.000.00000000  0x00000001 0x00000000 1735855564
   0x05  9 0x00  0x6e79a  0x0016  0x00000000192ea4d1  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855590
   0x06  9 0x00  0x6e792  0x000a  0x00000000192ea472  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855584
   0x07  9 0x00  0x6e7a6  0x0009  0x00000000192ea3fb  0x010001ff 0x0000.000.00000000  0x00000001 0x00000000 1735855576
   0x08  9 0x00  0x6e7a7  0xffff  0x00000000192ea609  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855611
   0x09  9 0x00  0x6e79c  0x0003  0x00000000192ea41a  0x010001ff 0x0000.000.00000000  0x00000001 0x00000000 1735855578
   0x0a  9 0x00  0x6e79e  0x0015  0x00000000192ea481  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855585
   0x0b  9 0x00  0x6e7a3  0x0019  0x00000000192ea44c  0x010000a1 0x0000.000.00000000  0x00000002 0x00000000 1735855581
   0x0c  9 0x00  0x6e785  0x0018  0x00000000192ea391  0x010001ff 0x0000.000.00000000  0x00000001 0x00000000 1735855569
   0x0d  9 0x00  0x6e7ab  0x001a  0x00000000192ea355  0x010001ff 0x0000.000.00000000  0x00000001 0x00000000 1735855565
   0x0e  9 0x00  0x6e793  0x0000  0x00000000192ea51f  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855596
   0x0f  9 0x00  0x6e7ad  0x000e  0x00000000192ea4fa  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855593
   0x10  9 0x00  0x6e794  0x0013  0x00000000192ea550  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855599
   0x11  9 0x00  0x6e792  0x0021  0x00000000192ea5df  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855609
   0x12  9 0x00  0x6e766  0x001e  0x00000000192ea5c2  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855607
   0x13  9 0x00  0x6e787  0x001c  0x00000000192ea561  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855600
   0x14  9 0x00  0x6e7a3  0x0007  0x00000000192ea3d2  0x010001ff 0x0000.000.00000000  0x00000001 0x00000000 1735855574
   0x15  9 0x00  0x6e7a5  0x0020  0x00000000192ea498  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855586
   0x16  9 0x00  0x6e791  0x0001  0x00000000192ea4dd  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855591
   0x17  9 0x00  0x6e79f  0x0008  0x00000000192ea5f6  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855610
   0x18  9 0x00  0x6e78a  0x001d  0x00000000192ea3a7  0x010001ff 0x0000.000.00000000  0x00000001 0x00000000 1735855571
   0x19  9 0x00  0x6e797  0x0006  0x00000000192ea45d  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855583
   0x1a  9 0x00  0x6e796  0x001b  0x00000000192ea368  0x010001ff 0x0000.000.00000000  0x00000001 0x00000000 1735855567
   0x1b  9 0x00  0x6e78e  0x000c  0x00000000192ea37d  0x010001ff 0x0000.000.00000000  0x00000001 0x00000000 1735855568
   0x1c  9 0x00  0x6e798  0x0002  0x00000000192ea56f  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855601
   0x1d  9 0x00  0x6e7a1  0x0014  0x00000000192ea3b5  0x010001ff 0x0000.000.00000000  0x00000001 0x00000000 1735855572
   0x1e  9 0x00  0x6e7a2  0x0011  0x00000000192ea5ce  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855607
   0x1f  9 0x00  0x6e794  0x0012  0x00000000192ea5a0  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855604
   0x20  9 0x00  0x6e7a2  0x0005  0x00000000192ea4a5  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855587
   0x21  9 0x00  0x6e7a5  0x0017  0x00000000192ea5e9  0x010000a1 0x0000.000.00000000  0x00000001 0x00000000 1735855609
  EXT TRN CTL::
  usn: 3
  sp1:0x00000000 sp2:0x00000000 sp3:0x00000000 sp4:0x00000000
  sp5:0x00000000 sp6:0x00000000 sp7:0x00000000 sp8:0x00000000
  EXT TRN TBL::
  index      extflag    extHash    extPdbid/Spare1 extSpare2
  -----------------------------------------------------------
   0x00    0x00000000   0x00000000   0x00000001    0x00000000
   0x01    0x00000000   0x00000000   0x00000001    0x00000000
   0x02    0x00000000   0x00000000   0x00000001    0x00000000
   0x03    0x00000000   0x00000000   0x00000001    0x00000000
   0x04    0x00000000   0x00000000   0x00000001    0x00000000
   0x05    0x00000000   0x00000000   0x00000001    0x00000000
   0x06    0x00000000   0x00000000   0x00000001    0x00000000
   0x07    0x00000000   0x00000000   0x00000001    0x00000000
   0x08    0x00000000   0x00000000   0x00000001    0x00000000
   0x09    0x00000000   0x00000000   0x00000001    0x00000000
   0x0a    0x00000000   0x00000000   0x00000001    0x00000000
   0x0b    0x00000000   0x00000000   0x00000001    0x00000000
   0x0c    0x00000000   0x00000000   0x00000001    0x00000000
   0x0d    0x00000000   0x00000000   0x00000001    0x00000000
   0x0e    0x00000000   0x00000000   0x00000001    0x00000000
   0x0f    0x00000000   0x00000000   0x00000001    0x00000000
   0x10    0x00000000   0x00000000   0x00000001    0x00000000
   0x11    0x00000000   0x00000000   0x00000001    0x00000000
   0x12    0x00000000   0x00000000   0x00000001    0x00000000
   0x13    0x00000000   0x00000000   0x00000001    0x00000000
   0x14    0x00000000   0x00000000   0x00000001    0x00000000
   0x15    0x00000000   0x00000000   0x00000001    0x00000000
   0x16    0x00000000   0x00000000   0x00000001    0x00000000
   0x17    0x00000000   0x00000000   0x00000001    0x00000000
   0x18    0x00000000   0x00000000   0x00000001    0x00000000
   0x19    0x00000000   0x00000000   0x00000001    0x00000000
   0x1a    0x00000000   0x00000000   0x00000001    0x00000000
   0x1b    0x00000000   0x00000000   0x00000001    0x00000000
   0x1c    0x00000000   0x00000000   0x00000001    0x00000000
   0x1d    0x00000000   0x00000000   0x00000001    0x00000000
   0x1e    0x00000000   0x00000000   0x00000001    0x00000000
   0x1f    0x00000000   0x00000000   0x00000001    0x00000000
   0x20    0x00000000   0x00000000   0x00000001    0x00000000
   0x21    0x00000000   0x00000000   0x00000001    0x00000000
Dump of buffer cache for pdb 1 tsn 2 rdba 0x10000a0 at level 10 done.

基于上述信息基本上可以确认是由于我们修复的undo文件和seq$中的file#=1 block#=1161不匹配导致该问题,对其进行处理之后,数据库open成功

2025-03-11T13:12:52.837511+08:00
alter database open
2025-03-11T13:12:52.860336+08:00
Smart fusion block transfer is disabled:
  instance mounted in exclusive mode.
Buffer Cache Full DB Caching mode changing from FULL CACHING DISABLED to FULL CACHING ENABLED 
2025-03-11T13:12:52.864333+08:00
Crash Recovery excluding pdb 2 which was cleanly closed.
2025-03-11T13:12:52.864333+08:00
Crash Recovery excluding pdb 3 which was cleanly closed.
Endian type of dictionary set to little
2025-03-11T13:12:52.886321+08:00
Redo log for group 1, sequence 7 is not located on DAX storage
Thread 1 opened at log sequence 7
  Current log# 1 seq# 7 mem# 0: H:\TEMP\ORCLCDB\REDO01.LOG
Successful open of redo thread 1
2025-03-11T13:12:52.894310+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Stopping change tracking
Undo initialization recovery: Parallel FPTR complete: start:824531531 end:824531531 diff:0 ms (0.0 seconds)
Undo initialization recovery: err:0 start: 824531531 end: 824531531 diff: 0 ms (0.0 seconds)
[13696] Successfully onlined Undo Tablespace 5.
Undo initialization online undo segments: err:0 start: 824531531 end: 824531609 diff: 78 ms (0.1 seconds)
Undo initialization finished serial:0 start:824531531 end:824531609 diff:78 ms (0.1 seconds)
Database Characterset is AL32UTF8
No Resource Manager plan active
Starting background process RCBG
2025-03-11T13:12:53.502478+08:00
RCBG started with pid=48, OS id=17560_12292 
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process AQPC
2025-03-11T13:12:53.724661+08:00
AQPC started with pid=49, OS id=17560_12948 
PDB$SEED(2):Pluggable database PDB$SEED opening in read only
PDB$SEED(2):Autotune of undo retention is turned on. 
PDB$SEED(2):Endian type of dictionary set to little
PDB$SEED(2):Undo initialization finished serial:0 start:824532375 end:824532375 diff:0 ms (0.0 seconds)
PDB$SEED(2):Database Characterset for PDB$SEED is AL32UTF8
PDB$SEED(2):*********************************************************************
PDB$SEED(2):WARNING: The following temporary tablespaces in container(PDB$SEED)
PDB$SEED(2):         contain no files.
PDB$SEED(2):         This condition can occur when a backup controlfile has
PDB$SEED(2):         been restored.  It may be necessary to add files to these
PDB$SEED(2):         tablespaces.  That can be done using the SQL statement:
PDB$SEED(2): 
PDB$SEED(2):         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
PDB$SEED(2): 
PDB$SEED(2):         Alternatively, if these temporary tablespaces are no longer
PDB$SEED(2):         needed, then they can be dropped.
PDB$SEED(2):           Empty temporary tablespace: TEMP
PDB$SEED(2):*********************************************************************
2025-03-11T13:12:53.936415+08:00
PDB$SEED(2):SUPLOG: Set PDB SUPLOG SGA at PDB OPEN, old 0x0, new 0x0 (no suplog)
PDB$SEED(2):Opening pdb with no Resource Manager plan active
Starting background process CJQ0
2025-03-11T13:12:54.249096+08:00
CJQ0 started with pid=77, OS id=17560_17968 
Completed: alter database open

所幸客户需要的数据直接在cdb中,直接使用expdp导出客户需要的数据,完成本次恢复

删除redo导致ORA-00313 ORA-00312故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：删除redo导致ORA-00313 ORA-00312故障处理

有客户由于误操作直接rm 删除了redo文件,导致数据库启动报ORA-00313 ORA-00312错

2025-03-07T14:49:16.325723+08:00
ALTER DATABASE OPEN
2025-03-07T14:50:00.124620+08:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
2025-03-07T14:50:00.198907+08:00
Crash Recovery excluding pdb 2 which was cleanly closed.
2025-03-07T14:50:00.238450+08:00
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
 Thread 1: Recovery starting at checkpoint rba (logseq 2966 block 74686), scn 0
2025-03-07T14:50:00.325246+08:00
Started redo scan
2025-03-07T14:50:00.341193+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2681.trc:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/u01/app/oracle/oradata/orcl/redo02.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
2025-03-07T14:50:00.372632+08:00
Slave encountered ORA-10388 exception during crash recovery
…………
2025-03-07T14:50:00.385698+08:00
Slave encountered ORA-10388 exception during crash recovery
2025-03-07T14:50:00.388594+08:00
Aborting crash recovery due to error 313
2025-03-07T14:50:00.388739+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2681.trc:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/u01/app/oracle/oradata/orcl/redo02.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
2025-03-07T14:50:00.389243+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2681.trc:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/u01/app/oracle/oradata/orcl/redo02.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
ORA-313 signalled during: ALTER DATABASE OPEN...

然后客户把历史的redo文件拷贝过来,尝试恢复数据库,报ORA-00314 ORA-00312错误

2025-03-07T15:07:30.784759+08:00
ALTER DATABASE OPEN
Ping without log force is disabled:
  instance mounted in exclusive mode.
2025-03-07T15:07:30.808497+08:00
Crash Recovery excluding pdb 2 which was cleanly closed.
2025-03-07T15:07:30.838664+08:00
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
 Thread 1: Recovery starting at checkpoint rba (logseq 2966 block 74686), scn 0
2025-03-07T15:07:30.897547+08:00
Started redo scan
2025-03-07T15:07:30.898222+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_4106.trc:
ORA-00314: log 2 of thread 1, expected sequence# 2966 doesn't match 1646
ORA-00312: online log 2 thread 1: '/u01/app/oracle/oradata/orcl/redo02.log'
2025-03-07T15:07:30.930089+08:00
Slave encountered ORA-10388 exception during crash recovery
…………
2025-03-07T15:07:30.940051+08:00
Slave encountered ORA-10388 exception during crash recovery
2025-03-07T15:07:30.942274+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_mz00_4138.trc:
ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log'
2025-03-07T15:07:30.945509+08:00
Slave encountered ORA-10388 exception during crash recovery
2025-03-07T15:07:30.945512+08:00
Slave encountered ORA-10388 exception during crash recovery
2025-03-07T15:07:30.948369+08:00
Aborting crash recovery due to error 314
2025-03-07T15:07:30.948488+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_4106.trc:
ORA-00314: log 2 of thread 1, expected sequence# 2966 doesn't match 1646
ORA-00312: online log 2 thread 1: '/u01/app/oracle/oradata/orcl/redo02.log'
2025-03-07T15:07:30.949390+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_4106.trc:
ORA-00314: log 2 of thread 1, expected sequence# 2966 doesn't match 1646
ORA-00312: online log 2 thread 1: '/u01/app/oracle/oradata/orcl/redo02.log'
ORA-314 signalled during: ALTER DATABASE OPEN...

使用Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)脚本收集信息之后数据文件头状态和所需要redo信息
df_header

数据库需要sequence#为2966的redo日志,但是当前已经被删除,基于当前情况,只能进行强制非一致性恢复,尝试强制打开库

SQL> recover database;                 
ORA-00283: recovery session canceled due to errors
ORA-00314: log 2 of thread 1, expected sequence# 2966 doesn't match 1646
ORA-00312: online log 2 thread 1: '/u01/app/oracle/oradata/orcl/redo02.log'

QL> select group#,status,sequence# from v$log;

	  GROUP# STATUS 		 SEQUENCE#
---------------- ---------------- ----------------
	       1 UNUSED 			 0
	       3 CURRENT		      2967
	       2 ACTIVE 		      2966

SQL> 
SQL> 
SQL> recover database until cancel;
ORA-00279: change 163033183 generated at 03/07/2025 14:04:20 needed for thread 1
ORA-00289: suggestion : /u01/app/oracle/recovery_area/orcl/archivelog/2025_03_08/o1_mf_1_2966_%u_.arc
ORA-00280: change 163033183 for thread 1 is in sequence #2966


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/u01/app/oracle/oradata/orcl/system01.dbf'


ORA-01112: media recovery not started


SQL> alter database open resetlogs;

Database altered.

运气不错,直接打开数据库成功,然后逻辑导出数据,完成此处恢复.这个让我想起来了一些类似案例:
Oracle 23ai rm redo*.log恢复
 清空redo,导致ORA-27048: skgfifi: file header information is invalid
由于默认情况下oracle的redo文件扩展名是.log,然后被当做是不重要文件从而被清理导致数据库故障,在oracle服务器上清理数据之前建议查询v$datafile,v$logfile,v$tempfile,v$controlfile来确认是否是数据库文件

linux rm -rf 删除数据文件恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：linux rm -rf 删除数据文件恢复

有客户由于误操作删除了oracle的部分数据文件(rm -rf 方式删除),然后自己尝试进行恢复操作,对部分文件执行了offline,导致比较麻烦的后果

offline-file

接手故障之后,第一时间对其进行了镜像(因为有部分文件句柄已经释放,为了方式覆盖进一步破坏),对于没有释放的句柄可以通过类似方法进行恢复,参考以前类似恢复:
Oracle误删除数据文件恢复
Solaris rm datafile recovery—利用句柄误删除数据文件恢复

!cp  269  /u01/app/oracle/oradata/orcl/XXXXXX_DATA01.dbf
alter database datafile 12 offline;
recover datafile 12;
alter database datafile 12 online;

对于删除文件,而且句柄已经释放的文件,通过文件系统层面反删除进行恢复,参考以前类似恢复:
rm -rf误删Oracle数据库恢复
 记录一次rm -rf 删除数据文件异常恢复
 rm -rf 删除数据文件恢复方法—文件系统反删除+oracle碎片重组

在这个恢复过程中,由于客户linux是物理机,而且本地空间不足,无法对其进行镜像,采用dd命令直接写镜像到其他的linux机器上（通过nfs方式),然后在win机器上直接挂载该nfs,记录下win上挂载nfs操作
nfs

CDM备份缺少归档打开数据库报ORA-600 kcbzib_kcrsds_1故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：CDM备份缺少归档打开数据库报ORA-600 kcbzib_kcrsds_1故障处理

有客户联系我们,说一个19c的库,由于产生归档较多在备份过程中把部分归档删除而导致没有备份成功,从而使得他们那边一个误操作需要恢复使用备份做不完全恢复,备份厂商进行恢复并尝试强制打开库报ORA-600 kcbzib_kcrsds_1错误
ORA-600-kcbzibz_kcrsds_1

由于客户那边使用的是CDM方式备份,可以较快的准备出来一个新环境,观察客户在应用日志过程中,文件头的scn一直不变,怀疑文件头由于begin backup冻结,对其进行dump,发现确实做了hot backup操作,而且没有end backup
scn

由于缺少归档,不完全恢复没有成功,begin backup 也无法正常结束,对于这种情况,先尝试调整到文件头最大的scn值,尝试打开库

SQL> alter database open resetlogs ;
alter database open resetlogs 
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [],
[], [], [], [], [], [], []
Process ID: 3949615
Session ID: 5111 Serial number: 30040


--重启到mount状态
SQL> set numw 16
col CHECKPOINT_TIME for a40
set lines 150
set pages 1000
SELECT status,
to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,FUZZY,checkpoint_change#,
count(*) ROW_NUM
FROM v$datafile_header
GROUP BY status, checkpoint_change#, to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss'),fuzzy
ORDER BY status, checkpoint_change#, checkpoint_time;SQL> SQL> SQL> SQL>   2    3    4    5    6  

STATUS  CHECKPOINT_TIME                          FUZ CHECKPOINT_CHANGE#          ROW_NUM
------- ---------------------------------------- --- ------------------ ----------------
ONLINE  2025-02-08 22:43:01                      YES     15626238353558               56

打开库失败,只能找出来数据库最大的block中最大scn,然后调整文件头scn的值,实现数据库open

SQL> oradebug setmypid
Statement processed.
SQL> oradebug DUMPvar SGA kcsgscn_
SQL> kscn8 kcsgscn_ [060017E98, 060017EA0) = 00000000 00000000
SQL> oradebug DUMPvar SGA kcsgscn_
kscn8 kcsgscn_ [060017E98, 060017EA0) = 8CD9C896 00000E4D
SQL> 
SQL> 
SQL> 
SQL> alter database open ;
alter database open 
*
ERROR at line 1:
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: '/cdmbak/db/xifenfei/ob_data_D-XIFENFEI-SYSTEM_FNO-1_t13930nd'

SQL> recover database;
Media recovery complete.
SQL> alter database open;

Database altered.

SQL>

使用exp导出客户需要的表,完成本次恢复任务
对于ORA-600 kcbzib_kcrsds_1恢复的情况,以前有过大量恢复案例,修改数据库scn即可
kcbzib_kcrsds_1报错汇总
 12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
 存储故障,强制拉库报ORA-600 kcbzib_kcrsds_1处理
 Patch SCN工具一键恢复ORA-600 kcbzib_kcrsds_1
此类故障处理太多,不一一列举,解决这个错误之后,数据库open成功,然后安排逻辑迁移即可

ORA-07445: exception encountered: core dump [expgod()+43] [IN_PAGE_ERROR]

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-07445: exception encountered: core dump [expgod()+43] [IN_PAGE_ERROR]

数据库在运行过程中报O/S-Error: (OS 23) 数据错误(循环冗余检查)错误

Thu Jan 30 22:00:02 2025
Begin automatic SQL Tuning Advisor run for special tuning task  "SYS_AUTO_SQL_TUNING_TASK"
Thu Jan 30 22:00:04 2025
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_j000_12576.trc:
ORA-12012: error on auto execute of job 155962
ORA-01115: IO error reading block from file  (block # )
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'
ORA-27070: async read/write failed
OSD-04006: ReadFile() 失败, 无法读取文件
O/S-Error: (OS 23) 数据错误(循环冗余检查)。
ORA-06512: at "SYS.DBMS_STATS", line 25836
ORA-06512: at "SYS.DBMS_STATS", line 26171
End automatic SQL Tuning Advisor run for special tuning task  "SYS_AUTO_SQL_TUNING_TASK"
Fri Jan 31 02:00:00 2025
Clearing Resource Manager plan via parameter
Fri Jan 31 08:15:46 2025
Thread 1 advanced to log sequence 4420 (LGWR switch)
  Current log# 1 seq# 4420 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Fri Jan 31 10:53:57 2025
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process 
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_cjq0_1140.trc:
Fri Jan 31 10:53:57 2025
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_j000_7916.trc:
ORA-27102: out of memory
OSD-00043: 附加错误信息
O/S-Error: (OS 1455) 页面文件太小，无法完成操作。
Process J000 died, see its trace file
kkjcre1p: unable to spawn jobq slave process 
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_cjq0_1140.trc:
Fri Jan 31 10:54:03 2025
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0x18] [PC:0xB778B02, clsdcxini()+90]
ERROR: Unable to normalize symbol name for the following short stack (at offset 199):
dbgexProcessError()+193<-dbgeExecuteForError()+65<-dbgePostErrorKGE()+1726<-dbkePostKGE_kgsf()+75
<-kgeade()+560<-kgerev()+125<-kgerec5()+60<-sss_xcpt_EvalFilterEx()+1869<-sss_xcpt_EvalFilter()+174
<-.1.4_5+59<-00007FFD0245F306<-00007FFD024735AF<-00007FFD023D4AAF<-00007FFD0247231E<-clsdcxini()+90
<-clsdinit()+124<-ksdnfy()+225<-kscnfy()+778<-opirip()+86<-opidrv()+909<-sou2o()+98<-opimai_real()+299
<-opimai()+191<-BackgroundThreadStart()+693<-00007FFD020E7E94<-00007FFD02437AD1
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_j000_9920.trc  (incident=39621):
ORA-07445: exception encountered: core dump [clsdcxini()+90][ACCESS_VIOLATION][ADDR:0x18][PC:0xB778B02][UNABLE_TO_READ]
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_39621\orcl_j000_9920_i39621.trc

然后无法正常启动,报Exception [type: IN_PAGE_ERROR, ] [] [PC:0x2C9C015, expgod()+43]错误

Wed Feb 05 09:43:51 2025
Sweep [inc][39621]: completed
Successful mount of redo thread 1, with mount id 1720066005
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: alter database mount exclusive
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 3 processes
Started redo scan
Completed redo scan
 read 140 KB redo, 62 data blocks need recovery
Started redo application at
 Thread 1: logseq 4420, block 42375
Wed Feb 05 09:44:00 2025
Recovery of Online Redo Log: Thread 1 Group 1 Seq 4420 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Completed redo application of 0.09MB
Completed crash recovery at
 Thread 1: logseq 4420, block 42656, scn 94456019
 62 data blocks read, 62 data blocks written, 140 redo k-bytes read
Wed Feb 05 09:44:01 2025
Exception [type: IN_PAGE_ERROR, ] [] [PC:0x2C9C015, expgod()+43]
ERROR: Unable to normalize symbol name for the following short stack (at offset 199):
dbgexProcessError()+193<-dbgeExecuteForError()+65<-dbgePostErrorKGE()+1726
<-dbkePostKGE_kgsf()+75<-kgeade()+560<-kgerev()+125<-kgerec5()+60<-sss_xcpt_EvalFilterEx()+1869
<-sss_xcpt_EvalFilter()+174<-.1.4_5+59<-00007FFD0245F306<-00007FFD024735AF<-00007FFD023D4AAF
<-00007FFD0247231E<-expgod()+43<-xtyopncb()+241<-qctcopn()+613<-qctcopn()+392<-qctcpqb()+290
<-qctcpqbl()+52<-xtydrv()+148<-opitca()+1091<-kksLoadChild()+9008<-kxsGetRuntimeLock()+2320
<-kksfbc()+15225<-kkspsc0()+2117<-kksParseCursor()+181<-opiosq0()+2538<-opiosq()+23<-opiodr()+1662
<-rpidrus()+862<-rpidru()+154<-rpiswu2()+2757<-rpidrv()+6105<-rpisplu()+1607<-kqldFixedTableLoadCols()+345
<-kqldcor()+2534<-kglslod()+352<-kqlslod()+52<-PGOSF455_kqlsublod()+125<-kqllod()+7284<-kglobld()+1354
<-kglobpn()+1900<-kglpim()+336<-qcdlgtd()+260<-qcsfplob()+166<-qcsprfro()+903<-qcsprfro_tree()
+292<-qcsprfro_tree()+373<-qcspafq()+96
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_mmon_15468.trc  (incident=39749):
ORA-07445: exception encountered: core dump [expgod()+43] [IN_PAGE_ERROR] [] [PC:0x2C9C015] [] []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_39749\orcl_mmon_15468_i39749.trc
Wed Feb 05 09:44:02 2025
Thread 1 advanced to log sequence 4421 (thread open)
Thread 1 opened at log sequence 4421
  Current log# 2 seq# 4421 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Feb 05 09:44:02 2025
SMON: enabling cache recovery
Wed Feb 05 09:44:11 2025
Exception [type: IN_PAGE_ERROR, ] [] [PC:0x2C9C015, expgod()+43]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_9308.trc  (incident=39781):
ORA-07445: ??????: ???? [expgod()+43] [IN_PAGE_ERROR] [] [PC:0x2C9C015] [] []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_39781\orcl_ora_9308_i39781.trc
Wed Feb 05 09:44:19 2025
PMON (ospid: 12376): terminating the instance due to error 397
Instance terminated by PMON, pid = 12376

基于上述的Exception [type: IN_PAGE_ERROR, ] [] [PC:0x2C9C015, expgod()+43]错误,第一反应就是可能由于底层损坏导致数据块损坏,dbv检查文件是否报错
dbv-system

检查系统日志确认异常
20250208203059

尝试拷贝文件也报错

已经比较明确由于底层问题,解决给问题之前,需要先对文件系统进行处理,然后再对恢复出来的数据文件恢复数据

2025年第一起ORA-600 16703故障恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：2025年第一起ORA-600 16703故障恢复

又有一个客户数据库启动报ORA-600 16703错误
ora-600-16703

查看alert日志信息

Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_ora_8784.trc  (incident=20617):
ORA-00600: 内部错误代码, 参数: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\incident\incdir_20617\xff_ora_8784_i20617.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_ora_8784.trc:
ORA-00704: 引导程序进程失败
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_ora_8784.trc:
ORA-00704: 引导程序进程失败
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 8784): terminating the instance due to error 704
Instance terminated by USER, pid = 8784
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (8784) as a result of ORA-1092

这个故障一般是由于oracle安装介质被注入了恶意脚本,在数据库运行300天之后重启数据库就会遭遇该问题:
警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703
对于这个问题,我们这边经过多年的恢复经验积累和研究,可以实现直接open库并且不用做逻辑迁移,完美恢复之后业务直接使用,最大限度恢复数据和最小限度减少停机时间
以往类似恢复case汇总:
tab$恢复错误汇总
 ORA-600 16703故障再现
 tab$异常被处理之后报ORA-600 13304故障处理
 ORA-600 16703直接把orachk备份表插入到tab$恢复
 最近遇到几起ORA-600 16703故障(tab$被清空),请引起重视
 ORA-00600: internal error code, arguments: [16703], [1403], [32]
aix平台tab$被删除可能出现ORA-600 [16703], [1403], [28]错误
 ORA-00600: internal error code, arguments: [16703], [1403], [4] 原因
 ORA-00600: internal error code, arguments: [16703], [1403], [4] 故障处理

ORA-600 16703故障,客户找人恢复数据库,数据库被进一步恶意破坏—ORA-00704 ORA-00922

ORA-600 ktuPopDictI_1恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 ktuPopDictI_1恢复

数据库启动报ORA-00600: internal error code, arguments: [ktuPopDictI_1], [4]错误

[oracle@ora19c:/home/oracle]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jan 20 21:42:28 2025
Version 19.24.0.0.0

Copyright (c) 1982, 2024, Oracle.  All rights reserved.

Connected to an idle instance.

sys@ORA19C 21:38:22> startup
ORACLE instance started.

Total System Global Area  763359928 bytes
Fixed Size                  9183928 bytes
Variable Size             457179136 bytes
Database Buffers          289406976 bytes
Redo Buffers                7589888 bytes
Database mounted.
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [ktuPopDictI_1], [4], [], [], [], [], [], [], [], [], [], []
Process ID: 3254475
Session ID: 410 Serial number: 22754

数据库alert日志报错

2025-01-20T21:38:30.411924+08:00
ALTER DATABASE OPEN
2025-01-20T21:38:30.437769+08:00
Smart fusion block transfer is disabled:
  instance mounted in exclusive mode.
2025-01-20T21:38:30.445071+08:00
Crash Recovery excluding pdb 2 which was cleanly closed.
2025-01-20T21:38:30.445125+08:00
Crash Recovery excluding pdb 3 which was cleanly closed.
2025-01-20T21:38:30.445172+08:00
Crash Recovery excluding pdb 4 which was cleanly closed.
Endian type of dictionary set to little
2025-01-20T21:38:30.459107+08:00
LGWR (PID:3254425): STARTING ARCH PROCESSES
2025-01-20T21:38:30.466458+08:00
TT00 (PID:3254477): Gap Manager starting
Starting background process ARC0
2025-01-20T21:38:30.474126+08:00
ARC0 started with pid=39, OS id=3254479 
2025-01-20T21:38:30.484228+08:00
LGWR (PID:3254425): ARC0: Archival started
LGWR (PID:3254425): STARTING ARCH PROCESSES COMPLETE
2025-01-20T21:38:30.484325+08:00
ARC0 (PID:3254479): Becoming a 'no FAL' ARCH
ARC0 (PID:3254479): Becoming the 'no SRL' ARCH
2025-01-20T21:38:30.495886+08:00
Redo log for group 3, sequence 447 is not located on DAX storage
Thread 1 opened at log sequence 447
  Current log# 3 seq# 447 mem# 0: /data/oradata/ORA19C/redo03.log
Successful open of redo thread 1
2025-01-20T21:38:30.512816+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Stopping change tracking
Undo initialization recovery: Parallel FPTR complete: start:2947972792 end:2947972831 diff:39 ms (0.0 seconds)
Undo initialization recovery: err:0 start: 2947972791 end: 2947972831 diff: 40 ms (0.0 seconds)
[3254475] Successfully onlined Undo Tablespace 2.
Undo initialization online undo segments: err:0 start: 2947972831 end: 2947972933 diff: 102 ms (0.1 seconds)
ORA-00600: internal error code, arguments: [ktuPopDictI_1], [4], [], [], [], [], [], [], [], [], [], []
Incident details in: /data/app/oracle/diag/rdbms/ora19c/ora19c/incident/incdir_38705/ora19c_ora_3254475_i38705.trc
2025-01-20T21:38:31.482586+08:00
TMON (PID:3254467): STARTING ARCH PROCESSES
Starting background process ARC1
2025-01-20T21:38:31.491744+08:00
ARC1 started with pid=41, OS id=3254483 
Starting background process ARC2
2025-01-20T21:38:31.500274+08:00
ARC2 started with pid=42, OS id=3254485 
Starting background process ARC3
2025-01-20T21:38:31.508426+08:00
ARC3 started with pid=43, OS id=3254487 
TMON (PID:3254467): ARC1: Archival started
TMON (PID:3254467): ARC2: Archival started
TMON (PID:3254467): ARC3: Archival started
TMON (PID:3254467): STARTING ARCH PROCESSES COMPLETE
2025-01-20T21:38:31.715480+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-01-20T21:38:31.798619+08:00
Errors in file /data/app/oracle/diag/rdbms/ora19c/ora19c/trace/ora19c_ora_3254475.trc:
ORA-00600: internal error code, arguments: [ktuPopDictI_1], [4], [], [], [], [], [], [], [], [], [], []
2025-01-20T21:38:31.798666+08:00
Errors in file /data/app/oracle/diag/rdbms/ora19c/ora19c/trace/ora19c_ora_3254475.trc:
ORA-00600: internal error code, arguments: [ktuPopDictI_1], [4], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [ktuPopDictI_1], [4], [], [], [], [], [], [], [], [], [], []
Incident details in: /data/app/oracle/diag/rdbms/ora19c/ora19c/incident/incdir_38706/ora19c_ora_3254475_i38706.trc
opiodr aborting process unknown ospid (3254475) as a result of ORA-603
2025-01-20T21:38:32.356148+08:00
ORA-603 : opitsk aborting process
License high water mark = 1
USER(prelim) (ospid: 3254475): terminating the instance due to ORA error 600

分析trace信息

[TOC00001]
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [ktuPopDictI_1], [4], [], [], [], [], [], [], [], [], [], []

[TOC00001-END]
[TOC00002]
========= Dump for incident 38706 (ORA 603) ========

*** 2025-01-20T21:38:31.823904+08:00
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
[TOC00003]
----- Current SQL Statement for this session (sql_id=1h50ks4ncswfn) -----
ALTER DATABASE OPEN
[TOC00003-END]

[TOC00004]
----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedst1()+95         call     kgdsdst()            7FFEA4EB7360 000000002
                                                   7FFEA4EB15B0 ? 7FFEA4EB16C8 ?
                                                   000000000 000000000
ksedst()+58          call     ksedst1()            000000000 000000001
                                                   7FFEA4EB15B0 ? 7FFEA4EB16C8 ?
                                                   000000000 ? 000000000 ?
dbkedDefDump()+2434  call     ksedst()             000000000 000000001 ?
7                                                  7FFEA4EB15B0 ? 7FFEA4EB16C8 ?
                                                   000000000 ? 000000000 ?
ksedmp()+577         call     dbkedDefDump()       000000003 000000002
                                                   7FFEA4EB15B0 ? 7FFEA4EB16C8 ?
                                                   000000000 ? 000000000 ?
dbgexPhaseII()+2092  call     ksedmp()             0000003EB 000000002 ?
                                                   7FFEA4EB15B0 ? 7FFEA4EB16C8 ?
                                                   000000000 ? 000000000 ?
dbgexProcessError()  call     dbgexPhaseII()       7F2A059ED6D8 7F2A002BF148
+1871                                              7FFEA4EB8E00 7FFEA4EB16C8 ?
                                                   000000000 ? 000000000 ?
dbgePostErrorKGE()+  call     dbgexProcessError()  7F2A059ED6D8 7F2A002BF148
1851                                               000000001 000000000
                                                   000000000 ? 000000000 ?
dbkePostKGE_kgsf()+  call     dbgePostErrorKGE()   7F2A05A2D9C0 7F2A058D0050
71                                                 00000025B 000000000 ?
                                                   000000000 ? 000000000 ?
kgeade()+339         call     dbkePostKGE_kgsf()   7F2A05A2D9C0 7F2A058D0050
                                                   00000025B 000000000 ?
                                                   000000000 ? 000000000 ?
kgefecl()+184        call     kgeade()             7F2A05A2D9C0 ? 7F2A05A2DC08 ?
                                                   7F2A058D0050 ? 00000025B ?
                                                   000000000 000000000
adbdrv_options()+48  call     kgefecl()            7F2A05A2D9C0 7F2A058D0050
548                                                000000444 000000001 ?
                                                   014971F9C ? 014973858 ?
opiexe()+31984       call     adbdrv_options()     000000000 7F2A058D0050 ?
                                                   000000444 ? 000000001 ?
                                                   014971F9C ? 014973858 ?
opiosq0()+4560       call     opiexe()             000000004 7F2A058D0050 ?
                                                   7FFEA4EC16D0 000000001 ?
                                                   014971F9C ? 014973858 ?
kpooprx()+287        call     opiosq0()            000000003 7F2A058D0050 ?
                                                   7F2A05A2D9C0 ? 0000000A4
                                                   000000000 000000023
kpoal8()+838         call     kpooprx()            7FFEA4EC5824 7FFEA4EC2F40
                                                   000000013 000000001 000000000
                                                   0000000A4
opiodr()+1253        call     kpoal8()             00000005E 000000026
                                                   7FFEA4EC5820 000000001 ?
                                                   000000000 ? 0000000A4 ?
ttcpip()+1216        call     opiodr()             00000005E 000000026
                                                   7FFEA4EC5820 ? 000000000
                                                   000000000 ? 0000000A4 ?
opitsk()+1916        call     ttcpip()             7F2A05A57B30 ? 000000026 ?
                                                   7FFEA4EC5820 000000000 ?
                                                   7FFEA4EC5280 7FFEA4EC5A80 ?
opiino()+936         call     opitsk()             000000000 000000000
                                                   7FFEA4EC5820 ? 000000000 ?
                                                   7FFEA4EC5280 ? 7FFEA4EC5A80 ?
opiodr()+1253        call     opiino()             00000003C 000000004
                                                   7FFEA4EC7418 000000000 ?
                                                   7FFEA4EC5280 ? 7FFEA4EC5A80 ?
opidrv()+1067        call     opiodr()             00000003C 000000004
                                                   7FFEA4EC7418 ? 000000000
                                                   7FFEA4EC5280 ? 7FFEA4EC5A80 ?
sou2o()+165          call     opidrv()             00000003C 000000004
                                                   7FFEA4EC7418 000000000 ?
                                                   7FFEA4EC5280 ? 7FFEA4EC5A80 ?
opimai_real()+422    call     sou2o()              7FFEA4EC73F0 00000003C
                                                   000000004 7FFEA4EC7418
                                                   7FFEA4EC5280 ? 7FFEA4EC5A80 ?
ssthrdmain()+417     call     opimai_real()        000000000 7FFEA4EC7C08
                                                   000000004 ? 7FFEA4EC7418 ?
                                                   7FFEA4EC5280 ? 7FFEA4EC5A80 ?
main()+256           call     ssthrdmain()         000000000 000000002
                                                   7FFEA4EC7C08 000000001
                                                   000000000 7FFEA4EC5A80 ?
__libc_start_main()  call     main()               000000002 7FFEA4EC7E58
+243                                               7FFEA4EC7C08 ? 000000001 ?
                                                   000000000 ? 7FFEA4EC5A80 ?
_start()+46          call     __libc_start_main()  000DFEF50 000000002
                                                   7FFEA4EC7E58 00746DD60 ?
                                                   000000000 ? 7FFEA4EC5A80 ?
[TOC00004-END]


[TOC00005]
--------------------- Binary Stack Dump ---------------------

对启动过程进行跟踪,确认报错具体位置

PARSING IN CURSOR #140457326129448 len=45 dep=1 tim=11537999627952 hv=2164165332 ad='69329ae8'sqlid='8su8qaa0gx2qn'
select dataobj# from obj$ where name like :1
END OF STMT
PARSE #140457326129448:c=15,e=15,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=1478545678,tim=11537999627952
BINDS #140457326129448:

 Bind#0
  oacdty=01 mxl=32(07) mxlc=00 mal=00 scl=00 pre=00
  oacflg=20 fl2=0000 frm=01 csi=873 siz=32 off=0
  kxsbbbfp=7fbec507eb20  bln=32  avl=07  flg=05
  value="I_UNDO2"
EXEC #140457326129448:c=51,e=51,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=1478545678,tim=11537999628044
FETCH #140457326129448:c=175,e=175,p=0,cr=11,cu=0,mis=0,r=1,dep=1,og=4,plh=1478545678,tim=11537999628226
STAT #140457326129448 id=1 cnt=1 pid=0 pos=1 obj=18 op='TABLE ACCESS BY INDEX ROWID BATCHED OBJ$ (cr=11  card=1)'
STAT #140457326129448 id=2 cnt=1 pid=1 pos=1 obj=37 op='INDEX SKIP SCAN I_OBJ2 (cr=10 time=177 us cost=26 size=0 card=1)'
CLOSE #140457326129448:c=40,e=40,dep=1,type=1,tim=11537999628290
PARSE #140457326129448:c=5,e=5,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=1478545678,tim=11537999628302
BINDS #140457326129448:

 Bind#0
  oacdty=01 mxl=32(09) mxlc=00 mal=00 scl=00 pre=00
  oacflg=20 fl2=0000 frm=01 csi=873 siz=32 off=0
  kxsbbbfp=7fbec507eb20  bln=32  avl=09  flg=05
  value="UNDOHIST$"
EXEC #140457326129448:c=31,e=31,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=1478545678,tim=11537999628342
WAIT #140457326129448: nam='db file sequential read' ela= 6 file#=1 block#=243 blocks=1 obj#=18 tim=11537999628370
FETCH #140457326129448:c=33,e=33,p=1,cr=5,cu=0,mis=0,r=1,dep=1,og=4,plh=1478545678,tim=11537999628381
CLOSE #140457326129448:c=4,e=4,dep=1,type=3,tim=11537999628409
PARSE #140457326129448:c=6,e=6,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=1478545678,tim=11537999628430
BINDS #140457326129448:

 Bind#0
  oacdty=01 mxl=32(11) mxlc=00 mal=00 scl=00 pre=00
  oacflg=20 fl2=0000 frm=01 csi=873 siz=32 off=0
  kxsbbbfp=7fbec507eb20  bln=32  avl=11  flg=05
  value="I_UNDOHIST1"
EXEC #140457326129448:c=39,e=39,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=1478545678,tim=11537999628485
WAIT #140457326129448: nam='db file sequential read' ela= 649 file#=1 block#=78527 blocks=1 obj#=37 tim=11537999629346
WAIT #140457326129448: nam='db file sequential read' ela= 747 file#=1 block#=78505 blocks=1 obj#=37 tim=11537999630122
WAIT #140457326129448: nam='db file sequential read' ela= 628 file#=1 block#=23459 blocks=1 obj#=37 tim=11537999630775
WAIT #140457326129448: nam='db file sequential read' ela= 470 file#=1 block#=78552 blocks=1 obj#=37 tim=11537999631268
WAIT #140457326129448: nam='db file sequential read' ela= 502 file#=1 block#=92776 blocks=1 obj#=37 tim=11537999631797
WAIT #140457326129448: nam='db file sequential read' ela= 512 file#=1 block#=78585 blocks=1 obj#=37 tim=11537999632331
WAIT #140457326129448: nam='db file sequential read' ela= 538 file#=1 block#=78557 blocks=1 obj#=37 tim=11537999632892
WAIT #140457326129448: nam='db file sequential read' ela= 520 file#=1 block#=19641 blocks=1 obj#=37 tim=11537999633440
WAIT #140457326129448: nam='db file sequential read' ela= 502 file#=1 block#=23486 blocks=1 obj#=37 tim=11537999633971
WAIT #140457326129448: nam='db file sequential read' ela= 465 file#=1 block#=23504 blocks=1 obj#=37 tim=11537999634461
WAIT #140457326129448: nam='db file sequential read' ela= 656 file#=1 block#=23488 blocks=1 obj#=37 tim=11537999635141
WAIT #140457326129448: nam='db file sequential read' ela= 429 file#=1 block#=23501 blocks=1 obj#=37 tim=11537999635596
WAIT #140457326129448: nam='db file sequential read' ela= 621 file#=1 block#=92686 blocks=1 obj#=37 tim=11537999636240
WAIT #140457326129448: nam='db file sequential read' ela= 1027 file#=1 block#=92678 blocks=1 obj#=37 tim=11537999637307
WAIT #140457326129448: nam='db file sequential read' ela= 494 file#=1 block#=92680 blocks=1 obj#=37 tim=11537999637831
WAIT #140457326129448: nam='db file sequential read' ela= 515 file#=1 block#=92682 blocks=1 obj#=37 tim=11537999638377
WAIT #140457326129448: nam='db file sequential read' ela= 559 file#=1 block#=92684 blocks=1 obj#=37 tim=11537999638975
WAIT #140457326129448: nam='db file sequential read' ela= 478 file#=1 block#=92683 blocks=1 obj#=37 tim=11537999639502
WAIT #140457326129448: nam='db file sequential read' ela= 402 file#=1 block#=92687 blocks=1 obj#=37 tim=11537999639951
WAIT #140457326129448: nam='db file sequential read' ela= 465 file#=1 block#=92691 blocks=1 obj#=37 tim=11537999640453
WAIT #140457326129448: nam='db file sequential read' ela= 629 file#=1 block#=92694 blocks=1 obj#=37 tim=11537999641112
WAIT #140457326129448: nam='db file sequential read' ela= 507 file#=1 block#=109829 blocks=1 obj#=37 tim=11537999641651
WAIT #140457326129448: nam='db file sequential read' ela= 467 file#=1 block#=109831 blocks=1 obj#=37 tim=11537999642150
WAIT #140457326129448: nam='db file sequential read' ela= 525 file#=1 block#=109833 blocks=1 obj#=37 tim=11537999642695
WAIT #140457326129448: nam='db file sequential read' ela= 823 file#=1 block#=109837 blocks=1 obj#=37 tim=11537999643540
WAIT #140457326129448: nam='db file sequential read' ela= 553 file#=1 block#=109834 blocks=1 obj#=37 tim=11537999644111
WAIT #140457326129448: nam='db file sequential read' ela= 509 file#=1 block#=109835 blocks=1 obj#=37 tim=11537999644650
FETCH #140457326129448:c=1777,e=16184,p=27,cr=38,cu=0,mis=0,r=0,dep=1,og=4,plh=1478545678,tim=11537999644675
2025-01-20T21:40:02.951778+08:00
ORA-00600: internal error code, arguments: [ktuPopDictI_1], [4], [], [], [], [], [], [], [], [], [], []

<error barrier> at 0x7ffe9566b3c0 placed dbsdrv.c@5141
ORA-00600: internal error code, arguments: [ktuPopDictI_1], [4], [], [], [], [], [], [], [], [], [], []
<error barrier> at 0x7ffe9566b3c0 placed dbsdrv.c@5141
ORA-00600: internal error code, arguments: [ktuPopDictI_1], [4], [], [], [], [], [], [], [], [], [], []
2025-01-20T21:40:04.951374+08:00
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [ktuPopDictI_1], [4], [], [], [], [], [], [], [], [], [], []

通过上述定位确认是select dataobj# from obj$ where name like :1这个sql在查询记录时报错,通过一些技巧绕过该sql,实现数据库正常open

19c非归档数据库断电导致ORA-00742故障恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：19c非归档数据库断电导致ORA-00742故障恢复

客户一套运行在win平台，非归档的19c数据库,由于异常断电导致数据库启动报ORA-01113,进行recover操作之后报ORA-00742

open之时alert日志报错信息

2025-01-18T00:17:52.205669+08:00
alter database open
2025-01-18T00:17:53.417839+08:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
2025-01-18T00:17:53.436858+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_4428.trc:
ORA-01113: 文件 1 需要介质恢复
ORA-01110: 数据文件 1: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSTEM01.DBF'
2025-01-18T00:17:53.442863+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_4428.trc:
ORA-01113: 文件 1 需要介质恢复
ORA-01110: 数据文件 1: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSTEM01.DBF'
ORA-1113 signalled during: alter database open...

recover database 数据库的alert日志报错Media Recovery failed with error 742和
ORA-01110 ORA-01208等错误

2025-01-18T00:20:10.196227+08:00
ALTER DATABASE RECOVER  database  
2025-01-18T00:20:10.221244+08:00
Media Recovery Start
 Started logmerger process
2025-01-18T00:20:10.459413+08:00
WARNING! Recovering data file 1 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 3 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 4 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 7 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 60 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 64 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 65 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 66 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 67 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
2025-01-18T00:20:10.599512+08:00
Parallel Media Recovery started with 12 slaves
2025-01-18T00:20:10.664559+08:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 2097 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO03.LOG
2025-01-18T00:20:12.644962+08:00
Media Recovery failed with error 742
2025-01-18T00:20:12.759043+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_4036.trc:
ORA-01110: 数据文件 1: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSTEM01.DBF'
ORA-01208: 数据文件是旧的版本 - 不能访问当前版本
2025-01-18T00:20:13.135309+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_4036.trc:
ORA-01110: 数据文件 3: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSAUX01.DBF'
ORA-01208: 数据文件是旧的版本 - 不能访问当前版本
2025-01-18T00:20:13.455536+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_4036.trc:
ORA-01110: 数据文件 4: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\UNDOTBS01.DBF'
ORA-01208: 数据文件是旧的版本 - 不能访问当前版本
2025-01-18T00:20:14.408212+08:00
ORA-283 signalled during: ALTER DATABASE RECOVER  database  ...

尝试recover datafile 1

SQL> RECOVER DATAFILE 1;
ORA-00283: 恢复会话因错误而取消
ORA-00742: 日志读取在线程 1 序列 2097 块 296728 中检测到写入丢失情况
ORA-00312: 联机日志 3 线程 1: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO03.LOG'

对于这种情况,比较明显是redo文件有写丢失,导致数据库无法正常的应用redo日志进行恢复,从而无法正常open.这种情况,只能只能选择屏蔽一致性,尝试强制打开数据库

C:\Users\Administrator\Desktop\check_db>sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Jan 18 00:59:28 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> recover database using backup controlfile until cancel;
ORA-00279: ?? 10103231167 (? 01/17/2025 09:20:12 ??) ???? 1 ????
ORA-00289: ??:
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000002097_1079211060.0001
ORA-00280: ?? 10103231167 (???? 1) ??? #2097 ?


指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO03.LOG
ORA-00283: ??????????????????????????????
ORA-00742: ????????????????????? 1 ?????? 2097 ??? 296960
??????????????????????????????
ORA-00334: ????????????: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO03.LOG'


ORA-01112: ???????


SQL> alter database open resetlogs;

Database altered.

alert日志对应的信息

2025-01-18T01:00:15.756033+08:00
alter database open resetlogs
2025-01-18T01:00:15.903141+08:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 10103247614 time 
.... (PID:4824): Clearing online redo logfile 1 D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG
.... (PID:4824): Clearing online redo logfile 2 D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO02.LOG
.... (PID:4824): Clearing online redo logfile 3 D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO03.LOG
Clearing online log 1 of thread 1 sequence number 2098
Clearing online log 2 of thread 1 sequence number 2096
Clearing online log 3 of thread 1 sequence number 2097
2025-01-18T01:00:19.381697+08:00
.... (PID:4824): Clearing online redo logfile 1 complete
.... (PID:4824): Clearing online redo logfile 2 complete
.... (PID:4824): Clearing online redo logfile 3 complete
Resetting resetlogs activation ID 3599991024 (0xd69380f0)
Online log D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG: Thread 1 Group 1 was previously cleared
Online log D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO02.LOG: Thread 1 Group 2 was previously cleared
Online log D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO03.LOG: Thread 1 Group 3 was previously cleared
2025-01-18T01:00:19.550821+08:00
Setting recovery target incarnation to 2
2025-01-18T01:00:20.418458+08:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
Initializing SCN for created control file
Database SCN compatibility initialized to 3
2025-01-18T01:00:25.466167+08:00
Endian type of dictionary set to little
2025-01-18T01:00:25.476174+08:00
Assigning activation ID 3711463735 (0xdd387137)
2025-01-18T01:00:25.491185+08:00
TT00 (PID:3332): Gap Manager starting
2025-01-18T01:00:25.507197+08:00
Redo log for group 1, sequence 1 is not located on DAX storage
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG
Successful open of redo thread 1
2025-01-18T01:00:26.162679+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2025-01-18T01:00:26.183694+08:00
TT03 (PID:1896): Sleep 5 seconds and then try to clear SRLs in 2 time(s)
2025-01-18T01:00:27.465636+08:00
Undo initialization recovery: err:0 start: 3158125 end: 3158390 diff: 265 ms (0.3 seconds)
Undo initialization online undo segments: err:0 start: 3158390 end: 3158390 diff: 0 ms (0.0 seconds)
Undo initialization finished serial:0 start:3158125 end:3158406 diff:281 ms (0.3 seconds)
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Verifying minimum file header compatibility for tablespace encryption..
Verifying file header compatibility for tablespace encryption completed for pdb 0
Database Characterset is AL32UTF8
2025-01-18T01:00:28.790609+08:00
No Resource Manager plan active
2025-01-18T01:00:30.086561+08:00
replication_dependency_tracking turned off (no async multimaster replication found)
2025-01-18T01:00:31.185369+08:00
TT03 (PID:1896): Sleep 10 seconds and then try to clear SRLs in 3 time(s)
2025-01-18T01:00:31.536626+08:00
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Starting background process AQPC
2025-01-18T01:00:31.638701+08:00
AQPC started with pid=38, OS id=4340 
2025-01-18T01:00:32.717495+08:00
Starting background process CJQ0
2025-01-18T01:00:32.726501+08:00
CJQ0 started with pid=40, OS id=1236 
Completed: alter database open resetlogs

数据库没有明显报错,直接resetlogs成功,直接逻辑导出数据,导入新库,完成本次恢复工作

2025年首个故障恢复—ORA-600 kcbzib_kcrsds_1

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：2025年首个故障恢复—ORA-600 kcbzib_kcrsds_1

一个12.2.0.1的库由于某种原因引起的双机切换,导致数据库无法正常mount

2025-01-04T15:45:44.424193+08:00
alter database mount
2025-01-04T15:45:48.491054+08:00
Network throttle feature is disabled as mount time

2025-01-04T15:45:48.601366+08:00
LGWR (ospid: 34014): terminating the instance
2025-01-04T15:45:48.602480+08:00
System state dump requested by (instance=1, osid=34014 (LGWR)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/xifenfei/trace/xifenfei_diag_33978_20250104154548.trc
2025-01-04T15:45:48.790674+08:00
Dumping diagnostic data in directory=[cdmp_20250104154548], requested by (instance=1, osid=34014 (LGWR))
2025-01-04T15:45:49.915068+08:00
Instance terminated by LGWR, pid = 34014

这个错误相对比较明显,是由于ctl异常导致,通过重建ctl,然后mount库,利用Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)脚本进行检测发现所有数据文件头的checkpoint 信息被冻结在 2024-11-29 19:00:29 （scn 2112302221),分析alert日志数据库在此后20天中正常提供服务,业务运行都正常,客户反馈在这个冻结checkpoint信息的时间点,使用备份一体机发起过备份,之后就没有再备份了.
当时急着恢复数据库,没有对文件头进行dump不然应该可以发现类似begin backup的信息,类似这样（测试环境重现）：

DATA FILE #1:
  name #7: /u01/app/oracle/oradata/xifenfei/system01.dbf
creation size=0 block size=8192 status=0xe head=7 tail=7 dup=1
 tablespace 0, index=1 krfil=1 prev_file=0
 unrecoverable scn: 0x0000.00000000 01/01/1988 00:00:00
 Checkpoint cnt:625 scn: 0x0105.0106deef 01/04/2025 22:02:50
 Stop scn: 0xffff.ffffffff 12/14/2024 08:15:07
 Creation Checkpointed at scn:  0x0000.00000007 08/24/2013 11:37:33
 thread:0 rba:(0x0.0.0)
 enabled  threads:  00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000
 Offline scn: 0x0000.000e2005 prev_range: 0
 Online Checkpointed at scn:  0x0000.000e2006 03/20/2024 20:53:56
 thread:1 rba:(0x1.2.0)
 enabled  threads:  01000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000
 Hot Backup end marker scn: 0x0000.00000000
 aux_file is NOT DEFINED
 Plugged readony: NO
 Plugin scnscn: 0x0000.00000000
 Plugin resetlogs scn/timescn: 0x0000.00000000 01/01/1988 00:00:00
 Foreign creation scn/timescn: 0x0000.00000000 01/01/1988 00:00:00
 Foreign checkpoint scn/timescn: 0x0000.00000000 01/01/1988 00:00:00
 Online move state: 0
 V10 STYLE FILE HEADER:
        Compatibility Vsn = 186647552=0xb200400
        Db ID=1780931490=0x6a26dba2, Db Name='XIFENFEI'
        Activation ID=0=0x0
        Control Seq=32953021=0x1f6d2bd, File size=98560=0x18100
        File Number=1, Blksiz=8192, File Type=3 DATA
Tablespace #0 - SYSTEM  rel_fn:1
Creation   at   scn: 0x0000.00000007 08/24/2013 11:37:33
Backup taken at scn: 0x0105.0106deef 01/04/2025 22:02:50 thread:1    <====注意
 reset logs count:0x45636764 scn: 0x0000.000e2006
 prev reset logs count:0x3121c97a scn: 0x0000.00000001
 recovered at 12/14/2024 08:36:35
 status:0x2001 root dba:0x00400208 chkpt cnt: 625 ctl cnt:624
begin-hot-backup file size: 98560                        <====注意
Checkpointed at scn:  0x0105.0106deef 01/04/2025 22:02:50
 thread:1 rba:(0x205.fdd9.10)
 enabled  threads:  01000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000
Backup Checkpointed at scn:  0x0105.0106df14 01/04/2025 22:03:20   <====注意
 thread:1 rba:(0x209.2.10)
 enabled  threads:  01000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000
External cache id: 0x0 0x0 0x0 0x0
Absolute fuzzy scn: 0x0000.00000000
Recovery fuzzy scn: 0x0000.00000000 01/01/1988 00:00:00
Terminal Recovery Stamp  01/01/1988 00:00:00
Platform Information:    Creation Platform ID: 13
Current Platform ID: 13 Last Platform ID: 13

基于上述情况,尝试强制打开库,报ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1]错误
ora-600 kzbzib_kcrsds_1

对于这个情况,以前有过大量恢复案例,修改数据库scn即可
kcbzib_kcrsds_1报错汇总
12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
存储故障,强制拉库报ORA-600 kcbzib_kcrsds_1处理
https://www.xifenfei.com/2023/12/patch-scn-ora-600-kcbzib_kcrsds_1.html
此类故障处理太多,不一一列举,解决这个错误之后,数据库open成功,然后安排逻辑迁移即可

分类目录归档：Oracle备份恢复