Database SOS – Professional Database Recovery Support@+8617813235971

docker回收和mysql备份导入导致数据丢失恢复

最近遇到两例MySQL异常被删除的案例,一例是在docker环境中,由于对docker执行了删除操作,并回收了相关的挂载卷,导致数据彻底丢失

另外一个客户使用备份导入生产库,导致生产库的数据全部被重置为了当时备份的状态,这是由于mysqldump导出数据的时候,默认带有DROP TABLE IF EXISTS `xifenfei`;语句,因此导入备份的时候会先删除掉存在的表,然后创建新表,再insert插入数据.

上述的这两个case,故障发生之后,都没有第一时间保护现场,反而对数据所在分区进行了不少的写入操作,导致覆盖概率相对增加很多.对于这样的故障,一般处理思路:
1. 停掉对该分区写入的业务,如果可以尽可能umount分区,然后做快照或者进项
2. 使用反删除软件对镜像的或者快照的分区进行分析,尝试恢复出来没有被覆盖的MySQL数据,主要是ibd和frm等文件
3. 使用碎片工具对镜像的或者快照的分区进行扫描,根据数据类型生产index和blob的page文件

4. 对于2中恢复的ibd,frm文件,可以尝试通过DISCARD TABLESPACE/IMPORT TABLESPACE方式进行恢复,如果不行对ibd文件进行解析恢复,参考:又一起mysql rm删除数据库目录事故
5. 对于3中恢复出来的page文件,利用工具结合表结构对其进行解析,恢复数据
通过上述恢复,基本上是对于MySQL数据的drop table/truncate table/drop database/rm -rf/格式化等相关误操作的终极恢复思路,对于类似MySQL故障,我们可以实现比较好的恢复效果,如果需要专业恢复技术支持请联系我们:
电话/微信:17813235971 Q Q:107644445 E-Mail:dba@xifenfei.com

服务器断电一起的一例ORA-01207故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：服务器断电一起的一例ORA-01207故障处理

有一个客户反馈服务器异常断电,导致数据库无法正常启动(启动报ORA-01207: file is more recent than control file – old control file 错误)

SQL> startup mount
ORACLE 例程已经启动。

Total System Global Area 5.4784E+10 bytes
Fixed Size                  2188768 bytes
Variable Size            4.2010E+10 bytes
Database Buffers         1.2751E+10 bytes
Redo Buffers               20566016 bytes
数据库装载完毕。
SQL> recover database
ORA-00283: recovery session canceled due to errors
ORA-01122: database file 6 failed verification check
ORA-01110: data file 6: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\HSHIS1'
ORA-01207: file is more recent than control file - old control file

尝试拷贝alert日志到本地进行分析,发现无法正常拷贝

怀疑文件系统或者底层磁盘有损坏,进一步检查系统日志,确认发现大量ntfs层面异常提示（而且多个盘都有）

对于这样的情况,先对数据文件进行了备份,运气还不错,文件全部备份成功,并通过Oracle Database Recovery Check 脚本进行检查当前故障库状态,确认很多文件的checkpoint点相差很远

对于这样的一个非归档数据库,肯定无法正常打开,只能通过屏蔽一致性尝试强制打开库,结果在打开过程中出现ORA-600 4194错误

SQL> alter database open ;
alter database open 
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
进程 ID: 7820
会话 ID: 177 序列号: 147

解决异常undo,数据库正常open成功,但是在数据库导出数据过程中报ORA-600 2662和ORA-600 2663错误(这种情况一般比较少见,大部分ORA-600 2662/2663都是在open过程中,这种库已经open成功,后续查询报该错误的比较少见,但是原理上有可能[启动的时候没有读到这些block,后面放到这些表的block scn大于文件头scn,所以报该错误])

Fri Sep 05 01:04:12 2025
QMNC started with pid=54, OS id=7708 
Completed: alter database open
Fri Sep 05 01:04:13 2025
Errors in file d:\app\administrator\diag\rdbms\his\his\trace\his_m001_8624.trc  (incident=10058):
ORA-00600: internal error code, arguments: [2662], [2], [1384717872], [2], [1385084764], [41943042],
Incident details in: d:\app\administrator\diag\rdbms\his\his\incident\incdir_10058\his_m001_8624_i10058.trc
Fri Sep 05 01:04:13 2025
Starting background process CJQ0
Fri Sep 05 01:04:13 2025
CJQ0 started with pid=62, OS id=6852 
Fri Sep 05 01:04:13 2025
db_recovery_file_dest_size of 3912 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Errors in file d:\app\administrator\diag\rdbms\his\his\trace\his_ora_9220.trc  (incident=10084):
ORA-00600: 内部错误代码, 参数: [2662], [2], [1384717989], [2], [1385446402], [37802453], [], [], [],
Fri Sep 05 01:04:18 2025
Errors in file d:\app\administrator\diag\rdbms\his\his\trace\his_j017_11248.trc  (incident=10059):
ORA-00600: internal error code, arguments: [2662], [2], [1384717990], [2], [1385446402], [37802453],
Errors in file d:\app\administrator\diag\rdbms\his\his\trace\his_j017_11248.trc:
ORA-00600: internal error code, arguments: [2662], [2], [1384717990], [2], [1385446402], [37802453],
ORA-06512: at "HOSPITAL.RPT_CASE", line 5684
ORA-06512: at line 3
Errors in file d:\app\administrator\diag\rdbms\his\his\trace\his_ora_9220.trc  (incident=10085):
ORA-00600: 内部错误代码, 参数: [2662], [2], [1384717990], [2], [1385446402], [37802453], [], [], [],

这个问题相对比较简单,通过Patch SCN小工具修改正确的Oracle SCN即可

RMAN SBT_TAPE备份通过小程序修改实现直接DISK通道还原

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：RMAN SBT_TAPE备份通过小程序修改实现直接DISK通道还原

有朋友把oracle rman备份从cv带库里拷贝到文件系统,然后希望把他还原到数据库中.通过尝试发现直接带库的文件无法直接在文件系统中被rman调用还原

C:\Users\XFF>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on 星期三 9月 3 18:07:58 2025

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

已连接到空闲例程。

SQL> startup nomount pfile='d:/pfile.txt'
ORACLE 例程已经启动。

Total System Global Area 4275781632 bytes
Fixed Size                  2288080 bytes
Variable Size             939525680 bytes
Database Buffers         3321888768 bytes
Redo Buffers               12079104 bytes
SQL> exit
从 Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options 断开

C:\Users\XFF>
C:\Users\XFF>
C:\Users\XFF>
C:\Users\XFF>rman target /

恢复管理器: Release 11.2.0.4.0 - Production on 星期三 9月 3 18:08:38 2025

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

已连接到目标数据库: ORCL (未装载)

RMAN> restore controlfile from 'H:\TEMP\Oracle_Restore\405\c-1737595250-20250822-00';

启动 restore 于 03-9月 -25
使用目标数据库控制文件替代恢复目录
分配的通道: ORA_DISK_1
通道 ORA_DISK_1: SID=28 设备类型=DISK

通道 ORA_DISK_1: 正在还原控制文件
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: restore 命令 (在 09/03/2025 18:08:54 上) 失败
ORA-19870: 还原备份片段 H:\TEMP\ORACLE_RESTORE\405\C-1737595250-20250822-00 时出错
ORA-19505: 无法识别文件"H:\TEMP\ORACLE_RESTORE\405\C-1737595250-20250822-00"
ORA-27046: 文件大小不是逻辑块大小的倍数
OSD-04000: 逻辑块大小不匹配 (OS 512)

以前也写过相关的文章,由于tape的备份和disk通道备份的文件头本身格式都不一样,具体参考:RMAN SBT_TAPE备份无法被DISK通道识别
尝试使用dbms_backup_restore还原依旧失败

C:\Users\XFF>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on 星期三 9月 3 18:19:23 2025

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> DECLARE
  2   devtype varchar2(256);
  3   done boolean;
  4   BEGIN
  5   devtype:=sys.dbms_backup_restore.deviceAllocate (type=>'',ident=>'t1');
  6   sys.dbms_backup_restore.restoreSetDatafile;
  7   sys.dbms_backup_restore.restoreControlfileTo(cfname=>'H:\CONTROL01.CTL');
  8   sys.dbms_backup_restore.restoreBackupPiece(done=>done,
  9     handle=>'H:\TEMP\Oracle_Restore\405\c-1737595250-20250822-00', params=>null);
 10  sys.dbms_backup_restore.deviceDeallocate;
 11  END;
 12  /
DECLARE
*
第 1 行出现错误:
ORA-19624: 操作失败, 如果可能请重试
ORA-19870: 还原备份片段 H:\TEMP\ORACLE_RESTORE\405\C-1737595250-20250822-00
时出错
ORA-19505: 无法识别文件"H:\TEMP\ORACLE_RESTORE\405\C-1737595250-20250822-00"
ORA-27046: 文件大小不是逻辑块大小的倍数
OSD-04000: 逻辑块大小不匹配 (OS 512)
ORA-06512: 在 "SYS.X$DBMS_BACKUP_RESTORE", line 5940
ORA-06512: 在 line 8

对于这种情况,我尝试自己开发一个小程序,来模拟disk通道备份集,欺骗数据库来实现还原操作.

通过程序模拟出来的disk通道的文件,每个后缀加上了disk

再次通过尝试通过disk通道还原操作

C:\Users\XFF>rman target /

恢复管理器: Release 11.2.0.4.0 - Production on 星期三 9月 3 18:23:14 2025

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

已连接到目标数据库: ORCL (未装载)

RMAN> restore controlfile from 'H:\TEMP\Oracle_Restore\405\c-1737595250-20250822-00.disk';

启动 restore 于 03-9月 -25
使用目标数据库控制文件替代恢复目录
分配的通道: ORA_DISK_1
通道 ORA_DISK_1: SID=28 设备类型=DISK

通道 ORA_DISK_1: 正在还原控制文件
通道 ORA_DISK_1: 还原完成, 用时: 00:00:01
输出文件名=H:\CONTROL01.CTL
完成 restore 于 03-9月 -25

RMAN> alter database mount;

数据库已装载
释放的通道: ORA_DISK_1

RMAN> catalog start with 'H:\TEMP\Oracle_Restore\405\*.disk';

搜索与样式 H:\TEMP\Oracle_Restore\405\*.disk 匹配的所有文件

数据库未知文件的列表
=====================================
文件名: H:\TEMP\ORACLE_RESTORE\405\322_ORCL_0141onds_1_1.disk
文件名: H:\TEMP\ORACLE_RESTORE\405\322_ORCL_0241onec_1_1.disk
文件名: H:\TEMP\ORACLE_RESTORE\405\322_ORCL_0441onev_1_1.disk
文件名: H:\TEMP\ORACLE_RESTORE\405\c-1737595250-20250822-00.disk
文件名: H:\TEMP\ORACLE_RESTORE\405\c-1737595250-20250822-01.disk

是否确实要将上述文件列入目录 (输入 YES 或 NO)? yes
正在编制文件目录...
目录编制完毕

已列入目录的文件的列表
=======================
文件名: H:\TEMP\ORACLE_RESTORE\405\322_ORCL_0141onds_1_1.disk
文件名: H:\TEMP\ORACLE_RESTORE\405\322_ORCL_0241onec_1_1.disk
文件名: H:\TEMP\ORACLE_RESTORE\405\322_ORCL_0441onev_1_1.disk
文件名: H:\TEMP\ORACLE_RESTORE\405\c-1737595250-20250822-00.disk
文件名: H:\TEMP\ORACLE_RESTORE\405\c-1737595250-20250822-01.disk

RMAN>

RMAN> report schema;

RMAN-06139: 警告: 控制文件对于 REPORT SCHEMA 不是最新
db_unique_name 为 ORCL 的数据库的数据库方案报表

永久数据文件列表
===========================
文件大小 (MB) 表空间           回退段数据文件名称
---- -------- -------------------- ------- ------------------------
1    0        SYSTEM               ***     /data/oracle/oradata/orcl/system01.dbf
2    0        SYSAUX               ***     /data/oracle/oradata/orcl/sysaux01.dbf
3    0        UNDOTBS1             ***     /data/oracle/oradata/orcl/undotbs01.dbf
4    0        USERS                ***     /data/oracle/oradata/orcl/users01.dbf
5    0        EXAMPLE              ***     /data/oracle/oradata/orcl/example01.dbf

临时文件列表
=======================
文件大小 (MB) 表空间           最大大小 (MB) 临时文件名称
---- -------- -------------------- ----------- --------------------
1    20       TEMP                 32767       /data/oracle/oradata/orcl/temp01.dbf

RMAN> run
2> {
3> set newname for datafile 1 to 'h:\system01.dbf';
4> set newname for datafile 2 to 'h:\sysaux01.dbf';
5> set newname for datafile 3 to 'h:\undotbs01.dbf';
6> set newname for datafile 4 to 'h:\users01.dbf';
7> set newname for datafile 5 to 'h:\example01.dbf';
8> restore database;
9> switch datafile all;
10> }

正在执行命令: SET NEWNAME

正在执行命令: SET NEWNAME

正在执行命令: SET NEWNAME

正在执行命令: SET NEWNAME

正在执行命令: SET NEWNAME

启动 restore 于 03-9月 -25
分配的通道: ORA_DISK_1
通道 ORA_DISK_1: SID=28 设备类型=DISK

通道 ORA_DISK_1: 正在开始还原数据文件备份集
通道 ORA_DISK_1: 正在指定从备份集还原的数据文件
通道 ORA_DISK_1: 将数据文件 00001 还原到 h:\system01.dbf
通道 ORA_DISK_1: 将数据文件 00002 还原到 h:\sysaux01.dbf
通道 ORA_DISK_1: 将数据文件 00003 还原到 h:\undotbs01.dbf
通道 ORA_DISK_1: 将数据文件 00004 还原到 h:\users01.dbf
通道 ORA_DISK_1: 将数据文件 00005 还原到 h:\example01.dbf
通道 ORA_DISK_1: 正在读取备份片段 H:\TEMP\ORACLE_RESTORE\405\322_ORCL_0141ONDS_1_1.DISK
通道 ORA_DISK_1: 段句柄 = H:\TEMP\ORACLE_RESTORE\405\322_ORCL_0141ONDS_1_1.DISK 标记 = TAG20250822T124236
通道 ORA_DISK_1: 已还原备份片段 1
通道 ORA_DISK_1: 还原完成, 用时: 00:00:01
完成 restore 于 03-9月 -25

数据文件 1 已转换成数据文件副本
输入数据文件副本 RECID=7 STAMP=1210875892 文件名=H:\SYSTEM01.DBF
数据文件 2 已转换成数据文件副本
输入数据文件副本 RECID=8 STAMP=1210875892 文件名=H:\SYSAUX01.DBF
数据文件 3 已转换成数据文件副本
输入数据文件副本 RECID=9 STAMP=1210875892 文件名=H:\UNDOTBS01.DBF
数据文件 4 已转换成数据文件副本
输入数据文件副本 RECID=10 STAMP=1210875892 文件名=H:\USERS01.DBF
数据文件 5 已转换成数据文件副本
输入数据文件副本 RECID=11 STAMP=1210875892 文件名=H:\EXAMPLE01.DBF

RMAN>

RMAN> run
2> {
3> set archivelog destination to 'h:/';
4> restore archivelog all;
5> }

正在执行命令: SET ARCHIVELOG DESTINATION

启动 restore 于 03-9月 -25
使用通道 ORA_DISK_1

通道 ORA_DISK_1: 正在开始将归档日志还原到用户指定的目标
归档日志目标=h:/
通道 ORA_DISK_1: 正在还原归档日志
归档日志线程=1 序列=6
通道 ORA_DISK_1: 正在还原归档日志
归档日志线程=1 序列=7
通道 ORA_DISK_1: 正在读取备份片段 H:\TEMP\ORACLE_RESTORE\405\322_ORCL_0441ONEV_1_1.DISK
通道 ORA_DISK_1: 段句柄 = H:\TEMP\ORACLE_RESTORE\405\322_ORCL_0441ONEV_1_1.DISK 标记 = TAG20250822T124311
通道 ORA_DISK_1: 已还原备份片段 1
通道 ORA_DISK_1: 还原完成, 用时: 00:00:01
完成 restore 于 03-9月 -25

通过上述操作,证明写的小程序oracle rman 备份集从tape通道备份方式修改为直接走disk 通道还原完整没有问题.

删除数据库文件并部分覆盖情况下Oracle恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：删除数据库文件并部分覆盖情况下Oracle恢复

有客户由于磁盘空间满了,人工删除了数据库30多个数据文件,导致数据库无法正常工作,然后又被人offline这些文件启动数据库,并运行了一段时间,导致写入了大量的trace和部分数据库归档日志,导致被删除的数据文件发生了覆盖,对于这样情况,通过底层反删除工具对磁盘进行扫描,发现了部分被删除文件,但是大小基本上显示0kb

对于这种情况,os层面反删除恢复,肯定无法恢复出来合适的数据文件,只能做底层数据块扫描恢复,参考以前类似case:
rm -rf误删Oracle数据库恢复
win系统删除oracle数据文件恢复
Oracle 数据文件大小为0kb或者文件丢失恢复
解决一次硬件恢复之后数据文件0kb的故障恢复case
这个客户的情况相对复杂一些:
1. 该磁盘分区中有历史库(也就是说单纯的软件直接按照rdba方式无法直接区分出来合适的数据块,儿实现数据重组)
2. 删除的文件比较多(33个数据文件),分区较大(5T+)
3. 删除文件之后,分区还写入了不少数据,会引起一些覆盖和导致碎片数量增加,导致工作量增加和恢复效果变差
通过对客户alert日志分析,发现一个好消息,客户数据每个数据文件是固定大小(没有设置自增长),这种情况,一般来说数据比较连续,碎片相对比较容易区分出来.

通过工具扫描识别出来oracle block,并把结果记录到数据库中,然后通过人工在数据库中对其进行挑选识别,然后生成dd语句恢复出来数据文件,比如这个只是被覆盖了文件头的22号文件,就比较容易恢复

对于一些碎片严重的文件,就需要人工生成大量dd语句来恢复

对于所有恢复出来的文件,使用工具检查坏块情况

然后把这些数据文件中的数据恢复到新库中,完成本次数据恢复工作,最大限度抢救客户数据.

Oracle dul 最新版(12.2.0.2.10)

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle dul 最新版(12.2.0.2.10)

Oracle官方的数据文件离线unload工具Data UnLoader(DUL),依旧保持强盛生命力,虽然开发者Bernard van Duijnen已经退休,但是产品依旧保持更新,这个唯一一款从Oracle 6开始支持的离线unload数据的产品.

[oracle@www.xifenfei.com dul]$ ./dul

Data UnLoader: 12.2.0.2.10 - Internal Only - on Sun Aug 31 21:04:44 2025
with 64-bit io functions and the decompression option

Copyright (c) 1994 2025 Bernard van Duijnen All rights reserved.

 Strictly Oracle Internal Use Only


DUL: Warning: ulimit process stack size is only 33554432
Found db_id = 1780931490
Found db_name = XIFENFEI
DUL> bootstrap;
Probing file = 1, block = 520
. unloading table                BOOTSTRAP$
DUL: Warning: block number is non zero but marked deferred trying to process it anyhow
      60 rows unloaded
Reading BOOTSTRAP.dat 60 entries loaded
Parsing Bootstrap$ contents
Generating dict.ddl for version 11
 OBJ$: segobjno 18, file 1 block 240
 TAB$: segobjno 2, tabno 1, file 1  block 144
 COL$: segobjno 2, tabno 5, file 1  block 144
 USER$: segobjno 10, tabno 1, file 1  block 208
Running generated file "@dict.ddl" to unload the dictionary tables
. unloading table                      OBJ$   86361 rows unloaded
. unloading table                      TAB$    2898 rows unloaded
. unloading table                      COL$   94610 rows unloaded
. unloading table                     USER$      86 rows unloaded
Reading USER.dat 86 entries loaded
Reading OBJ.dat 86361 entries loaded and sorted 86361 entries
Reading TAB.dat 2898 entries loaded
Reading COL.dat 94610 entries loaded and sorted 94610 entries
Reading BOOTSTRAP.dat 60 entries loaded

DUL: Warning: Recreating file "dict.ddl"
Generating dict.ddl for version 11
 OBJ$: segobjno 18, file 1 block 240
 TAB$: segobjno 2, tabno 1, file 1  block 144
 COL$: segobjno 2, tabno 5, file 1  block 144
 USER$: segobjno 10, tabno 1, file 1  block 208
 TABPART$: segobjno 591, file 1 block 4000
 INDPART$: segobjno 596, file 1 block 4040
 TABCOMPART$: segobjno 613, file 1 block 4176
 INDCOMPART$: segobjno 618, file 1 block 4216
 TABSUBPART$: segobjno 603, file 1 block 4096
 INDSUBPART$: segobjno 608, file 1 block 4136
 IND$: segobjno 2, tabno 3, file 1  block 144
 ICOL$: segobjno 2, tabno 4, file 1  block 144
 LOB$: segobjno 2, tabno 6, file 1  block 144
 COLTYPE$: segobjno 2, tabno 7, file 1  block 144
 TYPE$: segobjno 518, tabno 1, file 1  block 3464
 COLLECTION$: segobjno 518, tabno 2, file 1  block 3464
 ATTRIBUTE$: segobjno 518, tabno 3, file 1  block 3464
 LOBFRAG$: segobjno 624, file 1 block 4264
 LOBCOMPPART$: segobjno 627, file 1 block 4288
 UNDO$: segobjno 15, file 1 block 224
 TS$: segobjno 6, tabno 2, file 1  block 176
 PROPS$: segobjno 98, file 1 block 800
Running generated file "@dict.ddl" to unload the dictionary tables
. unloading table                      OBJ$
DUL: Warning: Recreating file "OBJ.ctl"
   86361 rows unloaded
. unloading table                      TAB$
DUL: Warning: Recreating file "TAB.ctl"
    2898 rows unloaded
. unloading table                      COL$
DUL: Warning: Recreating file "COL.ctl"
   94610 rows unloaded
. unloading table                     USER$
DUL: Warning: Recreating file "USER.ctl"
      86 rows unloaded
. unloading table                  TABPART$     104 rows unloaded
. unloading table                  INDPART$     126 rows unloaded
. unloading table               TABCOMPART$       1 row  unloaded
. unloading table               INDCOMPART$       0 rows unloaded
. unloading table               TABSUBPART$      32 rows unloaded
. unloading table               INDSUBPART$       0 rows unloaded
. unloading table                      IND$    4931 rows unloaded
. unloading table                     ICOL$    7644 rows unloaded
. unloading table                      LOB$    1031 rows unloaded
. unloading table                  COLTYPE$    2565 rows unloaded
. unloading table                     TYPE$    2909 rows unloaded
. unloading table               COLLECTION$    1002 rows unloaded
. unloading table                ATTRIBUTE$   11328 rows unloaded
. unloading table                  LOBFRAG$       1 row  unloaded
. unloading table              LOBCOMPPART$       0 rows unloaded
. unloading table                     UNDO$      21 rows unloaded
. unloading table                       TS$       6 rows unloaded
. unloading table                    PROPS$      36 rows unloaded
Reading USER.dat 86 entries loaded
Reading OBJ.dat 86361 entries loaded and sorted 86361 entries
Reading TAB.dat 2898 entries loaded
Reading COL.dat 94610 entries loaded and sorted 94610 entries
Reading TABPART.dat 104 entries loaded and sorted 104 entries
Reading TABCOMPART.dat 1 entries loaded and sorted 1 entries
Reading TABSUBPART.dat 32 entries loaded and sorted 32 entries
Reading INDPART.dat 126 entries loaded and sorted 126 entries
Reading INDCOMPART.dat 0 entries loaded and sorted 0 entries
Reading INDSUBPART.dat 0 entries loaded and sorted 0 entries
Reading IND.dat 4931 entries loaded
Reading LOB.dat
DUL: Notice: Increased the size of DC_LOBS from 1024 to 8192 entries
 1031 entries loaded
Reading ICOL.dat 7644 entries loaded
Reading COLTYPE.dat 2565 entries loaded
Reading TYPE.dat 2909 entries loaded
Reading ATTRIBUTE.dat 11328 entries loaded
Reading COLLECTION.dat 1002 entries loaded
Reading BOOTSTRAP.dat 60 entries loaded
Reading LOBFRAG.dat 1 entries loaded and sorted 1 entries
Reading LOBCOMPPART.dat 0 entries loaded and sorted 0 entries
Reading UNDO.dat 21 entries loaded
Reading TS.dat 6 entries loaded
Reading PROPS.dat 36 entries loaded
Database character set is ZHS16GBK
Database national character set is AL16UTF16
DUL> unload table sys.obj$;
. unloading table                      OBJ$   86361 rows unloaded
DUL>

存储掉电强制拉库引起ORA-01555和ORA-01189/ORA-01190故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：存储掉电强制拉库引起ORA-01555和ORA-01189/ORA-01190故障处理

机房存储突然掉电导致Oracle数据库访问存储异常,数据库报出大量的ORA-27072: File I/O error,Linux-x86_64 Error: 5: Input/output error,ORA-15081: failed to submit an I/O operation to a disk等错误,实例直接crash

Wed Aug 27 07:11:53 2025
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_m000_17596.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 6297632
Additional information: -1
WARNING: Read Failed. group:1 disk:0 AU:3075 offset:16384 size:16384
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-00202: control file: '+DG/xff/controlfile/current.284.918834897'
ORA-15081: failed to submit an I/O operation to a disk
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of 
  file 284 in group [1.2747812198] from disk DG_0000  allocation unit 3075 reason error; 
  if possible, will try another mirror side
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_m000_17596.trc:
ORA-00202: control file: '+DG/xff/controlfile/current.284.918834897'
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 16384
WARNING: Write Failed. group:1 disk:0 AU:3080 offset:49152 size:16384
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 16384
WARNING: Write Failed. group:1 disk:0 AU:3075 offset:49152 size:16384
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 0 logical 
 extent 0 of file 284 in group 1 on disk 0 allocation unit 3075 
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0
  of file 283 in group 1 on disk 0 allocation unit 3080 
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG/xff/controlfile/current.283.918834897'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG/xff/controlfile/current.284.918834897'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG/xff/controlfile/current.283.918834897'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG/xff/controlfile/current.284.918834897'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
CKPT (ospid: 6165): terminating the instance due to error 221
Wed Aug 27 07:11:53 2025
ORA-1092 : opitsk aborting process

存储恢复之后,尝试open数据库报ORA-00333错误（该错误一般是由于redo写丢失导致)

Wed Aug 27 16:36:32 2025
ALTER DATABASE OPEN
This instance was first to open
Beginning crash recovery of 2 threads
 parallel recovery started with 31 processes
Started redo scan
Incomplete read from log member '+DG/xff/onlinelog/group_2.287.918834905'. Trying next member.
Incomplete read from log member '+DG/xff/onlinelog/group_2.288.918834911'. Trying next member.
Incomplete read from log member '+DG/xff/onlinelog/group_2.287.918834905'. Trying next member.
Abort recovery for domain 0
Aborting crash recovery due to error 333
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_10257.trc:
ORA-00333: redo log read error block 1275904 count 5721
Abort recovery for domain 0
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_10257.trc:
ORA-00333: redo log read error block 1275904 count 5721
ORA-333 signalled during: ALTER DATABASE OPEN...
1
现场人员使用隐含参数,尝试直接拉库操作报ORA-00704 ORA-01555错误,导致拉库失败
1
Wed Aug 27 16:47:11 2025
ALTER DATABASE RECOVER  database until cancel  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 32 slaves
ORA-279 signalled during: ALTER DATABASE RECOVER  database until cancel  ...
Wed Aug 27 16:47:56 2025
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log +DG
Wed Aug 27 16:47:56 2025
Errors with log +DG
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_pr00_24154.trc:
ORA-00308: cannot open archived log '+DG'
ORA-17503: ksfdopn:2 Failed to open file +DG
ORA-15045: ASM file name '+DG' is not in reference form
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER CANCEL 
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_pr00_24154.trc:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '+DG/xff/datafile/system.279.918834827'
Slave exiting with ORA-1547 exception
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_pr00_24154.trc:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '+DG/xff/datafile/system.279.918834827'
ORA-1547 signalled during: ALTER DATABASE RECOVER CANCEL ...
Wed Aug 27 16:48:09 2025
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 25330611827626
Resetting resetlogs activation ID 3307041102 (0xc51d714e)
Deleted Oracle managed file +DG/xff/onlinelog/group_1.285.918834899
Deleted Oracle managed file +DG/xff/onlinelog/group_1.286.918834901
Deleted Oracle managed file +DG/xff/onlinelog/group_2.287.918834905
Deleted Oracle managed file +DG/xff/onlinelog/group_2.288.918834911
Wed Aug 27 16:48:28 2025
Deleted Oracle managed file +DG/xff/onlinelog/group_3.289.918834917
Deleted Oracle managed file +DG/xff/onlinelog/group_3.290.918834923
Deleted Oracle managed file +DG/xff/onlinelog/group_4.293.918835035
Deleted Oracle managed file +DG/xff/onlinelog/group_4.294.918835037
Wed Aug 27 16:48:48 2025
Deleted Oracle managed file +DG/xff/onlinelog/group_5.295.918835041
Deleted Oracle managed file +DG/xff/onlinelog/group_5.296.918835047
Deleted Oracle managed file +DG/xff/onlinelog/group_6.297.918835055
Wed Aug 27 16:48:58 2025
Deleted Oracle managed file +DG/xff/onlinelog/group_6.298.918835061
Wed Aug 27 16:49:10 2025
Setting recovery target incarnation to 3
Wed Aug 27 16:49:10 2025
This instance was first to open
Picked broadcast on commit scheme to generate SCNs
Wed Aug 27 16:49:10 2025
Assigning activation ID 3598492411 (0xd67ca2fb)
Thread 2 opened at log sequence 1
  Current log# 4 seq# 1 mem# 0: +DG/xff/onlinelog/group_4.294.1210265317
  Current log# 4 seq# 1 mem# 1: +DG/xff/onlinelog/group_4.293.1210265323
Successful open of redo thread 2
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Aug 27 16:49:10 2025
SMON: enabling cache recovery
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
ORA-01555 caused by SQL statement below (SQL ID: 4krwuz0ctqxdt, SCN: 0x1709.be1eb3b1):
select ctime, mtime, stime from obj$ where obj# = :1
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_23787.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 20 with name "_SYSSMU20_1295954159$" too small
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_23787.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 20 with name "_SYSSMU20_1295954159$" too small
Error 704 happened during db open, shutting down database
USER (ospid: 23787): terminating the instance due to error 704
Instance terminated by USER, pid = 23787
ORA-1092 signalled during: alter database open resetlogs...
opiodr aborting process unknown ospid (23787) as a result of ORA-1092

现场进行了一系列尝试操作,最后我接手数据库之时报错为:ORA-01190 ORA-01110,无法recover,也无法重建controlfile,陷入了死局

Completed: ALTER DATABASE   MOUNT
Sat Aug 30 10:03:20 2025
ALTER DATABASE OPEN
This instance was first to open
Abort recovery for domain 0
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_6435.trc:
ORA-01190: control file or data file 1 is from before the last RESETLOGS
ORA-01110: data file 1: '+DG/xff/datafile/system0829.dbf'
ORA-1190 signalled during: ALTER DATABASE OPEN...

Sat Aug 30 00:56:32 2025
NOTE: Loaded library: System 
SUCCESS: diskgroup DG was mounted
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_17302.trc:
ORA-01189: file is from a different RESETLOGS than previous files
ORA-01110: data file 2: '+DG/xff/datafile/sysaux.280.918834827'
ORA-1503 signalled during: create controlfile reuse database xff noarchivelog noresetlogs

对于这种情况,通过Oracle recovery check脚本可以直接发现异常(WRONG RESETLOGS)

使用Oracle Recovery Tools小工具实现快速恢复

再尝试重建ctl成功

然后修改数据库scn信息,顺利open数据库

后续建议客户逻辑迁移该库

ORA-600 kcratr_nab_less_than_odr和ORA-600 2662故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 kcratr_nab_less_than_odr和ORA-600 2662故障处理

异常断电之后,oracle启动报ORA-600 kcratr_nab_less_than_odr错误

Sun Aug 17 11:06:09 2025
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 11 processes
Started redo scan
Completed redo scan
 read 0 KB redo, 0 data blocks need recovery
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [40785], [33267], [40630]
Sun Aug 17 11:06:20 2025
Aborting crash recovery due to error 600
Sun Aug 17 11:06:20 2025
Trace dumping is performing id=[cdmp_20250817110620]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_2920.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [40785], [33267], [40630]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_2920.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_nab_less_than_odr], [1], [40785], [33267], [40630]
ORA-600 signalled during: ALTER DATABASE OPEN...

这个错误处理有多次处理经验
12c启动报kcratr_nab_less_than_odr
又一例ORA-600 kcratr_nab_less_than_odr
ORA-600 kcratr_nab_less_than_odr故障解决
 差点被误操作的ORA-600 kcratr_nab_less_than_odr故障
 11.2.0.4库中遇到ORA-600 kcratr_nab_less_than_odr报错
 ORA-600 kcratr_nab_less_than_odr和ORA-600 4194故障处理
一般重建ctl或者using backup ctl方式恢复即可实现0丢失打开库,但是这个库尝试打开报ORA-600 2662错误

SQL> startup nomount;
ORACLE 例程已经启动。

Total System Global Area 6847938560 bytes
Fixed Size                  2188768 bytes
Variable Size            4680845856 bytes
Database Buffers         2147483648 bytes
Redo Buffers               17420288 bytes
SQL> @rectl.sql

控制文件已创建。

SQL>
SQL>
SQL> recover database;
完成介质恢复。
SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [4413], [1200914792],
[4413], [1201104184], [12583040], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [2662], [4413], [1200914791],
[4413], [1201104184], [12583040], [], [], [], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [4413], [1200914789],
[4413], [1201104184], [12583040], [], [], [], [], [], []
进程 ID: 3132
会话 ID: 3018 序列号: 1

对于这样的ORA-600 2662错误是比较常见的问题,直接通过Patch_SCN工具修改scn即可正常打开库

Mon Aug 18 03:30:41 2025
ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 12 slaves
Mon Aug 18 03:30:41 2025
Recovery of Online Redo Log: Thread 1 Group 2 Seq 40787 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Completed: ALTER DATABASE RECOVER  database  
alter database open 
Beginning crash recovery of 1 threads
 parallel recovery started with 11 processes
Started redo scan
Completed redo scan
 read 1 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 40787, block 2, scn 18954891612040
Recovery of Online Redo Log: Thread 1 Group 2 Seq 40787 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 40787, block 4, scn 18954891632047
 0 data blocks read, 0 data blocks written, 1 redo k-bytes read
Mon Aug 18 03:30:47 2025
Thread 1 advanced to log sequence 40788 (thread open)
Thread 1 opened at log sequence 40788
  Current log# 3 seq# 40788 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Aug 18 03:30:47 2025
SMON: enabling cache recovery
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Completed: alter database open

后续增加tempfile,导出数据完成本次恢复任务

joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin)

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin)

expdp导出数据报ORA-39127 ORA-29548错误

ORA-39127: 调用 BEGIN :1 := SYS.DBMS_JAVA.START_EXPORT(:2, :3, :4, :5, :6, :7, :8, :9 ,:10); END; 时发生意外错误
ORA-29548: Java 系统类报告: release of Java system classes in the database (19.0.0.0.0 1.8) does not match that of the o
racle executable (19.3.0.0.0 1.8)
ORA-06512: 在 "SYS.DBMS_JAVA", line 520
ORA-06512: 在 line 1
ORA-06512: 在 "SYS.DBMS_JAVA", line 520
ORA-06512: 在 line 1
ORA-06512: 在 "SYS.DBMS_METADATA", line 11827
ORA-06512: 在 "SYS.DBMS_SYS_ERROR", line 95

对应的alert日志报错

2025-08-14T11:37:10.886745+08:00
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0 
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0 
2025-08-14T11:37:11.379211+08:00
Resize operation completed for file# 68, old size 83776K, new size 84800K
Resize operation completed for file# 68, old size 84800K, new size 85824K
2025-08-14T11:37:12.333138+08:00
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0 
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0 
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0 
joxcsys: release mismatch, 19.0.0.0.0 1.8 in database (classes.bin) vs 19.3.0.0.0 1.8 in executable pid 2624 cid 0

数据库版本（19.3版本,没有打任何其他patch）

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on 星期四 8月 14 11:17:08 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


连接到:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

出现这个错误一般是由于Oracle JVM组件和Oracle JVM库不同步导致,对于这个初始版本（没有打任何patch）,理论上不应该发生该问题,既然已经发生了,那就解决问题

SQL> startup upgrade;
ORACLE 例程已经启动。

Total System Global Area 1.5435E+10 bytes
Fixed Size                 16124312 bytes
Variable Size            4563402752 bytes
Database Buffers         1.0838E+10 bytes
Redo Buffers               17428480 bytes
数据库装载完毕。
数据库已经打开。
SQL> alter system set java_jit_enabled = FALSE;

系统已更改。

SQL> alter system set "_system_trig_enabled"=FALSE;

系统已更改。

SQL> alter system set JOB_QUEUE_PROCESSES=0;

系统已更改。

SQL> create or replace java system
  2          /

Java 已创建。

SQL> alter system set java_jit_enabled = true;

系统已更改。

SQL> alter system set "_system_trig_enabled"=TRUE;

系统已更改。

SQL> alter system set JOB_QUEUE_PROCESSES=100;

系统已更改。

SQL>  @?/rdbms/admin/utlrp.sql

会话已更改。

TIMESTAMP
--------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_BGN              2025-08-14 12:42:59

……………………

PL/SQL 过程已成功完成。

SQL> SELECT dbms_java.longname('TEST') long_name FROM dual;

LONG_NAME
--------------------------------------------------------------------------------
TEST

SQL> shutdown immediate;
数据库已经关闭。
已经卸载数据库。
ORACLE 例程已经关闭。
SQL> startup
ORACLE 例程已经启动。

Total System Global Area 1.5435E+10 bytes
Fixed Size                 16124312 bytes
Variable Size            4563402752 bytes
Database Buffers         1.0838E+10 bytes
Redo Buffers               17428480 bytes
数据库装载完毕。
数据库已经打开。

如果有其他情况,具体参考:ORA-29548 Error While Running Java Code or Datapatch In Database (Doc ID 2934546.1)文档

win环境断电强制拉库报ORA-600 kcbzib_kcrsds_1故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：win环境断电强制拉库报ORA-600 kcbzib_kcrsds_1故障处理

客户环境异常断电,导致数据库无法正常启动,执行recover提示缺少归档（ORA-00279 ORA-00289 ORA-00280错误)

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on 星期四 8月 14 11:17:08 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


连接到:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0


SQL> select open_mode from v$database;

OPEN_MODE
----------------------------------------
MOUNTED

SQL> select group#，sequence#,status from v$log;

    GROUP#  SEQUENCE# STATUS
---------- ---------- --------------------------------
         1      15082 ACTIVE
         3      15084 CURRENT
         2      15083 ACTIVE

SQL> recover database;
ORA-00279: 更改 885427209 (在 08/11/2025 22:14:56 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000015080_1051367880.0001
ORA-00280: 更改 885427209 (用于线程 1) 在序列 #15080 中


指定日志: {<RET>=suggested | filename | AUTO | CANCEL}

ORA-00308: 无法打开归档日志
'D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000015080_1051367880.0001'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件。

z这个库需要15080归档,但是由于非归档环境,redo最小为15082,所以这种情况,只能先恢复可以正常recover的数据文件，然后强制拉库

SQL> recover datafile 1;
完成介质恢复。
SQL> recover datafile 3,4,7;
完成介质恢复。
SQL> recover datafile 62;
完成介质恢复。

SQL> recover database until cancel;
ORA-00279: 更改 885427209 (在 08/11/2025 22:14:56 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000015080_1051367880.0001
ORA-00280: 更改 885427209 (用于线程 1) 在序列 #15080 中


指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: 警告: RECOVER 成功但 OPEN RESETLOGS 将出现如下错误
ORA-01152: 文件 1 没有从过旧的备份中还原
ORA-01110: 数据文件 1: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSTEM01.DBF'


ORA-01112: 未启动介质恢复


SQL> alter database open resetlogs ;
alter database open resetlogs
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [],
[], [], [], [], [], [], []
进程 ID: 6272
会话 ID: 3873 序列号: 46847

alert日志报错

2025-08-14T11:24:08.246798+08:00
alter database open resetlogs
2025-08-14T11:24:08.425971+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 28: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\PRESCRIPTION_P.DBF'
2025-08-14T11:24:08.643178+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 29: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\PRESCRIPTION_DETAIL_P.DBF'
2025-08-14T11:24:08.958486+08:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 885427209 time 
Resetting resetlogs activation ID 3572089731 (0xd4e9c383)
2025-08-14T11:24:08.988512+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 30: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\ORDER_LIST_P.DBF'
2025-08-14T11:24:12.029432+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 31: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\CHARGE_STATIS_P.DBF'
2025-08-14T11:24:13.687023+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 32: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\ECASETAB01.DBF'
2025-08-14T11:24:13.725060+08:00
Setting recovery target incarnation to 2
Checker run found 31 new persistent data failures
2025-08-14T11:24:14.701994+08:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
Endian type of dictionary set to little
2025-08-14T11:24:14.793084+08:00
Assigning activation ID 3729861866 (0xde512cea)
2025-08-14T11:24:14.848137+08:00
TT00 (PID:8068): Gap Manager starting
2025-08-14T11:24:14.870158+08:00
Redo log for group 1, sequence 1 is not located on DAX storage
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG
Successful open of redo thread 1
2025-08-14T11:24:14.912199+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2025-08-14T11:24:15.849098+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc(incident=717057):
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Please look for redo dump in pinned buffers history in incident trace file, 
if not dumped for what so ever reason, use the following command to dump it at the earliest.
ALTER SYSTEM DUMP REDO DBA MIN 4 128 DBA MAX 4 128 SCN MIN 1;
2025-08-14T11:24:17.402582+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Undo initialization recovery: err:600 start: 2392187 end: 2394046 diff: 1859 ms (1.9 seconds)
2025-08-14T11:24:17.528703+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2025-08-14T11:24:17.529703+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc(incident=717058):
ORA-00603: ORACLE 服务器会话因致命错误而终止
ORA-01092: ORACLE 实例终止。强制断开连接
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2025-08-14T11:24:18.995110+08:00
opiodr aborting process unknown ospid (6272) as a result of ORA-603
2025-08-14T11:24:18.997112+08:00
ORA-603 : opitsk aborting process
License high water mark = 42
USER (ospid: (prelim)): terminating the instance due to ORA error 
2025-08-14T11:24:23.749679+08:00
Instance terminated by USER(prelim), pid = 6272

遭遇到比较明显的ORA-600 kcbzib_kcrsds_1错误,通过自研的Patch_SCN小工具,以前有过类似恢复案例：
Patch SCN工具一键恢复ORA-600 kcbzib_kcrsds_1

然后直接open数据库

SQL> recover database;
完成介质恢复。
SQL> oradebug setmypid
已处理的语句
SQL> oradebug DUMPvar SGA kcsgscn_
kscn8 kcsgscn_ [7FF6623FF310, 7FF6623FF318) = 40B272E2 00000000
SQL>
SQL> alter database open;

数据库已更改。

然后安排导出数据,完成本次恢复任务.

log_archive_dest_1配置不当,当时standby redo无法归档引起adg不能实时同步

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：log_archive_dest_1配置不当,当时standby redo无法归档引起adg不能实时同步

oracle database dataguard无法利用standby redo实时同步

2025-08-12T18:41:57.474031+08:00
PR00 (PID:245113): Media Recovery Log /u01/app/oracle/fast_recovery_area/……/2025_08_12/o1_mf_1_34206_n9p6mmkb_.arc
PR00 (PID:245113): Media Recovery Waiting for T-1.S-34207 (in transit)

clear standby redo之后，实时同步一会儿，又无法实时同步

2025-08-12T18:43:28.183908+08:00
PR00 (PID:245113): MRP0: Background Media Recovery cancelled with status 16037
2025-08-12T18:43:28.184131+08:00
Errors in file /u01/app/oracle/diag/rdbms/dclhrdg2/DCLHR/trace/DCLHR_pr00_245113.trc:
ORA-16037: user requested cancel of managed recovery operation
PR00 (PID:245113): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Stopping change tracking
2025-08-12T18:43:28.356862+08:00
Errors in file /u01/app/oracle/diag/rdbms/dclhrdg2/DCLHR/trace/DCLHR_pr00_245113.trc:
ORA-16037: user requested cancel of managed recovery operation
2025-08-12T18:43:28.482234+08:00
Background Media Recovery process shutdown (DCLHR)
2025-08-12T18:43:29.178708+08:00
Managed Standby Recovery Canceled (DCLHR)
2025-08-12T18:43:44.724806+08:00
Clearing online log 11 of thread 1 sequence number 34111
2025-08-12T18:44:00.545234+08:00
Clearing online log 12 of thread 1 sequence number 34112
2025-08-12T18:44:10.770626+08:00
Clearing online log 13 of thread 1 sequence number 34113
2025-08-12T18:44:31.911565+08:00
Clearing online log 14 of thread 1 sequence number 34114
2025-08-12T18:44:42.111444+08:00
Clearing online log 15 of thread 1 sequence number 34108
2025-08-12T18:45:06.031673+08:00
Warning: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE has been deprecated.
Warning: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE has been deprecated.
2025-08-12T18:45:06.047722+08:00
Attempt to start background Managed Standby Recovery process (DCLHR)
Starting background process MRP0
2025-08-12T18:45:06.062512+08:00
MRP0 started with pid=219, OS id=245503
2025-08-12T18:45:06.064300+08:00
Background Managed Standby Recovery process started (DCLHR)
2025-08-12T18:45:11.086271+08:00
 Started logmerger process
2025-08-12T18:45:11.100434+08:00

IM on ADG: Start of Empty Journal

IM on ADG: End of Empty Journal
PR00 (PID:245509): Managed Standby Recovery starting Real Time Apply
max_pdb is 3
2025-08-12T18:45:13.275444+08:00
Parallel Media Recovery started with 80 slaves
2025-08-12T18:45:13.392342+08:00
Stopping change tracking
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34207 (in transit)
2025-08-12T18:46:22.717320+08:00
 rfs (PID:245728): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is Foreground (PID:15566)
2025-08-12T18:46:31.950359+08:00
 rfs (PID:244324): Archived Log entry 99 added for B-1149239150.T-1.S-34207 ID 0x57544f44 LAD:32
2025-08-12T18:46:32.185391+08:00
 rfs (PID:245735): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is ASYNC (PID:3334853)
 rfs (PID:245735): Primary database is in MAXIMUM PERFORMANCE mode
2025-08-12T18:46:32.331928+08:00
 rfs (PID:245735): Selected LNO:11 for T-1.S-34208 dbid 1299764411 branch 1149239150
2025-08-12T18:46:32.885400+08:00
PR00 (PID:245509): Media Recovery Log /u01/app/oracle/fast_recovery_area/……/2025_08_12/o1_mf_1_34207_n9p6mo8f_.arc
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34208 (in transit)
2025-08-12T18:46:33.669769+08:00
Recovery of Online Redo Log: Thread 1 Group 11 Seq 34208 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/DCLHR/std_redo11.log
2025-08-12T18:50:03.011923+08:00
Expanded controlfile section 28 from 141 to 282 records
Requested to grow by 141 records; added 2 blocks of records
2025-08-12T19:16:29.515866+08:00
 rfs (PID:245735): Selected LNO:12 for T-1.S-34209 dbid 1299764411 branch 1149239150
2025-08-12T19:16:29.515943+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34209 (in transit)
2025-08-12T19:16:30.522048+08:00
Recovery of Online Redo Log: Thread 1 Group 12 Seq 34209 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/DCLHR/std_redo12.log
2025-08-12T19:46:29.852369+08:00
 rfs (PID:245735): Selected LNO:13 for T-1.S-34210 dbid 1299764411 branch 1149239150
2025-08-12T19:46:29.852443+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34210 (in transit)
2025-08-12T19:46:30.858423+08:00
Recovery of Online Redo Log: Thread 1 Group 13 Seq 34210 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/DCLHR/std_redo13.log
2025-08-12T20:16:29.978563+08:00
 rfs (PID:245735): Selected LNO:14 for T-1.S-34211 dbid 1299764411 branch 1149239150
2025-08-12T20:16:30.012723+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34211 (in transit)
2025-08-12T20:16:30.013045+08:00
Recovery of Online Redo Log: Thread 1 Group 14 Seq 34211 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/DCLHR/std_redo14.log
2025-08-12T20:46:30.307116+08:00
 rfs (PID:245735): Selected LNO:15 for T-1.S-34212 dbid 1299764411 branch 1149239150
2025-08-12T20:46:30.339601+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34212 (in transit)
2025-08-12T20:46:30.339893+08:00
Recovery of Online Redo Log: Thread 1 Group 15 Seq 34212 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/DCLHR/std_redo15.log
2025-08-12T21:16:27.517565+08:00
 rfs (PID:245735): No SRLs available for T-1
2025-08-12T21:16:27.560282+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34213 (in transit)
2025-08-12T21:16:27.615513+08:00
 rfs (PID:245735): Opened log for T-1.S-34213 dbid 1299764411 branch 1149239150
2025-08-12T21:46:27.633777+08:00
 rfs (PID:245735): Archived Log entry 100 added for B-1149239150.T-1.S-34213 ID 0x57544f44 LAD:32
 rfs (PID:245735): No SRLs available for T-1
2025-08-12T21:46:27.787746+08:00
 rfs (PID:245735): Opened log for T-1.S-34214 dbid 1299764411 branch 1149239150
2025-08-12T21:46:28.439910+08:00
PR00 (PID:245509): Media Recovery Log /u01/app/oracle/fast_recovery_area/……/2025_08_12/o1_mf_1_34213_n9phockd_.arc
2025-08-12T21:46:32.804540+08:00
PR00 (PID:245509): Media Recovery Waiting for T-1.S-34214 (in transit)
2025-08-12T22:16:27.968668+08:00
 rfs (PID:245735): Archived Log entry 101 added for B-1149239150.T-1.S-34214 ID 0x57544f44 LAD:32
 rfs (PID:245735): No SRLs available for T-1

从报错信息看是由于无可用的standby redo导致（No SRLs available for T-1）,查询standby redo状态

[oracle@db03 trace]$ ss

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Aug 13 15:58:46 2025
Version 19.20.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

s
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.20.0.0.0

SQL> elect group#,status from v$standby_log;

    GROUP# STATUS
---------- ----------
        11 ACTIVE
        12 ACTIVE
        13 ACTIVE
        14 ACTIVE
        15 ACTIVE

所有的standby redo都是active状态,导致无可用standby redo,出现该问题的原因,初步怀疑是standby redo无法归档导致,引起该问题的原因怀疑是由于该备库的log_archive_dest_N参数设置不当导致,检查相关参数

*.db_unique_name='DCLHRDG2'
*.fal_client='DCLHRDG2'
*.fal_server='DCLHRDG','DCLHR'
*.log_archive_config='dg_config=(DCLHR,DCLHRDG,DCLHRDG2)'
*.log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST valid_for=(all_logfiles,all_roles) db_unique_name=DCLHRDG'
*.log_archive_dest_2='service=DCLHR lgwr async valid_for=(online_logfiles,primary_role) db_unique_name=DCLHR'
*.log_archive_dest_3='service=DCLHRDG2 lgwr async valid_for=(online_logfiles,primary_role) db_unique_name=DCLHRDG2'

比较明显由于log_archive_dest_1和log_archive_dest_3的相关参数配置不对导致standby redo无法通过log_archive_dest_1进行归档导致,正确的配置方式为

*.log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST valid_for=(all_logfiles,all_roles) db_unique_name=DCLHRDG'
*.log_archive_dest_3='service=DCLHRDG lgwr async valid_for=(online_logfiles,primary_role) db_unique_name=DCLHRDG'

这样配置之后,该dg库同步日志正常.