Database SOS – Professional Database Recovery Support@+8617813235971

ORA-600 krsm_mrp_rebuild_lf_list.lno_mismatch

标题：ORA-600 krsm_mrp_rebuild_lf_list.lno_mismatch

客户由于断电,导致一个19c数据库无法正常启动,接手时候报错为：ORA-600 krsm_mrp_rebuild_lf_list.lno_mismatch

2026-09-15T06:36:24.908317-07:00
ALTER DATABASE OPEN RESETLOGS
2026-09-15T06:36:24.912251-07:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 22297188690 time 
NET  (PID:36760): Unable to create archive log file '/u19/dbs/arch1_1_1243990641.dbf'
2026-09-15T06:36:24.919696-07:00
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_36760.trc:
ORA-19504: failed to create file "/u19/dbs/arch1_1_1243990641.dbf"
ORA-27038: created file already exists
Additional information: 1
NET  (PID:36760): Error 19504 Creating archive log file to '/u19/app/dbs/arch1_1_1243990641.dbf'
NET  (PID:36760): Archiving not possible: error count exceeded
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_36760.trc  (incident=119876):
ORA-00600: internal error code, arguments: [krsm_mrp_rebuild_lf_list.lno_mismatch], [3], [1], [1], [0], []
Incident details in: /u19/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_119876/orcl_ora_36760_i119876.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
NET  (PID:36760): Archive all ORLs failed, error=600
ORA-600 signalled during: ALTER DATABASE OPEN RESETLOGS...

通过alert日志大概回溯现场人员的恢复过程,最初数据库断电重启之后报ORA-00742错误

2026-09-14T20:52:37.294285-07:00
ALTER DATABASE OPEN
Ping without log force is disabled:
  instance mounted in exclusive mode.
2026-09-14T20:52:37.313266-07:00
Beginning crash recovery of 1 threads
 Thread 1: Recovery starting at checkpoint rba (logseq 63111 block 286483), scn 0
2026-09-14T20:52:37.370330-07:00
Started redo scan
2026-09-14T20:52:37.419944-07:00
Aborting crash recovery due to error 742
2026-09-14T20:52:37.420039-07:00
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_68055.trc:
ORA-00742: Log read detects lost write in thread 1 sequence 63111 block 326157
ORA-00312: online log 3 thread 1: '/u19/app/oracle/oradata/ORCL/redo03.log'
2026-09-14T20:52:37.420977-07:00
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_68055.trc:
ORA-00742: Log read detects lost write in thread 1 sequence 63111 block 326157
ORA-00312: online log 3 thread 1: '/u19/app/oracle/oradata/ORCL/redo03.log'
ORA-742 signalled during: ALTER DATABASE OPEN...

现场人员加上隐含参数强制拉库,然后数据库报ORA-600 kcbzib_kcrsds_1错误

2026-09-14T21:35:21.425310-07:00
ALTER DATABASE OPEN RESETLOGS
2026-09-14T21:35:21.429136-07:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 22297188666 time 
Resetting resetlogs activation ID 1680209845 (0x6425f7b5)
2026-09-14T21:35:34.017515-07:00
Setting recovery target incarnation to 3
2026-09-14T21:35:34.028360-07:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
Endian type of dictionary set to little
2026-09-14T21:35:34.043771-07:00
Assigning activation ID 1771732361 (0x699a7d89)
2026-09-14T21:35:34.125473-07:00
TT00 (PID:47466): Gap Manager starting
2026-09-14T21:35:34.127554-07:00
Redo log for group 1, sequence 1 is not located on DAX storage
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /u19/app/oracle/oradata/ORCL/redo01.log
Successful open of redo thread 1
2026-09-14T21:35:34.165332-07:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2026-09-14T21:35:35.507453-07:00
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1],  , [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
SYSTEM DUMP REDO DBA MIN 4 128 DBA MAX 4 128 SCN MIN 1;
2026-09-14T21:35:37.154962-07:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Undo initialization recovery: err:600 start: 3433752 end: 3435865 diff: 2113 ms (2.1 seconds)
2026-09-14T21:35:37.475752-07:00
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_46217.trc:
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1],  , [], [], [], [], []
2026-09-14T21:35:37.475831-07:00
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_46217.trc:
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1],  , [], [], [], [], []
Error 600 happened during db open, shutting down database
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_46217.trc  (incident=48128):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1],  , [], [], [], [], []

现场人员重建控制文件,然后启动库报ORA-600 krsm_mrp_rebuild_lf_list.lno_mismatch

2026-09-15T04:26:41.297637-07:00
Successful mount of redo thread 1, with mount id 1771762176
Completed: CREATE CONTROLFILE REUSE SET DATABASE "ORCL" RESETLOGS ARCHIVELOG
    MAXLOGFILES 16
    MAXLOGMEMBERS 3
    MAXDATAFILES 100
    MAXINSTANCES 1
    MAXLOGHISTORY 16
LOGFILE
  GROUP 1 '/u19/app/oracle/oradata/ORCL_NEW/redo01.log' SIZE 200M,
  GROUP 2 '/u19/app/oracle/oradata/ORCL_NEW/redo02.log' SIZE 200M
DATAFILE
  '/u19/app/oracle/oradata/ORCL_NEW/system01.dbf',
  '/u19/app/oracle/oradata/ORCL_NEW/sysaux01.dbf',
  '/u19/app/oracle/oradata/ORCL_NEW/undotbs01.dbf',
…………
2026-09-15T04:26:52.121793-07:00
ALTER DATABASE OPEN RESETLOGS
2026-09-15T04:26:52.126848-07:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 22297188686 time 
NET  (PID:27130): Unable to create archive log file '/u19/dbs/arch1_1_1243990641.dbf'
2026-09-15T04:26:52.233538-07:00
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_27130.trc:
ORA-19504: failed to create file "/u19/dbs/arch1_1_1243990641.dbf"
ORA-27038: created file already exists
Additional information: 1
NET  (PID:27130): Error 19504 Creating archive log file to '/u19/dbs/arch1_1_1243990641.dbf'
NET  (PID:27130): Archiving not possible: error count exceeded
Errors in file /u19/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_27130.trc  (incident=102102):
ORA-00600: internal error code, arguments: [krsm_mrp_rebuild_lf_list.lno_mismatch], [2], [1], [1], [0], []
Incident details in: /u19/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_102102/orcl_ora_27130_i102102.trc

接手故障开始处理,尝试open库,重现ORA-600 krsm_mrp_rebuild_lf_list.lno_mismatch错误

SQL> recover database using backup controlfile;
ORA-00279: change 22297188689 generated at 09/15/2026 00:57:22 needed for thread 1
ORA-00289: suggestion : /u19/dbs/arch/1_1_1243990641.dbf
ORA-00280: change 22297188689 for thread 1 is in sequence #1


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/u19/app/oracle/oradata/ORCL/redo01.log
Log applied.
Media recovery complete.
SQL>        
SQL> 
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [krsm_mrp_rebuild_lf_list.lno_mismatch], [3], [1], [1], [0]

分析alert日志,是由于归档无法被正常创建导致该问题,对/u19/dbs/arch/进行创建并授权之后,再次尝试打开数据库,数据库报ORA-600 kcbzib_kcrsds_1错误

SQL> startup nomount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area       6442448984 bytes
Fixed Size			  8910936 bytes
Variable Size		       3858759680 bytes
Database Buffers	       2566914048 bytes
Redo Buffers			  7864320 bytes
SQL> alter database mount;

Database altered.

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], []
Process ID: 42259
Session ID: 1 Serial number: 19675

使用obet的patch scn功能修改scn之后（obet patch_scn使用说明）,顺利打开库

记录一次0丢失的ORA-00354: 损坏重做日志块标头故障恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：记录一次0丢失的ORA-00354: 损坏重做日志块标头故障恢复

由于服务器断电导致数据库启动报错

Mon Sep 07 12:47:52 2026
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 32 processes
Started redo scan
ORA-00354: 损坏重做日志块标头
ORA-00353: 日志损坏接近块 41735 更改 2385313823 时间 09/07/2026 10:35:04
ORA-00312: 联机日志 5 线程 1: 'D:\APP\LIMSADMIN\ORADATA\ORCL\REDO05.LOG'
Mon Sep 07 12:47:58 2026
Trace dumping is performing id=[cdmp_20260907124758]
Errors in file d:\app\limsadmin\diag\rdbms\primary\orcl\trace\orcl_ora_3040.trc:
ORA-00354: 损坏重做日志块标头
ORA-00353: 日志损坏接近块 41735 更改 2385313823 时间 09/07/2026 10:35:04
ORA-00312: 联机日志 5 线程 1: 'D:\APP\LIMSADMIN\ORADATA\ORCL\REDO05.LOG'
ORA-354 signalled during: ALTER DATABASE OPEN...

从这里看,数据库在启动的时候实例恢复需要读取REDO05.LOG日志,但是发现该日志中有block损坏,而且损坏日志的时间点为09/07/2026 10:35:04,scn为2385313823（ORA-00353: 日志损坏接近块 41735 更改 2385313823 时间 09/07/2026 10:35:04）,导致实例恢复无法完成,从而数据库无法正常启动.通过Oracle Recovery Check脚本检查数据库当前状态,数据库的checkpoint time为:09/07/2026 10:37:37,checkpoint scn为:2385314357

这个就比较明显,数据库文件正在需要的日志是sequence 19001 REDO01.LOG(而不是报错的sequence 19000 REDO05.LOG),基于这样的情况,出现这种问题,是由于control中记录的Thread Checkpoint RBA不正确导致.解决这个问题相对比较简单,重建ctl即可

由于redo05.log有损坏,该库为归档模式,需要先clear redo05.log（不然会导致redo无法归档,数据库hang住）,然后对全库发起备份

不当数据库恢复操作导致一个月数据丢失

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：不当数据库恢复操作导致一个月数据丢失

今天分享一个本不该发生的故障,结果由于现场人员没有合适的判断导致客户丢失近一个月数据,事故经过大概是这样的
1. 客户数据库由于断电,启动之后报ORA-600 kcratr_scan_lastbwr错误

2026-09-03T11:22:58.998379+08:00
Beginning crash recovery of 2 threads
 parallel recovery started with 31 processes
 Thread 1: Recovery starting at checkpoint rba (logseq 37738 block 7724), scn 0
 Thread 2: Recovery starting at checkpoint rba (logseq 43821 block 37937), scn 0
2026-09-03T11:22:59.158232+08:00
Started redo scan
Hex dump of (file 20, block 2720) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_34049.trc

Reading datafile '+DATA/ORCL/DATAFILE/undotbs01.dbf' for corrupt data at rdba: 0x05000aa0 (file 20, block 2720)
Reread Not Encrypted (file 20, block 2720) found same data 
physically corrupt:No, logically corrupt:No, and is not expected version by upper layer. 
Write verification failed for File 20 Block 2720 (rdba 0x5000aa0)
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_34049.trc  (incident=430645):
ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_430645/orcl1_ora_34049_i430645.trc
2026-09-03T11:23:00.484486+08:00
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2026-09-03T11:23:00.510512+08:00
Slave encountered ORA-10388 exception during crash recovery

这个是一个比较常见的错误一般是由于redo写丢失导致(数据库open报ORA-600 kcratr_scan_lastbwr故障处理)

2. 现场操作的技术人员看了下这个错误,觉得数据库有备份，直接决定使用备份进行恢复

结果发现读取的是20260719的备份,而且原始的数据文件在asm里面，已经被覆盖。这个时候如果归档或者增量备份完整,还能够最大限度抢救数据,检查备份发现只有8月7日的增量备份,没有任何归档的备份(使用crontab 直接清理3天之前的归档)

基于这样的情况,应用到最后一个增量备份(也就是8月7日),后面的数据由于归档已经被清理(而且没有备份)

出现这个问题的原因是由于备份脚本本身备份归档,导致备份无法按照策略清理历史备份,导致磁盘空间满,从而后续备份无法继续(直接清理归档了)

恢复只能到此为止,查询最终的文件头恢复时间点

3. 基于这种情况只能尝试强制打开数据库,结果报ORA-600 kcbzibmlt_kcrsds_1错误

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzibmlt_kcrsds_1], [], [], []
Process ID: 30365
Session ID: 2180 Serial number: 43414

4. 这种错误类似ORA-600 kcbzib_kcrsds_1的处理思路,使用obet的patch_scn功能直接进行处理

OBET> patch_scn 89103 0x60017e98 3182707622

=== Oracle SCN Patch ===
PID:     89103
Address: 0x60017e98
New Value: 0xbdb443a6 (3182707622)
Original SCN: 0x0

Are you sure you want to modify Oracle SCN? (yes/no): y

SCN modified: 0x0 -> 0xbdb443a6
Oracle SCN successfully modified.

然后打开数据库

SQL> alter database open;

Database altered.

导出数据,导入新库完成本次恢复任务,可惜没有办法帮客户找回来丢失的那一个月数据.

案例总结:
1. rman备份脚本尽量不要有直接删除归档的操作,尽可能是先备份然后再删除(确认备份成功之后删除)
2. 备份还原操作之前,需要看备份文件和日志,大概评估下备份的可用性
3. 尽可能不要在损坏的现场环境上直接做rman还原操作(万一备份无法正常还原,比如中途有备份文件异常呢？)
4. 这个故障,如果不做还原操作,直接异常恢复,基本上也就丢失了当前redo中的数据,损失是最小的

obet快速修复oracle 位图损坏块

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：obet快速修复oracle 位图损坏块

近期有客户由于虚拟化平台系统负载过高,重启之后导致数据库出现一些坏块

其中被框起来的坏块比较好处理,属于数据库中间某个对象的block,对异常对象进行处理即可,但是有一个block 2，block 10 这种都属于数据文件中位图块(用来记录哪些block被使用,哪些block是没有使用),这种block如果损坏会导致该文件无法分配新的空间(也不能回收空间,无法统计表空间使用情况等)

alert日志有大量报错

对于这种情况,直接使用obet的坏块修复功能

OBET> set file 116
filename set to: /oradata/NNC_DATA207.dbf (file# 116)
endian auto-detected: little

OBET> set block 10
block set to: 10

OBET> backup
Backing up file #116, block 10
Successfully backed up block 10 from /oradata/NNC_DATA207.dbf to backup_blk/NNC_DATA207.dbf.10_20260902200547.blk

OBET> repair block
Warning: Missing value for 'block', using global setting: 10

Repairing block 10 in file /oradata/NNC_DATA207.dbf...

Repair analysis for block 10:
1. seq_kcbh check: 0xFF -> needs repair (0x01)
2. Tailchk check: 0x00001EFF -> needs repair (0x00001E01)
3. Checksum check: 0xCBC7 -> OK

Confirm repair operations:
File: /oradata/NNC_DATA207.dbf
Block: 10
Operations needed: fix offset14, fix tailchk
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x00001E01 OK
3. Checksum: 0xCBC7 OK

Block 10 repair completed successfully.

OBET>

OBET> set file 119
filename set to: /oradata/NNC_DATA210.dbf (file# 119)
endian auto-detected: little

OBET> set block 2
block set to: 2

OBET> backup
Backing up file #119, block 2
Successfully backed up block 2 from /oradata/NNC_DATA210.dbf to backup_blk/NNC_DATA210.dbf.2_20260902200645.blk

OBET> repair block
Warning: Missing value for 'block', using global setting: 2

Repairing block 2 in file /oradata/NNC_DATA210.dbf...

Repair analysis for block 2:
1. seq_kcbh check: 0xFF -> needs repair (0x01)
2. Tailchk check: 0x00001DFF -> needs repair (0x00001D01)
3. Checksum check: 0x80EC -> OK

Confirm repair operations:
File: /oradata/NNC_DATA210.dbf
Block: 2
Operations needed: fix offset14, fix tailchk
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x00001D01 OK
3. Checksum: 0x80EC OK

Block 2 repair completed successfully.

OBET> exit
Exiting OBET.

使用dbv检测,位图坏块全部修复

[oracle@nccdb ~]$ dbv file=/oradata/NNC_DATA210.dbf

DBVERIFY: Release 11.2.0.4.0 - Production on Wed Sep 2 20:07:15 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /oradata/NNC_DATA210.dbf


DBVERIFY - Verification complete

Total Pages Examined         : 4194302
Total Pages Processed (Data) : 3472440
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 205988
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 515867
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 7
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 751241252 (22.751241252)
[oracle@nccdb ~]$ dbv file=/oradata/NNC_DATA207.dbf

DBVERIFY: Release 11.2.0.4.0 - Production on Wed Sep 2 20:08:23 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /oradata/NNC_DATA207.dbf


DBVERIFY - Verification complete

Total Pages Examined         : 4194302
Total Pages Processed (Data) : 3471468
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 204185
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 518642
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 7
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 751241266 (22.751241266)

后续数据顺利导出,完成这次恢复

不太常见的10.2.0.1的oracle redo损坏恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：不太常见的10.2.0.1的oracle redo损坏恢复

曾经恢复过大量10g的库,现在一年也恢复不了几个10g的了,而10.2.0.1的64位库更是少之又少了.近期有幸处理过一个这样的case,重温了当年的感觉

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bi
PL/SQL Release 10.2.0.1.0 - Production
CORE 10.2.0.1.0 Production
TNS for 64-bit Windows: Version 10.2.0.1.0 - Production
NLSRTL Version 10.2.0.1.0 - Production

由于系统断电导致该版本的erp系统的数据库无法启动,看alert日志由于redo损坏导致异常(ORA-00354 ORA-00353)

Thu Jun 04 10:27:40 2026
ALTER DATABASE RECOVER  datafile 5  
Thu Jun 04 10:27:40 2026
Media Recovery Start
WARNING! Recovering data file 5 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
 parallel recovery started with 7 processes
Thu Jun 04 10:27:40 2026
Recovery of Online Redo Log: Thread 1 Group 3 Seq 26215 Reading mem 0
  Mem# 0 errs 0: D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG
Thu Jun 04 10:28:06 2026
Errors in file d:\oracle\product\10.2.0\admin\ORCL\udump\ORCL_ora_3856.trc:
ORA-00354: 损坏重做日志块头部
ORA-00353: 日志损坏接近块 93816 更改 1049070732 时间 05/27/2025 18:00:16
ORA-00334: 归档日志: 'D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG'

接手之后再次验证了该错误

这种错误基本上要不bbed/obet修改文件头,要不直接屏蔽一致性强制打开,我直接选择了强制拉库,先做不完全恢复

然后强制拉库报ORA-600 2662错误

直接使用_minimum_giga_scn修改scn,数据库open成功,并使用expdp成功导出数据,完成本次恢复工作

obet forcecopy功能抢救硬件故障中的数据文件

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：obet forcecopy功能抢救硬件故障中的数据文件

客户raid5发现有盘离线,并且其他盘有坏道,进行了多次换盘操作替换了大部分有坏道磁盘,但是部分文件访问依旧会报io错误

直接cp文件报

cp: error reading 'datafile': Input/output error

数据库层面操作报

ORA-01115: IO error reading block from file %s (block # %s)
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 21571487880

对于这样的情况,使用obet的forcecopy功能直接强制拷贝文件,可以最大限度求救数据obet forcecopy

把所有损坏文件拷贝出来之后传输到其他机器上,检查所有文件坏块情况

运气不错,整体坏块非常少,而且没有涉及数据库字典文件,然后尝试恢复

虽然recover database成功,但是不幸open 库报ORA-600 kcbzib_kcrsds_1错误

SQL> ALTER DATABASE OPEN ;
ALTER DATABASE OPEN 
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], []
Process ID: 15786
Session ID: 3054 Serial number: 46983

通过obet的patch_scn功能进行修改scn

顺利打开数据库,并导出数据

kcratr_nab_less_than_odr和system坏块故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：kcratr_nab_less_than_odr和system坏块故障处理

学校客户由于机房断电,导致数据库异常.现场工程师进行了现场备份,然后尝试强制拉库,结果没有成功,大概报错和操作过程如下:
断电之后启动数据库,数据库报ORA-600 kcratr_nab_less_than_odr故障解决错误,这个是一种非常常见的操作,一般是由于写丢失导致,以前有过很多类似恢复经历:
ORA-600 kcratr_nab_less_than_odr故障解决
 差点被误操作的ORA-600 kcratr_nab_less_than_odr故障
 ORA-600 kcratr_nab_less_than_odr和ORA-600 2662故障处理
 ORA-600 kcratr_nab_less_than_odr和ORA-600 4193故障处理

Tue Aug 18 23:59:19 2026
ALTER DATABASE OPEN
This instance was first to open
Beginning crash recovery of 2 threads
 parallel recovery started with 19 processes
Started redo scan
Completed redo scan
 read 21634 KB redo, 800 data blocks need recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_275950.trc  (incident=703037):
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6916], [3], [4], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_703037/orcl2_ora_275950_i703037.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Abort recovery for domain 0
Aborting crash recovery due to error 600
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_275950.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6916], [3], [4], [], [], [], []
Abort recovery for domain 0
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_275950.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6916], [3], [4], [], [], [], []
ORA-600 signalled during: ALTER DATABASE OPEN...

现场恢复人员上来之后,直接尝试做强制resetlogs操作,数据库报ORA-600 ORA-600 krsi_al_hdr_update.15错误,主要是由于redo写丢失导致无法resetlogs成功,具体参考：Alter Database Open Resetlogs returns error ORA-00600: [krsi_al_hdr_update.15], (Doc ID 2026541.1) Oracle断电故障处理

Wed Aug 19 00:28:47 2026
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 29721342127
Archived Log entry 38935 added for thread 1 sequence 6916 ID 0xceea62af dest 1:
ARCH: All Archive destinations made inactive due to error 742
ARCH: Closing local archive destination LOG_ARCHIVE_DEST_1: '+ARCHDG/2_19951_1224781365.arc' (error 742)(orcl2)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_287355.trc  (incident=727038):
ORA-00600: internal error code, arguments: [krsi_al_hdr_update.15], [4294967295], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_727038/orcl2_ora_287355_i727038.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Master archival failure: 600
Archive all online redo logfiles failed:600
ORA-600 signalled during: alter database open resetlogs...

通过 ALTER DATABASE RECOVER database using backup controlfile until cancel之后,继续尝试强制打开库，报ORA-600 2662错误.

Wed Aug 19 00:40:25 2026
Checker run found 32 new persistent data failures
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 29721342127
Archived Log entry 38936 added for thread 1 sequence 6915 ID 0xceea62af dest 1:
Archived Log entry 38937 added for thread 1 sequence 6916 ID 0xceea62af dest 1:
Archived Log entry 38938 added for thread 2 sequence 19951 ID 0xceea62af dest 1:
Archived Log entry 38939 added for thread 2 sequence 19950 ID 0xceea62af dest 1:
Clearing online redo logfile 1 +DATADG/orcl/onlinelog/group_1.319.1224781365
Clearing online log 1 of thread 1 sequence number 6915
Wed Aug 19 00:40:34 2026
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 +DATADG/orcl/onlinelog/group_2.320.1224781367
Clearing online log 2 of thread 1 sequence number 6916
Clearing online redo logfile 2 complete
Clearing online redo logfile 3 +DATADG/orcl/onlinelog/group_3.323.1224781463
Clearing online log 3 of thread 2 sequence number 19951
Clearing online redo logfile 3 complete
Clearing online redo logfile 4 +DATADG/orcl/onlinelog/group_4.324.1224781465
Clearing online log 4 of thread 2 sequence number 19950
Clearing online redo logfile 4 complete
Resetting resetlogs activation ID 3471467183 (0xceea62af)
Online log +DATADG/orcl/onlinelog/group_1.319.1224781365: Thread 1 Group 1 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_1.4574.1224781367: Thread 1 Group 1 was previously cleared
Online log +DATADG/orcl/onlinelog/group_2.320.1224781367: Thread 1 Group 2 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_2.10729.1224781369: Thread 1 Group 2 was previously cleared
Online log +DATADG/orcl/onlinelog/group_3.323.1224781463: Thread 2 Group 3 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_3.12866.1224781463: Thread 2 Group 3 was previously cleared
Online log +DATADG/orcl/onlinelog/group_4.324.1224781465: Thread 2 Group 4 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_4.7527.1224781465: Thread 2 Group 4 was previously cleared
Wed Aug 19 00:40:43 2026
Setting recovery target incarnation to 3
Wed Aug 19 00:40:43 2026
Assigning activation ID 3488382552 (0xcfec7e58)
Thread 2 opened at log sequence 1
  Current log# 3 seq# 1 mem# 0: +DATADG/orcl/onlinelog/group_3.323.1224781463
  Current log# 3 seq# 1 mem# 1: +ARCHDG/orcl/onlinelog/group_3.12866.1224781463
Successful open of redo thread 2
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Aug 19 00:40:43 2026
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_291405.trc  (incident=739007):
ORA-00600: internal error code, arguments: [2662], [6], [3951555809], [6], [3951556555], [12583040], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_739007/orcl2_ora_291405_i739007.trc
Wed Aug 19 00:40:45 2026
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_291405.trc:
ORA-00600: internal error code, arguments: [2662], [6], [3951555809], [6], [3951556555], [12583040], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_291405.trc:
ORA-00600: internal error code, arguments: [2662], [6], [3951555809], [6], [3951556555], [12583040], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 291405): terminating the instance due to error 600
Instance terminated by USER, pid = 291405
ORA-1092 signalled during: alter database open resetlogs...

到这一步,现场停止了继续尝试,我接手故障处理.先dbv检测坏块,发现system有两个坏块

[oracle@db3 ~]$ dbv userid=sys/oracle file=/datapool/cold_backup_20260818/SYSTEM.314.1224781265

DBVERIFY: Release 11.2.0.4.0 - Production on Sat Aug 22 10:39:26 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /datapool/cold_backup_20260818/SYSTEM.314.1224781265
Page 94587 is marked corrupt
Corrupt block relative dba: 0x0041717b (file 1, block 94587)
Bad header found during dbv: 
Data in bad block:
 type: 11 format: 2 rdba: 0x00400001
 last change scn: 0x0000.00000000 seq: 0x1 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x00000b01
 check value in block header: 0xd49
 computed block checksum: 0x0

Page 95021 is marked corrupt
Corrupt block relative dba: 0x0041732d (file 1, block 95021)
Bad header found during dbv: 
Data in bad block:
 type: 11 format: 2 rdba: 0x00400001
 last change scn: 0x0000.00000000 seq: 0x1 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x00000b01
 check value in block header: 0xd49
 computed block checksum: 0x0



DBVERIFY - Verification complete

Total Pages Examined         : 157440
Total Pages Processed (Data) : 72079
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 21854
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 49484
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 14021
Total Pages Marked Corrupt   : 2
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 0 (0.0)

OBET> dbv file 1

===============================================
DBV (Data Block Verification)
Block Size: 8192 bytes
Endian:     little-endian (x86)
Target File: #1(only)
===============================================

Verifying file #1: /datapool/cold_backup_20260818/SYSTEM.314.1224781265 (157441 blocks) - Started: 2026-08-22 11:14:48
File #32: rfile=1 (0x00000001)  header_block_num=157440 (0x00026700)  filesize_status:OK
  Progress: 100000 / 157441 blocks checked...
  File #1completed: 0 all zero, 0 soft corrupted, 2 tailchk error, 0 checksum error, 0 rdba error

DBV completed at: 2026-08-22 11:14:59

===============================================
DBV Summary:
Total blocks checked: 157439
Total all zero blocks found: 0
Total all rdba error blocks found: 0
Total all tailchk error blocks found: 2
Total all soft corrupted blocks found: 0
Total all checksum error blocks found: 0
Total bad blocks found: 2
Execution time: 11.00 seconds
Throughput: 111.82 MB/s
===============================================

Detailed report saved to: dbv_file_32_20260822111448.log
Files processed: 1

OBET> list corrupt
file 1(/datapool/cold_backup_20260818/SYSTEM.314.1224781265) total bad blocks: 2
block#   bad block type
94587   tailchk
95021   tailchk

使用obet修复坏块
Oracle Block Editor Tool使用手册

OBET> set file 32
filename set to: /datapool/cold_backup_20260818/SYSTEM.314.1224781265 (file#1)

OBET> set block 94587
block set to: 94587

OBET> set mode edit
mode set to: edit

OBET> repair block
Warning: Missing value for 'block', using global setting: 94587

Repairing block 94587 in file /datapool/cold_backup_20260818/SYSTEM.314.1224781265...

Repair analysis for block 94587:
1. seq_kcbh check: 0x01 -> OK
2. Tailchk check: 0x0106C0E4 -> needs repair (0xE4C00601)
3. Checksum check: 0x5EC5 -> needs repair (0x7DE6)

Confirm repair operations:
File: /datapool/cold_backup_20260818/SYSTEM.314.1224781265
Block: 94587
Operations needed: fix tailchk, fix checksum
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x0106C0E4 OK
3. Checksum: 0x7DE6 OK

Block 94587 repair completed successfully.

OBET> set block 95021
block set to: 95021

OBET> repair block 
Warning: Missing value for 'block', using global setting: 95021

Repairing block 95021 in file /datapool/cold_backup_20260818/SYSTEM.314.1224781265...

Repair analysis for block 95021:
1. seq_kcbh check: 0x01 -> OK
2. Tailchk check: 0xF8FA0601 -> needs repair (0xE06C0601)
3. Checksum check: 0x81BA -> OK

Confirm repair operations:
File: /datapool/cold_backup_20260818/SYSTEM.314.1224781265
Block: 95021
Operations needed: fix tailchk
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x01066CE0 OK
3. Checksum: 0x81BA OK

Block 95021 repair completed successfully.

dbv检查确认坏块修复成功

[oracle@db3 ~]$ dbv file=/datapool/cold_backup_20260818/SYSTEM.314.1224781265

DBVERIFY: Release 11.2.0.4.0 - Production on Sat Aug 22 11:17:30 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /datapool/cold_backup_20260818/SYSTEM.314.1224781265


DBVERIFY - Verification complete

Total Pages Examined         : 157440
Total Pages Processed (Data) : 72080
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 21855
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 49484
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 14021
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3951556890 (6.3951556890)

后面的恢复比较简单,使用客户恢复之前的备份,直接重建ctl,然后打开库成功,并且做rman校验没有异常,直接把恢复之后的库备份还原到asm里面,完成本次恢复任务

通过obet 恢复system坏块,打开数据库

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：通过obet 恢复system坏块,打开数据库

客户由于误操作直接在虚拟化平台点击电源键,强制关闭了正在运行的数据库服务器虚机,导致数据库无法正常启动,检查发现ntfs文件系统损坏

通过obet工具检测数据文件坏块(obet dbv功能完整说明),发现核心的system文件上面有一些坏块

File #1: D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF (225281 blocks) - Started: 2026-08-21 16:40:23
File #1: rfile=1 (0x00000001)  header_block_num=225280 (0x00037000)  filesize_status:OK
file 1, block 212238: tailchk error (expected 0x0106276F, got 0x01065E92), bad block
file 1, block 213507: tailchk error (expected 0x0106276F, got 0x01065E92), bad block
file 1, block 213515: tailchk error (expected 0x01065E92, got 0x0106276F), bad block
file 1, block 213547: tailchk error (expected 0x0106276F, got 0x01065E92), bad block
file 1, block 213555: tailchk error (expected 0x01065E92, got 0x0106276F), bad block
file 1, block 221387: tailchk error (expected 0x0106276F, got 0x01065E92), bad block
file 1, block 221403: tailchk error (expected 0x01065E92, got 0x0106B19C), bad block
file 1, block 221419: tailchk error (expected 0x01065E92, got 0x0106C7E3), bad block
file 1, block 222117: tailchk error (expected 0x01065E92, got 0x0106276F), bad block
file 1, block 222133: tailchk error (expected 0x0106276F, got 0x01065E92), bad block
file 1, block 222141: tailchk error (expected 0x01065E92, got 0x0106276F), bad block
file 1, block 222173: tailchk error (expected 0x0106C7E3, got 0x01065E92), bad block
  File #1 completed: 0 all zero, 0 soft corrupted, 12 tailchk error, 0 checksum error, 0 rdba error

这个12个坏块的数量和dbv检测结果一致


DBVERIFY: Release 19.0.0.0.0 - Production on 星期五 8月 21 17:37:02 2026

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.


DBVERIFY - 开始验证: FILE = D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF
页 212238 流入 - 很可能是介质损坏
Corrupt block relative dba: 0x00433d0e (file 1, block 212238)
Fractured block found during dbv: 
Data in bad block:
 type: 6 format: 2 rdba: 0x00433d0e
 last change scn: 0x0000.0002.d71a6f27 seq: 0x1 flg: 0x06
 spare3: 0x0
 consistency value in tail: 0x925e0601
 check value in block header: 0x93bb
 computed block checksum: 0xfd79

…………

页 222173 流入 - 很可能是介质损坏
Corrupt block relative dba: 0x004363dd (file 1, block 222173)
Fractured block found during dbv: 
Data in bad block:
 type: 6 format: 2 rdba: 0x004363dd
 last change scn: 0x0000.0002.d708e3c7 seq: 0x1 flg: 0x06
 spare3: 0x0
 consistency value in tail: 0x925e0601
 check value in block header: 0xe089
 computed block checksum: 0x7199


DBVERIFY - 验证完成

检查的页总数: 225280
处理的页总数 (数据): 125451
失败的页总数 (数据): 0
处理的页总数 (索引): 29984
失败的页总数 (索引): 0
处理的页总数 (其他): 48214
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 21619
标记为损坏的总页数: 12
流入的页总数: 12
加密的总页数        : 0
最高块 SCN            : 12199365950 (2.3609431358)

使用obet的reair block功能进行修复(obet repair block使用说明)

OBET> set file 1
filename set to: D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF (file#1)

OBET> set block 212238
block set to: 212238

OBET> d

File: D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF
Block: 212238                Offsets:     0 to    31
--------------------------------------------------------------------------------
67A1C000 06A20000 0E3D4300 276F1AD7 02000106 BB930000 02000000 D0020000 E46E1AD7

<32 bytes read>

OBET> tailchk
Check tailchk for File D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF, Block 212238:
current = 0x925E0601, required = 0x6F270601

OBET> backup
Backing up file #1, block 212238
Successfully backed up block 212238 from D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF 
   to backup_blk\SYSTEM01.DBF.212238_20260821173825.blk

OBET> set mode edit
mode set to: edit

OBET> repair
Usage: repair [subcommand]
  repair block [file N] [X]  - Repair tailchk/checksum (optional: file N, block X)
  repair blkscn [file N] [X] - Repair block SCN (optional: file N, block X)

OBET> repair block
Warning: Missing value for 'block', using global setting: 212238

Repairing block 212238 in file D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF...

Repair analysis for block 212238:
1. seq_kcbh check: 0x01 -> OK
2. Tailchk check: 0x925E0601 -> needs repair (0x6F270601)
3. Checksum check: 0xBB93 -> OK

Confirm repair operations:
File: D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF
Block: 212238
Operations needed: fix tailchk
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x6F270601 OK
3. Checksum: 0xBB93 OK

Block 212238 repair completed successfully.

所有坏块依次进行修复,然后再次使用dbv检测


DBVERIFY: Release 19.0.0.0.0 - Production on 星期五 8月 21 17:42:02 2026

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.


DBVERIFY - 开始验证: FILE = D:\APP\ADMINISTRATOR\ORADATA\NHIS\SYSTEM01.DBF
Block Checking: DBA = 4407811, Block Type = KTB-managed data block
**** actual rows locked by itl 2  = 0 != # in trans. header = 1
**** actual rows locked by itl 3  = 0 != # in trans. header = 1
---- end index block validation
页 213507 失败, 校验代码为 6401
Block Checking: DBA = 4407819, Block Type = KTB-managed data block
**** row 120: key out of order
**** actual rows locked by itl 2  = 0 != # in trans. header = 1
**** actual rows marked deleted = 1 != kdxlende = 0
---- end index block validation
页 213515 失败, 校验代码为 6401

DBVERIFY - 验证完成

检查的页总数: 225280
处理的页总数 (数据): 125451
失败的页总数 (数据): 0
处理的页总数 (索引): 29996
失败的页总数 (索引): 2
处理的页总数 (其他): 48214
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 21619
标记为损坏的总页数: 0
流入的页总数: 0
加密的总页数        : 0
最高块 SCN            : 12199365950 (2.3609431358)

有两个block有少量逻辑错误,可以通过数据库级别设置进行跳过,基本上实现了这个12个坏块的自动修复.后续就是数据库的打开过程

SQL> startup mount pfile='d:/pfile.txt'
ORACLE 例程已经启动。

Total System Global Area 1.2885E+10 bytes
Fixed Size                 16120656 bytes
Variable Size            7751073792 bytes
Database Buffers         5100273664 bytes
Redo Buffers               17432576 bytes
数据库装载完毕。
SQL> recover database;
ORA-00283: 恢复会话因错误而取消
ORA-00742: 日志读取在线程 1 序列 19828 块 11511 中检测到写入丢失情况
ORA-00312: 联机日志 2 线程 1: 'D:\APP\ADMINISTRATOR\ORADATA\NHIS\REDO02.LOG'

SQL> RECOVER DATABASE UNTIL TIME '2026-08-21:13:28:56' USING BACKUP CONTROLFILE;
ORA-00279: change 12198905983 generated at 08/21/2026 12:43:18 needed for
thread 1
ORA-00289: suggestion :
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000019827_1199133733.0001
ORA-00280: change 12198905983 for thread 1 is in sequence #19827


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ADMINISTRATOR\ORADATA\NHIS\REDO01.LOG
ORA-00279: change 12199301614 generated at 08/21/2026 13:27:06 needed for
thread 1
ORA-00289: suggestion :
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000019828_1199133733.0001
ORA-00280: change 12199301614 for thread 1 is in sequence #19828
ORA-00278: log file 'D:\APP\ADMINISTRATOR\ORADATA\NHIS\REDO01.LOG' no longer
needed for this recovery


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ADMINISTRATOR\ORADATA\NHIS\REDO02.LOG
ORA-00756: 鎭㈠鎿嶄綔妫€娴嬪埌鏁版嵁鍧楀啓鍏ヤ涪澶?ORA-10567:
Redo is inconsistent with data block (file# 4, block# 31663, file offset is 259383296 bytes)
ORA-10564: tablespace UNDOTBS1
ORA-01110: 鏁版嵁鏂囦欢 4: 'D:\APP\ADMINISTRATOR\ORADATA\NHIS\UNDOTBS01.DBF'
ORA-10560: block type 'KTU UNDO BLOCK'

ORA-01112: media recovery not started

SQL> alter database open resetlogs;

Database altered.

使用expdp导出数据,完成本次恢复任务

obet dbv功能完整说明

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：obet dbv功能完整说明

大量的恢复经验和实战总结,逐步了完善obet(Oracle Block Edit Tool)功能和实现恢复的便利性.对于数据库恢复其中一个重要功能就是检测数据块的损坏情况(这个直接关系到恢复的实际效果),Oracle本身提供的dbv功能有一些不足,obet工具的dbv功能是对oracle本身的dbv功能的深度完善,弥补其本身的几个功能不足点:
1. Oracle dbv 需要安装oracle 软件,对于有些时候为了一个dbv检测需要安装Oracle 软件，成本和动作太大,obet工具直接单个文件即可运行
2. 如果数据文件头损坏,或者文件大小不对,或者block 0损坏,Oracle自带dbv均无法检测
3. Oracle dbv无法实现跨字节序检测(比如需要在win平台上检测aix系统的数据文件的坏块情况[比如硬件恢复场景需要])
4. Oracle dbv有些版本无法实现低版本dbv检测高版本数据库文件
5. Oracle dbv无法一条命令实现多个数据文件检测(需要人工写脚本来实现)

在使用obet的dbv功能之前需要准备listfile文件(内容格式:No Name)
listfile文件操作
1. 在可以mount的库中可以直接通过查询v$datafile(select file#,name from v$datafile),把结果保存到listfile.txt中
2. 通过obet的listfile功能直接获取

OBET> listfile
Usage: listfile <path|remove|edit>
  listfile <path>   - Scan directory and write to listfile.txt
  listfile remove   - Delete listfile.txt
  listfile edit [path] - Edit listfile.txt (notepad on Win, vi on Linux)

listfile 支持*通配符

OBET> listfile e:\oradata\orcl\sys*.dbf
Listed 2 files from 'e:\oradata\orcl' to listfile.txt (starting from #1).

OBET> open listfile.txt
Loaded 2 files from  datafile list 'listfile.txt'.

OBET> info

Loaded files (2 total):
----------------------------------------
Number  Path
----------------------------------------
     1  e:\oradata\orcl\SYSAUX01.DBF
     2  e:\oradata\orcl\SYSTEM01.DBF
----------------------------------------

listfile remove 直接删除掉当前目录下面的listfile.txt文件
listfile edit 直接调用notepad/vi 编辑模式打开当前目录下的listfile.txt文件(也可以指定具体文件名称)
3. 使用open命令加载listfile

OBET> open listfile.txt
Loaded 2 files from  datafile list 'listfile.txt'.

OBET> info

Loaded files (2 total):
----------------------------------------
Number  Path
----------------------------------------
     1  e:\oradata\orcl\SYSAUX01.DBF
     2  e:\oradata\orcl\SYSTEM01.DBF
----------------------------------------

dbv之前确认参数设置
在执行dbv之前需要先确认当前一些配置,和dbv相关的主要是blocksize(默认值为8192),endian(默认值为little),可以通过show命令查看(如果是检测linux/win环境下默认的oracle数据文件,理论上不用这修改这些值)

OBET> show

Current settings:
File: (not set)
Blocksize: 8192 bytes
Block: 1
Offset in block: 0 (file offset: 0x00002000)
Count: 32 bytes
Mode: browse
Endian: little (x86)
Loaded files: 2 (use 'info' to list)

可以根据情况通过set命令修改相关值,具体语法为:set

OBET> set
Usage: set <param> <value>
  set filename <path>    - Set target file path (required)
  set file <num>         - Set filename using loaded file number (from open list)
  set blocksize <size>  - Set block size (2048,4096,[8192],16384,32768)
  set block <num>        - Set block number (starts from 0, default: 1)
  set offset <offset>   - Set offset within block (< blocksize, default: 0)
  set count <bytes>      - Set number of bytes to read (default: 32)
  set mode edit/browse   - Enable edit/browse mode
  set endian big/little  - Set byte order (default: little, big for AIX files)

dbv 检测数据文件
具体语法为:dbv [file N [block M [blocks X]]]
dbv 检测open listfile中的所有数据文件的所有block
dbv file N 检测listfile中编号为N的数据文件的所有block
dbv file N block M 检测listfile中编号为N的数据文件的block M开始到文件结束
dbv file N block M block X 检测listfile中编号为N的数据文件的block M开始,检测X个block

OBET> dbv

===============================================
DBV (Data Block Verification)
Block Size: 8192 bytes
Endian:     little-endian (x86)
Target File: ALL files in listfile
===============================================

Verifying file #1: e:\oradata\orcl\SYSAUX01.DBF (139521 blocks) - Started: 2026-08-16 17:37:25
File #1: rfile=2 (0x00000002)  header_block_num=139520 (0x00022100)  filesize_status:OK
  Progress: 100000 / 139521 blocks checked...
  File #1 completed: no bad blocks found
Verifying file #2: e:\oradata\orcl\SYSTEM01.DBF (126721 blocks) - Started: 2026-08-16 17:37:26
File #2: rfile=1 (0x00000001)  header_block_num=126720 (0x0001EF00)  filesize_status:OK
  Progress: 100000 / 126721 blocks checked...
  File #2 completed: no bad blocks found

DBV completed at: 2026-08-16 17:37:26

===============================================
DBV Summary:
Total blocks checked: 266242
Total all zero blocks found: 0
Total all rdba error blocks found: 0
Total all tailchk error blocks found: 0
Total all soft corrupted blocks found: 0
Total all checksum error blocks found: 0
Total bad blocks found: 0
Execution time: 1.00 seconds
Throughput: 2080.02 MB/s
===============================================

Detailed report saved to: dbv_20260816173725.log
Files processed: 2

OBET> dbv file 1

===============================================
DBV (Data Block Verification)
Block Size: 8192 bytes
Endian:     little-endian (x86)
Target File: #1 (only)
===============================================

Verifying file #1: e:\oradata\orcl\SYSAUX01.DBF (139521 blocks) - Started: 2026-08-16 17:37:31
File #1: rfile=2 (0x00000002)  header_block_num=139520 (0x00022100)  filesize_status:OK
  Progress: 100000 / 139521 blocks checked...
  File #1 completed: no bad blocks found

DBV completed at: 2026-08-16 17:37:32

===============================================
DBV Summary:
Total blocks checked: 139521
Total all zero blocks found: 0
Total all rdba error blocks found: 0
Total all tailchk error blocks found: 0
Total all soft corrupted blocks found: 0
Total all checksum error blocks found: 0
Total bad blocks found: 0
Execution time: 1.00 seconds
Throughput: 1090.01 MB/s
===============================================

Detailed report saved to: dbv_file_1_20260816173731.log
Files processed: 1

OBET> dbv file 1 block 128

===============================================
DBV (Data Block Verification)
Block Size: 8192 bytes
Endian:     little-endian (x86)
Target File: #1, block 128 to end
===============================================

Verifying file #1: e:\oradata\orcl\SYSAUX01.DBF (block 128 to 139520, 139393 blocks) - Started: 2026-08-16 17:37:42
File #1: rfile=2 (0x00000002)  header_block_num=139520 (0x00022100)  filesize_status:OK
  Progress: 100128 / 139521 blocks checked...
  File #1 completed: no bad blocks found

DBV completed at: 2026-08-16 17:37:43

===============================================
DBV Summary:
Total blocks checked: 139393
Total all zero blocks found: 0
Total all rdba error blocks found: 0
Total all tailchk error blocks found: 0
Total all soft corrupted blocks found: 0
Total all checksum error blocks found: 0
Total bad blocks found: 0
Execution time: 1.00 seconds
Throughput: 1090.01 MB/s
===============================================

Detailed report saved to: dbv_file_1_block_128_20260816173742.log
Files processed: 1

OBET> dbv file 1 block 128 blocks 10

===============================================
DBV (Data Block Verification)
Block Size: 8192 bytes
Endian:     little-endian (x86)
Target File: #1, block 128 - 137 (10 blocks)
===============================================

Verifying file #1: e:\oradata\orcl\SYSAUX01.DBF (block 128 to 137, 10 blocks) - Started: 2026-08-16 17:37:47
File #1: rfile=2 (0x00000002)  header_block_num=139520 (0x00022100)  filesize_status:OK
  File #1 completed: no bad blocks found

DBV completed at: 2026-08-16 17:37:47

===============================================
DBV Summary:
Total blocks checked: 10
Total all zero blocks found: 0
Total all rdba error blocks found: 0
Total all tailchk error blocks found: 0
Total all soft corrupted blocks found: 0
Total all checksum error blocks found: 0
Total bad blocks found: 0
Execution time: 0.00 seconds
Throughput: 0.00 MB/s
===============================================

Detailed report saved to: dbv_file_1_block_128_20260816173747.log
Files processed: 1

检测的具体明细可以看obet所在的当前目录中的dbv*.log文件,里面有坏块类型:
rdba error:表示block#和该块所在位置不匹配
tailchk/checksum error:表示该block不满足Oracle的校验规则
soft corrupted:表示该block在Oracle数据库标记为坏块
zero block:表示该block中全部为0,一般是由于底层损坏,或者硬件恢复过程使用空块替代引起
filesize_status:NO表示文件实际大小和文件头记录大小不匹配
下载obet obet使用说明

分享一例运行在aix上的sap系统数据库恢复过程

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：分享一例运行在aix上的sap系统数据库恢复过程

情况描述
客户sap系统运行在aix系统,ibm v7000存储上,数据存放在三个1T的lun组成的vg的多个lv中.异常断电之后,但是给数据库使用的三个lun丢失了2个,从而使得所有vg/lv异常,通过硬件恢复出来异常的2个lun和剩余的1个lun一起,把所有的数据文件恢复出来.但是由于某种原因,出现部分block被覆盖(其中还包括两个文件头损坏).

坏块检测
对于恢复的所有文件,为了快速做一遍坏块检查,直接在恢复的win机器上使用过obet做了一次dbv检查obet实现对数据文件坏块检测功能(使用obet的dbv检查有几个好处:1>可以在win上面检测aix的数据文件;2>可以检测数据文件头损坏的数据文件其他block;3>检测速度比原生dbv快[每个文件内部加了并行检测]),检测结果如下

--其中两个文件头损坏
File #62: E:\sr3_54\sr3.data54 (1280000 blocks) - Started: 2026-07-28 23:57:53
File #62: rfile=0 (0x00000000)  header_block_num=0 (0x00000000)  filesize_status:NO
file 62, block 0: block all zero
file 62, block 1: block all zero

File #84: E:\sr3_76\sr3.data76 (4185601 blocks) - Started: 2026-07-29 00:55:32
File #84: rfile=638937491 (0x26156993)  header_block_num=77152556 (0x0499412C)  filesize_status:NO
file 84, block 1: rdba error (expected 1, got 69760), bad block
file 84, block 2: rdba error (expected 2, got 589836), bad block

--坏块汇总
DBV completed at: 2026-07-29 00:58:10
===============================================
DBV Summary:
Total blocks checked: 196287237
Total all zero blocks found: 337417
Total all rdba error blocks found: 1062468
Total all tailchk error blocks found: 1
Total all soft corrupted blocks found: 0
Total all checksum error blocks found: 0
Total bad blocks found: 1399886
Execution time: 10507.00 seconds
===============================================

这个统计下来好的block在99.3%左右,证明硬件层面的会效果已经非常好.

碎片工具进一步恢复
恢复公司文件系统层面恢复有数据块遗漏的可能,通过碎片工具(OraScan(Oracle 碎片扫描工具) 使用说明)进一步扫描

通过确认62号文件还有少量block可以进一步恢复（也就是说碎片扫描到的62号文件的block多于硬件公司恢复出来的62号文件里面好的block数量）,通过obet的merge功能进行填补

基于上述操作,对于lun里面的数据文件实现了最大效果恢复.

数据库恢复操作
1. 上次恢复文件到aix,offline掉异常文件头数据文件打开数据库

sapprd2:oraprd 8> sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Mon Aug 3 18:41:28 2026

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select status from v$instance;

STATUS
------------------------------------
STARTED

SQL> alter database mount;
alter database mount
*
ERROR at line 1:
ORA-00214: control file '/oracle/PRD/origlogA/cntrl/cntrlPRD.dbf' version
9742943 inconsistent with file '/oracle/PRD/sapdata1/cntrl/cntrlPRD.dbf'
version 9742931

解决ctl不一致问题之后继续mount库恢复

sapprd2:oraprd 11> sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Mon Aug 3 18:42:53 2026

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> alter database mount;

Database altered.

SQL> alter database datafile 62,84 offline;

Database altered.

SQL> recover database ;
Media recovery complete.
SQL> alter database open;

Database altered.

2.对于两个损坏的数据文件头进行修复
由于现在aix环境的客户比较少,obet没有在aix环境下进行编译,因此直接使用bbed工具进行文件头修复（列举了主要操作过程）

BBED> copy file 83 block 1 to file 84 block 1
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:    0 to   31           Dba:0x15000001
------------------------------------------------------------------------
 0ba20000 14c00001 00000000 00000104 22250000 00000000 0b200000 72bc991d

 <32 bytes per line>

BBED> set offset 368
        OFFSET          368

BBED> d
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:  368 to  399           Dba:0x15000001
------------------------------------------------------------------------
 00000053 00000000 00000000 495dc097 00000000 00000000 00000000 00000000

 <32 bytes per line>

BBED> m /x 00000054
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:  368 to  399           Dba:0x15000001
------------------------------------------------------------------------
 00000054 00000000 00000000 495dc097 00000000 00000000 00000000 00000000

 <32 bytes per line>

BBED> set offset 52
        OFFSET          52

BBED> d
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:   52 to   83           Dba:0x15000001
------------------------------------------------------------------------
 00530003 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 <32 bytes per line>

BBED> m /x 0054
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:   52 to   83           Dba:0x15000001
------------------------------------------------------------------------
 00540003 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 <32 bytes per line>

BBED> set offset 4
        OFFSET          4

BBED> d
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:    4 to   35           Dba:0x15000001
------------------------------------------------------------------------
 14c00001 00000000 00000104 22250000 00000000 0b200000 72bc991d 50524400

 <32 bytes per line>

BBED> m /x 15000001
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:    4 to   35           Dba:0x15000001
------------------------------------------------------------------------
 15000001 00000000 00000104 22250000 00000000 0b200000 72bc991d 50524400

 <32 bytes per line>


BBED> m /x E1B0
Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) y
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:  100 to  611           Dba:0x15000001
------------------------------------------------------------------------
 e1b0853f 00040000 4083fcfe 33606b63 02502452 0000ca28 49680fb3 93b9f6b9

 <32 bytes per line>

BBED> set offset +2
        OFFSET          102

BBED> m /x 8542
 File: /oracle/PRD/sapdata3/sr3_76/sr3.data76 (84)
 Block: 1                Offsets:  102 to  613           Dba:0x15000001
------------------------------------------------------------------------
 85420004 00004083 fcfe3360 6b630250 24520000 ca284968 0fb393b9 f6b90005

 <32 bytes per line>


BBED> sum
Check value for File 84, Block 1:
current = 0x2225, required = 0x23e5

BBED> sum apply
Check value for File 84, Block 1:
current = 0x23e5, required = 0x23e5

BBED> verify
DBVERIFY - Verification starting
FILE = /oracle/PRD/sapdata3/sr3_76/sr3.data76
BLOCK = 1


DBVERIFY - Verification complete

Total Blocks Examined         : 1
Total Blocks Processed (Data) : 0
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED

修改完成之后,还出现过几个错误

SQL> alter database open ;
alter database open
*
ERROR at line 1:
ORA-01122: database file 62 failed verification check
ORA-01110: data file 62: '/oracle/PRD/sapdata2/sr3_54/sr3.data54'
ORA-01200: actual file size of 1279999 is smaller than correct size of 1280000

ORA-01200是由于数据文件比文件头记录信息小一个block,通过补上这个block解决

Read of datafile '/oracle/PRD/sapdata2/sr3_54/sr3.data54' (fno 62) header failed with ORA-01202
Rereading datafile 62 header failed with ORA-01202
Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_11272278.trc:
ORA-01122: database file 62 failed verification check
ORA-01110: data file 62: '/oracle/PRD/sapdata2/sr3_54/sr3.data54'
ORA-01202: wrong incarnation of this file - wrong creation time

ORA-01202是由于create time没有修改正确导致,重新修改解决

Rereading datafile 84 header failed with ORA-01203
Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_17105252.trc:
ORA-01122: database file 84 failed verification check
ORA-01110: data file 84: '/oracle/PRD/sapdata3/sr3_76/sr3.data76'
ORA-01203: wrong incarnation of this file - wrong creation SCN

ORA-01203是由于create scn没有修改正确导致,重新修改解决

Errors in file /oracle/PRD/saptrace/diag/rdbms/prd/PRD/trace/PRD_ora_17105278.trc:
ORA-01122: database file 62 failed verification check
ORA-01110: data file 62: '/oracle/PRD/sapdata2/sr3_54/sr3.data54'
ORA-01207: file is more recent than control file - old control file
ORA-1122 signalled during: alter database open .

ORA-01207是由于数据文件的ckp信息比控制文件的新,重建ctl解决,解决这些问题之后,顺利打开数据库

SQL> startup mount;
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 3.7548E+10 bytes
Fixed Size                  2254136 bytes
Variable Size            1.9193E+10 bytes
Database Buffers         1.8254E+10 bytes
Redo Buffers               98996224 bytes
Database mounted.
SQL> alter database open;

Database altered.

SQL> select status,count(1) from v$datafile_header group by status;

STATUS    COUNT(1)
------- ----------
ONLINE          84

然后跳过坏块,导出数据,对于无法导出的异常表进行特殊处理,完成本次恢复工作,最终恢复结果总结