记录block 0损坏,数据文件大量坏块,使用不当数据库版本恢复等各种操作之后的故障处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:记录block 0损坏,数据文件大量坏块,使用不当数据库版本恢复等各种操作之后的故障处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

今天一个恢复行业的朋友让我帮忙看一个oracle故障,说是vmdk文件被加密(加密破坏的很少),然后他从里面恢复出来了oracle的数据库文件,客户那边拿到数据文件然后说有三个文件头(被重命名为.bak)损坏了,无法打开数据库,让我这边给他们分析处理.分析他们说的被破坏文件,确实有损坏(初步看是block 0)
block0


使用oracle自带的dbv进行检测

C:\Users\XFF>dbv file=H:\BaiduNetdisk\D\SYSTEM06.DBF.bak

DBVERIFY: Release 11.2.0.4.0 - Production on 星期五 7月 3 21:06:28 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.


DBV-00107: 未知标头格式 (85) (1311029317)

由于block 0损坏,oracle自带的dbv无法检测,使用obet的dbv功能进行检测(obet实现对数据文件坏块检测功能)

===============================================
DBV (Data Block Verification) Report
Started: 2026-07-03 21:04:22
Block Size: 8192 bytes
Target File: ALL files in listfile
===============================================

File #1: H:\BaiduNetdisk\D\SYSTEM06.DBF.bak (12801 blocks) - Started: 2026-07-03 21:04:22
File #1: rfile=19 (0x00000013)  header_block_num=12800 (0x00003200)  filesize_status:OK
file 1, block 0: rdba error (expected 0, got 4083969), bad block
  File #1 completed: 0 all zero, 0 soft corrupted, 0 tailchk error, 0 checksum error, 1 rdba error

File #1: H:\BaiduNetdisk\D\TSP_XXXXS06.DBF.bak (4194303 blocks) - Started: 2026-07-03 21:04:22
File #1: rfile=24 (0x00000018)  header_block_num=4194302 (0x003FFFFE)  filesize_status:OK
file 1, block 0: rdba error (expected 0, got 1367117), bad block
  File #1 completed: 0 all zero, 0 soft corrupted, 0 tailchk error, 0 checksum error, 1 rdba error

File #1: H:\BaiduNetdisk\D\TSP_XXXXS07.DBF.bak (1933313 blocks) - Started: 2026-07-03 21:04:42
File #1: rfile=25 (0x00000019)  header_block_num=1933312 (0x001D8000)  filesize_status:OK
file 1, block 0: rdba error (expected 0, got 2445572), bad block
  File #1 completed: 0 all zero, 0 soft corrupted, 0 tailchk error, 0 checksum error, 1 rdba error


DBV completed at: 2026-07-03 21:04:51
===============================================
DBV Summary:
Total blocks checked: 6140414
Total all zero blocks found: 0
Total all rdba error blocks found: 3
Total all tailchk error blocks found: 0
Total all soft corrupted blocks found: 0
Total all checksum error blocks found: 0
Total bad blocks found: 3
Execution time: 29.00 seconds
===============================================

通过检测确认这三个数据文件就是block 0损坏其他的数据块是ok的.检查其他数据文件发现有一个数据文件有近2w个坏块

DBVERIFY: Release 11.2.0.4.0 - Production on 星期五 7月 3 12:55:08 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.


DBVERIFY - 开始验证: FILE = H:\BAIDUNETDISK\ORCL\TSP_XXXX.DBF
页 409601 标记为损坏
Corrupt block relative dba: 0x01864001 (file 6, block 409601)
Bad header found during dbv: 
Data in bad block:
 type: 219 format: 1 rdba: 0xdf6e738f
 last change scn: 0xd7ed.50be2d8b seq: 0x44 flg: 0x52
 spare1: 0x39 spare2: 0x92 spare3: 0xb5c5
 consistency value in tail: 0x6e7debc9
 check value in block header: 0x6d54
 block checksum disabled

………………
DBVERIFY - 验证完成

检查的页总数: 4194302
处理的页总数 (数据): 1842107
失败的页总数 (数据): 0
处理的页总数 (索引): 2143604
失败的页总数 (索引): 0
处理的页总数 (其他): 13763
处理的总页数 (段)  : 0
失败的总页数 (段)  : 0
空的页总数: 437
标记为损坏的总页数: 194391
流入的页总数: 2
加密的总页数        : 0
最高块 SCN            : 1869110545 (9.1869110545)

对于这个文件,由于客户有部分rman备份,尝试通过备份找出来这个文件历史文件

RMAN> run
2> {
3> set newname for datafile 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\TSP_XXXX.DBF' 
4>   to 'H:\BaiduNetdisk\orcl\TSP_XXXX.DBF_rman';
5> restore datafile 6;
6> }

正在执行命令: SET NEWNAME

启动 restore 于 03-7月 -26
使用通道 ORA_DISK_1

通道 ORA_DISK_1: 正在开始还原数据文件备份集
通道 ORA_DISK_1: 正在指定从备份集还原的数据文件
通道 ORA_DISK_1: 将数据文件 00006 还原到 H:\BaiduNetdisk\orcl\TSP_XXXX.DBF_rman
通道 ORA_DISK_1: 正在读取备份片段 H:\BAIDUNETDISK\BACKDB\FULLDB_ORCL_20250704_TC3TM4LJ_1_1.BAK.RESTORED0
通道 ORA_DISK_1: ORA-19870: 还原备份片段 H:\BAIDUNETDISK\BACKDB\FULLDB_ORCL_20250704_TC3TM4LJ_1_1.BAK.RESTORED0 时出错
ORA-19599: 块编号 1065697 已在 backup piece H:\BAIDUNETDISK\BACKDB\FULLDB_ORCL_20250704_TC3TM4LJ_1_1.BAK.RESTORED0 中损 坏

故障转移到上一个备份

通道 ORA_DISK_1: 正在开始还原数据文件备份集
通道 ORA_DISK_1: 正在指定从备份集还原的数据文件
通道 ORA_DISK_1: 将数据文件 00006 还原到 H:\BaiduNetdisk\orcl\TSP_XXXX.DBF_rman
通道 ORA_DISK_1: 正在读取备份片段 H:\BAIDUNETDISK\BACKDB\FULLDB_ORCL_20250703_T33TJG9L_1_1.BAK
通道 ORA_DISK_1: ORA-19870: 还原备份片段 H:\BAIDUNETDISK\BACKDB\FULLDB_ORCL_20250703_T33TJG9L_1_1.BAK 时出错
ORA-19599: 块编号 95352 已在 backup piece H:\BAIDUNETDISK\BACKDB\FULLDB_ORCL_20250703_T33TJG9L_1_1.BAK 中损坏

故障转移到上一个备份

通道 ORA_DISK_1: 正在开始还原数据文件备份集
通道 ORA_DISK_1: 正在指定从备份集还原的数据文件
通道 ORA_DISK_1: 将数据文件 00006 还原到 H:\BaiduNetdisk\orcl\TSP_XXXX.DBF_rman
通道 ORA_DISK_1: 正在读取备份片段 H:\BAIDUNETDISK\BACKDB\FULLDB_ORCL_20250702_SQ3TGRTU_1_1.BAK
通道 ORA_DISK_1: 段句柄 = H:\BAIDUNETDISK\BACKDB\FULLDB_ORCL_20250702_SQ3TGRTU_1_1.BAK 标记 = TAG20250702T000518
通道 ORA_DISK_1: 已还原备份片段 1
通道 ORA_DISK_1: 还原完成, 用时: 00:28:56
完成 restore 于 03-7月 -26

RMAN> exit

运气还不错,从备份里面找一份好的该文件的备份,然后通过使用该文件好的block替换最初损坏的block,实现该文件无坏块(由于这个文件比较靠前,而且已经写满,所以只差3天左右数据可能改变很小,因此可以采用这种替代方法最大限度恢复数据).数据文件检测和明显坏块处理完成,接下来开始打开数据库操作.
由于我这边的数据文件路径和原库的不一致,修改ctl中的datafile 路径报ORA-17503错误
rename


对于这些情况,通过修复block 0,然后尝试重命名数据文件成功

SQL> alter database rename file 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\TSP_XXX06.DBF' to 'H:\BaiduNetdisk\orcl\TSP_XXX06.DBF'
  2  ;

数据库已更改。

SQL> alter database rename file 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\TSP_XXX07.DBF' to 'H:\BaiduNetdisk\orcl\TSP_XXX07.DBF'
  2  ;

数据库已更改。

SQL> alter database rename file 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM06.DBF' to 'H:\BaiduNetdisk\orcl\SYSTEM06.DBF'
  2  ;

数据库已更改。

然后尝试打开数据库成功

SQL> recover database;
完成介质恢复。
SQL> alter database open;

数据库已更改。

尝试expdp导出数据,结果报UDE-22303

C:\Users\XFF>expdp "'/ as sysdba'" full=y dumpfile=expdp_0703_%U.dmp DIRECTORY=expdp_dir logfile=expdp_0703.log 
 parallel=4 compression=all EXCLUDE=STATISTICS,AUDIT

Export: Release 11.2.0.4.0 - Production on 星期五 7月 3 13:10:12 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

连接到: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

UDE-22303: 操作产生了 ORACLE 错误 22303
OCI-22303: 未找到类型 "SYS"."KU$_STATUS1020"

正常打开的数据库,没有明显坏块,出现这个错误,理论上不太应该,怀疑是客户的版本问题,查询组件版本
v1


确认该数据库版本可能是11.2.0.1而不是我现在看到的数据文件头为11.2.0.4,查询WRM$_DATABASE_INSTANCE,确认数据库之前版本是11.2.0.1.对于这种情况,直接简单的把compatible从11.2.0.4修改为11.2.0.0肯定不行,因为ctl,dbf,redo里面都写了11.2.0.4的信息
1. ctl报版本不匹配

SQL> startup mount pfile='i:/pfile.txt';
ORACLE 例程已经启动。

Total System Global Area 3206836224 bytes
Fixed Size                  2180024 bytes
Variable Size             654314568 bytes
Database Buffers         2533359616 bytes
Redo Buffers               16982016 bytes
ORA-00201: ?????? 11.2.0.4.0 ? ORACLE ?? 11.2.0.0.0 ???
ORA-00202: ????: ''H:\BAIDUNETDISK\ORCL\CONTROL01.CTL''

2. 这种情况需要rectl,然后报dbf版本不对

CREATE CONTROLFILE REUSE DATABASE "ORCL" NORESETLOGS  NOARCHIVELOG
*
第 1 行出现错误:
ORA-01503: CREATE CONTROLFILE ??
ORA-01130: ??????? 11.2.0.4.0 ? ORACLE ?? 11.2.0.0.0 ???
ORA-01110: ???? 1: 'H:\BAIDUNETDISK\ORCL\SYSTEM01.DBF'

这种情况,dbf中的版本信息无法通过重建解决,只能使用obet工具修改,每个文件类似修改(Oracle数据块编辑工具( Oracle Block Editor Tool)-obet)

OBET> set file 25
filename set to: H:\BAIDUNETDISK\ORCL\TSP_XXXX07.DBF (file#25)

OBET> set offset 24
offset set to: 24

OBET> m 0000

Confirm modification:
File: H:\BAIDUNETDISK\ORCL\TSP_XXXX07.DBF
Block: 1
Offset: 24 (file offset: 0x00002018)
Original value: 0004
New value:      0000
Confirm? (Y/YES to proceed): y
Verification successful: Data written correctly.
Modified 2 bytes at offset 0x00002018 successfully.

OBET> sum apply

Confirm applying checksum:
File: H:\BAIDUNETDISK\ORCL\TSP_XXXX07.DBF
Block: 1
Offset in block: 16 (file offset: 0x00002010)
Original value: 0x94C3
New value:      0x94C7
Confirm? (Y/YES to proceed): y
Verification successful: Stored checksum matches calculated value (0x94C7).
Checksum applied successfully.

OBET>

然后重建ctl报redo版本不兼容ORA-00331

CREATE CONTROLFILE REUSE DATABASE "ORCL" NORESETLOGS  NOARCHIVELOG
*
第 1 行出现错误:
ORA-01503: CREATE CONTROLFILE 失败
ORA-00331: 日志版本 0.0.0.0.0 与 ORACLE 版本 11.2.0.0.0 不兼容
ORA-01517: 日志成员: 'H:\BAIDUNETDISK\ORCL\REDO01.LOG'

采用resetlogs方式重建(不读取redo信息),重建ctl成功,然后直接打开数据库

 37  CHARACTER SET AL32UTF8
 38  ;

控制文件已创建。

SQL> alter database open resetlogs;

数据库已更改。

然后expdp导出数据完成本次恢复任务

dd破坏包含50多个pdb的asm 磁盘组恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:dd破坏包含50多个pdb的asm 磁盘组恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

前段时间刚刚恢复了一个客户dd了34个磁盘中的2个磁盘的1-2G多的数据(在生产环境错误执行dd命令破坏asm磁盘故障恢复),这次又遇到一个客户dd了asm 的两个磁盘的100m和10m,这次故障麻烦的是由于dd了disk 0和disk 1,ausize为4M,有50多个pdb在该磁盘组中,相对恢复比较麻烦,通过不懈努力,最后终于最大限度恢复客户数据,避免了进一步的损失
故障误操作
近期又一个可以误执行了dd命令,破坏了生产库的两个asm disk磁盘
1


上述截图中可以确认两个信息
1.sdh盘被dd掉了10m,sdg盘被dd掉了100M
2.sdg对应的是asm-data1,sdh对应的是asm-data2
dd误操作之后,data磁盘组开始报错,然后直接dismount.
2

分析原磁盘组中asm磁盘情况
1.确定损坏磁盘在磁盘组中位置
通过asm的alert日志分析损坏的盘对应的磁盘组信息
3

基于这个信息,我们可以确认asm-data1 对应的是DATA磁盘组的disk 0,asm-data2对应的是DATA磁盘组的disk 1,也就是说现在DATA磁盘组的0号盘被dd了100M,1号盘被dd了10m
2.确认ausize大小
通过分析该磁盘组的其他磁盘确认ausize大小
kfdhdb.ausize: 4194304 ; 0x0bc: 0×00400000
可以确认该磁盘组的au大小为4M
3.磁盘组后续进行了一次加盘扩容
4

在25年6月份,对data磁盘组加了三块盘
4.sdh(asm-data2)磁盘还被加入到了arch磁盘组中
5

比较幸运由于在另外一个节点上sdh磁盘权限没有正确修改,导致该增加没有完全成功,也就是该磁盘没有被reblance(如果加入成功并正常reblance,那后果比现在严重很多)
5.通过结合asm日志以及kfed,磁盘物理大小等信息,并且通过kfed构造出来损坏的磁盘头信息,列出故障之前DATA磁盘组所有磁盘的情况(为了避免udev别名带来的影响,我直接使用物理磁盘名称来显示)
6

客户现场情况说明
1.客户有一个大概1年多之前的dataguard(后续没有继续同步),已经进行了failover激活
2.客户的备份系统中有一个大概1个月之前的备份,但是备份库缺少归档,我接手之时,已经被维护厂商强制拉起
3.在我恢复之前,有专业的工程师已经对其这个现场进行了分析,但是没有拿出好的恢复方案
恢复难点说明
1.该磁盘组的disk 0 被dd掉了100M,这个导致kfdhdb.f1b1locn记录被清空,也就无法获取到存储指向ASM file 1 文件目录表,这个值虽然被清空,但是根据经验或者对比其他磁盘组的disk 0 可以确认指定aun为10
2.f1b1locn指向的au中存储中asm元数据1-255以及256-1023的file的前面60个au的extent映射表,由于这个在aun为10(也就是aun*aus=40M)的位置,但是这个位置也已经被dd掉了,从原理上业务文件256-1023的前面60个au的extent映射表彻底丢失
3.通过工具扫描,发现存储别名信息的au也在disk 0的前面100M之内(也被dd掉了),导致通过别名直接定位文件的起点的恢复思路也不可行,而且别名丢失导致以别名的数据文件后续识别有一定的难度(无法获取asm里面文件的完整路径)
4.该库有50多套pdb组成,也就意味着通过数据库碎片扫描的方式无法恢复(因为每个pdb里面都是由默认的种子创建而成,也就意味着rfile 1,4,9是重复的)
5.由于中途加过一次磁盘,因引起这个asm里面的文件进行重新reblance,使得部分文件同一个block记录在不同磁盘的au上(数据文件块可能重复)
6.由于该磁盘组中asm disk 大小不等,导致文件au分布在各个磁盘上不均匀,无明显规律可循
恢复操作
1.通过kfed对损坏的asm-data1、asm-disk2的磁盘头进行构造,便于后续的恢复工具识别
2.通过工具对data中所有磁盘进行扫描,主要扫描文件extent映射表信息和ACD中关于asm文件的分配信息
2.1> 通过对这些信息进行综合分析,确认asm file >=1024的文件的extent映射表信息完整,数据可以直接恢复,效果类似
7

2.2> 对于256-1023号文件通过asm file的extent信息缺少0-59 au的数据,结合acd中获取的部分au分配信息,可以尽可能完整的恢复出来这些数据(由于disk 0中的acd信息丢失,所以不是100%完整)
8

3.对于绝对文件号非1,4,9的文件,而且asm file 小于1024的数据文件,结合rdba碎片重组的方式再一次进行恢复,避免上面2.2中恢复中前面240M(60个au),有部分au信息丢失导致文件不完整的情况
通过上述多种方法恢复,整体恢复数据文件类似(由于文件较多,存放多个目录和根据规则取了多种名字)
9

4.由于asm的文件目录,别名信息全部丢失,而且该库有50多个pdb,无法确认恢复出来的上千个文件和pdb对应的关系,对于这种情况,临时写了一个小程序,对这些文件进行读取,获取file#,rfile#,ts#,tsname,文件大小,dbid,dbname,scn等信息(obet(Oracle Block Editor Tool)第二版发布
10

通过把这些信息和历史的控制文件中的信息进行匹配,确认各个文件所属的pdb关系
5.通过4中获取的文件和pdb对应关系然后通过dbms_pdb.recover包实现把恢复的文件插入到新的cdb库中,在这个插入和open库过程中遇到各种错误,都一一解决
11
12
13
14
15

6.最终完成客户数据恢复要求(为了保证数据不被再次修改,客户要求所有恢复的pdb不能打开到读写模式)
16

然后由客户的运维厂商或者应用厂商把需要的数据迁移或者整合到新库中并恢复业务,完成本次恢复任务

Oracle数据库系统回滚段异常处理-ORA-600 4137/4193

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:Oracle数据库系统回滚段异常处理-ORA-600 4137/4193

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

最初是由于数据库sysaux文件无法正常恢复,重建ctl抛弃sysaux文件,然后打开数据库,但是无法expdp导出数据

Export: Release 12.2.0.1.0 - Production on Wed Jun 24 17:18:04 2026
Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.
Connected to: Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
ORA-31626: job does not exist
ORA-31637: cannot create job SYS_EXPORT_SCHEMA_02 for user SYS
ORA-06512: at "SYS.KUPV$FT", line 1140
ORA-06512: at "SYS.KUPV$FT", line 1741
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 95
ORA-06512: at "SYS.KUPV$FT_INT", line 823
ORA-39080: failed to create queues "KUPC$C_1_20260624171804" and "" for Data Pump job
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 95
ORA-06512: at "SYS.KUPC$QUE_INT", line 1541
ORA-00376: file 3 cannot be read at this time
ORA-06512: at "SYS.DBMS_AQADM", line 742
ORA-06512: at "SYS.DBMS_AQADM_SYS", line 8060
ORA-01110: data file 3: '/u01/app/oracle/product/12.2.0.1/dbhome_2/dbs/MISSING00003'
ORA-06512: at "SYS.DBMS_AQADM_SYSCALLS", line 912
ORA-06512: at "SYS.DBMS_AQADM_SYS", line 8036
ORA-06512: at "SYS.DBMS_AQADM", line 737
ORA-06512: at "SYS.KUPC$QUE_INT", line 1461
ORA-06512: at line 1
ORA-06512: at "SYS.KUPC$QUEUE_INT", line 158
ORA-06512: at "SYS.KUPV$FT_INT", line 758
ORA-06512: at "SYS.KUPV$FT", line 1645
ORA-06512: at "SYS.KUPV$FT", line 1101

然后通过各方人员一顿操作猛如虎,导致数据库启动报ORA-600 4137和ORA-600 4193错误,数据库无法open成功

2026-06-24T18:38:50.158906+08:00
alter database open
2026-06-24T18:38:50.182720+08:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
2026-06-24T18:38:50.219449+08:00
…………
2026-06-24T18:38:50.514016+08:00
ARC3: Archival started
ARCH: STARTING ARCH PROCESSES COMPLETE
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_48840.trc  (incident=304968):
ORA-00600: internal error code, arguments: [4137], [0.77.1546], [0], [0], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_304968/orcl1_ora_48840_i304968.trc
Use ADRCI or Support Workbench to package the incident.
ORACLE Instance orcl1 (pid = 53) - Error 600 encountered while recovering transaction (0, 77).
2026-06-24T18:38:51.313973+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_48840.trc:
ORA-00600: internal error code, arguments: [4137], [0.77.1546], [0], [0], [], [], [], [], [], [], [], []
2026-06-24T18:38:51.649361+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_48840.trc  (incident=304969):
ORA-00600: internal error code, arguments: [4193], [1112], [1122], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_304969/orcl1_ora_48840_i304969.trc
2026-06-24T18:38:53.412782+08:00
opiodr aborting process unknown ospid (48840) as a result of ORA-603

需要open故障库,并且正常导出数据,需要处理两个问题
1. 被抛弃的sysaux文件需要正常online起来,不然expdp无法正常导出用户或者全库数据
2. 需要解决open过程的ORA-600 4137/ORA-600 4193错误
对于sysaux文件进行检查,由于重建ctl没有包含异常的sysaux文件,因此无法直接从库中查询到当前各种文件头相关情况,通过obet直接解析文件头获取相关信息(Oracle数据块编辑工具( Oracle Block Editor Tool)-obet)
res


对于这种情况,可以使用obet的修改文件头checkpoint scn和resetlogs scn功能进行快速修复

OBET> set file 2
filename set to: /u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_sysaux_go991cmw_.dbf (file#2)

OBET> backup
Created backup directory: backup_blk
Successfully backed up block 1 from current file to /tmp/backup_blk/o1_mf_sysaux_go991cmw_.dbf_1.20260624191357

OBET> copy chkscn file 1 to file 2
Error: Edit mode not enabled. Use 'set mode edit' first.

OBET> set mode edit
mode set to: edit

OBET> copy chkscn file 1 to file 2

Confirm Modify chkscn:
Source: file#1 (/u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_system_go990lcg_.dbf)
Target: file#2 (/u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_sysaux_go991cmw_.dbf)
Proceed? (Y/YES to confirm): yes
Successfully copied checkpoint SCN information from file#1 to file#2.

OBET> copy resetlogscn file 1 to file 2

Confirm Modify resetlogscn:
Source: file#1 (/u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_system_go990lcg_.dbf)
Target: file#2 (/u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_sysaux_go991cmw_.dbf)
Proceed? (Y/YES to confirm): yes
Successfully copied resetlog SCN information from file#1 to file#2.

OBET> sum
Check value for File /u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_sysaux_go991cmw_.dbf, Block 1:
current = 0xF21B, required = 0x6651

OBET> sum apply

Confirm applying checksum:
File: /u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_sysaux_go991cmw_.dbf
Block: 1
Offset in block: 16 (file offset: 0x00002010)
Original value: 0xF21B
New value:      0x6651
Confirm? (Y/YES to proceed): y
Verification successful: Stored checksum matches calculated value (0x6651).
Checksum applied successfully.

OBET> tailchk
Check tailchk for File /u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_sysaux_go991cmw_.dbf, Block 1:
current = 0x010B0000, required = 0x010B0000

OBET>

然后重建ctl,包含该sysaux,尝试打开数据库,报ORA-600 4193错误

SYS@ORCL> alter database open ;
alter database open 
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4193], [1112], [1122], [], [], [], [], [], [], [], [], []
Process ID: 93550
Session ID: 1123 Serial number: 55884

进一步跟踪启动过程,确认报错在update undo$上


PARSING IN CURSOR 
#140446136869016
 len=160 dep=1 uid=0 oct=6 lid=0 tim=3161302405543 hv=1292341136 ad='9bbd4828' sqlid='8vyjutx6hg3wh'
update /*+ rule */ undo$ set name=:2,file#=:3,block#=:4,status$=:5,user#=:6,undosqn=:7,
xactsqn=:8,scnbas=:9,scnwrp=:10,inst#=:11,ts#=:12,spare1=:13 where us#=:1
END OF STMT
PARSE 
#140446136869016
:c=11966,e=11918,p=18,cr=94,cu=0,mis=1,r=0,dep=1,og=3,plh=0,tim=3161302405542
BINDS 
#140446136869016
:
 Bind
#0
  oacdty=01 mxl=32(21) mxlc=00 mal=00 scl=00 pre=00
  oacflg=18 fl2=0001 frm=01 csi=852 siz=32 off=0
  kxsbbbfp=9bbdac32  bln=32  avl=21  flg=09
  value="_SYSSMU12_3861134380$"
 Bind
#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fbc23eda370  bln=24  avl=02  flg=05
  value=5
 Bind
#2
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fbc23eda340  bln=24  avl=03  flg=05
  value=144
 Bind
#3
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fbc23eda310  bln=24  avl=02  flg=05
  value=5
 Bind
#4
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fbc23eda2e0  bln=24  avl=02  flg=05
  value=1
 Bind
#5
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fbc23eda2b0  bln=24  avl=04  flg=05
  value=46221
 Bind
#6
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fbc23eda280  bln=24  avl=05  flg=05
  value=30810931
 Bind
#7
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fbc23eda250  bln=24  avl=06  flg=05
  value=3399756014
 Bind
#8
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fbc23eda220  bln=24  avl=03  flg=05
  value=2429
 Bind
#9
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fbc23eda1f0  bln=24  avl=02  flg=05
  value=2
 Bind
#10
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fbc23eda1c0  bln=24  avl=02  flg=05
  value=4
 Bind
#11
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fbc23eda190  bln=24  avl=02  flg=05
  value=2
 Bind
#12
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fbc23eda3a0  bln=22  avl=02  flg=05
  value=12
WAIT 
#140446136869016
: nam='db file sequential read' ela= 16 file#=1 block#=547 blocks=1 obj#=0 tim=3161302406306
2026-06-24T19:59:40.979075+08:00
ORA-00600: internal error code, arguments: [4193], [1112], [1122], [], [], [], [], [], [], [], [], []

alert日志中还有ORA-600 4137等错误

ORACLE Instance orcl1 (pid = 53) - Error 600 encountered while recovering transaction (0, 77).
2026-06-24T19:59:40.387459+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_83245.trc:
ORA-00600: internal error code, arguments: [4137], [0.77.1546], [0], [0], [], [], [], [], [], [], [], []

通过这个报错,可以确认是由于0号回滚段,也就是rollback中事务异常,获取相关的trace

[TOC00003]
----- Beginning of Customized Incident Dump(s) -----
XID passed in = xid: 0x0000.04d.0000060a
XID from Undo block = xid: 0x0000.060.00000600
Dump of buffer cache at level 7 for pdb=0 tsn=0 rdba=4194432
BH (0x3ddfd26b8) file#: 1 rdba: 0x00400080 (1/128) class: 15 ba: 0x3ddb80000
  set: 166 pool: 3 bsz: 8192 bsi: 0 sflg: 2 pwc: 0,0
  dbwrid: 3 obj: -1 objn: 0 tsn: [0/0] afn: 1 hint: f
  hash: [0x3ddece808,0xc6bdc2d8] lru: [0xbc1db108,0xbc1db108]
  ckptq: [NULL] fileq: [NULL]
  objq: [0xa2267bc0,0xa2267bc0] objaq: [0xa2267bb0,0xa2267bb0]
  st: XCURRENT md: NULL fpin: 'ktuwh72: ktugus:ktuswr1' fscn: 0x980cfff669f tch: 1
  flags:
  LRBA: [0x0.0.0] LSCN: [0x0] HSCN: [0x0] HSUB: [65535]
  Printing buffer operation history (latest change first):
  cnt: 10
  01. sid:00 L353:gcur:set:MEXCL      02. sid:00 L145:zib:mk:EXCL       
  03. sid:00 L212:zib:bic:FSQ         04. sid:00 L122:zgb:set:st        
  05. sid:00 L830:olq1:clr:WRT+CKT    06. sid:00 L951:zgb:lnk:objq      
  07. sid:00 L372:zgb:set:MEXCL       08. sid:00 L123:zgb:no:FEN        
  09. sid:00 L083:zgb:ent:fn          10. sid:01 L203:w_ini_dc:bic:FVB  
  buffer tsn: 0 rdba: 0x00400080 (1/128)
  scn: 0x980cffc5958 seq: 0x01 flg: 0x04 tail: 0x59580e01
  frmt: 0x02 chkval: 0x2688 type: 0x0e=KTU UNDO HEADER W/UNLIMITED EXTENTS

基于这样的情况,可以判断通过清理undo$中的相关记录,让其重新分配新的回滚块

Block Header:
block type=0x0e (KTU UNDO HEADER W/UNLIMITED EXTENTS)
block format=0xa2 (oracle 10+)
block rdba=0x00400080 (file#=1, block#=128)
scn=0x0980.cff7c56d, seq=1, tail=0xc56d0e01
block checksum value=0x2683=9859, flag=4
  Extent Control Header
  -------------------------------------------------------------
  Extent Header:: extents: 10  blocks: 79
                  last map: 0x00000000  
#maps
: 0  offset: 4128
      Highwater:: 0x00400225  (rfile#=1,block#=549)
                  ext#: 6  blk#: 5   ext size:8
      
#blocks
 in seg. hdr's freelists: 0
      
#blocks
 below: 0
      mapblk: 0x00000000   offset: 6
      Map Header:: next: 0x00000000   
#extents
: 10  obj#: 0  flag: 0x40000000
  Extent Control Header
  -------------------------------------------------------------
   0x00400081  length: 7
   0x004206a8  length: 8
   0x004206b0  length: 8
   0x00400088  length: 8
   0x00400210  length: 8
   0x00400218  length: 8
   0x00400220  length: 8
   0x00400228  length: 8
   0x004206a0  length: 8
   0x00400230  length: 8
  TRN CTL:: seq: 0x0462 chd: 0x005e ctl: 0x000d inc: 0x00000000 nbf: 0x0000
            mgc: 0x8002 xts: 0x0068 flg: 0x0001 opt: 2147483646(0x7ffffffe)
            uba: 0x00000225.0462.1d scn: 0x0980.cf1f2121
Version: 0x01
  FREE BLOCK POOL::
    uba: 0x00000000.0462.1c ext: 0x6  spc: 0x11a2
    uba: 0x00000000.0462.26 ext: 0x6  spc: 0xc86
    uba: 0x00000000.0462.03 ext: 0x6  spc: 0x1e5c
    uba: 0x00000000.0460.03 ext: 0x4  spc: 0x1e5c
    uba: 0x00000000.043c.21 ext: 0x8  spc: 0xd2c

然后数据库打开成功,使用expdp完美导出数据,完成本次恢复任务

硬件故障后数据文件大小不对故障处理—Oracle碎片扫描恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:硬件故障后数据文件大小不对故障处理—Oracle碎片扫描恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有硬件恢复圈朋友找到我,说硬件恢复之后dbv报dbv-00102错误,让我给看看是否可以处理
dbv-00102


这个是oracle dbv中一种常见错误,一般是由于block 0 不对,或者是由于文件大小不对引起,让把恢复文件发给我,进行检查

SQL> select name,bytes/1024/1024/1024 from v$datafile_header;

NAME                                                                             BYTES/1024/1024/1024
-------------------------------------------------------------------------------- --------------------
H:\BAIDUNETDISK\ORADATA\XXXXORCL\SYSTEM01.DBF                                             2.080078125
H:\BAIDUNETDISK\ORADATA\XXXXORCL\SYSAUX01.DBF                                             2.880859375
H:\BAIDUNETDISK\ORADATA\XXXXORCL\UNDOTBS01.DBF                                           9.0087890625
H:\BAIDUNETDISK\ORADATA\XXXXORCL\USERS01.DBF                                          31.993408203125
H:\BAIDUNETDISK\ORADATA\XXXXORCL\USERS02.DBF                                                8.1640625
H:\BAIDUNETDISK\ORADATA\XXXXORCL\USERS03.DBF                                              7.958984375
H:\BAIDUNETDISK\ORADATA\XXXXORCL\USERS04.DBF                                              7.958984375
H:\BAIDUNETDISK\ORADATA\XXXXORCL\USERS05.DBF                                                 7.890625

已选择8行。

确定USER02-USERS05的dbf文件实际大小(数据文件头记录)在8G左右,但是目前恢复出来的文件大小只有4G左右
4g


在恢复工具中直接查看文件大小情况
rs

这里比较明显rs中虽然显示文件状态良好,但是实际大小也不对(得出经验:以后恢复中不能太依赖这个状态),根据反馈现场是三个盘的raid5,中途做了一次强制上线,然后客户也使用win pe拷贝过一次数据,大小和现在一样,也是少了近4G.第一反应可能是由于raid盘弄的不对,但是经过对其他文件的确认,多完全没有问题,排除了盘错误的问题,怀疑是由于文件系统异常导致,对于这种的情况,文件系统层面肯定无法恢复,考虑使用自研的OraScan工具进行扫描(OraScan(Oracle 碎片扫描工具) 使用说明)
ora1
ora2

通过OraScan扫描找到相关block,并提取出来数据文件
file

使用dbv检测文件

C:\Users\XFF>dbv file=H:\BaiduNetdisk\xff\YFKJORCL.USERS.4.7.4.N.DBF

DBVERIFY: Release 11.2.0.4.0 - Production on 星期日 6月 7 18:06:30 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = H:\BAIDUNETDISK\XFF\YFKJORCL.USERS.4.7.4.N.DBF


DBVERIFY - 验证完成

检查的页总数: 1043200
处理的页总数 (数据): 67167
失败的页总数 (数据): 0
处理的页总数 (索引): 37995
失败的页总数 (索引): 0
处理的页总数 (其他): 861109
处理的总页数 (段)  : 0
失败的总页数 (段)  : 0
空的页总数: 76929
标记为损坏的总页数: 0
流入的页总数: 0
加密的总页数        : 0
最高块 SCN            : 347454063 (0.347454063)

把文件拷贝替换掉之前恢复的USERS02-USER05.DBF,然后尝试打开数据库

SQL> recover database ;
完成介质恢复。
SQL> alter database open ;
alter database open
*
第 1 行出现错误:
ORA-03113: 通信通道的文件结尾
进程 ID: 3308
会话 ID: 14 序列号: 3

查看alert日志分析原因

Sun Jun 07 14:43:51 2026
Recovery of Online Redo Log: Thread 1 Group 2 Seq 36464 Reading mem 0
  Mem# 0: H:\BAIDUNETDISK\ORADATA\YFKJORCL\REDO02.LOG
Completed: ALTER DATABASE RECOVER  database   
alter database open 
Beginning crash recovery of 1 threads
 parallel recovery started with 19 processes
Started redo scan
Completed redo scan
 read 2353 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 36464, block 15876
Recovery of Online Redo Log: Thread 1 Group 2 Seq 36464 Reading mem 0
  Mem# 0: H:\BAIDUNETDISK\ORADATA\YFKJORCL\REDO02.LOG
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 36464, block 20582, scn 347475303
 0 data blocks read, 0 data blocks written, 2353 redo k-bytes read
Sun Jun 07 14:43:57 2026
Errors in file c:\app\xff\diag\rdbms\yfkjorcl\o11201\trace\o11201_lgwr_2204.trc:
ORA-00314: ?? 3 (???? 1) ??? sequence# 36462 ? 32025 ???
ORA-00312: ???? 3 ?? 1: 'H:\BAIDUNETDISK\ORADATA\YFKJORCL\REDO03.LOG'
Errors in file c:\app\xff\diag\rdbms\yfkjorcl\o11201\trace\o11201_lgwr_2204.trc:
ORA-00314: ?? 3 (???? 1) ??? sequence# 36462 ? 32025 ???
ORA-00312: ???? 3 ?? 1: 'H:\BAIDUNETDISK\ORADATA\YFKJORCL\REDO03.LOG'
Errors in file c:\app\xff\diag\rdbms\yfkjorcl\o11201\trace\o11201_ora_3308.trc:
ORA-00314: 日志 1 (用于线程 ) 要求的 sequence#  与  不匹配
ORA-00312: 联机日志 3 线程 1: 'H:\BAIDUNETDISK\ORADATA\YFKJORCL\REDO03.LOG'
USER (ospid: 3308): terminating the instance due to error 314
Sun Jun 07 14:44:02 2026
Instance terminated by USER, pid = 3308

由于redo group 异常导致库无法正常open,但是由于已经recover database成功,因此大概率可以clear该redo 组

SQL> select group#,status from v$log;

    GROUP# STATUS
---------- ----------------
         1 INACTIVE
         3 INACTIVE
         2 CURRENT

SQL> alter database clear logfile group 3;

数据库已更改。

SQL> alter database open;

数据库已更改。

数据库open成功,然后使用expdp导出数据,完成本次恢复任务.

记录一次win删除数据文件完美恢复案例

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:记录一次win删除数据文件完美恢复案例

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户在操作系统层面删除了四个数据文件(数据库在关闭情况下直接删除物理文件),然后offline,启动数据库,结果发现被删除的数据文件是业务表空间的,导致大量业务访问报错
ORA-00376


通过数据库层面查询确认这些文件已经不存在
121

对于这样的情况,先尝试从文件系统层面尝试反删除恢复文件
0

由于删除文件之后,启动了数据库并写入了一些日志和数据导致被删除的文件的元数据信息彻底丢失,这种情况下基于文件系统的反删除恢复肯定无法需要的数据,对于这种情况,考虑进行碎片层面恢复,以前有过类似案例:
win系统删除oracle数据文件恢复
Oracle 数据文件大小为0kb或者文件丢失恢复
删除数据库文件并部分覆盖情况下Oracle恢复
通过底层扫描发现需要恢复的文件头部信息丢失,其他block完整(该文件本身不大,只有100M),扫描出来的数据块类似这样的结果
413
414

这里比较幸运,丢失的block为1-3,也就是涉及文件头和一些位图信息,通过以前自研的OraFHR工具(Oracle数据库被勒索加密一键open工具–OraFHR)进行重构文件头,再通过oracle recovery tools工具进行文件头的一些修复工作
orarec

再重建控制文件打开数据库,并完美导出数据,实现数据0丢失恢复
expdp

文件系统损坏导致数据库异常故障处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:文件系统损坏导致数据库异常故障处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户做了双机rose,由于某种故障导致共享存储在两个主机之间相互频繁挂载(甚至出现了同时挂载的情况),使得该文件系统发生损坏
ntfs


修复双机故障之后,数据库启动ORA-01122 ORA-01110 ORA-01200错误
ora-1200

初步看这个报错,block差距有点大,文件头中记录为419840个block,现在实际有的block数量为384000,使用obet查看文件头记录block number情况

OBET> p kcvfh.kcvfhhdr
File: E:\TEMP\20260219\SYSTEM01.DBF
Size: 8192 bytes
Block: 1
Offset: 20

struct kcvfhhdr, 76 bytes                   @20
   ub4 kccfhswv                            @20      0x00000000
   ub4 kccfhcvn                            @24      0x0B200400
   ub4 kccfhdbi                            @28      0x85D98FAB
   text kccfhdbn[8]                        @32-39   XXXX
   ub4 kccfhcsq                            @40      0x00091079
   ub4 kccfhfsz                            @44      0x00066800 <<--16转换为10禁止为419840
   s_blkz kccfhbsz                         @48      0x00
   ub2 kccfhfno                            @52      0x0001
   ub2 kccfhtyp                            @54      0x0003
   ub4 kccfhacid                           @56      0x00000000
   ub4 kccfhcks                            @60      0x00000000
   text kccfhtag[32]                       @64-95

<kcvfh.kcvfhhdr structure printed successfully>

对于这种情况,以前有过很多次处理经验(一般办法2个:1>修改文件头的block数量记录;2>修改现在的文件大小和实际文件有匹配),以前类似的处理记录:
bbed处理ORA-01200故障
记录一次ORA-01200完美恢复
ORA-01122 ORA-01200故障处理
处理完成system文件异常之后,sysaux文件继续异常

SQL> recover datafile 1;
完成介质恢复。
SQL> recover datafile 2;
ORA-00283: 恢复会话因错误而取消
ORA-01110: 数据文件 2: 'Z:\APP\ADMINISTRATOR\ORADATA\XXXX\SYSAUX01.DBF'
ORA-01122: 数据库文件 2 验证失败
ORA-01110: 数据文件 2: 'Z:\APP\ADMINISTRATOR\ORADATA\XXXX\SYSAUX01.DBF'
ORA-01200: 149760 的实际文件大小小于 153600 块的正确大小

类似处理该故障之后,由于文件系统故障导致不少文件出现大量连续坏块(全0或者记录了其他文件内容的坏块),这种是由于文件系统元数据异常导致,通过文件系统层面恢复继续无法正常处理,对于这样的情况,通过碎片扫描工具按照oracle block级别的文件重组(其实就是基于rdba信息进行重组),获取正确的数据块信息然后重新重组成数据文件
QQ20260220-005159


然后打开数据库,顺利导出数据,实现客户数据最大限度恢复

在生产环境错误执行dd命令破坏asm磁盘故障恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:在生产环境错误执行dd命令破坏asm磁盘故障恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

由于ssh登录错误,客户对生产环境进行了误操作把系统的一块磁盘dd到另外两个磁盘上,由于及时发现立马进行了终止操作,但是还是分别破坏了一点数据(一块盘破坏了2G多,另外一块盘破坏了1G多)
dd


通过分析udev的绑定关系确认被破坏的asm disk名称
udev

再通过asm alert日志确认破坏磁盘在asm disk中情况
asmdisk

通过上述信息基本上可以确认,asmdisk13被分别dd到了asmdisk11和asmdisk26中了部分11
26

基于这种情况,由于asm disk被破坏了1-2G多,直接修复然后正常mount磁盘组基本上没有希望,经过分析以及与客户沟通,确认他们改系统是4节点组成的集群,1/2节点上面跑2套库,3/4节点上跑2套库,数据整体放在data_dg磁盘组中,需要恢复的库是第二个顺序创建的1套库(4套库中只需恢复一套即可),由于破坏的数据本身不多,而且需要恢复的数据不是最初写入asm磁盘组,基于这样的情况,需要恢复的数据库机会比较大.

由于现在三个磁盘头信息一致(一个磁盘被dd到另外两个磁盘上),因此第一步先把损坏的两个磁盘头进行简单修复,为了便于amdu(找回ASM中数据文件)等之类数据可以识别到正确的磁盘头信息,然后进行后续的数据文件提取恢复.使用工具对数据文件进行了批量提取,提取数据完成之后,尝试recover和open库
open

虽然数据库正常打开了,不过很不幸,后台还是有一些坏块报错,通过dbv检查发现有文件有一部分坏块,类似dbv报错信息
1_1

通过分析该文件在磁盘组中各个磁盘的分布情况
map

确认该文件确实有部分block分布在被dd磁盘的破坏的范围内这个部分的数据丢失无法挽回,只能是定位到具体对象然后由业务想办法处理.相对以往的各种dd破坏的案例恢复而言,这个应该是效果比较好的一个,而且也是恢复比较容易的一个,没有使用到asm disk 基于asm au/oracle block 扫描的级别,而且system表空间没有任何损坏,数据库甚至直接open成功了,以往的一些dd案例列举:
asm磁盘dd破坏恢复
dd破坏asm磁盘头恢复
asm disk 磁盘部分被清空恢复

expdp dmp 导出不完整导入ORA-39059 ORA-39246 故障抢救数据

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:expdp dmp 导出不完整导入ORA-39059 ORA-39246 故障抢救数据

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户一套nc系统,由于安装时候把库建在了比较小的分区上,运行一些时间之后,出现空间不足,现场技术人员对oracle不太熟悉,经过一系列操作(删除业务表空间,复制pdb,创建表空间等等操作),无法恢复数据库,准备使用备份的dmp进行还原,结果分析发现仅保留的最后一份dmp,是一份导出不完全的dmp文件,无法正常导入(以前处理过一个类似case:ORA-39773: parse of metadata stream failed故障处理,尝试导入报ORA-39246错:

C:\Users\XFF>impdp system/oracle@127.0.0.1/orapdb directory=expdp_dir dumpfile=xxxxx_2025-12-01_0230.dmp logfile=1.log

Import: Release 19.0.0.0.0 - Production on 星期三 12月 3 21:00:19 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

连接到: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
ORA-39002: 操作无效
ORA-39059: 转储文件集不完整
ORA-39246: 无法在提供的转储文件中定位主表

分析当时当初的dmp日志,由于expdp的job表所在表空间不足导致expdp导出失败
dmp1


TABLE:"XIFENFEI"."EOM_MEASURE_POINT"
ORA-30032: 挂起的 (可恢复) 语句已超时
ORA-01691: Lob 段 XIFENFEI.SYS_LOB0000161267C00111$$ 无法通过 32 (在表空间 NNC_DATA01 中) 扩展
ORA-06512: 在 "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: 在 "SYS.KUPW$WORKER", line 12620
ORA-06512: 在 "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: 在 "SYS.KUPW$WORKER", line 11414
----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0xda5dae50     33476  package body SYS.KUPW$WORKER.WRITE_ERROR_INFORMATION
0xda5dae50     12641  package body SYS.KUPW$WORKER.DETERMINE_FATAL_ERROR
0xda5dae50     11602  package body SYS.KUPW$WORKER.CREATE_OBJECT_ROWS
0xda5dae50     15268  package body SYS.KUPW$WORKER.FETCH_XML_OBJECTS
0xda5dae50      3907  package body SYS.KUPW$WORKER.UNLOAD_METADATA
0xda5dae50     13736  package body SYS.KUPW$WORKER.DISPATCH_WORK_ITEMS
0xda5dae50      2429  package body SYS.KUPW$WORKER.MAIN
0x6524a4f0         2  anonymous block
KUPW: Object row index into parse items is: 1
KUPW: Parse item count is: 19
KUPW: In function CHECK_FOR_REMAP_NETWORK
KUPW: Nothing to remap
KUPW: In procedure BUILD_OBJECT_STRINGS - non-base info
KUPW: In procedure BUILD_SUBNAME_LIST with TABLE:XIFENFEI.EOM_MEASURE_POINT
KUPW: In function NEXT_PO_NUMBER
KUPW: PO number assigned: 34198
FORALL
KUPW: In procedure DETERMINE_FATAL_ERROR with ORA-30032: 挂起的 (可恢复) 语句已超时
ORA-01691: Lob 段 XIFENFEI.SYS_LOB0000161267C00111$$ 无法通过 32 (在表空间 NNC_DATA01 中) 扩展
作业 "XIFENFEI"."SYS_EXPORT_SCHEMA_01" 因致命错误于 星期一 12月 1 06:33:21 2025 elapsed 0 04:03:18 停止

从导出日志看,在导出大量”0 KB 0 行”记录之后提示表空间不足,expdp的job表无法扩展导致导出挂起然后超时导出终止(这个导出操作没有完全完成),从而在导入的时候出现了ORA-39059: 转储文件集不完整 ORA-39246: 无法在提供的转储文件中定位主表 的错误.对于这种故障,分析导出日志,发现运气不错,所有有数据的表都导出完成,基于这个心中就有了第一层底气,所有表数据不会丢失(因为都导出到了这个dmp中),但是非表的字典数据不完整,要想业务完整跑起来,需要找到一个完整的业务字典信息.对于大量的备份dmp被删除,然后对应分区还写入了很多数据,只能尝试看运气,通过对磁盘文件镜像,然后进行反删除恢复,找出来一个11月26日的dmp的压缩文件是完整的
good-dmp


通过这个dmp导入业务字典信息,然后再利用expdp dmp解析工具(expdp dmp被加密破坏恢复)把所有表数据出来,经过这两者组合,顺利完成数恢复,可以测试业务完全正常

system表空间丢失部分文件恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:system表空间丢失部分文件恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户因为system表空间有一个数据文件放在其他位置,当时没有正常拷贝出来(备份了oradata路径下面文件,遗漏了一个system文件),尝试启动库报ORA-01157 ORA-01147等错误

[oracle@xifenfei check_db]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sun Oct 5 21:13:28 2025

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> recover datafile 1;
Media recovery complete. 
SQL> recover datafile 2,3,4,5,6,7,8,9,10;   
Media recovery complete.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01157: cannot identify/lock data file 11 - see DBWR trace file
ORA-01110: data file 11:
'/u01/app/oracle/product/11.2.0.4/db_1/dbs/path_to_datafile.dbf'

SQL> alter database datafile 11 offline drop;

Database altered.

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01147: SYSTEM tablespace file 11 is offline
ORA-01110: data file 11:
'/u01/app/oracle/product/11.2.0.4/db_1/dbs/path_to_datafile.dbf'

alert日志报错信息

Sun Oct 05 22:35:01 2025
alter database open
Sun Oct 05 22:35:01 2025
Errors in file /data/app/oracle/diag/rdbms/mtxdb1/mtxdb1/trace/mtxdb1_dbw0_5946.trc:
ORA-01157: cannot identify/lock data file 11 - see DBWR trace file
ORA-01110: data file 11: '/u01/app/oracle/product/11.2.0.4/db_1/dbs/path_to_datafile.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /data/app/oracle/diag/rdbms/mtxdb1/mtxdb1/trace/mtxdb1_ora_11264.trc:
ORA-01157: cannot identify/lock data file 11 - see DBWR trace file
ORA-01110: data file 11: '/u01/app/oracle/product/11.2.0.4/db_1/dbs/path_to_datafile.dbf'
ORA-1157 signalled during: alter database open...
Sun Oct 05 22:35:25 2025
alter database datafile 11 offline 
ORA-1145 signalled during: alter database datafile 11 offline ...
alter database datafile 11 offline drop
Completed: alter database datafile 11 offline drop
alter database open
Errors in file /data/app/oracle/diag/rdbms/mtxdb1/mtxdb1/trace/mtxdb1_ora_11264.trc:
ORA-01147: SYSTEM tablespace file 11 is offline
ORA-01110: data file 11: '/u01/app/oracle/product/11.2.0.4/db_1/dbs/path_to_datafile.dbf'
ORA-1147 signalled during: alter database open...

由于11号文件是system表空间的一个数据文件,对于这种数据文件丢失无法offline该数据文件,然后open库(也就是说在open库的时候,system表空间的数据文件必须全部online,如果有部分文件offline就会报ORA-01147).对于这样的情况,以前有过类似恢复经历:bbed打开丢失部分system数据文件库,这次的编写了一个m_scn程序实现快速处理

[oracle@xifenfei  tmp]$ cat 1.txt
1@/data/app/oracle/oradata/mtxdb1/system01.dbf
11@/tmp/11.dbf
[oracle@xifenfei  tmp]$ ./m_scn 1.txt

-------------Is processing datafile:/tmp/11.dbf-------------
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.000835728 s, 1.3 GB/s

[oracle@xifenfei tmp]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed Oct 8 11:27:32 2025

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> set numw 16
SQL> col CHECKPOINT_TIME for a40
SQL> set lines 150
SQL> set pages 1000
SQL> SELECT status,
  2  to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,FUZZY,checkpoint_change#,
  3  count(*) ROW_NUM
  4  FROM v$datafile_header
  5  GROUP BY status, checkpoint_change#, to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss'),fuzzy
  6  ORDER BY status, checkpoint_change#, checkpoint_time;

STATUS  CHECKPOINT_TIME                          FUZ CHECKPOINT_CHANGE#          ROW_NUM
------- ---------------------------------------- --- ------------------ ----------------
OFFLINE 2025-10-02 06:50:06                      NO      17328662858685                1
ONLINE  2025-10-02 06:50:06                      NO      17328662858685               10


SQL> alter database datafile 11 online;

Database altered.

然后重建ctl,并尝试打开库
ctl_re


然后查询11号文件中涉及的对象情况

SQL> select distinct owner,segment_name,segment_type from dba_extents where file_id=11;

OWNER                          SEGMENT_NAME                           SEGMENT_TYPE
------------------------------ -------------------------------------- ------------------
SYS                            SYSTEM                                 ROLLBACK
SYS                            I_COL1                                 INDEX
SYS                            AUD$                                   TABLE

SQL> select owner,segment_name from dba_segments where HEADER_FILE=11;

no rows selected

证明丢失的11号文件(system表空间文件),涉及的对象较少,而且不涉及核心字典,比如tab$,obj$,col$等非常核心对象,评估理论上应该不涉业务数据丢失,尝试直接expdp导出数据,但是很不幸,报ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018]错误

. . exported "XFF020"."OTHERBILLDETAIL_DEL"              6.405 MB  126048 rows
. . exported "XFF020"."POSSOLDOUT"                       7.784 MB  281413 rows
ORA-31693: Table data object "XFF020"."MATERIELTRAN" failed to load/unload and is being skipped due to error:
ORA-39068: invalid master table data in row with PROCESS_ORDER=159:1000001
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPF$FILE", line 3720
ORA-06512: at line 1
ORA-39126: Worker unexpected fatal error in KUPW$WORKER.UNLOAD_DATA [TABLE_DATA:"XFF020"."MATERIELTRAN"] 
UPDATE "SYS"."SYS_EXPORT_FULL_01" SET processing_state = :1, processing_status = :2
    WHERE process_order = :3 AND duplicate = 0
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPW$WORKER", line 7866
ORA-31693: Table data object "XFF020"."MATERIELTRAN" failed to load/unload and is being skipped due to error:
ORA-39068: invalid master table data in row with PROCESS_ORDER=159:1000001
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPF$FILE", line 3720
ORA-06512: at line 1

ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.KUPW$WORKER", line 9721

----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0xef2fc508     21979  package body SYS.KUPW$WORKER
0xef2fc508      9742  package body SYS.KUPW$WORKER
0xef2fc508      3437  package body SYS.KUPW$WORKER
0xef2fc508     10436  package body SYS.KUPW$WORKER
0xef2fc508      1824  package body SYS.KUPW$WORKER
0xef2feb20         2  anonymous block

ORA-39097: Data Pump job encountered unexpected error -607
ORA-39065: unexpected master process exception in DISPATCH
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []

ORA-31693: Table data object "XFF020"."ANALYSEREPORT" failed to load/unload and is being skipped due to error:
ORA-39068: invalid master table data in row with PROCESS_ORDER=161:1000001
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPF$FILE", line 3720
ORA-06512: at line 1
ORA-39126: Worker unexpected fatal error in KUPW$WORKER.UNLOAD_DATA [TABLE_DATA:"XFF020"."ANALYSEREPORT"] 
UPDATE "SYS"."SYS_EXPORT_FULL_01" SET processing_state = :1, processing_status = :2
   WHERE process_order = :3 AND duplicate = 0
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPW$WORKER", line 7866
ORA-31693: Table data object "XFF020"."ANALYSEREPORT" failed to load/unload and is being skipped due to error:
ORA-39068: invalid master table data in row with PROCESS_ORDER=161:1000001
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPF$FILE", line 3720
ORA-06512: at line 1

ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.KUPW$WORKER", line 9721

----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0xef2fc508     21979  package body SYS.KUPW$WORKER
0xef2fc508      9742  package body SYS.KUPW$WORKER
0xef2fc508      3437  package body SYS.KUPW$WORKER
0xef2fc508     10436  package body SYS.KUPW$WORKER
0xef2fc508      1824  package body SYS.KUPW$WORKER
0xef2feb20         2  anonymous block

ORA-31693: Table data object "XFF020CW"."MATERIELTRAN" failed to load/unload and is being skipped due to error:
ORA-39068: invalid master table data in row with PROCESS_ORDER=160:1000001
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPF$FILE", line 3720
ORA-06512: at line 1
ORA-39126: Worker unexpected fatal error in KUPW$WORKER.UNLOAD_DATA [TABLE_DATA:"XFF020CW"."MATERIELTRAN"] 
UPDATE "SYS"."SYS_EXPORT_FULL_01" SET processing_state = :1, processing_status = :2
   WHERE process_order = :3 AND duplicate = 0
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPW$WORKER", line 7866
ORA-31693: Table data object "XFF020CW"."MATERIELTRAN" failed to load/unload and is being skipped due to error:
ORA-39068: invalid master table data in row with PROCESS_ORDER=160:1000001
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPF$FILE", line 3720
ORA-06512: at line 1

ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.KUPW$WORKER", line 9721

----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0xef2fc508     21979  package body SYS.KUPW$WORKER
0xef2fc508      9742  package body SYS.KUPW$WORKER
0xef2fc508      3437  package body SYS.KUPW$WORKER
0xef2fc508     10436  package body SYS.KUPW$WORKER
0xef2fc508      1824  package body SYS.KUPW$WORKER
0xef2feb20         2  anonymous block

Job "SYS"."SYS_EXPORT_FULL_01" stopped due to fatal error at Wed Oct 8 11:59:29 2025 elapsed 0 00:18:48

对ORA-600 kdBlkCheckError进行分析分析(11表示文件号,3表示block),是由于导出生成的master表写入在system表空间,而system表空间中的file# 11是人工构造出来的,block 3 是位图分配信息(该信息和实际字典中存储信息不匹配),所以导致出现该错误,对于这个问题解决方法为expdp写master表不在system表空间即可,通过该操作,顺利导出数据,完成本次恢复任务
expdp_ok


ORA-704 ORA-604 ORA-1426故障分析处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-704 ORA-604 ORA-1426故障分析处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

服务器异常断电,通过分析alert日志发现现场的一些操作,数据库启动最初报ORA-00322 ORA-00312错

Tue Sep 23 20:06:52 2025
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_m000_6056.trc:
ORA-00322: log 1 of thread 1 is not current copy
ORA-00312: online log 1 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG'
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_m000_6056.trc:
ORA-00322: log 2 of thread 1 is not current copy
ORA-00312: online log 2 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG'
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_m000_6056.trc:
ORA-00322: log 3 of thread 1 is not current copy
ORA-00312: online log 3 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG'

通过隐含参数强制拉库报ORA-00704 ORA-00604 ORA-01426错误

Tue Sep 23 23:53:52 2025
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 444541390
Resetting resetlogs activation ID 1705450279 (0x65a71b27)
Online log D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG: Thread 1 Group 3 was previously cleared
Tue Sep 23 23:53:53 2025
Setting recovery target incarnation to 6
Tue Sep 23 23:53:53 2025
Assigning activation ID 1740400222 (0x67bc665e)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Sep 23 23:53:53 2025
SMON: enabling cache recovery
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_7792.trc:
ORA-00704: 引导程序进程失败
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-01426: 数字溢出
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_7792.trc:
ORA-00704: 引导程序进程失败
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-01426: 数字溢出
Error 704 happened during db open, shutting down database
USER (ospid: 7792): terminating the instance due to error 704
Instance terminated by USER, pid = 7792
ORA-1092 signalled during: alter database open resetlogs...

我接手故障之后尝试启动库,依旧报ORA-01092 ORA-00704 ORA-00604 ORA-01426错误无法启动库

C:\Users\XFF>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on 星期四 9月 25 22:50:21 2025

Copyright (c) 1982, 2010, Oracle.  All rights reserved.


连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options


SQL> recover database;
完成介质恢复。
SQL> alter database open ;
alter database open
*
第 1 行出现错误:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01426: numeric overflow
进程 ID: 18152
会话 ID: 14 序列号: 1

这个报错是number数据溢出,那就可能是由于number类型的数据值不对,通过对启动过程跟踪

PARSING IN CURSOR #5 len=74 dep=1 uid=0 oct=3 lid=0 tim=1677733680660 hv=3309402135 
    ad='7fffeef07300' sqlid='5n1fs4m2n2y0r'
select pos#,intcol#,col#,spare1,bo#,spare2,spare3 from icol$ where obj#=:1
END OF STMT
BINDS #5:
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=1c1d70b8  bln=22  avl=03  flg=05
  value=425
EXEC #5:c=0,e=69,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=299250003,tim=1677733680725
WAIT #5: nam='db file sequential read' ela= 152 file#=1 block#=386 blocks=1 obj#=42 tim=1677733680892
FETCH #5:c=0,e=178,p=1,cr=3,cu=0,mis=0,r=1,dep=1,og=4,plh=299250003,tim=1677733680913
FETCH #5:c=0,e=3,p=0,cr=2,cu=0,mis=0,r=1,dep=1,og=4,plh=299250003,tim=1677733680931
FETCH #5:c=0,e=2,p=0,cr=1,cu=0,mis=0,r=0,dep=1,og=4,plh=299250003,tim=1677733680943
CLOSE #5:c=0,e=1,dep=1,type=3,tim=1677733680957

*** 2025-09-25 21:29:42.634
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=10, mask=0x0)
----- Error Stack Dump -----
ORA-01426: 数字溢出
----- Current SQL Statement for this session (sql_id=bkdusjx00dsmc) -----
select i.obj#,i.ts#,i.file#,i.block#,i.intcols,i.type#,i.flags, i.property,i.pctfree$,i.initrans,
i.maxtrans,i.blevel,i.leafcnt,i.distkey, i.lblkkey,i.dblkkey,i.clufac,i.cols,i.analyzetime,i.samplesize,
i.dataobj#, nvl(i.degree,1),nvl(i.instances,1),i.rowcnt,mod(i.pctthres$,256),i.indmethod#,
i.trunccnt,nvl(c.unicols,0),nvl(c.deferrable#+c.valid#,0), nvl(i.spare1,i.intcols),i.spare4,
spare2,spare6, decode(i.pctthres$,null,null, mod(trunc(i.pctthres$/256),256)) 
from ind$ i, (select enabled, min(cols) unicols, min(to_number(bitand(defer,1))) 
deferrable#, min(to_number(bitand(defer,4))) valid# from cdef$ where obj#=:1 and enabled > 1 group by enabled)
 c where i.obj#=c.enabled(+) and i.bo#=:1 order by i.obj#

通过上述可以是在执行上述sql的时候,遭遇异常,进一步查看trace,发现在该block中有异常信息

Block header dump:  0x004000a5
 Object id on Block? Y
 seg/obj: 0x2  csc: 0x00.10daa102  itc: 2  flg: -  typ: 1 - DATA
     fsl: 0  fnx: 0x0 ver: 0x01
 
 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x0006.01b.00000009  0x00c001b2.0003.09  C---    0  scn 0x0000.000021b5
0x02   0x000a.010.0003827e  0x00c00349.35ea.2f  --U-    2  fsc 0x0000.10daa11d
bdba: 0x004000a5

这个部分可以确认,异常对象是dataobj#为2(c_obj#的cluster),rdba为0x004000a5(file 1 block 165),对应的block dump中有以下信息异常

tab 3, row 4, @0x19a6
tl: 366 fb: -CH-FL-- lb: 0x0  cc: 28 cki: 4
col  0: [ 3]  c2 05 1b
col  1: [ 3]  c2 1d 1b
col  2: [44]
 80 02 c1 02 03 c2 1d 59 01 80 02 c1 03 02 c1 0b 02 c1 03 03 c2 03 38 ff 02
 c1 02 03 c2 15 33 01 80 02 c1 02 03 c2 02 3d 04 c3 04 62
col  3: [35]
 02 c1 02 02 c1 02 03 c2 17 10 07 78 7c 07 1b 0f 04 24 04 c3 04 62 23 04 c3
 04 62 23 02 c1 03 ff ff ff 02
col  4: [193]
 03 6c 00 1c 04 03 c2 05 1a 03 c2 05 1a 01 80 02 c1 02 03 c2 1d 51 01 80 02
 c1 03 02 c1 0b 02 c1 03 03 c2 03 38 ff 02 c1 02 03 c2 15 33 01 80 02 c1 02
 03 c2 02 3d 04 c3 04 5c 3d 02 c1 02 02 c1 02 03 c2 17 13 07 78 7c 07 1b 0f
 04 24 04 c3 04 62 23 04 c3 04 62 23 02 c1 03 ff ff ff 02 c1 03 6c 00 1c 02
 03 c2 05 16 03 c2 05 16 01 80 02 c1 02 03 c2 1d 11 01 80 02 c1 03 02 c1 0b
 02 c1 03 03 c2 03 38 ff 02 c1 04 02 c1 03 02 c1 02 02 c1 02 02 c1 0f 03 c2
 19 3b 02 c1 02 02 c1 02 03 c2 19 3b 07 78 7c 07 1b 17 04 23 03 c2 19 3b 03
 c2 19 3b 02 c1 03 ff ff ff 02 c1 03 6c 00 1c 01 03 c2
col  5: [ 5]  14 03 c2 05 14
col  6: [ 1]  80
col  7: [ 2]  c1 02
col  8: [ 3]  c2 1a 35
col  9: [ 1]  80
col 10: [ 2]  c1 03
col 11: [ 2]  c1 0b
col 12: [ 2]  c1 03
col 13: [ 3]  c2 03 38
col 14: *NULL*
col 15: [ 2]  c1 02
col 16: [ 3]  c2 15 33
col 17: [ 2]  c1 02
col 18: [ 2]  c1 02
col 19: [ 2]  c1 05
col 20: [ 3]  c2 08 4c
col 21: [ 2]  c1 02
col 22: [ 2]  c1 02
col 23: [ 2]  c1 05
col 24: [ 7]  78 6e 03 1e 0b 12 32
col 25: [ 3]  c2 08 4c
col 26: [ 3]  c2 08 4c
col 27: [ 2]  c1 03

这个里面显示是c_obj#这个簇中的第三个表第四行记录,通过查询正常库,确认第三个表是什么对象

SQL> select obj# from tab$ where dataobj#=2 and tab#=3;

      OBJ#
----------
        19

SQL> select name from obj$ where obj#=19;

NAME
------------------------------
IND$

通过上述可以去人tab 3为ind$对象,进一步分析ind$的表结构

SQL> desc ind$
 名称                                      是否为空? 类型
 ----------------------------------------- -------- ----------------------------
 OBJ#                                      NOT NULL NUMBER
 DATAOBJ#                                           NUMBER
 TS#                                       NOT NULL NUMBER
 FILE#                                     NOT NULL NUMBER
 BLOCK#                                    NOT NULL NUMBER
 BO#                                       NOT NULL NUMBER
 INDMETHOD#                                NOT NULL NUMBER
 COLS                                      NOT NULL NUMBER
 PCTFREE$                                  NOT NULL NUMBER
 INITRANS                                  NOT NULL NUMBER
 MAXTRANS                                  NOT NULL NUMBER
 PCTTHRES$                                          NUMBER
 TYPE#                                     NOT NULL NUMBER
 FLAGS                                     NOT NULL NUMBER
 PROPERTY                                  NOT NULL NUMBER
 BLEVEL                                             NUMBER
 LEAFCNT                                            NUMBER
 DISTKEY                                            NUMBER
 LBLKKEY                                            NUMBER
 DBLKKEY                                            NUMBER
 CLUFAC                                             NUMBER
 ANALYZETIME                                        DATE
 SAMPLESIZE                                         NUMBER
 ROWCNT                                             NUMBER
 INTCOLS                                   NOT NULL NUMBER
 DEGREE                                             NUMBER
 INSTANCES                                          NUMBER
 TRUNCCNT                                           NUMBER
 SPARE1                                             NUMBER
 SPARE2                                             NUMBER
 SPARE3                                             NUMBER
 SPARE4                                             VARCHAR2(1000)
 SPARE5                                             VARCHAR2(1000)
 SPARE6                                             DATE

对dump出来的block记录进行转换为实际值

SQL> select utl_raw.cast_to_number('c2051b') value from dual;

     VALUE
----------
       426

SQL> select utl_raw.cast_to_number('c21d1b') value from dual;

     VALUE
----------
      2826

SQL>  select utl_raw.cast_to_number('02c10202c10203c2171007787c071b0f042404c304622304c304622302c103ffffff02')
  2  value from dual;
 select utl_raw.cast_to_number('02c10202c10203c2171007787c071b0f042404c304622304c304622302c103ffffff02')
        *
第 1 行出现错误:
ORA-06502: PL/SQL: 数字或值错误
ORA-06512: 在 "SYS.UTL_RAW", line 388

通过上述分析证明,在ind$表的obj#为426行的记录的第三列无法转换为正常number记录,证明该值异常,从而导致数据库在执行sql_id=bkdusjx00dsmc这个sql的时候报错,从而无法open库,dbv对system文件进行检测发现有一些逻辑层面损坏

C:\Users\XFF>dbv file=H:\BaiduNetdisk\orcl\SYSTEM01.DBF

DBVERIFY: Release 11.2.0.4.0 - Production on 星期四 9月 25 19:17:17 2025

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = H:\BAIDUNETDISK\ORCL\SYSTEM01.DBF
Block Checking: DBA = 4194469, Block Type = KTB-managed data block
data header at 0x5c2425c
kdbchk: row does not end within block
        table=1  slot=0
        len=923  offset=7757  dtl=8096 aclen=923
页 165 失败, 校验代码为 6103
itl[2] has higher commit scn(0x0000.000ac920) than block scn (0x0000.000a4a37)
页 21478 失败, 校验代码为 6056
Block Checking: DBA = 4263330, Block Type = KTB-managed data block
**** row 100: row length 65627 past end of block
**** row 100: row skipped so other stats may be wrong
**** row 102: key out of order
**** row 102: lock value 4 is larger than maximum itl 2
**** row 102: bad flag value 193
**** row 105: key out of order
**** row 105: lock value 4 is larger than maximum itl 2
**** row 105: bad flag value 193
**** row 106: row length 65537 past end of block
**** row 106: row skipped so other stats may be wrong
**** row 108: row length 65627 past end of block
**** row 108: row skipped so other stats may be wrong
**** row 109: key out of order
**** row 109: lock value 4 is larger than maximum itl 2
**** row 109: bad flag value 193
**** row 110: key out of order
**** row 110: lock value 239 is larger than maximum itl 2
**** row 112: row length 65627 past end of block
**** row 112: row skipped so other stats may be wrong
**** row 113: key out of order
**** row 113: lock value 4 is larger than maximum itl 2
**** row 113: bad flag value 193
**** row 114: row length 65537 past end of block
**** row 114: row skipped so other stats may be wrong
**** row 116: key out of order
**** row 116: lock value 4 is larger than maximum itl 2
**** row 116: bad flag value 193
**** key (begin=0x1a14, len=17) overlaps with another
        begin = 0x1a1c len = 11
---- end index block validation
页 69026 失败, 校验代码为 6401
Block Checking: DBA = 4263331, Block Type = KTB-managed data block
**** row 121: row offset 3123 out of valid range
**** row 122: row offset 3106 out of valid range
**** row 123: lock value 214 is larger than maximum itl 2
**** row 123: bad flag value 128
**** row 124: row length 65537 past end of block
**** row 124: row skipped so other stats may be wrong
**** row 125: key out of order
**** row 129: row offset 2979 out of valid range
**** row 130: row offset 2962 out of valid range
**** row 131: row offset 3225 out of valid range
**** row 135: committed with rsl and/or ras flag
**** row 136: key out of order
**** row 136: lock value 100 is larger than maximum itl 2
**** row 137: lock value 193 is larger than maximum itl 2
**** row 141: row length 65537 past end of block
**** row 141: row skipped so other stats may be wrong
**** row 142: rsl is 0 with ras flag
**** row 143: key out of order
**** row 143: lock value 195 is larger than maximum itl 2
**** row 147: key out of order
**** row 149: key out of order
**** row 154: key out of order
**** row 154: lock value 72 is larger than maximum itl 2
**** row 155: key out of order
**** row 155: lock value 73 is larger than maximum itl 2
**** row 157: key out of order
**** row 157: lock value 195 is larger than maximum itl 2
**** row 159: row offset 8168 out of valid range
**** row 161: row offset 8138 out of valid range
**** row 163: row length 65943 past end of block
**** row 163: row skipped so other stats may be wrong
**** actual free space = 2756 < kdxcoavs = 3155
**** key (begin=0x11fa, len=17) overlaps with another
        begin = 0x1209 len = 9
---- end index block validation
页 69027 失败, 校验代码为 6401
itl[1] has higher commit scn(0x0000.000c021f) than block scn (0x0000.0009a123)
页 72609 失败, 校验代码为 6056
Block Checking: DBA = 4273153, Block Type = KTB-managed data block
data header at 0x5ac525c
kdbchk: row locked by non-existent transaction
        table=0   slot=93
        lockid=3   ktbbhitc=2
页 78849 失败, 校验代码为 6101
Block Checking: DBA = 4273154, Block Type = KTB-managed data block
data header at 0x5ac725c
kdbchk: bad row offset slot 113 offs 473 fseo 474 dtl 8168 bhs 72
页 78850 失败, 校验代码为 6135
Block Checking: DBA = 4273155, Block Type = KTB-managed data block
data header at 0x5ac925c
kdbchk: bad row offset slot 17 offs 342 fseo 394 dtl 8168 bhs 72
页 78851 失败, 校验代码为 6135
Block Checking: DBA = 4273185, Block Type = KTB-managed data block
data header at 0x5b0525c
kdbchk: row locked by non-existent transaction
        table=0   slot=14
        lockid=4   ktbbhitc=2
页 78881 失败, 校验代码为 6101
Block Checking: DBA = 4273189, Block Type = KTB-managed data block
data header at 0x5b0d25c
kdbchk: row locked by non-existent transaction
        table=0   slot=0
        lockid=2   ktbbhitc=2
页 78885 失败, 校验代码为 6101
Block Checking: DBA = 4273192, Block Type = KTB-managed data block
data header at 0x5b1325c
kdbchk: bad row offset slot 62 offs 569 fseo 1122 dtl 8168 bhs 72
页 78888 失败, 校验代码为 6135
Block Checking: DBA = 4282166, Block Type = KTB-managed data block
**** row 0: row offset 1457 out of valid range
**** row 1: row offset 1440 out of valid range
**** row 2: row offset 1423 out of valid range
**** row 3: row offset 1406 out of valid range
**** row 4: row offset 1389 out of valid range
**** row 5: row offset 1372 out of valid range
**** row 6: row offset 1355 out of valid range
**** row 7: row offset 1338 out of valid range
**** row 25: row offset 3055 out of valid range
**** row 26: row offset 3038 out of valid range
**** row 27: row offset 3021 out of valid range
**** row 28: row offset 3004 out of valid range
**** row 29: row offset 2987 out of valid range
**** row 30: row offset 2970 out of valid range
**** row 31: row offset 2953 out of valid range
**** row 32: row offset 2936 out of valid range
**** row 33: row offset 2919 out of valid range
**** row 34: row offset 2902 out of valid range
**** row 35: row offset 2885 out of valid range
**** row 36: row offset 2868 out of valid range
**** row 37: row offset 2851 out of valid range
**** row 38: row offset 2834 out of valid range
**** row 39: row offset 1321 out of valid range
**** row 40: row offset 2817 out of valid range
**** row 41: row offset 2800 out of valid range
**** row 42: row offset 2783 out of valid range
**** row 43: row offset 2766 out of valid range
**** row 44: row offset 2749 out of valid range
**** row 45: row offset 2732 out of valid range
**** row 46: row offset 2715 out of valid range
**** row 47: row offset 2698 out of valid range
**** row 48: row offset 2681 out of valid range
**** row 49: row offset 2664 out of valid range
**** row 50: row offset 2647 out of valid range
**** row 51: row offset 2630 out of valid range
**** row 52: row offset 2613 out of valid range
**** row 53: row offset 2596 out of valid range
**** row 54: row offset 2579 out of valid range
**** row 55: row offset 2562 out of valid range
**** row 56: row offset 2545 out of valid range
**** row 57: row offset 2528 out of valid range
**** row 58: row offset 2511 out of valid range
**** row 59: row offset 2494 out of valid range
**** row 60: row offset 2477 out of valid range
**** row 61: row offset 2460 out of valid range
**** row 62: row offset 2443 out of valid range
**** row 63: row offset 2426 out of valid range
**** row 64: row offset 2409 out of valid range
**** row 65: row offset 2392 out of valid range
**** row 66: row offset 2375 out of valid range
**** row 67: row offset 2358 out of valid range
**** row 68: row offset 2341 out of valid range
**** row 69: row offset 2324 out of valid range
**** row 70: row offset 2307 out of valid range
**** row 71: row offset 2290 out of valid range
**** row 72: row offset 2273 out of valid range
**** row 73: row offset 2256 out of valid range
**** row 74: row offset 2239 out of valid range
**** row 75: row offset 2222 out of valid range
**** row 76: row offset 2205 out of valid range
**** row 77: row offset 2188 out of valid range
**** row 78: row offset 2171 out of valid range
**** row 79: row offset 2154 out of valid range
**** row 80: row offset 2137 out of valid range
**** row 81: row offset 2120 out of valid range
**** row 82: row offset 2103 out of valid range
**** row 83: row offset 2086 out of valid range
**** row 84: row offset 2069 out of valid range
**** row 85: row offset 2052 out of valid range
**** row 86: row offset 2035 out of valid range
**** row 87: row offset 2018 out of valid range
**** row 88: row offset 2001 out of valid range
**** row 89: row offset 1984 out of valid range
**** row 90: row offset 1967 out of valid range
**** row 91: row offset 1950 out of valid range
**** row 92: row offset 1933 out of valid range
**** row 93: row offset 1916 out of valid range
**** row 94: row offset 1899 out of valid range
**** row 95: row offset 1882 out of valid range
**** row 96: row offset 1865 out of valid range
**** row 97: row offset 1848 out of valid range
**** row 98: row offset 1831 out of valid range
**** row 99: row offset 1814 out of valid range
**** row 100: row offset 1797 out of valid range
**** row 101: row offset 1780 out of valid range
**** row 102: row offset 1763 out of valid range
**** row 103: row offset 1746 out of valid range
**** row 104: row offset 1729 out of valid range
**** row 105: row offset 1712 out of valid range
**** row 106: row offset 1695 out of valid range
**** row 107: row offset 1678 out of valid range
**** row 108: row offset 1661 out of valid range
**** row 109: row offset 1644 out of valid range
**** row 110: row offset 1627 out of valid range
**** row 111: row offset 1610 out of valid range
**** row 112: row offset 1593 out of valid range
**** row 113: row offset 1576 out of valid range
**** row 114: row offset 1559 out of valid range
**** row 115: row offset 1542 out of valid range
**** row 116: row offset 1525 out of valid range
**** row 117: row offset 1508 out of valid range
**** row 118: row offset 1491 out of valid range
**** row 119: row offset 1474 out of valid range
**** actual rows locked by itl 2  = 95 != # in trans. header = 198
**** actual rows marked deleted = 95 != kdxlende = 123
---- end index block validation
页 87862 失败, 校验代码为 6401
Block Checking: DBA = 4282170, Block Type = KTB-managed data block
**** kdxcofbo = 434 != 590
---- end index block validation
页 87866 失败, 校验代码为 6401
Block Checking: DBA = 4301433, Block Type = KTB-managed data block
data header at 0x5bb525c
kdbchk: avsp(899) > tosp(867)
页 107129 失败, 校验代码为 6128


DBVERIFY - 验证完成

检查的页总数: 956160
处理的页总数 (数据): 919530
失败的页总数 (数据): 10
处理的页总数 (索引): 13114
失败的页总数 (索引): 4
处理的页总数 (其他): 3222
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 20294
标记为损坏的总页数: 0
流入的页总数: 0
加密的总页数        : 0
最高块 SCN            : 443681161 (0.443681161)

基于上述情况,现在基本上可以确定是由于datafile 1 block 165中的ind$记录的第四行出现损坏,导致数据库无法正常查询ind$记录,从而使得数据库无法open.已经定位到该问题了,处理起来相对比较简单,使用bbed对其进行修复,让数据库可以正常查询ind$表记录,从而正常open库

SQL> recover database;
完成介质恢复。
SQL> shutdown immediate;
ORA-01109: ??????


已经卸载数据库。
ORACLE 例程已经关闭。
SQL> startup mount pfile='d:/pfile.txt'
ORACLE 例程已经启动。

Total System Global Area 4275781632 bytes
Fixed Size                  2182592 bytes
Variable Size             822084160 bytes
Database Buffers         3439329280 bytes
Redo Buffers               12185600 bytes
数据库装载完毕。
SQL> alter database open;

数据库已更改。

然后到粗数据,完成本次恢复任务.