Database SOS – 第2页 – Professional Database Recovery Support@+8617813235971

客户一套nc系统,由于安装时候把库建在了比较小的分区上,运行一些时间之后,出现空间不足,现场技术人员对oracle不太熟悉,经过一系列操作(删除业务表空间,复制pdb,创建表空间等等操作),无法恢复数据库,准备使用备份的dmp进行还原,结果分析发现仅保留的最后一份dmp,是一份导出不完全的dmp文件,无法正常导入(以前处理过一个类似case:ORA-39773: parse of metadata stream failed故障处理,尝试导入报ORA-39246错:

C:\Users\XFF>impdp system/oracle@127.0.0.1/orapdb directory=expdp_dir dumpfile=xxxxx_2025-12-01_0230.dmp logfile=1.log

Import: Release 19.0.0.0.0 - Production on 星期三 12月 3 21:00:19 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

连接到: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
ORA-39002: 操作无效
ORA-39059: 转储文件集不完整
ORA-39246: 无法在提供的转储文件中定位主表

分析当时当初的dmp日志,由于expdp的job表所在表空间不足导致expdp导出失败

TABLE:"XIFENFEI"."EOM_MEASURE_POINT"
ORA-30032: 挂起的 (可恢复) 语句已超时
ORA-01691: Lob 段 XIFENFEI.SYS_LOB0000161267C00111$$ 无法通过 32 (在表空间 NNC_DATA01 中) 扩展
ORA-06512: 在 "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: 在 "SYS.KUPW$WORKER", line 12620
ORA-06512: 在 "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: 在 "SYS.KUPW$WORKER", line 11414
----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0xda5dae50     33476  package body SYS.KUPW$WORKER.WRITE_ERROR_INFORMATION
0xda5dae50     12641  package body SYS.KUPW$WORKER.DETERMINE_FATAL_ERROR
0xda5dae50     11602  package body SYS.KUPW$WORKER.CREATE_OBJECT_ROWS
0xda5dae50     15268  package body SYS.KUPW$WORKER.FETCH_XML_OBJECTS
0xda5dae50      3907  package body SYS.KUPW$WORKER.UNLOAD_METADATA
0xda5dae50     13736  package body SYS.KUPW$WORKER.DISPATCH_WORK_ITEMS
0xda5dae50      2429  package body SYS.KUPW$WORKER.MAIN
0x6524a4f0         2  anonymous block
KUPW: Object row index into parse items is: 1
KUPW: Parse item count is: 19
KUPW: In function CHECK_FOR_REMAP_NETWORK
KUPW: Nothing to remap
KUPW: In procedure BUILD_OBJECT_STRINGS - non-base info
KUPW: In procedure BUILD_SUBNAME_LIST with TABLE:XIFENFEI.EOM_MEASURE_POINT
KUPW: In function NEXT_PO_NUMBER
KUPW: PO number assigned: 34198
FORALL
KUPW: In procedure DETERMINE_FATAL_ERROR with ORA-30032: 挂起的 (可恢复) 语句已超时
ORA-01691: Lob 段 XIFENFEI.SYS_LOB0000161267C00111$$ 无法通过 32 (在表空间 NNC_DATA01 中) 扩展
作业 "XIFENFEI"."SYS_EXPORT_SCHEMA_01" 因致命错误于 星期一 12月 1 06:33:21 2025 elapsed 0 04:03:18 停止

从导出日志看,在导出大量”0 KB 0 行”记录之后提示表空间不足,expdp的job表无法扩展导致导出挂起然后超时导出终止(这个导出操作没有完全完成),从而在导入的时候出现了ORA-39059: 转储文件集不完整 ORA-39246: 无法在提供的转储文件中定位主表的错误.对于这种故障,分析导出日志,发现运气不错,所有有数据的表都导出完成,基于这个心中就有了第一层底气,所有表数据不会丢失(因为都导出到了这个dmp中),但是非表的字典数据不完整,要想业务完整跑起来,需要找到一个完整的业务字典信息.对于大量的备份dmp被删除,然后对应分区还写入了很多数据,只能尝试看运气,通过对磁盘文件镜像,然后进行反删除恢复,找出来一个11月26日的dmp的压缩文件是完整的

通过这个dmp导入业务字典信息,然后再利用expdp dmp解析工具(expdp dmp被加密破坏恢复)把所有表数据出来,经过这两者组合,顺利完成数恢复,可以测试业务完全正常

mysql drop database 恢复思路

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：mysql drop database 恢复思路

今天有一个客户在在虚拟机环境中使用drop database删除一个业务mysql的库,删除之后,第一时间关闭了该虚拟机系统.接到该case请求之后,让客户提供了虚拟机文件(vmdk),通过工具分析,mysql数据库是放在/分区下面的/data里面(lvm结构，xfs文件系统),但是已经看不到他们删除的数据库(mysql一个库对应一个目录)

对于这种情况,我们第一步尝试文件系统层面反删除恢复(对数据库所在分区进行文件系统层面扫描,尝试恢复被删除的mysql表文件),运气不错,找到了一些被删除库的ibd文件

对于这些文件,选择出来客户需要的表,采用discard+import方式进行恢复,类似命令

alter table tablename discard tablespace;
alter table tablename import tablespace;

对于有些部分block损坏的ibd文件,直接这样操作可能会报Data structure corruption错误,这样的情况,可以通过工具解析ibd文件进行恢复

对于有些表,可能在文件系统层面反删除无法恢复,我们通过对磁盘进行mysql的数据块层面扫描(识别在磁盘上没有覆盖的mysql的block),按照pageid的方式进行重组成一个个page文件

然后基于这些page进行恢复需要的表数据

基本上通过上述的方法实现最大限度mysql删除数据的恢复(比如drop database，drop table，truncate table)
如果MySQL数据库恢复问题,需要专业恢复技术支持，请联系我们提供专业MySql数据库恢复服务:
电话/微信:17813235971 Q Q:107644445 E-Mail:dba@xifenfei.com

PRVG-11975 : The I/O scheduler parameter of device “/dev/sdm” did not match the expected value on nodes

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：PRVG-11975 : The I/O scheduler parameter of device “/dev/sdm” did not match the expected value on nodes

今天在oracle linux 7.9的系统中装Oracle 19.29集群采用gridSetup applyRU 的方式安装

su – grid
cd $ORACLE_HOME
./gridSetup.sh -applyRU /soft/38298204/

在检测的步骤I/O scheduler warning

具体内容类似:

I/O scheduler - This task checks the I/O scheduler parameter configured Error: 
PRVG-11975:The I/O scheduler parameter of device "/dev/sdm" did not match the expected value on nodes "hisdb2,hisdb2". 
[Expected scheduler = "none" ; Found scheduler = "mq-deadline"] ? Cause:?The I/O scheduler parameter of the indicated 
device was not  the expected value on the indicated nodes. ? Action:?Change the I/O scheduler parameter using 
'echo deadline > /sys/block/<device>/queue/scheduler' command of the indicated device to ensure it is the expected value.

大概的意思是磁盘的I/O scheduler parameter被检查出来是mq-deadline,实际要求为:deadline,并建议使用echo deadline > /sys/block//queue/scheduler命令来修改,检查存储磁盘的实际值

[grid@hisdb1 grid]$ cat /sys/block/sdm/queue/scheduler
[mq-deadline] kyber bfq none

确实I/O scheduler为mq-deadline,查询了相关描述:
mq-deadline 是 deadline 调度器的多队列升级版，核心设计目标一致（按 “截止时间” 调度以保证 I/O 延迟），但适配的硬件场景、性能表现差异显著 —— 简单说：deadline 是为传统单队列设备（如 HDD）设计的老版本，mq-deadline 是为现代多队列设备（如 SSD/NVMe）优化的新版本。

客户这边刚好是华为的ssd存储,使用mq-deadline是非常合理的,如果调整为deadline反而不太合理了.初步感觉这个问题应该是Oracle 本身检测不合理导致,进一步查询MOS,Oracle自己也承认了该问题:

对于ssd环境,可以忽略该警告,继续安装

obet(Oracle Block Editor Tool)第二版发布

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：obet(Oracle Block Editor Tool)第二版发布

在几天之前发布了obet的第一个版本（Oracle数据块编辑工具( Oracle Block Editor Tool)-obet）,最近对功能进行了一些完善,并发布第二版（下载地址:下载obet），主要增加了以下几个功能:
0) corrupt [block x] 命令主要用于标记坏块功能(在第一版中已经完善)
1) repair [block x] 命令对于坏块的自动修复,主要包括checksum,tailchk,seq_kcbh等
2）增加copy chkscn 命令主要用于对文件头的checkpoint scn相关信息修复
3）增加copy resetlogscn命令主要用于对文件头的resetlogs相关信息修复
4）把之前的copy命令调整为copy data命令格式
5）增加copy block命令主要用于数据文件之间的数据块直接拷贝
该版本的help命令提示

OBET (Oracle Block Editor Tool) commands:
  open <config_file>    - Load file list from config file (format: <num> <path>)
  info                  - Show loaded file list (from open command)
  set filename <path>   - Set target file path (required)
  set file <num>        - Set filename using loaded file number (from open list)
  set blocksize <size>  - Set block size (2048,4096,[8192],16384,32768)
  set block <num>       - Set block number (starts from 0, default: 1)
  set offset <offset>   - Set offset within block (< blocksize, default: 0)
  set count <bytes>     - Set number of bytes to read (default: 32)
  set mode edit/browse  - Enable edit/browse mode
  d/dump [options]      - Display data (options: block X, offset Y, count N)
  m/modify <hex> [opts] - Modify data with hex (opts: block X, offset Y)
  undo                  - Undo last modification
  sum [block X]         - Calculate checksum for block (default: current block)
  sum apply [block X]   - Apply checksum: write calculated value to block
  tailchk [block X]     - Calculate tailchk for block (default: current block)
  tailchk apply [block X] - Apply tailchk: write calculated value to block
  repair [block X]      - Repair bad block (fix seq_kcbh, tailchk, checksum)
  copy data <src> to <dest>  - Copy data between files
    <src> format: file,block,offset,count (e.g., 1,1,10,64)
    <dest> format: file[,block][,offset] (e.g., 3 or 3,1 or 3,1,128)
  copy block file#,block# to file#,block#     - Copy entire data block
  copy chkscn file n to file m         - Copy datafile header checkpoint SCN info
  copy resetlogscn file n to file m    - Copy datafile header resetlogs info
  corrupt [block X]     - Mark block as corrupted (default: current block)
  show                  - Display current settings (filename, blocksize, block, offset, count, mode)
  license               - Show/manage software license (registration code required)
  version               - Show software version and developer information
  p/print <param>       - Print Oracle structure,Use the 'p/print' command to see details
  undo                  - Undo the last copy chkscn or copy resetlogscn operation
  help                  - Show this help message
  quit/exit             - Exit OBET

copy block功能演示
随便把一个block拷贝到文件的另外位置,也可以拷贝到不同文件的其他位置,根据需要调整

OBET> copy block 1,1 to 1,5

Confirm copy block:
Source: file#1 (/u01/xifenfei/system01.dbf), block 1 (entire 8192-byte block)
Target: file#1 (/u01/xifenfei/system01.dbf), block 5 (entire 8192-byte block)
Proceed? (Y/YES to confirm): y
Successfully copied block 1 from file#1 to block 5 in file#1 (8192 bytes).

OBET> set file 1 
filename set to: /u01/xifenfei/system01.dbf (file#1)

OBET> set count 128
count set to: 128

OBET> dump block 1 offset 0

File: /u01/xifenfei/system01.dbf
Block: 1                Offsets:     0 to   127
--------------------------------------------------------------------------------
00002000 01230000 01004000 00000000 00000104 28C60000 00000000 0004200B A2DB266A 
00002020 58494645 4E464549 AC020000 00720100 00200000 01000300 00000000 00000000 
00002040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00002060 08024000 07000000 00000000 7DC92131 12345678 87654321 00000000 00000000 

<128 bytes read>

OBET> dump block 5 offset 0

File: /u01/xifenfei/system01.dbf
Block: 5                Offsets:     0 to   127
--------------------------------------------------------------------------------
0000A000 01230000 01004000 00000000 00000104 28C60000 00000000 0004200B A2DB266A 
0000A020 58494645 4E464549 AC020000 00720100 00200000 01000300 00000000 00000000 
0000A040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
0000A060 08024000 07000000 00000000 7DC92131 12345678 87654321 00000000 00000000 

<128 bytes read>

copy chkscn功能演示

OBET> p kcvfh.kcvfhckp
File: H:\xifenfei\system01.dbf
Size: 8192 bytes
Block: 1
Offset: 484

struct kcvfhckp, 36 bytes                   @484
   struct kcvcpscn, 8 bytes                 @484
      ub4 kscnbas                          @484     0x1BDDBF09
      ub2 kscnwrp                          @488     0x163D
      ub2 kscnwrp2                         @490     0x0000
   ub4 kcvcptim                            @492     0x488361FC
   ub2 kcvcpthr                            @496     0x0001
   union u, 12 bytes                       @500
      struct kcvcprba, 12 bytes            @500
         ub4 kcrbaseq                     @500     0x0006E3D3
         ub4 kcrbabno                     @504     0x0000B44B
         ub2 kcrbabof                     @508     0x0010
   ub1 kcvcpetb[8]                         @512-519 02 00 00 00 00 00 00 00


OBET> p kcvfh.kcvfhckp
File: H:\xifenfei\sysaux01.dbf
Size: 8192 bytes
Block: 1
Offset: 484

struct kcvfhckp, 36 bytes                   @484
   struct kcvcpscn, 8 bytes                 @484
      ub4 kscnbas                          @484     0x67452301
      ub2 kscnwrp                          @488     0x0000
      ub2 kscnwrp2                         @490     0x0000
   ub4 kcvcptim                            @492     0x00000000
   ub2 kcvcpthr                            @496     0x0000
   union u, 12 bytes                       @500
      struct kcvcprba, 12 bytes            @500
         ub4 kcrbaseq                     @500     0x0006E3D3
         ub4 kcrbabno                     @504     0x0000B44B
         ub2 kcrbabof                     @508     0x0010
   ub1 kcvcpetb[8]                         @512-519 02 00 00 00 00 00 00 00

<kcvfh.kcvfhckp structure printed successfully>



OBET> copy chkscn file 1 to file 2

Confirm Modify chkscn:
Source: file#1 (H:\xifenfei\system01.dbf)
Target: file#2 (H:\xifenfei\sysaux01.dbf)
Proceed? (Y/YES to confirm): y
Successfully copied checkpoint SCN information from file#1 to file#2.

OBET> p kcvfh.kcvfhckp
File: H:\xifenfei\sysaux01.dbf
Size: 8192 bytes
Block: 1
Offset: 484

struct kcvfhckp, 36 bytes                   @484
   struct kcvcpscn, 8 bytes                 @484
      ub4 kscnbas                          @484     0x1BDDBF09
      ub2 kscnwrp                          @488     0x163D
      ub2 kscnwrp2                         @490     0x0000
   ub4 kcvcptim                            @492     0x488361FC
   ub2 kcvcpthr                            @496     0x0001
   union u, 12 bytes                       @500
      struct kcvcprba, 12 bytes            @500
         ub4 kcrbaseq                     @500     0x0006E3D3
         ub4 kcrbabno                     @504     0x0000B44B
         ub2 kcrbabof                     @508     0x0010
   ub1 kcvcpetb[8]                         @512-519 02 00 00 00 00 00 00 00

<kcvfh.kcvfhckp structure printed successfully>

copy resetlogscn功能演示

OBET> p kcvfh.kcvfhrlc
File: H:\xifenfei\system01.dbf
Size: 8192 bytes
Block: 1
Offset: 112

ub4 kcvfhrlc                             @112     0x3215EC5C

<kcvfh.kcvfhrlc structure printed successfully>

OBET> p kcvfh.kcvfhrls
File: H:\xifenfei\system01.dbf
Size: 8192 bytes
Block: 1
Offset: 116

struct kcvfhrls, 8 bytes                    @116
   ub4 kscnbas                             @116     0x000E74FF
   ub2 kscnwrp                             @120     0x0000
   ub2 kscnwrp2                            @122     0x0000

<kcvfh.kcvfhrls structure printed successfully>


OBET> p kcvfh.kcvfhrlc
File: H:\xifenfei\sysaux01.dbf
Size: 8192 bytes
Block: 1
Offset: 112

ub4 kcvfhrlc                             @112     0x67452301

<kcvfh.kcvfhrlc structure printed successfully>

OBET> p kcvfh.kcvfhrls
File: H:\xifenfei\sysaux01.dbf
Size: 8192 bytes
Block: 1
Offset: 116

struct kcvfhrls, 8 bytes                    @116
   ub4 kscnbas                             @116     0x43658709
   ub2 kscnwrp                             @120     0x0021
   ub2 kscnwrp2                            @122     0x0000

<kcvfh.kcvfhrls structure printed successfully>

OBET> copy resetlogscn file 1 to file 2

Confirm Modify resetlogscn:
Source: file#1 (H:\xifenfei\system01.dbf)
Target: file#2 (H:\xifenfei\sysaux01.dbf)
Proceed? (Y/YES to confirm): y
Successfully copied resetlog SCN information from file#1 to file#2.

OBET> p kcvfh.kcvfhrlc
File: H:\xifenfei\sysaux01.dbf
Size: 8192 bytes
Block: 1
Offset: 112

ub4 kcvfhrlc                             @112     0x3215EC5C

<kcvfh.kcvfhrlc structure printed successfully>

OBET> p kcvfh.kcvfhrls
File: H:\xifenfei\sysaux01.dbf
Size: 8192 bytes
Block: 1
Offset: 116

struct kcvfhrls, 8 bytes                    @116
   ub4 kscnbas                             @116     0x000E74FF
   ub2 kscnwrp                             @120     0x0000
   ub2 kscnwrp2                            @122     0x0000

<kcvfh.kcvfhrls structure printed successfully>

corrupt [block x]标记坏块功能

C:\Users\XFF>dbv file=h:/xifenfei/users01.dbf

DBVERIFY: Release 11.2.0.4.0 - Production on 星期三 11月 12 21:40:55 2025

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = H:\XIFENFEI\USERS01.DBF


DBVERIFY - 验证完成

检查的页总数: 26656
处理的页总数 (数据): 23569
失败的页总数 (数据): 0
处理的页总数 (索引): 309
失败的页总数 (索引): 0
处理的页总数 (其他): 720
处理的总页数 (段)  : 0
失败的总页数 (段)  : 0
空的页总数: 2058
标记为损坏的总页数: 0
流入的页总数: 0
加密的总页数        : 0
最高块 SCN            : 2255414715 (18.2255414715)


OBET> corrupt block 10

Confirm modification:
File: h:/xifenfei/users01.dbf
Block: 10
Offset: 14 (file offset: 0x0001400E)
Original value: 01
New value:      FF
Are you sure to set this block corrupted? (Y/YES to proceed): y
Verification successful: Block 10 marked as corrupted (offset 14 set to 0xFF).
Modification successful.

C:\Users\XFF>dbv file=h:/xifenfei/users01.dbf

DBVERIFY: Release 11.2.0.4.0 - Production on 星期三 11月 12 21:41:22 2025

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = H:\XIFENFEI\USERS01.DBF
页 10 流入 - 很可能是介质损坏
Corrupt block relative dba: 0x0100000a (file 4, block 10)
Fractured block found during dbv:
Data in bad block:
 type: 30 format: 2 rdba: 0x0100000a
 last change scn: 0x0000.00003e78 seq: 0xff flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x3e781e01
 check value in block header: 0x1a6
 computed block checksum: 0xfe



DBVERIFY - 验证完成

检查的页总数: 26656
处理的页总数 (数据): 23569
失败的页总数 (数据): 0
处理的页总数 (索引): 309
失败的页总数 (索引): 0
处理的页总数 (其他): 719
处理的总页数 (段)  : 0
失败的总页数 (段)  : 0
空的页总数: 2058
标记为损坏的总页数: 1
流入的页总数: 1
加密的总页数        : 0
最高块 SCN            : 2255414715 (18.2255414715)

repair [block x]修复坏块功能

C:\Users\XFF>dbv file=h:/xifenfei/undo01.dbf

DBVERIFY: Release 11.2.0.4.0 - Production on 星期日 11月 16 00:06:59 2025

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = H:\XIFENFEI\UNDO01.DBF
页 2 流入 - 很可能是介质损坏
Corrupt block relative dba: 0x02000002 (file 8, block 2)
Fractured block found during dbv:
Data in bad block:
 type: 29 format: 2 rdba: 0x02000002
 last change scn: 0x163d.1bddbcfa seq: 0xff flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0xbcfa1d02
 check value in block header: 0xec11
 computed block checksum: 0xfd



DBVERIFY - 验证完成

检查的页总数: 4880
处理的页总数 (数据): 0
失败的页总数 (数据): 0
处理的页总数 (索引): 0
失败的页总数 (索引): 0
处理的页总数 (其他): 4372
处理的总页数 (段)  : 11
失败的总页数 (段)  : 0
空的页总数: 507
标记为损坏的总页数: 1
流入的页总数: 1
加密的总页数        : 0
最高块 SCN            : 467517187 (5693.467517187)

OBET> repair

Repairing block 2 in file H:\xifenfei\undo01.dbf...

Repair analysis for block 2:
1. seq_kcbh check: 0xFF -> needs repair (0x01)
2. Tailchk check: 0x021DFABC -> needs repair (0x011DFABC)
3. Checksum check: 0x11EC -> OK

Confirm repair operations:
File: H:\xifenfei\undo01.dbf
Block: 2
Operations needed: fix offset14, fix tailchk
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x011DFABC OK
3. Checksum: 0x11EC OK

Block 2 repair completed successfully.


C:\Users\XFF>dbv file=h:/xifenfei/undo01.dbf

DBVERIFY: Release 11.2.0.4.0 - Production on 星期日 11月 16 00:07:29 2025

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = H:\XIFENFEI\UNDO01.DBF


DBVERIFY - 验证完成

检查的页总数: 4880
处理的页总数 (数据): 0
失败的页总数 (数据): 0
处理的页总数 (索引): 0
失败的页总数 (索引): 0
处理的页总数 (其他): 4373
处理的总页数 (段)  : 11
失败的总页数 (段)  : 0
空的页总数: 507
标记为损坏的总页数: 0
流入的页总数: 0
加密的总页数        : 0
最高块 SCN            : 467517187 (5693.467517187)

Oracle数据块编辑工具( Oracle Block Editor Tool)-obet

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle数据块编辑工具( Oracle Block Editor Tool)-obet

由于oracle后续版本对于bbed的支持不是太友好(从10g之后无法直接编译使用,win版本偏移量错误等),基于这些情况,结合当前ai的便利,自己动手写了一个基础版的obet(Oracle Block Editor Tool),用来实现在某些情况下Oracle 数据块的编辑工作.目前(2025年11月7日)发布第一版主要功能有:
1. 通过16进制查看数据文件任何偏移量位置的数据(d/dump)
2. 通过16进制编辑数据文件任何偏移量位置的数据(m/modify)
3. 对oracle 数据块的tailchk自动修复(tailchk apply)
4. 对oracle 数据块的checksum自动修复(sum apply)
5. 数据文件之间部分数据拷贝功能(coopy)

启动软件界面

[root@iZbp11c0qyuuo1gr7j98upZ tmp]# ./obet
=============================================
Welcome to Oracle Block Editor Tool (OBET)
=============================================

***** !!! For Oracle Internal Use only !!! *****

[Software Function Description]
- View and edit data block in hexadecimal format
- Automatically repair tailchk and checksum
- Copy data between different data files
- Mark data block as corrupted block

[Developer Information]
- Name: XiFenFei
- Phone: +86-17813235971
- Email: dba@xifenfei
- Q Q: 107644445
- WeChat: 17813235971
- Website: https://www.xifenfei.com

[Version Details]
- Software Version: v2025.11.001
- Build Date: 2025.11.07

=============================================
Type 'help' for command list | 'exit' to quit
=============================================

==================================================
Software License Status: Authorized
==================================================

使用说明

OBET> help
OBET (Oracle Block Editor Tool) commands:
  open <config_file>    - Load file list from config file (format: <num> <path>)
  info                  - Show loaded file list (from open command)
  set filename <path>   - Set target file path (required)
  set file <num>        - Set filename using loaded file number (from open list)
  set blocksize <size>  - Set block size (2048,4096,[8192],16384,32768)
  set block <num>       - Set block number (starts from 0, default: 1)
  set offset <offset>   - Set offset within block (< blocksize, default: 0)
  set count <bytes>     - Set number of bytes to read (default: 32)
  set mode edit/browse  - Enable edit/browse mode
  d/dump [options]      - Display data (options: block X, offset Y, count N)
  m/modify <hex> [opts] - Modify data with hex (opts: block X, offset Y)
  undo                  - Undo last modification
  sum [block X]         - Calculate checksum for block (default: current block)
  sum apply [block X]   - Apply checksum: write calculated value to block
  tailchk [block X]     - Calculate tailchk for block (default: current block)
  tailchk apply [block X] - Apply tailchk: write calculated value to block
  copy <src> to <dest>  - Copy data between files
    <src> format: file,block,offset,count (e.g., 1,1,10,64)
    <dest> format: file[,block][,offset] (e.g., 3 or 3,1 or 3,1,128)
  corrupt [block X]     - Mark block as corrupted (default: current block)
  show                  - Display current settings (filename, blocksize, block, offset, count, mode)
  license               - Show/manage software license (registration code required)
  version               - Show software version and developer information
  help                  - Show this help message
  quit/exit             - Exit OBET

加载数据文件

--使用open打开数据文件列表(格式: 编号  路径)
OBET> open /tmp/3.txt
Loaded 4 files from config file '/tmp/3.txt'.

OBET> info

Loaded files (4 total):
----------------------------------------
Number  Path
----------------------------------------
     1  /u01/xifenfei/system01.dbf
     2  /u01/xifenfei/sysaux01.dbf
     3  /u01/xifenfei/undotbs01.dbf
     4  /u01/xifenfei/users01.dbf
----------------------------------------

OBET> set file 1
filename set to: /u01/xifenfei/system01.dbf (file#1)

--或者直接使用 set filename
OBET> set filename /tmp/system01.dbf
filename set to: /tmp/system01.dbf

进入数据文件特定位置

OBET> set file 2
filename set to: /u01/xifenfei/sysaux01.dbf (file#2)

OBET> set block 5
block set to: 5

OBET> set offset 128
offset set to: 128

16进制方式查看数据

OBET> d

File: /u01/xifenfei/system01.dbf
Block: 1                Offsets:     0 to    31
--------------------------------------------------------------------------------
00002000 0BA20000 01004000 00000000 00000104 224D0000 00000000 0004200B A2DB266A 

<32 bytes read>

OBET> dump block 521 offset 128

File: /u01/xifenfei/system01.dbf
Block: 521                Offsets:   128 to   159
--------------------------------------------------------------------------------
00412080 5E068D05 C6040000 00000000 00000000 00000000 00000000 00000000 00000000 

<32 bytes read>

OBET> set count 128
count set to: 128

OBET> d

File: /u01/xifenfei/system01.dbf
Block: 1                Offsets:     0 to   127
--------------------------------------------------------------------------------
00002000 0BA20000 01004000 00000000 00000104 224D0000 00000000 0004200B A2DB266A 
00002020 58494645 4E464549 AC020000 00720100 00200000 01000300 00000000 00000000 
00002040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00002060 08024000 07000000 00000000 7DC92131 64676345 06200E00 00000000 00000000 

<128 bytes read>

16进制方式修改数据块内容(一般修改数据块内容之后建议校验tailchk和sum)

OBET> d

File: /u01/xifenfei/system01.dbf
Block: 1                Offsets:     0 to   127
--------------------------------------------------------------------------------
00002000 0BA20000 01004000 00000000 00000104 224D0000 00000000 0004200B A2DB266A 
00002020 58494645 4E464549 AC020000 00720100 00200000 01000300 00000000 00000000 
00002040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00002060 08024000 07000000 00000000 7DC92131 64676345 06200E00 00000000 00000000 

<128 bytes read>

OBET> m 0123

Confirm modification:
File: /u01/xifenfei/system01.dbf
Block: 1
Offset: 0 (file offset: 0x00002000)
Original value: 0B
New value:      0123
Confirm? (Y/YES to proceed): y
Verification successful: Data written correctly.
Modified 2 bytes at offset 0x00002000 successfully.

OBET> d

File: /u01/xifenfei/system01.dbf
Block: 1                Offsets:     0 to   127
--------------------------------------------------------------------------------
00002000 01230000 01004000 00000000 00000104 224D0000 00000000 0004200B A2DB266A 
00002020 58494645 4E464549 AC020000 00720100 00200000 01000300 00000000 00000000 
00002040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00002060 08024000 07000000 00000000 7DC92131 64676345 06200E00 00000000 00000000 

<128 bytes read>

tail和checksum修改

OBET> tailchk
Check tailchk for File /u01/xifenfei/system01.dbf, Block 1:
current = 0x010B0000, required = 0x01010000

OBET> sum
Check value for File /u01/xifenfei/system01.dbf, Block 1:
current = 0x224D, required = 0x28CC

OBET> tailchk apply

Confirm applying tailchk:
File: /u01/xifenfei/system01.dbf
Block: 1
Offset in block: 8188 (file offset: 0x00003FFC)
Original value: 0x010B0000
New value:      0x01010000
Confirm? (Y/YES to proceed): y
Verification successful: Stored tailchk matches calculated value (0x01010000).
Tailchk applied successfully.

OBET> sum apply

Confirm applying checksum:
File: /u01/xifenfei/system01.dbf
Block: 1
Offset in block: 16 (file offset: 0x00002010)
Original value: 0x224D
New value:      0x28C6
Confirm? (Y/YES to proceed): y
Verification successful: Stored checksum matches calculated value (0x28C6).
Checksum applied successfully.

两个数据文件之前拷贝数据(一般copy数据之后建议校验tailchk和sum)
一般情况下文件之间的拷贝就是数据号不一样,比如修改checkpoint,resetlog信息等,这里支持不一样偏移量,不一样数据块的拷贝

OBET> copy 1,1,0,128 to 3,5,128

Confirm copy:
Source: file#1 (/u01/xifenfei/system01.dbf), block 1, offset 0, 128 bytes
Target: file#3 (/u01/xifenfei/undotbs01.dbf), block 5, offset 128
Proceed? (Y/YES to confirm): y
Copy successful: 128 bytes copied from file #1 to file #3.

OBET> set file 3
filename set to: /u01/xifenfei/undotbs01.dbf (file#3)

OBET> d block 5 offset 128

File: /u01/xifenfei/undotbs01.dbf
Block: 5                Offsets:   128 to   255
--------------------------------------------------------------------------------
0000A080 0BA20000 01004000 00000000 00000104 224D0000 00000000 0004200B A2DB266A 
0000A0A0 58494645 4E464549 AC020000 00720100 00200000 01000300 00000000 00000000 
0000A0C0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
0000A0E0 08024000 07000000 00000000 7DC92131 64676345 06200E00 00000000 00000000 

<128 bytes read>

OBET> set file 1
filename set to: /u01/xifenfei/system01.dbf (file#1)

OBET> dump block 1 offset 0

File: /u01/xifenfei/system01.dbf
Block: 1                Offsets:     0 to   127
--------------------------------------------------------------------------------
00002000 0BA20000 01004000 00000000 00000104 224D0000 00000000 0004200B A2DB266A 
00002020 58494645 4E464549 AC020000 00720100 00200000 01000300 00000000 00000000 
00002040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00002060 08024000 07000000 00000000 7DC92131 64676345 06200E00 00000000 00000000 

<128 bytes read>

标记数据块为坏块功能


OBET> show 

Current settings:
File: /u01/xifenfei/system01.dbf
Blocksize: 8192 bytes
Block: 1
Offset in block: 0 (file offset: 0x00002000)
Count: 128 bytes
Mode: edit
Loaded files: 4 (use 'info' to list)

OBET> corrupt

Confirm modification:
File: /u01/xifenfei/system01.dbf
Block: 1
Offset: 14 (file offset: 0x0000200E)
Original value: 01
New value:      FF
Are you sure to set this block corrupted? (Y/YES to proceed): y
Verification successful: Block 1 marked as corrupted (offset 14 set to 0xFF).
Modification successful.

由于该工具直接编辑Oracle 底层数据块操作具有一定的破坏性和风险性,所以在没有授权的情况下无法对数据块进行修改(只能查看),具体授权操作

OBET> license

========================================
           Software Registration        
========================================

Your Hardware ID: XXXXXXXX  ----->提供给我

Please send your Hardware ID to XiFenFei to register.
Website: https://www.xifenfei.com 
Tel/WX: +86-17813235971 

Enter Registration Code: XXXXXX-XXXXXXXX <-----输入注册码进行授权
Registration successful!

Oracle坏块修复工具:Patch_blk

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle坏块修复工具:Patch_blk

在win平台上开发了Oracle Recovery Tools小工具,可以实现坏块的快速恢复功能

具体参考相关文章:自研Oracle恢复小工具
由于之前工具使用c#开发不太方便实现跨平台(而且图形化在linux等操作系统上使用起来不方便),现在使用c语言写了小工具:Patch_blk(主要是seq_kcbh tailchk checksum类似坏块处理),模拟两个坏块

BBED> verify
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/xifenfei/system01.dbf
BLOCK = 521

Block 521 is corrupt
Corrupt block relative dba: 0x00400209 (file 0, block 521)
Fractured block found during verification
Data in bad block:
 type: 6 format: 2 rdba: 0x00400209
 last change scn: 0x0000.000001d7 seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x01d706fe
 check value in block header: 0x5205
 computed block checksum: 0x0


DBVERIFY - Verification complete

Total Blocks Examined         : 1
Total Blocks Processed (Data) : 0
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 1
Total Blocks Influx           : 2
Message 531 not found;  product=RDBMS; facility=BBED


BBED> verify block 522
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/xifenfei/system01.dbf
BLOCK = 522

Block 522 is corrupt
Corrupt block relative dba: 0x0040020a (file 0, block 522)
Fractured block found during verification
Data in bad block:
 type: 6 format: 2 rdba: 0x0040020a
 last change scn: 0x0000.000001d7 seq: 0xff flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x01d70601
 check value in block header: 0x1e16
 computed block checksum: 0x0


DBVERIFY - Verification complete

Total Blocks Examined         : 1
Total Blocks Processed (Data) : 0
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 1
Total Blocks Influx           : 2
Message 531 not found;  product=RDBMS; facility=BBED

配置坏块修复列表文件

[root@iZbp11c0qyuuo1gr7j98upZ tmp]# cat 1.txt
/u01/app/oracle/oradata/xifenfei/system01.dbf  521 8192 N
/u01/app/oracle/oradata/xifenfei/system01.dbf  522 8192 Y 

--列表文件说明
数据文件路径 文件号 数据块大小 是否人工干预(N表示不需要,Y表示需要)

执行修复操作

[root@iZbp11c0qyuuo1gr7j98upZ tmp]# ./Patch_blk 1.txt


===== Processing line 1 =====
=====================================================
Processing: File=/u01/app/oracle/oradata/xifenfei/system01.dbf, Block=521, Size=8192, Mode=N
=====================================================

=== Step 1: Check seq_kcbh ===
  Current value: 0x01
  [OK] seq_kcbh is normal

=== Step 2: Check tailchk ===
  Current (reversed): 0x01D706FE
  Expected (reversed): 0x01D70601
  [Auto-repair] tailchk mismatch, will fix
  [Backup] Block saved to: ./20251104_230616/system01.dbf_block521
  [Success] tailchk fixed
  [Verify] New tailchk (reversed): 0x01D70601

=== Step 3: Check checksum  ===
  Current checksum: 0x0552
  Computed checksum: 0xFA52
  [Auto-repair] Checksum mismatch, will fix
  [Success] Checksum fixed
  [Verify] New checksum: 0xFA52

=== Step 4: Final Verification ===
  seq_kcbh: 0x01 (not 0xFF: PASS)
  tailchk: 0x01D70601 (expected 0x01D70601: PASS)
  Checksum: 0xFA52 (expected 0xFA52: PASS)

[Result] Block repair completed successfully


===== Processing line 2 =====
=====================================================
Processing: File=/u01/app/oracle/oradata/xifenfei/system01.dbf, Block=522, Size=8192, Mode=Y
=====================================================

=== Step 1: Check seq_kcbh ===
  Current value: 0xFF
  [WARNING] Block is marked as BAD. Repair? (yes/no): y
  [Backup] Block saved to: ./20251104/system01.dbf_block522
  [Success] seq_kcbh updated to 0x01

=== Step 2: Check tailchk ===
  Current (reversed): 0x01D70601
  Expected (reversed): 0x01D70601
  [OK] tailchk is normal

=== Step 3: Check checksum ===
  Current checksum: 0x161E
  Computed checksum: 0xE81E
  [WARNING] Checksum mismatch. Repair? (yes/no): y
  [Success] Checksum fixed
  [Verify] New checksum: 0xE81E

=== Step 4: Final Verification ===
  seq_kcbh: 0x01 (not 0xFF: PASS)
  tailchk: 0x01D70601 (expected 0x01D70601: PASS)
  Checksum: 0xE81E (expected 0xE81E: PASS)

[Result] Block repair completed successfully


=====================================
Processing complete. Total: 2
  Modify Success: 2
  Modify None: 0
  Skipped/Failed: 0
  Blocks Backed Up: 2
=====================================

在修复坏块之前会对相关block进行备份

[root@iZbp11c0qyuuo1gr7j98upZ 20251104]# ls -ltra
total 24
-rw-r--r--   1 root root 8192 Nov  4 22:42 system01.dbf_block521
-rw-r--r--   1 root root 8192 Nov  4 22:42 system01.dbf_block522
drwxr-xr-x   2 root root 4096 Nov  4 22:42 .
drwxrwxrwt. 14 root root 4096 Nov  4 23:39 ..
[root@iZbp11c0qyuuo1gr7j98upZ 20251104]#

Patch_blk修复坏块之后,检查坏块正常

BBED> verify
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/xifenfei/system01.dbf
BLOCK = 521


DBVERIFY - Verification complete

Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED


BBED> verify block 522
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/xifenfei/system01.dbf
BLOCK = 522


DBVERIFY - Verification complete

Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED

对于不太熟悉bbed的客户,可以通过这个工具快速实现常见坏块类型恢复

ORA-01172 ORA-01151故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-01172 ORA-01151故障处理

接手客户一个云平台硬件故障恢复之后,数据库无法启动的case,通过分析alert日志,发现数据库在open过程中报ORA-01172: recovery of thread 1 stuck at block 2220167 of file 262,ORA-01151: use media recovery to recover block, restore backup if needed等相关错误(其实也就是在做实例恢复的过程中报了logically corrupt导致无法完成实例恢复)

Sat Nov 01 14:29:10 2025
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 7034 KB redo, 937 data blocks need recovery
Started redo application at
 Thread 1: logseq 296553, block 389408
Recovery of Online Redo Log: Thread 1 Group 3 Seq 296553 Reading mem 0
  Mem# 0: /data/orcl/onlinelog/redo03a.log
Sat Nov 01 14:29:11 2025
Hex dump of (file 262, block 2220584) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p009_2648.trc
Sat Nov 01 14:29:11 2025
Hex dump of (file 262, block 2218886) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p007_2644.trc
Reading datafile '/data/orcl/datafile/xifenfei12.dbf' for corruption at rdba: 0x41a1db86 (file 262, block 2218886)
Reading datafile '/data/orcl/datafile/xifenfei12.dbf' for corruption at rdba: 0x41a1e228 (file 262, block 2220584)
Sat Nov 01 14:29:11 2025
Hex dump of (file 262, block 2219845) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p008_2646.trc
Reading datafile '/data/orcl/datafile/xifenfei12.dbf' for corruption at rdba: 0x41a1df45 (file 262, block 2219845)
Sat Nov 01 14:29:11 2025
Hex dump of (file 262, block 2220167) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p001_2632.trc
Reading datafile '/data/orcl/datafile/xifenfei12.dbf' for corruption at rdba: 0x41a1e087 (file 262, block 2220167)
Reread (file 262, block 2218886) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 2218886 OF FILE 262
Reread (file 262, block 2220584) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 2220584 OF FILE 262
Reread (file 262, block 2219845) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 2219845 OF FILE 262
Reread (file 262, block 2220167) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 2220167 OF FILE 262
Sat Nov 01 14:29:26 2025
Slave exiting with ORA-1172 exception
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p007_2644.trc:
ORA-01172: recovery of thread 1 stuck at block 2218886 of file 262
ORA-01151: use media recovery to recover block, restore backup if needed
Sat Nov 01 14:29:26 2025
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p008_2646.trc:
ORA-10388: parallel query server interrupt (failure)
Sat Nov 01 14:29:26 2025
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p009_2648.trc:
ORA-10388: parallel query server interrupt (failure)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p008_2646.trc:
ORA-10388: parallel query server interrupt (failure)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p009_2648.trc:
ORA-10388: parallel query server interrupt (failure)
Sat Nov 01 14:29:26 2025
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p001_2632.trc:
ORA-10388: parallel query server interrupt (failure)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p001_2632.trc:
ORA-10388: parallel query server interrupt (failure)
Sat Nov 01 14:29:26 2025
Aborting crash recovery due to slave death, attempting serial crash recovery
Beginning crash recovery of 1 threads
Started redo scan
Completed redo scan
 read 7034 KB redo, 937 data blocks need recovery
Started redo application at
 Thread 1: logseq 296553, block 389408
Recovery of Online Redo Log: Thread 1 Group 3 Seq 296553 Reading mem 0
  Mem# 0: /data/orcl/onlinelog/redo03a.log
Hex dump of (file 262,block 2220167) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2606.trc
Reading datafile '/data/orcl/datafile/xifenfei12.dbf'for corruption at rdba: 0x41a1e087 (file 262,block 2220167)
Reread (file 262, block 2220167) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 2220167 OF FILE 262
Aborting crash recovery due to error 1172
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2606.trc:
ORA-01172: recovery of thread 1 stuck at block 2220167 of file 262
ORA-01151: use media recovery to recover block, restore backup if needed
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2606.trc:
ORA-01172: recovery of thread 1 stuck at block 2220167 of file 262
ORA-01151: use media recovery to recover block, restore backup if needed
ORA-1172 signalled during: ALTER DATABASE OPEN...

接手故障之后,尝试recover database恢复,结果报ORA-600 4552错误

SQL> recover database;
ORA-10562: Error occurred while applying redo to data block (file# 262, block#
2222153)
ORA-10564: tablespace XIFENFEI
ORA-01110: data file 262: '/data/orcl/datafile/xifenfei12.dbf'
ORA-10560: block type '0'
ORA-00600: internal error code, arguments: [4552], [1], [0], [], [], [], [],
[], [], [], [], []

关于ORA-600 4552对应的alert日志信息

Sat Nov 01 17:49:58 2025
ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 16 slaves
Sat Nov 01 17:49:59 2025
Recovery of Online Redo Log: Thread 1 Group 3 Seq 296553 Reading mem 0
  Mem# 0: /data/orcl/onlinelog/redo03a.log
Sat Nov 01 17:49:59 2025
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr0c_28770.trc  (incident=1018821):
ORA-00600: internal error code, arguments: [4552], [1], [0], [], [], [], [], [], [], [], [], []
Incident details in:/u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_1018821/orcl_pr0c_28770_i1018821.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Sat Nov 01 17:50:03 2025
Sweep [inc][1018821]: completed
Sweep [inc2][1018821]: completed
Slave exiting with ORA-10562 exception
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr0c_28770.trc:
ORA-10562: Error occurred while applying redo to data block (file# 262, block# 2222153)
ORA-10564: tablespace xifenfei
ORA-01110: data file 262: '/data/orcl/datafile/xifenfei12.dbf'
ORA-10560: block type '0'
ORA-00600: internal error code, arguments: [4552], [1], [0], [], [], [], [], [], [], [], [], []
Recovery Slave PR0C previously exited with exception 10562
Media Recovery failed with error 448
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr00_28732.trc:
ORA-00283: recovery session canceled due to errors
ORA-00448: normal completion of background process
ORA-10562 signalled during: ALTER DATABASE RECOVER  database  ...

无法整个库级别recover,尝试数据文件recover操作

SQL> recover datafile 1;
Media recovery complete.
…………
SQL> recover datafile 22,23,24,26,25,27,28,29,30;
Media recovery complete.
…………
SQL>    recover datafile  251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261;
Media recovery complete.
SQL> recover datafile 262;
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [3020], [262], [2215808],
[1101123456], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 262, block# 2215808,
file offset is 972029952 bytes)
ORA-10564: tablespace XIFENFEI
ORA-01110: data file 262: '/data/orcl/datafile/xifenfei12.dbf'
ORA-10560: block type '0'

SQL> recover datafile 263;
Media recovery complete.

出file# 262数据文件之外,其他文件全部recover成功,对应的ORA-600 3020错误相关alert日志信息

Sat Nov 01 17:53:37 2025
ALTER DATABASE RECOVER  datafile 262  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 3 Seq 296553 Reading mem 0
  Mem# 0: /data/orcl/onlinelog/redo03a.log
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_28561.trc  (incident=1018717):
ORA-00600: internal error code, arguments: [3020], [262], [2215808], [1101123456], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 262, block# 2215808, file offset is 972029952 bytes)
ORA-10564: tablespace xifenfei
ORA-01110: data file 262: '/data/orcl/datafile/xifenfei12.dbf'
ORA-10560: block type '0'
Incident details in:/u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_1018717/orcl_ora_28561_i1018717.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Media Recovery failed with error 600
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 262  ...

对于这种情况,有两种处理方式:
1)在recover过程中对于报错的block标记为坏块,然后继续恢复,这样正常应用日志成功,再把标记的坏块修复好
2)直接修改该文件头跳过该文件跳过这些block的应用日志,直接骗过数据库
在本case中由于客户急着恢复业务,需要尽快处理,所以采用了第一个方案,这里我使用自研的m_scn(modify_scn)工具快速修改相关数据文件信息

[oracle@host-172-18-50-10 tmp]$ cat 1.txt
1@/data/orcl/datafile/system01.dbf
262@/data/orcl/datafile/xifenfei12.dbf
[oracle@host-172-18-50-10 tmp]$ ./m_scn 1.txt
Please Enter Password: 

===== Starting Datafile Header modification program =====
Datafile list file: 1.txt
Operation Mode: Only Modify Datafile Header CheckPoint
Block Size: 8192
Log Path: /tmp/modify_scn
---------------------------------------------------------
Preparing Datafile list file...
Verifying Datafile existence...
Datafile verification passed
Initializing working directory...
Recovery script created: /tmp/modify_scn/backup/recover_datafile.sh
---------------------------------------------------------
Starting Datafile Header processing (total 2 files)...
[1/2] Processing Datafile Header: /data/orcl/datafile/system01.dbf (File number: 1)
  - Skipping file number 1 (control file)
---------------------------------------------------------
[2/2] Processing Datafile Header: /data/orcl/datafile/xifenfei12.dbf (File number: 262)
  - Backing up Datafile header...
  - Executing Datafile Header modification with block size 8192...
  - Datafile Header processing completed
---------------------------------------------------------
Cleaning up temporary files...
================= All operations completed =================

Note: Execute /tmp/modify_scn/backup/recover_datafile.sh operation for rollback

然后查询相关scn信息,确认修改文件信息没有问题并尝试recover 262号文件

[oracle@host-172-18-50-10 tmp]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sat Nov 1 18:02:18 2025

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount;
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 4.1823E+10 bytes
Fixed Size                  2262368 bytes
Variable Size            4294970016 bytes
Database Buffers         3.7447E+10 bytes
Redo Buffers               78614528 bytes
Database mounted.
SQL> set pages 10000
SQL> set numw 16
SQL> SELECT status,
  2  checkpoint_change#,
  3  to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,
  4  last_change#,
  5  count(*) ROW_NUM
FROM v$datafile
  6    7  GROUP BY status, checkpoint_change#, checkpoint_time,last_change#
ORDER BY status, checkpoint_change#, checkpoint_time;
  8  

set numw 16
col CHECKPOINT_TIME for a40
set lines 150
set pages 1000
SELECT status,
to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,FUZZY,checkpoint_change#,
count(*) ROW_NUM
FROM v$datafile_header
GROUP BY status, checkpoint_change#, to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss'),fuzzy
ORDER BY status, checkpoint_change#, checkpoint_time;

SELECT dd.FILE#,
dd.NAME,
dd.STATUS,
dd.checkpoint_change# dfile_chkp_change,
dh.checkpoint_change# dfile_hed_chkp_change,
dh.recover,
dh.fuzzy
FROM v$datafile dd,
v$datafile_header dh
WHERE dd.FILE#=dh.FILE#
AND dd.checkpoint_change#<>dh.checkpoint_change#;


STATUS  CHECKPOINT_CHANGE# CHECKPOINT_TIME         LAST_CHANGE#          ROW_NUM
------- ------------------ ------------------- ---------------- ----------------
ONLINE      16816934458875 2025-11-01 17:58:28   16816934458875              258
RECOVER     16816934368799 2025-11-01 05:29:39   16816934456943                1
SYSTEM      16816934458875 2025-11-01 17:58:28   16816934458875                4

SQL> SQL> SQL> SQL> SQL> SQL> SQL>   2    3    4    5    6  
STATUS  CHECKPOINT_TIME                          FUZ CHECKPOINT_CHANGE#          ROW_NUM
------- ---------------------------------------- --- ------------------ ----------------
OFFLINE 2025-11-01 17:58:28                      NO      16816934458875                1
ONLINE  2025-11-01 17:58:28                      NO      16816934458875              262

SQL> SQL>   2    3    4    5    6    7    8    9   10   11  
           FILE#
----------------
NAME
----------------------------------------------------------------------------------
STATUS  DFILE_CHKP_CHANGE DFILE_HED_CHKP_CHANGE REC FUZ
------- ----------------- --------------------- --- ---
             262
/data/orcl/datafile/xifenfei12.dbf
RECOVER    16816934368799        16816934458875 YES NO

SQL> recover datafile 262;
Media recovery complete.

open数据库成功

SQL> alter database open;

Database altered.

Sat Nov 01 18:06:00 2025
ALTER DATABASE OPEN
Thread 1 opened at log sequence 296554
  Current log# 1 seq# 296554 mem# 0: /data/orcl/onlinelog/redo01a.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
[33941] Successfully onlined Undo Tablespace 143.
Undo initialization finished serial:0 start:12793234 end:12793304 diff:70 (0 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Sat Nov 01 18:06:01 2025
QMNC started with pid=20, OS id=33973 
Completed: ALTER DATABASE OPEN

至此基本上完成本次恢复任务,后续根据alert日志,有个别表可能由于在file# 262中丢失一些数据导致不一致的问题进行单独,其他没有太大问题,最快帮客户恢复了业务

C_OBJ#_INTCOL#坏块导致数据库无法open故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：C_OBJ#_INTCOL#坏块导致数据库无法open故障处理

客户通过硬件恢复出来数据文件之后,尝试启动数据库报错,他们经过多轮尝试依旧无法open数据库,还原到恢复出来文件的初始状态,通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)脚本检测,发现file# 3 需要sequence#为3200的日志,其他文件需要sequence#为3205的日志

检查发现数据库为非归档模式,而且sequence#为3200的redo已经被覆盖

基于上述情况,可以确认除file# 3之外,其他文件可以正常recover

[oracle@oracledb check_db]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Oct 15 16:08:33 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> recover datafile 1;
Media recovery complete.
SQL> recover datafile 2,4,5,7,8,9,10,11,12,13,14,15,16,17;
Media recovery complete.
SQL> recover datafile 18,19,20,21;
Media recovery complete.

对于缺少日志的数据文件,直接使用自研的m_scn(modify scn)工具进行修改

[oracle@oracledb check_db]$ chmod +x m_scn
[oracle@oracledb check_db]$ ./m_scn 1.txt
Please Enter Password: 

===== Starting Datafile Header modification program =====
Datafile list file: 1.txt
Operation Mode: Only Modify Datafile Header CheckPoint
Block Size: 8192
Log Path: /tmp/modify_scn
---------------------------------------------------------
Preparing Datafile list file...
Verifying Datafile existence...
ERROR: Datafile  does not exist!
Datafile verification passed
Initializing working directory...
Recovery script created: /tmp/modify_scn/backup/recover_datafile.sh
---------------------------------------------------------
Starting Datafile Header processing (total 3 files)...
[1/3] Processing Datafile Header: /data/20251014HF/oradata/system01.dbf (File number: 1)
  - Skipping file number 1 (control file)
---------------------------------------------------------
[2/3] Processing Datafile Header: /data/20251014HF/oradata/sysaux01.dbf (File number: 3)
  - Backing up Datafile header...
  - Executing Datafile Header modification with block size 8192...
  - Datafile Header processing completed
---------------------------------------------------------
Cleaning up temporary files...
================= All operations completed =================

Note: Execute /tmp/modify_scn/backup/recover_datafile.sh operation for rollback

[oracle@oracledb check_db]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Oct 15 16:15:12 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> recover datafile 3;
Media recovery complete.

然后尝试打开数据库报ORA-01092 ORA-01578等错误

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00604: error occurred at recursive SQL level 2
ORA-01578: ORACLE data block corrupted (file # 1, block # 132585)
ORA-01110: data file 1: '/data/app/oracle/oradata/ORCL/system01.dbf'
Process ID: 1617
Session ID: 1 Serial number: 5

报错比较明显是由于坏块导致数据库无法打开,进一步检查alert日志

2025-10-14T21:38:22.889985+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2025-10-14T21:38:22.909799+08:00
Starting background process SMCO
2025-10-14T21:38:22.930114+08:00
SMCO started with pid=57, OS id=1630 
2025-10-14T21:38:23.000833+08:00
ARC0 (PID:1616): Archived Log entry 3 added for T-1.S-3205 ID 0x6539a8b7 LAD:1
2025-10-14T21:38:23.524409+08:00
Undo initialization recovery: err:0 start: 19911054 end: 19911123 diff: 69 ms (0.1 seconds)
2025-10-14T21:38:23.826920+08:00
[26765] Successfully onlined Undo Tablespace 2.
Undo initialization online undo segments: err:0 start: 19911123 end: 19911426 diff: 303 ms (0.3 seconds)
Undo initialization finished serial:0 start:19911054 end:19911454 diff:400 ms (0.4 seconds)
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Verifying minimum file header compatibility for tablespace encryption..
Verifying file header compatibility for tablespace encryption completed for pdb 0
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is AL32UTF8
No Resource Manager plan active
2025-10-14T21:38:24.645423+08:00
Hex dump of (file 1, block 132585) in trace file /u01/app/diag/rdbms/orcl/ORCL/trace/ORCL_ora_26765.trc

Invalid temporary block relative dba: 0x004205e9 (file 1, block 132585)
Bad header found during buffer read
Data in bad block:
 type: 66 format: 2 rdba: 0x4c444242
 last change scn: 0x4c44.4242.4c444242 seq: 0x44 flg: 0x4c
 spare3: 0x4c44
 consistency value in tail: 0xfcc60601
 check value in block header: 0x4242
 computed block checksum: 0x9e0a

Reading datafile '/data/app/oracle/oradata/ORCL/system01.dbf' for corrupt data at rdba:0x004205e9(file 1,block 132585)
Reread (file 1, block 132585) found same corrupt data (no logical check)
2025-10-14T21:38:24.838291+08:00
Errors in file /u01/app/diag/rdbms/orcl/ORCL/trace/ORCL_ora_26765.trc  (incident=51502):
ORA-01578: ORACLE data block corrupted (file # 1, block # 132585)
ORA-01110: data file 1: '/data/app/oracle/oradata/ORCL/system01.dbf'
Incident details in: /u01/app/diag/rdbms/orcl/ORCL/incident/incdir_51502/ORCL_ora_26765_i51502.trc
2025-10-14T21:38:24.861113+08:00
Corrupt Block Found
         TIME STAMP (GMT) = 10/14/2025 21:38:23
         CONT = 0, TSN = 0, TSNAME = SYSTEM
         RFN = 1, BLK = 132585, RDBA = 4326889
         OBJN = 66, OBJD = 64, OBJECT = C_OBJ#_INTCOL#, SUBOBJECT = 
         SEGMENT OWNER = SYS, SEGMENT TYPE = Cluster Segment
2025-10-14T21:38:26.554220+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2025-10-14T21:38:26.769742+08:00
Errors in file /u01/app/diag/rdbms/orcl/ORCL/trace/ORCL_ora_26765.trc:
ORA-00604: error occurred at recursive SQL level 2
ORA-01578: ORACLE data block corrupted (file # 1, block # 132585)
ORA-01110: data file 1: '/data/app/oracle/oradata/ORCL/system01.dbf'
2025-10-14T21:38:26.769940+08:00
Errors in file /u01/app/diag/rdbms/orcl/ORCL/trace/ORCL_ora_26765.trc:
ORA-00604: error occurred at recursive SQL level 2
ORA-01578: ORACLE data block corrupted (file # 1, block # 132585)
ORA-01110: data file 1: '/data/app/oracle/oradata/ORCL/system01.dbf'
Error 604 happened during db open, shutting down database
Errors in file /u01/app/diag/rdbms/orcl/ORCL/trace/ORCL_ora_26765.trc  (incident=51503):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00604: error occurred at recursive SQL level 2
ORA-01578: ORACLE data block corrupted (file # 1, block # 132585)
ORA-01110: data file 1: '/data/app/oracle/oradata/ORCL/system01.dbf'
Incident details in: /u01/app/diag/rdbms/orcl/ORCL/incident/incdir_51503/ORCL_ora_26765_i51503.trc
Invalid temporary block relative dba: 0x004205e9 (file 1, block 132585)
Bad header found during validation
Data in bad block:
 type: 66 format: 2 rdba: 0x4c444242
 last change scn: 0x4c44.4242.4c444242 seq: 0x44 flg: 0x4c
 spare3: 0x4c44
 consistency value in tail: 0xfcc60601
 check value in block header: 0x4242
 computed block checksum: 0x9e0a

Reread of blocknum=132585, file=/data/app/oracle/oradata/ORCL/system01.dbf. found same corrupt data
Reread of blocknum=132585, file=/data/app/oracle/oradata/ORCL/system01.dbf. found same corrupt data
Reread of blocknum=132585, file=/data/app/oracle/oradata/ORCL/system01.dbf. found same corrupt data
Reread of blocknum=132585, file=/data/app/oracle/oradata/ORCL/system01.dbf. found same corrupt data
Reread of blocknum=132585, file=/data/app/oracle/oradata/ORCL/system01.dbf. found same corrupt data
Checker run found 1 new persistent data failures
2025-10-14T21:38:27.515191+08:00
opiodr aborting process unknown ospid (26765) as a result of ORA-603
2025-10-14T21:38:27.528063+08:00
ORA-603 : opitsk aborting process
License high water mark = 18
USER (ospid: (prelim)): terminating the instance due to ORA error 
2025-10-14T21:38:28.576271+08:00
Instance terminated by USER(prelim), pid = 26765

通过分析alert日志可以确定是由于file 1, block 132585损坏,对应的对象为C_OBJ#_INTCOL#,该对象是histgrm$表的cluster,非数据库必须核心对象,可以在启动的时候跳过该对象,启动数据库,然后设置event对该对象进行处理

SQL> alter database open;

Database altered.

SQL>  create table good_histgrm$ as select * from histgrm$;

Table created.

SQL> truncate cluster c_obj#_intcol#;

Cluster truncated.

SQL> insert into histgrm$ select * from good_histgrm$ ;

286071 rows created.

SQL> commit;

Commit complete.

SQL> select /*+ full(a) */ count(*) from histgrm$ a ;

  COUNT(*)
----------
    286071

SQL> select /*+ full(a) */ count(*) from histgrm$ a ;

  COUNT(*)
----------
    286071

至此整体上完成该库的恢复任务

ORA-600 kkkicreatecgmap:!efn3

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 kkkicreatecgmap:!efn3

raid故障恢复之后,数据库recover成功,但是open报ORA-03113: end-of-file on communication channel错误

SQL> recover database;
Media recovery complete.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 20394
Session ID: 191 Serial number: 3

对应的alert日志错误为ORA-600 [kkkicreatecgmap:!efn3]错误

ALTER DATABASE RECOVER  database
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 4 slaves
Mon Oct 20 18:51:06 2025
Recovery of Online Redo Log: Thread 1 Group 1 Seq 32119 Reading mem 0
  Mem# 0: /u01/oradata/redo01.log
Media Recovery Complete (orcl)
Completed: ALTER DATABASE RECOVER  database
Mon Oct 20 18:51:16 2025
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 3 processes
Started redo scan
Completed redo scan
 read 41 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 32119, block 34978
Recovery of Online Redo Log: Thread 1 Group 1 Seq 32119 Reading mem 0
  Mem# 0: /u01/oradata/redo01.log
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 32119, block 35061, scn 17375938230308
 0 data blocks read, 0 data blocks written, 41 redo k-bytes read
Mon Oct 20 18:51:16 2025
Thread 1 advanced to log sequence 32120 (thread open)
Thread 1 opened at log sequence 32120
  Current log# 2 seq# 32120 mem# 0: /u01/oradata/redo02.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Oct 20 18:51:16 2025
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Re-creating tempfile /u01/oradata/temp01.dbf
Database Characterset is ZHS16GBK
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x3999DC33][PC:0x2297750, kgegpa()+40][flags: 0x0, count: 1]
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x3999DC33][PC:0x229597B, kgebse()+279][flags: 0x2, count: 2]
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x3999DC33][PC:0x229597B, kgebse()+279][flags: 0x2, count: 2]
No Resource Manager plan active
Mon Oct 20 18:51:16 2025
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20286.trc  (incident=3649):
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_3649/orcl_dbrm_20286_i3649.trc
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20286.trc:
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
DBRM (ospid: 20286): terminating the instance due to error 56710
Instance terminated by DBRM, pid = 20286

对应的trace文件内容


----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
skdstdst()+36        call     kgdsdst()            000000000 ? 000000000 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   7FFD08773E78 ? 000000000 ?
ksedst1()+98         call     skdstdst()           000000000 ? 000000000 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
ksedst()+34          call     ksedst1()            000000000 ? 000000001 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbkedDefDump()+2736  call     ksedst()             000000000 ? 000000001 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
ksedmp()+36          call     dbkedDefDump()       000000003 ? 000000002 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
ksfdmp()+64          call     ksedmp()             000000003 ? 000000002 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbgexPhaseII()+1764  call     ksfdmp()             000000003 ? 000000002 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbgexProcessError()  call     dbgexPhaseII()       7F16CCF3E6F0 ? 7F16CA6F2598 ?
+2279                                              7FFD0877BC68 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbgeExecuteForError  call     dbgexProcessError()  7F16CCF3E6F0 ? 7F16CA6F2598 ?
()+83                                              000000001 ? 000000000 ?
                                                   7FFD00000000 ? 000000000 ?
dbgePostErrorKGE()+  call     dbgeExecuteForError  7F16CCF3E6F0 ? 7F16CA6F2598 ?
1615                          ()                   000000001 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbkePostKGE_kgsf()+  call     dbgePostErrorKGE()   000000000 ? 7F16CA560040 ?
63                                                 000000258 ? 7F16CA6F2598 ?
                                                   000000000 ? 000000000 ?
kgeadse()+383        call     dbkePostKGE_kgsf()   00A99D360 ? 7F16CA560040 ?
                                                   000000258 ? 7F16CA6F2598 ?
                                                   000000000 ? 000000000 ?
kgerinv_internal()+  call     kgeadse()            00A99D360 ? 7F16CA560040 ?
45                                                 000000258 ? 000000000 ?
                                                   000000000 ? 000000000 ?
kgerinv()+33         call     kgerinv_internal()   00A99D360 ? 7F16CA560040 ?
                                                   877420000000000 ? 000000258 ?
                                                   000000000 ? 000000000 ?
kgeasnmierr()+143    call     kgerinv()            00A99D360 ? 7F16CA560040 ?
                                                   877420000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
kkkicreatecgmap()+3  call     kgeasnmierr()        00A99D360 ? 7F16CA560040 ?
125                                                877420000000000 ? 000000000 ?
                                                   000000000 ? 00000057B ?
kskirefreshcgmap()+  call     kkkicreatecgmap()    121DC3050 ? 7F16CA560040 ?
104                                                877420000000000 ? 000000000 ?
                                                   000000000 ? 00000057B ?
kskreload()+1150     call     kskirefreshcgmap()   121DC3050 ? 7F16CA560040 ?
                                                   877420000000000 ? 000000000 ?
                                                   000000000 ? 00000057B ?
kskdbrmpa()+378      call     kskreload()          7FFD0877D428 ? 000000001 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
ksbabs()+465         call     kskdbrmpa()          7FFD0877D418 ? 000000058 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
ksbrdp()+923         call     ksbabs()             7FFD0877D418 ? 000000058 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
opirip()+618         call     ksbrdp()             7FFD0877D418 ? 000000058 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
opidrv()+598         call     opirip()             000000032 ? 000000004 ?
                                                   7FFD0877E598 ? 000000001 ?
                                                   000000000 ? 00000020A ?
sou2o()+98           call     opidrv()             000000032 ? 000000004 ?
                                                   7FFD0877E598 ? 000000001 ?
                                                   000000000 ? 00000020A ?
opimai_real()+261    call     sou2o()              7FFD0877E570 ? 000000032 ?
                                                   000000004 ? 7FFD0877E598 ?
                                                   000000000 ? 00000020A ?
ssthrdmain()+209     call     opimai_real()        000000000 ? 7FFD0877E760 ?
                                                   000000004 ? 7FFD0877E598 ?
                                                   000000000 ? 00000020A ?
main()+196           call     ssthrdmain()         000000003 ? 7FFD0877E760 ?
                                                   000000001 ? 000000000 ?
                                                   000000000 ? 00000020A ?
__libc_start_main()  call     main()               000000003 ? 7FFD0877E900 ?
+253                                               000000001 ? 000000000 ?
                                                   000000000 ? 00000020A ?
_start()+36          call     __libc_start_main()  0009D3D94 ? 000000001 ?
                                                   7FFD0877E8F8 ? 000000000 ?
                                                   000000000 ? 00000020A ?

虽然ORA-600 kkkicreatecgmap:!efn3没有见过但是数据库open过程中kgegpa、kgebse的故障还是遇到不少,大部分可能和undo有一定关系,处理undo问题之后,继续尝试open库,依旧报ORA-03113: end-of-file on communication channel

SQL> alter database Open;
ERROR:
ORA-03113: end-of-file on communication channel
Process ID: 20586
Session ID: 191 Serial number: 3

但是后台的alert日志已经改变ORA-600 4193,ORA-600 kkkicreatecgmap:!efn3错误

Mon Oct 20 18:54:39 2025
Thread 1 advanced to log sequence 32121 (thread open)
Thread 1 opened at log sequence 32121
  Current log# 3 seq# 32121 mem# 0: /u01/oradata/redo03.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Oct 20 18:54:39 2025
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_smon_20492.trc  (incident=4905):
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_4905/orcl_smon_20492_i4905.trc
No Resource Manager plan active
Mon Oct 20 18:54:39 2025
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20478.trc  (incident=4849):
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_4849/orcl_dbrm_20478_i4849.trc
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20478.trc:
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
DBRM (ospid: 20478): terminating the instance due to error 56710
Some DDE async actions failed or were cancelled
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_smon_20492.trc  (incident=4906):
ORA-00353: log corruption near block 8 change 17375938190767 time 10/03/2025 00:20:34
ORA-00312: online log 1 thread 1: '/u01/oradata/redo01.log'
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_4906/orcl_smon_20492_i4906.trc
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_smon_20492.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 8 change 17375938190767 time 10/03/2025 00:20:34
ORA-00312: online log 1 thread 1: '/u01/oradata/redo01.log'
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Instance terminated by DBRM, pid = 20478

虽然ORA-600 kkkicreatecgmap:!efn3还在,但是看到了比较熟悉的ORA-600 4193错误,处理undo异常回滚段,数据库open成功,重建undo,尝试导出数据,完成数据恢复任务.

补充说明,对于ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403]网络上没有任何资料,查询了mos发现一个有一点类似的报错信息
Bug 28167557 – bigscn_dbim_tm_def – trc – kkkdchkcriticalobj – ORA-700 [kkkdchkcriticalob (Doc ID 28167557.8)

Description
Information about new symptoms:
 
- Signaling function: kkkdchkcriticalobj (kkkd.c)
- Symptom: ORA-700 [kkkdchkcriticalobj:fail]
- Owner: SUELEE
- Special Run:  BIGSCN_DBIM_TM_DEF (owner: WAI-SZE.TAM, sr_id: 2709)
- Release: 19.1
- Job id: 22488605
- Base label:  RDBMS_MAIN_LINUX.X64_180607
 
ORA-600 [KKKICHKRMAPPRI:0=NR PEND] 
 ORA-600 kkkicreatecgmap:!group
 ORA-700 kkkdchkcriticalobj
  
 ORA-600 [KKKICHKRMAPPRI:0=NR PEND] 
 ORA-600 kkkicreatecgmap:!group
 ORA-700 kkkdchkcriticalobj
  
 
 REDISCOVERY INFORMATION:
 Symptoms are any of the following:
 ORA-600 [KKKICHKRMAPPRI:0=NR PEND] 
 ORA-600 kkkicreatecgmap:!group
 ORA-700 kkkdchkcriticalobj
  
 .
 WORKAROUND:
 None

由于这个是硬件故障恢复出来的数据文件（涉及磁盘坏道，磁盘顺序，磁盘在raid中均衡,raid的cache等因素可能会一起Oracle各种非常规问题）,分析原因意义不大,重点是快速解决问题,不做过多分析

Oracle 19c 202510补丁（RUs+OJVM）-19.29

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle 19c 202510补丁（RUs+OJVM）-19.29

Description	Database Update	GI Update	Windows Bundle Patch
OCT2025 (19.29.0.0.0.)	38291812	38298204	38111211
JUL2025 (19.28.0.0.0)	37960098	37957391	37962957
APR2025 (19.27.0.0.0)	37642901	37641958	37532350
JAN2025 (19.26.0.0.0)	37260974	37257886	37486199
OCT2024 (19.25.0.0.0)	36912597	36916690	36878821
JUL2024 (19.24.0.0.0)	36582781	36582629	36521936
APR2024 (19.23.0.0.0)	36233263	36233126	36219938
JAN2024 (19.22.0.0.0)	35943157	35940989	35962832
OCT2023 (19.21.0.0.0)	35643107	35642822	35681552
JUL2023 (19.20.0.0.0)	35320081	35319490	35348034
APR2023 (19.19.0.0.0)	35042068	35037840	35046439
JAN2023 (19.18.0.0.0)	34765931	34762026	34750795
Oct2022 (19.17.0.0.0)	34419443	34416665	34468114
JUL2022 (19.16.0.0.0)	34133642	34130714	34110685
APR2022 (19.15.0.0.0)	33806152	33803476	33829175
JAN2022 (19.14.0.0.0)	33515361	33509923	33575656
OCT2021(19.13.0.0.0)	33192793	33182768	33155330
JUL2021 (19.12.0.0.0)	32904851	32895426	32832237
APR2021 (19.11.0.0.0)	32545013	32545008	32409154
JAN2021 (19.10.0.0.0)	32218454	32226239	32062765
OCT2020 (19.9.0.0.0)	31771877	31750108	31719903
JUL2020 (19.8.0.0.0)	31281355	31305339	31247621
APR2020 (19.7.0.0.0)	30869156	30899722	30901317
JAN2020 (19.6.0.0.0)	30557433	30501910	30445947
OCT2019 (19.5.0.0.0)	30125133	30116789	30151705
JUL2019 (19.4.0.0.0)	29834717	29708769	NA
APR2019 (19.3.0.0.0)	29517242	29517302	NA

Description	OJVM Update	OJVM + DB Update	OJVM + GI Update
OCT2025 (19.29.0.0.251021)	38194382	38273545	38273558
JUL2025 (19.28.0.0.250715)	37847857	37952354	37952382
APR2025 (19.27.0.0.250415)	37499406	37591483	37591516
JAN2025 (19.26.0.0.250121)	37102264	37262172	37262208
OCT2024 (19.25.0.0.241015)	36878697	36866623	36866740
JUL2024 (19.24.0.0.240716)	36414915	36522340	36522439
APR2024 (19.23.0.0.240416)	36199232	36209492	36209493
JAN2024 (19.22.0.0.240116)	35926646	36031426	36031453
OCT2023 (19.21.0.0.231017)	35648110	35742413	35742441
JUL2023 (19.20.0.0.230718)	35354406	35370174	35370167
APR2023 (19.19.0.0.230418)	35050341	35058163	35058172
JAN2023 (19.18.0.0.230117)	34786990	34773489	34773504
OCT2022 (19.17.0.0.221018)	34411846	34449114	34449117
JUL2022 (19.16.0.0.220719)	34086870	34160831	34160854
APR2022 (19.15.0.0.220419)	33808367	33859194	33859214
JAN2022 (19.14.0.0.220118)	33561310	33567270	33567274
OCT2021 (19.13.0.0.211019)	33192694	33248420	33248471
JUL2021 (19.12.0.0.210720)	32876380	32900021	32900083
APR2021 (19.11.0.0.210420)	32399816	32578972	32578973
JAN2021 (19.10.0.0.210119)	32067171	32126828	32126842
OCT2020 (19.9.0.0.201020)	31668882	31720396	31720429
JUL2020 (19.8.0.0.200714)	31219897	31326362	31326369
APR2020 (19.7.0.0.200414)	30805684	30783543	30783556
JAN2020 (19.6.0.0.200114)	30484981	30463595	30463609
OCT2019 (19.5.0.0.191015)	30128191	30133124	30133178
JUL2019 (19.4.0.0.190716)	29774421	29699079	29699097
APR2019 (19.3.0.0.190416)	29548437	29621253	29621299

expdp dmp 导出不完整导入ORA-39059 ORA-39246 故障抢救数据