国产信创库fio破坏主备库以及备份故障处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:国产信创库fio破坏主备库以及备份故障处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

去年恢复过一个case(达梦数据库,主备节点同时被fio破坏fio测试io,导致磁盘文件系统损坏故障恢复),这两天又接到一个类似的case

故障背景描述
1. 客户达梦数据库运行在arm服务器的linux环境
2. 两台机器做了达梦的DataWatch(类似oracle的dataguard)
3. 主库上有多份数据库备份,但是备份和数据库文件放都放在同一块磁盘的同一个分区上
4. 由于数据库运行较慢,应用厂商先在主库上进行了fio性能测试,然后数据库发生自动切换,切换到备库(当时现场没有发现异常),然后还继续在主库上进行了一次fio测试
5. 再在数据库备库(现在已经切换为主库)的机器上又进行了一次fio测试,然后数据库也异常,至此达梦的主备环境全部异常,三次的fio具体操作命令:

fio --direct=1 --iodepth=32 --rw=randrw --rwmixread=70 --bs=4k --ioengine=libaio --numjobs=4 --group_reporting \
--time_based --runtime=60 --filename=/dev/vdb --name=4k_iops_test

6. 根据当时测试的fio结果显示主库两次测试一共写入数据大概2G左右,备库写入大概1.2G左右数据
7. 当前数据库的block size设置为32k(也就是说在32k里面随机写中任何4k的数据这个block就破坏)
8. 虽然客户是央企但是由于商务招标等原因导致达梦现在不是他们的数据库中标厂商,而且这个库没有正式授权和任何售后服务,甚至安装实施都非达梦工程师.基于这样的情况,应用厂商通过多种途径联系上达梦管理人员,才给予了一定的技术支持.

恢复过程和思路
1. 对于当前的情况,是选择优先恢复主库还是备库的磁盘考虑:虽然主库破坏写入的多一些,但是由于主库中有多份有效备份,因此考虑先让客户对现场主库环境被破坏的vdb磁盘进行镜像,然后使用专业的软件对文件系统进行分析后,可以直接看到达梦相关的数据文件,而且文件系统元数据较为完整
r1


另外分析备份文件,确认备份文件文件数量正确,而且状态良好
bak

2. 恢复出来达梦相关数据文件和所有有效备份文件,上传到新准备的和以前操作系统,数据库版本一样的机器上,然后进行尝试恢复.
3. 非常不幸所有备份集通过dmrman的check backupset命令检测返回无效备份包,但是达梦官方无法提供进一步信息,这个想通过备份集来恢复的思路基本上走死.
4. 考虑让达梦厂商基于恢复的数据文件强制拉库,但是比较不幸由于fio的随机写入破坏导致拉库过程中报page check error,而且达梦工程师经过多次尝试均无法跳过,推断可能是涉及核心字典对象异常,无法规避,数据库无法打开.
r3
r2
r4

5. 达梦工程师考虑通过dmdul工具进行提取,结果无法成功加载字典信息,反馈给研发说该工具不支持当前数据库版本,短期内无法让工具支持,这个希望也放弃.
6. 基于这种情况,再次对备库的fio破坏的磁盘进行恢复,然后通过备库恢复出来的数据文件和主库的数据文件的启动坏块进行互补,然后由达梦工程师打开数据库.
7. 然后尝试dmexpdp导出数据,由于还有字典异常导致导出失败(虽然可以进一步通过替代的方式修复一些坏块,也许可以绕过但是每次报错一个坏块,然后修复导出效率太低,放弃该方法)
8. 尝试通过dmdbcheck检查全库坏块情况,也非常不幸,这个命令也需要检查很多字典,虽然通过人工修复了一些字典报错数据块,但是还是无法执行,最后放弃
9. 在达梦工程师建议下,他们采用了达梦的数据迁移工具按照表迁移到新库中,对于报错的块进行修复,然后再次尝试迁移,这样不停的尝试完成了大部分的迁移工作
10. 少量表通过block修复之后依旧报坏块,而且达梦工程师那边反馈当前版本无法通过表级别和全局方式跳过坏块,导致这些表如果有一个坏块无法修复,数据就无法正常迁移
11. 对于这种情况,提供给他们类似oracle rowid抢救数据的思路进一步抢救数据(打开的主备库都进行类似操作,然后进行对比获取最大限度的数据恢复)
12. 再由开发商进行整合调试业务,最大限度完成本次恢复任务

故障恢复回顾
1. 在客户没有购买最终授权和服务,甚至可能最终整体出局的情况下,达梦厂商给予了非常大的支持,这点值的表扬
2. 达梦数据库对于异常库的诊断分析功能不够完善,主要体现在:
2.1)在数据库非正常关闭的情况下无法检测数据文件坏块情况,对于这个故障,如果有类似oracle的dbv功能,然后配合脚本可以快速的实现坏块填补,会节省大量的反复尝试报错,然后修复,再尝试,再修复的工作
2.2)数据库在open的过程中提示信息不够明确友好,不便于恢复调试,比如类似oracle数据库open过程的明确日志,如果有整个启动过程的类似10046跟踪到具体sql和数据块更好
2.3)达梦的离线提取工具,对版本依赖太强,没有更好的兼容低版本,使得极端情况下,达梦售后工程师缺少兜底工具和底气
2.4)现场达梦工程师反馈当前客户的达梦数据库版本,无法全局和当个表的跳过坏块,严重不合理
2.5)dmdbcheck工具对系统字典依赖太强,如果部分字典不能被正常解析(比如有坏块),直接导致检查终止,不合理
2.6)在我接触另外一些国产库中,研发的响应速度要比达梦的迅速,也许是当前客户协调的资源不够导致(以前有客户其他国产库故障比这个小,客户也比这个小,但是直接协调不错的研发资源快速解决问题)
3. 又一次在主备库上面同时执行了fio,又是把备份和生产数据放在同一个磁盘上,这些非常不合理,希望所有人引以为戒

.wman扩展名勒索mysql数据库恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:.wman扩展名勒索mysql数据库恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户mysql数据库被勒索加密,扩展名为.[[9BZyIXRkRaQ1F]].[[dawsones@cock.li]].wman
wman


3

通过分析,确认ibd文件破坏较少
2

可以通过人工处理直接打开数据库并导出数据
4

对于类似这种被加密的勒索的数据库(oracle,mysql,sql server)等被加密,需要专业恢复技术支持,请联系我们:
电话/微信:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com
系统安全防护措施建议:
1.多台机器,不要使用相同的账号和口令
2.登录口令要有足够的长度和复杂性,并定期更换登录口令
3.重要资料的共享文件夹应设置访问权限控制,并进行定期备份
4.定期检测系统和软件中的安全漏洞,及时打上补丁。
5.定期到服务器检查是否存在异常。
6.安装安全防护软件,并确保其正常运行。
7.从正规渠道下载安装软件。
8.对不熟悉的软件,如果已经被杀毒软件拦截查杀,不要添加信任继续运行。
9.保存良好的备份习惯,尽量做到每日备份,异地备份。

Oracle数据库被勒索加密一键open工具–OraFHR

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:Oracle数据库被勒索加密一键open工具–OraFHR

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

软件概述
1)OraFHR是由惜分飞自主研发的专业 Oracle 数据库数据文件头重构工具,通过重构文件头直接打开数据库,专为解决由于勒索病毒加密数据文件,导致Oracle数据库正常运行的情况恢复。
2)软件版权:惜分飞(www.xifenfei.com)所有,未经授权不得擅自传播或修改。
3)技术支持(包含软件授权和使用过程中的问题咨询)
QQ:107644445
邮箱:dba@xifenfei.com
微信 / 电话:+86-17813235971
4)软件使用前提:
在数据文件被加密之后,可以通过obet工具(obet实现对数据文件坏块检测功能)进行坏块检测,只有少量block损坏可以通过此工具进行恢复直接打开数据库,如果无法自行评估可以发给我进行评估确认
5)软件下载地址和使用说明
下载地址使用说明

数据文件默认值配置
orafhr1


根据实际情况进行调整,如果不太懂,可以保持默认值
DBID:输入10进制数字,可以参考正常库的v$database.dbid值
DBNAME:输入长度不超过8个字符串,可以参考正常库的v$database.name值
版本输入值参考:同版本正常库偏移量block_size+24-28位置倒序,如果不确定当前数据库版本信息或者无法确认版本值该如何填写可以联系我进行分析

11.2.0.1-11.2.0.3  0B200000
11.2.0.4           0B200400
10.2.0.1           0A200100
10.2.0.3           0A200300
10.2.0.4           0A200400(有些平台依旧为0A200300)
10.2.0.5           0A200500
12.1.0.2           0C100200
12.2.0.1           0C200000
19.x               13000000
21.x               15000000
23.4               17040000  

字符集:可以通过业务或者运维人员确认,如果没有办法确认可以恢复数据库sys.props$表进行确认

选择需要处理的数据文件
orafhr2


正常情况下,选择数据文件之后,会自动根据数据文件中前面block的信息,计算出来Rfile#信息
orafhr3

如果有Rfile#为0,表示需要人工根据实际情况分析进行修改,一般扩展名在选择数据文件的时候,会自动补充完成,如果不对可以人工进行调整.

完善表格中其他记录
orafhr4

选择wrh$_datafile文件,注意列使用|分割开,然后点击补全表格,就会自动根据当前情况进行完善表格相关信息
某些版本或者由于某些原因导致wrh$_datafile信息不全,可以通过ts$,file$进行完善(这种情况选择第二套参照方案,需要注册软件)
orafhr5


重构数据文件头
补全表格完成之后,可以点击重构数据文件头按钮,进行文件头重构。
orafhr6


后续数据库打开操作
点击后续操作命令按钮,自动生成open数据库相关操作语句文件
orafhr7

生成数据库open操作步骤和相关语句
orafhr8

参照这些文件中内容对于修复的文件进行操作,直接open库导出数据

通过alert日志回顾其他dba oracle异常恢复故障处理以及后续open数据库操作

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:通过alert日志回顾其他dba oracle异常恢复故障处理以及后续open数据库操作

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户有一个数据库故障,是其他工程师进行恢复操作,最后搞不定通过朋友介绍找到我的.我通过分析alert日志,大概追述故障经过
1. 数据库断电之后启动报ORA-01172 ORA-01151错误,直接启动数据库失败,从报错看是由于数据库在open过程中前滚redo日志异常导致

Thu Feb 26 06:48:35 2026
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 23 processes
Started redo scan
Completed redo scan
 read 73194 KB redo, 37226 data blocks need recovery
Thu Feb 26 06:48:49 2026
Started redo application at
 Thread 1: logseq 869366, block 3
Recovery of Online Redo Log: Thread 1 Group 2 Seq 869366 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Thu Feb 26 06:48:50 2026
RECOVERY OF THREAD 1 STUCK AT BLOCK 16938 OF FILE 3
Slave exiting with ORA-1172 exception
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_p016_4672.trc:
ORA-01172: recovery of thread 1 stuck at block 16938 of file 3
ORA-01151: use media recovery to recover block, restore backup if needed
Aborting crash recovery due to slave death, attempting serial crash recovery
Beginning crash recovery of 1 threads
Started redo scan
Thu Feb 26 06:49:00 2026
Completed redo scan
 read 73194 KB redo, 37226 data blocks need recovery
Started redo application at
 Thread 1: logseq 869366, block 3
Thu Feb 26 06:49:10 2026
Recovery of Online Redo Log: Thread 1 Group 2 Seq 869366 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
RECOVERY OF THREAD 1 STUCK AT BLOCK 16938 OF FILE 3
Aborting crash recovery due to error 1172
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_9260.trc:
ORA-01172: recovery of thread 1 stuck at block 16938 of file 3
ORA-01151: use media recovery to recover block, restore backup if needed
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_9260.trc:
ORA-01172: recovery of thread 1 stuck at block 16938 of file 3
ORA-01151: use media recovery to recover block, restore backup if needed
ORA-1172 signalled during: alter database open...

2. 尝试recover database报ORA-600 3020、ORA-600 17147、ORA-600 17114、ORA-600 17182等错误,这个报错比较明确是由于redo的block信息和datafile的block不一致,导致实例recover database失败

Thu Feb 26 06:50:54 2026
ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 24 slaves
Thu Feb 26 06:50:57 2026
Recovery of Online Redo Log: Thread 1 Group 2 Seq 869366 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Thu Feb 26 06:50:59 2026
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr0i_9232.trc  (incident=79538):
ORA-00600: internal error code, arguments: [3020], [3], [16934], [12599846], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 3, block# 16934, file offset is 138723328 bytes)
ORA-10564: tablespace UNDOTBS1
ORA-01110: data file 3: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\UNDOTBS01.DBF'
ORA-10560: block type 'KTU UNDO BLOCK'
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_79538\orcl_pr0i_9232_i79538.trc
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr0i_9232.trc  (incident=79539):
ORA-00600: internal error code, arguments: [17114], [0x0381C4C60], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [17182], [0x0381C6948], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [17147], [0x0381C4C60], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [3020], [3], [16934], [12599846], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 3, block# 16934, file offset is 138723328 bytes)
ORA-10564: tablespace UNDOTBS1
ORA-01110: data file 3: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\UNDOTBS01.DBF'
ORA-10560: block type 'KTU UNDO BLOCK'

3. 使用隐含参数尝试强制拉库,报ORA-600 2662错误,导致强制拉库没有成功,这个错误相对比较简单,一般修改数据库scn即可

Thu Feb 26 07:04:39 2026
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 16794964253372
Resetting resetlogs activation ID 1548038913 (0x5c453301)
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_8712.trc:
ORA-00367: checksum error in log file header
ORA-00322: log 1 of thread 1 is not current copy
ORA-00312: online log 1 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG'
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_8712.trc:
ORA-00367: checksum error in log file header
ORA-00322: log 2 of thread 1 is not current copy
ORA-00312: online log 2 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG'
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_8712.trc:
ORA-00367: checksum error in log file header
ORA-00322: log 3 of thread 1 is not current copy
ORA-00312: online log 3 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG'
Thu Feb 26 07:04:49 2026
Setting recovery target incarnation to 3
Thu Feb 26 07:04:50 2026
Assigning activation ID 1753992092 (0x688bcb9c)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Successful open of redo thread 1
Thu Feb 26 07:04:50 2026
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu Feb 26 07:04:50 2026
SMON: enabling cache recovery
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_8712.trc  (incident=82031):
ORA-00600: internal error code, arguments: [2662], [3910], [1642126020], [3910], [1642126047], [4194432]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_8712.trc:
ORA-00600: internal error code, arguments: [2662], [3910], [1642126020], [3910], [1642126047], [4194432]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_8712.trc:
ORA-00600: internal error code, arguments: [2662], [3910], [1642126020], [3910], [1642126047], [4194432]
Error 600 happened during db open, shutting down database
USER (ospid: 8712): terminating the instance due to error 600
Thu Feb 26 07:05:00 2026
Instance terminated by USER, pid = 8712
ORA-1092 signalled during: alter database open resetlogs...
opiodr aborting process unknown ospid (8712) as a result of ORA-1092

4. 尝试重新open库,数据库open成功但是报ORA-600 kturbleurec1、ORA-600 kcbgtcr_13错误,数据库运行一会就直接crash,这个错误一般是由于undo异常导致

Thu Feb 26 07:09:10 2026
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 23 processes
Started redo scan
Completed redo scan
 read 0 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 1, block 3, scn 16794964253378
Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 1, block 3, scn 16794964273379
 0 data blocks read, 0 data blocks written, 0 redo k-bytes read
Thu Feb 26 07:09:14 2026
Thread 1 advanced to log sequence 2 (thread open)
Thread 1 opened at log sequence 2
  Current log# 2 seq# 2 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu Feb 26 07:09:14 2026
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Dictionary check beginning
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
Thu Feb 26 07:09:21 2026
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_p002_10096.trc  (incident=84847):
ORA-00600: internal error code, arguments: [kturbleurec1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_84847\orcl_p002_10096_i84847.trc
Thu Feb 26 07:09:21 2026
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_4600.trc  (incident=84831):
ORA-00600: internal error code, arguments: [kcbgtcr_13], [], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_84831\orcl_ora_4600_i84831.trc
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_4600.trc  (incident=84832):
ORA-00600: internal error code, arguments: [kcbgtcr_13], [], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_84832\orcl_ora_4600_i84832.trc

客户那边还做了各种恢复尝试,最终依旧无法正常open库,让我这边进行恢复支持.由于客户库不大,而且可以提供数据进行恢复,我让客户发生我数据之后,在本地电脑上进行恢复,下载文件之后,重命名相关路径然后尝试open库

SQL> recover database;
ORA-00283: 恢复会话因错误而取消
ORA-16433: 必须以读/写模式打开数据库。

重建控制文件

C:\Users\XFF>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on 星期五 2月 27 14:36:57 2026

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

已连接到空闲例程。

SQL> startup nomount pfile='d:/pfile.txt'
ORACLE 例程已经启动。

Total System Global Area 4275781632 bytes
Fixed Size                  2182592 bytes
Variable Size             973079104 bytes
Database Buffers         3288334336 bytes
Redo Buffers               12185600 bytes
SQL> @rectl.sql

控制文件已创建。

SQL> recover database;
完成介质恢复。

尝试open库报ORA-600 2663错误

SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2663], [3910], [1642323772],
[3910], [1642327019], [], [], [], [], [], [], []
进程 ID: 27296
会话 ID: 14 序列号: 3

使用Patch_SCN工具修改数据库scn
ora-600-2663-patch_scn


再次尝试open数据库

SQL> recover database;
完成介质恢复。
SQL> alter database open ;

数据库已更改。

alert日志报ORA-600 6856错误

Fri Feb 27 14:40:03 2026
QMNC started with pid=62, OS id=18360 
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Completed: alter database open 
Fri Feb 27 14:40:03 2026
Errors in file c:\app\xff\diag\rdbms\orcl\orcl\trace\orcl_p001_21540.trc  (incident=15787):
ORA-00600: 内部错误代码, 参数: [6856], [0], [479], [], [], [], [], [], [], [], [], []
Incident details in: c:\app\xff\diag\rdbms\orcl\orcl\incident\incdir_15787\orcl_p001_21540_i15787.trc
Fri Feb 27 14:40:04 2026
Starting background process CJQ0
Fri Feb 27 14:40:04 2026
CJQ0 started with pid=63, OS id=15592 
Doing block recovery for file 8 block 1541549
Resuming block recovery (PMON) for file 8 block 1541549
Block recovery from logseq 3, block 108 to scn 16794965431265
Recovery of Online Redo Log: Thread 1 Group 3 Seq 3 Reading mem 0
  Mem# 0: H:\BAIDUNETDISK\20260227\REDO03.LOG
Block recovery completed at rba 3.16415.16, scn 3910.1643303906
Fri Feb 27 14:40:04 2026
Trace dumping is performing id=[cdmp_20260227144004]
SMON: ignoring slave err,downgrading to serial rollback
Errors in file c:\app\xff\diag\rdbms\orcl\orcl\trace\orcl_smon_21696.trc  (incident=15723):
ORA-00600: 内部错误代码, 参数: [6856], [0], [479], [], [], [], [], [], [], [], [], []
Incident details in: c:\app\xff\diag\rdbms\orcl\orcl\incident\incdir_15723\orcl_smon_21696_i15723.trc
………………
ORACLE Instance orcl (pid = 15) - Error 607 encountered while recovering transaction (3, 10) on object 81310.
Errors in file c:\app\xff\diag\rdbms\orcl\orcl\trace\orcl_smon_21696.trc:
ORA-00607: 当更改数据块时出现内部错误
ORA-00600: 内部错误代码, 参数: [6856], [0], [479], [], [], [], [], [], [], [], [], []
Process debug not enabled via parameter _debug_enable
Trace dumping is performing id=[cdmp_20260227144011]
PMON (ospid: 23760): terminating the instance due to error 474

该错误是undo异常引起,屏蔽掉异常undo之后,正常open,并顺利导出所有数据,完成本次恢复任务
dmp


年前几例Oracle数据库被加密为.wman的数据库故障恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:年前几例Oracle数据库被加密为.wman的数据库故障恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

年前连续恢复了4个Oracle数据库被勒索加密的case(大小分别为80g,160G,680G,1600G),扩展名是.[[dawsones@cock.li]].wman和[[YATESNET@cock.li]].wman,其中有些库是所有数据文件均被加密并且修改扩展名
1


还有些数据库只是不部分文件扩展名被改变,部分文件保持以前扩展名
2

但是他们都有一个共同的特点,就是这些数据文件均为前面63个block损坏(obet实现对数据文件坏块检测功能)

File #1: D:\app\Administrator\oradata\orcl\BHDATA.DBF (1540353 blocks) - Started: 2026-02-11 14:57:12
File #1: rfile=270931750 (0x10261726)  header_block_num=3344637312 (0xC75B1D80)  filesize_status:NO
file 1, block 0: rdba error (expected 0, got 2951434), bad block
file 1, block 1: rdba error (expected 1, got 298248), bad block
file 1, block 2: rdba error (expected 2, got 1732212), bad block
file 1, block 3: rdba error (expected 3, got 2259600), bad block
file 1, block 4: rdba error (expected 4, got 1628690), bad block
………………
file 1, block 54: rdba error (expected 54, got 1652119), bad block
file 1, block 55: rdba error (expected 55, got 3661515), bad block
file 1, block 56: rdba error (expected 56, got 2412272), bad block
file 1, block 57: rdba error (expected 57, got 3848900), bad block
file 1, block 58: rdba error (expected 58, got 3415387), bad block
file 1, block 59: rdba error (expected 59, got 3649462), bad block
file 1, block 60: rdba error (expected 60, got 1651981), bad block
file 1, block 61: rdba error (expected 61, got 99182), bad block
file 1, block 62: rdba error (expected 62, got 513782), bad block
file 1, block 63: rdba error (expected 63, got 269316), bad block
  File #1 completed: 0 all zero, 0 soft corrupted, 0 tailchk error, 0 checksum error, 64 rdba error

由于被损坏的数据库都是11g版本(而且不是从低版本升级上来的),也就是说实际业务数据是从block 128开始的(block 1为datafile header,block 2-127为数据块分配的位图信息),因此对于这个故障直接通过自研的Oracle数据文件勒索加密恢复工具对文件头进行重构
QQ20250113-220625


然后直接打开数据库,导出数据,但是由于这些库中有大量的xml类型,使得导出有点麻烦,特别是遇到了expdp导出xml列报ORA-22924故障处理,还有一些诡异问题(expdp导出正常,impdp导入报错)进行额外处理

. . 导出了 "ZLCHS"."保险结算记录"                            196.8 MB  463090 行
ORA-31693: 表数据对象 "ZLCHS"."保险结算记录" 无法加载/卸载并且被跳过, 错误如下:
ORA-29913: 执行 ODCIEXTTABLEFETCH 调出时出错
ORA-00600: 内部错误代码, 参数: [qmcxeRemap82], [], [], [], [], [], [], [], [], [], [], []

文件系统损坏导致数据库异常故障处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:文件系统损坏导致数据库异常故障处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户做了双机rose,由于某种故障导致共享存储在两个主机之间相互频繁挂载(甚至出现了同时挂载的情况),使得该文件系统发生损坏
ntfs


修复双机故障之后,数据库启动ORA-01122 ORA-01110 ORA-01200错误
ora-1200

初步看这个报错,block差距有点大,文件头中记录为419840个block,现在实际有的block数量为384000,使用obet查看文件头记录block number情况

OBET> p kcvfh.kcvfhhdr
File: E:\TEMP\20260219\SYSTEM01.DBF
Size: 8192 bytes
Block: 1
Offset: 20

struct kcvfhhdr, 76 bytes                   @20
   ub4 kccfhswv                            @20      0x00000000
   ub4 kccfhcvn                            @24      0x0B200400
   ub4 kccfhdbi                            @28      0x85D98FAB
   text kccfhdbn[8]                        @32-39   XXXX
   ub4 kccfhcsq                            @40      0x00091079
   ub4 kccfhfsz                            @44      0x00066800 <<--16转换为10禁止为419840
   s_blkz kccfhbsz                         @48      0x00
   ub2 kccfhfno                            @52      0x0001
   ub2 kccfhtyp                            @54      0x0003
   ub4 kccfhacid                           @56      0x00000000
   ub4 kccfhcks                            @60      0x00000000
   text kccfhtag[32]                       @64-95

<kcvfh.kcvfhhdr structure printed successfully>

对于这种情况,以前有过很多次处理经验(一般办法2个:1>修改文件头的block数量记录;2>修改现在的文件大小和实际文件有匹配),以前类似的处理记录:
bbed处理ORA-01200故障
记录一次ORA-01200完美恢复
ORA-01122 ORA-01200故障处理
处理完成system文件异常之后,sysaux文件继续异常

SQL> recover datafile 1;
完成介质恢复。
SQL> recover datafile 2;
ORA-00283: 恢复会话因错误而取消
ORA-01110: 数据文件 2: 'Z:\APP\ADMINISTRATOR\ORADATA\XXXX\SYSAUX01.DBF'
ORA-01122: 数据库文件 2 验证失败
ORA-01110: 数据文件 2: 'Z:\APP\ADMINISTRATOR\ORADATA\XXXX\SYSAUX01.DBF'
ORA-01200: 149760 的实际文件大小小于 153600 块的正确大小

类似处理该故障之后,由于文件系统故障导致不少文件出现大量连续坏块(全0或者记录了其他文件内容的坏块),这种是由于文件系统元数据异常导致,通过文件系统层面恢复继续无法正常处理,对于这样的情况,通过碎片扫描工具按照oracle block级别的文件重组(其实就是基于rdba信息进行重组),获取正确的数据块信息然后重新重组成数据文件
QQ20260220-005159


然后打开数据库,顺利导出数据,实现客户数据最大限度恢复

expdp导出xml列报ORA-22924故障处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:expdp导出xml列报ORA-22924故障处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

expdp导出xml列类型表报错

连接到: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
启动 "SYS"."EXPDP_TBA1":  "/******** AS SYSDBA" tables=XFF.XML_TAB dumpfile=XML_TAB.dmp
    DIRECTORY=expdp_dir logfile=expdp_XML_TAB.log EXCLUDE=STATISTICS job_name=expdp_tba1
正在使用 BLOCKS 方法进行估计...
处理对象类型 TABLE_EXPORT/TABLE/TABLE_DATA
使用 BLOCKS 方法的总估计: 13.12 GB
处理对象类型 TABLE_EXPORT/TABLE/TABLE
处理对象类型 TABLE_EXPORT/TABLE/INDEX/INDEX
处理对象类型 TABLE_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
处理对象类型 TABLE_EXPORT/TABLE/CONSTRAINT/REF_CONSTRAINT
ORA-31693: 表数据对象 "XFF"."XML_TAB" 无法加载/卸载并且被跳过, 错误如下:
ORA-29913: 执行 ODCIEXTTABLEPOPULATE 调出时出错
ORA-22924: 快照太旧
已成功加载/卸载了主表 "SYS"."EXPDP_TBA1"
******************************************************************************
SYS.EXPDP_TBA1 的转储文件集为:
  D:\XML_TAB.DMP
作业 "SYS"."EXPDP_TBA1" 已经完成, 但是有 1 个错误 (于 星期六 2月 14 09:01:26 2026 elapsed 0 00:04:15 完成)

一般对于这些问题的解决思路是找出来异常的行的rowid,然后导出跳过异常行或者把对应的xml列置空,关键就是如何找出来该记录,这里自己写了一个plsql来找出来异常rowid

SQL> declare
  2  page number;
  3  len number;
  4  c varchar2(10);
  5  charpp number := 8132/2;
  6  n number;
  7  v_sqlerrm varchar2(100);
  8  begin
  9  n := 0;
 10  for r in (select rowid rid from XFF.XML_TAB order by rowid) loop
 11  begin
 12  select dbms_lob.getlength (xmltype.getclobval(XML_COL)) into c from XFF.XML_TAB     where rowid = r.rid;
 13  v_sqlerrm := SQLERRM;
 14  insert into result values (n, r.rid, 'good', page, v_sqlerrm);
 15  commit;
 16  exception
 17  when others then
 18  dbms_output.put_line ('Error on rowid ' ||r.rid||' page '||page);
 19  dbms_output.put_line (sqlerrm);
 20  v_sqlerrm := SQLERRM;
 21  insert into result values (n, r.rid, 'bad',page,v_sqlerrm);
 22  commit;
 23  end;
 24  end loop;
 25  end;
 26  /
Error on rowid AAASO1AAKAAAH5bAAC page
ORA-22924: 快照太旧
ORA-06512: 在 "SYS.XMLTYPE", line 138
Error on rowid AAASO1AAKAAAIPtAAZ page
ORA-22924: 快照太旧
ORA-06512: 在 "SYS.XMLTYPE", line 138
Error on rowid AAASO1AAKAAAIRfAAU page
ORA-22924: 快照太旧
ORA-06512: 在 "SYS.XMLTYPE", line 138
Error on rowid AAASO1AAKAAAIS9AAW page
ORA-22924: 快照太旧
ORA-06512: 在 "SYS.XMLTYPE", line 138
Error on rowid AAASO1AAKAAAIT7AAW page
ORA-22924: 快照太旧
ORA-06512: 在 "SYS.XMLTYPE", line 138
Error on rowid AAASO1AAKAAAIT/AAG page
ORA-22924: 快照太旧
ORA-06512: 在 "SYS.XMLTYPE", line 138

PL/SQL 过程已成功完成。

然后expdp导出数据跳过这些异常的rowid的par文件

directory=expdp_dir
dumpfile=expdp_XFF.XML_TAB.dmp
logfile=expdp_XFF.XML_TAB.log
tables=XFF.XML_TAB
query="XFF.XML_TAB:WHERE rowid not IN  ('AAASO1AAKAAAH5bAAC','AAASO1AAKAAAIPtAAZ','AAASO1AAKAAAIT7AAW',
'AAASO1AAKAAAIT/AAG', 'AAASO1AAKAAAIRfAAU','AAASO1AAKAAAIS9AAW')"
job_name=exp_1

数据可以正常到导出
expdp


也可以通过update语句直接把异常的xml值直接置空

update XFF.XML_TAB
set XML_COL = XMLType.createXML('')
where rowid in (select checked_rowid from result where status='bad');

obet处理ORA-704 ORA-604 ORA-1578故障

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:obet处理ORA-704 ORA-604 ORA-1578故障

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户数据库启动报ORA-704 ORA-604 ORA-1578错误,导致启动失败
ORA-704-ORA-1578


alert日志报错

Fri Feb 06 23:38:43 2026
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_239325.trc  (incident=1008437):
ORA-01578: ORACLE data block corrupted (file # 1, block # 241)
ORA-01110: data file 1: '+DATADG/orcl/datafile/system.270.1015415183'
Fri Feb 06 23:38:46 2026
Redo thread 2 internally disabled at seq 6 (CKPT)
Archived Log entry 182212 added for thread 2 sequence 5 ID 0xd68b4385 dest 1:
ARC0: Archiving disabled thread 2 sequence 6
Archived Log entry 182213 added for thread 2 sequence 6 ID 0xd68b4385 dest 1:
Thread 1 advanced to log sequence 9 (LGWR switch)
  Current log# 1 seq# 9 mem# 0: +DATADG/orcl/onlinelog/group_1.302.1223816761
  Current log# 1 seq# 9 mem# 1: +ARCHDG/orcl/onlinelog/group_1.257.1223816761
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_239325.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01578: ORACLE data block corrupted (file # 1, block # 241)
ORA-01110: data file 1: '+DATADG/orcl/datafile/system.270.1015415183'
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_239325.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01578: ORACLE data block corrupted (file # 1, block # 241)
ORA-01110: data file 1: '+DATADG/orcl/datafile/system.270.1015415183'
Error 704 happened during db open, shutting down database
USER (ospid: 239325): terminating the instance due to error 704
Fri Feb 06 23:38:48 2026
Instance terminated by USER, pid = 239325
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (239325) as a result of ORA-1092
Fri Feb 06 23:38:48 2026
ORA-1092 : opitsk aborting process

对system文件进行dbv检测(客户通过asmcmd cp命令拷贝出来system文件)

DBVERIFY - Verification starting : FILE = /asm_backup/orcl/datafile/SYSTEM.270.1015415183

DBV-00200: Block, DBA 4194545, already marked corrupt
csc(0x0006.d75a0ed7) higher than block scn(0x0000.00000000)
Page 241 failed with check code 6054

DBV-00200: Block, DBA 4306619, already marked corrupt
csc(0x0006.d75a14f4) higher than block scn(0x0000.00000000)
Page 112315 failed with check code 6054


DBVERIFY - Verification complete

Total Pages Examined         : 738560
Total Pages Processed (Data) : 424109
Total Pages Failing   (Data) : 1
Total Pages Processed (Index): 287403
Total Pages Failing   (Index): 1
Total Pages Processed (Other): 3408
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 23640
Total Pages Marked Corrupt   : 2
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3613010282 (6.3613010282)

这里可以看到有两个block被标记为坏块原因是由于:csc(0×0006.d75a0ed7) higher than block scn(0×0000.00000000),坏块错误代码为:Page 241 failed with check code 6054.这种问题的原因是由于block的scn小于数据块的csc(一般是由于写丢失导致),对于这个问题,最近在obet中刚刚增加了repair blkscn功能进行修复,具体参考:obet修复csc higher than block scn类型坏块,使用工具修复操作(其他block类似修改)

OBET> set filename /datapool/asm_backup/orcl/datafile/SYSTEM.270.1015415183
filename set to: /datapool/asm_backup/orcl/datafile/SYSTEM.270.1015415183

OBET> set block 241
block set to: 241

OBET> backup
Created backup directory: backup_blk
Successfully backed up current block from current file to 
    /tmp/backup_blk/SYSTEM.270.1015415183_241.20260209122822

OBET> repair blkscn

Repairing SCN for block 241...

Current block SCN information:
  scn_base: 0x00000000 (0)
  scn_wrap: 0x0000 (0)

Block 1 reference information:
  hdr_base: 0xD75FF65E (3613390430)
  hdr_wrap: 0x0006 (6)

Current block CSC information:
  csc_base: 0xD75A0ED7 (3613003479)
  csc_wrap: 0x0006 (6)

Calculated values:
  blkcsc: 29382807255 (0x00000006D75A0ED7)
  blkscn: 0 (0x0000000000000000)

blkscn < blkcsc, SCN repair required.

SCN repair requires edit mode.
Please enable edit mode with 'set mode edit' first.

OBET> set mode edit
mode set to: edit

OBET> repair blkscn

Repairing SCN for block 241...

Current block SCN information:
  scn_base: 0x00000000 (0)
  scn_wrap: 0x0000 (0)

Block 1 reference information:
  hdr_base: 0xD75FF65E (3613390430)
  hdr_wrap: 0x0006 (6)

Current block CSC information:
  csc_base: 0xD75A0ED7 (3613003479)
  csc_wrap: 0x0006 (6)

Calculated values:
  blkcsc: 29382807255 (0x00000006D75A0ED7)
  blkscn: 0 (0x0000000000000000)

blkscn < blkcsc, SCN repair required.

New SCN values to write:
  new_scn_base: 0xD75FF65D (3613390429)
  new_scn_wrap: 0x0006 (6)

Confirm SCN repair operations:
File: /datapool/asm_backup/orcl/datafile/SYSTEM.270.1015415183
Block: 241
Operations:
  1. Write new scn_base (0xD75FF65D)
  2. Write new scn_wrap (0x0006)
  3. Fix seq_kcbh (0xFF -> 0x01)

Confirm? (Y/YES to proceed): y

[Undo information saved - use 'undo' command to restore original values]

Verification after SCN repair:
1. scn_base: 5DF65FD7 (expected: 5DF65FD7)
2. scn_wrap: 0600 (expected: 0600)
3. seq_kcbh: 0x01 OK

Block 241 SCN repair completed successfully.
Execute tailchk apply and sum apply for this block.

OBET> tailchk apply

Confirm applying tailchk:
File: /datapool/asm_backup/orcl/datafile/SYSTEM.270.1015415183
Block: 241
Offset in block: 8188 (file offset: 0x001E3FFC)
Original value: 0xFF060000
New value:      0x01065DF6
Confirm? (Y/YES to proceed): y
Verification successful: Stored tailchk matches calculated value (0x01065DF6).
Tailchk applied successfully.

OBET> sum apply

Confirm applying checksum:
File: /datapool/asm_backup/orcl/datafile/SYSTEM.270.1015415183
Block: 241
Offset in block: 16 (file offset: 0x001E2010)
Original value: 0x7330
New value:      0x2AE7
Confirm? (Y/YES to proceed): y
Verification successful: Stored checksum matches calculated value (0x2AE7).
Checksum applied successfully.

然后dbv检查数据文件

[oracle@db4 tmp]$ dbv file=/datapool/asm_backup/orcl/datafile/SYSTEM.270.1015415183

DBVERIFY: Release 11.2.0.4.0 - Production on Mon Feb 9 20:37:54 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE =
         /datapool/asm_backup/orcl/datafile/SYSTEM.270.1015415183


DBVERIFY - Verification complete

Total Pages Examined         : 738560
Total Pages Processed (Data) : 424109
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 287403
Total Pages Failing   (Index): 1
Total Pages Processed (Other): 3408
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 23640
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3613451063 (6.3613451063)

然后直接顺利打开数据库

SQL> alter database open;

Database altered.

由于该库在我接手之前已经做了大量的强制拉库等各种恢复尝试,因此对该库做逻辑导出,导入新库完成本次恢复任务

obet修复csc higher than block scn类型坏块

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:obet修复csc higher than block scn类型坏块

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

最近有客户数据库报ORA-01092 ORA-01578错误导致数据库无法open

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01578: ORACLE data block corrupted (file # 1, block # 2123)
ORA-01110: data file 1: '+DATADG/orcl/datafile/system.270.1015415183'
Process ID: 239325
Session ID: 1599 Serial number: 7

通过dbv检查确认是csc higher than block scn故障

DBVERIFY: Release 11.2.0.4.0 - Production on Sat Feb 7 03:47:12 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = +DATADG/orcl/datafile/SYSTEM.270.1015415183

DBV-00200: Block, DBA 4306619, already marked corrupt
csc(0x0006.d75a14f4) higher than block scn(0x0000.00000000)
Page 2123 failed with check code 6054


DBVERIFY - Verification complete

Total Pages Examined         : 738560
Total Pages Processed (Data) : 424109
Total Pages Failing   (Data) : 1
Total Pages Processed (Index): 287403
Total Pages Failing   (Index): 1
Total Pages Processed (Other): 3408
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 23640
Total Pages Marked Corrupt   : 1
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3613010282 (6.3613010282)

对于这样的故障,最近把他整合到了obet工具中,执行命令为repair blkscn [block x]进行修复

C:\Users\XFF>dbv file=H:\TEMP\SYSTEM01.DBF

DBVERIFY: Release 11.2.0.4.0 - Production on 星期日 2月 8 11:07:51 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = H:\TEMP\SYSTEM01.DBF
csc(0x0012.a0d783c4) higher than block scn(0x0000.00000000)
页 241 失败, 校验代码为 6054

DBVERIFY - 验证完成

检查的页总数: 325120
处理的页总数 (数据): 99921
失败的页总数 (数据): 1
处理的页总数 (索引): 34013
失败的页总数 (索引): 0
处理的页总数 (其他): 174241
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 16945
标记为损坏的总页数: 0
流入的页总数: 0
加密的总页数        : 0
最高块 SCN            : 2698836688 (18.2698836688)

使用obet修复csc higher than block scn 故障

OBET> set filename H:\TEMP\SYSTEM01.DBF
filename set to: H:\TEMP\SYSTEM01.DBF

OBET> set block 241
block set to: 241

OBET> set mode edit
mode set to: edit

OBET> repair blkscn

Repairing SCN for block 241...

Current block SCN information:
  scn_base: 0x00000000 (0)
  scn_wrap: 0x0000 (0)

Block 1 reference information:
  hdr_base: 0xA0DCE449 (2698830921)
  hdr_wrap: 0x0012 (18)

Current block CSC information:
  csc_base: 0xA0D783C4 (2698478532)
  csc_wrap: 0x0012 (18)

Calculated values:
  blkcsc: 80007889860 (0x00000012A0D783C4)
  blkscn: 0 (0x0000000000000000)

blkscn < blkcsc, SCN repair required.

New SCN values to write:
  new_scn_base: 0xA0DCE448 (2698830920)
  new_scn_wrap: 0x0012 (18)

Confirm SCN repair operations:
File: H:\TEMP\SYSTEM01.DBF
Block: 241
Operations:
  1. Write new scn_base (0xA0DCE448)
  2. Write new scn_wrap (0x0012)

Confirm? (Y/YES to proceed): y

[Undo information saved - use 'undo' command to restore original values]

Verification after SCN repair:
1. scn_base: 48E4DCA0 (expected: 48E4DCA0)
2. scn_wrap: 1200 (expected: 1200)
3. seq_kcbh: 0x01 OK

Block 241 SCN repair completed successfully.
Execute tailchk apply and sum apply for this block.

OBET> tailchk apply

Confirm applying tailchk:
File: H:\TEMP\SYSTEM01.DBF
Block: 241
Offset in block: 8188 (file offset: 0x001E3FFC)
Original value: 0x01060000
New value:      0x010648E4
Confirm? (Y/YES to proceed): y
Verification successful: Stored tailchk matches calculated value (0x010648E4).
Tailchk applied successfully.

OBET> sum apply

Confirm applying checksum:
File: H:\TEMP\SYSTEM01.DBF
Block: 241
Offset in block: 16 (file offset: 0x001E2010)
Original value: 0xAD0F
New value:      0x63AF
Confirm? (Y/YES to proceed): y
Verification successful: Stored checksum matches calculated value (0x63AF).
Checksum applied successfully.

dbv验证该错误已经修复

C:\Users\XFF>dbv file=H:\TEMP\SYSTEM01.DBF

DBVERIFY: Release 11.2.0.4.0 - Production on 星期日 2月 8 11:11:12 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = H:\TEMP\SYSTEM01.DBF


DBVERIFY - 验证完成

检查的页总数: 325120
处理的页总数 (数据): 99921
失败的页总数 (数据): 0
处理的页总数 (索引): 34013
失败的页总数 (索引): 0
处理的页总数 (其他): 174241
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 16945
标记为损坏的总页数: 0
流入的页总数: 0
加密的总页数        : 0
最高块 SCN            : 2698836688 (18.2698836688)

obet工具下载地址:obet下载
obet工具使用说明:obet使用说明

ORA-600 kcratr_nab_less_than_odr和ORA-600 4193故障处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-600 kcratr_nab_less_than_odr和ORA-600 4193故障处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

数据库启动报ORA-600 kcratr_nab_less_than_odr

alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 2930 KB redo, 1357 data blocks need recovery
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_19096.trc  (incident=576634):
ORA-00600: ??????, ??: [kcratr_nab_less_than_odr], [1], [7859], [62180], [63172], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_576634\orcl_ora_19096_i576634.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Aborting crash recovery due to error 600
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_19096.trc:
ORA-00600: ??????, ??: [kcratr_nab_less_than_odr], [1], [7859], [62180], [63172], [], [], [], [], [], [], []
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_19096.trc:
ORA-00600: ??????, ??: [kcratr_nab_less_than_odr], [1], [7859], [62180], [63172], [], [], [], [], [], [], []
ORA-600 signalled during: alter database open...

这个错误一般常见的是11.2.0.1的数据库异常关机了容易遇到
又一例ORA-600 kcratr_nab_less_than_odr
ORA-600 kcratr_nab_less_than_odr故障解决
这个客户是11.2.0.4的库(在这个版本中该错误相对较少,虽然也遇到过几次)
11.2.0.4库中遇到ORA-600 kcratr_nab_less_than_odr报错
分析日志发现是由于之前io比较慢导致写入异常导致

Fri Feb 06 00:32:59 2026
********************* ATTENTION: ******************** 
 The controlfile header block returned by the OS
 has a sequence number that is too old. 
 The controlfile might be corrupted.
 PLEASE DO NOT ATTEMPT TO START UP THE INSTANCE 
 without following the steps below.
 RE-STARTING THE INSTANCE CAN CAUSE SERIOUS DAMAGE 
 TO THE DATABASE, if the controlfile is truly corrupted.
 In order to re-start the instance safely, 
 please do the following:
 (1) Save all copies of the controlfile for later 
     analysis and contact your OS vendor and Oracle support.
 (2) Mount the instance and issue: 
     ALTER DATABASE BACKUP CONTROLFILE TO TRACE;
 (3) Unmount the instance. 
 (4) Use the script in the trace file to
     RE-CREATE THE CONTROLFILE and open the database. 
*****************************************************
CKPT (ospid: 2600): terminating the instance
Fri Feb 06 00:33:00 2026
System state dump requested by (instance=1, osid=2600 (CKPT)), summary=[abnormal instance termination].
System State dumped to trace file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_diag_2488_20260206003300.trc
Fri Feb 06 00:33:01 2026
opiodr aborting process unknown ospid (3420) as a result of ORA-1092

处理这个错误相对比较简单,重建控制文件即可
11


但是后续数据库出现ORA-600 4193错误

Fri Feb 06 02:57:34 2026
ALTER DATABASE OPEN
Thread 1 opened at log sequence 7860
  Current log# 1 seq# 7860 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\orcl\REDO01.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
[4044] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:5166812 end:5167140 diff:328 (3 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_smon_3868.trc  (incident=640589):
ORA-00600: 内部错误代码, 参数: [4193], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_640589\orcl_smon_3868_i640589.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Starting background process QMNC
Fri Feb 06 02:57:36 2026
QMNC started with pid=21, OS id=5008 
Completed: ALTER DATABASE OPEN
Fri Feb 06 02:57:37 2026
db_recovery_file_dest_size of 4182 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Block recovery from logseq 7860, block 132954 to scn 55142160294
Recovery of Online Redo Log: Thread 1 Group 1 Seq 7860 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\orcl\REDO01.LOG
Block recovery completed at rba 7860.132998.16, scn 12.3602552743
Block recovery from logseq 7860, block 132954 to scn 55142160257
Recovery of Online Redo Log: Thread 1 Group 1 Seq 7860 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\orcl\REDO01.LOG
Block recovery completed at rba 7860.132954.16, scn 12.3602552707
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_smon_3868.trc:
ORA-01595: 释放区 (5) 回退段 (21) 时出错
ORA-00600: 内部错误代码, 参数: [4193], [], [], [], [], [], [], [], [], [], [], []
Flush retried for xcb 0x96c82600, pmd 0x8c6a70b8
Block recovery from logseq 7860, block 132954 to scn 55142160294
Recovery of Online Redo Log: Thread 1 Group 1 Seq 7860 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\orcl\REDO01.LOG
Block recovery completed at rba 7860.132998.16, scn 12.3602552743
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_pmon_2728.trc:
ORA-00600: 内部错误代码, 参数: [4193], [], [], [], [], [], [], [], [], [], [], []
PMON (ospid: 2728): terminating the instance due to error 472
System state dump requested by (instance=1, osid=2728 (PMON)), summary=[abnormal instance termination].
Fri Feb 06 02:58:17 2026
Instance terminated by PMON, pid = 2728

这个错误相对比较简单,由于undo回滚段异常,处理掉异常undo之后,数据库正常,完成本次恢复任务