联系：手机/微信(+86 17813235971) QQ(107644445)
标题：12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
有朋友由于主机断电,导致数据库无法启动,需要我们提供恢复支持.通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)检查分析,发现存放在一个磁盘的数据文件scn过小,数据库无法正常open,需要历史redo(已经被覆盖,而且数据库未归档).
联系：手机/微信(+86 17813235971) QQ(107644445)
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
12C版本使用pdb的Oracle数据库,由于在创建index的过程中强制终止,导致业务大量阻塞,然后重启数据库几次之后直接crash,最后直接无法open成功,报ORA-00600 6856
SQL> alter database open; alter database open * ERROR at line 1: ORA-01578: ORACLE data block corrupted (file # 22, block # 993741) ORA-01110: data file 22: '+DATA/XIFENFEI/96D12F7BA1E2CE57E0532506060A4A2D/DATAFILE/xff01.309.1050943869' ORA-10564: tablespace CWBASEMS01 ORA-01110: data file 22: '+DATA/XIFENFEI/96D12F7BA1E2CE57E0532506060A4A2D/DATAFILE/xff01.309.1050943869' ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 76286 ORA-00607: Internal error occurred while making a change to a data block ORA-00600: internal error code, arguments: [6856], [0], [12], [], [], [], [],[], [], [], [], []
2020-11-14T15:56:50.736722+08:00 Started redo scan 2020-11-14T15:56:50.977023+08:00 Completed redo scan read 82825 KB redo, 10769 data blocks need recovery 2020-11-14T15:56:51.147256+08:00 Started redo application at Thread 1: logseq 120309, block 48, offset 0 Thread 2: logseq 74989, block 2, offset 16, scn 0x00000000f69e1f8d 2020-11-14T15:56:51.151007+08:00 Recovery of Online Redo Log: Thread 1 Group 1 Seq 120309 Reading mem 0 Mem# 0: +DATA/XIFENFEI/ONLINELOG/group_1.262.1023806467 2020-11-14T15:56:51.153989+08:00 Recovery of Online Redo Log: Thread 2 Group 7 Seq 74989 Reading mem 0 Mem# 0: +DATA/XIFENFEI/ONLINELOG/group_7.274.1023806785 Errors in file /u01/app/oracle/……/xifenfei1/trace/xifenfei1_p00d_469777.trc(incident=10079552)(PDBNAME=CDB$ROOT): ORA-00600: internal error code, arguments: [6856], [0], [12], [], [], [], [], [], [], [], [], [] 2020-11-14T15:56:52.089726+08:00 (3):Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details.
这个错误比较明显是由于ORA-600 6856错误导致数据库在启动的时候无法进行实例恢复,出现这个错误原因是由于客户创建index的过程中强制终止引起Bug 17437634 – ORA-1578 or ORA-600 [6856] transient in-memory corruption on TEMP segment during transaction recovery / ROLLBACK (eg: after Ctrl-C) – superseded (Doc ID 17437634.8),屏蔽该文件实例恢复,cdb启动成功,但是pdb无法正常open
SQL> alter session set container=pdb1; Session altered. SQL> alter database open; alter database open * ERROR at line 1: ORA-00603: ORACLE server session terminated by fatal error ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00600: internal error code, arguments: [kcffo_online_pdb_check:fno_system], [3], [], [] Process ID: 224476 Session ID: 13311 Serial number: 59525
这个错误比较明显是由于ORA-600 kcffo_online_pdb_check:fno_system,数据库未正常检测到pdb的system文件导致该问题,通过对pdb的system文件进行操作,让数据库识别到该文件,然后继续open库
SQL> alter database open; alter database open * ERROR at line 1: ORA-00603: ORACLE server session terminated by fatal error ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00600: internal error code, arguments: [kcvfdb_pdb_set_clean_scn: cleanckpt],[3],[4138003978],[4289274940], [2] Process ID: 224476 Session ID: 13311 Serial number: 59525
该错误是由于数据库在恢复的过程中推了scn,触发了oracle 某种bug导致该问题,通过一些操作之后,数据库可以open,尝试temp表空间增加临时数据文件报ORA-00600 [kcffo_add_tmpf-1] 错误（Bug 29379978 – ORA-00600 [kcffo_add_tmpf-1] when trying to add temp file (Doc ID 29379978.8)）.由于该文件无法加入,数据库无法导出
联系：手机/微信(+86 17813235971) QQ(107644445)
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
早上接手一个被人折腾了一个晚上的库,通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)收集信息发现有8个数据文件WRONG FILE TYPE
run{
set newname for datafile 31 to '+ARCH';
set newname for datafile 19 to '+ARCH';
set newname for datafile 20 to '+ARCH';
set newname for datafile 35 to '+ARCH';
set newname for datafile 52 to '+ARCH';
set newname for datafile 16 to '+ARCH';
set newname for datafile 30 to '+ARCH';
set newname for datafile 28 to '+ARCH';
restore datafile 31,19,20,35,52,16,30,28;
switch datafile all;
}
然后catalog注册日志,继续恢复,实现数据0丢失,数据库打开
联系：手机/微信(+86 17813235971) QQ(107644445)
标题：因为人工误操作导致resetlogs scn不一致恢复
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
有个朋友,数据库损坏有一段时间,前期找过我们,对其进行分析,判断强制打开库应该就可以了;后来不知道什么原因有几批人进行了恢复,把现场完全破坏,检查数据库发现,数据库文件有部分resetlogs scn不一致
SQL> recover database until cancel;
ORA-00279: change 15764982275935 generated at 11/06/2020 00:27:23 needed for thread 1
ORA-00289: suggestion : /jiaoh/oradata/orcl/archlog/arch_1_5_995131008.log
ORA-00280: change 15764982275935 for thread 1 is in sequence #5
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 3 needs more recovery to be consistent
ORA-01110: data file 3: '/u01/oracle/oradata/orcl/undotbs01.dbf'
ORA-01112: media recovery not started
SQL> alter database open resetlogs;
Database altered.
数据库alert日志报ORA-600 4097错误
un Nov 08 12:17:06 2020 Errors in file /u01/oracle/diag/rdbms/orcl/orcl/trace/orcl_j006_135739.trc (incident=2604819): ORA-00600: internal error code, arguments: [4097], [31], [2], [338312], [], [], [], [], [], [], [], [] Incident details in: /u01/oracle/diag/rdbms/orcl/orcl/incident/incdir_2604819/orcl_j006_135739_i2604819.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details.
对undo进行处理,数据库后台无明显报错,建议客户逻辑导出数据导入新库,恢复完成.
联系：手机/微信(+86 17813235971) QQ(107644445)
标题：ORA-15196: invalid ASM block header [kfc.c:26368]故障恢复
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
有客户对asm的data磁盘组增加磁盘进行扩容,在做reblance的过程中重启了主机,结果导致data磁盘组mount之后自动dismount
Fri Oct 09 20:48:06 2020 NOTE: PST enabling heartbeating (grp 1) Fri Oct 09 20:48:06 2020 NOTE: ASM did background COD recovery for group 1/0x739536c (DATA) NOTE: starting rebalance of group 1/0x739536c (DATA) at power 10 Starting background process ARB0 Fri Oct 09 20:48:07 2020 ARB0 started with pid=28, OS id=39278 NOTE: assigning ARB0 to group 1/0x739536c (DATA) with 10 parallel I/Os cellip.ora not found. WARNING:cache read a corrupt block:group=1(DATA) dsk=8 blk=7 disk=8(DATA_0008)incarn=3916014506 au=0 blk=7 count=1 Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_39278.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] NOTE: a corrupted block from group DATA was dumped to /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_39278.trc WARNING:cache read(retry)a corrupt block:group=1(DATA) dsk=8 blk=7 disk=8(DATA_0008)incarn=3916014506 au=0 blk=7 count=1 Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_39278.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] ERROR: cache failed to read group=1(DATA) dsk=8 blk=7 from disk(s): 8(DATA_0008) Fri Oct 09 20:48:13 2020 NOTE: GroupBlock outside rolling migration privileged region ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] NOTE: requesting all-instance membership refresh for group=1 NOTE: cache initiating offline of disk 8 group DATA NOTE: process _arb0_+asm1 (39278) initiating offline of disk 8.3916014506 (DATA_0008) with mask 0x7e in group 1 NOTE: initiating PST update: grp = 1, dsk = 8/0xe969a3aa, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 7 for pid 28, osid 39278 ERROR: Disk 8 cannot be offlined, since diskgroup has external redundancy. ERROR: too many offline disks in PST (grp 1) Fri Oct 09 20:48:13 2020 NOTE: cache dismounting (not clean) group 1/0x0739536C (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 39346, image: oracle@rac1 (B000) Fri Oct 09 20:48:13 2020 NOTE: halting all I/Os to diskgroup 1 (DATA) Fri Oct 09 20:48:13 2020 NOTE: LGWR doing non-clean dismount of group 1 (DATA) NOTE: LGWR sync ABA=32.4749 last written ABA 32.4749 WARNING: Offline for disk DATA_0008 in mode 0x7f failed. Fri Oct 09 20:48:13 2020 kjbdomdet send to inst 2 detach from dom 1, sending detach message to inst 2 Fri Oct 09 20:48:13 2020 List of instances: 1 2 Dirty detach reconfiguration started (new ddet inc 2, cluster inc 4) Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_39278.trc (incident=337185): ORA-15335: ASM metadata corruption detected in disk group 'DATA' ORA-15130: diskgroup "DATA" is being dismounted ORA-15066: offlining disk "DATA_0008" in group "DATA" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] Incident details in: /u01/grid/diag/asm/+asm/+ASM1/incident/incdir_337185/+ASM1_arb0_39278_i337185.trc Global Resource Directory partially frozen for dirty detach * dirty detach - domain 1 invalid = TRUE 2341 GCS resources traversed, 0 cancelled Dirty Detach Reconfiguration complete freeing rdom 1 Fri Oct 09 20:48:13 2020 WARNING: dirty detached from domain 1 NOTE: cache dismounted group 1/0x0739536C (DATA)
错误信息比较明显dsk=8 blk=7 au=0 blk=7 的check值不对,本来应该是2190395015现在变为了2182009786,通过kfed分析确实如此
C:\Users\Administrator>kfed read f:/temp/xff/2.dd blkn=7|more kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL kfbh.datfmt: 2 ; 0x003: 0x02 kfbh.block.blk: 7 ; 0x004: blk=7 kfbh.block.obj: 2147483656 ; 0x008: disk=8 kfbh.check: 2182009786 ; 0x00c: 0x820ed3ba kfbh.fcn.base: 2711248 ; 0x010: 0x00295ed0 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdatb.aunum: 2240 ; 0x000: 0x000008c0 kfdatb.shrink: 448 ; 0x004: 0x01c0 kfdatb.ub2pad: 0 ; 0x006: 0x0000 kfdatb.auinfo[0].link.next: 8 ; 0x008: 0x0008 kfdatb.auinfo[0].link.prev: 8 ; 0x00a: 0x0008 kfdatb.auinfo[1].link.next: 12 ; 0x00c: 0x000c kfdatb.auinfo[1].link.prev: 12 ; 0x00e: 0x000c kfdatb.auinfo[2].link.next: 16 ; 0x010: 0x0010 kfdatb.auinfo[2].link.prev: 16 ; 0x012: 0x0010 kfdatb.auinfo[3].link.next: 20 ; 0x014: 0x0014 kfdatb.auinfo[3].link.prev: 20 ; 0x016: 0x0014 kfdatb.auinfo[4].link.next: 24 ; 0x018: 0x0018 kfdatb.auinfo[4].link.prev: 24 ; 0x01a: 0x0018 kfdatb.auinfo[5].link.next: 28 ; 0x01c: 0x001c kfdatb.auinfo[5].link.prev: 28 ; 0x01e: 0x001c kfdatb.auinfo[6].link.next: 32 ; 0x020: 0x0020 kfdatb.auinfo[6].link.prev: 32 ; 0x022: 0x0020 kfdatb.spare: 0 ; 0x024: 0x00000000
修改该值之后,再次mount data磁盘组,报错如下
Sat Oct 10 13:49:22 2020 ARB0 started with pid=28, OS id=10329 NOTE: assigning ARB0 to group 1/0x3759521c (DATA) with 10 parallel I/Os cellip.ora not found. Sat Oct 10 13:49:26 2020 NOTE: GroupBlock outside rolling migration privileged region NOTE: requesting all-instance membership refresh for group=1 WARNING: cache read a corrupt block: group=1(DATA) dsk=8 blk=8 disk=8 (DATA_0008) incarn=3916014011 au=0 blk=8 count=1 Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_10329.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] NOTE: a corrupted block from group DATA was dumped to /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_10329.trc WARNING:cache read(retry)a corrupt block: group=1(DATA)dsk=8 blk=8 disk=8(DATA_0008)incarn=3916014011 au=0 blk=8 count=1 Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_10329.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] ERROR: cache failed to read group=1(DATA) dsk=8 blk=8 from disk(s): 8(DATA_0008) ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] NOTE: cache initiating offline of disk 8 group DATA NOTE: process _arb0_+asm1 (10329) initiating offline of disk 8.3916014011 (DATA_0008) with mask 0x7e in group 1 NOTE: initiating PST update: grp = 1, dsk = 8/0xe969a1bb, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 64 for pid 28, osid 10329 ERROR: Disk 8 cannot be offlined, since diskgroup has external redundancy. ERROR: too many offline disks in PST (grp 1) Sat Oct 10 13:49:28 2020 NOTE: cache dismounting (not clean) group 1/0x3759521C (DATA) WARNING: Offline for disk DATA_0008 in mode 0x7f failed. Sat Oct 10 13:49:28 2020 NOTE: halting all I/Os to diskgroup 1 (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 10346, image: oracle@rac1 (B000) Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_10329.trc (incident=363107): ORA-15335: ASM metadata corruption detected in disk group 'DATA' ORA-15130: diskgroup "DATA" is being dismounted ORA-15066: offlining disk "DATA_0008" in group "DATA" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] Incident details in: /u01/grid/diag/asm/+asm/+ASM1/incident/incdir_363107/+ASM1_arb0_10329_i363107.trc
该报错为:dsk=8 blk=7 au=0 blk=8异常,通过kfed查看发现
C:\Users\Administrator>kfed read f:/temp/xff/2.dd blkn=8
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
006BE8C00 00000000 00000000 00000000 00000000 [................]
Repeat 31 times
006BE8E00 012C0000 04AFFC07 003BFFCD 03F15BD0 [..,.......;..[..]
006BE8E10 012BFC30 00000000 00000002 00000002 [0.+.............]
006BE8E20 00008000 00008000 00002000 564F22AF [......... ..."OV]
006BE8E30 5F805293 FFFF0002 0001EF53 00000001 [.R._....S.......]
006BE8E40 545AE384 00000000 00000000 00000001 [..ZT............]
006BE8E50 00000000 0000000B 00000100 0000003C [............<...]
006BE8E60 00000242 0000007B 52438BA0 C44FFA90 [B...{.....CR..O.]
006BE8E70 33B6F381 919E2DBA 00000000 00000000 [...3.-..........]
006BE8E80 00000000 00000000 6361622F 0070756B [......../backup.]
006BE8E90 00000000 00000000 00000000 00000000 [................]
Repeat 2 times
006BE8EC0 00000000 00000000 00000000 03ED0000 [................]
006BE8ED0 00000000 00000000 00000000 00000000 [................]
006BE8EE0 00000008 00000000 00000000 AC3C87D6 [..............<.]
006BE8EF0 F1401174 F4F036BD 274FB92F 00000101 [t.@..6../.O'....]
006BE8F00 0000000C 00000000 545AE384 0002F30A [..........ZT....]
006BE8F10 00000004 00000000 00000000 00007FFF [................]
006BE8F20 02508000 00007FFF 00000001 0250FFFF [..P...........P.]
006BE8F30 00000000 00000000 00000000 00000000 [................]
006BE8F40 00000000 00000000 00000000 08000000 [................]
006BE8F50 00000000 00000000 00000000 001C001C [................]
006BE8F60 00000001 00000000 00000000 00000000 [................]
006BE8F70 00000000 00000004 A9AF72B9 0000003B [.........r..;...]
006BE8F80 00000000 00000000 00000000 00000000 [................]
Repeat 167 times
006BE9A00 00001CC4 00800101 00001CC9 00800101 [................]
006BE9A10 00001CCD 00800101 00001CD2 00800101 [................]
006BE9A20 00001CD7 00800101 00001CDE 00800101 [................]
006BE9A30 00001CE3 00800101 00001CE8 00800101 [................]
006BE9A40 00001CEC 00800101 00000000 00000000 [................]
006BE9A50 00000000 00000000 00000000 00000000 [................]
Repeat 26 times
KFED-00322:Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
该block完全损坏,基本上无直接修复的可能,通过对data 磁盘组进行patch操作,让其mount之后不再dismount
NOTE: GMON heartbeating for grp 1 GMON querying group 1 at 76 for pid 27, osid 14466 NOTE: cache opening disk 0 of grp 1: DATA_0000 path:/dev/emcpowere NOTE: F1X0 found on disk 0 au 2 fcn 0.2708382 NOTE: cache opening disk 1 of grp 1: DATA_0001 path:/dev/emcpowerf NOTE: cache opening disk 2 of grp 1: DATA_0002 path:/dev/emcpowerg NOTE: cache opening disk 3 of grp 1: DATA_0003 path:/dev/emcpowerh NOTE: cache opening disk 4 of grp 1: DATA_0004 path:/dev/emcpoweri NOTE: cache opening disk 5 of grp 1: DATA_0005 path:/dev/emcpowerj NOTE: cache opening disk 6 of grp 1: DATA_0006 path:/dev/emcpowerk NOTE: cache opening disk 7 of grp 1: DATA_0007 path:/dev/emcpowerl NOTE: cache opening disk 8 of grp 1: DATA_0008 path:/dev/emcpowerc NOTE: cache mounting (first) external redundancy group 1/0x47495222 (DATA) Sat Oct 10 13:59:38 2020 * allocate domain 1, invalid = TRUE Sat Oct 10 13:59:38 2020 NOTE: attached to recovery domain 1 NOTE: starting recovery of thread=1 ckpt=53.6778 group=1 (DATA) NOTE: advancing ckpt for group 1 (DATA) thread=1 ckpt=53.6778 NOTE: cache recovered group 1 to fcn 0.2961429 NOTE: redo buffer size is 256 blocks (1053184 bytes) Sat Oct 10 13:59:38 2020 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (DATA) NOTE: LGWR found thread 1 closed at ABA 53.6777 NOTE: LGWR mounted thread 1 for diskgroup 1 (DATA) NOTE: LGWR opening thread 1 at fcn 0.2961429 ABA 54.6778 NOTE: cache mounting group 1/0x47495222 (DATA) succeeded NOTE: cache ending mount (success) of group DATA number=1 incarn=0x47495222 NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1 SUCCESS: diskgroup DATA was mounted SUCCESS: alter diskgroup data mount
然后通过rman备份数据库,删除老磁盘组,创建新磁盘组,恢复数据,实现数据库完美恢复,数据0丢失.
联系：手机/微信(+86 17813235971) QQ(107644445)
标题：ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], []
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
数据库启动报ORA-00600 6006错误
Tue Sep 29 14:31:31 2020 SMON: enabling tx recovery Tue Sep 29 14:31:31 2020 Database Characterset is AL32UTF8 Opening with internal Resource Manager plan where NUMA PG = 2, CPUs = 4 Tue Sep 29 14:31:34 2020 SMON: Restarting fast_start parallel rollback Tue Sep 29 14:31:34 2020 replication_dependency_tracking turned off (no async multimaster replication found) Starting background process QMNC QMNC started with pid=64, OS id=2860 Tue Sep 29 14:31:39 2020 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_p000_1084.trc: ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [] Tue Sep 29 14:31:44 2020 SMON: Parallel transaction recovery slave got internal error SMON: Downgrading transaction recovery to serial Tue Sep 29 14:31:48 2020 Completed: ALTER DATABASE OPEN Tue Sep 29 14:31:48 2020 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_3084.trc: ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [] Tue Sep 29 14:31:48 2020 db_recovery_file_dest_size of 8192 MB is 0.00% used. This is a user-specified limit on the amount of space that will be used by this database for recovery-related files, and does not reflect the amount of space available in the underlying filesystem or ASM diskgroup. Tue Sep 29 14:31:53 2020 ORACLE Instance orcl (pid = 16) - Error 600 encountered while recovering transaction (1, 8) on object 53228. Tue Sep 29 14:31:54 2020 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_3084.trc: ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [] Tue Sep 29 14:32:03 2020 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_3084.trc: ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [] Tue Sep 29 14:32:07 2020 ORACLE Instance orcl (pid = 16) - Error 600 encountered while recovering transaction (1, 8) on object 53228. Tue Sep 29 14:32:07 2020 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_3084.trc: ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [] Tue Sep 29 14:32:13 2020 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_3084.trc: ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [] ORACLE Instance orcl (pid = 16) - Error 600 encountered while recovering transaction (1, 8) on object 53228. Tue Sep 29 14:32:15 2020 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_3084.trc: ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [] Tue Sep 29 14:32:21 2020 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_3084.trc: ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [] Tue Sep 29 14:32:23 2020 ORACLE Instance orcl (pid = 16) - Error 600 encountered while recovering transaction (1, 8) on object 53228. Tue Sep 29 14:32:23 2020 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_3084.trc: ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [] Tue Sep 29 14:32:30 2020 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_3084.trc: ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [] Tue Sep 29 14:32:31 2020 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_pmon_3860.trc: ORA-00474: SMON process terminated with error Tue Sep 29 14:32:31 2020 PMON: terminating instance due to error 474
因为这个错误提示比较明显“ORACLE Instance orcl (pid = 16) – Error 600 encountered while recovering transaction (1, 8) on object 53228.”和以前的文章:ORACLE Instance XFF (pid = 18) – Error 600 encountered while recovering transaction非常相似,由于数据库异常关闭导致事务无法正常回滚.通过屏蔽回滚（event 10513）,然后对相关对象进行处理(表导出数据,重新导入;index 进行重建),可以实现数据库的完美恢复
关于ORA-600 6006 ORA-600 6856解释
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [] Oracle is undoing an index leaf key operation. If the key is not found, --oracle 回滚index leaf key操作,如果这个值不存在,报ORA-600 6006错误 ORA-00600: internal error code, arguments: [6856], [0], [60], [], [], [] SMON is trying to recover a dead transaction. But the undo application runs into an internal error (trying to delete a row that is already deleted). --oracle 的回滚操作尝试删除一个已经删除的记录报ORA-600 6856错误
联系：手机/微信(+86 17813235971) QQ(107644445)
标题：Oracle Recovery Tools解决ORA-00279 ORA-00289 ORA-00280故障
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
对于ORA-00279 ORA-00289 ORA-00280故障,一般是由于缺少日志(比如数据库非归档模式,或者归档日志丢失),导致某些文件无法正常online,对此Oracle Recovery Tool能够一键式快速解决.
模拟故障
非归档环境,模拟datafile 4 offline,然后多次切换日志,导致该文件无法正常online
SQL> select owner,segment_name from dba_segments where tablespace_name='USERS' AND OWNER='SYSTEM';
OWNER SEGMENT_NAME
------------------------------ ------------------------------
SYSTEM T_XIFENFEI
SQL> SELECT COUNT(1) FROM SYSTEM.T_XIFENFEI;
COUNT(1)
----------
87752
SQL> ALTER DATABASE DATAFILE 4 OFFLINE DROP;
Database altered.
SQL> ALTER SYSTEM SWITCH LOGFILE;
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL> archive log list;
Database log mode No Archive Mode
Automatic archival Disabled
Archive destination D:\app\Administrator\product\11.2.0\dbhome_1\RDBMS
Oldest online log sequence 22
Current log sequence 24
SQL> select group#,status,sequence# from v$log;
GROUP# STATUS SEQUENCE#
---------- ---------------- ----------
1 INACTIVE 22
2 INACTIVE 23
3 CURRENT 24
SQL> RECOVER DATAFILE 4;
ORA-00279: change 1525881 generated at 09/06/2020 13:46:17 needed for thread 1
ORA-00289: suggestion :
D:\APP\ADMINISTRATOR\PRODUCT\11.2.0\DBHOME_1\RDBMS\ARC0000000019_1046476315.0001
ORA-00280: change 1525881 for thread 1 is in sequence #19
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
使用Oracle Recovery Tools快速修复
SQL> recover datafile 4;
Media recovery complete.
SQL> alter database datafile 4 online;
Database altered.
SQL> SELECT COUNT(1) FROM SYSTEM.T_XIFENFEI;
COUNT(1)
----------
87752
通过该工具就可以快速解决oracle丢失归档数据文件online的问题
联系：手机/微信(+86 17813235971) QQ(107644445)
标题：Oracle Recovery Tools—202008
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
近期对Oracle Recovery Tools小工具进行了一些完善,增加了一些小功能和修复了一些bug,现在对该小工具进行工具和使用说明:
具有功能
修复单个block 坏块
标记单个block为坏块
查看数据块内容
修改数据块中数据
修复数据文件头SCN信息
修复数据文件头resetlogs 信息
修复数据文件头fuzzy信息
数据块拷贝
软件界面
联系：手机/微信(+86 17813235971) QQ(107644445)
标题：Oracle恢复小工具—Oracle Recovery Tools
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
Oracle Recovery Tools可以实现指定block坏块修复,文件头scn修改,resetlogs scn修复等功能.实现数据库快速open,部分文件online或者坏块简单修复功能,简化的使用bbed操作的繁琐
联系：手机/微信(+86 17813235971) QQ(107644445)
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
有朋友反馈win环境下rac异常,asm无法正常mount,检查日志发现
Fri Jul 03 03:55:46 2020 Errors in file C:\APP\ADMINISTRATOR\diag\asm\+asm\+asm2\trace\+asm2_ora_7004.trc: ORA-15025: could not open disk "\\.\ORCLDISKDATA1" ORA-27041: unable to open file OSD-04002: 无法打开文件 O/S-Error: (OS 2) 系统找不到指定的文件。 Errors in file C:\APP\ADMINISTRATOR\diag\asm\+asm\+asm2\trace\+asm2_ora_7004.trc: ORA-15025: could not open disk "\\.\ORCLDISKDATA1" ORA-27041: unable to open file OSD-04002: 无法打开文件 O/S-Error: (OS 2) 系统找不到指定的文件。 WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 267 in group [2.2254399778] from disk DATA_0000 allocation unit 3502 reason error; if possible, will try another mirror side Errors in file C:\APP\ADMINISTRATOR\diag\asm\+asm\+asm2\trace\+asm2_ora_7004.trc: ORA-15081: failed to submit an I/O operation to a disk Fri Jul 03 03:59:46 2020 Errors in file C:\APP\ADMINISTRATOR\diag\asm\+asm\+asm2\trace\+asm2_ora_7328.trc: ORA-15025: could not open disk "\\.\ORCLDISKDATA1" ORA-27041: unable to open file OSD-04002: 无法打开文件 O/S-Error: (OS 2) 系统找不到指定的文件。 Errors in file C:\APP\ADMINISTRATOR\diag\asm\+asm\+asm2\trace\+asm2_ora_7328.trc: ORA-15025: could not open disk "\\.\ORCLDISKDATA1" ORA-27041: unable to open file OSD-04002: 无法打开文件 O/S-Error: (OS 2) 系统找不到指定的文件。 WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 267 in group [2.2254399778] from disk DATA_0000 allocation unit 3502 reason error; if possible, will try another mirror side Errors in file C:\APP\ADMINISTRATOR\diag\asm\+asm\+asm2\trace\+asm2_ora_7328.trc: ORA-15081: failed to submit an I/O operation to a disk
报错信息比较明显是由于无法找到\\.\ORCLDISKDATA1磁盘,因此异常,通过asmtool查看磁盘信息
C:\app\11.2.0\grid>asmtool -list
NTFS \Device\Harddisk0\Partition3 81920M
NTFS \Device\Harddisk0\Partition4 200000M
NTFS \Device\Harddisk0\Partition5 4293849M
\Device\Harddisk1\Partition2 4062M
\Device\Harddisk2\Partition2 2097022M
ORCLDISKFRA0 \Device\Harddisk3\Partition2 511870M
明显的发现ORCLDISKDATA1磁盘丢失,通过对磁盘dd到本地然后进行分析发现,asm disk header损坏
C:\Users\Administrator>kfed read F:\temp\disk3\1\disk2.dd kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 0 ; 0x001: 0x00 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 0 ; 0x008: file=0 kfbh.check: 0 ; 0x00c: 0x00000000 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 006B38C00 00000000 00000000 00000000 00000000 [................] Repeat 255 times KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0] C:\Users\Administrator>kfed read F:\temp\disk3\1\disk2.dd blkn=2 kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL kfbh.datfmt: 2 ; 0x003: 0x02 kfbh.block.blk: 2 ; 0x004: blk=2 kfbh.block.obj: 2147483648 ; 0x008: disk=0 kfbh.check: 2349305287 ; 0x00c: 0x8c078dc7 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdatb.aunum: 0 ; 0x000: 0x00000000 kfdatb.shrink: 448 ; 0x004: 0x01c0 kfdatb.ub2pad: 0 ; 0x006: 0x0000 kfdatb.auinfo[0].link.next: 8 ; 0x008: 0x0008 kfdatb.auinfo[0].link.prev: 8 ; 0x00a: 0x0008 kfdatb.auinfo[1].link.next: 12 ; 0x00c: 0x000c kfdatb.auinfo[1].link.prev: 12 ; 0x00e: 0x000c kfdatb.auinfo[2].link.next: 456 ; 0x010: 0x01c8 kfdatb.auinfo[2].link.prev: 456 ; 0x012: 0x01c8 kfdatb.auinfo[3].link.next: 488 ; 0x014: 0x01e8 kfdatb.auinfo[3].link.prev: 488 ; 0x016: 0x01e8 kfdatb.auinfo[4].link.next: 24 ; 0x018: 0x0018 kfdatb.auinfo[4].link.prev: 24 ; 0x01a: 0x0018 kfdatb.auinfo[5].link.next: 28 ; 0x01c: 0x001c kfdatb.auinfo[5].link.prev: 28 ; 0x01e: 0x001c kfdatb.auinfo[6].link.next: 552 ; 0x020: 0x0228 kfdatb.auinfo[6].link.prev: 3112 ; 0x022: 0x0c28 kfdatb.spare: 0 ; 0x024: 0x00000000 kfdate[0].discriminator: 1 ; 0x028: 0x00000001 kfdate[0].allo.lo: 0 ; 0x028: XNUM=0x0 kfdate[0].allo.hi: 8388608 ; 0x02c: V=1 I=0 H=0 FNUM=0x0 kfdate[1].discriminator: 1 ; 0x030: 0x00000001 kfdate[1].allo.lo: 0 ; 0x030: XNUM=0x0 kfdate[1].allo.hi: 8388608 ; 0x034: V=1 I=0 H=0 FNUM=0x0 kfdate[2].discriminator: 1 ; 0x038: 0x00000001 kfdate[2].allo.lo: 0 ; 0x038: XNUM=0x0 kfdate[2].allo.hi: 8388609 ; 0x03c: V=1 I=0 H=0 FNUM=0x1
fra磁盘虽然磁盘asm label信息存在,但是其他信息依旧损坏,但是也只是磁盘头信息损坏
