联系：手机/微信(+86 17813235971) QQ(107644445)
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
有朋友反馈win环境下rac异常,asm无法正常mount,检查日志发现
Fri Jul 03 03:55:46 2020 Errors in file C:\APP\ADMINISTRATOR\diag\asm\+asm\+asm2\trace\+asm2_ora_7004.trc: ORA-15025: could not open disk "\\.\ORCLDISKDATA1" ORA-27041: unable to open file OSD-04002: 无法打开文件 O/S-Error: (OS 2) 系统找不到指定的文件。 Errors in file C:\APP\ADMINISTRATOR\diag\asm\+asm\+asm2\trace\+asm2_ora_7004.trc: ORA-15025: could not open disk "\\.\ORCLDISKDATA1" ORA-27041: unable to open file OSD-04002: 无法打开文件 O/S-Error: (OS 2) 系统找不到指定的文件。 WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 267 in group [2.2254399778] from disk DATA_0000 allocation unit 3502 reason error; if possible, will try another mirror side Errors in file C:\APP\ADMINISTRATOR\diag\asm\+asm\+asm2\trace\+asm2_ora_7004.trc: ORA-15081: failed to submit an I/O operation to a disk Fri Jul 03 03:59:46 2020 Errors in file C:\APP\ADMINISTRATOR\diag\asm\+asm\+asm2\trace\+asm2_ora_7328.trc: ORA-15025: could not open disk "\\.\ORCLDISKDATA1" ORA-27041: unable to open file OSD-04002: 无法打开文件 O/S-Error: (OS 2) 系统找不到指定的文件。 Errors in file C:\APP\ADMINISTRATOR\diag\asm\+asm\+asm2\trace\+asm2_ora_7328.trc: ORA-15025: could not open disk "\\.\ORCLDISKDATA1" ORA-27041: unable to open file OSD-04002: 无法打开文件 O/S-Error: (OS 2) 系统找不到指定的文件。 WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 267 in group [2.2254399778] from disk DATA_0000 allocation unit 3502 reason error; if possible, will try another mirror side Errors in file C:\APP\ADMINISTRATOR\diag\asm\+asm\+asm2\trace\+asm2_ora_7328.trc: ORA-15081: failed to submit an I/O operation to a disk
报错信息比较明显是由于无法找到\\.\ORCLDISKDATA1磁盘,因此异常,通过asmtool查看磁盘信息
C:\app\11.2.0\grid>asmtool -list
NTFS \Device\Harddisk0\Partition3 81920M
NTFS \Device\Harddisk0\Partition4 200000M
NTFS \Device\Harddisk0\Partition5 4293849M
\Device\Harddisk1\Partition2 4062M
\Device\Harddisk2\Partition2 2097022M
ORCLDISKFRA0 \Device\Harddisk3\Partition2 511870M
明显的发现ORCLDISKDATA1磁盘丢失,通过对磁盘dd到本地然后进行分析发现,asm disk header损坏
C:\Users\Administrator>kfed read F:\temp\disk3\1\disk2.dd kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 0 ; 0x001: 0x00 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 0 ; 0x008: file=0 kfbh.check: 0 ; 0x00c: 0x00000000 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 006B38C00 00000000 00000000 00000000 00000000 [................] Repeat 255 times KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0] C:\Users\Administrator>kfed read F:\temp\disk3\1\disk2.dd blkn=2 kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL kfbh.datfmt: 2 ; 0x003: 0x02 kfbh.block.blk: 2 ; 0x004: blk=2 kfbh.block.obj: 2147483648 ; 0x008: disk=0 kfbh.check: 2349305287 ; 0x00c: 0x8c078dc7 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdatb.aunum: 0 ; 0x000: 0x00000000 kfdatb.shrink: 448 ; 0x004: 0x01c0 kfdatb.ub2pad: 0 ; 0x006: 0x0000 kfdatb.auinfo[0].link.next: 8 ; 0x008: 0x0008 kfdatb.auinfo[0].link.prev: 8 ; 0x00a: 0x0008 kfdatb.auinfo[1].link.next: 12 ; 0x00c: 0x000c kfdatb.auinfo[1].link.prev: 12 ; 0x00e: 0x000c kfdatb.auinfo[2].link.next: 456 ; 0x010: 0x01c8 kfdatb.auinfo[2].link.prev: 456 ; 0x012: 0x01c8 kfdatb.auinfo[3].link.next: 488 ; 0x014: 0x01e8 kfdatb.auinfo[3].link.prev: 488 ; 0x016: 0x01e8 kfdatb.auinfo[4].link.next: 24 ; 0x018: 0x0018 kfdatb.auinfo[4].link.prev: 24 ; 0x01a: 0x0018 kfdatb.auinfo[5].link.next: 28 ; 0x01c: 0x001c kfdatb.auinfo[5].link.prev: 28 ; 0x01e: 0x001c kfdatb.auinfo[6].link.next: 552 ; 0x020: 0x0228 kfdatb.auinfo[6].link.prev: 3112 ; 0x022: 0x0c28 kfdatb.spare: 0 ; 0x024: 0x00000000 kfdate[0].discriminator: 1 ; 0x028: 0x00000001 kfdate[0].allo.lo: 0 ; 0x028: XNUM=0x0 kfdate[0].allo.hi: 8388608 ; 0x02c: V=1 I=0 H=0 FNUM=0x0 kfdate[1].discriminator: 1 ; 0x030: 0x00000001 kfdate[1].allo.lo: 0 ; 0x030: XNUM=0x0 kfdate[1].allo.hi: 8388608 ; 0x034: V=1 I=0 H=0 FNUM=0x0 kfdate[2].discriminator: 1 ; 0x038: 0x00000001 kfdate[2].allo.lo: 0 ; 0x038: XNUM=0x0 kfdate[2].allo.hi: 8388609 ; 0x03c: V=1 I=0 H=0 FNUM=0x1
fra磁盘虽然磁盘asm label信息存在,但是其他信息依旧损坏,但是也只是磁盘头信息损坏
通过现场分析,基本上可以确定是由于某种原因导致win asm 的磁盘的所有磁盘头都损坏(两个磁盘头被置空,另外一个磁盘头基本上损坏),基于原因未知
基于客户现场的情况,以及他们有前一天的rman备份,而且客户有保障现场(进一步故障原因分析)的需求,未在现场环境进行恢复,而是在不对现场环境做任何修改的情况下,直接恢复fra里面的redo和归档日志,进而结合备份异地实现数据库恢复,实现数据0丢失,又不破坏现场的效果
以前遇到过类似我其他操作系统平台中asm disk header异常的case:
asm磁盘分区丢失恢复
pvid=yes导致asm无法mount
asm磁盘头全部损坏数据0丢失恢复
分区无法识别导致asm diskgroup无法mount
asm disk误设置pvid导致asm diskgroup无法mount恢复