又一例system大量坏块恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:又一例system大量坏块恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有朋友找到我们,说数据库服务可以启动,但是无法登陆,类似报错

C:\Users\XIFENFEI>D:\app\XIFENFEI\product\11.2.0.1\dbhome_2\bin\sqlplus / as sys
dba

SQL*Plus: Release 11.2.0.1.0 Production on 星期四 3月 12 15:04:32 2020

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

ERROR:
ORA-01075: 您现在已登录


请输入用户名:
ERROR:
ORA-01017: 用户名/口令无效; 登录被拒绝


请输入用户名:
ERROR:
ORA-01017: 用户名/口令无效; 登录被拒绝


SP2-0157: 在 3 次尝试之后无法连接到 ORACLE, 退出 SQL*Plus

通过分析发现数据库启动报错(未正常open成功)

C:\Users\XIFENFEI>D:\app\XIFENFEI\product\11.2.0.1\dbhome_2\bin\sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on 星期四 3月 12 14:58:28 2020

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

已连接到空闲例程。

SQL> startup mount pfile='F:\pfile.txt';
ORACLE 例程已经启动。

Total System Global Area 3307048960 bytes
Fixed Size                  2180264 bytes
Variable Size            1811942232 bytes
Database Buffers         1476395008 bytes
Redo Buffers               16531456 bytes
数据库装载完毕。

SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-01578: ORACLE 数据块损坏 (文件号 1, 块号 48396)
ORA-01110: 数据文件 1: 'F:\ORADATA\SYSTEM01.DBF'

数据库没有正常启动成功的原因是由于system文件有坏块导致,通过dbv检查system文件发现有大量连续坏块

DBVERIFY: Release 11.2.0.1.0 - Production on 星期四 3月 12 19:12:34 2020

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.


DBVERIFY - 开始验证: FILE = F:\ORADATA\SYSTEM01.DBF
页 48064 流入 - 很可能是介质损坏
Corrupt block relative dba: 0x0040bbc0 (file 1, block 48064)
Fractured block found during dbv: 
Data in bad block:
 type: 6 format: 2 rdba: 0x0040bbc0
 last change scn: 0x0000.0006f69c seq: 0x1 flg: 0x06
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x43455474
 check value in block header: 0x3893
 computed block checksum: 0xadfa

…………

页 48412 标记为损坏
Corrupt block relative dba: 0x0040bd1c (file 1, block 48412)
Bad header found during dbv: 
Data in bad block:
 type: 36 format: 2 rdba: 0x6dce856d
 last change scn: 0xfc44.d24c936f seq: 0xdc flg: 0x3b
 spare1: 0x9c spare2: 0x92 spare3: 0xcf67
 consistency value in tail: 0x43455474
 check value in block header: 0x2598
 block checksum disabled


DBVERIFY - 验证完成

检查的页总数: 249600
处理的页总数 (数据): 209467
失败的页总数 (数据): 0
处理的页总数 (索引): 13616
失败的页总数 (索引): 0
处理的页总数 (其他): 3369
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 22799
标记为损坏的总页数: 349
流入的页总数: 1
加密的总页数        : 0
最高块 SCN            : 84123103 (0.84123103)

数据库alert日志信息

Thu Mar 12 14:52:20 2020
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Hex dump of (file 1, block 48403) in trace file d:\app\xifenfei\diag\rdbms\qdbdc\o11201gbk\trace\o11201gbk_ora_9480.trc
Corrupt block relative dba: 0x0040bd13 (file 1, block 48403)
Bad header found during multiblock buffer read
Data in bad block:
 type: 36 format: 2 rdba: 0x6dce856d
 last change scn: 0xfc44.d24c936f seq: 0xdc flg: 0x3b
 spare1: 0x9c spare2: 0x92 spare3: 0xcf67
 consistency value in tail: 0x43455474
 check value in block header: 0x2598
 block checksum disabled
Reading datafile 'F:\ORADATA\SYSTEM01.DBF' for corruption at rdba: 0x0040bd13 (file 1, block 48403)
Reread (file 1, block 48403) found same corrupt data
Hex dump of (file 1, block 48404) in trace file d:\app\xifenfei\diag\rdbms\qdbdc\o11201gbk\trace\o11201gbk_ora_9480.trc
Corrupt block relative dba: 0x0040bd14 (file 1, block 48404)
Corrupt Block Found
Bad header found during multiblock buffer read
         TSN = 0, TSNAME = SYSTEM
Data in bad block:
         RFN = 1, BLK = 48403, RDBA = 4242707
 type: 36 format: 2 rdba: 0x6dce856d
 last change scn: 0xfc44.d24c936f seq: 0xdc flg: 0x3b
 spare1: 0x9c spare2: 0x92 spare3: 0xcf67
 consistency value in tail: 0x43455474
 check value in block header: 0x2598
 block checksum disabled
Reading datafile 'F:\ORADATA\SYSTEM01.DBF' for corruption at rdba: 0x0040bd14 (file 1, block 48404)
Reread (file 1, block 48404) found same corrupt data
Errors in file d:\app\xifenfei\diag\rdbms\qdbdc\o11201gbk\trace\o11201gbk_ora_9480.trc  (incident=3784):
ORA-01578: ORACLE 数据块损坏 (文件号 1, 块号 48404)
ORA-01110: 数据文件 1: 'F:\ORADATA\SYSTEM01.DBF'
Incident details in: d:\app\xifenfei\diag\rdbms\qdbdc\o11201gbk\incident\incdir_3784\o11201gbk_ora_9480_i3784.trc
         OBJN = 36, OBJD = 36, OBJECT = I_OBJ1, SUBOBJECT = 
         SEGMENT OWNER = SYS, SEGMENT TYPE = Index Segment
Corrupt Block Found
         TSN = 0, TSNAME = SYSTEM
         RFN = 1, BLK = 48404, RDBA = 4242708
         OBJN = 36, OBJD = 36, OBJECT = I_OBJ1, SUBOBJECT = 
         SEGMENT OWNER = SYS, SEGMENT TYPE = Index Segment
Errors in file d:\app\xifenfei\diag\rdbms\qdbdc\o11201gbk\trace\o11201gbk_ora_9480.trc:
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-01578: ORACLE 数据块损坏 (文件号 1, 块号 48404)
ORA-01110: 数据文件 1: 'F:\ORADATA\SYSTEM01.DBF'
Errors in file d:\app\xifenfei\diag\rdbms\qdbdc\o11201gbk\trace\o11201gbk_ora_9480.trc:
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-01578: ORACLE 数据块损坏 (文件号 1, 块号 48404)
ORA-01110: 数据文件 1: 'F:\ORADATA\SYSTEM01.DBF'
Error 604 happened during db open, shutting down database
USER (ospid: 9480): terminating the instance due to error 604

这里可以看出来数据库不能正常启动的原因,主要是由于I_OBJ1(obj$表的index)刚好被损坏,导致数据库无法,通过分析定位确定是如下sql导致启动失败

Dump continued from file: d:\app\xifenfei\diag\rdbms\qdbdc\o11201gbk\trace\o11201gbk_ora_9480.trc
ORA-01578: ORACLE 数据块损坏 (文件号 1, 块号 48404)
ORA-01110: 数据文件 1: 'F:\ORADATA\SYSTEM01.DBF'

========= Dump for incident 3784 (ORA 1578) ========

*** 2020-03-12 14:52:20.614
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=cq514nkrp38hv) -----
select distinct d.p_obj#,d.p_timestamp from sys.dependency$ d, obj$ o where d.p_obj#>=:1 and 
d.d_obj#=o.obj# and o.status not in (5,6)

通过对其i_obj1损坏block进行修复,数据库正启动成功

C:\Users\XIFENFEI>D:\app\XIFENFEI\product\11.2.0.1\dbhome_2\bin\sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on 星期四 3月 12 15:05:48 2020

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

已连接到空闲例程。

SQL> startup pfile='f:/pfile.txt' mount;
ORACLE 例程已经启动。

Total System Global Area 3307048960 bytes
Fixed Size                  2180264 bytes
Variable Size            1811942232 bytes
Database Buffers         1476395008 bytes
Redo Buffers               16531456 bytes
数据库装载完毕。
SQL> alter database open;

数据库已更改。

尝试导出数据

C:\Windows\system32>exp system/oracle owner=gis_sys file=f:/gis_sys.dmp FEEDBACK
=10000  COMPRESS=NO BUFFER=102400000 STATISTICS=none

Export: Release 11.2.0.1.0 - Production on 星期四 3月 12 15:46:19 2020

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.


连接到: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
已导出 ZHS16GBK 字符集和 AL16UTF16 NCHAR 字符集

即将导出指定的用户...
. 正在导出 pre-schema 过程对象和操作
. 正在导出用户 GIS_SYS 的外部函数库名
. 导出 PUBLIC 类型同义词
EXP-00008: 遇到 ORACLE 错误 604
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-01410: 无效的 ROWID
EXP-00000: 导出终止失败

分析trace文件

*** SESSION ID:(192.3) 2020-03-12 15:05:36.132
OBJD MISMATCH typ=6, seg.obj=18, diskobj=224, dsflg=100100, dsobj=18, tid=18, cls=1
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-01410: 无效的 ROWID
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-01410: 无效的 ROWID

由于obj id为18(obj$)的对象和对应的数据库中实际block存储的表的block为224(aud$)不匹配,从而出现该错误,通过分析是由于i_obj1记录错误导致该问题,通过修复该记录之后,数据实现完美导出.
20200312202743


类似system文件坏块案例有:
通过拷贝block实现system文件大量坏块恢复
记录一次system表空间坏块(ORA-01578)数据库恢复
SYSTEM表空间坏块恢复—C_TS#对象坏块恢复(file 1 block 60)
file 1 block 128 corrupted/坏块恢复—system rollback坏块修复

oracle文件被删除且部分被覆盖恢复案例

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:oracle文件被删除且部分被覆盖恢复案例

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户数据库异常,让我们对其进行分析,判断是否可以恢复,让客户通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)收集数据库信息,发现有三个数据文件异常
20200306223254


通过和客户确认,大概情况是这样的:由于/opt目录满了,客户把/opt/oracle整体迁移到/home目录中,然后通过link目录的方式实现迁移,但是在迁移之前把/home/oracle中的所有数据文件给rm掉了,然后把/opt中的数据拷贝到/home/中,在启动数据库的时候提示/home/oracle/JXWR.dbf文件丢失,然后又人工创建了一个该文件,从而出现了上述的三个文件异常(其实删除的数据库文件有十几个,涉及该库的有三个,客户只要恢复JXWR表空间数据即可).对于这种情况,有可能有覆盖的风险,让客户提供空间对现有/home 分区进行镜像,通过工具分析镜像文件
20200305142707
20200305142745

发现需要恢复文件大小/时间均不对,查看内容发现是oracle的审计trace,证明该文件对应的位置已经有覆盖,对于这样的情况,无法从os层面反删除进行恢复,考虑通过oracle碎片层面进行处理,对其分析发现大量block依旧存在(情况有点复杂,因为该目录涉及多个库,通过分析确认相关段为该数据库文件)
20200306230009

检查重组出来的数据文件效果
20200306224628


检查效果比较乐观,因为根据这样的情况,丢失15%左右的block算是非常理想的效果,然后通过oracle dul恢复客户需要的数据1

完成数据库恢复任务.
这次的恢复效果不是太好主要就是由于客户删除文件之后,对被删除文件所在分区进行了大量写操作导致不少数据库block被覆盖,最终只能抱着试试看的态度最大限度恢复,属于比较侥幸的恢复成功,再次提醒各位:在数据被误删除之后,应该先保护现场(不要对其分区进行写操作),覆盖的越少,恢复的效果越好.

数字后缀名加密数据库恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:数字后缀名加密数据库恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

一个oracle dmp文件被加密破坏的case,加密提示如下
20200306192501


20200306191213

通过工具分析发现该文件前面1M被破坏
20200306191553

通过我们工具对头部损坏的1M数据进行特殊处理,数据直接使用imp命令导入
20200306191709
如果你有各种被类似病毒加密的数据库(oracle,sql server,mysql),我们可以提供专业的恢复支持,实现不给黑客交钱的情况下,数据几乎完美恢复
Tel/微信:17813235971    Q Q:107644445 QQ咨询惜分飞    E-Mail:dba@xifenfei.com提供专业的恢复服务.

xfs删除数据文件恢复

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:xfs删除数据文件恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

在linux7的环境(xfs文件系统格式)中,由于误操作执行了rm -rf /操作,导致系统大部分文件被删除(oracle数据库相关文件).我们对其现场进行分析,确认相关数据在磁盘底层依旧存在
20191219042943


20191219043052


运气不错通过底层恢复,实现数据库完美open
20191219221348


oracle dmp被加密恢复

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:oracle dmp被加密恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

接到朋友恢复请求,oracle dmp文件被加密
20191124230422


通过分析文件发现该加密主要是对头尾部分block按照每16byte中8个byte置空和部分加密
20191124230702
20191124231600


通过进行恢复,对损坏部分进行跳过,剩余数据直接导入数据库,通过show=y测试数据可以正常入库,实现了dmp文件表数据的完美恢复
20191124231917
20191124232238


再一起asm disk被格式化成ext3文件系统故障恢复

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:再一起asm disk被格式化成ext3文件系统故障恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

国庆节前夕接到朋友求救电话asm disk被格式化成ext3格式了,具体操作如下
20191006205129


20191006205203


并且把这个分区直接挂载到/目录
20191006205239


由于/被挂载新格式化的控盘,导致asm磁盘组访问其他盘报错

Sun Sep 29 18:15:02 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_b000_8094.trc:
ORA-15025: could not open disk "/dev/asmdisk/sdh"
ORA-27041: unable to open file
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_b000_8094.trc:
ORA-15025: could not open disk "/dev/asmdisk/sdh"
ORA-27041: unable to open file
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
WARNING: cache failed reading from group=1(DATA) fn=9 blk=0 count=1 from
disk= 5 (DATA_0005) kfkist=0x20 status=0x02 osderr=0x0 file=kfc.c line=11596
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_b000_8094.trc:
ORA-15025: could not open disk "/dev/asmdisk/sdh"
ORA-27041: unable to open file
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-15080: synchronous I/O operation to a disk failed
WARNING: cache succeeded reading from group=1(DATA) fn=9 blk=0 count=1 from
disk= 7 (DATA_0007) kfkist=0x20 status=0x01 osderr=0x0 file=kfc.c line=11637
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_b000_8094.trc:
ORA-15025: could not open disk "/dev/asmdisk/sdh"
ORA-27041: unable to open file
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
WARNING: PST-initiated drop of 1 disk(s) in group 1(.2380027701))

重启系统之后,重试mount 磁盘组

GMON dismounting group 1 at 18 for pid 31, osid 44279
NOTE: Disk DATA_0000 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0001 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0002 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0003 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0004 in mode 0x1 marked for de-assignment
NOTE: Disk DATA_0005 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0006 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0007 in mode 0x7f marked for de-assignment
NOTE: Disk  in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0009 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0010 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0011 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0012 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0013 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0014 in mode 0x7f marked for de-assignment
ERROR: diskgroup DATA was not mounted
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "8" is missing from group number "1"
ERROR: ALTER DISKGROUP DATA MOUNT  /* asm agent *//* {1:2587:2} */

由于sdb(asm disk 8)被格式化,导致data磁盘组无法正常mount.这个客户运气比较好data 磁盘组是normal模式,但是由于mount到/,导致disk 4被强制drop,因此无法mount成功,但是通过一系列处理数据实现完美恢复,0数据丢失
20191006211707


如果磁盘组是外部冗余,请参考:
又一例asm格式化文件系统恢复
一次完美的asm disk被格式化ntfs恢复
oracle asm disk格式化恢复—格式化为ext4文件系统
oracle asm disk格式化恢复—格式化为ntfs文件系统

ORA-600 ktfbhget-4

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:ORA-600 ktfbhget-4

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户和我们反馈,他们数据库现在使用异常,查询dba_data_files报ORA-600 ktfbhget-4

*** 2019-09-21 13:56:42.944
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [ktfbhget-4], [384], [16], [], [], [], [], []
Current SQL statement for this session:
select tablespace_name from dba_data_files
----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp+0148          bl       ksedst               1028F23AC ?
ksfdmp+0018          bl       01FD7F4C
kgerinv+00e8         bl       _ptrgl
kgesinv+0020         bl       kgerinv              000000000 ? 000000000 ?
                                                   FFFFFFFFFFF8920 ? 000400002 ?
                                                   000000000 ?
ksesin+005c          bl       kgesinv              7000001637868E8 ? 110359FD8 ?
                                                   000000100 ? 000007FFF ?
                                                   000007FFF ?
ktfbhget+047c        bl       ksesin               1029B1D18 ? 200000002 ?
                                                   000000000 ? 000000180 ?
                                                   000000000 ? 000000010 ?
                                                   000000008 ? 000000000 ?
ktfbhcf+03c0         bl       ktfbhget             FFFFFFFFFFF8DA0 ? 1103589D8 ?
                                                   201035A57C ? 1C610005A40 ?
qerfxFetch+0c94      bl       01FD7AC4
qerjoFetch+037c      bl       _ptrgl
rwsfcd+0060          bl       _ptrgl
qersoFetch+0108      bl       _ptrgl
qerjoFetch+037c      bl       _ptrgl
qerjoFetch+037c      bl       _ptrgl
rwsfcd+0060          bl       _ptrgl
qeruaFetch+0140      bl       01FD7AC4
qervwFetch+008c      bl       01FD7AC4
opifch2+0c68         bl       01FD7AC4
opiall0+1244         bl       opifch2              7000001626B2F28 ? 100000001 ?
                                                   FFFFFFFFFFFA3D8 ?
kpoal8+0a68          bl       opiall0              5E1000C818 ? 2200000014 ?
                                                   FFFFFFFFFFFAAD8 ? 000000000 ?
                                                   FFFFFFFFFFFAA28 ? 11028C2C8 ?
                                                   080000000 ? 4000000007FFF ?
opiodr+08e8          bl       _ptrgl
ttcpip+0c54          bl       _ptrgl
opitsk+0c28          bl       ttcpip               11000C818 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
opiino+0798          bl       opitsk               000000000 ? 000000000 ?
opiodr+08e8          bl       _ptrgl
opidrv+032c          bl       opiodr               3C00000018 ? 4101F5E80 ?
                                                   FFFFFFFFFFFF4C0 ? 0A000FD98 ?
sou2o+0028           bl       opidrv               3C0C000000 ? 400000000 ?
                                                   FFFFFFFFFFFF4C0 ?
main+0138            bl       01FD758C
__start+0098         bl       main                 000000000 ? 000000000 ?

通过询问知道由于file#=17最初由于裸设备权限异常,导致无法访问,执行了delete from file$ where file#=17,然后遭遇异常又人工插入了一条file#=17的记录到file$中,但是由于不知道具体值,可能是由于部分值插入错误引起现在问题.
20190927225858


通过分析file$表定义

create table file$                                             /* file table */
( file#         number not null,                   /* file identifier number */
  status$       number not null,                      /* status (see KTS.H): */
                                               /* 1 = INVALID, 2 = AVAILABLE */
  blocks        number not null,                   /* size of file in blocks */
                                           /* zero for bitmapped tablespaces */
  ts#           number,                         /* tablespace that owns file */
  relfile#      number,                              /* relative file number */
  maxextend     number,                                 /* maximum file size */
  inc           number,                                  /* increment amount */
  crscnwrp      number,                                 /* creation SCN wrap */
  crscnbas      number,                                 /* creation SCN base */
  ownerinstance varchar("M_IDEN"),                    /* Owner instance name */
  spare1        number,      /* tablespace-relative DBA of space file header */
                                   /* NULL for dictionary-mapped tablespaces */
  spare2        number,
  spare3        varchar2(1000),
  spare4        date
)

通过dump文件头(文件创建大小/SCN等),结合一些计算和经验值,获取到相关值重新插入正确值,一切恢复正常

delete from file$ where file#=17 and ts#=384;
insert into sys.file$ (FILE#, STATUS$, BLOCKS, TS#, RELFILE#, MAXEXTEND, INC,
 CRSCNWRP, CRSCNBAS, OWNERINSTANCE, SPARE1, SPARE2, SPARE3, SPARE4)
values (17,2,256000,384,17,0,0,3087,1631091037,null,71303170, null, null, null);
commit;

再次提醒,file$中记录请勿人工修改,以前写过相关casefile$ 删除记录恢复(delete file$ recovery)

记录一次200T的数据库恢复经历

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:记录一次200T的数据库恢复经历

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有一个客户恢复请求,6个节点11.2.0.3 RAC,非归档模式,数据量近200T
df_size


由于存储掉电导致数据库6个节点全部宕机,恢复硬件之后,数据库无法正常启动,报错如下:

SQL> recover database;
ORA-00279: change 318472018583 generated at 05/04/2019 17:58:05 needed for
thread 4
ORA-00289: suggestion :
/u01/app/oracle/product/11.2.0/db_1/dbs/arch4_322810_870181839.dbf
ORA-00280: change 318472018583 for thread 4 is in sequence #322810
Wed Aug 28 11:19:55 2019
ALTER DATABASE RECOVER  DATABAE
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 14 Seq 552 Reading mem 0
  Mem# 0: +REDO/xff/log2.ora
Recovery of Online Redo Log: Thread 2 Group 15 Seq 126 Reading mem 0
  Mem# 0: +REDO/xff/log3.ora
Recovery of Online Redo Log: Thread 3 Group 18 Seq 122 Reading mem 0
  Mem# 0: +REDO/xff/log6.ora
ORA-279 signalled during: ALTER DATABASE RECOVER  database  ...
Wed Aug 28 11:21:31 2019
ALTER DATABASE RECOVER CANCEL
Media Recovery Canceled
Completed: ALTER DATABASE RECOVER CANCEL

数据库恢复需要thread 4 sequence #322810,查询redo信息
redo


redo已经被覆盖,数据库无法通过正常途径恢复实现数据库open,尝试屏蔽一致性强制拉库操作后

Wed Aug 28 12:40:15 2019
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_smon_51338.trc  (incident=244209):
ORA-00600: internal error code, arguments: [4137], [44.47.613406], [0], [0], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xff/xff1/incident/incdir_244209/xff1_smon_51338_i244209.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Wed Aug 28 12:40:16 2019
ORACLE Instance xff1 (pid = 26) - Error 600 encountered while recovering transaction (44, 47).
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_smon_51338.trc:
ORA-00600: internal error code, arguments: [4137], [44.47.613406], [0], [0], [], [], [], [], [], [], [], []
Wed Aug 28 12:40:20 2019
Exception[type: SIGSEGV,Address not mapped to object][ADDR:0x5122000000C8][PC:0xE1B4D3,ktugru()+87][flags:0x0,count:1]
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_p086_54066.trc  (incident=245017):
ORA-07445:exception encountered:core dump [ktugru()+87][SIGSEGV][ADDR:0x5122000000C8][Address not mapped to object]
Incident details in: /u01/app/oracle/diag/rdbms/xff/xff1/incident/incdir_245017/xff1_p086_54066_i245017.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Wed Aug 28 12:40:20 2019
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_p000_53873.trc  (incident=244305):
ORA-00600: internal error code, arguments: [4198], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xff/xff1/incident/incdir_244305/xff1_p000_53873_i244305.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

提示undo异常,屏蔽回滚段之后,数据库正常打开没有任何报错信息

Wed Aug 28 12:57:15 2019
SMON: enabling cache recovery
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
[57676] Successfully onlined Undo Tablespace 22.
Undo initialization finished serial:0 start:2386111306 end:2386112316 diff:1010 (10 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Wed Aug 28 12:57:17 2019
minact-scn: Inst 1 is now the master inc#:2 mmon proc-id:57624 status:0x7
minact-scn status: grec-scn:0x0000.00000000 gmin-scn:0x0000.00000000 gcalc-scn:0x0000.00000000
No Resource Manager plan active
Starting background process GTX0
Wed Aug 28 12:57:18 2019
GTX0 started with pid=45, OS id=57777
Starting background process RCBG
Wed Aug 28 12:57:18 2019
RCBG started with pid=46, OS id=57779
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Wed Aug 28 12:57:19 2019
QMNC started with pid=47, OS id=57788
Completed: ALTER DATABASE OPEN

后续涉及创建新undo,删除老undo并处理一些类似,基本上恢复正常
OPEN


一次完美的asm disk被格式化ntfs恢复

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:一次完美的asm disk被格式化ntfs恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

又一起win rac的asm disk被格式化为ntfs,导致数据库异常恢复的请求,客户描述有一个500G的data磁盘组(只有一个磁盘,被误操作进行了格式化).格式化asm disk之后
asm的alert日志报错

Tue Jul 09 17:09:45 2019
NOTE: ASM client orcl1:ORCL disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file D:\APP\ADMINISTRATOR\diag\asm\+asm\+asm1\trace\+asm1_ora_5376.trc
Tue Jul 09 17:10:19 2019
Errors in file D:\APP\ADMINISTRATOR\diag\asm\+asm\+asm1\trace\+asm1_lgwr_1448.trc:
ORA-27070: async read/write failed
OSD-04008: WriteFile() 失败, 无法写入文件
O/S-Error: (OS 21) 设备未就绪。
WARNING: Write Failed. group:1 disk:0 AU:15 offset:876544 size:4096
NOTE: unable to write any mirror side for diskgroup DATA
NOTE: cache initiating offline of disk 0 group DATA
NOTE: process _lgwr_+asm1 (3764:1448) initiating offline of disk 0.4042281525 (DATA_0000) with mask 0x7e in group 1
Tue Jul 09 17:10:19 2019
WARNING: Disk 0 (DATA_0000) in group 1 mode 0x7f is now being offlined
WARNING: Disk 0 (DATA_0000) in group 1 in mode 0x7f is now being taken offline on ASM inst 1
NOTE: initiating PST update: grp = 1, dsk = 0/0xf0f05235, mask = 0x6a, op = clear
GMON updating disk modes for group 1 at 10 for pid 15, osid 1448
ERROR: Disk 0 cannot be offlined, since diskgroup has external redundancy.
ERROR: too many offline disks in PST (grp 1)
WARNING: Disk 0 (DATA_0000) in group 1 mode 0x7f offline is being aborted
WARNING: Offline of disk 0 (DATA_0000) in group 1 and mode 0x7f failed on ASM inst 1
Tue Jul 09 17:10:20 2019
NOTE: halting all I/Os to diskgroup 1 (DATA)
NOTE: unable to offline disks after getting write error for diskgroup DATA
Tue Jul 09 17:10:20 2019
NOTE: cache dismounting (not clean) group 1/0xBDB0A2C0 (DATA)
NOTE: disk 0 had IO error
NOTE: messaging CKPT to quiesce pins Windows thread id: 520528, image: ORACLE.EXE (B000)
Tue Jul 09 17:10:20 2019
NOTE: Deferred communication with ASM instance
Errors in file D:\APP\ADMINISTRATOR\diag\asm\+asm\+asm1\trace\+asm1_ora_6140.trc:
ORA-15130: diskgroup "DATA" is being dismounted
NOTE: deferred map free for map id 4
NOTE: LGWR doing non-clean dismount of group 1 (DATA)
NOTE: LGWR sync ABA=38.3028 last written ABA 38.3029

数据库的alert日志报错

Tue Jul 09 17:09:12 2019
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl1\trace\orcl1_lgwr_5396.trc:
ORA-27072: 文件 I/O 错误
WARNING: IO Failed. group:1 disk(number.incarnation):0.0xf0f05235 disk_path:\\.\ORCLDISKDATA0
	 AU:1305 disk_offset(bytes):1368456704 io_size:512 operation:Write type:asynchronous
	 result:I/O error process_id:5396
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl1\trace\orcl1_lgwr_5396.trc:
ORA-15080: 与磁盘的同步 I/O 操作失败
WARNING: failed to write mirror side 1 of virtual extent 23 logical extent 0
   of file 261 in group 1 on disk 0 allocation unit 1305
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl1\trace\orcl1_lgwr_5396.trc:
ORA-00345: 重做日志写入块 47231 计数 1 出错
ORA-00312: 联机日志 1 线程 1: '+DATA/orcl/onlinelog/group_1.261.909498607'
ORA-15081: 无法将 I/O 操作提交到磁盘
ORA-15081: 无法将 I/O 操作提交到磁盘
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl1\trace\orcl1_lgwr_5396.trc:
ORA-27070: 异步读取/写入失败
OSD-04006: ReadFile() 失败, 无法读取文件
O/S-Error: (OS 21) 设备未就绪。

机器上查看磁盘信息
format_ntfs2
format_ntfs


这个客户比较幸运,asm disk被格式化之后,没有进行任何写操作,理解对现场进行了保护,没有任何的二次破坏.因为处理过多起类似故障(oracle asm disk格式化恢复—格式化为ntfs文件系统,oracle asm disk格式化恢复—格式化为ext4文件系统又一例asm格式化文件系统恢复),有一定的经验,一般三种方案恢复数据文件:
1)运气好直接通过kfed进行修复asm disk,然后mount起来,然后把数据文件拷贝到文件系统中
2)通过相关工具把asm disk中相关的数据文件拷贝到文件系统中
3)如果损坏的严重,通过底层碎片,把相关数据文件恢复到文件系统中
拷贝完成数据文件之后,然后根据文件的情况有几种可能性:
1)直接open数据库,处理可能的其他坏块
2)通过一些方法强制拉库,然后对其进行导出导入新库
3)通过工具直接恢复表数据,甚至恢复部分核心数据
这次的恢复运气不错,对asm disk进行一系列修复之后,asm 磁盘组mount成功

C:\Users\Administrator>asmtool -list
NTFS                             \Device\Harddisk0\Partition1            80000M
NTFS                             \Device\Harddisk0\Partition2           491133M
ORCLDISKOCR0                     \Device\Harddisk1\Partition1            34132M
ORCLDISKOCR1                     \Device\Harddisk1\Partition2            34132M
ORCLDISKOCR2                     \Device\Harddisk1\Partition3            34133M
ORCLDISKDATA0                    \Device\Harddisk2\Partition1           511997M
ORCLDISKFRA0                     \Device\Harddisk3\Partition2          1225000M
NTFS                             \Device\Harddisk3\Partition3          1225800M
                                 \Device\Harddisk3\Partition4          1314861M
C:\Users\Administrator>sqlplus / as sysasm
SQL*Plus: Release 11.2.0.1.0 Production on 星期六 7月 13 11:20:02 2019
Copyright (c) 1982, 2010, Oracle.  All rights reserved.
连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> alter diskgroup data mount;
Diskgroup altered.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64
bit Production
With the Real Application Clusters and Automatic Storage Management options

mount数据库拷贝文件

C:\Users\Administrator>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on 星期六 7月 13 16:02:08 2019
Copyright (c) 1982, 2010, Oracle.  All rights reserved.
已连接到空闲例程。
SQL> startup  mount;
ORACLE 例程已经启动。
Total System Global Area 3.4206E+10 bytes
Fixed Size                  2192864 bytes
Variable Size            7516195360 bytes
Database Buffers         2.6642E+10 bytes
Redo Buffers               45727744 bytes
数据库已装载
RMAN>  backup as copy   database format 'I:\rmanback\df_%U.dbf';
启动 backup 于 13-7月 -19
使用通道 ORA_DISK_1
使用通道 ORA_DISK_2
使用通道 ORA_DISK_3
使用通道 ORA_DISK_4
通道 ORA_DISK_1: 启动数据文件副本
输入数据文件: 文件号=00006 名称=+DATA/orcl/datafile/XFF5.dmp
通道 ORA_DISK_2: 启动数据文件副本
输入数据文件: 文件号=00016 名称=+DATA/orcl/datafile/XFF53
通道 ORA_DISK_3: 启动数据文件副本
输入数据文件: 文件号=00017 名称=+DATA/orcl/datafile/XFF54
通道 ORA_DISK_4: 启动数据文件副本
输入数据文件: 文件号=00018 名称=+DATA/orcl/datafile/XFF55
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-16_0VU6I1FO.DBF
标记=TAG20190713T120927 RECID=2 STAMP=1013517954
通道 ORA_DISK_2: 数据文件复制完毕, 经过时间: 00:36:29
通道 ORA_DISK_2: 启动数据文件副本
输入数据文件: 文件号=00019 名称=+DATA/orcl/datafile/XFF5.281.988042407
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-17_10U6I1FO.DBF
标记=TAG20190713T120927 RECID=1 STAMP=1013517951
通道 ORA_DISK_3: 数据文件复制完毕, 经过时间: 00:36:29
通道 ORA_DISK_3: 启动数据文件副本
输入数据文件: 文件号=00020 名称=+DATA/orcl/datafile/XFF5.282.988042943
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-18_11U6I1FO.DBF
标记=TAG20190713T120927 RECID=3 STAMP=1013517962
通道 ORA_DISK_4: 数据文件复制完毕, 经过时间: 00:36:37
通道 ORA_DISK_4: 启动数据文件副本
输入数据文件: 文件号=00021 名称=+DATA/orcl/datafile/XFF5.283.988043279
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-6_0UU6I1FO.DBF
标记=TAG20190713T120927 RECID=4 STAMP=1013518052
通道 ORA_DISK_1: 数据文件复制完毕, 经过时间: 00:38:12
通道 ORA_DISK_1: 启动数据文件副本
输入数据文件: 文件号=00022 名称=+DATA/orcl/datafile/XFF5.284.988043721
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-20_13U6I3K5.DBF
标记=TAG20190713T120927 RECID=5 STAMP=1013519584
通道 ORA_DISK_3: 数据文件复制完毕, 经过时间: 00:27:10
通道 ORA_DISK_3: 启动数据文件副本
输入数据文件: 文件号=00009 名称=+DATA/orcl/datafile/XFF51.dmp
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-21_14U6I3KD.DBF
标记=TAG20190713T120927 RECID=6 STAMP=1013519586
通道 ORA_DISK_4: 数据文件复制完毕, 经过时间: 00:27:03
通道 ORA_DISK_4: 启动数据文件副本
输入数据文件: 文件号=00010 名称=+DATA/orcl/datafile/XFF52.dmp
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-19_12U6I3K5.DBF
标记=TAG20190713T120927 RECID=7 STAMP=1013519599
通道 ORA_DISK_2: 数据文件复制完毕, 经过时间: 00:27:26
通道 ORA_DISK_2: 启动数据文件副本
输入数据文件: 文件号=00003 名称=+DATA/orcl/datafile/undotbs1.258.909498475
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-22_15U6I3NC.DBF
标记=TAG20190713T120927 RECID=8 STAMP=1013519695
通道 ORA_DISK_1: 数据文件复制完毕, 经过时间: 00:27:19
通道 ORA_DISK_1: 启动数据文件副本
输入数据文件: 文件号=00005 名称=+DATA/orcl/datafile/undotbs2.264.909498739
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-UNDOTBS1_FNO-3_18U6I57K.DB
F 标记=TAG20190713T120927 RECID=9 STAMP=1013520920
通道 ORA_DISK_2: 数据文件复制完毕, 经过时间: 00:22:02
通道 ORA_DISK_2: 启动数据文件副本
输入数据文件: 文件号=00012 名称=+DATA/orcl/datafile/XFF5.275.954036129
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-UNDOTBS2_FNO-5_19U6I5AJ.DB
F 标记=TAG20190713T120927 RECID=10 STAMP=1013521015
通道 ORA_DISK_1: 数据文件复制完毕, 经过时间: 00:22:03
通道 ORA_DISK_1: 启动数据文件副本
输入数据文件: 文件号=00013 名称=+DATA/orcl/datafile/XFF5.276.954036233
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-10_17U6I574.DBF
标记=TAG20190713T120927 RECID=11 STAMP=1013521231
通道 ORA_DISK_4: 数据文件复制完毕, 经过时间: 00:27:30
通道 ORA_DISK_4: 启动数据文件副本
输入数据文件: 文件号=00002 名称=+DATA/orcl/datafile/sysaux.257.909498475
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-9_16U6I574.DBF
标记=TAG20190713T120927 RECID=12 STAMP=1013521244
通道 ORA_DISK_3: 数据文件复制完毕, 经过时间: 00:27:37
通道 ORA_DISK_3: 启动数据文件副本
输入数据文件: 文件号=00023 名称=+DATA/orcl/datafile/XFF60.dmp
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-12_1AU6I6GU.DBF
标记=TAG20190713T120927 RECID=13 STAMP=1013522267
通道 ORA_DISK_2: 数据文件复制完毕, 经过时间: 00:22:25
通道 ORA_DISK_2: 启动数据文件副本
输入数据文件: 文件号=00024 名称=+DATA/orcl/datafile/XFF61.dmp
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-13_1BU6I6JU.DBF
标记=TAG20190713T120927 RECID=15 STAMP=1013522367
通道 ORA_DISK_1: 数据文件复制完毕, 经过时间: 00:22:26
通道 ORA_DISK_1: 启动数据文件副本
输入数据文件: 文件号=00011 名称=+DATA/orcl/datafile/system.dmp
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-SYSAUX_FNO-2_1CU6I6QM.DBF
标记=TAG20190713T120927 RECID=14 STAMP=1013522361
通道 ORA_DISK_4: 数据文件复制完毕, 经过时间: 00:18:50
通道 ORA_DISK_4: 启动数据文件副本
输入数据文件: 文件号=00001 名称=+DATA/orcl/datafile/system.256.909498475
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-23_1DU6I6QT.DBF
标记=TAG20190713T120927 RECID=16 STAMP=1013522400
通道 ORA_DISK_3: 数据文件复制完毕, 经过时间: 00:19:19
通道 ORA_DISK_3: 启动数据文件副本
输入数据文件: 文件号=00025 名称=+DATA/orcl/datafile/XFF62.dmp
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-25_1HU6I7V4.DBF
标记=TAG20190713T120927 RECID=17 STAMP=1013522490
通道 ORA_DISK_3: 数据文件复制完毕, 经过时间: 00:01:35
通道 ORA_DISK_3: 启动数据文件副本
输入数据文件: 文件号=00026 名称=+DATA/orcl/datafile/XFF63.dmp
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-SYSTEM_FNO-1_1GU6I7U0.DBF
标记=TAG20190713T120927 RECID=18 STAMP=1013522526
通道 ORA_DISK_4: 数据文件复制完毕, 经过时间: 00:02:47
通道 ORA_DISK_4: 启动数据文件副本
输入数据文件: 文件号=00007 名称=+DATA/orcl/datafile/precise.dbf
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-SYSTEM_FNO-11_1FU6I7U0.DBF
 标记=TAG20190713T120927 RECID=20 STAMP=1013522576
通道 ORA_DISK_1: 数据文件复制完毕, 经过时间: 00:03:33
通道 ORA_DISK_1: 启动数据文件副本
输入数据文件: 文件号=00004 名称=+DATA/orcl/datafile/users.259.909498477
RMAN-03009: backup 命令 (ORA_DISK_3 通道上, 在 07/13/2019 14:03:01 上) 失败
ORA-19566: 超出损坏块限制 0 (文件 +DATA/orcl/datafile/XFF63.dmp)
继续执行其他作业步骤, 将不重新运行失败的作业
通道 ORA_DISK_3: 启动数据文件副本
输入数据文件: 文件号=00014 名称=+DATA/orcl/datafile/test.280.972807149
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-PRECISE_FNO-7_1JU6I837.DBF
 标记=TAG20190713T120927 RECID=19 STAMP=1013522572
通道 ORA_DISK_4: 数据文件复制完毕, 经过时间: 00:00:46
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-USERS_FNO-4_1KU6I84L.DBF
标记=TAG20190713T120927 RECID=21 STAMP=1013522591
通道 ORA_DISK_1: 数据文件复制完毕, 经过时间: 00:00:15
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-TEST_FNO-14_1LU6I84L.DBF
标记=TAG20190713T120927 RECID=22 STAMP=1013522591
通道 ORA_DISK_3: 数据文件复制完毕, 经过时间: 00:00:15
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-24_1EU6I7R0.DBF
标记=TAG20190713T120927 RECID=23 STAMP=1013522974
通道 ORA_DISK_2: 数据文件复制完毕, 经过时间: 00:11:44
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: backup 命令 (ORA_DISK_3 通道上, 在 07/13/2019 14:03:01 上) 失败
ORA-19566: 超出损坏块限制 0 (文件 +DATA/orcl/datafile/XFF63.dmp)

运气不错,除+DATA/orcl/datafile/XFF63.dmp文件上面有坏块之外其他文件没有发现坏块,对该文件进行特殊方式拷贝处理
dbv检查该文件

C:\Users\Administrator>dbv file=i:/rmanback/26.dbf
DBVERIFY: Release 11.2.0.1.0 - Production on 星期六 7月 13 14:09:32 2019
Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.
DBVERIFY - 开始验证: FILE = I:\RMANBACK\26.DBF
页 132735 流入 - 很可能是介质损坏
Corrupt block relative dba: 0x0682067f (file 26, block 132735)
Fractured block found during dbv:
Data in bad block:
 type: 40 format: 2 rdba: 0x0682067f
 last change scn: 0x0000.333eaa21 seq: 0x2 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0xaa550000
 check value in block header: 0x4242
 computed block checksum: 0xb24f
DBVERIFY - 验证完成
检查的页总数: 153600
处理的页总数 (数据): 9283
失败的页总数 (数据): 0
处理的页总数 (索引): 4789
失败的页总数 (索引): 0
处理的页总数 (其他): 139463
处理的总页数 (段)  : 0
失败的总页数 (段)  : 0
空的页总数: 64
标记为损坏的总页数: 1
流入的页总数: 1
加密的总页数        : 0
最高块 SCN            : 1029666541 (0.1029666541)

这次运气非常好,该文件也只有一个坏块,整体来说,把正在运行的asm disk磁盘格式化为ntfs格式化,整个数据库文件只发现一个坏块.拷贝数据文件,redo,ctl等之后

通过重建控制文件尝试open数据库

SQL> @ctl.sql
控制文件已创建。
SQL> RECOVER DATABASE;
完成介质恢复。
SQL> ALTER DATABASE OPEN;
数据库已更改。

通过dba_extents 定位坏块对象,然后根据实际情况处理(index直接rebuild,表跳过,lob置空等方法),确定数据没有问题,重建磁盘组,数据回迁,恢复完美完成

ORA-00600: internal error code, arguments: [16703], [1403], [32]

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:ORA-00600: internal error code, arguments: [16703], [1403], [32]

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有网友咨询数据库启动报ORA-00600: internal error code, arguments: [16703], [1403], [32]错误,这个错误和以往遇到的不太一样,以前恢复的一些案例
tab$恢复错误汇总
10g数据库遭遇ORA-600 16703
12C数据库遭遇ORA-600 16703
tab$被恶意删除sys用户之外记录
ORA-600 16703故障解析—tab$表被清空
警告:互联网中有oracle介质被注入恶意程序导致—ORA-600 16703

SQL> startup
ORACLE 例程已经启动。
Total System Global Area 1.3892E+10 bytes
Fixed Size                  5420776 bytes
Variable Size            2281703704 bytes
Database Buffers         1.1576E+10 bytes
Redo Buffers               28131328 bytes
数据库装载完毕。
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [32], [], [], [],[], [], [], [], [], []
进程 ID: 9512
会话 ID: 272 序列号: 22801

查看alert日志

Sun Jun 30 14:47:55 2019
SMON: enabling cache recovery
Errors in file D:\APP\SHORCL\diag\rdbms\orcl\orcl\trace\orcl_ora_7824.trc  (incident=177881) (PDBNAME=CDB$ROOT):
ORA-00600: 内部错误代码, 参数: [16703], [1403], [32], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\SHORCL\diag\rdbms\orcl\orcl\incident\incdir_177881\orcl_ora_7824_i177881.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Sun Jun 30 14:47:57 2019
Errors in file D:\APP\SHORCL\diag\rdbms\orcl\orcl\trace\orcl_ora_7824.trc:
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [16703], [1403], [32], [], [], [], [], [], [], [], [], []
Sun Jun 30 14:47:57 2019
Errors in file D:\APP\SHORCL\diag\rdbms\orcl\orcl\trace\orcl_ora_7824.trc:
ORA-00704: 引导程序进程失败
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [16703], [1403], [32], [], [], [], [], [], [], [], [], []
Sun Jun 30 14:47:57 2019
Errors in file D:\APP\SHORCL\diag\rdbms\orcl\orcl\trace\orcl_ora_7824.trc:
ORA-00704: 引导程序进程失败
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [16703], [1403], [32], [], [], [], [], [], [], [], [], []
Sun Jun 30 14:47:57 2019
Error 704 happened during db open, shutting down database
USER (ospid: 7824): terminating the instance due to error 704
Sun Jun 30 14:48:02 2019
Instance terminated by USER, pid = 7824
ORA-1092 signalled during: ALTER DATABASE OPEN...

根据以往经验,这个很可能也是tab$数据被删除导致。经过分析,该库的区别是由于该库是12C的pdb
12.1.0.2


pdb


通过分析确认,确实是tab$数据被删除,通过bbed反向删除处理,实现时间完美恢复,open之后删除恶意脚本,数据库直接使用,实现完美恢复

SQL> startup mount
ORACLE 例程已经启动。
Total System Global Area 1.3892E+10 bytes
Fixed Size                  5420776 bytes
Variable Size            2281703704 bytes
Database Buffers         1.1576E+10 bytes
Redo Buffers               28131328 bytes
数据库装载完毕。
SQL> alter database open;
数据库已更改。
SQL> select 'drop '||object_type||' '||owner||'.'||object_name||';' from dba_obj
ects where object_name in('DBMS_SUPPORT_DBMONITOR','DBMS_SUPPORT_DBMONITORP');
'DROP'||OBJECT_TYPE||''||OWNER||'.'||OBJECT_NAME||';'
--------------------------------------------------------------------------------
drop TRIGGER SYS.DBMS_SUPPORT_DBMONITOR;
drop PROCEDURE SYS.DBMS_SUPPORT_DBMONITORP;
SQL>
SQL> drop TRIGGER SYS.DBMS_SUPPORT_DBMONITOR;
触发器已删除。
SQL> drop PROCEDURE SYS.DBMS_SUPPORT_DBMONITORP;
过程已删除。
SQL> shutdown immediate;
数据库已经关闭。
已经卸载数据库。
ORACLE 例程已经关闭。
SQL> startup
ORACLE 例程已经启动。
Total System Global Area 1.3892E+10 bytes
Fixed Size                  5420776 bytes
Variable Size            2281703704 bytes
Database Buffers         1.1576E+10 bytes
Redo Buffers               28131328 bytes
数据库装载完毕。
数据库已经打开。
SQL> show pdbs;
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDBORCL                        MOUNTED
SQL> alter session set  container=pdborcl;
会话已更改。
SQL> alter database open;
数据库已更改。