Database SOS – 第24页 – Professional Database Recovery Support@+8617813235971

asm disk被分区,格式化为ext4恢复

有客户因为没有认识到linux中的磁盘被asm使用,对其进行分区并且做成了ext4的文件系统，从history中获取客户操作命令

  600  fdisk -l
  601  fdisk /dev/sdb
  602  mkfs ext4 /dev/sdb1 
  603  fdisk -l
  604  mkfs -t ext4 /dev/sdb1 
  605  cd /
  606  mkdir u01
  607  mount /dev/sdb1 /u01
  608  df -h

确认磁盘情况,确认sdb直接被asm磁盘使用(asmdisk1)

[grid@racdb3 trace]$ ls -l /dev/asm*
brw-rw---- 1 grid asmadmin 8, 16 Sep 30 14:34 /dev/asmdisk1
[grid@racdb3 trace]$ ls -l /dev/sd*
brw-rw---- 1 root disk 8,  0 Jul 27  2021 /dev/sda
brw-rw---- 1 root disk 8,  1 Jul 27  2021 /dev/sda1
brw-rw---- 1 root disk 8,  2 Jul 27  2021 /dev/sda2
brw-rw---- 1 root disk 8, 16 Sep 30 11:23 /dev/sdb
brw-rw---- 1 root disk 8, 17 Sep 30 11:23 /dev/sdb1
brw-rw---- 1 root disk 8, 32 Jul 27  2021 /dev/sdc

asm日志报错

Fri Sep 30 11:31:41 2022
NOTE: SMON starting instance recovery for group DATA domain 1 (mounted)
NOTE: SMON skipping disk 0 - no header
NOTE: cache initiating offline of disk 0 group DATA
NOTE: process _smon_+asm3 (2989) initiating offline of disk 0.3915953109 (DATA_0000) with mask 0x7e in group 1
NOTE: initiating PST update: grp = 1, dsk = 0/0xe968b3d5, mask = 0x6a, op = clear
Fri Sep 30 11:31:41 2022
GMON updating disk modes for group 1 at 4 for pid 17, osid 2989
ERROR: Disk 0 cannot be offlined, since diskgroup has external redundancy.
ERROR: too many offline disks in PST (grp 1)
Fri Sep 30 11:31:41 2022
NOTE: cache dismounting (not clean) group 1/0x34F84324 (DATA) 
WARNING: Offline for disk DATA_0000 in mode 0x7f failed.
Fri Sep 30 11:31:41 2022
NOTE: halting all I/Os to diskgroup 1 (DATA)
ERROR: No disks with F1X0 found on disk group DATA
NOTE: aborting instance recovery of domain 1 due to diskgroup dismount
NOTE: SMON skipping lock domain (1) validation because diskgroup being dismounted

数据库日志报错

Fri Sep 30 11:31:44 2022
Errors in file /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei3/trace/xifenfei3_lmon_26356.trc:
ORA-00202: control file: '+DATA/xifenfei/controlfile/current.256.968794097'
ORA-15078: ASM diskgroup was forcibly dismounted
Fri Sep 30 11:31:45 2022
Errors in file /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei3/trace/xifenfei3_ckpt_26388.trc:
ORA-00206: error in writing (block 5, # blocks 1) of control file
ORA-00202: control file: '+DATA/xifenfei/controlfile/current.257.968794097'
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-00206: error in writing (block 5, # blocks 1) of control file
ORA-00202: control file: '+DATA/xifenfei/controlfile/current.256.968794097'
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei3/trace/xifenfei3_ckpt_26388.trc:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 5, # blocks 1) of control file
ORA-00202: control file: '+DATA/xifenfei/controlfile/current.257.968794097'
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-00206: error in writing (block 5, # blocks 1) of control file
ORA-00202: control file: '+DATA/xifenfei/controlfile/current.256.968794097'
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-15078: ASM diskgroup was forcibly dismounted
CKPT (ospid: 26388): terminating the instance due to error 221

通过kfed 查看asm disk被破坏情况

[root@racdb3 scsi_host]#  kfed read /dev/asmdisk1 
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
7F4FAAD45400 00000000 00000000 00000000 00000000  [................]
        Repeat 26 times
7F4FAAD455B0 00000000 00000000 45C222C8 01000000  [.........".E....]
7F4FAAD455C0 FE830001 003FFFFF E9D60000 0000FFFF  [......?.........]
7F4FAAD455D0 00000000 00000000 00000000 00000000  [................]
        Repeat 1 times
7F4FAAD455F0 00000000 00000000 00000000 AA550000  [..............U.]
7F4FAAD45600 00000000 00000000 00000000 00000000  [................]
  Repeat 223 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

[root@racdb3 scsi_host]#  kfed read /dev/asmdisk1  aun=2
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
7F64E77A0400 00000000 00000000 00000000 00000000  [................]
        Repeat 223 times
7F64E77A1200 000081F9 000181F9 000281F9 000381F9  [................]
7F64E77A1210 000481F9 000C81F9 000D81F9 001881F9  [................]
7F64E77A1220 002881F9 003E81F9 007981F9 00AB81F9  [..(...>...y.....]
7F64E77A1230 013881F9 016C81F9 044581F9 04B081F9  [..8...l...E.....]
7F64E77A1240 061A81F9 0CD081F9 1E8481F9 00000000  [................]
7F64E77A1250 00000000 00000000 00000000 00000000  [................]
  Repeat 26 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

[root@racdb3 scsi_host]#  kfed read /dev/asmdisk1  aun=3
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
7F8D101FF400 00000000 00000000 00000000 00000000  [................]
        Repeat 223 times
7F8D10200200 000082F9 000182F9 000282F9 000382F9  [................]
7F8D10200210 000482F9 000C82F9 000D82F9 001882F9  [................]
7F8D10200220 002882F9 003E82F9 007982F9 00AB82F9  [..(...>...y.....]
7F8D10200230 013882F9 016C82F9 044582F9 04B082F9  [..8...l...E.....]
7F8D10200240 061A82F9 0CD082F9 1E8482F9 00000000  [................]
7F8D10200250 00000000 00000000 00000000 00000000  [................]
  Repeat 26 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

[root@racdb3 scsi_host]#  kfed read /dev/asmdisk1  aun=4
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
7F142949C400 00000000 00000000 00000000 00000000  [................]
        Repeat 223 times
7F142949D200 000083F9 000183F9 000283F9 000383F9  [................]
7F142949D210 000483F9 000C83F9 000D83F9 001883F9  [................]
7F142949D220 002883F9 003E83F9 007983F9 00AB83F9  [..(...>...y.....]
7F142949D230 013883F9 016C83F9 044583F9 04B083F9  [..8...l...E.....]
7F142949D240 061A83F9 0CD083F9 1E8483F9 00000000  [................]
7F142949D250 00000000 00000000 00000000 00000000  [................]
  Repeat 26 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

[root@racdb3 scsi_host]#  kfed read /dev/asmdisk1  aun=5
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
7F0615CF6400 00000000 00000000 00000000 00000000  [................]
  Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

磁盘前几个au被破坏严重.而且相关的备份block都已经损坏,基于这种情况,直接参考:
asm磁盘dd破坏恢复
 asm disk header 彻底损坏恢复
 asm disk 磁盘部分被清空恢复
通过底层恢复出来相关数据文件,并检测正常

进一步通过au分配列表获恢复redo,ctl等文件

H:\TEMP\asm-ext4\other>dir 
 驱动器 H 中的卷是 SSD-SX
 卷的序列号是 84EB-F434

 H:\TEMP\asm-ext4\other 的目录

2022-09-30  21:52        25,165,824 256.dd
2022-09-30  21:52        25,165,824 257.dd
2022-09-30  23:52        52,429,312 258.dd.1
2022-09-30  23:54        52,429,312 259.dd.1
2022-09-30  23:55        52,429,312 260.dd.1
2022-09-30  23:55        52,429,312 261.dd.1
2022-09-30  23:56        52,429,312 270.dd.1
2022-09-30  23:57        52,429,312 271.dd.1
2022-09-30  23:57        52,429,312 272.dd.1
2022-09-30  23:57        52,429,312 273.dd.1
2022-09-30  23:58        52,429,312 274.dd.1
2022-10-01  00:01        52,429,312 275.dd.1
2022-10-01  00:00        52,429,312 276.dd.1
2022-10-01  00:00        52,429,312 277.dd.1
2022-10-01  00:00        52,429,312 278.dd.1
2022-09-30  23:59        52,429,312 279.dd.1
2022-09-30  23:59        52,429,312 280.dd.1
2022-09-30  23:59        52,429,312 281.dd.1

在另外的新机器上尝试恢复库

[oracle@xifenfei ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sat Oct 1 10:18:58 2022

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount pfile='/tmp/pfile'
ORACLE instance started.

Total System Global Area 1519898624 bytes
Fixed Size                  2253464 bytes
Variable Size             939527528 bytes
Database Buffers          570425344 bytes
Redo Buffers                7692288 bytes
ORA-00227: corrupt block detected in control file: (block 8, # blocks 1)
ORA-00202: control file: '/oradata/256.dd'

控制文件损坏,重建ctl

SQL> CREATE CONTROLFILE REUSE DATABASE "xifenfei" NORESETLOGS  NOARCHIVELOG  
  2      MAXLOGFILES 50  
  3      MAXLOGMEMBERS 5  
  4      MAXDATAFILES 100  
  5      MAXINSTANCES 8  
  6      MAXLOGHISTORY 226  
  7  LOGFILE  
  8  group 7  '/oradata/270.dd.1' size 50M,
  9  group 8  '/oradata/272.dd.1' size 50M,
 10  group 5  '/oradata/274.dd.1' size 50M,
 11  group 6  '/oradata/276.dd.1' size 50M,
 12  group 3  '/oradata/278.dd.1' size 50M,
 13  group 4  '/oradata/280.dd.1' size 50M,
 14  group 1  '/oradata/258.dd.1' size 50M,
 15  group 2  '/oradata/260.dd.1' size 50M
 16  DATAFILE  
 17  '/oradata/1',
 18  '/oradata/2',
 19  '/oradata/3',
 20  '/oradata/4',
 21  '/oradata/5',
 22  '/oradata/6',
 23  '/oradata/7',
 24  '/oradata/8',
 25  '/oradata/9',
 26  '/oradata/10',
 27  '/oradata/11'
 28  CHARACTER SET ZHS16GBK  
 29  ;  

Control file created.

尝试open库,报ORA-600 kqfidps_update_stats:2，ORA-600 4194等错误

SQL> recover database;
Media recovery complete.
SQL> alter database open ;
alter database open 
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kqfidps_update_stats:2],
[0x7FFCCBEB3EC0], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [19319], [l.ok

解决该异常，open数据库成功

SQL> startup mount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area 1519898624 bytes
Fixed Size                  2253464 bytes
Variable Size             939527528 bytes
Database Buffers          570425344 bytes
Redo Buffers                7692288 bytes
Database mounted.
SQL> alter database open;

Database altered.

导出数据库,遭遇个别表如下ORA-08103和ORA-01555两种错误，这种是由于个别block在做成文件系统的时候被损坏,底层恢复的时候block被置空导致,对其异常表进行单独处理即可

. . 正在导出表                           ALBUM
EXP-00056: 遇到 ORACLE 错误 8103
ORA-08103: 对象不再存在

. . 正在导出表                  M_PUSH_CONTENT
EXP-00056: 遇到 ORACLE 错误 1555
ORA-01555: 快照过旧: 回退段号  (名称为 "") 过小
ORA-22924: 快照太旧

通过上述操作,实现客户数据的恢复,最大限度挽回客户损坏,再次提醒对于asm disk进行了误操作,建议第一时间保护现场(不要有任何的写入操作,可以最大限度恢复数据)

ORA-15335 ORA-15130 ORA-15066 ORA-15196

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-15335 ORA-15130 ORA-15066 ORA-15196

客户反馈,数据库无法正常启动，通过分析asm的alert日志发现,data磁盘组mount成功之后,没有一会儿自动dismount掉

Mon Sep 26 16:40:14 2022
SQL> /* ASMCMD */ALTER DISKGROUP data MOUNT  
NOTE: cache registered group DATA number=2 incarn=0x9dfa705f
NOTE: cache began mount (first) of group DATA number=2 incarn=0x9dfa705f
NOTE: Assigning number (2,1) to disk (/dev/oracleasm/disks/DATA02)
NOTE: Assigning number (2,0) to disk (/dev/oracleasm/disks/DATA01)
Mon Sep 26 16:40:20 2022
NOTE: GMON heartbeating for grp 2
GMON querying group 2 at 68 for pid 25, osid 14650
NOTE: cache opening disk 0 of grp 2: DATA_0000 path:/dev/oracleasm/disks/DATA01
NOTE: F1X0 found on disk 0 au 2 fcn 0.0
NOTE: cache opening disk 1 of grp 2: DATA_0001 path:/dev/oracleasm/disks/DATA02
NOTE: cache mounting (first) external redundancy group 2/0x9DFA705F (DATA)
Mon Sep 26 16:40:20 2022
* allocate domain 2, invalid = TRUE 
kjbdomatt send to inst 2
Mon Sep 26 16:40:20 2022
NOTE: attached to recovery domain 2
NOTE: cache recovered group 2 to fcn 0.321845
NOTE: redo buffer size is 256 blocks (1053184 bytes)
Mon Sep 26 16:40:20 2022
NOTE: LGWR attempting to mount thread 1 for diskgroup 2 (DATA)
NOTE: LGWR found thread 1 closed at ABA 20.3546
NOTE: LGWR mounted thread 1 for diskgroup 2 (DATA)
NOTE: LGWR opening thread 1 at fcn 0.321845 ABA 21.3547
NOTE: cache mounting group 2/0x9DFA705F (DATA) succeeded
NOTE: cache ending mount (success) of group DATA number=2 incarn=0x9dfa705f
Mon Sep 26 16:40:20 2022
NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 2
SUCCESS: diskgroup DATA was mounted
SUCCESS: /* ASMCMD */ALTER DISKGROUP data MOUNT 
Mon Sep 26 16:40:22 2022
WARNING: failed to online diskgroup resource ora.DATA.dg (unable to communicate with CRSD/OHASD)
Mon Sep 26 16:40:47 2022
NOTE: client xff1:xff registered, osid 14742, mbr 0x0
Mon Sep 26 16:40:57 2022
WARNING: cache read  a corrupt block: group=2(DATA) dsk=1 blk=257 disk=1 (DATA_0001) 
incarn=3916071178 au=113792 blk=1 count=1
Errors in file /opt/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14778.trc:
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]
NOTE: a corrupted block from group DATA was dumped to /opt/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14778.trc
WARNING: cache read (retry) a corrupt block: group=2(DATA) dsk=1 blk=257 
disk=1 (DATA_0001) incarn=3916071178 au=113792 blk=1 count=1
Errors in file /opt/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14778.trc:
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]
ERROR: cache failed to read group=2(DATA) dsk=1 blk=257 from disk(s): 1(DATA_0001)
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]
NOTE: cache initiating offline of disk 1 group DATA
NOTE: process _user14778_+asm1 (14778) initiating offline of 
disk 1.3916071178 (DATA_0001) with mask 0x7e in group 2
NOTE: initiating PST update: grp = 2, dsk = 1/0xe96a810a, mask = 0x6a, op = clear
Mon Sep 26 16:40:58 2022
GMON updating disk modes for group 2 at 70 for pid 28, osid 14778
ERROR: Disk 1 cannot be offlined, since diskgroup has external redundancy.
ERROR: too many offline disks in PST (grp 2)
Mon Sep 26 16:40:58 2022
NOTE: cache dismounting (not clean) group 2/0x9DFA705F (DATA) 
WARNING: Offline for disk DATA_0001 in mode 0x7f failed.
NOTE: messaging CKPT to quiesce pins Unix process pid: 14782, image: oracle@oracle11grac1 (B000)
Mon Sep 26 16:40:58 2022
NOTE: halting all I/Os to diskgroup 2 (DATA)
Errors in file /opt/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14778.trc  (incident=144548):
ORA-15335: ASM metadata corruption detected in disk group 'DATA'
ORA-15130: diskgroup "DATA" is being dismounted
ORA-15066: offlining disk "DATA_0001" in group "DATA" may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]
Incident details in: /opt/grid/diag/asm/+asm/+ASM1/incident/incdir_144548/+ASM1_ora_14778_i144548.trc
Mon Sep 26 16:40:58 2022
Sweep [inc][144548]: completed
System State dumped to trace file /opt/grid/diag/asm/+asm/+ASM1/incident/incdir_144548/+ASM1_ora_14778_i144548.trc
Mon Sep 26 16:40:58 2022
NOTE: AMDU dump of disk group DATA created at /opt/grid/diag/asm/+asm/+ASM1/incident/incdir_144548
Mon Sep 26 16:41:00 2022
NOTE: LGWR doing non-clean dismount of group 2 (DATA)
NOTE: LGWR sync ABA=21.3550 last written ABA 21.3550
Mon Sep 26 16:41:00 2022
Sweep [inc2][144548]: completed
Mon Sep 26 16:41:00 2022
ERROR: ORA-15130 in COD recovery for diskgroup 2/0x9dfa705f (DATA)
ERROR: ORA-15130 thrown in RBAL for group number 2
Errors in file /opt/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_5162.trc:
ORA-15130: diskgroup "DATA" is being dismounted

这里看主要是由于asm 磁盘组需要做COD recovery导致无法正常稳定的mount,主要原因是遭遇到asm disk的逻辑坏块(存储物理上看是ok的,但是实际数据在asm中看是异常的)

数据库alert日志报错

Mon Sep 26 16:40:52 2022
Successful mount of redo thread 1, with mount id 1097279951
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Lost write protection disabled
Completed: alter database mount
alter database open
This instance was first to open
Picked broadcast on commit scheme to generate SCNs
LGWR: STARTING ARCH PROCESSES
Mon Sep 26 16:40:56 2022
ARC0 started with pid=40, OS id=14761 
ARC0: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Mon Sep 26 16:40:57 2022
ARC1 started with pid=41, OS id=14764 
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_lgwr_14479.trc:
ORA-00313: ??????? 1 (???? 1) ???
Mon Sep 26 16:40:57 2022
ARC2 started with pid=42, OS id=14766 
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_lgwr_14479.trc:
ORA-00313: ??????? 2 (???? 1) ???
Mon Sep 26 16:40:57 2022
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_14732.trc:
ORA-00313: open failed for members of log group 1 of thread 1
Mon Sep 26 16:40:57 2022
ARC3 started with pid=44, OS id=14770 
ARC1: Archival started
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_14732.trc:
ORA-00313: open failed for members of log group 1 of thread 1
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_arc2_14766.trc:
ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_arc1_14764.trc:
ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_14732.trc  (incident=180281):
ORA-15335: ASM metadata corruption detected in disk group 'DATA'
ORA-15130: diskgroup "DATA" is being dismounted
ORA-15066: offlining disk "DATA_0001" in group "DATA" may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_arc0_14761.trc:
ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员
ORA-00312: 联机日志 1 线程 1: '+DATA/xff/onlinelog/group_1.271.1025610215'
ORA-17503: ksfdopn: 2 未能打开文件 +DATA/xff/onlinelog/group_1.271.1025610215
ORA-15130: diskgroup "DATA" is being dismounted
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_arc3_14770.trc:
ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员
ORA-00312: 联机日志 1 线程 1: '+DATA/xff/onlinelog/group_1.271.1025610215'
ORA-17503: ksfdopn: 2 未能打开文件 +DATA/xff/onlinelog/group_1.271.1025610215
ORA-15130: diskgroup "DATA" is being dismounted
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_arc0_14761.trc:
ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员
ORA-00312: 联机日志 1 线程 1: '+DATA/xff/onlinelog/group_1.271.1025610215'
ORA-17503: ksfdopn: 2 未能打开文件 +DATA/xff/onlinelog/group_1.271.1025610215
ORA-15130: diskgroup "DATA" is being dismounted
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_arc3_14770.trc:
ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员
ORA-00312: 联机日志 1 线程 1: '+DATA/xff/onlinelog/group_1.271.1025610215'
ORA-17503: ksfdopn: 2 未能打开文件 +DATA/xff/onlinelog/group_1.271.1025610215
ORA-15130: diskgroup "DATA" is being dismounted
Unable to create archive log file '+DATA'
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_14732.trc:
ORA-19816: WARNING: Files may exist in db_recovery_file_dest that are not known to database.
ORA-17502: ksfdcre:4 Failed to create file +DATA
ORA-15335: ASM metadata corruption detected in disk group 'DATA'
ORA-15130: diskgroup "DATA" is being dismounted
ORA-15066: offlining disk "DATA_0001" in group "DATA" may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483649] [257] [0 != 1]
*************************************************************
WARNING: A file of type ARCHIVED LOG may exist in
db_recovery_file_dest that is not known to the database.
Use the RMAN command CATALOG RECOVERY AREA to re-catalog
any such files. If files cannot be cataloged, then manually
delete them using OS command. This is most likely the
result of a crash during file creation.
*************************************************************
ARCH: Error 19504 Creating archive log file to '+DATA'
NOTE: Deferred communication with ASM instance
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_14732.trc:
ORA-15130: diskgroup "DATA" is being dismounted
NOTE: deferred map free for map id 23
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_14732.trc:
ORA-16038: log 1 sequence# 14235 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 1 thread 1: '+DATA/xff/onlinelog/group_1.271.1025610215'
ORA-00312: online log 1 thread 1: '+ARCH/xff/onlinelog/group_1.279.1025610217'
Mon Sep 26 16:40:58 2022
Sweep [inc][180281]: completed
Sweep [inc2][180281]: completed
USER (ospid: 14732): terminating the instance due to error 16038
Mon Sep 26 16:40:59 2022
System state dump requested by (instance=1, osid=14732), summary=[abnormal instance termination].
Instance terminated by USER, pid = 14732

对于这类故障处理相对比较容易,通过patch asm，让data磁盘组稳定mount,然后open库,迁移数据,实现数据0丢失,完美恢复

ORA-01558: out of transaction ID’s in rollback segment SYSTEM

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-01558: out of transaction ID’s in rollback segment SYSTEM

有客户数据库启动报ORA-01558: out of transaction ID’s in rollback segment SYSTorcl无法open成功,而且对应的备库也无法正常open(报同样错误)

对ORA-01558错误分析

[oracle@iZbp1hx0enix3hix1kvyrxZ ~]$  oerr ora 1558
01558, 00000, "out of transaction ID's in rollback segment %s"
// *Cause: All the available transaction id's have been used
// *Action: Shutdown the instance and restart using other rollback segment(s),
//          then drop the rollback segment that has no more transaction id's.

官方对于该错误的解释为由于回滚段的事务ID被使用完,无法再分配从而报ORA-01558,解决方法是重启库使用其他回滚段,并删除这个异常回滚段.对于此次故障的报错,这个方法原则上行不通,因为异常的是systorcl里面的rollback回滚段(也就是我们通常说的系统回滚段).通过alert日志分析客户这个故障的前因后果。
正常运行的库,突然报该错误

Fri Jun 03 19:37:36 2022
Thread 1 advanced to log sequence 2766146 (LGWR switch)
  Current log# 3 seq# 2766146 morcl# 0: /oradata/orclr/group_3.310.962631159
Fri Jun 03 19:37:36 2022
LNS: Standby redo logfile selected for thread 1 sequence 2766146 for destination LOG_ARCHIVE_DEST_2
Fri Jun 03 19:37:37 2022
Archived Log entry 3560157 added for thread 1 sequence 2766145 ID 0xefc56f40 dest 1:
Fri Jun 03 19:37:58 2022
Errors in file /u01/app/oracle/diag/rdbms/orclr/orclr/trace/orclr_smon_9974.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTorcl
Errors in file /u01/app/oracle/diag/rdbms/orclr/orclr/trace/orclr_smon_9974.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTorcl
Errors in file /u01/app/oracle/diag/rdbms/orclr/orclr/trace/orclr_smon_9974.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTorcl
Errors in file /u01/app/oracle/diag/rdbms/orclr/orclr/trace/orclr_smon_9974.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTorcl
Errors in file /u01/app/oracle/diag/rdbms/orclr/orclr/trace/orclr_smon_9974.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTorcl
Fri Jun 03 19:38:13 2022
Errors in file /u01/app/oracle/diag/rdbms/orclr/orclr/trace/orclr_smon_9974.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTorcl

这个错误一直报了很长时间,等到有时间窗口,客户安排重启操作

Mon Sep 26 14:10:12 2022
Shutting down instance (immediate)
Shutting down instance: further logons disabled
License high water mark = 205
All dispatchers and shared servers shutdown
Errors in file /u01/app/oracle/diag/rdbms/orclr/orclr/trace/orclr_smon_9974.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTorcl
ALTER DATABASE CLOSE NORMAL
ORA-1558 signalled during: ALTER DATABASE CLOSE NORMAL...
Mon Sep 26 14:10:22 2022
Errors in file /u01/app/oracle/diag/rdbms/orclr/orclr/trace/orclr_smon_9974.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTorcl
Mon Sep 26 14:10:33 2022
Errors in file /u01/app/oracle/diag/rdbms/orclr/orclr/trace/orclr_smon_9974.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTorcl
Mon Sep 26 14:10:43 2022
Errors in file /u01/app/oracle/diag/rdbms/orclr/orclr/trace/orclr_smon_9974.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTorcl
Mon Sep 26 14:10:53 2022
Errors in file /u01/app/oracle/diag/rdbms/orclr/orclr/trace/orclr_smon_9974.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTorcl
Mon Sep 26 14:10:55 2022
Shutting down instance (abort)
License high water mark = 205
USER (ospid: 25049): terminating the instance
Instance terminated by USER, pid = 25049

正常shutdown immediate执行失败,直接abort方式关闭库,然后尝试重启库

Mon Sep 26 14:12:16 2022
ARC3 started with pid=44, OS id=25978 
ARC1: Archival started
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Thread 1 opened at log sequence 2782827
  Current log# 1 seq# 2782827 morcl# 0: /oradata/orclr/group_1.296.962631151
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Sep 26 14:12:16 2022
SMON: enabling cache recovery
Mon Sep 26 14:12:16 2022
NSA2 started with pid=46, OS id=25982 
Archived Log entry 3593518 added for thread 1 sequence 2782826 ID 0xefc56f40 dest 1:
ARC3: Standby redo logfile selected for thread 1 sequence 2782826 for destination LOG_ARCHIVE_DEST_2
Errors in file /u01/app/oracle/diag/rdbms/orclr/orclr/trace/orclr_ora_25879.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTorcl
Errors in file /u01/app/oracle/diag/rdbms/orclr/orclr/trace/orclr_ora_25879.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTorcl
Error 604 happened during db open, shutting down database
USER (ospid: 25879): terminating the instance due to error 604
Instance terminated by USER, pid = 25879
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (25879) as a result of ORA-1092
Mon Sep 26 14:12:17 2022
ORA-1092 : opitsk aborting process

数据库重启失败,报ORA-604和ORA-01558错误.
对于这种情况,对于数据库启动过程进行跟踪确认在UPDATE UNDO$的时候无法分配事务,导致无法继续

PARSING IN CURSOR #139696084476000 len=160 dep=1 uid=0 oct=6 lid=0 tim=1664353484223278 hv=1292341136 
ad='6ff1ffa8' sqlid='8vyjutx6hg3wh'
update /*+ rule */ undo$ set name=:2,file#=:3,block#=:4,status$=:5,user#=:6,undosqn=:7,xactsqn=:8,
scnbas=:9,scnwrp=:10,inst#=:11,ts#=:12,spare1=:13 where us#=:1
END OF STMT
PARSE #139696084476000:c=5998,e=10708,p=7,cr=53,cu=0,mis=1,r=0,dep=1,og=3,plh=0,tim=1664353484223277
BINDS #139696084476000:
 Bind#0
  oacdty=01 mxl=32(21) mxlc=00 mal=00 scl=00 pre=00
  oacflg=18 fl2=0001 frm=01 csi=852 siz=32 off=0
  kxsbbbfp=6ff376da  bln=32  avl=21  flg=09
  value="_SYSSMU14_3733658264$"
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7f0d877d7a68  bln=24  avl=02  flg=05
  value=15
 Bind#2
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7f0d877d7a38  bln=24  avl=03  flg=05
  value=336
 Bind#3
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7f0d877d7a00  bln=24  avl=02  flg=05
  value=5
 Bind#4
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7f0d877d79d0  bln=24  avl=02  flg=05
  value=1
 Bind#5
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7f0d877d79a0  bln=24  avl=03  flg=05
  value=5747
 Bind#6
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7f0d877d7970  bln=24  avl=04  flg=05
  value=22103
 Bind#7
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7f0d877d7938  bln=24  avl=06  flg=05
  value=1026171661
 Bind#8
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7f0d877d7680  bln=24  avl=03  flg=05
  value=3399
 Bind#9
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7f0d877d7650  bln=24  avl=02  flg=05
  value=2
 Bind#10
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7f0d877d7620  bln=24  avl=02  flg=05
  value=5
 Bind#11
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7f0d877d75f0  bln=24  avl=02  flg=05
  value=2
 Bind#12
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7f0d877d7a98  bln=22  avl=02  flg=05
  value=14
EXEC #139696084476000:c=1000,e=1713,p=0,cr=1,cu=2,mis=1,r=0,dep=1,og=3,plh=3078630091,tim=1664353484225072
ERROR #139696084476000:err=1558 tim=1664353484225094
STAT #139696084476000 id=1 cnt=0 pid=0 pos=1 obj=0 op='UPDATE  UNDO$ (cr=0 pr=0 pw=0 time=5 us)'
STAT #139696084476000 id=2 cnt=1 pid=1 pos=1 obj=34 op='INDEX UNIQUE SCAN I_UNDO1 (cr=1 pr=0 pw=0 time=6 us)'
KQRCMT: Write failed with error=604 po=0x6ff375d0 cid=3
diagnostics : cid=3 hash=f2114ab9 flag=2a
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTorcl
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTorcl

既然报了事务ID不足,那对block进行分析,确实几乎都达到了数据库设计的理论最大值

  index  state cflags  wrap#    uel         scn            dba            parent-xid    nub     stmt_num
  ------------------------------------------------------------------------------------------------
   0x00    9    0x00  0xfffe  0x0026  0x111f.4d598aae  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x01    9    0x00  0xfffe  0x0054  0x111f.4d598a63  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x02    9    0x00  0xfffe  0x001a  0x111f.4d598a77  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x03    9    0x00  0xfffe  0x005c  0x111f.4d598a7d  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x04    9    0x00  0xfffe  0x003d  0x111f.4d598a4d  0x00400185  0x0000.000.00000000  0x00000001    0x00000000
   0x05    9    0x00  0xfffe  0x0061  0x111f.4d598a74  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x06    9    0x00  0xfffe  0x0002  0x111f.4d598a76  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x07    9    0x00  0xfffe  0x001b  0x111f.4d598a90  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x08    9    0x00  0xfffe  0x0011  0x111f.4d598a8c  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x09    9    0x00  0xfffe  0x0042  0x111f.4d598a6e  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x0a    9    0x00  0xfffe  0x0003  0x111f.4d598a7c  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x0b    9    0x00  0xfffe  0x000a  0x111f.4d598a7b  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x0c    9    0x00  0xfffe  0x0005  0x111f.4d598a73  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x0d    9    0x00  0xfffe  0x005d  0x111f.4d598a87  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x0e    9    0x00  0xfffe  0x0050  0x111f.4d598a68  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x0f    9    0x00  0xfffe  0x0047  0x111f.4d598a6a  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x10    9    0x00  0xfffe  0x0033  0x111f.4d598a95  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x11    9    0x00  0xfffe  0x0031  0x111f.4d598a8d  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x12    9    0x00  0xfffe  0x0020  0x111f.4d598a81  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x13    9    0x00  0xfffe  0x0014  0x111f.4d598a85  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x14    9    0x00  0xfffe  0x000d  0x111f.4d598a86  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x15    9    0x00  0xfffe  0x0013  0x111f.4d598a83  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x16    9    0x00  0xfffe  0x003b  0x111f.4d598aa4  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x17    9    0x00  0xfffe  0x005b  0x111f.4d598a70  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x18    9    0x00  0xfffe  0x001f  0x111f.4d598a8a  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x19    9    0x00  0xfffe  0x0010  0x111f.4d598a94  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x1a    9    0x00  0xfffe  0x001c  0x111f.4d598a79  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x1b    9    0x00  0xfffe  0x0027  0x111f.4d598a91  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x1c    9    0x00  0xfffe  0x000b  0x111f.4d598a7a  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x1d    9    0x00  0xfffe  0x0012  0x111f.4d598a80  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x1e    9    0x00  0xfffe  0x0045  0x111f.4d598aba  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x1f    9    0x00  0xfffe  0x0008  0x111f.4d598a8b  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x20    9    0x00  0xfffe  0x0015  0x111f.4d598a82  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x21    9    0x00  0xfffe  0x0038  0x111f.4d598ab2  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x22    9    0x00  0xfffd  0x003e  0x111f.4d598ab4  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x23    9    0x00  0xfffe  0x0028  0x111f.4d598a9e  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x24    9    0x00  0xfffd  0x0060  0x111f.4d598a5e  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x25    9    0x00  0xfffe  0x0021  0x111f.4d598ab0  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x26    9    0x00  0xfffe  0x0025  0x111f.4d598aaf  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x27    9    0x00  0xfffe  0x0019  0x111f.4d598a93  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x28    9    0x00  0xfffe  0x0046  0x111f.4d598a9f  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x29    9    0x00  0xfffe  0x0023  0x111f.4d598a9d  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x2a    9    0x00  0xfffe  0x002c  0x111f.4d598a98  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x2b    9    0x00  0xfffe  0x0040  0x111f.4d598aa7  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x2c    9    0x00  0xfffe  0x0030  0x111f.4d598a99  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x2d    9    0x00  0xfffe  0x003c  0x111f.4d598aab  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x2e    9    0x00  0xfffe  0x0056  0x111f.4d598abf  0x00400189  0x0000.000.00000000  0x00000001    0x00000000
   0x2f    9    0x00  0xfffe  0x0037  0x111f.4d598aa2  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x30    9    0x00  0xfffe  0x0034  0x111f.4d598a9a  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x31    9    0x00  0xfffe  0x0007  0x111f.4d598a8f  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x32    9    0x00  0xfffe  0x001e  0x111f.4d598ab9  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x33    9    0x00  0xfffe  0x002a  0x111f.4d598a97  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x34    9    0x00  0xfffe  0x0029  0x111f.4d598a9c  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x35    9    0x00  0xfffd  0x005f  0x111f.4d598a5b  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x36    9    0x00  0xfffe  0x0049  0x111f.4d598ac6  0x00400189  0x0000.000.00000000  0x00000001    0x00000000
   0x37    9    0x00  0xfffe  0x0016  0x111f.4d598aa3  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x38    9    0x00  0xfffe  0x0022  0x111f.4d598ab3  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x39    9    0x00  0xfffd  0x0058  0x111f.4d598a52  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x3a    9    0x00  0xfffd  0x0048  0x111f.4d598a57  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x3b    9    0x00  0xfffe  0x002b  0x111f.4d598aa5  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x3c    9    0x00  0xfffe  0x0000  0x111f.4d598aac  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x3d    9    0x00  0xfffd  0x0043  0x111f.4d598a4f  0x00400185  0x0000.000.00000000  0x00000001    0x00000000
   0x3e    9    0x00  0xfffe  0x003f  0x111f.4d598ab6  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x3f    9    0x00  0xfffe  0x0032  0x111f.4d598ab7  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x40    9    0x00  0xfffe  0x005a  0x111f.4d598aa8  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x41    9    0x00  0xfffe  0x004b  0x111f.4d598abc  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x42    9    0x00  0xfffd  0x0017  0x111f.4d598a6f  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x43    9    0x00  0xfffd  0x004d  0x111f.4d598a50  0x00400185  0x0000.000.00000000  0x00000001    0x00000000
   0x44    9    0x00  0xfffe  0x005e  0x111f.4d598ac2  0x00400189  0x0000.000.00000000  0x00000001    0x00000000
   0x45    9    0x00  0xfffe  0x0041  0x111f.4d598abb  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x46    9    0x00  0xfffe  0x002f  0x111f.4d598aa0  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x47    9    0x00  0xfffd  0x0059  0x111f.4d598a6b  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x48    9    0x00  0xfffd  0x0052  0x111f.4d598a58  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x49    9    0x00  0xfffe  0xffff  0x111f.4d598ac7  0x00400189  0x0000.000.00000000  0x00000001    0x00000000
   0x4a    9    0x00  0xfffd  0x003a  0x111f.4d598a56  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x4b    9    0x00  0xfffe  0x002e  0x111f.4d598abe  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x4c    9    0x00  0xfffd  0x0024  0x111f.4d598a5d  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x4d    9    0x00  0xfffd  0x0039  0x111f.4d598a51  0x00400185  0x0000.000.00000000  0x00000001    0x00000000
   0x4e    9    0x00  0xfffd  0x0001  0x111f.4d598a62  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x4f    9    0x00  0xfffd  0x000e  0x111f.4d598a66  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x50    9    0x00  0xfffd  0x000f  0x111f.4d598a69  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x51    9    0x00  0xfffe  0x0036  0x111f.4d598ac5  0x00400189  0x0000.000.00000000  0x00000001    0x00000000
   0x52    9    0x00  0xfffd  0x0035  0x111f.4d598a59  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x53    9    0x00  0xfffd  0x004a  0x111f.4d598a55  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x54    9    0x00  0xfffd  0x0055  0x111f.4d598a64  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x55    9    0x00  0xfffd  0x004f  0x111f.4d598a65  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x56    9    0x00  0xfffe  0x0044  0x111f.4d598ac1  0x00400189  0x0000.000.00000000  0x00000001    0x00000000
   0x57    9    0x00  0xfffd  0x004e  0x111f.4d598a60  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x58    9    0x00  0xfffd  0x0053  0x111f.4d598a53  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x59    9    0x00  0xfffd  0x0009  0x111f.4d598a6c  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x5a    9    0x00  0xfffd  0x002d  0x111f.4d598aa9  0x00400188  0x0000.000.00000000  0x00000001    0x00000000
   0x5b    9    0x00  0xfffd  0x000c  0x111f.4d598a71  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x5c    9    0x00  0xfffd  0x001d  0x111f.4d598a7f  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x5d    9    0x00  0xfffd  0x0018  0x111f.4d598a88  0x00400187  0x0000.000.00000000  0x00000001    0x00000000
   0x5e    9    0x00  0xfffe  0x0051  0x111f.4d598ac3  0x00400189  0x0000.000.00000000  0x00000001    0x00000000
   0x5f    9    0x00  0xfffd  0x004c  0x111f.4d598a5c  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x60    9    0x00  0xfffd  0x0057  0x111f.4d598a5f  0x00400186  0x0000.000.00000000  0x00000001    0x00000000
   0x61    9    0x00  0xfffd  0x0006  0x111f.4d598a75  0x00400187  0x0000.000.00000000  0x00000001    0x00000000

通过bbed修改相关值,规避掉此类问题

m /x 6c000000 offset 6513
m /x 7c000000 offset 6752
m /x 8c000000 offset 7048

启动数据库成功

又一例asm disk 加入vg故障

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：又一例asm disk 加入vg故障

又一客户把asm disk加入到vg,并且扩容到lv中

通过asm层面查看

ASMCMD> lsdsk
Path
/dev/asmdisk/asm-arch01
/dev/asmdisk/asm-data01
/dev/asmdisk/asm-ocr01
/dev/asmdisk/asm-ocr02
/dev/asmdisk/asm-ocr03
dbsrv2-> ls -l /dev/asm*
/dev/asm:
total 0

/dev/asmdisk:
total 0
lrwxrwxrwx 1 root root 6 Sep 14 17:39 asm-arch01 -> ../sdb
lrwxrwxrwx 1 root root 6 Sep 14 17:35 asm-data01 -> ../sda
lrwxrwxrwx 1 root root 6 Sep 11 09:11 asm-ocr01 -> ../sdc
lrwxrwxrwx 1 root root 6 Sep 11 09:11 asm-ocr02 -> ../sdd
lrwxrwxrwx 1 root root 6 Sep 11 09:11 asm-ocr03 -> ../sde

对于这类情况,由于客户的系统是ext4,根据这个文件系统特性,每隔2G会有一点破坏,最终数据恢复效果看运气,运气好直接通过元数据恢复出来所有数据文件,然后open库,然后不好可能需要底层碎片等,参见类似恢复:
asm disk被加入vg恢复
 asm disk 磁盘部分被清空恢复
 文件系统重新分区oracle恢复
 删除分区 oracle asm disk 恢复
 pvcreate asm disk导致asm磁盘组异常恢复
对于使用asm的客户,在对文件系统进行操作时,一定要注意asm disk,别弄错磁盘（把asm disk磁盘给误操作掉了）,适当情况下linux平台可以考虑AFD(ASM FILTER DRIVER)

.Devos加密数据库恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：.Devos加密数据库恢复

有客户win文件系统中勒索加密,其中数据库文件被加密

通过资源的oracle勒索加密恢复工具进行处理

直接顺利open数据库,并且导出数据

C:\Windows\system32>expdp "'/ as sysdba'"  schemas=XFF  DIRECTORY=expdp_dir logfile=XFF.log  dumpfile=XFF.dmp

Export: Release 11.2.0.1.0 - Production on 星期四 9月 22 18:05:59 2022

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

连接到: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
启动 "SYS"."SYS_EXPORT_SCHEMA_01":  "/******** AS SYSDBA" schemas=XFF DIRECTORY=expdp_dir logfile=XFF.log dumpfile=XFF.dmp
正在使用 BLOCKS 方法进行估计...
处理对象类型 SCHEMA_EXPORT/TABLE/TABLE_DATA
使用 BLOCKS 方法的总估计: 4.089 GB
处理对象类型 SCHEMA_EXPORT/USER
处理对象类型 SCHEMA_EXPORT/SYSTEM_GRANT
处理对象类型 SCHEMA_EXPORT/ROLE_GRANT
处理对象类型 SCHEMA_EXPORT/DEFAULT_ROLE
处理对象类型 SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA
处理对象类型 SCHEMA_EXPORT/SEQUENCE/SEQUENCE
处理对象类型 SCHEMA_EXPORT/TABLE/TABLE
处理对象类型 SCHEMA_EXPORT/TABLE/INDEX/INDEX
处理对象类型 SCHEMA_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
处理对象类型 SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
处理对象类型 SCHEMA_EXPORT/TABLE/COMMENT
处理对象类型 SCHEMA_EXPORT/PACKAGE/PACKAGE_SPEC
处理对象类型 SCHEMA_EXPORT/FUNCTION/FUNCTION
处理对象类型 SCHEMA_EXPORT/PROCEDURE/PROCEDURE
处理对象类型 SCHEMA_EXPORT/PACKAGE/COMPILE_PACKAGE/PACKAGE_SPEC/ALTER_PACKAGE_SPEC
处理对象类型 SCHEMA_EXPORT/FUNCTION/ALTER_FUNCTION
处理对象类型 SCHEMA_EXPORT/PROCEDURE/ALTER_PROCEDURE
处理对象类型 SCHEMA_EXPORT/VIEW/VIEW
处理对象类型 SCHEMA_EXPORT/PACKAGE/PACKAGE_BODY
处理对象类型 SCHEMA_EXPORT/TABLE/CONSTRAINT/REF_CONSTRAINT
处理对象类型 SCHEMA_EXPORT/TABLE/TRIGGER
. . 导出了 "XFF"."LIS_R_IMAGE_7"                      608.4 MB   29616 行
. . 导出了 "XFF"."LIS_R_IMAGE_36"                     660.6 MB    8698 行
. . 导出了 "XFF"."LIS_REPORT"                         9.553 MB   36826 行
. . 导出了 "XFF"."LIS_R_DETAIL_10"                    584.9 KB   11816 行
. . 导出了 "XFF"."SCV_SAMPLE_INFO"                    9.428 MB   38148 行
. . 导出了 "XFF"."SVC_SAMPLE_INST"                    1.537 MB   47328 行
. . 导出了 "XFF"."LIS_REPORT_CLASS"                   79.91 MB 1226246 行
……………………………………
. . 导出了 "XFF"."SY_FILTERCASE"                          0 KB       0 行
. . 导出了 "XFF"."SY_FILTERDETAIL"                        0 KB       0 行
. . 导出了 "XFF"."SY_TABLE"                               0 KB       0 行
. . 导出了 "XFF"."XK_CODE_REASON"                         0 KB       0 行
已成功加载/卸载了主表 "SYS"."SYS_EXPORT_SCHEMA_01"
******************************************************************************
SYS.SYS_EXPORT_SCHEMA_01 的转储文件集为:
  E:\XFF.DMP
作业 "SYS"."SYS_EXPORT_SCHEMA_01" 已于 18:06:32 成功完成

对于类似这种被加密的勒索的数据文件（.id[0D160C61-3327].[ferguson@cock.li].Devos）,我们可以实现比较好的恢复效果,如果此类的数据库(oracle,mysql,sql server)等被加密,需要专业恢复技术支持，请联系我们:
电话/微信:17813235971 Q Q:107644445 E-Mail:dba@xifenfei.com
系统安全防护措施建议：
1.多台机器，不要使用相同的账号和口令
2.登录口令要有足够的长度和复杂性，并定期更换登录口令
3.重要资料的共享文件夹应设置访问权限控制，并进行定期备份
4.定期检测系统和软件中的安全漏洞，及时打上补丁。
5.定期到服务器检查是否存在异常。
6.安装安全防护软件，并确保其正常运行。
7.从正规渠道下载安装软件。
8.对不熟悉的软件，如果已经被杀毒软件拦截查杀，不要添加信任继续运行。
9.保存良好的备份习惯，尽量做到每日备份，异地备份。

oracle启动报ORA-600 kdBlkCheckError故障解决

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：oracle启动报ORA-600 kdBlkCheckError故障解决

数据库启动报ORA-600 kdBlkCheckError错误

SQL> alter database open ;
alter database open 
*
第 1 行出现错误:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [3], [144], [38504]
进程 ID: 17516
会话 ID: 14 序列号: 5

根据ORA-600 kdBlkCheckError的经验,这个错误是3号文件的144号block逻辑不一致导致.通过dbv检查该文件

Microsoft Windows [版本 10.0.19044.1949]
(c) Microsoft Corporation。保留所有权利。

C:\Users\XFF>dbv file=H:\BaiduNetdisk\oradata\XFF\UNDOTBS1.DBF

DBVERIFY: Release 11.2.0.4.0 - Production on 星期六 9月 17 10:51:32 2022

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = H:\BAIDUNETDISK\ORADATA\SPECTRA\UNDOTBS1.DBF
页 112 失败, 校验代码为 18018
Block Checking: DBA = 12583056, Block Type = System Managed Segment Header Block
ERROR: SMU Segment Header Corrupted.  Error Code = 38504
ktu4smck: SCN commited txn list is not sorted.
  previous txn slot=23, scn=0x0000.ee917d05
  offending txn slot=18, scn=0x0000.ee916272
  TRN CTL:: seq: 0x0c3f chd: 0x0017 ctl: 0x0018 inc: 0x00000000 nfb: 0x0001
            mgc: 0xb000 xts: 0x0068 flg: 0x0001 opt: 2147483646 (0x7ffffffe)
            uba: 0x00c087c8.0c3f.05 scn: 0x0000.ee9160d2
            Version: 0x01
  FREE BLOCK POOL::
    uba: 0x00c087c8.0c3f.05  ext: 0xe  spc: 0x1594
    uba: 0x00000000.0bfe.03  ext: 0x12 spc: 0x1eb8
    uba: 0x00000000.0b20.04  ext: 0x4  spc: 0x1d2e
    uba: 0x00000000.c6e5.01  ext: 0x2  spc: 0x1f84
    uba: 0x00000000.0000.00  ext: 0x0  spc: 0x0
  TRN TBL::
  index  state cflags  wrap#    uel         scn            dba            parent-xid    nub       bcl     cmt
  -----------------------------------------------------------------------------------------
   0x00  9 0x00  0x62878b  0xffe4  0x0000.ee917f13  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031369
   0x01 10 0x80  0x62887e  0x001f  0x0000.ee917efe  0x00c087d6  0x0000.000.00000000  0x00000001 0x00000000  38
   0x02  9 0x00  0x628871  0xffde  0x0000.ee917cce  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031373
   0x03  9 0x00  0x62865f  0x0000  0x0000.ee916279  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x04  9 0x00  0x628823  0x0011  0x0000.ee916083  0x00c087d4  0x0000.000.00000000  0x00000001 0x00000000  1663031371
   0x05  9 0x00  0x6287b3  0x0000  0x0000.ee917e56  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x06  9 0x00  0x628893  0x001c  0x0000.ee916288  0x00002338  0x0000.000.00000000  0x00000000 0x00000000  1663031371
   0x07  9 0x00  0x628820  0x0011  0x0000.ee917f66  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031373
   0x08  9 0x00  0x628833  0x000c  0x0000.ee917eaf  0x00c087d4  0x0000.000.00000000  0x00000001 0x00000000  1663031371
   0x09  9 0x00  0x628815  0x0000  0x0000.ee917e72  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x0a  9 0x00  0x628863  0x0002  0x0000.ee917cc1  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031373
   0x0b  9 0x00  0x62870c  0x0008  0x0000.ee916085  0x00c087d4  0x0000.000.00000000  0x00000001 0x00000000  1663031372
   0x0c  9 0x00  0x62881a  0x0005  0x0000.ee917ff2  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x0d  9 0x00  0x6289fb  0x000f  0x0000.ee917d1d  0x00c087d4  0x0000.000.00000000  0x00000001 0x00000000  1663031375
   0x0e  9 0x00  0x6287d8  0x000a  0x0000.ee91638a  0x00c087d6  0x0000.000.00000000  0x00000001 0x00000000  1663031371
   0x0f  9 0x00  0x62880c  0x001b  0x0000.ee91619e  0x00003003  0x0000.000.00000000  0x00000000 0x00000000  1663031370
   0x10  9 0x00  0x6287e6  0x0013  0x0000.ee9161fc  0x00c087d5  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x11  9 0x00  0x62863b  0x0019  0x0000.ee916354  0x00c087d5  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x12  9 0x00  0x6287d4  0x0010  0x0000.ee916272  0x00c087d6  0x0000.000.00000000  0x00000001 0x00000000  1663031371
   0x13  9 0x00  0x628470  0x0007  0x0000.ee9160b2  0x00c087d5  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x14  9 0x00  0x6287a4  0x001e  0x0000.ee91627f  0x00c087d6  0x0000.000.00000000  0x00000001 0x00000000  1663031372
   0x15  9 0x00  0x628797  0x000a  0x0000.ee9162bb  0x00c087c9  0x0000.000.00000000  0x00000001 0x00000000  1663031368
   0x16  9 0x00  0x6287ad  0x0005  0x0000.ee917f6c  0x00c087d4  0x0000.000.00000000  0x00000001 0x00000000  1663031371
   0x17  9 0x00  0x6287b5  0x0012  0x0000.ee917d05  0x00c087d7  0x0000.000.00000000  0x00000001 0x00000000  1663031373
   0x18  9 0x00  0x628719  0x000b  0x0000.ee916136  0x00c087d4  0x0000.000.00000000  0x00000001 0x00000000  1663031373
   0x19  9 0x00  0x628783  0x0006  0x0000.ee916363  0x00c087d5  0x0000.000.00000000  0x00000001 0x00000000  1663031370
   0x1a  9 0x00  0x6287d8  0xffff  0x0000.ee917d97  0x00c087cb  0x0000.000.00000000  0x00000001 0x00000000  1663031375
   0x1b  9 0x00  0x6287d7  0x0022  0x0000.ee916043  0x00c087d5  0x0000.000.00000000  0x00000001 0x00000000  1663031373
   0x1c  9 0x00  0x62880e  0x0005  0x0000.ee917db7  0x00002338  0x0000.000.00000000  0x00000000 0x00000000  1663031373
   0x1d  9 0x00  0x6287b7  0x0003  0x0000.ee9161e1  0x00c087d4  0x0000.000.00000000  0x00000001 0x00000000  1663031373
   0x1e  9 0x00  0x6287f6  0x0015  0x0000.ee9162e6  0x00002338  0x0000.000.00000000  0x00000000 0x00000000  1663031368
   0x1f  9 0x00  0x6287ad  0x0003  0x0000.ee917eae  0x00003003  0x0000.000.00000000  0x00000000 0x00000000  1663031372
   0x20  9 0x00  0x6287b0  0x0003  0x0000.ee9163a5  0x0000133b  0x0000.000.00000000  0x00000000 0x00000000  1663031368
   0x21  9 0x00  0x62886a  0x0001  0x0000.ee916056  0x00c087d5  0x0000.000.00000000  0x00000001 0x00000000  1663031373
  EXT TRN CTL::
  usn: 2
  sp1:0x00000000 sp2:0x00000000 sp3:0x00000000 sp4:0x00000000
  sp5:0x00000000 sp6:0x00000000 sp7:0x00000000 sp8:0x00000000
  EXT TRN TBL::
index extflag  extHash  extSpare1  extSpare2
---------------------------------------------
   0x00  0x00000000 0x00000000 0x00000000  0x00000000
   0x01  0x00000000 0x00000000 0x00000000  0x00000000
   0x02  0x00000000 0x00000000 0x00000000  0x00000000
   0x03  0x00000000 0x00000000 0x00000000  0x00000000
   0x04  0x00000000 0x00000000 0x00000000  0x00000000
   0x05  0x00000000 0x00000000 0x00000000  0x00000000
   0x06  0x00000000 0x00000000 0x00000000  0x00000000
   0x07  0x00000000 0x00000000 0x00000000  0x00000000
   0x08  0x00000000 0x00000000 0x00000000  0x00000000
   0x09  0x00000000 0x00000000 0x00000000  0x00000000
   0x0a  0x00000000 0x00000000 0x00000000  0x00000000
   0x0b  0x00000000 0x00000000 0x00000000  0x00000000
   0x0c  0x00000000 0x00000000 0x00000000  0x00000000
   0x0d  0x00000000 0x00000000 0x00000000  0x00000000
   0x0e  0x00000000 0x00000000 0x00000000  0x00000000
   0x0f  0x00000000 0x00000000 0x00000000  0x00000000
   0x10  0x00000000 0x00000000 0x00000000  0x00000000
   0x11  0x00000000 0x00000000 0x00000000  0x00000000
   0x12  0x00000000 0x00000000 0x00000000  0x00000000
   0x13  0x00000000 0x00000000 0x00000000  0x00000000
   0x14  0x00000000 0x00000000 0x00000000  0x00000000
   0x15  0x00000000 0x00000000 0x00000000  0x00000000
   0x16  0x00000000 0x00000000 0x00000000  0x00000000
   0x17  0x00000000 0x00000000 0x00000000  0x00000000
   0x18  0x00000000 0x00000000 0x00000000  0x00000000
   0x19  0x00000000 0x00000000 0x00000000  0x00000000
   0x1a  0x00000000 0x00000000 0x00000000  0x00000000
   0x1b  0x00000000 0x00000000 0x00000000  0x00000000
   0x1c  0x00000000 0x00000000 0x00000000  0x00000000
   0x1d  0x00000000 0x00000000 0x00000000  0x00000000
   0x1e  0x00000000 0x00000000 0x00000000  0x00000000
   0x1f  0x00000000 0x00000000 0x00000000  0x00000000
   0x20  0x00000000 0x00000000 0x00000000  0x00000000
   0x21  0x00000000 0x00000000 0x00000000  0x00000000
页 144 失败, 校验代码为 38504

…………

DBVERIFY - 验证完成

检查的页总数: 161280
处理的页总数 (数据): 0
失败的页总数 (数据): 0
处理的页总数 (索引): 0
失败的页总数 (索引): 0
处理的页总数 (其他): 161277
处理的总页数 (段)  : 9
失败的总页数 (段)  : 0
空的页总数: 1
标记为损坏的总页数: 4
流入的页总数: 2
加密的总页数        : 0
最高块 SCN            : 4002695098 (0.4002695098)

C:\Users\XFF>

可以确认是由于SMU Segment Header异常，导致数据库无法正常启动，通过数据库层面设置,规避数据库启动访问该block,数据库正常启动正常,并顺利导出数据

Thu Sep 15 11:02:23 2022
ALTER DATABASE   MOUNT
Successful mount of redo thread 1, with mount id 2863639551
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE   MOUNT
Thu Sep 15 11:02:31 2022
alter database open upgrade
Thread 1 opened at log sequence 660107
  Current log# 2 seq# 660107 mem# 0: H:\BAIDUNETDISK\ORADATA\XFF\REDO02.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Undo initialization finished serial:0 start:74439375 end:74439375 diff:0 (0 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Completed: alter database open

Oracle Recovery Tools快速解决sysaux文件不能online问题

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle Recovery Tools快速解决sysaux文件不能online问题

又一客户sysaux表空间对应的数据文件离线(file 6 为测试表空间数据可以不要)

Tue Jul 26 11:33:41 2022
alter database datafile 2 offline drop
Completed: alter database datafile 2 offline drop
Tue Jul 26 11:35:26 2022
alter database datafile 6 offline drop
Completed: alter database datafile 6 offline drop
Tue Jul 26 11:36:04 2022
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 19 processes
Started redo scan
Completed redo scan
 read 14595 KB redo, 954 data blocks need recovery
Started redo application at
 Thread 1: logseq 52560, block 31365
Recovery of Online Redo Log: Thread 1 Group 3 Seq 52560 Reading mem 0
  Mem# 0: D:\XXXX\DATAS\ORADATA\XXXX\REDO03.LOG
Completed redo application of 6.50MB
Completed crash recovery at
 Thread 1: logseq 52560, block 60555, scn 4397986801
 954 data blocks read, 954 data blocks written, 14595 redo k-bytes read
Tue Jul 26 11:36:11 2022
Thread 1 advanced to log sequence 52561 (thread open)
Thread 1 opened at log sequence 52561
  Current log# 1 seq# 52561 mem# 0: D:\XXXX\DATAS\ORADATA\XXXX\REDO01.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Jul 26 11:36:11 2022
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Tue Jul 26 11:36:14 2022
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Errors in file d:\XXXX\datas\diag\rdbms\XXXX\XXXX\trace\XXXX_ora_8476.trc  (incident=275156):
ORA-25319: 队列表重新分区已中止
Incident details in: d:\XXXX\datas\diag\rdbms\XXXX\XXXX\incident\incdir_275156\XXXX_ora_8476_i275156.trc
error 25319 happened during Queue table repartitioning
Starting background process QMNC
Tue Jul 26 11:36:23 2022
QMNC started with pid=50, OS id=11200 
Tue Jul 26 11:36:23 2022
Trace dumping is performing id=[cdmp_20220726113623]
 XDB UNINITIALIZED: XDB$SCHEMA not accessible 
Tue Jul 26 11:36:27 2022
Completed: ALTER DATABASE OPEN

SQL> select file#,status from v$datafile;

     FILE# STATUS
---------- -------
         1 SYSTEM
         2 OFFLINE
         3 ONLINE
         4 ONLINE
         5 ONLINE
         6 OFFLINE

7月份offline datafile 2,然后open数据库一直运行至今,数据库一直无法进行备份,需要我们进行解决

SQL> archive log list;
数据库日志模式             非存档模式
自动存档             禁用
存档终点            D:\APP\DATAS\product\11.2.0.4\dbhome_1\RDBMS
最早的联机日志序列     55557
当前日志序列           55559

SQL> recover datafile 2;
ORA-00279: 更改 4397905894 (在 07/25/2022 18:26:58 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\XXXX\DATAS\FLASH_RECOVERY_AREA\XXXX\ARCHIVELOG\2022_08_29\O1_MF_1_52560_%U_.ARC
ORA-00280: 更改 4397905894 (用于线程 1) 在序列 #52560 中


指定日志: {<RET>=suggested | filename | AUTO | CANCEL}

数据库为非归档,常规方法无法直接online datafile 2,对于这样的情况,使用Oracle Recovery Tools工具,进行快速修改文件头信息

查询文件头信息

SQL> set pages 1000
SQL> set linesize 150
SQL> select ts#,file#,TABLESPACE_NAME,status,
  2  to_char(CREATION_TIME,'yyyy-mm-dd hh24:mi:ss') CREATE_TIME,
  3  to_char(checkpoint_change#,'9999999999999999') "SCN",
  4  to_char(RESETLOGS_CHANGE#,'9999999999999999') "RESETLOGS SCN",FUZZY
  5  from v$datafile_header;

       TS#      FILE# TABLESPACE_NAME                STATUS  CREATE_TIME         SCN               RESETLOGS SCN     FUZ

---------- ---------- ------------------------------ ------- ------------------- ----------------- ----------------- ---

         0          1 SYSTEM                         ONLINE  2010-03-30 10:07:48        4599488977            947455 NO
         1          2 SYSAUX                         ONLINE  2010-03-30 10:07:52        4599488977            947455 YES
         2          3 UNDOTBS1                       ONLINE  2010-03-30 11:07:21        4599488977            947455 NO
         4          4 USERS                          ONLINE  2010-03-30 10:08:04        4599488977            947455 NO
         6          5 XXXX                           ONLINE  2020-05-29 09:45:48        4599488977            947455 NO

并且尝试online datafile 2

SQL> recover datafile 2;
ORA-00283: 恢复会话因错误而取消
ORA-01122: 数据库文件 2 验证失败
ORA-01110: 数据文件 2: 'D:\XXXX\DATAS\ORADATA\XXXX\SYSAUX01.DBF'
ORA-01207: 文件比控制文件更新 - 旧的控制文件

由于ctl中的关于datafile2 的信息没有更新，因此数据文件的信息比ctl中的新,无法正常recover,需要重建ctl

SQL> startup nomount;
ORACLE 例程已经启动。

Total System Global Area 1.3195E+10 bytes
Fixed Size                  2188168 bytes
Variable Size            1.0301E+10 bytes
Database Buffers         2885681152 bytes
Redo Buffers                5738496 bytes
SQL> CREATE CONTROLFILE REUSE DATABASE "XXXX" NORESETLOGS  NOARCHIVELOG
  2      MAXLOGFILES 16
  3      MAXLOGMEMBERS 3
  4      MAXDATAFILES 100
  5      MAXINSTANCES 8
  6      MAXLOGHISTORY 2336
  7  LOGFILE
  8    GROUP 1 'D:\XXXX\DATAS\ORADATA\XXXX\REDO01.LOG'  SIZE 50M BLOCKSIZE 512,
  9    GROUP 2 'D:\XXXX\DATAS\ORADATA\XXXX\REDO02.LOG'  SIZE 50M BLOCKSIZE 512,
 10    GROUP 3 'D:\XXXX\DATAS\ORADATA\XXXX\REDO03.LOG'  SIZE 50M BLOCKSIZE 512
 11  DATAFILE
 12    'D:\XXXX\DATAS\ORADATA\XXXX\SYSTEM01.DBF',
 13    'D:\XXXX\DATAS\ORADATA\XXXX\SYSAUX01.DBF',
 14    'D:\XXXX\DATAS\ORADATA\XXXX\UNDOTBS01.DBF',
 15    'D:\XXXX\DATAS\ORADATA\XXXX\USERS01.DBF',
 16    'D:\XXXX\DATAS\ZYSPACE\XXXX.DBF',
 17    'E:\XXXX\DATAS\BACKUP\XXXXX.DBF'
 18  CHARACTER SET ZHS16GBK
 19  ;
CREATE CONTROLFILE REUSE DATABASE "XXXX" NORESETLOGS  NOARCHIVELOG
*
第 1 行出现错误:
ORA-01503: CREATE CONTROLFILE ??
ORA-01229: ???? 2 ??????
ORA-01110: ???? 2: 'D:\XXXX\DATAS\ORADATA\XXXX\SYSAUX01.DBF'

由于redo中信息也不对,重建需要使用resetlogs方式进行

SQL> CREATE CONTROLFILE REUSE DATABASE "XXXX" RESETLOGS  NOARCHIVELOG
  2      MAXLOGFILES 16
  3      MAXLOGMEMBERS 3
  4      MAXDATAFILES 100
  5      MAXINSTANCES 8
  6      MAXLOGHISTORY 2336
  7  LOGFILE
  8    GROUP 1 'D:\XXXX\DATAS\ORADATA\XXXX\REDO01.LOG'  SIZE 50M BLOCKSIZE 512,
  9    GROUP 2 'D:\XXXX\DATAS\ORADATA\XXXX\REDO02.LOG'  SIZE 50M BLOCKSIZE 512,
 10    GROUP 3 'D:\XXXX\DATAS\ORADATA\XXXX\REDO03.LOG'  SIZE 50M BLOCKSIZE 512
 11  DATAFILE
 12    'D:\XXXX\DATAS\ORADATA\XXXX\SYSTEM01.DBF',
 13    'D:\XXXX\DATAS\ORADATA\XXXX\SYSAUX01.DBF',
 14    'D:\XXXX\DATAS\ORADATA\XXXX\UNDOTBS01.DBF',
 15    'D:\XXXX\DATAS\ORADATA\XXXX\USERS01.DBF',
 16    'D:\XXXX\DATAS\ZYSPACE\XXXX.DBF',
 17    'E:\XXXX\DATAS\BACKUP\XXXXX.DBF'
 18  CHARACTER SET ZHS16GBK
 19  ;

控制文件已创建。

后续处理

SQL> alter database datafile 6 offline drop;

数据库已更改。

SQL> recover database using backup controlfile;
ORA-00279: ?? 4599488977 (? 08/29/2022 20:59:25 ??) ???? 1 ????
ORA-00289: ??: D:\XXXX\DATAS\FLASH_RECOVERY_AREA\XXXX\ARCHIVELOG\2022_08_29\O1_MF_1_55279_%U_.ARC
ORA-00280: ?? 4599488977 (???? 1) ??? #55279 ?


指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
D:\XXXX\DATAS\ORADATA\XXXX\REDO01.LOG
已应用的日志。
完成介质恢复。
SQL> alter database open resetlogs;

数据库已更改。

SQL> ALTER TABLESPACE TEMP ADD TEMPFILE 'D:\XXXX\DATAS\ORADATA\XXXX\TEMP01.DBF' REUSE;

表空间已更改。

数据导出成功

Oracle Recovery Tools修复空闲坏块

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle Recovery Tools修复空闲坏块

我们经常遇到由于某种原因,表上面有坏块,通过event或者dbms包跳过坏块,然后重建该表之后,但是dbv和rman检查坏块依旧存在(而且导致常规情况下rman无法正常备份),最近在Oracle Recovery Tools工具中增加的这种异常数据块修复功能,通过试验重现类似故障:
创建表并进行破坏

C:\Users\XFF>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on 星期一 8月 8 14:00:34 2022

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> create tablespace ts_test datafile 'e:/oradata/test/ts_test.dbf' size 128M autoextend on;

表空间已创建。

SQL> create table t_xifenfei tablespace ts_test
  2  as
  3  select * from dba_objects;

表已创建。

SQL> alter system checkpoint;

系统已更改。

SQL> set pages 100
SQL> select file_id,block_id,blocks from dba_extents where segment_name='T_XIFENFEI';

   FILE_ID   BLOCK_ID     BLOCKS
---------- ---------- ----------
         5        128          8
         5        136          8
         5        144          8
         5        152          8
         5        160          8
         5        168          8
         5        176          8
         5        184          8
         5        192          8
         5        200          8
         5        208          8
         5        216          8
         5        224          8
         5        232          8
         5        240          8
         5        248          8
         5        256        128
         5        384        128
         5        512        128
         5        640        128
         5        768        128
         5        896        128
         5       1024        128
         5       1152        128
         5       1280        128

已选择25行。


SQL> SELECT COUNT(1) FROM T_XIFENFEI;

  COUNT(1)
----------
     86048

SQL> shutdown immediate;
数据库已经关闭。
已经卸载数据库。
ORACLE 例程已经关闭。


C:\Windows\system32>dd of=e:/oradata/test/ts_test.dbf  if=/dev/zero bs=1k seek=1419 conv=notrunc count=1
rawwrite dd for windows version 0.6beta3.
Written by John Newbigin <jn@it.swin.edu.au>
This program is covered by terms of the GPL Version 2.

notrunc
1+0 records in
1+0 records out


SQL> startup
ORACLE 例程已经启动。

Total System Global Area 3206836224 bytes
Fixed Size                  2285512 bytes
Variable Size             754974776 bytes
Database Buffers         2432696320 bytes
Redo Buffers               16879616 bytes
数据库装载完毕。
数据库已经打开。
SQL> select count(1) from t_xifenfei;
select count(1) from t_xifenfei
                     *
第 1 行出现错误:
ORA-01578: ORACLE 数据块损坏 (文件号 5, 块号 177)
ORA-01110: 数据文件 5: 'E:\ORADATA\TEST\TS_TEST.DBF'

跳过坏块重建该表

SQL> alter session set events '10231 trace name context forever, level 10';

会话已更改。

SQL> create table t_xifenfei_bak tablespace ts_test
  2  as select * from t_xifenfei;

表已创建。

SQL> select count(1) from t_xifenfei_bak;

  COUNT(1)
----------
     85968

SQL> drop table t_xifenfei purge;

表已删除。

SQL> rename t_xifenfei_bak to t_xifenfei;

表已重命名。

SQL> select count(1) from t_xifenfei;

  COUNT(1)
----------
     85968

检查坏块情况
通过rman和dbv检查,均表明file 5 block 177为坏块

C:\Users\XFF>dbv file=E:\ORADATA\TEST\TS_TEST.DBF

DBVERIFY: Release 11.2.0.4.0 - Production on 星期一 8月 8 17:25:57 2022

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = E:\ORADATA\TEST\TS_TEST.DBF
页 177 标记为损坏
Corrupt block relative dba: 0x014000b1 (file 5, block 177)
Bad check value found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x014000b1
 last change scn: 0x0000.000ebc27 seq: 0x2 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0xbc270602
 check value in block header: 0x5b2a
 computed block checksum: 0xbb32



DBVERIFY - 验证完成

检查的页总数: 16384
处理的页总数 (数据): 2456
失败的页总数 (数据): 0
处理的页总数 (索引): 0
失败的页总数 (索引): 0
处理的页总数 (其他): 155
处理的总页数 (段)  : 0
失败的总页数 (段)  : 0
空的页总数: 13772
标记为损坏的总页数: 1
流入的页总数: 0
加密的总页数        : 0
最高块 SCN            : 967616 (0.967616)

RMAN> backup validate check logical datafile 5;

启动 backup 于 08-8月 -22
使用目标数据库控制文件替代恢复目录
分配的通道: ORA_DISK_1
通道 ORA_DISK_1: SID=118 设备类型=DISK
通道 ORA_DISK_1: 正在启动全部数据文件备份集
通道 ORA_DISK_1: 正在指定备份集内的数据文件
输入数据文件: 文件号=00005 名称=E:\ORADATA\TEST\TS_TEST.DBF
通道 ORA_DISK_1: 备份集已完成, 经过时间:00:00:01
数据文件列表
=================
文件状态 标记为损坏 空块 已检查的块 高 SCN
---- ------ -------------- ------------ --------------- ----------
5    FAILED 0              13744        16384           967621
  文件名: E:\ORADATA\TEST\TS_TEST.DBF
  块类型 失败的块 已处理的块
  ---------- -------------- ----------------
  数据       1              2457
  索引      0              0
  其他      0              183

验证找到一个或多个损坏的块
有关详细信息, 请参阅跟踪文件 C:\APP\XFF\diag\rdbms\test\test\trace\test_ora_22284.trc
完成 backup 于 08-8月 -22

SQL> select * from v$database_block_corruption ;

     FILE#     BLOCK#     BLOCKS CORRUPTION_CHANGE# CORRUPTIO
---------- ---------- ---------- ------------------ ---------
         5        177          1                  0 CHECKSUM

查询坏块所属对象
没有查询到该坏块所属对象,证明该block为游离块[不属于任何数据对象,是空闲块,但是损坏]

SQL> SELECT OWNER, SEGMENT_NAME, SEGMENT_TYPE, TABLESPACE_NAME, A.PARTITION_NAME
  2    FROM DBA_EXTENTS A
  3   WHERE FILE_ID = &FILE_ID
  4     AND &BLOCK_ID BETWEEN BLOCK_ID AND BLOCK_ID + BLOCKS - 1;
输入 file_id 的值:  5
原值    3:  WHERE FILE_ID = &FILE_ID
新值    3:  WHERE FILE_ID = 5
输入 block_id 的值:  177
原值    4:    AND &BLOCK_ID BETWEEN BLOCK_ID AND BLOCK_ID + BLOCKS - 1
新值    4:    AND 177 BETWEEN BLOCK_ID AND BLOCK_ID + BLOCKS - 1

未选定行

通过Oracle Recovery Tools工具进行修复

再次检查坏块
通过工具修复之后,dbv和rman检查均正常

C:\Users\XFF>rman target /

恢复管理器: Release 11.2.0.4.0 - Production on 星期一 8月 8 17:59:26 2022

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

已连接到目标数据库: TEST (DBID=2410248200)

RMAN> backup validate check logical datafile 5;

启动 backup 于 08-8月 -22
使用目标数据库控制文件替代恢复目录
分配的通道: ORA_DISK_1
通道 ORA_DISK_1: SID=54 设备类型=DISK
通道 ORA_DISK_1: 正在启动全部数据文件备份集
通道 ORA_DISK_1: 正在指定备份集内的数据文件
输入数据文件: 文件号=00005 名称=E:\ORADATA\TEST\TS_TEST.DBF
通道 ORA_DISK_1: 备份集已完成, 经过时间:00:00:01
数据文件列表
=================
文件状态 标记为损坏 空块 已检查的块 高 SCN
---- ------ -------------- ------------ --------------- ----------
5    OK     0              13745        16384           967621
  文件名: E:\ORADATA\TEST\TS_TEST.DBF
  块类型 失败的块 已处理的块
  ---------- -------------- ----------------
  数据       0              2456
  索引      0              0
  其他      0              183

完成 backup 于 08-8月 -22


C:\Users\XFF>dbv file=E:\ORADATA\TEST\TS_TEST.DBF

DBVERIFY: Release 11.2.0.4.0 - Production on 星期一 8月 8 17:56:45 2022

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = E:\ORADATA\TEST\TS_TEST.DBF


DBVERIFY - 验证完成

检查的页总数: 16384
处理的页总数 (数据): 2456
失败的页总数 (数据): 0
处理的页总数 (索引): 0
失败的页总数 (索引): 0
处理的页总数 (其他): 183
处理的总页数 (段)  : 0
失败的总页数 (段)  : 0
空的页总数: 13745
标记为损坏的总页数: 0
流入的页总数: 0
加密的总页数        : 0
最高块 SCN            : 967621 (0.967621)


SQL> select * from v$database_block_corruption ;

未选定行

Oracle Recovery Tools 解决ORA-600 3020故障

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle Recovery Tools 解决ORA-600 3020故障

尝试recover datafile,部分文件报ORA-600 3020，其他文件recover成功

ALTER DATABASE RECOVER  datafile 1  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 3 Seq 24972 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_72232.trc  (incident=749532):
ORA-00600: 内部错误代码, 参数: [3020], [1], [272255], [4466559], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 1, block# 272255, file offset is 2230312960 bytes)
ORA-10564: tablespace SYSTEM
ORA-01110: 数据文件 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 383
Media Recovery failed with error 600
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 1  ...
Tue Aug 02 10:28:24 2022
Trace dumping is performing id=[cdmp_20220802102824]
Tue Aug 02 10:28:31 2022
ALTER DATABASE RECOVER  datafile 2  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 3 Seq 24972 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_72232.trc  (incident=749533):
ORA-00600: 内部错误代码, 参数: [3020], [2], [92323], [8480931], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 2, block# 92323, file offset is 756310016 bytes)
ORA-10564: tablespace SYSAUX
ORA-01110: 数据文件 2: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSAUX01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 12330
Media Recovery failed with error 600
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 2  ...

利用Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)检查文件头相关信息，发现recover 失败的两个文件异常

通过Oracle Recovery Tools工具进行修复

数据库recover 成功，并顺利open

Tue Aug 02 10:56:13 2022
ALTER DATABASE RECOVER  datafile 1  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 3 Seq 24972 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Completed: ALTER DATABASE RECOVER  datafile 1  
ALTER DATABASE RECOVER  datafile 2  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 3 Seq 24972 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Completed: ALTER DATABASE RECOVER  datafile 2  
Tue Aug 02 10:56:34 2022
alter database open 
Beginning crash recovery of 1 threads
 parallel recovery started with 7 processes
Started redo scan
Completed redo scan
 read 8504 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 24972, block 2, scn 177712270
Recovery of Online Redo Log: Thread 1 Group 3 Seq 24972 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 24972, block 17011, scn 177734679
 0 data blocks read, 0 data blocks written, 8504 redo k-bytes read
Tue Aug 02 10:56:35 2022
Thread 1 advanced to log sequence 24973 (thread open)
Thread 1 opened at log sequence 24973
  Current log# 1 seq# 24973 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Aug 02 10:56:35 2022
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
**********************************************************
WARNING: Files may exists in db_recovery_file_dest
that are not known to the database. Use the RMAN command
CATALOG RECOVERY AREA to re-catalog any such files.
If files cannot be cataloged, then manually delete them
using OS command.
One of the following events caused this:
1. A backup controlfile was restored.
2. A standby controlfile was restored.
3. The controlfile was re-created.
4. db_recovery_file_dest had previously been enabled and
   then disabled.
**********************************************************
replication_dependency_tracking turned off (no async multimaster replication found)
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Completed: alter database open

增加tempfile,导出数据该库恢复完成