Database SOS – 第81页 – Professional Database Recovery Support@+8617813235971

ORA-600 999 异常恢复

有网友数据库启动报ORA-600 999错误,无法正常open,让我们介入分析,帮忙恢复其中部分数据
数据库启动报ORA-600 999

Sun Jul 31 23:09:36 2016
SMON: enabling cache recovery
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_smon_3356.trc  (incident=179779):
ORA-00600: internal error code, arguments: [999], [0x7FFAE748013], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_179779\orcl_smon_3356_i179779.trc
No Resource Manager plan active
Starting background process QMNC
Sun Jul 31 23:09:37 2016
QMNC started with pid=20, OS id=5068
ORACLE Instance orcl (pid = 13) - Error 600 encountered while recovering transaction (7, 1).
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_smon_3356.trc:
ORA-00600: internal error code, arguments: [999], [0x7FFAE748013], [], [], [], [], [], [], [], [], [], []
Completed: alter database open
Sun Jul 31 23:09:38 2016
db_recovery_file_dest_size of 8680 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Trace dumping is performing id=[cdmp_20160731230939]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_smon_3356.trc  (incident=179785):
ORA-00600: internal error code, arguments: [999], [0x7FFAE748013], [], [], [], [], [], [], [], [], [], []
ORACLE Instance orcl (pid = 13) - Error 600 encountered while recovering transaction (7, 1).
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_smon_3356.trc:
ORA-00600: internal error code, arguments: [999], [0x7FFAE748013], [], [], [], [], [], [], [], [], [], []
Sun Jul 31 23:09:41 2016
Starting background process CJQ0
Sun Jul 31 23:09:41 2016
CJQ0 started with pid=25, OS id=2572
Process debug not enabled via parameter _debug_enable
Trace dumping is performing id=[cdmp_20160731230942]
PMON (ospid: 3948): terminating the instance due to error 474
Sun Jul 31 23:09:48 2016
opiodr aborting process unknown ospid (2592) as a result of ORA-1092
Sun Jul 31 23:09:48 2016
ORA-1092 : opitsk aborting process
Sun Jul 31 23:09:52 2016
Instance terminated by PMON, pid = 3948

设置_offline_rollback_segments数据库启动正常

Sun Jul 31 23:18:13 2016
ALTER DATABASE OPEN
Thread 1 opened at log sequence 16
  Current log# 1 seq# 16 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Successful open of redo thread 1
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 5.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_smon_4372.trc  (incident=182188):
ORA-00600: internal error code, arguments: [kdBlkCheckError], [3], [224], [38508], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_182188\orcl_smon_4372_i182188.trc
Doing block recovery for file 3 block 224
Resuming block recovery (PMON) for file 3 block 224
Block recovery from logseq 16, block 2945 to scn 15431544
Recovery of Online Redo Log: Thread 1 Group 1 Seq 16 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Trace dumping is performing id=[cdmp_20160731231815]
Block recovery stopped at EOT rba 16.2952.16
Block recovery completed at rba 16.2952.16, scn 0.15431543
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_smon_4372.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [3], [224], [38508], [], [], [], [], [], [], [], []
Sun Jul 31 23:18:19 2016
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_smon_4372.trc  (incident=182189):
ORA-00600: internal error code, arguments: [kdBlkCheckError], [3], [224], [38508], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_182189\orcl_smon_4372_i182189.trc
Starting background process QMNC
Sun Jul 31 23:18:19 2016
QMNC started with pid=20, OS id=4920
Doing block recovery for file 3 block 224
Resuming block recovery (PMON) for file 3 block 224
Block recovery from logseq 16, block 2945 to scn 15431544
Recovery of Online Redo Log: Thread 1 Group 1 Seq 16 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Block recovery completed at rba 16.2952.16, scn 0.15431545
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_smon_4372.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [3], [224], [38508], [], [], [], [], [], [], [], []
Starting background process SMCO
Sun Jul 31 23:18:19 2016
SMCO started with pid=21, OS id=3176
Sun Jul 31 23:18:20 2016
Trace dumping is performing id=[cdmp_20160731231820]
Completed: ALTER DATABASE OPEN

尝试删除异常回滚段

Sun Jul 31 23:15:07 2016
drop rollback segment "_SYSSMU7_1101470402$"
Sun Jul 31 23:15:07 2016
Corrupt Block Found
         TSN = 2, TSNAME = UNDOTBS1
         RFN = 3, BLK = 224, RDBA = 12583136
         OBJN = -1, OBJD = -1, OBJECT = , SUBOBJECT =
         SEGMENT OWNER = , SEGMENT TYPE =
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_5300.trc  (incident=181035):
ORA-00600: 内部错误代码, 参数: [kdBlkCheckError], [3], [224], [38508], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_181035\orcl_ora_5300_i181035.trc
Doing block recovery for file 3 block 224
Resuming block recovery (PMON) for file 3 block 224
Block recovery from logseq 14, block 8682 to scn 15397854
Recovery of Online Redo Log: Thread 1 Group 2 Seq 14 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Block recovery completed at rba 14.8688.16, scn 0.15397855
ORA-607 signalled during: drop rollback segment "_SYSSMU7_1101470402$"...
Corrupt Block Found
         TSN = 2, TSNAME = UNDOTBS1
         RFN = 3, BLK = 224, RDBA = 12583136
         OBJN = -1, OBJD = -1, OBJECT = , SUBOBJECT =
         SEGMENT OWNER = , SEGMENT TYPE =

从这里看,我们可以确定file 3 block 224异常,导致删除回滚段异常.和mos官方给出来的案例类似,由于undo坏块导致数据库报ORA-600 999错误

mos中ORA-600 999报错信息
官方的益处ORA-600[999]报错,也是由于undo坏块引起和本文的报错基本上一致
ORA-600-999

因为只要部分数据,直接屏蔽回滚段,数据库不再crash,导出需要对象即可

_ALLOW_RESETLOGS_CORRUPTION

我相信_ALLOW_RESETLOGS_CORRUPTION 这个参数一定很多人都熟悉,是redo异常恢复的杀手锏之一,以下文章是来自官方的解释

DB_Parameter _ALLOW_RESETLOGS_CORRUPTION
========================================
This documentation has been prepared avoiding the mention of the complex
structures from the code and to simply give an insight to the 'damage it could
cause'.  The usage of this parameter leads to an in-consistent Database with no
other alternative but to rebuild the complete Database.  This parameter could
be used when we realize that there are no stardard options available and are
convinced that the customer understands the implications of using the Oracle's
secret parameter.  The factors to be considered are ;--
1. Customer does not have a good backup.
2. A lot of time and money has been invested after the last good backup and
   there is no possibility for reproduction of the lost data.
3. The customer has to be ready to export the full database and import it
   back after creating a new one.
4. There is no 100% guarantee that by using this parameter the database would
   come up.
5. Oracle does not support the database after using this parameter for
   recovery.
6. ALL OPTIONS including the ones mentioned in the action part of the error
   message have been tried.
By setting _ALLOW_RESETLOGS_CORRUPTION=TRUE, certain consistency checks are
SKIPPED during database open stage.  This basically means it does not check
the datafile headers as to what the status was before the shutdown and how it
was shutdown.  The following cases mention few of the checks that were skipped.
Case-I
------
Verification that the datafile present has not been restored from a BACKUP
taken before the database was opened successfully by using RESETLOGS.
ORA-01190: control file or data file %s is from before the last RESETLOGS
    Cause: Attempting to use a data file when the log reset information in
           the file does not match the control file.  Either the data file or
           the control file is a backup that was made before the most recent
           ALTER DATABASE OPEN RESETLOGS.
   Action: Restore file from a more recent backup.
Case-II
-------
Verification that the status bit of the datafile is not in a FUZZY state.
The datafile could be in this state due to the database going down when the
 - Datafile was on-line and open
 - Datafile was not closed cleanly (maybe due to OS).
ORA-01194: file %s needs more recovery to be consistent
    Cause: An incomplete recover session was started, but an insufficient
           number of logs were applied to make the file consistent.  The
           reported file was not closed cleanly when it was last opened by
           the database.  It must be recovered to a time when it was not
           being updated.  The most likely cause of this error is forgetting
           to restore the file from a backup before doing incomplete
           recovery.
   Action: Either apply more logs until the file is consistent or restore the
           file from an older backup and repeat recovery.
Case-III
--------
Verification that the COMPLETE recover strategies have been applied for
recovering the datafile and not any of the INCOMPLETE recovery options.
Basically because the complete recovery is one in which we even apply the
ON-LINE redo log files and open the DB without reseting the logs.
ORA-01113: file '%s' needs media recovery starting at log sequence # %s
    Cause: An attempt was made to open a database file that is in need of
           media recovery.
   Action: First apply media recovery to the file.
Case-IV
-------
Verification that the datafile has been recovered through an END BACKUP if the
control file indicates that it was in backup mode.
This is useful when the DB has crashed while in hot backup mode and we lost
all log files in DB version's less than V7.2.
ORA-01195: on-line backup of file %s needs more recovery to be consistent"
    Cause: An incomplete recovery session was started, but an insufficient
           number of logs were applied to make the file consistent.  The
           reported file is an on-line backup which must be recovered to the
           time the backup ended.
   Action: Either apply more logs until the file is consistent or resotre
          the database files from an older backup and repeat recovery.
In version 7.2, we could simply issue the ALTER DATABASE DATAFILE xxxx END
BACKUP statement and proceed with the recovery.  But again to issue this
statement, we need to have the ON-LINE redo logs or else we still are forced to
use this parameter.
Case-V
------
Verification that the data file status is not still in (0x10) MEDIA recovery
FUZZY.
When recovery is started, a flag is set in the datafile header status flag to
indicate that the file is presently in media recovery.  This is reset when
recovery is completed and at times when it has not been reset we are forced to
use this paramter.
ORA-01196: file %s is inconsistent due to a failed media recovery session
    Cause: The file was being recovered but the recovery did not terminate
           normally.  This left the file in an inconsistent state.  No more
           recovery was successfully completed on this file.
   Action: Either apply more logs until the file is consistent or restore the
           backup again and repeat recovery.
Case-VI
-------
Verification that the datafile has been restored form a proper backup to
correspond with the log files.  This situation could happen when we have
decided that the data file is invalid since its SCN is ahead of the last
applied logs SCN but it has not failed on one of the ABOVE CHECKS.
ORA-01152: file '%s' was not restored from a sufficientluy old backup"
    Cause: A manual recovery session was started, but an insufficient number
           of logs were applied to make the database consistent.  This file is
           still in the future of the last log applied.  Note that this
           mistake can not always be caught.
   Action: Either apply more logs until the database is consistent or
           restore the database file from an older backup and repeat
           recovery.

使用_ALLOW_RESETLOGS_CORRUPTION 参数需谨慎,因为该参数可能导致数据库逻辑不一致,甚至可能把本来很简单的一个恢复弄的非常复杂甚至不可恢复的后果,建议在oracle support支持下使用.另外使用该参数resetlogs库之后,强烈建议通过逻辑方式重建库

数据库恢复的敏感性—重建控制文件使用不合适数据文件

有客户数据库由于某种原因无法open,请求我们技术支持.通过检查alert日志发现数据库是由于ORA-600 kccpb_sanity_check_2错误.并且他们已经重建控制文件,通过我们的Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)
datafile 6 异常
wrong-file1

wrong-file-redo

通过这里我们发现datafile 6 数据文件头是干净的,而且对应的redo seq为5270,而其他文件头都是fuzzy,而且对应的redo为295902613.这里怀疑datafile 6 可能是错误的.当然对于这样scn相距比较大的情况,我们可以通过隐含参数,修改scn等方法强制让该库起来(或者该文件online),但是从现在看到的情况,文件很可能异常,这样强制恢复,可能没有实际意义.
分析alert日志
alert-log

这个里面可以发现是先删除了sde表空间,然后创建了同一个表空间,只是数据文件路径不一样了.而且正好在seq为5270的地方操作的.现在出现datafile 6异常的原因已经清楚,就是创建数据控制文件的时候,使用了错误的数据文件导致.

完美恢复数据库

D:\>sqlplus / as sysdba
SQL*Plus: Release 10.2.0.1.0 - Production on 星期六 7月 30 16:31:01 2016
Copyright (c) 1982, 2005, Oracle.  All rights reserved.
连接到:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
SQL> shutdown immediate;
ORA-01109: 数据库未打开
已经卸载数据库。
ORACLE 例程已经关闭。
SQL> STARTUP NOMOUNT
ORACLE 例程已经启动。
Total System Global Area 5133828096 bytes
Fixed Size                  2011360 bytes
Variable Size            2382368544 bytes
Database Buffers         2734686208 bytes
Redo Buffers               14761984 bytes
SQL> CREATE CONTROLFILE REUSE DATABASE "LANDDB" NORESETLOGS  ARCHIVELOG
  2      MAXLOGFILES 16
  3      MAXLOGMEMBERS 3
  4      MAXDATAFILES 100
  5      MAXINSTANCES 8
  6      MAXLOGHISTORY 292
  7  LOGFILE
  8    GROUP 1 'F:\ORADATA\LANDDB\ONLINELOG\O1_MF_1_4JCM05KL_.LOG'  SIZE 50M,
  9    GROUP 2 'F:\ORADATA\LANDDB\ONLINELOG\O1_MF_2_4JCM064D_.LOG'  SIZE 50M,
 10    GROUP 3 'F:\ORADATA\LANDDB\ONLINELOG\O1_MF_3_4JCM06PG_.LOG'  SIZE 50M
 11  DATAFILE
 12    'F:\ORADATA\LANDDB\DATAFILE\O1_MF_SYSTEM_4JCLYY6T_.DBF',
 13    'F:\ORADATA\LANDDB\DATAFILE\O1_MF_UNDOTBS1_4JCLYY8S_.DBF',
 14    'F:\ORADATA\LANDDB\DATAFILE\O1_MF_SYSAUX_4JCLYY7B_.DBF',
 15    'F:\ORADATA\LANDDB\DATAFILE\O1_MF_USERS_4JCLYY98_.DBF',
 16    'F:\ORADATA\LANDDB\DATAFILE\FUJIAN',
 17    'D:\data\sde.dbf'
 18  CHARACTER SET ZHS16GBK
 19  ;
控制文件已创建。
SQL> recover database;
完成介质恢复。
SQL> alter database open;
数据库已更改。
SQL>

这个库比较幸运,客户发现异常之后,里面停止了有风险性的操作(比如使用_allow_resetlogs_corruption参数,resetlogs库等),使得数据完美恢复0丢失.如果条件允许最好使用老的控制文件来重建新控制文件,而不要通过人工去系统中找数据文件来实现恢复,这样很可能有遗落或者使用错误的数据文件

kfed找出来asm 磁盘组中数据文件别名对应的文件号—amdu恢复

前段时间有多个朋友问我,在amdu中,如果数据文件命名不是omf的方式,该如何找出来数据文件的asm file_number,从而实现通过amdu对不能mount的磁盘组中的数据文件进行恢复,这里通过测试给出来处理方法.根据我们对asm的理解,asm file_number 6为asm file的别名文件记录所在地,我们通过分析kfed这些au中的记录即可获得相关数据文件的别名对应的asm文件号

模拟各种别名

D:\app\product\10.2.0\db_1\bin>sqlplus / as sysdba
SQL*Plus: Release 10.2.0.3.0 - Production on 星期三 7月 27 22:48:48 2016
Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.
连接到:
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production
With the Partitioning, OLAP and Data Mining options
SQL> select name from v$datafile;
NAME
--------------------------------------------------------------------------------
+DATA/ora10g/datafile/system.256.914797317
+DATA/ora10g/datafile/undotbs1.258.914797317
+DATA/ora10g/datafile/sysaux.257.914797317
+DATA/ora10g/datafile/users.259.914797317
SQL> create tablespace xifenfei
  2  datafile '+data/xifenfei01.dbf' size 10M;
表空间已创建。
SQL> alter tablespace xifenfei add
  2  datafile '+data/ora10g/datafile/xifenfei02.dbf' size 10m;
表空间已更改。
SQL> alter tablespace xifenfei add
  2  datafile '+data/ora10g/xifenfei03.dbf' size 10m;
表空间已更改。
SQL> select name from v$datafile;
NAME
--------------------------------------------------------------------------------
+DATA/ora10g/datafile/system.256.914797317
+DATA/ora10g/datafile/undotbs1.258.914797317
+DATA/ora10g/datafile/sysaux.257.914797317
+DATA/ora10g/datafile/users.259.914797317
+DATA/xifenfei01.dbf
+DATA/ora10g/datafile/xifenfei02.dbf
+DATA/ora10g/xifenfei03.dbf
已选择7行。

分析磁盘组和别名信息

SQL> select name from v$asm_disk;
NAME
------------------------------
DATA_0000
DATA_0001
SQL> select path from v$asm_disk;
PATH
-----------------------------------------
H:\ASMDISK\ASMDISK1.DD
H:\ASMDISK\ASMDISK2.DD
SQL> SELECT NAME,FILE_NUMBER FROM V$ASM_ALIAS where file_number<>4294967295;
NAME                           FILE_NUMBER
------------------------------ -----------
SYSTEM.256.914797317                   256
SYSAUX.257.914797317                   257
UNDOTBS1.258.914797317                 258
USERS.259.914797317                    259
XIFENFEI.266.918341361                 266
XIFENFEI.267.918341389                 267
xifenfei02.dbf                         267
XIFENFEI.268.918341409                 268
Current.260.914797381                  260
group_1.261.914797385                  261
group_2.262.914797385                  262
group_3.263.914797387                  263
TEMP.264.914797393                     264
spfile.265.914797421                   265
spfileora10g.ora                       265
xifenfei03.dbf                         268
xifenfei01.dbf                         266
已选择17行。
SQL> SELECT NAME,FILE_NUMBER FROM V$ASM_ALIAS;
NAME                           FILE_NUMBER
------------------------------ -----------
ORA10G                          4294967295
DATAFILE                        4294967295
SYSTEM.256.914797317                   256
SYSAUX.257.914797317                   257
UNDOTBS1.258.914797317                 258
USERS.259.914797317                    259
XIFENFEI.266.918341361                 266
XIFENFEI.267.918341389                 267
xifenfei02.dbf                         267
XIFENFEI.268.918341409                 268
CONTROLFILE                     4294967295
Current.260.914797381                  260
ONLINELOG                       4294967295
group_1.261.914797385                  261
group_2.262.914797385                  262
group_3.263.914797387                  263
TEMPFILE                        4294967295
TEMP.264.914797393                     264
PARAMETERFILE                   4294967295
spfile.265.914797421                   265
spfileora10g.ora                       265
xifenfei03.dbf                         268
xifenfei01.dbf                         266
已选择23行。

从sql查询,我们可以确定xifenfei0n.dbf对应的文件号分别为:xifenfei01.dbf==>266,xifenfei02.dbf==>267,xifenfei03.dbf==>268

通过kfed file 6所在位置

www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD |grep f1b1
kfdhdb.f1b1locn:                      2 ; 0x0d4: 0x00000002
kfdhdb.f1b1fcn.base:                  0 ; 0x100: 0x00000000
kfdhdb.f1b1fcn.wrap:                  0 ; 0x104: 0x00000000
www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=2 blkn=6|grep kfffde|more
kfffde[0].xptr.au:                   26 ; 0x4a0: 0x0000001a
kfffde[0].xptr.disk:                  0 ; 0x4a4: 0x0000
kfffde[0].xptr.flags:                 0 ; 0x4a6: L=0 E=0 D=0 S=0
kfffde[0].xptr.chk:                  48 ; 0x4a7: 0x30
kfffde[1].xptr.au:           4294967295 ; 0x4a8: 0xffffffff
kfffde[1].xptr.disk:              65535 ; 0x4ac: 0xffff

从这里我们可以确定别名的au只有一个位于disk 0, au 26(0x1a)的位置
通过kfed分析别名

www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=26 |more
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                           11 ; 0x002: KFBTYP_ALIASDIR
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       6 ; 0x008: file=6
kfbh.check:                  1563703526 ; 0x00c: 0x5d3438e6
kfbh.fcn.base:                     3461 ; 0x010: 0x00000d85
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kffdnd.bnode.incarn:                  1 ; 0x000: A=1 NUMM=0x0
kffdnd.bnode.frlist.number:  4294967295 ; 0x004: 0xffffffff
kffdnd.bnode.frlist.incarn:           0 ; 0x008: A=0 NUMM=0x0
kffdnd.overfl.number:        4294967295 ; 0x00c: 0xffffffff
kffdnd.overfl.incarn:                 0 ; 0x010: A=0 NUMM=0x0
kffdnd.parent.number:                 0 ; 0x014: 0x00000000
kffdnd.parent.incarn:                 1 ; 0x018: A=1 NUMM=0x0
kffdnd.fstblk.number:                 0 ; 0x01c: 0x00000000
kffdnd.fstblk.incarn:                 1 ; 0x020: A=1 NUMM=0x0
kfade[0].entry.incarn:                1 ; 0x024: A=1 NUMM=0x0
kfade[0].entry.hash:         2080305534 ; 0x028: 0x7bfef17e
kfade[0].entry.refer.number:          1 ; 0x02c: 0x00000001
kfade[0].entry.refer.incarn:          1 ; 0x030: A=1 NUMM=0x0
kfade[0].name:                   ORA10G ; 0x034: length=6
kfade[0].fnum:               4294967295 ; 0x064: 0xffffffff
kfade[0].finc:               4294967295 ; 0x068: 0xffffffff
kfade[0].flags:                       4 ; 0x06c: U=0 S=0 S=1 U=0 F=0
kfade[0].ub1spare:                    0 ; 0x06d: 0x00
kfade[0].freeblock:                   0 ; 0x06e: 0x0000
kfade[1].entry.incarn:                1 ; 0x070: A=1 NUMM=0x0
kfade[1].entry.hash:         3085841201 ; 0x074: 0xb7ee3331
kfade[1].entry.refer.number: 4294967295 ; 0x078: 0xffffffff
kfade[1].entry.refer.incarn:          0 ; 0x07c: A=0 NUMM=0x0
kfade[1].name:           xifenfei01.dbf ; 0x080: length=14
kfade[1].fnum:                      266 ; 0x0b0: 0x0000010a
kfade[1].finc:                918341361 ; 0x0b4: 0x36bcc6f1
kfade[1].flags:                      17 ; 0x0b8: U=1 S=0 S=0 U=0 F=1
kfade[1].ub1spare:                    0 ; 0x0b9: 0x00
kfade[1].freeblock:                   0 ; 0x0ba: 0x0000
kfade[2].entry.incarn:                0 ; 0x0bc: A=0 NUMM=0x0
kfade[2].entry.hash:                  0 ; 0x0c0: 0x00000000
kfade[2].entry.refer.number:          0 ; 0x0c4: 0x00000000
kfade[2].entry.refer.incarn:          0 ; 0x0c8: A=0 NUMM=0x0
kfade[2].name:                          ; 0x0cc: length=0
kfade[2].fnum:                        0 ; 0x0fc: 0x00000000
kfade[2].finc:                        0 ; 0x100: 0x00000000
kfade[2].flags:                       0 ; 0x104: U=0 S=0 S=0 U=0 F=0
kfade[2].ub1spare:                    0 ; 0x105: 0x00
kfade[2].freeblock:                   0 ; 0x106: 0x0000
www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=26 |grep name
kfade[0].name:                   ORA10G ; 0x034: length=6
kfade[1].name:           xifenfei01.dbf ; 0x080: length=14
kfade[2].name:                          ; 0x0cc: length=0
kfade[3].name:                          ; 0x118: length=0
kfade[4].name:                          ; 0x164: length=0
www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=26 blkn=1|more
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                           11 ; 0x002: KFBTYP_ALIASDIR
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       1 ; 0x004: blk=1
kfbh.block.obj:                       6 ; 0x008: file=6
kfbh.check:                   239000469 ; 0x00c: 0x0e3edb95
kfbh.fcn.base:                     3536 ; 0x010: 0x00000dd0
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kffdnd.bnode.incarn:                  1 ; 0x000: A=1 NUMM=0x0
kffdnd.bnode.frlist.number:  4294967295 ; 0x004: 0xffffffff
kffdnd.bnode.frlist.incarn:           0 ; 0x008: A=0 NUMM=0x0
kffdnd.overfl.number:        4294967295 ; 0x00c: 0xffffffff
kffdnd.overfl.incarn:                 0 ; 0x010: A=0 NUMM=0x0
kffdnd.parent.number:                 0 ; 0x014: 0x00000000
kffdnd.parent.incarn:                 1 ; 0x018: A=1 NUMM=0x0
kffdnd.fstblk.number:                 1 ; 0x01c: 0x00000001
kffdnd.fstblk.incarn:                 1 ; 0x020: A=1 NUMM=0x0
kfade[0].entry.incarn:                1 ; 0x024: A=1 NUMM=0x0
kfade[0].entry.hash:          710518681 ; 0x028: 0x2a59a799
kfade[0].entry.refer.number:          2 ; 0x02c: 0x00000002
kfade[0].entry.refer.incarn:          1 ; 0x030: A=1 NUMM=0x0
kfade[0].name:                 DATAFILE ; 0x034: length=8
kfade[0].fnum:               4294967295 ; 0x064: 0xffffffff
kfade[0].finc:               4294967295 ; 0x068: 0xffffffff
kfade[0].flags:                       4 ; 0x06c: U=0 S=0 S=1 U=0 F=0
kfade[0].ub1spare:                    0 ; 0x06d: 0x00
kfade[0].freeblock:                   0 ; 0x06e: 0x0000
kfade[1].entry.incarn:                3 ; 0x070: A=1 NUMM=0x1
kfade[1].entry.hash:         4053320104 ; 0x074: 0xf198c1a8
kfade[1].entry.refer.number:          3 ; 0x078: 0x00000003
kfade[1].entry.refer.incarn:          3 ; 0x07c: A=1 NUMM=0x1
kfade[1].name:              CONTROLFILE ; 0x080: length=11
kfade[1].fnum:               4294967295 ; 0x0b0: 0xffffffff
kfade[1].finc:               4294967295 ; 0x0b4: 0xffffffff
kfade[1].flags:                       4 ; 0x0b8: U=0 S=0 S=1 U=0 F=0
kfade[1].ub1spare:                    0 ; 0x0b9: 0x00
kfade[1].freeblock:                   0 ; 0x0ba: 0x0000
kfade[2].entry.incarn:                1 ; 0x0bc: A=1 NUMM=0x0
kfade[2].entry.hash:         2803485489 ; 0x0c0: 0xa719cb31
kfade[2].entry.refer.number:          4 ; 0x0c4: 0x00000004
kfade[2].entry.refer.incarn:          1 ; 0x0c8: A=1 NUMM=0x0
kfade[2].name:                ONLINELOG ; 0x0cc: length=9
kfade[2].fnum:               4294967295 ; 0x0fc: 0xffffffff
kfade[2].finc:               4294967295 ; 0x100: 0xffffffff
kfade[2].flags:                       4 ; 0x104: U=0 S=0 S=1 U=0 F=0
kfade[2].ub1spare:                    0 ; 0x105: 0x00
kfade[2].freeblock:                   0 ; 0x106: 0x0000
kfade[3].entry.incarn:                1 ; 0x108: A=1 NUMM=0x0
kfade[3].entry.hash:         2905271101 ; 0x10c: 0xad2aeb3d
kfade[3].entry.refer.number:          5 ; 0x110: 0x00000005
kfade[3].entry.refer.incarn:          1 ; 0x114: A=1 NUMM=0x0
kfade[3].name:                 TEMPFILE ; 0x118: length=8
kfade[3].fnum:               4294967295 ; 0x148: 0xffffffff
kfade[3].finc:               4294967295 ; 0x14c: 0xffffffff
kfade[3].flags:                       4 ; 0x150: U=0 S=0 S=1 U=0 F=0
kfade[3].ub1spare:                    0 ; 0x151: 0x00
kfade[3].freeblock:                   0 ; 0x152: 0x0000
kfade[4].entry.incarn:                1 ; 0x154: A=1 NUMM=0x0
kfade[4].entry.hash:         3261836913 ; 0x158: 0xc26bae71
kfade[4].entry.refer.number:          6 ; 0x15c: 0x00000006
kfade[4].entry.refer.incarn:          1 ; 0x160: A=1 NUMM=0x0
kfade[4].name:            PARAMETERFILE ; 0x164: length=13
kfade[4].fnum:               4294967295 ; 0x194: 0xffffffff
kfade[4].finc:               4294967295 ; 0x198: 0xffffffff
kfade[4].flags:                       4 ; 0x19c: U=0 S=0 S=1 U=0 F=0
kfade[4].ub1spare:                    0 ; 0x19d: 0x00
kfade[4].freeblock:                   0 ; 0x19e: 0x0000
kfade[5].entry.incarn:                1 ; 0x1a0: A=1 NUMM=0x0
kfade[5].entry.hash:         3373604202 ; 0x1a4: 0xc9151d6a
kfade[5].entry.refer.number: 4294967295 ; 0x1a8: 0xffffffff
kfade[5].entry.refer.incarn:          0 ; 0x1ac: A=0 NUMM=0x0
kfade[5].name:         spfileora10g.ora ; 0x1b0: length=16
kfade[5].fnum:                      265 ; 0x1e0: 0x00000109
kfade[5].finc:                914797421 ; 0x1e4: 0x3686b36d
kfade[5].flags:                      17 ; 0x1e8: U=1 S=0 S=0 U=0 F=1
kfade[5].ub1spare:                    0 ; 0x1e9: 0x00
kfade[5].freeblock:                   0 ; 0x1ea: 0x0000
kfade[6].entry.incarn:                1 ; 0x1ec: A=1 NUMM=0x0
kfade[6].entry.hash:         3992241470 ; 0x1f0: 0xedf4c53e
kfade[6].entry.refer.number: 4294967295 ; 0x1f4: 0xffffffff
kfade[6].entry.refer.incarn:          0 ; 0x1f8: A=0 NUMM=0x0
kfade[6].name:           xifenfei03.dbf ; 0x1fc: length=14
kfade[6].fnum:                      268 ; 0x22c: 0x0000010c
kfade[6].finc:                918341409 ; 0x230: 0x36bcc721
kfade[6].flags:                      17 ; 0x234: U=1 S=0 S=0 U=0 F=1
kfade[6].ub1spare:                    0 ; 0x235: 0x00
kfade[6].freeblock:                   0 ; 0x236: 0x0000
kfade[7].entry.incarn:                0 ; 0x238: A=0 NUMM=0x0
kfade[7].entry.hash:                  0 ; 0x23c: 0x00000000
kfade[7].entry.refer.number:          0 ; 0x240: 0x00000000
kfade[7].entry.refer.incarn:          0 ; 0x244: A=0 NUMM=0x0
kfade[7].name:                          ; 0x248: length=0
kfade[7].fnum:                        0 ; 0x278: 0x00000000
kfade[7].finc:                        0 ; 0x27c: 0x00000000
kfade[7].flags:                       0 ; 0x280: U=0 S=0 S=0 U=0 F=0
kfade[7].ub1spare:                    0 ; 0x281: 0x00
kfade[7].freeblock:                   0 ; 0x282: 0x0000
www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=26 blkn=1|grep name
kfade[0].name:                 DATAFILE ; 0x034: length=8
kfade[1].name:              CONTROLFILE ; 0x080: length=11
kfade[2].name:                ONLINELOG ; 0x0cc: length=9
kfade[3].name:                 TEMPFILE ; 0x118: length=8
kfade[4].name:            PARAMETERFILE ; 0x164: length=13
kfade[5].name:         spfileora10g.ora ; 0x1b0: length=16
kfade[6].name:           xifenfei03.dbf ; 0x1fc: length=14
kfade[7].name:                          ; 0x248: length=0
kfade[8].name:                          ; 0x294: length=0
www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=26 blkn=2
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                           11 ; 0x002: KFBTYP_ALIASDIR
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       2 ; 0x004: blk=2
kfbh.block.obj:                       6 ; 0x008: file=6
kfbh.check:                  3937052433 ; 0x00c: 0xeaaaa711
kfbh.fcn.base:                     3535 ; 0x010: 0x00000dcf
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kffdnd.bnode.incarn:                  1 ; 0x000: A=1 NUMM=0x0
kffdnd.bnode.frlist.number:  4294967295 ; 0x004: 0xffffffff
kffdnd.bnode.frlist.incarn:           0 ; 0x008: A=0 NUMM=0x0
kffdnd.overfl.number:        4294967295 ; 0x00c: 0xffffffff
kffdnd.overfl.incarn:                 0 ; 0x010: A=0 NUMM=0x0
kffdnd.parent.number:                 1 ; 0x014: 0x00000001
kffdnd.parent.incarn:                 1 ; 0x018: A=1 NUMM=0x0
kffdnd.fstblk.number:                 2 ; 0x01c: 0x00000002
kffdnd.fstblk.incarn:                 1 ; 0x020: A=1 NUMM=0x0
kfade[0].entry.incarn:                1 ; 0x024: A=1 NUMM=0x0
kfade[0].entry.hash:         1410293950 ; 0x028: 0x540f60be
kfade[0].entry.refer.number: 4294967295 ; 0x02c: 0xffffffff
kfade[0].entry.refer.incarn:          0 ; 0x030: A=0 NUMM=0x0
kfade[0].name:                   SYSTEM ; 0x034: length=6
kfade[0].fnum:                      256 ; 0x064: 0x00000100
kfade[0].finc:                914797317 ; 0x068: 0x3686b305
kfade[0].flags:                      18 ; 0x06c: U=0 S=1 S=0 U=0 F=1
kfade[0].ub1spare:                    0 ; 0x06d: 0x00
kfade[0].freeblock:                   0 ; 0x06e: 0x0000
kfade[1].entry.incarn:                1 ; 0x070: A=1 NUMM=0x0
kfade[1].entry.hash:         1052386617 ; 0x074: 0x3eba2539
kfade[1].entry.refer.number: 4294967295 ; 0x078: 0xffffffff
kfade[1].entry.refer.incarn:          0 ; 0x07c: A=0 NUMM=0x0
kfade[1].name:                   SYSAUX ; 0x080: length=6
kfade[1].fnum:                      257 ; 0x0b0: 0x00000101
kfade[1].finc:                914797317 ; 0x0b4: 0x3686b305
kfade[1].flags:                      18 ; 0x0b8: U=0 S=1 S=0 U=0 F=1
kfade[1].ub1spare:                    0 ; 0x0b9: 0x00
kfade[1].freeblock:                   0 ; 0x0ba: 0x0000
kfade[2].entry.incarn:                1 ; 0x0bc: A=1 NUMM=0x0
kfade[2].entry.hash:         2341166852 ; 0x0c0: 0x8b8b5f04
kfade[2].entry.refer.number: 4294967295 ; 0x0c4: 0xffffffff
kfade[2].entry.refer.incarn:          0 ; 0x0c8: A=0 NUMM=0x0
kfade[2].name:                 UNDOTBS1 ; 0x0cc: length=8
kfade[2].fnum:                      258 ; 0x0fc: 0x00000102
kfade[2].finc:                914797317 ; 0x100: 0x3686b305
kfade[2].flags:                      18 ; 0x104: U=0 S=1 S=0 U=0 F=1
kfade[2].ub1spare:                    0 ; 0x105: 0x00
kfade[2].freeblock:                   0 ; 0x106: 0x0000
kfade[3].entry.incarn:                1 ; 0x108: A=1 NUMM=0x0
kfade[3].entry.hash:           18985629 ; 0x10c: 0x0121b29d
kfade[3].entry.refer.number: 4294967295 ; 0x110: 0xffffffff
kfade[3].entry.refer.incarn:          0 ; 0x114: A=0 NUMM=0x0
kfade[3].name:                    USERS ; 0x118: length=5
kfade[3].fnum:                      259 ; 0x148: 0x00000103
kfade[3].finc:                914797317 ; 0x14c: 0x3686b305
kfade[3].flags:                      18 ; 0x150: U=0 S=1 S=0 U=0 F=1
kfade[3].ub1spare:                    0 ; 0x151: 0x00
kfade[3].freeblock:                   0 ; 0x152: 0x0000
kfade[4].entry.incarn:                1 ; 0x154: A=1 NUMM=0x0
kfade[4].entry.hash:          379856949 ; 0x158: 0x16a42835
kfade[4].entry.refer.number: 4294967295 ; 0x15c: 0xffffffff
kfade[4].entry.refer.incarn:          0 ; 0x160: A=0 NUMM=0x0
kfade[4].name:                 XIFENFEI ; 0x164: length=8
kfade[4].fnum:                      266 ; 0x194: 0x0000010a
kfade[4].finc:                918341361 ; 0x198: 0x36bcc6f1
kfade[4].flags:                      18 ; 0x19c: U=0 S=1 S=0 U=0 F=1
kfade[4].ub1spare:                    0 ; 0x19d: 0x00
kfade[4].freeblock:                   0 ; 0x19e: 0x0000
kfade[5].entry.incarn:                1 ; 0x1a0: A=1 NUMM=0x0
kfade[5].entry.hash:          889929475 ; 0x1a4: 0x350b3f03
kfade[5].entry.refer.number: 4294967295 ; 0x1a8: 0xffffffff
kfade[5].entry.refer.incarn:          0 ; 0x1ac: A=0 NUMM=0x0
kfade[5].name:                 XIFENFEI ; 0x1b0: length=8
kfade[5].fnum:                      267 ; 0x1e0: 0x0000010b
kfade[5].finc:                918341389 ; 0x1e4: 0x36bcc70d
kfade[5].flags:                      18 ; 0x1e8: U=0 S=1 S=0 U=0 F=1
kfade[5].ub1spare:                    0 ; 0x1e9: 0x00
kfade[5].freeblock:                   0 ; 0x1ea: 0x0000
kfade[6].entry.incarn:                1 ; 0x1ec: A=1 NUMM=0x0
kfade[6].entry.hash:         3416790953 ; 0x1f0: 0xcba817a9
kfade[6].entry.refer.number: 4294967295 ; 0x1f4: 0xffffffff
kfade[6].entry.refer.incarn:          0 ; 0x1f8: A=0 NUMM=0x0
kfade[6].name:           xifenfei02.dbf ; 0x1fc: length=14
kfade[6].fnum:                      267 ; 0x22c: 0x0000010b
kfade[6].finc:                918341389 ; 0x230: 0x36bcc70d
kfade[6].flags:                      17 ; 0x234: U=1 S=0 S=0 U=0 F=1
kfade[6].ub1spare:                    0 ; 0x235: 0x00
kfade[6].freeblock:                   0 ; 0x236: 0x0000
kfade[7].entry.incarn:                1 ; 0x238: A=1 NUMM=0x0
kfade[7].entry.hash:         3200622536 ; 0x23c: 0xbec59fc8
kfade[7].entry.refer.number: 4294967295 ; 0x240: 0xffffffff
kfade[7].entry.refer.incarn:          0 ; 0x244: A=0 NUMM=0x0
kfade[7].name:                 XIFENFEI ; 0x248: length=8
kfade[7].fnum:                      268 ; 0x278: 0x0000010c
kfade[7].finc:                918341409 ; 0x27c: 0x36bcc721
kfade[7].flags:                      18 ; 0x280: U=0 S=1 S=0 U=0 F=1
kfade[7].ub1spare:                    0 ; 0x281: 0x00
kfade[7].freeblock:                   0 ; 0x282: 0x0000
www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=26 blkn=2|grep name
kfade[0].name:                   SYSTEM ; 0x034: length=6
kfade[1].name:                   SYSAUX ; 0x080: length=6
kfade[2].name:                 UNDOTBS1 ; 0x0cc: length=8
kfade[3].name:                    USERS ; 0x118: length=5
kfade[4].name:                 XIFENFEI ; 0x164: length=8
kfade[5].name:                 XIFENFEI ; 0x1b0: length=8
kfade[6].name:           xifenfei02.dbf ; 0x1fc: length=14
kfade[7].name:                 XIFENFEI ; 0x248: length=8
kfade[8].name:                          ; 0x294: length=0
www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=26 blkn=3
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                           11 ; 0x002: KFBTYP_ALIASDIR
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       3 ; 0x004: blk=3
kfbh.block.obj:                       6 ; 0x008: file=6
kfbh.check:                   362685464 ; 0x00c: 0x159e2418
kfbh.fcn.base:                     1938 ; 0x010: 0x00000792
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kffdnd.bnode.incarn:                  3 ; 0x000: A=1 NUMM=0x1
kffdnd.bnode.frlist.number:  4294967295 ; 0x004: 0xffffffff
kffdnd.bnode.frlist.incarn:           0 ; 0x008: A=0 NUMM=0x0
kffdnd.overfl.number:        4294967295 ; 0x00c: 0xffffffff
kffdnd.overfl.incarn:                 0 ; 0x010: A=0 NUMM=0x0
kffdnd.parent.number:                 1 ; 0x014: 0x00000001
kffdnd.parent.incarn:                 1 ; 0x018: A=1 NUMM=0x0
kffdnd.fstblk.number:                 3 ; 0x01c: 0x00000003
kffdnd.fstblk.incarn:                 3 ; 0x020: A=1 NUMM=0x1
kfade[0].entry.incarn:                3 ; 0x024: A=1 NUMM=0x1
kfade[0].entry.hash:         2951411460 ; 0x028: 0xafeaf704
kfade[0].entry.refer.number: 4294967295 ; 0x02c: 0xffffffff
kfade[0].entry.refer.incarn:          0 ; 0x030: A=0 NUMM=0x0
kfade[0].name:                  Current ; 0x034: length=7
kfade[0].fnum:                      260 ; 0x064: 0x00000104
kfade[0].finc:                914797381 ; 0x068: 0x3686b345
kfade[0].flags:                      18 ; 0x06c: U=0 S=1 S=0 U=0 F=1
kfade[0].ub1spare:                    0 ; 0x06d: 0x00
kfade[0].freeblock:                   0 ; 0x06e: 0x0000
kfade[1].entry.incarn:                0 ; 0x070: A=0 NUMM=0x0
kfade[1].entry.hash:                  0 ; 0x074: 0x00000000
kfade[1].entry.refer.number:          0 ; 0x078: 0x00000000
kfade[1].entry.refer.incarn:          0 ; 0x07c: A=0 NUMM=0x0
kfade[1].name:                          ; 0x080: length=0
kfade[1].fnum:                        0 ; 0x0b0: 0x00000000
kfade[1].finc:                        0 ; 0x0b4: 0x00000000
kfade[1].flags:                       0 ; 0x0b8: U=0 S=0 S=0 U=0 F=0
kfade[1].ub1spare:                    0 ; 0x0b9: 0x00
kfade[1].freeblock:                   0 ; 0x0ba: 0x0000
www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=26 blkn=3|grep name
kfade[0].name:                  Current ; 0x034: length=7
kfade[1].name:                          ; 0x080: length=0
kfade[2].name:                          ; 0x0cc: length=0
kfade[3].name:                          ; 0x118: length=0
kfade[4].name:                          ; 0x164: length=0
www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=26 blkn=4|more
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                           11 ; 0x002: KFBTYP_ALIASDIR
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       4 ; 0x004: blk=4
kfbh.block.obj:                       6 ; 0x008: file=6
kfbh.check:                  3581479529 ; 0x00c: 0xd5790a69
kfbh.fcn.base:                     2167 ; 0x010: 0x00000877
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kffdnd.bnode.incarn:                  1 ; 0x000: A=1 NUMM=0x0
kffdnd.bnode.frlist.number:  4294967295 ; 0x004: 0xffffffff
kffdnd.bnode.frlist.incarn:           0 ; 0x008: A=0 NUMM=0x0
kffdnd.overfl.number:        4294967295 ; 0x00c: 0xffffffff
kffdnd.overfl.incarn:                 0 ; 0x010: A=0 NUMM=0x0
kffdnd.parent.number:                 1 ; 0x014: 0x00000001
kffdnd.parent.incarn:                 1 ; 0x018: A=1 NUMM=0x0
kffdnd.fstblk.number:                 4 ; 0x01c: 0x00000004
kffdnd.fstblk.incarn:                 1 ; 0x020: A=1 NUMM=0x0
kfade[0].entry.incarn:                1 ; 0x024: A=1 NUMM=0x0
kfade[0].entry.hash:         1017821950 ; 0x028: 0x3caabafe
kfade[0].entry.refer.number: 4294967295 ; 0x02c: 0xffffffff
kfade[0].entry.refer.incarn:          0 ; 0x030: A=0 NUMM=0x0
kfade[0].name:                  group_1 ; 0x034: length=7
kfade[0].fnum:                      261 ; 0x064: 0x00000105
kfade[0].finc:                914797385 ; 0x068: 0x3686b349
kfade[0].flags:                      18 ; 0x06c: U=0 S=1 S=0 U=0 F=1
kfade[0].ub1spare:                    0 ; 0x06d: 0x00
kfade[0].freeblock:                   0 ; 0x06e: 0x0000
kfade[1].entry.incarn:                1 ; 0x070: A=1 NUMM=0x0
kfade[1].entry.hash:         1570256801 ; 0x074: 0x5d9837a1
kfade[1].entry.refer.number: 4294967295 ; 0x078: 0xffffffff
kfade[1].entry.refer.incarn:          0 ; 0x07c: A=0 NUMM=0x0
kfade[1].name:                  group_2 ; 0x080: length=7
kfade[1].fnum:                      262 ; 0x0b0: 0x00000106
kfade[1].finc:                914797385 ; 0x0b4: 0x3686b349
kfade[1].flags:                      18 ; 0x0b8: U=0 S=1 S=0 U=0 F=1
kfade[1].ub1spare:                    0 ; 0x0b9: 0x00
kfade[1].freeblock:                   0 ; 0x0ba: 0x0000
kfade[2].entry.incarn:                1 ; 0x0bc: A=1 NUMM=0x0
kfade[2].entry.hash:          157707762 ; 0x0c0: 0x09666df2
kfade[2].entry.refer.number: 4294967295 ; 0x0c4: 0xffffffff
kfade[2].entry.refer.incarn:          0 ; 0x0c8: A=0 NUMM=0x0
kfade[2].name:                  group_3 ; 0x0cc: length=7
kfade[2].fnum:                      263 ; 0x0fc: 0x00000107
kfade[2].finc:                914797387 ; 0x100: 0x3686b34b
kfade[2].flags:                      18 ; 0x104: U=0 S=1 S=0 U=0 F=1
kfade[2].ub1spare:                    0 ; 0x105: 0x00
kfade[2].freeblock:                   0 ; 0x106: 0x0000
kfade[3].entry.incarn:                0 ; 0x108: A=0 NUMM=0x0
kfade[3].entry.hash:                  0 ; 0x10c: 0x00000000
kfade[3].entry.refer.number:          0 ; 0x110: 0x00000000
kfade[3].entry.refer.incarn:          0 ; 0x114: A=0 NUMM=0x0
kfade[3].name:                          ; 0x118: length=0
kfade[3].fnum:                        0 ; 0x148: 0x00000000
kfade[3].finc:                        0 ; 0x14c: 0x00000000
kfade[3].flags:                       0 ; 0x150: U=0 S=0 S=0 U=0 F=0
kfade[3].ub1spare:                    0 ; 0x151: 0x00
kfade[3].freeblock:                   0 ; 0x152: 0x0000
www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=26 blkn=4|grep name
kfade[0].name:                  group_1 ; 0x034: length=7
kfade[1].name:                  group_2 ; 0x080: length=7
kfade[2].name:                  group_3 ; 0x0cc: length=7
kfade[3].name:                          ; 0x118: length=0
kfade[4].name:                          ; 0x164: length=0
kfade[5].name:                          ; 0x1b0: length=0
kfade[6].name:                          ; 0x1fc: length=0
www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=26 blkn=5|more
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                           11 ; 0x002: KFBTYP_ALIASDIR
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       5 ; 0x004: blk=5
kfbh.block.obj:                       6 ; 0x008: file=6
kfbh.check:                  1153372471 ; 0x00c: 0x44bf1137
kfbh.fcn.base:                     2212 ; 0x010: 0x000008a4
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kffdnd.bnode.incarn:                  1 ; 0x000: A=1 NUMM=0x0
kffdnd.bnode.frlist.number:  4294967295 ; 0x004: 0xffffffff
kffdnd.bnode.frlist.incarn:           0 ; 0x008: A=0 NUMM=0x0
kffdnd.overfl.number:        4294967295 ; 0x00c: 0xffffffff
kffdnd.overfl.incarn:                 0 ; 0x010: A=0 NUMM=0x0
kffdnd.parent.number:                 1 ; 0x014: 0x00000001
kffdnd.parent.incarn:                 1 ; 0x018: A=1 NUMM=0x0
kffdnd.fstblk.number:                 5 ; 0x01c: 0x00000005
kffdnd.fstblk.incarn:                 1 ; 0x020: A=1 NUMM=0x0
kfade[0].entry.incarn:                1 ; 0x024: A=1 NUMM=0x0
kfade[0].entry.hash:         3699413877 ; 0x028: 0xdc809375
kfade[0].entry.refer.number: 4294967295 ; 0x02c: 0xffffffff
kfade[0].entry.refer.incarn:          0 ; 0x030: A=0 NUMM=0x0
kfade[0].name:                     TEMP ; 0x034: length=4
kfade[0].fnum:                      264 ; 0x064: 0x00000108
kfade[0].finc:                914797393 ; 0x068: 0x3686b351
kfade[0].flags:                      18 ; 0x06c: U=0 S=1 S=0 U=0 F=1
kfade[0].ub1spare:                    0 ; 0x06d: 0x00
kfade[0].freeblock:                   0 ; 0x06e: 0x0000
kfade[1].entry.incarn:                0 ; 0x070: A=0 NUMM=0x0
kfade[1].entry.hash:                  0 ; 0x074: 0x00000000
kfade[1].entry.refer.number:          0 ; 0x078: 0x00000000
kfade[1].entry.refer.incarn:          0 ; 0x07c: A=0 NUMM=0x0
kfade[1].name:                          ; 0x080: length=0
kfade[1].fnum:                        0 ; 0x0b0: 0x00000000
kfade[1].finc:                        0 ; 0x0b4: 0x00000000
kfade[1].flags:                       0 ; 0x0b8: U=0 S=0 S=0 U=0 F=0
kfade[1].ub1spare:                    0 ; 0x0b9: 0x00
kfade[1].freeblock:                   0 ; 0x0ba: 0x0000
www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=26 blkn=5|grep name
kfade[0].name:                     TEMP ; 0x034: length=4
kfade[1].name:                          ; 0x080: length=0
kfade[2].name:                          ; 0x0cc: length=0
kfade[3].name:                          ; 0x118: length=0
kfade[4].name:                          ; 0x164: length=0
www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=26 blkn=6|more
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                           11 ; 0x002: KFBTYP_ALIASDIR
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       6 ; 0x004: blk=6
kfbh.block.obj:                       6 ; 0x008: file=6
kfbh.check:                  1230193442 ; 0x00c: 0x49534322
kfbh.fcn.base:                     2267 ; 0x010: 0x000008db
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kffdnd.bnode.incarn:                  1 ; 0x000: A=1 NUMM=0x0
kffdnd.bnode.frlist.number:  4294967295 ; 0x004: 0xffffffff
kffdnd.bnode.frlist.incarn:           0 ; 0x008: A=0 NUMM=0x0
kffdnd.overfl.number:        4294967295 ; 0x00c: 0xffffffff
kffdnd.overfl.incarn:                 0 ; 0x010: A=0 NUMM=0x0
kffdnd.parent.number:                 1 ; 0x014: 0x00000001
kffdnd.parent.incarn:                 1 ; 0x018: A=1 NUMM=0x0
kffdnd.fstblk.number:                 6 ; 0x01c: 0x00000006
kffdnd.fstblk.incarn:                 1 ; 0x020: A=1 NUMM=0x0
kfade[0].entry.incarn:                1 ; 0x024: A=1 NUMM=0x0
kfade[0].entry.hash:         3897004393 ; 0x028: 0xe8479169
kfade[0].entry.refer.number: 4294967295 ; 0x02c: 0xffffffff
kfade[0].entry.refer.incarn:          0 ; 0x030: A=0 NUMM=0x0
kfade[0].name:                   spfile ; 0x034: length=6
kfade[0].fnum:                      265 ; 0x064: 0x00000109
kfade[0].finc:                914797421 ; 0x068: 0x3686b36d
kfade[0].flags:                      18 ; 0x06c: U=0 S=1 S=0 U=0 F=1
kfade[0].ub1spare:                    0 ; 0x06d: 0x00
kfade[0].freeblock:                   0 ; 0x06e: 0x0000
kfade[1].entry.incarn:                0 ; 0x070: A=0 NUMM=0x0
kfade[1].entry.hash:                  0 ; 0x074: 0x00000000
kfade[1].entry.refer.number:          0 ; 0x078: 0x00000000
kfade[1].entry.refer.incarn:          0 ; 0x07c: A=0 NUMM=0x0
kfade[1].name:                          ; 0x080: length=0
kfade[1].fnum:                        0 ; 0x0b0: 0x00000000
kfade[1].finc:                        0 ; 0x0b4: 0x00000000
kfade[1].flags:                       0 ; 0x0b8: U=0 S=0 S=0 U=0 F=0
kfade[1].ub1spare:                    0 ; 0x0b9: 0x00
kfade[1].freeblock:                   0 ; 0x0ba: 0x0000
www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=26 blkn=6|grep name
kfade[0].name:                   spfile ; 0x034: length=6
kfade[1].name:                          ; 0x080: length=0
kfade[2].name:                          ; 0x0cc: length=0
kfade[3].name:                          ; 0x118: length=0
kfade[4].name:                          ; 0x164: length=0
www.xifenfei.com>kfed read H:\ASMDISK\ASMDISK1.DD aun=26 blkn=7|grep name
kfade[0].name:                          ; 0x034: length=0
kfade[1].name:                          ; 0x080: length=0
kfade[2].name:                          ; 0x0cc: length=0
kfade[3].name:                          ; 0x118: length=0
kfade[4].name:                          ; 0x164: length=0
kfade[5].name:                          ; 0x1b0: length=0
kfade[6].name:                          ; 0x1fc: length=0

通过上述分析我们发现目前数据主要分布在au=26,block in(0-6)的几个block中,通过kfed已经找出来了所有的asm中文件的file_number

非win平台脚本实现

for (( i=0; i<255; i++ ))
do
   kfed read H:\ASMDISK\ASMDISK1.DD aun=26 blknum=$i
  \|egrep 'name|fnum'|grep -v length=0 |grep -v 0x00000000 >>asm_file.out
done

注意需要按照file 6的au依次处理,否则会不全,更加简单的方法,直接通过dul扫描磁盘获取相关file number

mount数据库也可能有LOCAL=NO的进程

在一次无意中发现mount状态的数据库也有LOCAL=NO的进程,经过分析确定是由于主库连接到备库的nls或者arch进程连接到备库引起的
发现mount库中有LOCAL=NO的进程

[oracle@localhost ~]$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.5.0 - Production on Fri Jul 29 11:59:57 2016
Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> select database_role ,open_mode from v$database;
DATABASE_ROLE    OPEN_MODE
---------------- ----------
PHYSICAL STANDBY MOUNTED
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@localhost ~]$ ps -ef|grep LOCAL
oracle   11394     1  0 Apr27 ?        08:08:41 oracleorcl (LOCAL=NO)
oracle   11398     1  0 Apr27 ?        15:36:29 oracleorcl (LOCAL=NO)
oracle   18854 18752  0 12:00 pts/2    00:00:00 grep LOCAL
[oracle@localhost ~]$ ps -ef|grep pmon
oracle   14374     1  0  2015 ?        00:10:54 ora_pmon_orcl
oracle   18893 18752  0 12:01 pts/2    00:00:00 grep pmon
SQL>  select sid,status,username from v$session where paddr in
   2  (select addr from v$process where spid in(11394,11398));
       SID STATUS   USERNAME
---------- -------- ------------------------------
       510 INACTIVE PUBLIC
       507 INACTIVE PUBLIC

查看备库进程连接

[oracle@localhost ~]$ netstat -natp|grep -E '11394|11398'
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 192.168.160.22:1521         192.168.160.23:42783        ESTABLISHED 11394/oracleorcl
tcp        0      0 192.168.160.22:1521         192.168.160.23:42785        ESTABLISHED 11398/oracleorcl

主库上查看,确定192.168.160.22是备库

SQL> show parameter log_archive_dest_2;
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_2                   string      service=orcl lgwr async valid_
                                                 for=(online_logfiles,primary_r
                                                 ole) db_unique_name=orcl
SQL> !tnsping orcl
TNS Ping Utility for Linux: Version 10.2.0.5.0 - Production on 29-JUL-2016 12:20:01
Copyright (c) 1997,  2010, Oracle.  All rights reserved.
Used parameter files:
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST =192.168.160.22)(PORT = 1521))
 (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = orcl)))
OK (0 msec)

查看主库连接

[oracle@localhost ~]$ netstat -natp|grep "192.168.160.22"
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 192.168.160.23:42785        192.168.160.22:1521         ESTABLISHED 12394/ora_arc1_orcl
tcp        0      0 192.168.160.23:42783        192.168.160.22:1521         ESTABLISHED 12400/ora_lns1_orcl

通过分析确定在mount情况的备库中,会有LOCAL=NO的进程,他们是主库arch和lns进程对应的服务进程

_OFFLINE_ROLLBACK_SEGMENTS/_CORRUPTED_ROLLBACK_SEGMENTS

对于oracle undo异常的时候恢复中,经常需要使用的_OFFLINE_ROLLBACK_SEGMENTS和_CORRUPTED_ROLLBACK_SEGMENTS参数,关于这两个参数的区别进行说明
_OFFLINE_ROLLBACK_SEGMENTS 参数说明

_CORRUPTED_ROLLBACK_SEGMENTS 参数说明

_OFFLINE_ROLLBACK_SEGMENTS 和 _CORRUPTED_ROLLBACK_SEGMENTS 区别
offline_corrupted

这两个参数属于oracle隐含参数,在没有oracle support的情况下,请慎用.该相关参数可能导致数据库逻辑不一致风险,如果使用了,建议逻辑方式导出导入库

oracle asm系列文章汇总

ORA-600 4193 错误说明和解决

ORA-600 4193 解释说明

ERROR:
  Format: ORA-600 [4193] [a] [b]
VERSIONS:
  versions 6.0 to 12.1
DESCRIPTION:
  A mismatch has been detected between Redo records and Rollback (Undo)
  records.
  We are validating the Undo block sequence number in the undo block against
  the Redo block sequence number relating to the change being applied.
  This error is reported when this validation fails.
ARGUMENTS:
  Arg [a] Undo record seq number
  Arg [b] Redo record seq number
FUNCTIONALITY:
  KERNEL TRANSACTION UNDO
ORA-600 [4193] [a] [b] [ ] [ ]  [ ]
Versions: 7.2.2  - 9.2.0                              Source: ktuc.c
===========================================================================
Meaning: seq# mismatch while adding an undo record to an undo block. This
         is done by the application of redo.
---------------------------------------------------------------------------
Argument Description:
    a. (ktubhseq): undo record seq# - this is the seq# of the block that
                                      this undo record WILL BE APPLIED TO.
                                      This is from the Undo Block. It is
                                      NOT the seq# of the undo block itself.
    b. (ktudbseq): redo RECORD seq# - this is the seq# number in the block
                                      that this redo WILL BE APPLIED TO.
                                      This is from the Redo Record.
---------------------------------------------------------------------------
Diagnosis:
    This error is raised in kturdb which handles the adding of undo records
    by the application of redo.
    When we try to apply redo to an undo block (forward changes are made by
    the application of redo to a block) we check that the seq# in the undo
    record matches the seq# in the redo record. These seq# should be the
    same because when we apply a redo record we must apply it to the
    correct version of the block. We can only apply a redo record to a
    block that contains the same seq# as in the redo record.
    If the seq# do not match then this error is raised. This implies some
    kind of block corruption in either the redo or the undo block.
7.3.x - 8.1.7.x
ASSERT2(ubh->ktubhseq == db->ktudbseq, OERI(4193), KSESVSGN,
            ubh->ktubhseq, db->ktudbseq);
9.2.x
ksesic2(OERI(4193), ksenrg(ubh->ktubhseq), ksenrg(db->ktudbseq));
struct ktubh
{
  kxid  ktubhxid;      /* txid of tx currently using or last used this block */
  ub2   ktubhseq;                              /* undo block sequence number */
  ub1   ktubhcnt;    /* high water mark record index, number of undo entries */
  ub1   ktubhirb;  /* rollback record index, rec index to start the rollback */
  ub1   ktubhicl;  /* collecting record index, rec index to start retrieving col info */
  ub1   ktubhflg;                                                 /* dummy */
  ub2   ktubhidx[1];     /* byte offset of record in block, grows at runtime */
};
struct ktudb   Kernel Transaction Undo Data operation Block (redo)
{
  ub2    ktudbsiz;                                          /* size of entry */
  ub2    ktudbspc;                 /* verification: space left in undo block */
  ub2    ktudbflg;            /* flag to indicate the kind of redo operation */
  kxid   ktudbxid;                                          /* current tx id */
  ub2    ktudbseq;                                  /* block sequence number */
  ub1    ktudbrec;                       /* new record index for this change */
};

ORA 600 4193 处理方法同How to resolve ORA-600 [4194] errors

How to resolve ORA-600 [4194] errors

在oracle恢复中ORA-600 4194是一个非常常见的错误,该错误的主要原因是由于redo记录和undo(rollback)记录不匹配.
ORA 600 4194错误原因以及含义

ERROR:
  Format: ORA-600 [4194] [a] [b]
VERSIONS:
  versions 6.0 to 12.1
DESCRIPTION:
  A mismatch has been detected between Redo records and rollback (Undo)
  records.
  We are validating the Undo record number relating to the change being
  applied against the maximum undo record number recorded in the undo block.
  This error is reported when the validation fails.
ARGUMENTS:
  Arg [a] Maximum Undo record number in Undo block
  Arg [b] Undo record number from Redo block

ORA 600 4194 错误处理思路
第一步

Confirm whether the database is up and running or not.  If the database fails to start or crashes shortly
after startup due to this error occurring, then try setting event 10513 at level 2 in the init.ora/spfile
to disable transaction recovery and restart the instance, e.g.:
      event = "10513 trace name context forever, level 2"
This may allow the database to successfully open and stay up so that
the required diagnostics/actions can be performed.

第二步

In the trace file there should be an undo segment header dump, and so check
to see if the undo segment header shows an active transaction after recovery, e.g.:
TRN TBL    <---- Represents the Transaction table for the particular undo segment
index state cflags wrap# uel scn dba
---------------------------------------------------------------------------------------------
0x41 9 0x80 0x35ab6 0xffff 0x0695.38f6b959 0x1081e796
0x42 9 0x80 0x35bb1 0x000e 0x0695.38f6b028 0x1081e793
0x43 9 0x80 0x35b11 0x005d 0x0695.38f6b7ae 0x1081e795
0x44 9 0x80 0x359f0 0x0036 0x0695.38f69a91 0x1081e78e
0x45 10 0x80 0x35b1b 0x0000 0x0695.3a0aba4d 0x1081e796
0x46 9 0x80 0x35bb7 0x001c 0x0695.38f69bde 0x1081e78f
===================================
State ---> This column specifies the status of the transaction
                  9 -----> represents a commited transaction
                  10 ---> Represents a active transaction
Dba -----> Undo block containing the undo records
                  Strictly speaking this is the block at the end of the undo chain.
You can see from the transaction table that there is an active transaction
for this particular rollback/undo segment after recovery.
Therefore this rollback/undo segment and/or undo tablespace cannot be dropped without corrupting the database!
Therefore recreating the UNDO tablespace is not an option.

第三步

From the trace file determine the affected undo segment, e.g.:
Block image after block recovery:
UNDO BLK:
xid: 0x0015.02b.0001544b seq: 0x163e cnt: 0x12 irb: 0x12 icl: 0x0 flg: 0x0000
XID ==> Undo segment no + Slot no + Sequence no
Therefore, in this case the Undo Segment is:
USN# 0x15 (Hex) ==> 21 (Dec)  ==> _SYSSMU21$
So if and ONLY IF the transaction table shows no active transaction can the
 rollback/undo segment be offlined and dropped.Note however,
that before you can confirm if the entire UNDO tablespace can be dropped, you would need to check the
transaction tables of ALL active rollback/undo segments in the same wasy as the above.
The steps required to drop the rollback/undo segment are fully detailed in Note:179952.1,
but are briefly listed here for completeness:
If using Automatic Undo Management
Offline the undo segment using the _OFFLINE_ROLLBACK_SEGMENTS parameter and bounce the database as follows:
1.  Create  and edit the init.ora file for the instance to set the following parameters:
UNDO_MANAGEMENT=MANUAL
_OFFLINE_ROLLBACK_SEGMENTS=(_SYSSMU21$)
2.  Open the database in restricted mode to prevent user access, e.g.:
connect / as sysdba
startup restrict pfile = '<Full path to init.ora file>';
3.  Drop the rollback/undo segment, e.g.:
drop rollback segment "_SYSSMU21";
4.  Shutdown the instance, and remove the init.ora parameters added in point 1 and restart the instance, e.g.:
shutdown immediate
startup
If SMON was recovering the transaction then this may not work as we cannot open the database if it is determined
to be in an inconsistent state. I have reviewed a number of SRs where this approach was successful,
so it is important to try it first but understand that it may fail and you will have to resort to
a point in time recovery or forcing open the DB and recreating it.

第四步

Now we need to dump the undo block to see which object was affected.
We noted in Step 2 that this is the active transaction (from the trace file):
TRN TBL
index state cflags wrap# uel scn dba
0x45 10 0x80 0x35b1b 0x0000 0x0695.3a0aba4d 0x1081e796
Dba----------------> Undo block containing the undo records
dba--->0x1081e796 is the block containing the active transaction .
Use the WebIV tools to convert this RDBA to block number (block#) and file number (file#), e.g.:
V SPLIT ==> DBA (Hex) = File#,Block# (Hex File#,Block#)
= ===== === ===== ============
V8 10,10 ==> 276948886 (0x1081e796) = 66,124822 (0x42 0x1e796)
So the file# is 66 and the block# is 124822, so dump the block by issuing:
SQL> Alter system dump datafile 66 block 124822;
This will generate a trace file in the user_dump_dest.  The following is a sample of the information in the undo block:
UNDO BLK:
xid: 0x000c.045.00035b1b seq: 0x1e14 cnt: 0x17 irb: 0x17 icl: 0x0 flg: 0x0000
Rec Offset Rec Offset Rec Offset Rec Offset Rec Offset
---------------------------------------------------------------------------
0x01 0x1f8c 0x02 0x1f30 0x03 0x1ed4 0x04 0x1e78 0x05 0x1e1c
0x06 0x1dc0 0x07 0x1d64 0x08 0x1d08 0x09 0x1cac 0x0a 0x1c50
0x0b 0x1bf4 0x0c 0x1b98 0x0d 0x1b3c 0x0e 0x1ae0 0x0f 0x1a74
0x10 0x1a18 0x11 0x19bc 0x12 0x1960 0x13 0x1904 0x14 0x187c
0x15 0x181c 0x16 0x1798 0x17 0x173c
* Rec #0x16 slt: 0x45 objn: 1485619(0x0016ab33) objd: 1485619 tblspc: 71(0x00000047)
* Layer: 11 (Row) opc: 1 rci 0x00
Undo type: Regular undo Begin trans Last buffer split: No
Temp Object: No
Tablespace Undo: No
rdba: 0x00000000
*-----------------------------
uba: 0x1081e796.1e14.14 ctl max scn: 0x0695.38f69853 prv tx scn: 0x0695.38f698a1
KDO undo record:
KTB Redo
op: 0x04 ver: 0x01
op: L itl: scn: 0x0019.009.00034237 uba: 0x36c0cce4.1d2f.19
flg: C--- lkc: 0 scn: 0x0695.38f6b96b
KDO Op code: URP xtype: XA bdba: 0x35406893 hdba: 0x35406892
itli: 1 ispac: 0 maxfr: 4863
tabn: 0 slot: 0(0x0) flag: 0x2c lock: 0 ckix: 0
ncol: 1 nnew: 1 size: -1
col 0: [ 4] c3 0e 36 2e
*-----------------------------
* Rec #0x17 slt: 0x45 objn: 1485619(0x0016ab33) objd: 1485619 tblspc: 71(0x00000047)
* Layer: 11 (Row) opc: 1 rci 0x16
Undo type: Regular undo Last buffer split: No
Temp Object: No
Tablespace Undo: No
rdba: 0x00000000
*-----------------------------
From the trace file above:
UNDO BLK:
xid: 0x000c.045.00035b1b seq: 0x1e14 cnt: 0x17 irb: 0x17 icl: 0x0 flg: 0x0000
The undo segment with the active transaction is segment is 0x000c (Hex) which is 12 (Dec) as the XID is:
      Undo segment no + Slot no + Sequence no
This step is often skipped because it was performed earlier in step 3, however it is a good idea to do this
again now to make sure that the XID from the UNDO block matches the UNDO SEGMENT HEADER,
this way you have followed all the chain, from the UNDO SEGMENT to UNDO BLOCK, back and forth.
If there is a conflict here please check and make sure that the customer dumped the correct undo block.
Check for the value of irb which is an index which points you to the latest change done to the undo block.
This is the point from which a rollback would begin if one was issued.
From the trace file we see: 'irb: 0x17' so this points to record 0x17,
so search for this particular string i.e 0x17 and it will take you to undo record 'REC #0x17', e.g.:
* Rec #0x17 slt: 0x45 objn: 1485619(0x0016ab33) objd: 1485619 tblspc: 71(0x00000047)
* Layer: 11 (Row) opc: 1 rci 0x16
Undo type: Regular undo Last buffer split: No
Temp Object: No
Tablespace Undo: No
rdba: 0x00000000
*-----------------------------
Note the slot number (slt) is 0x45, the object number (objn) is the OBJECT_ID from dba_objects
and data object number (objd) is the DATA_OBJECT_ID from dba_objects.
These numbers may be the same but not necessarily, and so if the database is open then identify this object, e.g.:
        select object_name, owner, object_type, data_object_id from dba_objects where object_id = <objn>;
This is the object, which has an active transaction.  Note in the above trace file extract that rci
has a value of 0x16 which means that this record is at the end of an undo chain.
This means that the chain continues in another UNDO BLOCK.
Please refer to unpublished Note:281504.1 for information on Undo chains.
So the next record that needs to be rolled back is present in REC #X016.
If rci is 0x00 then it means that this is the first record present in the undo chain
and so you can check to see if there is rdba info, e.g.:
* Rec #0x16 slt: 0x45 objn: 1485619(0x0016ab33) objd: 1485619 tblspc: 71(0x00000047)
* Layer: 11 (Row) opc: 1 rci 0x00
Undo type: Regular undo Begin trans Last buffer split: No
Temp Object: No
Tablespace Undo: No
rdba: 0x00000000
*-----------------------------
uba: 0x1081e796.1e14.14 ctl max scn: 0x0695.38f69853 prv tx scn: 0x0695.38f698a1
KDO undo record:
KTB Redo
op: 0x04 ver: 0x01
op: L itl: scn: 0x0019.009.00034237 uba: 0x36c0cce4.1d2f.19
flg: C--- lkc: 0 scn: 0x0695.38f6b96b
KDO Op code: URP xtype: XA bdba: 0x35406893 hdba: 0x35406892
itli: 1 ispac: 0 maxfr: 4863
tabn: 0 slot: 0(0x0) flag: 0x2c lock: 0 ckix: 0
ncol: 1 nnew: 1 size: -1
col 0: [ 4] c3 0e 36 2e
*-----------------------------
If the object is an Index, drop and recreate it.  If it is a table,
then again the table would need to be dropped and recreated (or truncated)
so that its object number changes and hence the rollback/undo is no longer required.
If this isn't possible, then you have two options:
First take a backup of the database in its current state.
This is critical in case anything goes wrong and you lose the opportunity to salvage the data!
Option 1
 - Restore the undo segment datafile and the datafile containing the object and perform a full recovery.
   This can only be done if you have all the archived redo as you will need to do full recovery on these files.
OR
Option 2
If option 1 is not possible, you can use the unsupported method, e.g.:
Specify the undo segment in the _OFFLINE_ROLLBACK_SEGMENTS parameter and try to drop the rollback segment.
If there is an active transaction then this is not likely to work and you will probably need
to set the _CORRUPTED_ROLLBACK_SEGMENTS parameter as well

温馨提示：
1.隐含参数_OFFLINE_ROLLBACK_SEGMENTS/_CORRUPTED_ROLLBACK_SEGMENTS属于Oracle内部隐含参数,建议在Oracle support认可的情况下使用,因为使用之后可能导致数据库事务完整性彻底损坏
2.进行屏蔽事务之前,如果条件允许建议使用txchecker检查
2.使用上述方法恢复数据库之后，建议通过逻辑方式导出导入重建数据库

Exadata火线救援：10TB级数据恢复—强制拉库篇

这个库的恢复有一些历史故事(【力荐】Exadata火线救援：10TB级数据修复经典案例详解！)：xx运营商x2的1/4配置的oracle exadata机器，跑了近6年，最近有一个cell节点主机异常,在rebalance过程中,只有两个节点的cell其中一个节点坏了一个硬盘导致.导致asm diskgroup无法正常mount,最后该运营商运维三方通过amdu把该一体机中的数据文件全部抽出来,然后在恢复过程中出现大量错误无法解决,请求我们支持
数据库open过程报ORA-01555错误

Thu Jul  14 00:01:04 2016
alter database open
Thu Jul  14 00:01:04 2016
Thread 1 advanced to log sequence 2 (thread open)
Thread 1 opened at log sequence 2
  Current log# 2 seq# 2 mem# 0: /data/amdu/redo/DATA_EC_260.f
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu Jul  14 00:01:05 2016
SMON: enabling cache recovery
ORA-01555 caused by SQL statement below (SQL ID: 4krwuz0ctqxdt, SCN: 0x0b26.9f080238):
select ctime, mtime, stime from obj$ where obj# = :1
Errors in file /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_59546.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 83 with name &quot;_SYSSMU83_1078760807$&quot; too small
Errors in file /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_59546.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 83 with name &quot;_SYSSMU83_1078760807$&quot; too small
Error 704 happened during db open, shutting down database
USER (ospid: 59546): terminating the instance due to error 704
Instance terminated by USER, pid = 59546
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (59546) as a result of ORA-1092

这个错误比较常见，可以通过推scn就可以解决,由于已经安装了scn patch,通过oradebug推scn解决该问题.

ORA-600 4194
这个ora 600 4194相对比较特殊,在SMON: enabling cache recovery之后立马报出来,然后实例直接open失败.

Thu Jul  14 00:06:15 2016
alter database open
Thu Jul  14 00:06:15 2016
Thread 1 advanced to log sequence 3 (thread open)
Thread 1 opened at log sequence 3
  Current log# 3 seq# 3 mem# 0: /data/amdu/redo/DATA_EC_263.f
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu Jul  14 00:06:15 2016
SMON: enabling cache recovery
Errors in file /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_60038.trc  (incident=1080450):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Block recovery from logseq 3, block 3 to scn 12269096776739
Recovery of Online Redo Log: Thread 1 Group 3 Seq 3 Reading mem 0
  Mem# 0: /data/amdu/redo/DATA_EC_263.f
Block recovery stopped at EOT rba 3.5.16
Block recovery completed at rba 3.5.16, scn 2856.2670179361
Block recovery from logseq 3, block 3 to scn 12269096776736
Recovery of Online Redo Log: Thread 1 Group 3 Seq 3 Reading mem 0
  Mem# 0: /data/amdu/redo/DATA_EC_263.f
Block recovery completed at rba 3.5.16, scn 2856.2670179361
Errors in file /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_60038.trc:
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Errors in file /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_60038.trc:
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 60038): terminating the instance due to error 600
Instance terminated by USER, pid = 60038
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (60038) as a result of ORA-1092

trace文件分析
打开数据库报ORA-600[4194]错误,对启动过程进行10046跟踪并且分析trace文件发现

PARSING IN CURSOR #140375370511672 len=148 dep=1 uid=0 oct=6 lid=0 tim=3501342849457766 hv=3540833987
ad='a47df47a8' sqlid='5ansr7r9htpq3'
update undo$ set name=:2,file#=:3,block#=:4,status$=:5,user#=:6,undosqn=:7,xactsqn=:8,scnbas=:9,
scnwrp=:10,inst#=:11,ts#=:12,spare1=:13 where us#=:1
END OF STMT
PARSE #140375370511672:c=27996,e=28041,p=66,cr=224,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=3501342849457765
BINDS #140375370511672:
 Bind#0
  oacdty=01 mxl=32(20) mxlc=00 mal=00 scl=00 pre=00
  oacflg=18 fl2=0001 frm=01 csi=178 siz=32 off=0
  kxsbbbfp=a47e093ca  bln=32  avl=20  flg=09
  value=&quot;_SYSSMU1_2856534670$&quot;
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fabae3b92b0  bln=24  avl=03  flg=05
  value=1024
 Bind#2
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fabae3b9280  bln=24  avl=03  flg=05
  value=128
 Bind#3
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fabae3b9248  bln=24  avl=02  flg=05
  value=5
 Bind#4
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fabae3b9218  bln=24  avl=02  flg=05
  value=1
 Bind#5
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fabae3b91e8  bln=24  avl=03  flg=05
  value=3398
 Bind#6
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fabae3b91b8  bln=24  avl=05  flg=05
  value=1485261
 Bind#7
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fabae3b9180  bln=24  avl=06  flg=05
  value=1946693999
 Bind#8
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fabae3b8ec8  bln=24  avl=03  flg=05
  value=2847
 Bind#9
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fabae3b8e98  bln=24  avl=02  flg=05
  value=1
 Bind#10
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fabae3b8e68  bln=24  avl=02  flg=05
  value=2
 Bind#11
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fabae3b8e38  bln=24  avl=02  flg=05
  value=2
 Bind#12
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fabae3b92e0  bln=22  avl=02  flg=05
  value=1
WAIT #140375370511672: nam='db file sequential read' ela= 21 file#=1 block#=179020 blocks=1 obj#=0 tim=3501342849459353
*** 2016-07-14 03:14:09.548
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []

很明显数据库是在update undo$的时候,读取到file 1 block 179020的时候报错,继续分析trace文件

Error 600 in redo application callback
Dump of change vector:
TYP:0 CLS:16 AFN:1 DBA:0x0042bb4c OBJ:4294967295 SCN:0x0b26.a88e815e SEQ:1 OP:5.1 ENC:0 RBL:0
ktudb redo: siz: 272 spc: 4906 flg: 0x0012 seq: 0x0f4c rec: 0x0d
            xid:  0x0000.01b.00000b63
ktubl redo: slt: 27 rci: 0 opc: 11.1 [objn: 15 objd: 15 tsn: 0]
Undo type:  Regular undo        Begin trans    Last buffer split:  No
Temp Object:  No
Tablespace Undo:  No
             0x00000000  prev ctl uba: 0x0042bb4c.0f4c.0c
prev ctl max cmt scn:  0x0b23.ccb9eb89  prev tx cmt scn:  0x0b23.ccb9ebac
txn start scn:  0xffff.ffffffff  logon user: 0  prev brb: 4373321  prev bcl: 0 BuExt idx: 0 flg2: 0
KDO undo record:
KTB Redo
op: 0x04  ver: 0x01
compat bit: 4 (post-11) padding: 1
op: L  itl: xid:  0x0000.01c.00000b65 uba: 0x0042bb4a.0f4c.1d
                      flg: C---    lkc:  0     scn: 0x0b25.7391ee5c
KDO Op code: URP row dependencies Disabled
  xtype: XA flags: 0x00000000  bdba: 0x004000e1  hdba: 0x004000e0
itli: 2  ispac: 0  maxfr: 4863
tabn: 0 slot: 1(0x1) flag: 0x2c lock: 0 ckix: 0
ncol: 17 nnew: 12 size: 0
col  1: [20]  5f 53 59 53 53 4d 55 31 5f 32 38 35 36 35 33 34 36 37 30 24
col  2: [ 2]  c1 02
col  3: [ 3]  c2 0b 19
col  4: [ 3]  c2 02 1d
col  5: [ 6]  c5 14 2f 46 28 64
col  6: [ 3]  c2 1d 30
col  7: [ 5]  c4 02 31 35 3e
col  8: [ 3]  c2 22 63
col  9: [ 2]  c1 02
col 10: [ 2]  c1 04
col 11: [ 2]  c1 03
col 16: [ 2]  c1 03
Block after image is corrupt:
buffer tsn: 0 rdba: 0x0042bb4c (1/179020)
scn: 0x0b26.a88e815e seq: 0x01 flg: 0x04 tail: 0x815e0201
frmt: 0x02 chkval: 0xf022 type: 0x02=KTU UNDO BLOCK
Hex dump of block: st=0, typ_found=1
Dump of memory from 0x000000094E07A000 to 0x000000094E07C000
94E07A000 0000A202 0042BB4C A88E815E 04010B26  [....L.B.^...&amp;...]
94E07A010 0000F022 004E0000 00000B5B 1D1D0F4C  [&quot;.....N.[...L...]

我们知道在file 1 block 179020的时候redo和undo信息不匹配,出现了上述的ORA 600 4194的错误.进一步分析

Block image after block recovery:
buffer tsn: 0 rdba: 0x00400080 (1/128)
scn: 0x0b26.6389e19d seq: 0x01 flg: 0x04 tail: 0xe19d0e01
frmt: 0x02 chkval: 0x7c95 type: 0x0e=KTU UNDO HEADER W/UNLIMITED EXTENTS
Hex dump of block: st=0, typ_found=1
Dump of memory from 0x000000094DAB2000 to 0x000000094DAB4000
94DAB2000 0000A20E 00400080 6389E19D 04010B26  [......@....c&amp;...]
94DAB2010 00007C95 00000000 00000000 00000000  [.|..............]
94DAB2020 00000000 00000015 000002FF 00001020  [............ ...]
94DAB2030 0000000D 0000004C 00000080 0042BB4C  [....L.......L.B.]
  Extent Control Header
  -----------------------------------------------------------------
  Extent Header:: spare1: 0      spare2: 0      #extents: 21     #blocks: 767
                  last map  0x00000000  #maps: 0      offset: 4128
      Highwater::  0x0042bb4c  ext#: 13     blk#: 76     ext size: 128
  #blocks in seg. hdr's freelists: 0
  #blocks below: 0
  mapblk  0x00000000  offset: 13
                   Unlocked
     Map Header:: next  0x00000000  #extents: 21   obj#: 0      flag: 0x40000000
  Extent Map
  -----------------------------------------------------------------
   0x00400081  length: 7
   0x00413a38  length: 8
   0x00400088  length: 8
   0x00413a30  length: 8
   0x0042b888  length: 8
   0x0042b890  length: 8
   0x0042b898  length: 8
   0x0042b8a0  length: 8
   0x0042b8a8  length: 8
   0x0042b8b0  length: 8
   0x0042b8b8  length: 8
   0x0042b8c0  length: 8
   0x0042ba80  length: 128
   0x0042bb00  length: 128
   0x0042bc00  length: 128
   0x0042bc80  length: 128
   0x0042bb80  length: 128
   0x00400210  length: 8
   0x00400218  length: 8
   0x00400220  length: 8
   0x00400228  length: 8
  TRN CTL:: seq: 0x0f4c chd: 0x001b ctl: 0x0043 inc: 0x00000000 nfb: 0x0001
            mgc: 0x8002 xts: 0x0068 flg: 0x0001 opt: 2147483646 (0x7ffffffe)
            uba: 0x0042bb4c.0f4c.0c scn: 0x0b23.ccb9eb89
Version: 0x01
  FREE BLOCK POOL::
    uba: 0x0042bb4c.0f4c.0c ext: 0xd  spc: 0x132a
    uba: 0x00000000.0f4c.0c ext: 0xd  spc: 0x12f6
    uba: 0x00000000.0f4c.01 ext: 0xd  spc: 0x1ec8
    uba: 0x00000000.0f4c.04 ext: 0xd  spc: 0x1b86
    uba: 0x00000000.0f4c.09 ext: 0xd  spc: 0x162c
  TRN TBL::
  index  state cflags  wrap#    uel         scn            dba            parent-xid    nub     stmt_num
  ------------------------------------------------------------------------------------------------
   0x00    9    0x00  0x0b62  0x0011  0x0b25.2dacaf61  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x01    9    0x00  0x0b64  0x0024  0x0b24.a6a2cf7b  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x02    9    0x00  0x0b65  0x0036  0x0b25.7391eda0  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x03    9    0x00  0x0b4f  0x0007  0x0b24.337bf49b  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x04    9    0x00  0x0b64  0x0051  0x0b23.ff22c637  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x05    9    0x00  0x0b64  0x0022  0x0b26.4393eb1e  0x0042bb4c  0x0000.000.00000000  0x00000001   0x00000000
   0x06    9    0x00  0x0b66  0x0058  0x0b24.335c794d  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x07    9    0x00  0x0b4f  0x001d  0x0b24.4e05f2af  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x08    9    0x00  0x0b65  0x005e  0x0b23.ff22c618  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x09    9    0x00  0x0b5f  0x0035  0x0b24.337bf3d9  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x0a    9    0x00  0x0b64  0x004f  0x0b25.7391ee5f  0x0042bb4c  0x0000.000.00000000  0x00000001   0x00000000
   0x0b    9    0x00  0x0b64  0x0040  0x0b24.335c7bd7  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x0c    9    0x00  0x0b65  0x0002  0x0b25.7391e929  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x0d    9    0x00  0x0b4f  0x0033  0x0b24.a6a2caa5  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x0e    9    0x00  0x0b65  0x0008  0x0b23.ff22c616  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x0f    9    0x00  0x0b61  0x0038  0x0b26.6389e195  0x0042bb4c  0x0000.000.00000000  0x00000001   0x00000000
   0x10    9    0x00  0x0b4f  0x002a  0x0b24.bcff18b3  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x11    9    0x00  0x0b5c  0x0059  0x0b25.2dacaf69  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x12    9    0x00  0x0b65  0x0026  0x0b25.2dacb0a8  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x13    9    0x00  0x0b66  0x0021  0x0b24.a6a2caaa  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x14    9    0x00  0x0b62  0x0009  0x0b24.337bf3d7  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x15    9    0x00  0x0b63  0x0031  0x0b25.1b4e13ba  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x16    9    0x00  0x0b66  0x003b  0x0b25.2dacee5d  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x17    9    0x00  0x0b63  0x0034  0x0b26.6389e199  0x0042bb4c  0x0000.000.00000000  0x00000001   0x00000000
   0x18    9    0x00  0x0b5d  0x002f  0x0b24.bcff18af  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x19    9    0x00  0x0b5b  0x004d  0x0b24.d60f78e3  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x1a    9    0x00  0x0b60  0x005b  0x0b25.1b4e13be  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x1b    9    0x00  0x0b62  0x003e  0x0b23.ccb9ebac  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x1c    9    0x00  0x0b65  0x000a  0x0b25.7391ee5c  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x1d    9    0x00  0x0b64  0x002c  0x0b24.4e05f2b1  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x1e    9    0x00  0x0b64  0x0045  0x0b24.33255ae9  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x1f    9    0x00  0x0b64  0x0015  0x0b25.1b4e13b5  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x20    9    0x00  0x0b63  0x0050  0x0b24.335c79a4  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x21    9    0x00  0x0b5d  0x0001  0x0b24.a6a2caf0  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x22    9    0x00  0x0b65  0x000f  0x0b26.4393eb20  0x0042bb4c  0x0000.000.00000000  0x00000001   0x00000000
   0x23    9    0x00  0x0b62  0x0042  0x0b24.337bf3d2  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x24    9    0x00  0x0b65  0x003c  0x0b24.a6a2d137  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x25    9    0x00  0x0b62  0x0020  0x0b24.335c795d  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x26    9    0x00  0x0b63  0x0052  0x0b25.2dacee48  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x27    9    0x00  0x0b4e  0x003a  0x0b25.7391ee58  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x28    9    0x00  0x0b65  0x0049  0x0b25.2dacb089  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x29    9    0x00  0x0b61  0x0030  0x0b24.bcff18bb  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x2a    9    0x00  0x0b65  0x0057  0x0b24.bcff18b5  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x2b    9    0x00  0x0b64  0x0054  0x0b25.2dacee55  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x2c    9    0x00  0x0b61  0x000d  0x0b24.4e05f2b3  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x2d    9    0x00  0x0b64  0x000e  0x0b23.ff22c611  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x2e    9    0x00  0x0b65  0x0053  0x0b25.7391e78a  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x2f    9    0x00  0x0b66  0x0010  0x0b24.bcff18b1  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x30    9    0x00  0x0b63  0x0019  0x0b24.d60f78e1  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x31    9    0x00  0x0b65  0x001a  0x0b25.1b4e13bc  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x32    9    0x00  0x0b65  0x0037  0x0b24.a6a2d369  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x33    9    0x00  0x0b4f  0x0013  0x0b24.a6a2caa7  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x34    9    0x00  0x0b63  0x0043  0x0b26.6389e19b  0x0042bb4c  0x0000.000.00000000  0x00000001   0x00000000
   0x35    9    0x00  0x0b65  0x0046  0x0b24.337bf409  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x36    9    0x00  0x0b52  0x0061  0x0b25.7391eda2  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x37    9    0x00  0x0b64  0x0018  0x0b24.bcff18ad  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x38    9    0x00  0x0b65  0x0017  0x0b26.6389e197  0x0042bb4c  0x0000.000.00000000  0x00000001   0x00000000
   0x39    9    0x00  0x0b62  0x0006  0x0b24.335c7947  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x3a    9    0x00  0x0b64  0x001c  0x0b25.7391ee5a  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x3b    9    0x00  0x0b4d  0x002e  0x0b25.7391e730  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x3c    9    0x00  0x0b65  0x0032  0x0b24.a6a2d144  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x3d    9    0x00  0x0b65  0x0016  0x0b25.2dacee59  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x3e    9    0x00  0x0b63  0x002d  0x0b23.ff22c60b  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x3f    9    0x00  0x0b63  0x005f  0x0b24.335c7b57  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x40    9    0x00  0x0b65  0x0044  0x0b24.335c7bd9  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x41    9    0x00  0x0b65  0x0029  0x0b24.bcff18b9  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x42    9    0x00  0x0b61  0x0014  0x0b24.337bf3d4  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x43    9    0x00  0x0b5b  0xffff  0x0b26.6389e19d  0x0042bb4c  0x0000.000.00000000  0x00000001   0x00000000
   0x44    9    0x00  0x0b4c  0x004b  0x0b24.335c7bdb  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x45    9    0x00  0x0b61  0x0039  0x0b24.335c7945  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x46    9    0x00  0x0b65  0x005a  0x0b24.337bf421  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x47    9    0x00  0x0b65  0x0027  0x0b25.7391eda8  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x48    9    0x00  0x0b62  0x004e  0x0b24.335c7953  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x49    9    0x00  0x0b63  0x0012  0x0b25.2dacb09f  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x4a    9    0x00  0x0b63  0x0023  0x0b24.335c7bf8  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x4b    9    0x00  0x0b5b  0x004a  0x0b24.335c7bf3  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x4c    9    0x00  0x0b63  0x003f  0x0b24.335c7b55  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x4d    9    0x00  0x0b61  0x001f  0x0b25.1b4e13b3  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x4e    9    0x00  0x0b5a  0x0025  0x0b24.335c795b  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x4f    9    0x00  0x0b5f  0x005c  0x0b25.8ff588bb  0x0042bb4c  0x0000.000.00000000  0x00000001   0x00000000
   0x50    9    0x00  0x0b63  0x004c  0x0b24.335c79a7  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x51    9    0x00  0x0b64  0x005d  0x0b24.0c84cbf4  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x52    9    0x00  0x0b65  0x002b  0x0b25.2dacee49  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x53    9    0x00  0x0b64  0x000c  0x0b25.7391e927  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x54    9    0x00  0x0b63  0x003d  0x0b25.2dacee56  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x55    9    0x00  0x0b64  0x0005  0x0b26.4393eada  0x0042bb4c  0x0000.000.00000000  0x00000001   0x00000000
   0x56    9    0x00  0x0b64  0x0055  0x0b26.4393ead3  0x0042bb4c  0x0000.000.00000000  0x00000001   0x00000000
   0x57    9    0x00  0x0b60  0x0041  0x0b24.bcff18b7  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x58    9    0x00  0x0b65  0x0060  0x0b24.335c794f  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x59    9    0x00  0x0b60  0x0028  0x0b25.2dacaf74  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x5a    9    0x00  0x0b61  0x0003  0x0b24.337bf499  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
   0x5b    9    0x00  0x0b62  0x0000  0x0b25.1b4e13c0  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x5c    9    0x00  0x0b62  0x0056  0x0b25.950a6e4b  0x0042bb4c  0x0000.000.00000000  0x00000001   0x00000000
   0x5d    9    0x00  0x0b63  0x001e  0x0b24.33255ae7  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x5e    9    0x00  0x0b5f  0x0004  0x0b23.ff22c635  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x5f    9    0x00  0x0b64  0x000b  0x0b24.335c7bd5  0x0042bb49  0x0000.000.00000000  0x00000001   0x00000000
   0x60    9    0x00  0x0b61  0x0048  0x0b24.335c7950  0x0042bb4b  0x0000.000.00000000  0x00000001   0x00000000
   0x61    9    0x00  0x0b65  0x0047  0x0b25.7391eda6  0x0042bb4a  0x0000.000.00000000  0x00000001   0x00000000
KQRCMT: Write failed with error=600 po=0xa47e092c0 cid=3
diagnostics : cid=3 hash=35e74caf flag=2a
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []

到这里我们基本上明白了报错的file 1 block 179020是由FREE BLOCK POOL分配出来的,现在解决给问题的思路就是直接使用bbed分配一个新块即可.

ORA-600 6711
数据库无法正常open报ORA 600 6711错误

Thu Jul  14 04:04:28 2016
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 32 processes
Started redo scan
Completed redo scan
 read 1 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 2, block 3
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: /data/amdu/redo/DATA_EC_260.f
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 2, block 5, scn 12269633687653
 0 data blocks read, 0 data blocks written, 1 redo k-bytes read
Thu Jul  14 04:04:29 2016
Thread 1 advanced to log sequence 3 (thread open)
Thread 1 opened at log sequence 3
  Current log# 3 seq# 3 mem# 0: /data/amdu/redo/DATA_EC_263.f
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
After successful startup of the database, please remove
the parameters _allow_error_simulation and _smu_debug_mode
and restart the database
Thu Jul  14 04:04:29 2016
SMON: enabling cache recovery
Undo initialization finished serial:0 start:270626334 end:270626544 diff:210 (2 seconds)
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Corrected file 19 plugged in read-only status in control file
Corrected file 81 plugged in read-only status in control file
Corrected file 88 plugged in read-only status in control file
Corrected file 93 plugged in read-only status in control file
Corrected file 128 plugged in read-only status in control file
Corrected file 130 plugged in read-only status in control file
Corrected file 131 plugged in read-only status in control file
Corrected file 163 plugged in read-only status in control file
Corrected file 181 plugged in read-only status in control file
Corrected file 184 plugged in read-only status in control file
Corrected file 186 plugged in read-only status in control file
Corrected file 191 plugged in read-only status in control file
Corrected file 214 plugged in read-only status in control file
Corrected file 220 plugged in read-only status in control file
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
         ALTER TABLESPACE &lt;tablespace_name&gt; ADD TEMPFILE
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Updating character set in controlfile to ZHS16GBK
WARNING: event 8105 is set. This event disables failed
         online index [re]build cleanup
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Thu Jul  14 04:04:30 2016
QMNC started with pid=57, OS id=3549
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Errors in file /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_3401.trc  (incident=1440450):
ORA-00600: internal error code, arguments: [6711], [4293062], [1], [4318348], [0], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_1440450/xifenfei_ora_3401_i1440450.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Error 600 in kwqmnpartition(), aborting txn
Thu Jul  14 04:04:32 2016
Dumping diagnostic data in directory=[cdmp_20801214040432], requested by (instance=1, osid=3401), summary=[incident=1440450].
Thu Jul  14 04:04:32 2016
Errors in file /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_mmon_3329.trc  (incident=1440178):
ORA-00600: internal error code, arguments: [6711], [4293062], [1], [4318348], [0], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_1440178/xifenfei_mmon_3329_i1440178.trc
Errors in file /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_3401.trc  (incident=1440451):
ORA-00600: internal error code, arguments: [6711], [4293062], [1], [4318348], [0], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_1440451/xifenfei_ora_3401_i1440451.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_mmon_3329.trc  (incident=1440179):
ORA-00600: internal error code, arguments: [6711], [4293062], [1], [4318348], [0], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_1440179/xifenfei_mmon_3329_i1440179.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ORA-600 signalled during: alter database open...

数据库正常open失败,但是可以upgrade启动.根据对trace文件的分析,定位到问题在HISTGRM$表上面,进一步分析该表结构为

CREATE TABLE SYS.HISTGRM$
(
  OBJ#      NUMBER,
  COL#      NUMBER,
  ROW#      NUMBER,
  BUCKET    NUMBER,
  ENDPOINT  NUMBER,
  INTCOL#   NUMBER,
  EPVALUE   VARCHAR2(1000 BYTE),
  SPARE1    NUMBER,
  SPARE2    NUMBER
)
CLUSTER SYS.C_OBJ#_INTCOL#(OBJ#, INTCOL#);

这个和ORA-600 6711错误相匹配了

ERROR:
ORA-600 [6711] [a] [b]  [d]
VERSIONS:
versions 6.0 to 12.1
DESCRIPTION:
This error is generated when we find more blocks on a cluster key
chain than are supposed to be there.
Usually this indicates that the chain contains a loop within itself. We
cannot have more than 65535 blocks in a chain.
ARGUMENTS:
Arg [a] beginning DBA
Arg [b] table slot number (in table index)
Arg {c} dba of key next in chain
Arg [d] row slot of key next in chain

需要处理该错误,也就是需要处理CLUSTER SYS.C_OBJ#_INTCOL#,这个可以通过重建来实现,但是当重建之时发生

ORA-00600: internal error code, arguments: [kkoipt:invalid aptyp], [0], [0], [], [], [], [], [], [], [], [], []
ORA-08102: index key not found, obj# 39, file 1, block 1374829 (2)

这里错误比较明显obj#=39为obj$的i_obj4的index的记录和表不匹配,导致任何ddl无法执行,因此如果要处理C_OBJ#_INTCOL#就必须要先处理i_obj4的问题.通过一些技巧重建i_obj4,然后重建C_OBJ#_INTCOL#,数据库终于可以正常打开.由于大量数据字典不一致,exp/expdp导出依旧有问题,通过dblink直接拉数据到新库,完成本次恢复
补充说明：1. 在这个库的恢复过程中,我们还使用了大量的event和隐含参数,因为比较常规而且不涉及核心环节,因为未列举出来;2. 由于当时操作记录未能够保留日志因此相关操作步骤无法贴出来,本文只能提供恢复处理思路
再次提醒各位数据库做好备份,做好巡检工作,哪怕是强大的Oracle exadata也禁不起无备份折腾,数据重于一切