联系：手机/微信(+86 17813235971) QQ(107644445)
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
做一个oracle数据库被加密，所有数据文件名称全部被重命名,类似
E-Mail:dba@xifenfei.com
联系：手机/微信(+86 17813235971) QQ(107644445)
标题：How to resolve ORA-600 [4194] errors
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
在oracle恢复中ORA-600 4194是一个非常常见的错误,该错误的主要原因是由于redo记录和undo(rollback)记录不匹配.
ORA 600 4194错误原因以及含义
ERROR: Format: ORA-600 [4194] [a] [b] VERSIONS: versions 6.0 to 12.1 DESCRIPTION: A mismatch has been detected between Redo records and rollback (Undo) records. We are validating the Undo record number relating to the change being applied against the maximum undo record number recorded in the undo block. This error is reported when the validation fails. ARGUMENTS: Arg [a] Maximum Undo record number in Undo block Arg [b] Undo record number from Redo block
ORA 600 4194 错误处理思路
第一步
Confirm whether the database is up and running or not. If the database fails to start or crashes shortly
after startup due to this error occurring, then try setting event 10513 at level 2 in the init.ora/spfile
to disable transaction recovery and restart the instance, e.g.:
event = "10513 trace name context forever, level 2"
This may allow the database to successfully open and stay up so that
the required diagnostics/actions can be performed.
第二步
In the trace file there should be an undo segment header dump, and so check
to see if the undo segment header shows an active transaction after recovery, e.g.:
TRN TBL <---- Represents the Transaction table for the particular undo segment
index state cflags wrap# uel scn dba
---------------------------------------------------------------------------------------------
0x41 9 0x80 0x35ab6 0xffff 0x0695.38f6b959 0x1081e796
0x42 9 0x80 0x35bb1 0x000e 0x0695.38f6b028 0x1081e793
0x43 9 0x80 0x35b11 0x005d 0x0695.38f6b7ae 0x1081e795
0x44 9 0x80 0x359f0 0x0036 0x0695.38f69a91 0x1081e78e
0x45 10 0x80 0x35b1b 0x0000 0x0695.3a0aba4d 0x1081e796
0x46 9 0x80 0x35bb7 0x001c 0x0695.38f69bde 0x1081e78f
===================================
State ---> This column specifies the status of the transaction
9 -----> represents a commited transaction
10 ---> Represents a active transaction
Dba -----> Undo block containing the undo records
Strictly speaking this is the block at the end of the undo chain.
You can see from the transaction table that there is an active transaction
for this particular rollback/undo segment after recovery.
Therefore this rollback/undo segment and/or undo tablespace cannot be dropped without corrupting the database!
Therefore recreating the UNDO tablespace is not an option.
第三步
From the trace file determine the affected undo segment, e.g.: Block image after block recovery: UNDO BLK: xid: 0x0015.02b.0001544b seq: 0x163e cnt: 0x12 irb: 0x12 icl: 0x0 flg: 0x0000 XID ==> Undo segment no + Slot no + Sequence no Therefore, in this case the Undo Segment is: USN# 0x15 (Hex) ==> 21 (Dec) ==> _SYSSMU21$ So if and ONLY IF the transaction table shows no active transaction can the rollback/undo segment be offlined and dropped.Note however, that before you can confirm if the entire UNDO tablespace can be dropped, you would need to check the transaction tables of ALL active rollback/undo segments in the same wasy as the above. The steps required to drop the rollback/undo segment are fully detailed in Note:179952.1, but are briefly listed here for completeness: If using Automatic Undo Management Offline the undo segment using the _OFFLINE_ROLLBACK_SEGMENTS parameter and bounce the database as follows: 1. Create and edit the init.ora file for the instance to set the following parameters: UNDO_MANAGEMENT=MANUAL _OFFLINE_ROLLBACK_SEGMENTS=(_SYSSMU21$) 2. Open the database in restricted mode to prevent user access, e.g.: connect / as sysdba startup restrict pfile = '<Full path to init.ora file>'; 3. Drop the rollback/undo segment, e.g.: drop rollback segment "_SYSSMU21"; 4. Shutdown the instance, and remove the init.ora parameters added in point 1 and restart the instance, e.g.: shutdown immediate startup If SMON was recovering the transaction then this may not work as we cannot open the database if it is determined to be in an inconsistent state. I have reviewed a number of SRs where this approach was successful, so it is important to try it first but understand that it may fail and you will have to resort to a point in time recovery or forcing open the DB and recreating it.
第四步
Now we need to dump the undo block to see which object was affected.
We noted in Step 2 that this is the active transaction (from the trace file):
TRN TBL
index state cflags wrap# uel scn dba
0x45 10 0x80 0x35b1b 0x0000 0x0695.3a0aba4d 0x1081e796
Dba----------------> Undo block containing the undo records
dba--->0x1081e796 is the block containing the active transaction .
Use the WebIV tools to convert this RDBA to block number (block#) and file number (file#), e.g.:
V SPLIT ==> DBA (Hex) = File#,Block# (Hex File#,Block#)
= ===== === ===== ============
V8 10,10 ==> 276948886 (0x1081e796) = 66,124822 (0x42 0x1e796)
So the file# is 66 and the block# is 124822, so dump the block by issuing:
SQL> Alter system dump datafile 66 block 124822;
This will generate a trace file in the user_dump_dest. The following is a sample of the information in the undo block:
UNDO BLK:
xid: 0x000c.045.00035b1b seq: 0x1e14 cnt: 0x17 irb: 0x17 icl: 0x0 flg: 0x0000
Rec Offset Rec Offset Rec Offset Rec Offset Rec Offset
---------------------------------------------------------------------------
0x01 0x1f8c 0x02 0x1f30 0x03 0x1ed4 0x04 0x1e78 0x05 0x1e1c
0x06 0x1dc0 0x07 0x1d64 0x08 0x1d08 0x09 0x1cac 0x0a 0x1c50
0x0b 0x1bf4 0x0c 0x1b98 0x0d 0x1b3c 0x0e 0x1ae0 0x0f 0x1a74
0x10 0x1a18 0x11 0x19bc 0x12 0x1960 0x13 0x1904 0x14 0x187c
0x15 0x181c 0x16 0x1798 0x17 0x173c
* Rec #0x16 slt: 0x45 objn: 1485619(0x0016ab33) objd: 1485619 tblspc: 71(0x00000047)
* Layer: 11 (Row) opc: 1 rci 0x00
Undo type: Regular undo Begin trans Last buffer split: No
Temp Object: No
Tablespace Undo: No
rdba: 0x00000000
*-----------------------------
uba: 0x1081e796.1e14.14 ctl max scn: 0x0695.38f69853 prv tx scn: 0x0695.38f698a1
KDO undo record:
KTB Redo
op: 0x04 ver: 0x01
op: L itl: scn: 0x0019.009.00034237 uba: 0x36c0cce4.1d2f.19
flg: C--- lkc: 0 scn: 0x0695.38f6b96b
KDO Op code: URP xtype: XA bdba: 0x35406893 hdba: 0x35406892
itli: 1 ispac: 0 maxfr: 4863
tabn: 0 slot: 0(0x0) flag: 0x2c lock: 0 ckix: 0
ncol: 1 nnew: 1 size: -1
col 0: [ 4] c3 0e 36 2e
*-----------------------------
* Rec #0x17 slt: 0x45 objn: 1485619(0x0016ab33) objd: 1485619 tblspc: 71(0x00000047)
* Layer: 11 (Row) opc: 1 rci 0x16
Undo type: Regular undo Last buffer split: No
Temp Object: No
Tablespace Undo: No
rdba: 0x00000000
*-----------------------------
From the trace file above:
UNDO BLK:
xid: 0x000c.045.00035b1b seq: 0x1e14 cnt: 0x17 irb: 0x17 icl: 0x0 flg: 0x0000
The undo segment with the active transaction is segment is 0x000c (Hex) which is 12 (Dec) as the XID is:
Undo segment no + Slot no + Sequence no
This step is often skipped because it was performed earlier in step 3, however it is a good idea to do this
again now to make sure that the XID from the UNDO block matches the UNDO SEGMENT HEADER,
this way you have followed all the chain, from the UNDO SEGMENT to UNDO BLOCK, back and forth.
If there is a conflict here please check and make sure that the customer dumped the correct undo block.
Check for the value of irb which is an index which points you to the latest change done to the undo block.
This is the point from which a rollback would begin if one was issued.
From the trace file we see: 'irb: 0x17' so this points to record 0x17,
so search for this particular string i.e 0x17 and it will take you to undo record 'REC #0x17', e.g.:
* Rec #0x17 slt: 0x45 objn: 1485619(0x0016ab33) objd: 1485619 tblspc: 71(0x00000047)
* Layer: 11 (Row) opc: 1 rci 0x16
Undo type: Regular undo Last buffer split: No
Temp Object: No
Tablespace Undo: No
rdba: 0x00000000
*-----------------------------
Note the slot number (slt) is 0x45, the object number (objn) is the OBJECT_ID from dba_objects
and data object number (objd) is the DATA_OBJECT_ID from dba_objects.
These numbers may be the same but not necessarily, and so if the database is open then identify this object, e.g.:
select object_name, owner, object_type, data_object_id from dba_objects where object_id = <objn>;
This is the object, which has an active transaction. Note in the above trace file extract that rci
has a value of 0x16 which means that this record is at the end of an undo chain.
This means that the chain continues in another UNDO BLOCK.
Please refer to unpublished Note:281504.1 for information on Undo chains.
So the next record that needs to be rolled back is present in REC #X016.
If rci is 0x00 then it means that this is the first record present in the undo chain
and so you can check to see if there is rdba info, e.g.:
* Rec #0x16 slt: 0x45 objn: 1485619(0x0016ab33) objd: 1485619 tblspc: 71(0x00000047)
* Layer: 11 (Row) opc: 1 rci 0x00
Undo type: Regular undo Begin trans Last buffer split: No
Temp Object: No
Tablespace Undo: No
rdba: 0x00000000
*-----------------------------
uba: 0x1081e796.1e14.14 ctl max scn: 0x0695.38f69853 prv tx scn: 0x0695.38f698a1
KDO undo record:
KTB Redo
op: 0x04 ver: 0x01
op: L itl: scn: 0x0019.009.00034237 uba: 0x36c0cce4.1d2f.19
flg: C--- lkc: 0 scn: 0x0695.38f6b96b
KDO Op code: URP xtype: XA bdba: 0x35406893 hdba: 0x35406892
itli: 1 ispac: 0 maxfr: 4863
tabn: 0 slot: 0(0x0) flag: 0x2c lock: 0 ckix: 0
ncol: 1 nnew: 1 size: -1
col 0: [ 4] c3 0e 36 2e
*-----------------------------
If the object is an Index, drop and recreate it. If it is a table,
then again the table would need to be dropped and recreated (or truncated)
so that its object number changes and hence the rollback/undo is no longer required.
If this isn't possible, then you have two options:
First take a backup of the database in its current state.
This is critical in case anything goes wrong and you lose the opportunity to salvage the data!
Option 1
- Restore the undo segment datafile and the datafile containing the object and perform a full recovery.
This can only be done if you have all the archived redo as you will need to do full recovery on these files.
OR
Option 2
If option 1 is not possible, you can use the unsupported method, e.g.:
Specify the undo segment in the _OFFLINE_ROLLBACK_SEGMENTS parameter and try to drop the rollback segment.
If there is an active transaction then this is not likely to work and you will probably need
to set the _CORRUPTED_ROLLBACK_SEGMENTS parameter as well
温馨提示：
1.隐含参数_OFFLINE_ROLLBACK_SEGMENTS/_CORRUPTED_ROLLBACK_SEGMENTS属于Oracle内部隐含参数,建议在Oracle support认可的情况下使用,因为使用之后可能导致数据库事务完整性彻底损坏
2.进行屏蔽事务之前,如果条件允许建议使用txchecker检查
2.使用上述方法恢复数据库之后，建议通过逻辑方式导出导入重建数据库
联系：手机/微信(+86 17813235971) QQ(107644445)
标题：aix平台 ORA-01115 ORA-01110 ORA-27067 故障恢复
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
接到朋友恢复请求，aix 5.3,Oracle 10.2.0.1平台，数据库启动报ORA-01115 ORA-01110 ORA-27067错误，数据库无法正常打开，通过分析，是由于10201在aix上面的bug导致，通过技巧规避，完美解决给问题，数据0丢失
数据库报错alert日志
Mon Aug 10 13:25:22 2015 ALTER DATABASE MOUNT Mon Aug 10 13:25:29 2015 Setting recovery target incarnation to 1 Mon Aug 10 13:25:29 2015 Successful mount of redo thread 1, with mount id 432339141 Mon Aug 10 13:25:29 2015 Database mounted in Exclusive Mode Completed: ALTER DATABASE MOUNT Mon Aug 10 13:25:36 2015 alter database open Mon Aug 10 13:25:36 2015 Beginning crash recovery of 1 threads parallel recovery started with 15 processes Mon Aug 10 13:25:37 2015 Started redo scan Mon Aug 10 13:25:52 2015 Completed redo scan 7889582 redo blocks read, 75305 data blocks need recovery Mon Aug 10 13:25:53 2015 Errors in file /dc/admin/datacent/bdump/datacent_p002_144124.trc: ORA-01115: IO error reading block from file 2 (block # 40704) ORA-01110: data file 2: '/dc/oradata/datacent/undotbs01.dbf' ORA-27067: size of I/O buffer is invalid Additional information: 2 Additional information: 1572864 Mon Aug 10 13:25:53 2015 Aborting crash recovery due to slave death, attempting serial crash recovery Mon Aug 10 13:25:53 2015 Beginning crash recovery of 1 threads Mon Aug 10 13:25:53 2015 Started redo scan Mon Aug 10 13:26:09 2015 Completed redo scan 7889582 redo blocks read, 75305 data blocks need recovery Mon Aug 10 13:26:12 2015 Aborting crash recovery due to error 1115 Mon Aug 10 13:26:12 2015 Errors in file /dc/admin/datacent/udump/datacent_ora_123384.trc: ORA-01115: IO error reading block from file 2 (block # 39077) ORA-01110: data file 2: '/dc/oradata/datacent/undotbs01.dbf' ORA-27067: size of I/O buffer is invalid Additional information: 2 Additional information: 1310720 ORA-1115 signalled during: alter database open...
这里报的前面两个错误ORA-01115 ORA-01110我们都非常熟悉，类似数据库启动遇到坏块或者io错误之时可能就会报如此错误。但是ORA-27067确实不多见，从mos上看，很多是由于rman备份之时的bug可能导致该错误。
dbv检测undo坏块文件
DBVERIFY: Release 10.2.0.1.0 - Production on Mon Aug 10 23:18:15 2015 Copyright (c) 1982, 2003, Oracle and/or its affiliates. All rights reserved. DBVERIFY - Verification starting : FILE = /dc/oradata/datacent/undotbs01.dbf DBVERIFY - Verification complete Total Pages Examined : 329600 Total Pages Processed (Data) : 0 Total Pages Failing (Data) : 0 Total Pages Processed (Index): 0 Total Pages Failing (Index): 0 Total Pages Processed (Other): 327504 Total Pages Processed (Seg) : 17 Total Pages Failing (Seg) : 0 Total Pages Empty : 2096 Total Pages Marked Corrupt : 0 Total Pages Influx : 0 Total Pages Encrypted : 0 Highest block SCN : 1887888 (0.1887888)
这里可以看到，undo文件本身并没有逻辑和物理的坏块,证明因为数据库异常的原因，可能是由于ORA-27067: size of I/O buffer is invalid导致。根据官方文档ORA-01115 ORA-27067 DURING PARALLEL INSTANCE RECOVERY AFTER INSTANCE CRASH中的解释，我们基本上可以确定很可能是由于10.2.0.1在aix平台的jfs2系统中，由于大量事务操作，突然abort掉数据库（也可能断电），从而数据库在启动的时候进行实例恢复，而由于内部的bug，导致实例恢复无法成功。通过我们处理后的，数据库完美启动，数据0丢失
数据库启动日志
Mon Aug 10 16:34:14 2015 alter database open Mon Aug 10 16:34:14 2015 Beginning crash recovery of 1 threads parallel recovery started with 15 processes Mon Aug 10 16:34:14 2015 Started redo scan Mon Aug 10 16:34:27 2015 Completed redo scan 7889582 redo blocks read, 0 data blocks need recovery Mon Aug 10 16:34:27 2015 Started redo application at Thread 1: logseq 664704, block 1286922 Mon Aug 10 16:34:27 2015 Recovery of Online Redo Log: Thread 1 Group 4 Seq 664704 Reading mem 0 Mem# 0 errs 0: /dev/rredo04 Mon Aug 10 16:34:32 2015 Recovery of Online Redo Log: Thread 1 Group 5 Seq 664705 Reading mem 0 Mem# 0 errs 0: /dev/rredo05 Mon Aug 10 16:34:38 2015 Recovery of Online Redo Log: Thread 1 Group 6 Seq 664706 Reading mem 0 Mem# 0 errs 0: /dev/rredo06 Mon Aug 10 16:34:40 2015 Completed redo application Mon Aug 10 16:34:40 2015 Completed crash recovery at Thread 1: logseq 664706, block 1017805, scn 8554793334 0 data blocks read, 0 data blocks written, 7889582 redo blocks read Mon Aug 10 16:34:40 2015 Thread 1 advanced to log sequence 664707 Thread 1 opened at log sequence 664707 Current log# 1 seq# 664707 mem# 0: /dev/rredo01 Successful open of redo thread 1 Mon Aug 10 16:34:40 2015 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Mon Aug 10 16:34:40 2015 SMON: enabling cache recovery Mon Aug 10 16:34:40 2015 Successfully onlined Undo Tablespace 1. Mon Aug 10 16:34:40 2015 SMON: enabling tx recovery Mon Aug 10 16:34:41 2015 Database Characterset is ZHS32GB18030 replication_dependency_tracking turned off (no async multimaster replication found) WARNING: AQ_TM_PROCESSES is set to 0. System operation might be adversely affected. Mon Aug 10 16:34:41 2015 SMON: Parallel transaction recovery tried Mon Aug 10 16:34:42 2015 db_recovery_file_dest_size of 2048 MB is 0.00% used. This is a user-specified limit on the amount of space that will be used by this database for recovery-related files, and does not reflect the amount of space available in the underlying filesystem or ASM diskgroup. Mon Aug 10 16:34:42 2015 Completed: alter database open
联系：手机/微信(+86 17813235971) QQ(107644445)
标题：分享一例由于主库逻辑坏块导致dataguard容灾失效
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
有朋友数据库配置了物理dataguard，但是由于触发了某种Oracle bug/或者其他原因导致block出现大量逻辑坏块，结果是主备库(主库启动后就crash,备库failover后也启动后就crash)都无法正常工作,请求给予技术支持。
数据库配置了物理dataguard，但是主库在进行了某些操作之后，主库直接crash.重启主库发现，数据库启动之后，稍后数据库继续CRASH
Wed Jul 08 16:32:22 2015 Thread 1 advanced to log sequence 401531 (LGWR switch) Current log# 6 seq# 401531 mem# 0: /opt/oracle/database/fast_recovery_area/xifenfei/onlinelog/o1_mf_6_b9p7qhrm_.log Archived Log entry 605194 added for thread 1 sequence 401530 ID 0xfbe74bc9 dest 1: ARC3: Standby redo logfile selected for thread 1 sequence 401530 for destination LOG_ARCHIVE_DEST_2 ****************************************************************** LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2 ****************************************************************** Wed Jul 08 16:33:02 2015 Errors in file /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_p032_413611.trc (incident=201910): ORA-00600: internal error code, arguments: [17114], [0x7FB03BB6DBD0], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7FB03BB6DBF8], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/incident/incdir_201910/xifenfei_p032_413611_i201910.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_p032_413611.trc (incident=201911): ORA-00600: internal error code, arguments: [kghfrempty:ds], [0x7FB03BB6DBE8], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17114], [0x7FB03BB6DBD0], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7FB03BB6DBF8], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/incident/incdir_201911/xifenfei_p032_413611_i201911.trc Wed Jul 08 16:34:23 2015 SMON: slave died unexpectedly, downgrading to serial recovery Errors in file /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_smon_413207.trc (incident=201547): ORA-00600: internal error code, arguments: [17182], [0x7F24D680D7A0], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/incident/incdir_201547/xifenfei_smon_413207_i201547.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Block recovery from logseq 401530, block 119 to scn 73226510040 Recovery of Online Redo Log: Thread 1 Group 5 Seq 401530 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/xifenfei/onlinelog/o1_mf_5_b9p7qhl0_.log Recovery of Online Redo Log: Thread 1 Group 6 Seq 401531 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/xifenfei/onlinelog/o1_mf_6_b9p7qhrm_.log Block recovery completed at rba 401531.1882.16, scn 17.212066009 ORACLE Instance xifenfei (pid = 16) - Error 600 encountered while recovering transaction (13, 14) on object 135520. Errors in file /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_smon_413207.trc: ORA-00600: internal error code, arguments: [17182], [0x7F24D680D7A0], [], [], [], [], [], [], [], [], [], [] Wed Jul 08 16:34:24 2015 Dumping diagnostic data in directory=[cdmp_20150708163424], requested by (instance=1, osid=413207 (SMON)), summary=[incident=201547]. Errors in file /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_smon_413207.trc (incident=201548): ORA-00600: internal error code, arguments: [KSMFPG2], [0x7F24D680D000], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7F24D680D7A0], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/incident/incdir_201548/xifenfei_smon_413207_i201548.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Fatal internal error happened while SMON was doing active transaction recovery. Errors in file /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_smon_413207.trc: ORA-00600: internal error code, arguments: [KSMFPG2], [0x7F24D680D000], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7F24D680D7A0], [], [], [], [], [], [], [], [], [], [] SMON (ospid: 413207): terminating the instance due to error 474 System state dump requested by (instance=1, osid=413207 (SMON)), summary=[abnormal instance termination]. System State dumped to trace file /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_diag_413167_20150708163426.trc Dumping diagnostic data in directory=[cdmp_20150708163426], requested by (instance=1, osid=413207 (SMON)), summary=[abnormal instance termination]. Instance terminated by SMON, pid = 413207
由于主库不能正常open，备库直接failover方式激活
Wed Jul 08 17:56:41 2015 alter database recover managed standby database finish Terminal Recovery: request posted (xffdb) Wed Jul 08 17:56:45 2015 Begin: Standby Redo Logfile archival End: Standby Redo Logfile archival Terminal Recovery timestamp is '07/08/2015 17:56:45' Terminal Recovery: applying standby redo logs. Terminal Recovery: thread 1 seq# 401533 redo required Terminal Recovery: Recovery of Online Redo Log: Thread 1 Group 7 Seq 401533 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_7_bb7x31lk_.log Identified End-Of-Redo (failover) for thread 1 sequence 401533 at SCN 0xffff.ffffffff Incomplete Recovery applied until change 73226530800 time 07/08/2015 16:56:40 Terminal Recovery: successful completion Wed Jul 08 17:56:45 2015 ARCH: Archival stopped, error occurred. Will continue retrying Forcing ARSCN to IRSCN for TR 17:212086768 ORACLE Instance xffdb - Archival Error Attempt to set limbo arscn 17:212086768 irscn 17:212086768 ORA-16014: log 7 sequence# 401533 not archived, no available destinations ORA-00312: online log 7 thread 1: '/opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_7_bb7x31lk_.log' Resetting standby activation ID 4226239433 (0xfbe74bc9) Wed Jul 08 17:56:45 2015 MRP0: Media Recovery Complete (xffdb) MRP0: Background Media Recovery process shutdown (xffdb) Terminal Recovery: completion detected (xffdb) Completed: alter database recover managed standby database finish Wed Jul 08 17:56:58 2015 alter database commit to switchover to primary ALTER DATABASE SWITCHOVER TO PRIMARY (xffdb) Maximum wait for role transition is 15 minutes. Backup controlfile written to trace file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_ora_485094.trc Standby terminal recovery start SCN: 73226530482 RESETLOGS after incomplete recovery UNTIL CHANGE 73226530800 Online logfile pre-clearing operation disabled by switchover Online log /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_1_bb7x30gw_.log: Thread 1 Group 1 was previously cleared Online log /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_2_bb7x30js_.log: Thread 1 Group 2 was previously cleared Online log /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_3_bb7x310q_.log: Thread 1 Group 3 was previously cleared Online log /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_4_bb7x312r_.log: Thread 1 Group 4 was previously cleared Online log /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_5_bb7x317f_.log: Thread 1 Group 5 was previously cleared Online log /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_6_bb7x31cz_.log: Thread 1 Group 6 was previously cleared Standby became primary SCN: 73226530481 Wed Jul 08 17:56:58 2015 Setting recovery target incarnation to 3 Switchover: Complete - Database mounted as primary Completed: alter database commit to switchover to primary
数据库激活成功后，重启激活之后数据库发现和主库出现类似情况
Wed Jul 08 17:57:25 2015 ALTER DATABASE MOUNT Successful mount of redo thread 1, with mount id 4243462021 Database mounted in Exclusive Mode Lost write protection disabled Completed: ALTER DATABASE MOUNT Wed Jul 08 17:57:30 2015 ALTER DATABASE OPEN Assigning activation ID 4243462021 (0xfcee1785) LGWR: STARTING ARCH PROCESSES Wed Jul 08 17:57:30 2015 ARC0 started with pid=23, OS id=485230 ARC0: Archival started LGWR: STARTING ARCH PROCESSES COMPLETE ARC0: STARTING ARCH PROCESSES Thread 1 advanced to log sequence 2 (thread open) Wed Jul 08 17:57:31 2015 ARC1 started with pid=24, OS id=485236 Wed Jul 08 17:57:31 2015 ARC2 started with pid=25, OS id=485240 Wed Jul 08 17:57:31 2015 ARC3 started with pid=26, OS id=485244 ARC1: Archival started ARC2: Archival started ARC1: Becoming the 'no FAL' ARCH ARC1: Becoming the 'no SRL' ARCH ARC2: Becoming the heartbeat ARCH Thread 1 opened at log sequence 2 Current log# 2 seq# 2 mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_2_bb7x30js_.log Successful open of redo thread 1 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set SMON: enabling cache recovery Wed Jul 08 17:57:31 2015 NSA2 started with pid=27, OS id=485248 [485226] Successfully onlined Undo Tablespace 2. Undo initialization finished serial:0 start:3453451748 end:3453452018 diff:270 (2 seconds) Dictionary check beginning Dictionary check complete Verifying file header compatibility for 11g tablespace encryption.. Verifying 11g file header compatibility for tablespace encryption completed SMON: enabling tx recovery Database Characterset is AL32UTF8 No Resource Manager plan active replication_dependency_tracking turned off (no async multimaster replication found) Wed Jul 08 17:57:32 2015 Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_p032_485380.trc (incident=64481): ORA-00600: internal error code, arguments: [17182], [0x7FE96B50DBF8], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xffdb/xffdb/incident/incdir_64481/xffdb_p032_485380_i64481.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Starting background process QMNC Wed Jul 08 17:57:32 2015 QMNC started with pid=92, OS id=485512 LOGSTDBY: Validating controlfile with logical metadata LOGSTDBY: Validation complete Dumping diagnostic data in directory=[cdmp_20150708175733], requested by (instance=1, osid=485380 (P032)), summary=[incident=64481]. Thread 1 advanced to log sequence 3 (LGWR switch) Current log# 3 seq# 3 mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_3_bb7x310q_.log ARC3: STARTING ARCH PROCESSES Wed Jul 08 17:57:34 2015 ARC4 started with pid=93, OS id=485516 Wed Jul 08 17:57:35 2015 db_recovery_file_dest_size of 204800 MB is 0.41% used. This is a user-specified limit on the amount of space that will be used by this database for recovery-related files, and does not reflect the amount of space available in the underlying filesystem or ASM diskgroup. ARC4: Archival started ARC3: STARTING ARCH PROCESSES COMPLETE krsk_srl_archive_int: Enabling archival of deferred physical standby SRLs Archived Log entry 273963 added for thread 1 sequence 2 ID 0xfcee1785 dest 1: Archived Log entry 273964 added for thread 1 sequence 401533 ID 0xfbe74bc9 dest 1: Completed: ALTER DATABASE OPEN Wed Jul 08 17:57:36 2015 Starting background process CJQ0 Wed Jul 08 17:57:36 2015 CJQ0 started with pid=95, OS id=485554 Shutting down archive processes ARCH shutting down ARC4: Archival stopped Wed Jul 08 17:57:41 2015 Thread 1 advanced to log sequence 4 (LGWR switch) Current log# 4 seq# 4 mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_4_bb7x312r_.log Wed Jul 08 17:57:41 2015 Archived Log entry 273965 added for thread 1 sequence 3 ID 0xfcee1785 dest 1: Wed Jul 08 17:58:30 2015 Sweep [inc][64481]: completed Sweep [inc2][64481]: completed Wed Jul 08 17:58:31 2015 Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_p032_485380.trc (incident=64482): ORA-00600: internal error code, arguments: [17114], [0x7FE96B50DBD0], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7FE96B50DBF8], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xffdb/xffdb/incident/incdir_64482/xffdb_p032_485380_i64482.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Wed Jul 08 17:58:32 2015 Dumping diagnostic data in directory=[cdmp_20150708175832], requested by (instance=1, osid=485380 (P032)), summary=[incident=64482]. Wed Jul 08 17:58:36 2015 Thread 1 advanced to log sequence 5 (LGWR switch) Current log# 5 seq# 5 mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_5_bb7x317f_.log Wed Jul 08 17:59:02 2015 Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_p032_485380.trc (incident=64483): ORA-00600: internal error code, arguments: [17114], [0x7FE96B50DBD0], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17114], [0x7FE96B50DBD0], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7FE96B50DBF8], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xffdb/xffdb/incident/incdir_64483/xffdb_p032_485380_i64483.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Wed Jul 08 17:59:04 2015 Dumping diagnostic data in directory=[cdmp_20150708175904], requested by (instance=1, osid=485380 (P032)), summary=[incident=64483]. Wed Jul 08 17:59:29 2015 Sweep [inc][64483]: completed Sweep [inc][64482]: completed Sweep [inc2][64483]: completed Sweep [inc2][64482]: completed Wed Jul 08 17:59:30 2015 Block recovery from logseq 2, block 104 to scn 73226531646 Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_2_bb7x30js_.log Recovery of Online Redo Log: Thread 1 Group 3 Seq 3 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_3_bb7x310q_.log Recovery of Online Redo Log: Thread 1 Group 4 Seq 4 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_4_bb7x312r_.log Recovery of Online Redo Log: Thread 1 Group 5 Seq 5 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_5_bb7x317f_.log Block recovery stopped at EOT rba 5.765.16 Block recovery completed at rba 5.765.16, scn 17.212087614 Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_p032_485380.trc (incident=64484): ORA-00600: internal error code, arguments: [KSMFPG2], [0x7FE96B50D000], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17114], [0x7FE96B50DBD0], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17114], [0x7FE96B50DBD0], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7FE96B50DBF8], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xffdb/xffdb/incident/incdir_64484/xffdb_p032_485380_i64484.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Dumping diagnostic data in directory=[cdmp_20150708175934], requested by (instance=1, osid=485380 (P032)), summary=[incident=64487]. Wed Jul 08 17:59:36 2015 SMON: slave died unexpectedly, downgrading to serial recovery Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_smon_485190.trc (incident=64129): ORA-00600: internal error code, arguments: [17182], [0x7F5AED10D7A0], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xffdb/xffdb/incident/incdir_64129/xffdb_smon_485190_i64129.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Block recovery from logseq 2, block 104 to scn 73226531646 Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_2_bb7x30js_.log Recovery of Online Redo Log: Thread 1 Group 3 Seq 3 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_3_bb7x310q_.log Recovery of Online Redo Log: Thread 1 Group 4 Seq 4 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_4_bb7x312r_.log Recovery of Online Redo Log: Thread 1 Group 5 Seq 5 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_5_bb7x317f_.log Block recovery completed at rba 5.765.16, scn 17.212087615 ORACLE Instance xffdb (pid = 16) - Error 600 encountered while recovering transaction (13, 14) on object 135520. Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_smon_485190.trc: ORA-00600: internal error code, arguments: [17182], [0x7F5AED10D7A0], [], [], [], [], [], [], [], [], [], [] Dumping diagnostic data in directory=[cdmp_20150708175937], requested by (instance=1, osid=485190 (SMON)), summary=[incident=64129]. Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_smon_485190.trc (incident=64130): ORA-00600: internal error code, arguments: [KSMFPG2], [0x7F5AED10D000], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7F5AED10D7A0], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xffdb/xffdb/incident/incdir_64130/xffdb_smon_485190_i64130.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Fatal internal error happened while SMON was doing active transaction recovery. Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_smon_485190.trc: ORA-00600: internal error code, arguments: [KSMFPG2], [0x7F5AED10D000], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7F5AED10D7A0], [], [], [], [], [], [], [], [], [], [] SMON (ospid: 485190): terminating the instance due to error 474 System state dump requested by (instance=1, osid=485190 (SMON)), summary=[abnormal instance termination]. System State dumped to trace file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_diag_485150_20150708175939.trc Dumping diagnostic data in directory=[cdmp_20150708175939], requested by (instance=1, osid=485190 (SMON)), summary=[abnormal instance termination]. Instance terminated by SMON, pid = 485190
通过进一步分析确定是67号文件有异常,使用dbv检查该文件发现
DBVERIFY: Release 11.2.0.4.0 - Production on Thu Jul 9 11:08:27 2015
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
DBVERIFY - Verification starting : FILE = /opt/oracle/database/oradata/xffdb/dk_adv_ts_main_25.dbf
Block Checking: DBA = 283035270, Block Type = KTB-managed data block
data header at 0x7f87813ea064
kdbchk: the amount of space used is not equal to block size
used=7383 fsc=0 avsp=873 dtl=8088
Page 2016902 failed with check code 6110
Block Checking: DBA = 283035271, Block Type = KTB-managed data block
data header at 0x7f87813ec064
kdbchk: the amount of space used is not equal to block size
used=8676 fsc=0 avsp=832 dtl=8088
Page 2016903 failed with check code 6110
Block Checking: DBA = 283035272, Block Type = KTB-managed data block
data header at 0x7f87813ee064
kdbchk: avsp(1306) > tosp(894)
Page 2016904 failed with check code 6128
Block Checking: DBA = 283035273, Block Type = KTB-managed data block
data header at 0x7f87813f0064
kdbchk: the amount of space used is not equal to block size
used=7506 fsc=0 avsp=815 dtl=8088
Page 2016905 failed with check code 6110
Block Checking: DBA = 283035274, Block Type = KTB-managed data block
data header at 0x7f87813f2064
kdbchk: the amount of space used is not equal to block size
used=7892 fsc=0 avsp=884 dtl=8088
Page 2016906 failed with check code 6110
…………
Block Checking: DBA = 283035368, Block Type = KTB-managed data block
data header at 0x7f87814ae064
kdbchk: the amount of space used is not equal to block size
used=7934 fsc=0 avsp=837 dtl=8088
Page 2017000 failed with check code 6110
Block Checking: DBA = 283035369, Block Type = KTB-managed data block
data header at 0x7f87814b0064
kdbchk: the amount of space used is not equal to block size
used=7683 fsc=0 avsp=883 dtl=8088
Page 2017001 failed with check code 6110
Block Checking: DBA = 283035370, Block Type = KTB-managed data block
data header at 0x7f87814b2064
kdbchk: the amount of space used is not equal to block size
used=8556 fsc=0 avsp=841 dtl=8088
Page 2017002 failed with check code 6110
Block Checking: DBA = 283035371, Block Type = KTB-managed data block
data header at 0x7f87814b4064
kdbchk: the amount of space used is not equal to block size
used=7460 fsc=0 avsp=822 dtl=8088
Page 2017003 failed with check code 6110
DBVERIFY - Verification complete
Total Pages Examined : 4063232
Total Pages Processed (Data) : 2724435
Total Pages Failing (Data) : 102
Total Pages Processed (Index): 1064839
Total Pages Failing (Index): 0
Total Pages Processed (Other): 273957
Total Pages Processed (Seg) : 0
Total Pages Failing (Seg) : 0
Total Pages Empty : 1
Total Pages Marked Corrupt : 0
Total Pages Influx : 0
Total Pages Encrypted : 0
Highest block SCN : 212065913 (17.212065913)
把datafile 67给offline之后，数据库open正常，也不再crash。这里比较明显，导致主库和被动都出现异常,主库直接crash，然后重启后，一会儿就crash；备库激活后，重启后一会儿也就crash；两个库现象相同。最后通过dbv定位到是由于某个文件出现大量逻辑坏块，导致数据库open之后，进行回滚之时crash.对于此类故障，可以通过屏蔽事务回滚，并且通过alert日志和trace文件定位到异常对象，可以删除异常对象可以暂时解决该问题。而导致数据库出现类似问题。
通过这个案例，可以的出来，由于oracle某种bug或者其他原因，导致block 逻辑损坏，而且这个损坏会传输到备库，导致备库也异常，oracle的备份，容灾不能全部依赖物理dataguard容灾。因此在条件允许情况下，建议增加物理备份和逻辑容灾（类似OGG)
联系：手机/微信(+86 17813235971) QQ(107644445)
标题：通过bbed修改obj$中dataobj$重现I_OBJ4索引报ORA-08102错误
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
在最近的数据库恢复中,经历了多次11.2库由于各种原因,数据库打开后,报ORA-8102错误,而且错误对象是OBJ$上的I_OBJ4这个index上,而且不能创建新表,周末开会闲着无事,进行了一个简单的模拟重现
数据库版本信息11.2.0.4
SQL> select * from v$version; BANNER -------------------------------------------------------------------------------- Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production PL/SQL Release 11.2.0.4.0 - Production CORE 11.2.0.4.0 Production TNS for Linux: Version 11.2.0.4.0 - Production NLSRTL Version 11.2.0.4.0 - Production
定位需要破坏的OBJ$上记录,为了使之和I_OBJ4中记录不一致,从而实现ORA-8102错误
SQL> select object_id,object_type from dba_objects where object_name='I_OBJ4';
OBJECT_ID OBJECT_TYPE
---------- -------------------
87404 INDEX
SQL> select max(DATAOBJ#) from obj$;
MAX(DATAOBJ#)
-------------
87420
SQL> select dump(87420,16) from dual;
DUMP(87420,16)
-----------------------
Typ=2 Len=4: c3,9,4b,15
SQL> select dbms_rowid.rowid_relative_fno(rowid) file#,dbms_rowid.rowid_block_number(rowid) block#,
dbms_rowid.rowid_row_number(rowid) row#
2 3 from obj$ where DATAOBJ#=87420;
FILE# BLOCK# ROW#
---------- ---------- ----------
1 98085 40
SQL> alter system dump datafile 1 block 98085;
System altered.
SQL> select value from v$diag_info where name='Default Trace File';
VALUE
--------------------------------------------------------------------------------
/u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_26373.trc
--dump该记录显示
tab 0, row 40, @0x11fc
tl: 72 fb: --H-FL-- lb: 0x0 cc: 18
col 0: [ 2] c1 02
col 1: [ 4] c3 09 4b 15
col 2: [ 1] 80
col 3: [12] 5f 4e 45 58 54 5f 4f 42 4a 45 43 54
col 4: [ 2] c1 02
col 5: *NULL*
col 6: [ 1] 80
col 7: [ 7] 78 71 08 18 0c 26 24
col 8: [ 7] 78 73 03 0d 15 2e 2b
col 9: [ 7] 78 71 08 18 0c 26 24
col 10: [ 1] 80
col 11: *NULL*
col 12: *NULL*
col 13: [ 1] 80
col 14: *NULL*
col 15: [ 1] 80
col 16: [ 4] c3 07 38 24
col 17: [ 1] 80
tab 0, row 41, @0x9af
tl: 2 fb: --HDFL-- lb: 0x2
这里我们知道i_obj4中的dataobj#最大值为87420对应的16进制记录为04 c3 09 4b 15
使用bbed破坏记录,修改dataobj#中的值,使得obj$.dataobj#和i_obj4中的dataobj#不匹配
SQL> select name from v$datafile where file#=1;
NAME
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/xifenfei/system01.dbf
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
[oracle@localhost ~]$ bbed blocksize=8192 mode=edit filename='/u01/app/oracle/oradata/xifenfei/system01.dbf'
Password:
BBED: Release 2.0.0.0.0 - Limited Production on Sat Mar 14 14:23:02 2015
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
************* !!! For Oracle Internal Use only !!! ***************
BBED> show all
FILE# 0
BLOCK# 1
OFFSET 0
DBA 0x00000000 (0 0,1)
FILENAME /u01/app/oracle/oradata/xifenfei/system01.dbf
BIFILE bifile.bbd
LISTFILE
BLOCKSIZE 8192
MODE Edit
EDIT Unrecoverable
IBASE Dec
OBASE Dec
WIDTH 80
COUNT 512
LOGFILE log.bbd
SPOOL No
BBED> set block 98085
BLOCK# 98085
BBED> p *kdbr[40]
rowdata[2446]
-------------
ub1 rowdata[2446] @4696 0x2c
BBED> x /rnnncnnncc
rowdata[2446] @4696
-------------
flag@4696: 0x2c (KDRHFL, KDRHFF, KDRHFH)
lock@4697: 0x00
cols@4698: 18
col 0[2] @4699: 1
col 1[4] @4702: 87420
col 2[1] @4707: 0
col 3[12] @4709: _NEXT_OBJECT
col 4[2] @4722: 1
col 5[0] @4725: *NULL*
col 6[1] @4726: 0
col 7[7] @4728: xq...&$
col 8[7] @4736: xs....+
col 9[7] @4744: xq...&$
col 10[1] @4752: .
col 11[0] @4754: *NULL*
col 12[0] @4755: *NULL*
col 13[1] @4756: .
col 14[0] @4758: *NULL*
col 15[1] @4759: .
col 16[4] @4761: Ã.8$
col 17[1] @4766: .
BBED> set block 98085
BLOCK# 98085
BBED> set offset 4702
OFFSET 4702
BBED> set count 32
COUNT 32
BBED> d
File: /u01/app/oracle/oradata/xifenfei/system01.dbf (0)
Block: 98085 Offsets: 4702 to 4733 Dba:0x00000000
------------------------------------------------------------------------
04c3094b 1501800c 5f4e4558 545f4f42 4a454354 02c102ff 01800778 7108180c
<32 bytes per line>
BBED> set offset +4
OFFSET 4706
BBED> d
File: /u01/app/oracle/oradata/xifenfei/system01.dbf (0)
Block: 98085 Offsets: 4706 to 4737 Dba:0x00000000
------------------------------------------------------------------------
1501800c 5f4e4558 545f4f42 4a454354 02c102ff 01800778 7108180c 26240778
<32 bytes per line>
BBED> m /x 17
File: /u01/app/oracle/oradata/xifenfei/system01.dbf (0)
Block: 98085 Offsets: 4706 to 4737 Dba:0x00000000
------------------------------------------------------------------------
1701800c 5f4e4558 545f4f42 4a454354 02c102ff 01800778 7108180c 26240778
<32 bytes per line>
BBED> sum apply
Check value for File 0, Block 98085:
current = 0xd361, required = 0xd361
BBED> verify
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/xifenfei/system01.dbf
BLOCK = 98085
DBVERIFY - Verification complete
Total Blocks Examined : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing (Index): 0
Total Blocks Empty : 0
Total Blocks Marked Corrupt : 0
Total Blocks Influx : 0
Message 531 not found; product=RDBMS; facility=BBED
使用bbed 修改04 c3 09 4b 15为04 c3 09 4b 17
重现在obj$的I_OBJ4 index上报ORA-8102错误,而且不能创建新对象
SQL> startup
ORACLE instance started.
Total System Global Area 1570009088 bytes
Fixed Size 2253584 bytes
Variable Size 469765360 bytes
Database Buffers 1090519040 bytes
Redo Buffers 7471104 bytes
Database mounted.
Database opened.
SQL> create table t1 as select * from dba_users;
create table t1 as select * from dba_users
*
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-08102: index key not found, obj# 87404, file 1, block 97266 (2)
SQL> col OBJECT_NAME for a30
SQL> select object_name,object_type from dba_objects where object_id=87404;
OBJECT_NAME OBJECT_TYPE
------------------------------ -------------------
I_OBJ4 INDEX
如果修复该问题请见:使用bbed 修复I_OBJ4 index 报ORA-8102
联系：手机/微信(+86 17813235971) QQ(107644445)
标题：记录一次由于坏块和不恰当恢复引起各种ORA-600案例
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
朋友让我帮忙处理一个不能open的库,打开alert日志一看,傻眼了,里面是各种ORA-600的错误应有尽有,被折腾的够惨
故障后重启,无法启动主要表现在block坏块,引起的各种ORA-600等错误
Mon Mar 02 16:09:27 2015 ALTER DATABASE OPEN Beginning crash recovery of 1 threads parallel recovery started with 23 processes Started redo scan Completed redo scan read 962 KB redo, 256 data blocks need recovery Started redo application at Thread 1: logseq 726, block 37343 Recovery of Online Redo Log: Thread 1 Group 3 Seq 726 Reading mem 0 Mem# 0: /u01/app/oracle/oradata/oa/redo03.log Mon Mar 02 16:09:27 2015 RECOVERY OF THREAD 1 STUCK AT BLOCK 1673 OF FILE 3 Completed redo application of 0.27MB Mon Mar 02 16:09:27 2015 RECOVERY OF THREAD 1 STUCK AT BLOCK 3104 OF FILE 3 Mon Mar 02 16:09:27 2015 RECOVERY OF THREAD 1 STUCK AT BLOCK 3613 OF FILE 3 Mon Mar 02 16:09:28 2015 RECOVERY OF THREAD 1 STUCK AT BLOCK 272 OF FILE 3 Mon Mar 02 16:09:28 2015 RECOVERY OF THREAD 1 STUCK AT BLOCK 2512 OF FILE 3 Hex dump of (file 2, block 92889) in trace file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_dbw2_4158.trc Corrupt block relative dba: 0x00816ad9 (file 2, block 92889) Bad header found during preparing block for write Data in bad block: type: 0 format: 0 rdba: 0x6ad90000 last change scn: 0x0000.00c6a052 seq: 0x1 flg: 0x00 spare1: 0x6 spare2: 0xa2 spare3: 0x5d7e consistency value in tail: 0xa0520001 check value in block header: 0x0 block checksum disabled Mon Mar 02 16:09:28 2015 Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_p007_4196.trc (incident=3833): ORA-00600: internal error code, arguments: [4502], [1], [], [], [], [], [], [], [], [], [], [] Mon Mar 02 16:09:28 2015 Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_p013_4208.trc (incident=3881): ORA-00600: internal error code, arguments: [2037], [4259067], [4244307968], [159], [243], [0], [2162032704], [100728832], [], [], [], [] Slave exiting with ORA-1172 exception Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_p009_4200.trc: ORA-01172: recovery of thread 1 stuck at block 3613 of file 3 ORA-01151: use media recovery to recover block, restore backup if needed Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_p001_4184.trc: ORA-01172: recovery of thread 1 stuck at block 2512 of file 3 ORA-01151: use media recovery to recover block, restore backup if needed Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_p021_4224.trc: ORA-10388: parallel query server interrupt (failure) Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_p021_4224.trc: Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_dbw2_4158.trc (incident=3697): ORA-00600: internal error code, arguments: [kcbzpbuf_1], [4], [1], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/oa/oa/incident/incdir_3697/oa_dbw2_4158_i3697.trc Exception [type: SIGSEGV, SI_KERNEL(general_protection)] [ADDR:0x0] [PC:0xD2DDB7, kcbs_shrink_pool()+705] [flags: 0x0, count: 1] Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_mman_4152.trc (incident=3673): ORA-07445: exception encountered: core dump [kcbs_shrink_pool()+705] [SIGSEGV] [ADDR:0x0] [PC:0xD2DDB7] [SI_KERNEL(general_protection)] [] Incident details in: /u01/app/oracle/diag/rdbms/oa/oa/incident/incdir_3673/oa_mman_4152_i3673.trc Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_dbw2_4158.trc: Mon Mar 02 16:09:34 2015 Instance terminated by DBW2, pid = 4158
第二次重启后增加新错误ORA-00600[17182]
Mon Mar 02 16:39:50 2015 Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_p002_4321.trc (incident=4993): ORA-00600: internal error code, arguments: [17182], [0x7F548C2BDBA8], [], [], [], [], [], [], [], [], [], []
进行了一些恢复处理后,日志中报错
主要体现在进行了不完全恢复,而且应该是对redo进行了重命名或者redo头损坏锁引起的一系列提示
Beginning crash recovery of 1 threads Started redo scan Completed redo scan read 962 KB redo, 256 data blocks need recovery Started redo application at Thread 1: logseq 726, block 37343 Recovery of Online Redo Log: Thread 1 Group 3 Seq 726 Reading mem 0 Mem# 0: /u01/app/oracle/oradata/oa/redo03.log RECOVERY OF THREAD 1 STUCK AT BLOCK 1673 OF FILE 3 Aborting crash recovery due to error 1172 Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_ora_6644.trc: ORA-01172: recovery of thread 1 stuck at block 1673 of file 3 ORA-01151: use media recovery to recover block, restore backup if needed Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_ora_6644.trc: ORA-01172: recovery of thread 1 stuck at block 1673 of file 3 ORA-01151: use media recovery to recover block, restore backup if needed ORA-1172 signalled during: alter database open... Tue Mar 03 11:17:59 2015 Sweep [inc][17178]: completed Sweep [inc][17177]: completed Sweep [inc2][17178]: completed Tue Mar 03 11:18:00 2015 ALTER DATABASE RECOVER database until cancel Media Recovery Start started logmerger process Parallel Media Recovery started with 24 slaves ORA-279 signalled during: ALTER DATABASE RECOVER database until cancel ... ALTER DATABASE RECOVER CONTINUE DEFAULT Tue Mar 03 11:18:06 2015 Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_pr00_6701.trc: ORA-00266: name of archived log file needed ORA-266 signalled during: ALTER DATABASE RECOVER CONTINUE DEFAULT ... ALTER DATABASE RECOVER CANCEL Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_pr00_6701.trc: ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below ORA-01194: file 1 needs more recovery to be consistent ORA-01110: data file 1: '/u01/app/oracle/oradata/oa/system01.dbf' Slave exiting with ORA-1547 exception Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_pr00_6701.trc: ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below ORA-01194: file 1 needs more recovery to be consistent ORA-01110: data file 1: '/u01/app/oracle/oradata/oa/system01.dbf' ORA-10879 signalled during: ALTER DATABASE RECOVER CANCEL ... Tue Mar 03 11:18:06 2015 Checker run found 4 new persistent data failures Tue Mar 03 11:18:13 2015 alter database open resetlogs RESETLOGS is being done without consistancy checks. This may result in a corrupted database. The database should be recreated. RESETLOGS after incomplete recovery UNTIL CHANGE 12986989 Resetting resetlogs activation ID 3278679642 (0xc36cae5a) Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_ora_6644.trc: ORA-00367: checksum error in log file header ORA-00322: log 1 of thread 1 is not current copy ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/oa/redo01.log' Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_ora_6644.trc:
再一步折腾,增加了_allow_resetlogs_corruption= TRUE之后数据库报ORA-600[2662]
Tue Mar 03 11:19:26 2015 SMON: enabling cache recovery Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_ora_6864.trc (incident=18195): ORA-00600: internal error code, arguments: [2662], [0], [13007002], [0], [13016626], [4194545], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/oa/oa/incident/incdir_18195/oa_ora_6864_i18195.trc Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_ora_6864.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [2662], [0], [13007002], [0], [13016626], [4194545], [], [], [], [], [], [] Error 704 happened during db open, shutting down database USER (ospid: 6864): terminating the instance due to error 704 Instance terminated by USER, pid = 6864 ORA-1092 signalled during: alter database open... opiodr aborting process unknown ospid (6864) as a result of ORA-1092 Tue Mar 03 11:19:29 2015 ORA-1092 : opitsk aborting process
进一步折腾,可以看出来undo已经被其offline,无法正常访问,导致系统报ORA-704和ORA-00376
Wed Mar 04 21:10:58 2015 SMON: enabling cache recovery Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_ora_17074.trc: ORA-00704: bootstrap process failure ORA-00604: error occurred at recursive SQL level 2 ORA-00376: file 3 cannot be read at this time ORA-01110: data file 3: '/u01/app/oracle/oradata/oa/undotbs01.dbf' Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_ora_17074.trc: ORA-00704: bootstrap process failure ORA-00604: error occurred at recursive SQL level 2 ORA-00376: file 3 cannot be read at this time ORA-01110: data file 3: '/u01/app/oracle/oradata/oa/undotbs01.dbf' Error 704 happened during db open, shutting down database USER (ospid: 17074): terminating the instance due to error 704 Instance terminated by USER, pid = 17074 ORA-1092 signalled during: alter database open... opiodr aborting process unknown ospid (17074) as a result of ORA-1092 Wed Mar 04 21:11:00 2015 ORA-1092 : opitsk aborting process
通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)检测结果见附件(xifenfei_db_recover_20150304),这里可以知道undo 不知道怎么折腾的数据文件scn较大而且还offline,
通过一些列方法(bbed,隐含参数等)调整数据库scn,强制启动数据库,报如下错误
Wed Mar 04 22:50:23 2015 SMON: enabling cache recovery ORA-01555 caused by SQL statement below (SQL ID: 3nkd3g3ju5ph1, SCN: 0x0000.4000003e): select obj#,type#,ctime,mtime,stime, status, dataobj#, flags, oid$, spare1, spare2 from obj$ where owner#=:1 and name=:2 and namespace=:3 and remoteowner is null and linkname is null and subname is null Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_ora_17807.trc: ORA-00704: bootstrap process failure ORA-00604: error occurred at recursive SQL level 2 ORA-01555: snapshot too old: rollback segment number 10 with name "_SYSSMU10_3550978943$" too small Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_ora_17807.trc: ORA-00704: bootstrap process failure ORA-00604: error occurred at recursive SQL level 2 ORA-01555: snapshot too old: rollback segment number 10 with name "_SYSSMU10_3550978943$" too small Error 704 happened during db open, shutting down database USER (ospid: 17807): terminating the instance due to error 704 Instance terminated by USER, pid = 17807 ORA-1092 signalled during: alter database open resetlogs... opiodr aborting process unknown ospid (17807) as a result of ORA-1092
根据经验,该错误怀疑是文件头scn不够大,块延迟清理导致,进一步增加scn尝试,最后依旧是ORA-00704/ORA-00604/ORA-01555错误
Wed Mar 04 22:50:23 2015 SMON: enabling cache recovery ORA-01555 caused by SQL statement below (SQL ID: 3nkd3g3ju5ph1, SCN: 0x0000.4000003e): select obj#,type#,ctime,mtime,stime, status, dataobj#, flags, oid$, spare1, spare2 from obj$ where owner#=:1 and name=:2 and namespace=:3 and remoteowner is null and linkname is null and subname is null Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_ora_17807.trc: ORA-00704: bootstrap process failure ORA-00604: error occurred at recursive SQL level 2 ORA-01555: snapshot too old: rollback segment number 10 with name "_SYSSMU10_3550978943$" too small Errors in file /u01/app/oracle/diag/rdbms/oa/oa/trace/oa_ora_17807.trc: ORA-00704: bootstrap process failure ORA-00604: error occurred at recursive SQL level 2 ORA-01555: snapshot too old: rollback segment number 10 with name "_SYSSMU10_3550978943$" too small Error 704 happened during db open, shutting down database USER (ospid: 17807): terminating the instance due to error 704 Instance terminated by USER, pid = 17807 ORA-1092 signalled during: alter database open resetlogs... opiodr aborting process unknown ospid (17807) as a result of ORA-1092
根据经验,在scn上做手脚估计难以解决给问题,对其启动过程做10046和errorstack分析发现
PARSING IN CURSOR #3 len=202 dep=2 uid=0 oct=3 lid=0 tim=1425481940448439 hv=3819099649 ad='64ff91af8' sqlid='3nkd3g3ju5ph1' select obj#,type#,ctime,mtime,stime, status, dataobj#, flags, oid$, spare1, spare2 from obj$ where owner#=:1 and name=:2 and namespace=:3 and remoteowner is null and linkname is null and subname is null END OF STMT PARSE #3:c=1000,e=334,p=0,cr=0,cu=0,mis=1,r=0,dep=2,og=4,plh=0,tim=1425481940448439 BINDS #3: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=7f5b3253a6f0 bln=22 avl=01 flg=05 value=0 Bind#1 oacdty=01 mxl=32(06) mxlc=00 mal=00 scl=00 pre=00 oacflg=18 fl2=0001 frm=01 csi=852 siz=32 off=0 kxsbbbfp=7f5b3253a6b8 bln=32 avl=06 flg=05 value="PROPS$" Bind#2 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=7f5b3253a688 bln=24 avl=02 flg=05 value=1 EXEC #3:c=0,e=640,p=0,cr=0,cu=0,mis=1,r=0,dep=2,og=4,plh=2853959010,tim=1425481940449147 WAIT #3: nam='db file sequential read' ela= 5 file#=1 block#=345 blocks=1 obj#=37 tim=1425481940449186 WAIT #3: nam='db file sequential read' ela= 4 file#=1 block#=44528 blocks=1 obj#=37 tim=1425481940449221 WAIT #3: nam='db file sequential read' ela= 3 file#=1 block#=5505 blocks=1 obj#=37 tim=1425481940449247 *** 2015-03-04 23:12:20.450 dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=3, mask=0x0) ----- Error Stack Dump ----- ORA-00604: error occurred at recursive SQL level 2 ORA-01555: snapshot too old: rollback segment number 10 with name "_SYSSMU10_3550978943$" too small ----- Current SQL Statement for this session (sql_id=g64r07v2jn8nq) ----- SELECT NULL FROM PROPS$ WHERE NAME='BOOTSTRAP_UPGRADE_ERROR'
这里可以发现是数据库在启动的过程中需要执行SELECT NULL FROM PROPS$ WHERE NAME=’BOOTSTRAP_UPGRADE_ERROR’语句，而该语句递归调用了select obj#,type#,ctime,mtime,stime, status, dataobj#, flags, oid$, spare1, spare2 from obj$ where owner#=:1 and name=:2 and namespace=:3 and remoteowner is null and linkname is null and subname is null 语句。既然这样通过一些方法避免数据库启动之时查询SELECT NULL FROM PROPS$ WHERE NAME=’BOOTSTRAP_UPGRADE_ERROR’语句，果然数据库启动成功。
知识点补充
ORA-600 [4502] [a]
Arg [a] ITL entry with a lock count Meaning: During ITL cleanout we clear all row locks but the ITL entry still thinks there is an uncleared lock. Ie: ITL has a locked row but there are no locked rows in the block
大体意思是数据库发现undo 的itl已经被清除，但是block中的itl依然存在，从而出现ORA-600[4502],引起该问题除bug外主要原因是坏块
ORA-600 [2037] [a] [b] {c} [d] [e] [f] [g]
Arg [a] Relative Data Block Address (RDBA) that the redo vector is for
Arg [b] The Block format
Arg {c} RDBA in the block itself
Arg [d] The block type
Arg [e] The sequence number
Arg [f] Flags, if set
Arg [g] The return value from the block head/tail checker.
DESCRIPTION:
During recovery we are examining a block to ensure that it is not
corrupt prior to applying any change vectors.
The block has failed this check and this exception is raised
大体意思是在恢复过程中，正在检查的块，以确保它在应用任何变化向量之前不损坏。如果检查失败排除该异常ORA-600[2037],引起该问题除bug外主要原因是坏块
ORA-600 [kcbzpbuf_1],[a],[b]
Arg [a] Corruption reason
Arg [b] Calculate checksum flag
Corruption reason:
#define KCBH_GOOD 0 /* block is valid */
#define KCBH_ZERO 1 /* block header was entirely zero on disk */
#define KCBH_BROKEN 2 /* corruption could be from a partial disk write */
#define KCBH_CHKVAL 3 /* The check value for the block failed */
#define KCBH_CORRUPT 4 /* this is the wrong block or is not a data block */
#define KCBH_ZERONG 5 /* all zero block and it is not allowed */
Calculate checksum flag:
The possible values are 1 (Generate Checksum - db_block_checksum is enabled - default value)
0 (do not generate checksum - db_block_checksum=false)
kcbzpbuf_1是该错误的源码函数
ORA-600 [17182] [a] [b] {c} [d] [e]
DESCRIPTION: Oracle has detected that the magic number in a memory chunk header has been overwritten. This is a heap (in memory) corruption and there is no underlying data corruption. The error may occur in the one of the process specific heaps (the Call heap, PGA heap, or session heap) or in the shared heap (SGA).
ORACLE 发现在内存中重要的块头被重新,但是没有基础数据损坏,大部分和数据块或者内存损坏有关系.
ORA-600 [4552] [a] [b] {c} [d] [e]
DESCRIPTION: This assertion is raised because we are trying to unlock the rows in a block, but receive an incorrect block type. The second argument is the block type received.
ORACLE尝试对某行进行解锁但是接收到了不正确的数据块类型,Arg [b]是接收到的数据块类型
ORA-600 [2662] [a] [b] {c} [d] [e]
DESCRIPTION:
A data block SCN is ahead of the current SCN.
The ORA-600 [2662] occurs when an SCN is compared to the dependent SCN
stored in a UGA variable.
If the SCN is less than the dependent SCN then we signal the ORA-600 [2662]
internal error.
ARGUMENTS:
Arg [a] Current SCN WRAP
Arg [b] Current SCN BASE
Arg {c} dependent SCN WRAP
Arg [d] dependent SCN BASE
Arg [e] Where present this is the DBA where the dependent SCN came from.
主要的含义就是oracle文件头scn比某个block dependent scn小从而出现该问题
联系：手机/微信(+86 17813235971) QQ(107644445)
标题：通过多次resetlogs规避类似ORA-01248: file N was created in the future of incomplete recovery错误
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
数据库现状
控制文件
控制文件中数据文件信息
数据文件头信息
redo信息
根据当前数据库恢复检查脚本(Oracle Database Recovery Check)收集的信息,数据库的是非归档状态,而且redo已经覆盖,数据库datafile 5 无法直接online.遇到这样情况,可以使用bbed修改文件头scn实现online(使用bbed让rac中的sysaux数据文件online),也可以通过使用_allow_resetlogs_corruption等隐含参数实现online.本恢复案例中有180个数据文件,160个offline,然后open数据库,所以大量数据文件无法正常online,bbed工作量太大.在恢复过程中不幸遇到ORA-01248
数据库resetlogs出现ORA-01248错误
SQL> alter database open resetlogs; alter database open resetlogs * ERROR at line 1: ORA-01248: file 5 was created in the future of incomplete recovery ORA-01110: data file 5: 'F:\TTDATA\PUBRTS.DAT'
alert日志记录
Fri Oct 10 15:09:26 2014 alter database open resetlogs Fri Oct 10 15:09:26 2014 RESETLOGS is being done without consistancy checks. This may result in a corrupted database. The database should be recreated. ORA-1248 signalled during: alter database open resetlogs... Fri Oct 10 15:15:22 2014 alter database open Fri Oct 10 15:15:22 2014 ORA-1589 signalled during: alter database open... Fri Oct 10 15:15:30 2014 alter database open resetlogs RESETLOGS is being done without consistancy checks. This may result in a corrupted database. The database should be recreated. ORA-1248 signalled during: alter database open resetlogs...
尝试offline文件然后resetlogs
SQL>ALTER DATABASE DATAFILE 5 OFFLINE; Database altered. sql>ALTER DATABASE OPEN RESETLOGS； ERROR at line 1: ORA-01245: ffline file 5 will be lost if resetlogs is done ORA-01110: data file 5: 'F:\TTDATA\PUBRTS.DAT'
alert日志
Fri Oct 10 15:19:37 2014 ALTER DATABASE DATAFILE 5 offline Fri Oct 10 15:19:37 2014 Completed: ALTER DATABASE DATAFILE 5 offline Fri Oct 10 15:19:40 2014 alter database open resetlogs RESETLOGS is being done without consistancy checks. This may result in a corrupted database. The database should be recreated. ORA-1245 signalled during: alter database open resetlogs...
出现该错误原因是由于数据库是非归档模式,offline数据文件需要使用offline drop
Fri Oct 10 15:22:16 2014 alter database datafile 5 offline drop Fri Oct 10 15:22:17 2014 Completed: alter database datafile 5 offline drop Fri Oct 10 15:23:13 2014 alter database open resetlogs Fri Oct 10 15:23:14 2014 Fri Oct 10 15:23:49 2014 RESETLOGS after complete recovery through change 1422423346 Resetting resetlogs activation ID 3503292347 (0xd0cfffbb) Fri Oct 10 15:24:01 2014 Setting recovery target incarnation to 3 Fri Oct 10 15:24:04 2014 Assigning activation ID 3649065262 (0xd980512e) LGWR: STARTING ARCH PROCESSES ARC0 started with pid=23, OS id=3772 Fri Oct 10 15:24:04 2014 ARC0: Archival started ARC1: Archival started LGWR: STARTING ARCH PROCESSES COMPLETE ARC1 started with pid=24, OS id=3668 Fri Oct 10 15:24:05 2014 Thread 1 opened at log sequence 1 Current log# 1 seq# 1 mem# 0: D:\ORACLE\PRODUCT\10.2.0\ORADATA\CLTTDB\REDO01.LOG Successful open of redo thread 1 Fri Oct 10 15:24:05 2014 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Fri Oct 10 15:24:05 2014 ARC0: STARTING ARCH PROCESSES ARC2: Archival started ARC0: STARTING ARCH PROCESSES COMPLETE ARC0: Becoming the 'no FAL' ARCH ARC2 started with pid=25, OS id=636 Fri Oct 10 15:24:06 2014 ARC0: Becoming the 'no SRL' ARCH Fri Oct 10 15:24:06 2014 ARC1: Becoming the heartbeat ARCH Fri Oct 10 15:24:06 2014 SMON: enabling cache recovery Fri Oct 10 15:24:07 2014 Successfully onlined Undo Tablespace 1. Dictionary check beginning File #5 is offline, but is part of an online tablespace. data file 5: 'F:\TTDATA\PUBRTS.DAT' Dictionary check complete Fri Oct 10 15:24:19 2014 SMON: enabling tx recovery Fri Oct 10 15:24:19 2014 Database Characterset is ZHS16GBK replication_dependency_tracking turned off (no async multimaster replication found) Starting background process QMNC QMNC started with pid=26, OS id=868 Fri Oct 10 15:24:21 2014 LOGSTDBY: Validating controlfile with logical metadata Fri Oct 10 15:24:21 2014 LOGSTDBY: Validation complete Completed: alter database open resetlogs
open成功后,再次resetlogs库,实现数据文件online
Fri Oct 10 15:28:44 2014 ALTER DATABASE DATAFILE 5 online Fri Oct 10 15:28:44 2014 Completed: ALTER DATABASE DATAFILE 5 online Fri Oct 10 15:31:46 2014 alter database open resetlogs Fri Oct 10 15:31:46 2014 RESETLOGS is being done without consistancy checks. This may result in a corrupted database. The database should be recreated. Setting recovery target incarnation to 4 Fri Oct 10 15:32:00 2014 Assigning activation ID 3649091231 (0xd980b69f) LGWR: STARTING ARCH PROCESSES ARC0 started with pid=23, OS id=700 Fri Oct 10 15:32:00 2014 ARC0: Archival started ARC1: Archival started LGWR: STARTING ARCH PROCESSES COMPLETE ARC1 started with pid=24, OS id=3360 Fri Oct 10 15:32:01 2014 Thread 1 opened at log sequence 1 Current log# 1 seq# 1 mem# 0: D:\ORACLE\PRODUCT\10.2.0\ORADATA\CLTTDB\REDO01.LOG Successful open of redo thread 1 Fri Oct 10 15:32:01 2014 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Fri Oct 10 15:32:01 2014 ARC0: STARTING ARCH PROCESSES ARC2: Archival started ARC0: STARTING ARCH PROCESSES COMPLETE ARC0: Becoming the 'no FAL' ARCH ARC2 started with pid=25, OS id=2016 Fri Oct 10 15:32:02 2014 ARC0: Becoming the 'no SRL' ARCH Fri Oct 10 15:32:02 2014 ARC1: Becoming the heartbeat ARCH Fri Oct 10 15:32:02 2014 SMON: enabling cache recovery Fri Oct 10 15:32:03 2014 Successfully onlined Undo Tablespace 1. Dictionary check beginning Fri Oct 10 15:32:15 2014 Dictionary check complete Fri Oct 10 15:32:15 2014 SMON: enabling tx recovery Fri Oct 10 15:32:15 2014 Database Characterset is ZHS16GBK replication_dependency_tracking turned off (no async multimaster replication found) Starting background process QMNC QMNC started with pid=26, OS id=256 Fri Oct 10 15:32:17 2014 LOGSTDBY: Validating controlfile with logical metadata Fri Oct 10 15:32:17 2014 LOGSTDBY: Validation complete Completed: alter database open resetlogs
联系：手机/微信(+86 17813235971) QQ(107644445)
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
朋友数据库启动遭遇ORA-00600[2663]
Mon Sep 22 19:24:20 2014 Thread 1 advanced to log sequence 17 (thread open) Thread 1 opened at log sequence 17 Current log# 17 seq# 17 mem# 0: /u02/orayali2/redo17.log Successful open of redo thread 1 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Mon Sep 22 19:24:20 2014 SMON: enabling cache recovery Errors in file /u01/app/oracle/diag/rdbms/orayali2/orayali2/trace/orayali2_ora_20722.trc (incident=336180): ORA-00600: internal error code, arguments: [2663], [13], [3140023138], [13], [3141216403], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/orayali2/orayali2/incident/incdir_336180/orayali2_ora_20722_i336180.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /u01/app/oracle/diag/rdbms/orayali2/orayali2/trace/orayali2_ora_20722.trc: ORA-00600: internal error code, arguments: [2663], [13], [3140023138], [13], [3141216403], [], [], [], [], [], [], [] Errors in file /u01/app/oracle/diag/rdbms/orayali2/orayali2/trace/orayali2_ora_20722.trc: ORA-00600: internal error code, arguments: [2663], [13], [3140023138], [13], [3141216403], [], [], [], [], [], [], [] Error 600 happened during db open, shutting down database USER (ospid: 20722): terminating the instance due to error 600 Instance terminated by USER, pid = 20722 ORA-1092 signalled during: alter database open... opiodr aborting process unknown ospid (20722) as a result of ORA-1092 Mon Sep 22 19:24:24 2014 ORA-1092 : opitsk aborting process
ORA-600[2663]与常见的ORA-600[2662]类似,都是由于block的scn大于文件头的scn导致,只不过错误的对象不一样而已.对于该类问题,我们的处理方法一般就是简单的推scn,但是这个库比较特殊11.2.0.3.5版本,一般方法无法推scn,因为收集操作日志有限，贴出核心操作步骤
[oracle@orayali2 OPatch]$ uname -a Linux orayali2 2.6.32-279.el6.x86_64 #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux [oracle@orayali2 OPatch]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.3.0 Production on Mon Sep 22 19:09:18 2014 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to an idle instance. SQL> startup mount ORACLE instance started. Total System Global Area 1.6535E+10 bytes Fixed Size 2244792 bytes Variable Size 9898561352 bytes Database Buffers 6610223104 bytes Redo Buffers 24256512 bytes Database mounted. SQL> oradebug setmypid Statement processed. SQL> oradebug DUMPvar SGA kcsgscn_ kcslf kcsgscn_ [060019598, 0600195C8) = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 60019278 00000000 SQL> oradebug poke 0x060019598 8 0x0000000000000040 BEFORE: [060019598, 0600195A0) = 00000000 00000000 AFTER: [060019598, 0600195A0) = 00000040 00000000 SQL> oradebug DUMPvar SGA kcsgscn_ kcslf kcsgscn_ [060019598, 0600195C8) = 00000040 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 60019278 00000000 SQL> alter database open; alter database open * ERROR at line 1: ORA-01092: ORACLE instance terminated. Disconnection forced ORA-30012: undo tablespace 'SYSTEM' does not exist or of wrong type Process ID: 21174 Session ID: 1563 Serial number: 3
现在错误已经改变,而是出现了ORA-30012的错误
alter database open Beginning crash recovery of 1 threads parallel recovery started with 31 processes Started redo scan Completed redo scan read 4 KB redo, 0 data blocks need recovery Started redo application at Thread 1: logseq 17, block 2, scn 58974597984 Recovery of Online Redo Log: Thread 1 Group 17 Seq 17 Reading mem 0 Mem# 0: /u02/orayali2/redo17.log Completed redo application of 0.00MB Completed crash recovery at Thread 1: logseq 17, block 3, scn 58974617986 0 data blocks read, 0 data blocks written, 4 redo k-bytes read Mon Sep 22 19:30:05 2014 Thread 1 advanced to log sequence 18 (thread open) Thread 1 opened at log sequence 18 Current log# 18 seq# 18 mem# 0: /u02/orayali2/redo18.log Successful open of redo thread 1 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Mon Sep 22 19:30:05 2014 SMON: enabling cache recovery Undo initialization errored: err:30012 serial:0 start:1143146928 end:1143147338 diff:410 (4 seconds) Errors in file /u01/app/oracle/diag/rdbms/orayali2/orayali2/trace/orayali2_ora_21174.trc: ORA-30012: undo tablespace 'SYSTEM' does not exist or of wrong type Errors in file /u01/app/oracle/diag/rdbms/orayali2/orayali2/trace/orayali2_ora_21174.trc: ORA-30012: undo tablespace 'SYSTEM' does not exist or of wrong type Error 30012 happened during db open, shutting down database USER (ospid: 21174): terminating the instance due to error 30012 Instance terminated by USER, pid = 21174 ORA-1092 signalled during: alter database open... opiodr aborting process unknown ospid (21174) as a result of ORA-1092 Mon Sep 22 19:30:08 2014 ORA-1092 : opitsk aborting process
猜测原因是undo设置有问题导致,检查果然发现undo_management=auto,而undo_tablespace=SYSTEM
SQL> startup mount ORACLE instance started. Total System Global Area 1.6535E+10 bytes Fixed Size 2244792 bytes Variable Size 9898561352 bytes Database Buffers 6610223104 bytes Redo Buffers 24256512 bytes Database mounted. SQL> show parameter undo; NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ undo_management string AUTO undo_retention integer 10800 undo_tablespace string SYSTEM SQL> alter system set undo_management=manual scope=spfile; System altered. SQL> shutdown immediate; ORA-01109: database not open Database dismounted. ORACLE instance shut down. SQL> startup ORACLE instance started. Total System Global Area 1.6535E+10 bytes Fixed Size 2244792 bytes Variable Size 9898561352 bytes Database Buffers 6610223104 bytes Redo Buffers 24256512 bytes Database mounted. Database opened.
解决该问题修改undo_management=manual即可
联系：手机/微信(+86 17813235971) QQ(107644445)
标题：记录一次system表空间坏块(ORA-01578)数据库恢复
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
半夜朋友打来求救电话,说xx医院his系统因为存储异常导致system坏块无法正常启动,因为是win平台无法使用bbed,无法修复system 坏块,请求技术支持
dbv检查system文件报坏块
对应具体地址为:file 1 block 39041和66738
判断控制文件异常
通过数据库恢复检查脚本(Oracle Database Recovery Check)脚本检测数据库发现控制文件明显异常(checkpoint scn
尝试恢复数据库
因此对该库进行了不完全恢复,然后尝试resetlogs打开数据库,数据库报ORA-600 2662错误
Fri Aug 29 02:35:08 2014 alter database open resetlogs Fri Aug 29 02:35:11 2014 RESETLOGS after complete recovery through change 451371288 Resetting resetlogs activation ID 1232269761 (0x4972f1c1) Fri Aug 29 02:35:15 2014 Setting recovery target incarnation to 3 Fri Aug 29 02:35:15 2014 Assigning activation ID 1384652231 (0x52881dc7) LGWR: STARTING ARCH PROCESSES ARC0 started with pid=17, OS id=1084 Fri Aug 29 02:35:15 2014 ARC0: Archival started ARC1: Archival started LGWR: STARTING ARCH PROCESSES COMPLETE ARC1 started with pid=18, OS id=2836 Fri Aug 29 02:35:15 2014 Thread 1 opened at log sequence 1 Current log# 1 seq# 1 mem# 0: Z:\ORACLE\PRODUCT\10.2.0\ORCL\REDO01.LOG Successful open of redo thread 1 Fri Aug 29 02:35:15 2014 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Fri Aug 29 02:35:15 2014 ARC1: Becoming the 'no FAL' ARCH ARC1: Becoming the 'no SRL' ARCH Fri Aug 29 02:35:15 2014 ARC0: Becoming the heartbeat ARCH Fri Aug 29 02:35:15 2014 SMON: enabling cache recovery Fri Aug 29 02:35:16 2014 Errors in file d:\oracle\product\10.2.0\admin\orcl\udump\orcl_ora_4824.trc: ORA-00600: 内部错误代码, 参数: [2662], [0], [451371311], [0], [451374534], [8388977], [], [] Fri Aug 29 02:35:16 2014 Errors in file d:\oracle\product\10.2.0\admin\orcl\udump\orcl_ora_4824.trc: ORA-00600: 内部错误代码, 参数: [2662], [0], [451371311], [0], [451374534], [8388977], [], [] Fri Aug 29 02:35:16 2014 Error 600 happened during db open, shutting down database USER: terminating instance due to error 600 Fri Aug 29 02:35:17 2014 Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_2928.trc: ORA-00600: ??????, ??: [], [], [], [], [], [], [], [] Instance terminated by USER, pid = 4824 ORA-1092 signalled during: alter database open resetlogs...
ORA-600 2662 该错误解决思路很明显,推进scn,数据库报ORA-01578
Fri Aug 29 02:42:47 2014 SMON: enabling cache recovery Fri Aug 29 02:42:47 2014 Successfully onlined Undo Tablespace 1. Dictionary check beginning Dictionary check complete Fri Aug 29 02:42:49 2014 SMON: enabling tx recovery Fri Aug 29 02:42:49 2014 Database Characterset is ZHS16GBK Opening with internal Resource Manager plan where NUMA PG = 1, CPUs = 16 replication_dependency_tracking turned off (no async multimaster replication found) Fri Aug 29 02:42:50 2014 Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_4804.trc: ORA-00604: 递归 SQL 级别 1 出现错误 ORA-01578: ORACLE 数据块损坏 (文件号 1, 块号 39041) ORA-01110: 数据文件 1: 'Z:\ORACLE\PRODUCT\10.2.0\ORCL\SYSTEM01.DBF' Fri Aug 29 02:42:50 2014 LOGSTDBY: Validating controlfile with logical metadata Fri Aug 29 02:42:51 2014 LOGSTDBY: Validation complete ORA-604 signalled during: alter database open...
使用event跳过坏块，启动数据库成功
Fri Aug 29 02:48:59 2014 SMON: enabling cache recovery Fri Aug 29 02:49:00 2014 Successfully onlined Undo Tablespace 1. Fri Aug 29 02:49:00 2014 SMON: enabling tx recovery Fri Aug 29 02:49:00 2014 Database Characterset is ZHS16GBK Opening with internal Resource Manager plan where NUMA PG = 1, CPUs = 16 replication_dependency_tracking turned off (no async multimaster replication found) Starting background process QMNC QMNC started with pid=34, OS id=3096 Fri Aug 29 02:49:01 2014 db_recovery_file_dest_size of 4096 MB is 0.00% used. This is a user-specified limit on the amount of space that will be used by this database for recovery-related files, and does not reflect the amount of space available in the underlying filesystem or ASM diskgroup. Fri Aug 29 02:49:01 2014 Completed: alter database open
查询坏块对象
因为这些对象均不是核心对象,直接进行truncate然后插入老数据
后续还有大量错误修复
ORA-12012: error on auto execute of job 1 ORA-08102: index key not found, obj# 239, file 1, block 1674 (2) ORA-00600: 内部错误代码, 参数: [kcbz_check_objd_typ], [0], [0], [1], [], [], [], [] ORA-00600: internal error code, arguments: [6749], [3], [12606796], [173], [], [], [], [] ORA-00600: 内部错误代码, 参数: [13013], [52898], [52895], [38288618], [44], [38288618], [17], [] ORA-00600: 内部错误代码, 参数: [13013], [5001], [52895], [38286476], [5], [38286476], [17], []
再次说明,很多时候数据库恢复不要看成多神秘,就是几个参数搞定,更加不要神化有坏块就bbed修复,当然非常极端,使用N中工具,N种尝试的也存在.做好备份重于一切
联系：手机/微信(+86 17813235971) QQ(107644445)
标题：数据库恢复历史再次刷新到Oracle 7.3.2版本—redo异常恢复
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
有网友在QQ上找我,说Oracle 7.3的数据库,因为redo异常咨询我是否可以恢复
检查数据库得到以下信息
SVRMGR> select * from v$version; BANNER ---------------------------------------------------------------- Oracle7 Workgroup Server Release 7.3.2.2.1 - Production Release PL/SQL Release 2.3.2.2.0 - Production CORE Version 3.5.2.0.0 - Production TNS for 32-bit Windows: Version 2.3.2.1.0 - Production NLSRTL Version 3.2.2.0.0 - Production 已选择 5 行
数据文件信息
redo信息
跳过redo进行恢复,在resetlogs过程中报rbs表空间坏块,然后通过dul工具获得回滚段名称,然后使用隐含参数屏蔽掉
License high water mark = 2 Starting up ORACLE RDBMS Version: 7.3.2.2.1. System parameters with non-default values: processes = 800 shared_pool_size = 540000000 control_files = D:\ORANT\DATABASE\ctl1orcl.ora, D:\ORANT\DATABASE\ctl2orcl.ora compatible = 7.3.0.0.0 log_buffer = 327680 log_checkpoint_interval = 1000000 db_files = 40 db_file_simultaneous_writes= 1280 max_rollback_segments = 12800 _offline_rollback_segments= RB13, RB14, RB15, RB16, RB20 _corrupted_rollback_segments= RB13, RB14, RB15, RB16, RB20 sequence_cache_entries = 100 sequence_cache_hash_buckets= 100 remote_login_passwordfile= SHARED mts_servers = 0 mts_max_servers = 0 mts_max_dispatchers = 0 audit_trail = NONE sort_area_retained_size = 65536 sort_direct_writes = AUTO db_name = oracle open_cursors = 800 text_enable = TRUE snapshot_refresh_processes= 1 background_dump_dest = %RDBMS73%\trace user_dump_dest = %RDBMS73%\trace Mon Jun 16 16:46:57 2014 PMON started Mon Jun 16 16:46:57 2014 DBWR started Mon Jun 16 16:46:57 2014 LGWR started Mon Jun 16 16:46:57 2014 RECO started Mon Jun 16 16:46:57 2014 SNP0 started Mon Jun 16 16:46:57 2014 alter database mount exclusive Mon Jun 16 16:46:58 2014 Successful mount of redo thread 1. Mon Jun 16 16:46:58 2014 Completed: alter database mount exclusive Mon Jun 16 16:48:15 2014 alter database open Mon Jun 16 16:48:16 2014 Beginning crash recovery of 1 threads Crash recovery completed successfully Mon Jun 16 16:48:17 2014 Thread 1 advanced to log sequence 9 Current log# 1 seq# 9 mem# 0: D:\ORANT\DATABASE\LOG2ORCL.ORA Thread 1 opened at log sequence 9 Current log# 1 seq# 9 mem# 0: D:\ORANT\DATABASE\LOG2ORCL.ORA Successful open of redo thread 1. Mon Jun 16 16:48:18 2014 SMON: enabling cache recovery Mon Jun 16 16:48:19 2014 Completed: alter database open Mon Jun 16 16:48:20 2014 SMON: enabling tx recovery SMON: about to recover undo segment 14 SMON: mark undo segment 14 as needs recovery SMON: about to recover undo segment 15 SMON: mark undo segment 15 as needs recovery SMON: about to recover undo segment 16 SMON: mark undo segment 16 as needs recovery SMON: about to recover undo segment 17 SMON: mark undo segment 17 as needs recovery SMON: about to recover undo segment 18 SMON: mark undo segment 18 as needs recovery Mon Jun 16 16:48:20 2014 Errors in file D:\ORANT\RDBMS73\trace\orclSMON.TRC: ORA-00600: internal error code, arguments: [4306], [21], [2], [], [], [], [], []
数据库在启动过程中出现ORA-00600[4306]，导致smon异常。该错误是因为在数据库open过程中smon会清理临时段从而出现该错误,通过设置event跳过，数据库算整体打开，不过在恢复过程中还遇到了
Mon Jun 16 17:53:10 2014 Errors in file D:\ORANT\RDBMS73\trace\orclDBWR.TRC: ORA-00600: internal error code, arguments: [3600], [3], [14], [], [], [], [], [] Mon Jun 16 18:05:12 2014 Errors in file D:\ORANT\RDBMS73\trace\ORA06880.TRC: ORA-01578: ORACLE数据块有错(文件号12, 块号46644) ORA-01110: 文件'12'没有联机 ORA-00600: 内部错误码, 变元: [4194], [18], [5], [], [], [] ORA-00600: 内部错误码, 变元: [4194], [18], [5], [], [], []
ORA-00600[3600]是因为在offline 回滚段所在表空间锁出现的问题
ORA-00600[4194]是因为回滚段所在的表空间数据文件出现坏块所导致