联系：手机/微信(+86 17813235971) QQ(107644445)
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
前几天恢复了一个40多T的CASE:ORA-00600: internal error code, arguments: [16513], [1403] 恢复,又一个近30T的库由于硬件故障,通过其他人一系列恢复之后,无法正常open,让我们提供技术支持:
故障最初原因是由于存储异常
Fri Feb 19 09:03:49 2021 Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_ora_3460.trc: ORA-01114: 将块写入文件 849 时出现 IO 错误 (块 # 3871748) ORA-27070: 异步读取/写入失败 OSD-04016: 异步 I/O 请求排队时出错。 O/S-Error: (OS 1167) 设备没有连接。 ORA-01114: 将块写入文件 849 时出现 IO 错误 (块 # 3871748) ORA-27070: 异步读取/写入失败 OSD-04016: 异步 I/O 请求排队时出错。 O/S-Error: (OS 1167) 设备没有连接。
通过其他人一系列处理后,数据库报ORA-600 2662错误
Sat Feb 20 08:19:35 2021 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Sat Feb 20 08:19:35 2021 SMON: enabling cache recovery Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_ora_5304.trc(incident=1960181): ORA-00600:internal error code,arguments:[2662],[4],[2185364344], [4],[2185453722],[893388032],[],[],[],[],[],[] Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_ora_5304.trc: ORA-00600:internal error code,arguments:[2662],[4],[2185364344], [4],[2185453722],[893388032],[],[],[],[],[],[] Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_ora_5304.trc: ORA-00600:internal error code,arguments:[2662],[4],[2185364344], [4],[2185453722],[893388032],[],[],[],[],[],[] Error 600 happened during db open, shutting down database USER (ospid: 5304): terminating the instance due to error 600 Instance terminated by USER, pid = 5304 ORA-1092 signalled during: ALTER DATABASE OPEN... opiodr aborting process unknown ospid (5304) as a result of ORA-1092 Sat Feb 20 08:19:42 2021 ORA-1092 : opitsk aborting process
通过对scn处理,数据库顺利绕过该错误,然后报ORA-600 4194错误
Doing block recovery for file 213 block 4688 No block recovery was needed Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_smon_7048.trc(incident=1984136): ORA-00600: internal error code, arguments: [4194], [38.4.1381252], [0], [], [],[],[],[],[],[],[],[] Sat Feb 20 10:50:45 2021 Doing block recovery for file 213 block 4688 No block recovery was needed Fatal internal error happened while SMON was doing active transaction recovery. Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_smon_7048.trc: ORA-00600: internal error code, arguments: [4194], [38.4.1381252], [0], [], [],[],[],[],[],[],[],[] SMON (ospid: 7048): terminating the instance due to error 474 Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_ora_6652.trc(incident=1984185): ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] Sat Feb 20 10:50:52 2021 Instance terminated by SMON, pid = 7048
通过对异常事务进行处理,屏蔽smon进程进行回滚,数据库open成功,但是报ORA-600 4137错误
Sat Feb 20 10:53:46 2021 Sweep [inc][1992133]: completed Stopping background process MMNL Sat Feb 20 10:53:47 2021 Trace dumping is performing id=[cdmp_20210220105347] Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_smon_6576.trc(incident=1992134): ORA-00600: internal error code, arguments: [4137], [23.13.3094188], [0], [0], [], [], [], [], [], [], [], [] ORACLE Instance xifenfei (pid = 14) - Error 600 encountered while recovering transaction (23, 13). Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_smon_6576.trc: ORA-00600: internal error code, arguments: [4137], [23.13.3094188], [0], [0], [], [], [], [], [], [], [], [] Sat Feb 20 10:53:47 2021 Sweep [inc2][1992133]: completed Sat Feb 20 10:53:47 2021 Sweep [inc][1992134]: completed Stopping background process MMON Trace dumping is performing id=[cdmp_20210220105348] Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_smon_6576.trc(incident=1992135): ORA-00600: internal error code, arguments: [4137], [38.4.1381252], [0], [0], [], [], [], [], [], [], [], [] Starting background process MMON Starting background process MMNL Sat Feb 20 10:53:48 2021 MMON started with pid=16, OS id=6448 ALTER SYSTEM enable restricted session; Sat Feb 20 10:53:48 2021 MMNL started with pid=36, OS id=6840 ORACLE Instance xifenfei (pid = 14) - Error 600 encountered while recovering transaction (38, 4). Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_smon_6576.trc: ORA-00600: internal error code, arguments: [4137], [38.4.1381252], [0], [0], [], [], [], [], [], [], [], [] Sat Feb 20 10:53:49 2021 Sweep [inc][1992135]: completed Trace dumping is performing id=[cdmp_20210220105349] replication_dependency_tracking turned off (no async multimaster replication found) Completed: alter database open
对异常回滚段进行处理,数据库后端启动正常,不再报明显ORA-错误.通过hcheck.sql检查字典正常
HCheck Version 07MAY18 on 20-FEB-2021 11:35:11
----------------------------------------------
Catalog Version 11.2.0.1.0 (1102000100)
db_name: JYJG
Catalog Fixed
Procedure Name Version Vs Release Timestamp
Result
------------------------------ ... ---------- -- ---------- --------------
------
.- LobNotInObj ... 1102000100 <= *All Rel* 02/20 11:35:11 PASS
.- MissingOIDOnObjCol ... 1102000100 <= *All Rel* 02/20 11:35:11 PASS
.- SourceNotInObj ... 1102000100 <= *All Rel* 02/20 11:35:11 PASS
.- IndIndparMismatch ... 1102000100 <= 1102000100 02/20 11:35:12 PASS
.- InvCorrAudit ... 1102000100 <= 1102000100 02/20 11:35:12 PASS
.- OversizedFiles ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS
.- PoorDefaultStorage ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS
.- PoorStorage ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS
.- PartSubPartMismatch ... 1102000100 <= 1102000100 02/20 11:35:12 PASS
.- TabPartCountMismatch ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS
.- OrphanedTabComPart ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS
.- MissingSum$ ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS
.- MissingDir$ ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS
.- DuplicateDataobj ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS
.- ObjSynMissing ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- ObjSeqMissing ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- OrphanedUndo ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- OrphanedIndex ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- OrphanedIndexPartition ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- OrphanedIndexSubPartition ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- OrphanedTable ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- OrphanedTablePartition ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- OrphanedTableSubPartition ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- MissingPartCol ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- OrphanedSeg$ ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- OrphanedIndPartObj# ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- DuplicateBlockUse ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- FetUet ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- Uet0Check ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- ExtentlessSeg ... 1102000100 <= 1102000100 02/20 11:35:13 PASS
.- SeglessUET ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- BadInd$ ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- BadTab$ ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- BadIcolDepCnt ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- ObjIndDobj ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- TrgAfterUpgrade ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- ObjType0 ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- BadOwner ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- StmtAuditOnCommit ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- BadPublicObjects ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- BadSegFreelist ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- BadDepends ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS
.- CheckDual ... 1102000100 <= *All Rel* 02/20 11:35:14 PASS
.- ObjectNames ... 1102000100 <= *All Rel* 02/20 11:35:14 PASS
.- BadCboHiLo ... 1102000100 <= *All Rel* 02/20 11:35:14 PASS
.- ChkIotTs ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS
.- NoSegmentIndex ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS
.- BadNextObject ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS
.- DroppedROTS ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS
.- FilBlkZero ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS
.- DbmsSchemaCopy ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS
.- OrphanedObjError ... 1102000100 > 1102000000 02/20 11:35:15 PASS
.- ObjNotLob ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS
.- MaxControlfSeq ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS
.- SegNotInDeferredStg ... 1102000100 > 1102000000 02/20 11:35:18 PASS
.- SystemNotRfile1 ... 1102000100 > 902000000 02/20 11:35:18 PASS
.- DictOwnNonDefaultSYSTEM ... 1102000100 <= *All Rel* 02/20 11:35:19 PASS
.- OrphanTrigger ... 1102000100 <= *All Rel* 02/20 11:35:19 PASS
.- ObjNotTrigger ... 1102000100 <= *All Rel* 02/20 11:35:19 PASS
---------------------------------------
20-FEB-2021 11:35:19 Elapsed: 8 secs
---------------------------------------
Found 0 potential problem(s) and 0 warning(s)
PL/SQL procedure successfully completed.
Statement processed.
虽然字典正常,但是由于数据库屏蔽了一致性,建议客户在条件允许的情况下,进行逻辑迁移,排除风险隐患.