联系:手机/微信(+86 17813235971) QQ(107644445)
作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
前几天恢复了一个40多T的CASE:ORA-00600: internal error code, arguments: [16513], [1403] 恢复,又一个近30T的库由于硬件故障,通过其他人一系列恢复之后,无法正常open,让我们提供技术支持:
故障最初原因是由于存储异常
Fri Feb 19 09:03:49 2021 Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_ora_3460.trc: ORA-01114: 将块写入文件 849 时出现 IO 错误 (块 # 3871748) ORA-27070: 异步读取/写入失败 OSD-04016: 异步 I/O 请求排队时出错。 O/S-Error: (OS 1167) 设备没有连接。 ORA-01114: 将块写入文件 849 时出现 IO 错误 (块 # 3871748) ORA-27070: 异步读取/写入失败 OSD-04016: 异步 I/O 请求排队时出错。 O/S-Error: (OS 1167) 设备没有连接。
通过其他人一系列处理后,数据库报ORA-600 2662错误
Sat Feb 20 08:19:35 2021 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Sat Feb 20 08:19:35 2021 SMON: enabling cache recovery Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_ora_5304.trc(incident=1960181): ORA-00600:internal error code,arguments:[2662],[4],[2185364344], [4],[2185453722],[893388032],[],[],[],[],[],[] Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_ora_5304.trc: ORA-00600:internal error code,arguments:[2662],[4],[2185364344], [4],[2185453722],[893388032],[],[],[],[],[],[] Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_ora_5304.trc: ORA-00600:internal error code,arguments:[2662],[4],[2185364344], [4],[2185453722],[893388032],[],[],[],[],[],[] Error 600 happened during db open, shutting down database USER (ospid: 5304): terminating the instance due to error 600 Instance terminated by USER, pid = 5304 ORA-1092 signalled during: ALTER DATABASE OPEN... opiodr aborting process unknown ospid (5304) as a result of ORA-1092 Sat Feb 20 08:19:42 2021 ORA-1092 : opitsk aborting process
通过对scn处理,数据库顺利绕过该错误,然后报ORA-600 4194错误
Doing block recovery for file 213 block 4688 No block recovery was needed Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_smon_7048.trc(incident=1984136): ORA-00600: internal error code, arguments: [4194], [38.4.1381252], [0], [], [],[],[],[],[],[],[],[] Sat Feb 20 10:50:45 2021 Doing block recovery for file 213 block 4688 No block recovery was needed Fatal internal error happened while SMON was doing active transaction recovery. Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_smon_7048.trc: ORA-00600: internal error code, arguments: [4194], [38.4.1381252], [0], [], [],[],[],[],[],[],[],[] SMON (ospid: 7048): terminating the instance due to error 474 Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_ora_6652.trc(incident=1984185): ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] Sat Feb 20 10:50:52 2021 Instance terminated by SMON, pid = 7048
通过对异常事务进行处理,屏蔽smon进程进行回滚,数据库open成功,但是报ORA-600 4137错误
Sat Feb 20 10:53:46 2021 Sweep [inc][1992133]: completed Stopping background process MMNL Sat Feb 20 10:53:47 2021 Trace dumping is performing id=[cdmp_20210220105347] Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_smon_6576.trc(incident=1992134): ORA-00600: internal error code, arguments: [4137], [23.13.3094188], [0], [0], [], [], [], [], [], [], [], [] ORACLE Instance xifenfei (pid = 14) - Error 600 encountered while recovering transaction (23, 13). Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_smon_6576.trc: ORA-00600: internal error code, arguments: [4137], [23.13.3094188], [0], [0], [], [], [], [], [], [], [], [] Sat Feb 20 10:53:47 2021 Sweep [inc2][1992133]: completed Sat Feb 20 10:53:47 2021 Sweep [inc][1992134]: completed Stopping background process MMON Trace dumping is performing id=[cdmp_20210220105348] Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_smon_6576.trc(incident=1992135): ORA-00600: internal error code, arguments: [4137], [38.4.1381252], [0], [0], [], [], [], [], [], [], [], [] Starting background process MMON Starting background process MMNL Sat Feb 20 10:53:48 2021 MMON started with pid=16, OS id=6448 ALTER SYSTEM enable restricted session; Sat Feb 20 10:53:48 2021 MMNL started with pid=36, OS id=6840 ORACLE Instance xifenfei (pid = 14) - Error 600 encountered while recovering transaction (38, 4). Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_smon_6576.trc: ORA-00600: internal error code, arguments: [4137], [38.4.1381252], [0], [0], [], [], [], [], [], [], [], [] Sat Feb 20 10:53:49 2021 Sweep [inc][1992135]: completed Trace dumping is performing id=[cdmp_20210220105349] replication_dependency_tracking turned off (no async multimaster replication found) Completed: alter database open
对异常回滚段进行处理,数据库后端启动正常,不再报明显ORA-错误.通过hcheck.sql检查字典正常
HCheck Version 07MAY18 on 20-FEB-2021 11:35:11 ---------------------------------------------- Catalog Version 11.2.0.1.0 (1102000100) db_name: JYJG Catalog Fixed Procedure Name Version Vs Release Timestamp Result ------------------------------ ... ---------- -- ---------- -------------- ------ .- LobNotInObj ... 1102000100 <= *All Rel* 02/20 11:35:11 PASS .- MissingOIDOnObjCol ... 1102000100 <= *All Rel* 02/20 11:35:11 PASS .- SourceNotInObj ... 1102000100 <= *All Rel* 02/20 11:35:11 PASS .- IndIndparMismatch ... 1102000100 <= 1102000100 02/20 11:35:12 PASS .- InvCorrAudit ... 1102000100 <= 1102000100 02/20 11:35:12 PASS .- OversizedFiles ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS .- PoorDefaultStorage ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS .- PoorStorage ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS .- PartSubPartMismatch ... 1102000100 <= 1102000100 02/20 11:35:12 PASS .- TabPartCountMismatch ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS .- OrphanedTabComPart ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS .- MissingSum$ ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS .- MissingDir$ ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS .- DuplicateDataobj ... 1102000100 <= *All Rel* 02/20 11:35:12 PASS .- ObjSynMissing ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- ObjSeqMissing ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- OrphanedUndo ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- OrphanedIndex ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- OrphanedIndexPartition ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- OrphanedIndexSubPartition ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- OrphanedTable ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- OrphanedTablePartition ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- OrphanedTableSubPartition ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- MissingPartCol ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- OrphanedSeg$ ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- OrphanedIndPartObj# ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- DuplicateBlockUse ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- FetUet ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- Uet0Check ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- ExtentlessSeg ... 1102000100 <= 1102000100 02/20 11:35:13 PASS .- SeglessUET ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- BadInd$ ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- BadTab$ ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- BadIcolDepCnt ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- ObjIndDobj ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- TrgAfterUpgrade ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- ObjType0 ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- BadOwner ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- StmtAuditOnCommit ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- BadPublicObjects ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- BadSegFreelist ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- BadDepends ... 1102000100 <= *All Rel* 02/20 11:35:13 PASS .- CheckDual ... 1102000100 <= *All Rel* 02/20 11:35:14 PASS .- ObjectNames ... 1102000100 <= *All Rel* 02/20 11:35:14 PASS .- BadCboHiLo ... 1102000100 <= *All Rel* 02/20 11:35:14 PASS .- ChkIotTs ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS .- NoSegmentIndex ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS .- BadNextObject ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS .- DroppedROTS ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS .- FilBlkZero ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS .- DbmsSchemaCopy ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS .- OrphanedObjError ... 1102000100 > 1102000000 02/20 11:35:15 PASS .- ObjNotLob ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS .- MaxControlfSeq ... 1102000100 <= *All Rel* 02/20 11:35:15 PASS .- SegNotInDeferredStg ... 1102000100 > 1102000000 02/20 11:35:18 PASS .- SystemNotRfile1 ... 1102000100 > 902000000 02/20 11:35:18 PASS .- DictOwnNonDefaultSYSTEM ... 1102000100 <= *All Rel* 02/20 11:35:19 PASS .- OrphanTrigger ... 1102000100 <= *All Rel* 02/20 11:35:19 PASS .- ObjNotTrigger ... 1102000100 <= *All Rel* 02/20 11:35:19 PASS --------------------------------------- 20-FEB-2021 11:35:19 Elapsed: 8 secs --------------------------------------- Found 0 potential problem(s) and 0 warning(s) PL/SQL procedure successfully completed. Statement processed.
虽然字典正常,但是由于数据库屏蔽了一致性,建议客户在条件允许的情况下,进行逻辑迁移,排除风险隐患.