联系：手机/微信(+86 17813235971) QQ(107644445)
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
接到一个朋友恢复请求，由于硬件故障，通过底层技术恢复数据文件之后，数据库启动报错，但是dbv检查没有发现坏块
ORA-600 16513报错
[oracle@ora11g ~]$ ss SQL*Plus: Release 11.2.0.4.0 Production on Fri Jan 25 15:06:15 2019 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL> startup ORACLE instance started. Total System Global Area 3056513024 bytes Fixed Size 2257152 bytes Variable Size 704646912 bytes Database Buffers 2332033024 bytes Redo Buffers 17575936 bytes Database mounted. ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], [] Process ID: 20487 Session ID: 191 Serial number: 3
数据库alert日志报错
Fri Jan 25 14:15:18 2019 ALTER DATABASE OPEN Thread 1 opened at log sequence 11 Current log# 2 seq# 11 mem# 0: /u01/app/oracle/oradata/orcl11g/redo02.log Successful open of redo thread 1 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set SMON: enabling cache recovery Errors in file /u01/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_12712.trc (incident=4953): ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/orcl11g/orcl11g/incident/incdir_4953/orcl11g_ora_12712_i4953.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /u01/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_12712.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], [] Errors in file /u01/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_12712.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], [] Error 704 happened during db open, shutting down database USER (ospid: 12712): terminating the instance due to error 704 Instance terminated by USER, pid = 12712 ORA-1092 signalled during: ALTER DATABASE OPEN... opiodr aborting process unknown ospid (12712) as a result of ORA-1092
trace文件报错
---orcl11g_ora_12712.trc文件报错
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []
*** 2019-01-25 14:15:20.875
USER (ospid: 12712): terminating the instance due to error 704
---orcl11g_ora_12712_i4953.trc文件报错
Dump continued from file: /u01/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_12712.trc
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []
========= Dump for incident 4953 (ORA 600 [16513]) ========
*** 2019-01-25 14:15:19.247
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=1h50ks4ncswfn) -----
ALTER DATABASE OPEN
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
skdstdst()+41 call kgdsdst() 000000000 ? 000000000 ?
7FFD7A0B37B0 ? 7FFD7A0B3888 ?
7FFD7A0B8330 ? 000000002 ?
ksedst1()+103 call skdstdst() 000000000 ? 000000000 ?
7FFD7A0B37B0 ? 7FFD7A0B3888 ?
7FFD7A0B8330 ? 000000002 ?
ksedst()+39 call ksedst1() 000000000 ? 000000001 ?
7FFD7A0B37B0 ? 7FFD7A0B3888 ?
7FFD7A0B8330 ? 000000002 ?
dbkedDefDump()+2746 call ksedst() 000000000 ? 000000001 ?
7FFD7A0B37B0 ? 7FFD7A0B3888 ?
7FFD7A0B8330 ? 000000002 ?
ksedmp()+41 call dbkedDefDump() 000000003 ? 000000002 ?
7FFD7A0B37B0 ? 7FFD7A0B3888 ?
7FFD7A0B8330 ? 000000002 ?
ksfdmp()+69 call ksedmp() 000000003 ? 000000002 ?
7FFD7A0B37B0 ? 7FFD7A0B3888 ?
7FFD7A0B8330 ? 000000002 ?
dbgexPhaseII()+1764 call ksfdmp() 000000003 ? 000000002 ?
7FFD7A0B37B0 ? 7FFD7A0B3888 ?
7FFD7A0B8330 ? 000000002 ?
dbgexProcessError() call dbgexPhaseII() 7F06FBCD2730 ? 7F06FBCD5520 ?
+2680 7FFD7A0BCE08 ? 7FFD7A0B3888 ?
7FFD7A0B8330 ? 000000002 ?
dbgeExecuteForError call dbgexProcessError() 7F06FBCD2730 ? 7F06FBCD5520 ?
()+88 000000001 ? 000000000 ?
7FFD7A0B8330 ? 000000002 ?
dbgePostErrorKGE()+ call dbgeExecuteForError 7F06FBCD2730 ? 7F06FBCD5520 ?
2136 () 000000001 ? 000000001 ?
000000000 ? 000000002 ?
dbkePostKGE_kgsf()+ call dbgePostErrorKGE() 00C114E60 ? 7F06FB7B0040 ?
71 000000258 ? 000000001 ?
000000000 ? 000000002 ?
kgeade()+351 call dbkePostKGE_kgsf() 00C114E60 ? 7F06FB7B0040 ?
000000258 ? 000000001 ?
000000000 ? 000000002 ?
kgeriv_int()+125 call kgeade() 00C114E60 ? 00C115020 ?
7F06FB7B0040 ? 000000258 ?
000000000 ? 000000002 ?
kgeriv()+17 call kgeriv_int() 00C114E60 ? 00C115020 ?
7F06FB7B0040 ? 000000258 ?
000000000 ? 000000002 ?
kgesiv()+115 call kgeriv() 00C114E60 ? 00C115020 ?
7F06FB7B0040 ? 000000258 ?
000000000 ? 000000002 ?
ksesic2()+199 call kgesiv() 00C114E60 ? 7F06FB7B0040 ?
000004081 ? 000000002 ?
7FFD7A0BDAC0 ? 000000002 ?
kqdpts()+522 call ksesic2() 00C114E60 ? 000000000 ?
00000057B ? 000000000 ?
00000003B ? 000008000 ?
kqrlfc()+685 call kqdpts() 1158C5BC0 ? 000000000 ?
00000057B ? 000000000 ?
00000003B ? 000008000 ?
kqlbplc()+180 call kqrlfc() 1158C5BC0 ? 7FFD7A0BDEC0 ?
00000057B ? 000000000 ?
00000003B ? 000008000 ?
kqlblfc()+280 call kqlbplc() 000000000 ? 7FFD7A0BDEC0 ?
00000057B ? 000000000 ?
00000003B ? 000008000 ?
adbdrv()+57408 call kqlblfc() 000000000 ? 7FFD7A0C4BD4 ?
00000057B ? 000000000 ?
00000003B ? 000008000 ?
opiexe()+18724 call adbdrv() 000000000 ? 7FFD7A0C4BD4 ?
00000057B ? 000000000 ?
00000003B ? 000008000 ?
opiosq0()+4303 call opiexe() 000000004 ? 000000000 ?
00000057B ? 000000000 ?
00000003B ? 000008000 ?
kpooprx()+274 call opiosq0() 000000003 ? 00000000E ?
7FFD7A0C65B0 ? 0000000A4 ?
00000003B ? 000008000 ?
kpoal8()+842 call kpooprx() 7FFD7A0C9D94 ? 7FFD7A0C7DB8 ?
000000013 ? 000000001 ?
000000000 ? 000008000 ?
opiodr()+917 call kpoal8() 00000005E ? 7FFD7A0C7DB8 ?
000000013 ? 000000001 ?
000000000 ? 000008000 ?
ttcpip()+2183 call opiodr() 00000005E ? 00000001C ?
7FFD7A0C9D90 ? 000000001 ?
000000000 ? 000008000 ?
opitsk()+1710 call ttcpip() 00C132AB0 ? 0099D6610 ?
7FFD7A0C9D90 ? 000000000 ?
7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
opiino()+969 call opitsk() 00C132AB8 ? 000000000 ?
7FFD7A0C9D90 ? 000000000 ?
7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
opiodr()+917 call opiino() 00000003C ? 000000004 ?
7FFD7A0CB588 ? 000000000 ?
7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
opidrv()+570 call opiodr() 00000003C ? 000000004 ?
7FFD7A0CB588 ? 000000000 ?
7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
sou2o()+103 call opidrv() 00000003C ? 000000004 ?
7FFD7A0CB588 ? 000000000 ?
7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
opimai_real()+133 call sou2o() 7FFD7A0CB560 ? 00000003C ?
000000004 ? 7FFD7A0CB588 ?
7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
ssthrdmain()+265 call opimai_real() 000000002 ? 7FFD7A0CB750 ?
000000004 ? 7FFD7A0CB588 ?
7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
main()+201 call ssthrdmain() 000000002 ? 7FFD7A0CB750 ?
000000001 ? 000000000 ?
7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
__libc_start_main() call main() 000000002 ? 7FFD7A0CB8F8 ?
+253 000000001 ? 000000000 ?
7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
_start()+41 call __libc_start_main() 000A2A5B4 ? 000000002 ?
7FFD7A0CB8E8 ? 000000000 ?
7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
--------------------- Binary Stack Dump ---------------------
根据错误提示含ORA-00704错误,可以初步确定很可能是在数据库核心基表有问题导致该问题.trace数据库启动过程,发现在obj$中异常,通过跟踪定位到启动报错所在位置,然后通过bbed(参考:bbed 文章汇总)进行修改,再次尝试启动数据库
ORA-704 ORA-1555
SQL> alter database open; alter database open * ERROR at line 1: ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00604: error occurred at recursive SQL level 1 ORA-01555: snapshot too old: rollback segment number 10 with name "_SYSSMU10_1197734989$" too small Process ID: 17898 Session ID: 191 Serial number: 5
该错误比较熟悉,通过bbed继续修改相关信息(参考:在数据库open过程中常遇到ORA-01555汇总)启动数据库成功
SQL> startup mount; ORACLE instance started. Total System Global Area 3056513024 bytes Fixed Size 2257152 bytes Variable Size 704646912 bytes Database Buffers 2332033024 bytes Redo Buffers 17575936 bytes Database mounted. SQL> alter database open; Database altered.
尝试ddl操作报错误
ORA-600 kkdlcob-objn-exists
SQL> create table t1 as select * from dual;
create table t1 as select * from dual
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [kkdlcob-objn-exists], [87520], [],
[], [], [], [], [], [], [], [], []
该错误是由于字典obj$的dataobj#太小导致,修复该问题之后,有出现以下问题
ORA-8102
SQL> create table t1 as select * from dual;
create table t1 as select * from dual
*
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-08102: index key not found, obj# 39, file 1, block 54781 (2)
这个错误比较明显由于obj$上的index异常导致,通过文章:bootstrap$核心index(I_OBJ1,I_USER1,I_FILE#_BLOCK#,I_IND1,I_TS#,I_CDEF1等)异常恢复—ORA-00701错误解决修复之后一切恢复正常
SQL> CREATE TABLE T1 AS SELECT * FROM DUAL; Table created. SQL> DROP TABLE T1 PURGE; Table dropped.
至此数据库恢复基本完成，建议客户逻辑方式导出数据，导入到新库