ORA-600 16513 – Database SOS

标题：ORA-00600: internal error code, arguments: [16513], [1403] 恢复

接到客户请求,存储异常断电之后,一个40多T的经分数据库无法正常启动,通过各方一系列操作之后,数据库依旧无法open,报错信息为:ORA-00600: internal error code, arguments: [16513], [1403], [28]

Sun Feb 14 00:20:09 BEIST 2021
SMON: enabling cache recovery
Sun Feb 14 00:20:09 BEIST 2021
ORA-01555 caused by SQL statement below (SQL ID: 4krwuz0ctqxdt, SCN: 0x0f27.13cd4fc3):
Sun Feb 14 00:20:09 BEIST 2021
select ctime, mtime, stime from obj$ where obj# = :1
Sun Feb 14 00:20:09 BEIST 2021
Errors in file /oracle10g/db/admin/xifenfei/udump/xifenfei1_ora_177254.trc:
ORA-00600: internal error code, arguments: [16513], [1403], [28], [], [], [], [], []
Sun Feb 14 00:20:10 BEIST 2021
Errors in file /oracle10g/db/admin/xifenfei/udump/xifenfei1_ora_177254.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16513], [1403], [28], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Instance terminated by USER, pid = 177254
ORA-1092 signalled during: ALTER DATABASE OPEN...

通过对启动过程进行跟踪

=====================
PARSING IN CURSOR #5 len=52 dep=1 uid=0 oct=3 lid=0 tim=194171381576991 hv=429618617 ad='afcee60'
select ctime, mtime, stime from obj$ where obj# = :1
END OF STMT
PARSE #5:c=0,e=257,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,tim=194171381576990
BINDS #5:
kkscoacd
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=1104b8ed0  bln=22  avl=02  flg=05
  value=28
EXEC #5:c=0,e=422,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,tim=194171381577472
WAIT #5: nam='db file sequential read' ela= 274 file#=1 block#=110 blocks=1 obj#=36 tim=194171381577809
WAIT #5: nam='db file sequential read' ela= 257 file#=1 block#=78943 blocks=1 obj#=36 tim=194171381578123
WAIT #5: nam='db file sequential read' ela= 253 file#=1 block#=111 blocks=1 obj#=36 tim=194171381578416
WAIT #5: nam='db file sequential read' ela= 226 file#=1 block#=62 blocks=1 obj#=18 tim=194171381578692
=====================
PARSING IN CURSOR #6 len=142 dep=2 uid=0 oct=3 lid=0 tim=194171381579134 hv=361892850 ad='df87eb0'
select /*+ rule */ name,file#,block#,status$,user#,undosqn,xactsqn,scnbas,scnwrp,
DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1
END OF STMT
PARSE #6:c=0,e=368,p=0,cr=0,cu=0,mis=1,r=0,dep=2,og=3,tim=194171381579133
BINDS #6:
kkscoacd
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=1105af9a8  bln=22  avl=02  flg=05
  value=92
EXEC #6:c=0,e=531,p=0,cr=0,cu=0,mis=1,r=0,dep=2,og=3,tim=194171381579736
WAIT #6: nam='db file sequential read' ela= 232 file#=1 block#=102 blocks=1 obj#=34 tim=194171381580011
WAIT #6: nam='db file sequential read' ela= 251 file#=1 block#=54 blocks=1 obj#=15 tim=194171381580307
FETCH #6:c=0,e=615,p=2,cr=2,cu=0,mis=0,r=1,dep=2,og=3,tim=194171381580369
STAT #6 id=1 cnt=1 pid=0 pos=1 obj=15 op='TABLE ACCESS BY INDEX ROWID UNDO$ (cr=2 pr=2 pw=0 time=584 us)'
STAT #6 id=2 cnt=1 pid=1 pos=1 obj=34 op='INDEX UNIQUE SCAN I_UNDO1 (cr=1 pr=1 pw=0 time=282 us)'
WAIT #5: nam='db file sequential read' ela= 260 file#=4290 block#=365090 blocks=1 obj#=0 tim=194171381580721
FETCH #5:c=10000,e=3327,p=7,cr=7,cu=0,mis=0,r=0,dep=1,og=4,tim=194171381580817
*** 2021-02-14 02:32:40.430
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [16513], [1403], [28], [], [], [], [], []
Current SQL statement for this session:
alter database open
----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
ksedst+001c          bl       ksedst1              70000090FC00AE4 ? 000000000 ?
ksedmp+0290          bl       ksedst               104C1CA58 ?
ksfdmp+02d8          bl       03F4BD0C             
kgeriv+0108          bl       _ptrgl               
kgesiv+0080          bl       kgeriv               7000008F6DB5890 ? 00000000C ?
                                                   7000008F6DB5890 ? 0FFFFFDC3 ?
                                                   0FFFFFFBF ?
ksesic2+0060         bl       kgesiv               5FFFEC580 ? 500000000 ?
                                                   FFFFFFFFFFEC590 ? 0F5FAF808 ?
                                                   7000008F5FAF7E8 ?
kqdpts+0158          bl       ksesic2              408100004081 ? 000000000 ?
                                                   00000057B ? 000000000 ?
                                                   00000001C ?
                                                   A006150571B05EF0 ?
                                                   1100CF0A8 ? 000000001 ?
kqrlfc+0274          bl       kqdpts               000000000 ?
kqlbplc+00b4         bl       03F4E6A8             
kqlblfc+0230         bl       kqlbplc              00000009D ?
adbdrv+1a9c          bl       03F4B66C             
opiexe+2db4          bl       adbdrv               
opiosq0+1ac8         bl       opiexe               1103B418C ? 000000000 ?
                                                   FFFFFFFFFFF9008 ?
kpooprx+016c         bl       opiosq0              300F1E1D4 ? 000000000 ?
                                                   000000000 ? A4FFFFFFFF9798 ?
                                                   000000000 ?
kpoal8+03cc          bl       kpooprx              FFFFFFFFFFFB814 ?
                                                   FFFFFFFFFFFB5C0 ?
                                                   1300000013 ? 100000001 ?
                                                   000000000 ? A40000000000A4 ?
                                                   000000000 ? 1103B4378 ?
opiodr+0b2c          bl       _ptrgl               
ttcpip+1020          bl       _ptrgl               
opitsk+117c          bl       01FBD04C             
opiino+09d0          bl       opitsk               1EFFFFD7E0 ? 000000000 ?
opiodr+0b2c          bl       _ptrgl               
opidrv+04a4          bl       opiodr               3C102B1398 ? 404C6FF30 ?
                                                   FFFFFFFFFFFF7A0 ? 0102B1390 ?
sou2o+0090           bl       opidrv               3C02705B3C ? 440663000 ?
                                                   FFFFFFFFFFFF7A0 ?
opimai_real+01bc     bl       01FB9BF4             
main+0098            bl       opimai_real          000000000 ? 000000000 ?
__start+0098         bl       main                 000000000 ? 000000000 ?
 
--------------------- Binary Stack Dump ---------------------

报错比较明显,数据库在查询obj$取obj#=28的数据的时候无法正常获取到该数据,从而报错ORA-600 16513错误.通过对相关block进行分析,发现有事务异常

BBED>  p ktbbh.ktbbhitl[1]
struct ktbbhitl[1], 24 bytes                @44
   struct ktbitxid, 8 bytes                 @44
      ub2 kxidusn                           @44       0x5c00
      ub2 kxidslt                           @46       0x0a00
      ub4 kxidsqn                           @48       0x48b4fc01
   struct ktbituba, 8 bytes                 @52
      ub4 kubadba                           @52       0x22928531
      ub2 kubaseq                           @56       0xe383
      ub1 kubarec                           @58       0x01
   ub2 ktbitflg                             @60       0x0120 (NONE)
   union _ktbitun, 2 bytes                  @62
      b2 _ktbitfsc                          @62       0
      ub2 _ktbitwrp                         @62       0x0000
   ub4 ktbitbas                             @64       0x95c2ff13

通过一些技巧处理规避掉该事务,然后启动库报熟悉的ORA-01555错误

相对比较简单参考(在数据库open过程中常遇到ORA-01555汇总),数据库顺利open成功,完成春节后第一个大库的恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 16513故障恢复

接到一个朋友恢复请求，由于硬件故障，通过底层技术恢复数据文件之后，数据库启动报错，但是dbv检查没有发现坏块
ORA-600 16513报错

[oracle@ora11g ~]$ ss
SQL*Plus: Release 11.2.0.4.0 Production on Fri Jan 25 15:06:15 2019
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> startup
ORACLE instance started.
Total System Global Area 3056513024 bytes
Fixed Size                  2257152 bytes
Variable Size             704646912 bytes
Database Buffers         2332033024 bytes
Redo Buffers               17575936 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [],
[], [], [], [], [], []
Process ID: 20487
Session ID: 191 Serial number: 3

数据库alert日志报错

Fri Jan 25 14:15:18 2019
ALTER DATABASE OPEN
Thread 1 opened at log sequence 11
  Current log# 2 seq# 11 mem# 0: /u01/app/oracle/oradata/orcl11g/redo02.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_12712.trc  (incident=4953):
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl11g/orcl11g/incident/incdir_4953/orcl11g_ora_12712_i4953.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_12712.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_12712.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 12712): terminating the instance due to error 704
Instance terminated by USER, pid = 12712
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (12712) as a result of ORA-1092

trace文件报错

---orcl11g_ora_12712.trc文件报错
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []
*** 2019-01-25 14:15:20.875
USER (ospid: 12712): terminating the instance due to error 704
---orcl11g_ora_12712_i4953.trc文件报错
Dump continued from file: /u01/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_12712.trc
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []
========= Dump for incident 4953 (ORA 600 [16513]) ========
*** 2019-01-25 14:15:19.247
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=1h50ks4ncswfn) -----
ALTER DATABASE OPEN
----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
skdstdst()+41        call     kgdsdst()            000000000 ? 000000000 ?
                                                   7FFD7A0B37B0 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
ksedst1()+103        call     skdstdst()           000000000 ? 000000000 ?
                                                   7FFD7A0B37B0 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
ksedst()+39          call     ksedst1()            000000000 ? 000000001 ?
                                                   7FFD7A0B37B0 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
dbkedDefDump()+2746  call     ksedst()             000000000 ? 000000001 ?
                                                   7FFD7A0B37B0 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
ksedmp()+41          call     dbkedDefDump()       000000003 ? 000000002 ?
                                                   7FFD7A0B37B0 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
ksfdmp()+69          call     ksedmp()             000000003 ? 000000002 ?
                                                   7FFD7A0B37B0 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
dbgexPhaseII()+1764  call     ksfdmp()             000000003 ? 000000002 ?
                                                   7FFD7A0B37B0 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
dbgexProcessError()  call     dbgexPhaseII()       7F06FBCD2730 ? 7F06FBCD5520 ?
+2680                                              7FFD7A0BCE08 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
dbgeExecuteForError  call     dbgexProcessError()  7F06FBCD2730 ? 7F06FBCD5520 ?
()+88                                              000000001 ? 000000000 ?
                                                   7FFD7A0B8330 ? 000000002 ?
dbgePostErrorKGE()+  call     dbgeExecuteForError  7F06FBCD2730 ? 7F06FBCD5520 ?
2136                          ()                   000000001 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbkePostKGE_kgsf()+  call     dbgePostErrorKGE()   00C114E60 ? 7F06FB7B0040 ?
71                                                 000000258 ? 000000001 ?
                                                   000000000 ? 000000002 ?
kgeade()+351         call     dbkePostKGE_kgsf()   00C114E60 ? 7F06FB7B0040 ?
                                                   000000258 ? 000000001 ?
                                                   000000000 ? 000000002 ?
kgeriv_int()+125     call     kgeade()             00C114E60 ? 00C115020 ?
                                                   7F06FB7B0040 ? 000000258 ?
                                                   000000000 ? 000000002 ?
kgeriv()+17          call     kgeriv_int()         00C114E60 ? 00C115020 ?
                                                   7F06FB7B0040 ? 000000258 ?
                                                   000000000 ? 000000002 ?
kgesiv()+115         call     kgeriv()             00C114E60 ? 00C115020 ?
                                                   7F06FB7B0040 ? 000000258 ?
                                                   000000000 ? 000000002 ?
ksesic2()+199        call     kgesiv()             00C114E60 ? 7F06FB7B0040 ?
                                                   000004081 ? 000000002 ?
                                                   7FFD7A0BDAC0 ? 000000002 ?
kqdpts()+522         call     ksesic2()            00C114E60 ? 000000000 ?
                                                   00000057B ? 000000000 ?
                                                   00000003B ? 000008000 ?
kqrlfc()+685         call     kqdpts()             1158C5BC0 ? 000000000 ?
                                                   00000057B ? 000000000 ?
                                                   00000003B ? 000008000 ?
kqlbplc()+180        call     kqrlfc()             1158C5BC0 ? 7FFD7A0BDEC0 ?
                                                   00000057B ? 000000000 ?
                                                   00000003B ? 000008000 ?
kqlblfc()+280        call     kqlbplc()            000000000 ? 7FFD7A0BDEC0 ?
                                                   00000057B ? 000000000 ?
                                                   00000003B ? 000008000 ?
adbdrv()+57408       call     kqlblfc()            000000000 ? 7FFD7A0C4BD4 ?
                                                   00000057B ? 000000000 ?
                                                   00000003B ? 000008000 ?
opiexe()+18724       call     adbdrv()             000000000 ? 7FFD7A0C4BD4 ?
                                                   00000057B ? 000000000 ?
                                                   00000003B ? 000008000 ?
opiosq0()+4303       call     opiexe()             000000004 ? 000000000 ?
                                                   00000057B ? 000000000 ?
                                                   00000003B ? 000008000 ?
kpooprx()+274        call     opiosq0()            000000003 ? 00000000E ?
                                                   7FFD7A0C65B0 ? 0000000A4 ?
                                                   00000003B ? 000008000 ?
kpoal8()+842         call     kpooprx()            7FFD7A0C9D94 ? 7FFD7A0C7DB8 ?
                                                   000000013 ? 000000001 ?
                                                   000000000 ? 000008000 ?
opiodr()+917         call     kpoal8()             00000005E ? 7FFD7A0C7DB8 ?
                                                   000000013 ? 000000001 ?
                                                   000000000 ? 000008000 ?
ttcpip()+2183        call     opiodr()             00000005E ? 00000001C ?
                                                   7FFD7A0C9D90 ? 000000001 ?
                                                   000000000 ? 000008000 ?
opitsk()+1710        call     ttcpip()             00C132AB0 ? 0099D6610 ?
                                                   7FFD7A0C9D90 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
opiino()+969         call     opitsk()             00C132AB8 ? 000000000 ?
                                                   7FFD7A0C9D90 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
opiodr()+917         call     opiino()             00000003C ? 000000004 ?
                                                   7FFD7A0CB588 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
opidrv()+570         call     opiodr()             00000003C ? 000000004 ?
                                                   7FFD7A0CB588 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
sou2o()+103          call     opidrv()             00000003C ? 000000004 ?
                                                   7FFD7A0CB588 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
opimai_real()+133    call     sou2o()              7FFD7A0CB560 ? 00000003C ?
                                                   000000004 ? 7FFD7A0CB588 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
ssthrdmain()+265     call     opimai_real()        000000002 ? 7FFD7A0CB750 ?
                                                   000000004 ? 7FFD7A0CB588 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
main()+201           call     ssthrdmain()         000000002 ? 7FFD7A0CB750 ?
                                                   000000001 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
__libc_start_main()  call     main()               000000002 ? 7FFD7A0CB8F8 ?
+253                                               000000001 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
_start()+41          call     __libc_start_main()  000A2A5B4 ? 000000002 ?
                                                   7FFD7A0CB8E8 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
--------------------- Binary Stack Dump ---------------------

根据错误提示含ORA-00704错误,可以初步确定很可能是在数据库核心基表有问题导致该问题.trace数据库启动过程,发现在obj$中异常,通过跟踪定位到启动报错所在位置,然后通过bbed(参考:bbed 文章汇总)进行修改,再次尝试启动数据库
ORA-704 ORA-1555

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 10 with name
"_SYSSMU10_1197734989$" too small
Process ID: 17898
Session ID: 191 Serial number: 5

该错误比较熟悉,通过bbed继续修改相关信息(参考:在数据库open过程中常遇到ORA-01555汇总)启动数据库成功

SQL> startup mount;
ORACLE instance started.
Total System Global Area 3056513024 bytes
Fixed Size                  2257152 bytes
Variable Size             704646912 bytes
Database Buffers         2332033024 bytes
Redo Buffers               17575936 bytes
Database mounted.
SQL> alter database open;
Database altered.

尝试ddl操作报错误
ORA-600 kkdlcob-objn-exists

SQL>  create table t1 as select * from dual;
 create table t1 as select * from dual
                                  *
ERROR at line 1:
ORA-00600: internal error code, arguments: [kkdlcob-objn-exists], [87520], [],
[], [], [], [], [], [], [], [], []

该错误是由于字典obj$的dataobj#太小导致,修复该问题之后,有出现以下问题
ORA-8102

SQL> create table t1 as select * from dual;
create table t1 as select * from dual
                                 *
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-08102: index key not found, obj# 39, file 1, block 54781 (2)

这个错误比较明显由于obj$上的index异常导致,通过文章:bootstrap$核心index(I_OBJ1,I_USER1,I_FILE#_BLOCK#,I_IND1,I_TS#,I_CDEF1等)异常恢复—ORA-00701错误解决修复之后一切恢复正常

SQL> CREATE TABLE T1 AS SELECT * FROM DUAL;
Table created.
SQL> DROP TABLE T1 PURGE;
Table dropped.

至此数据库恢复基本完成，建议客户逻辑方式导出数据，导入到新库

标签归档：ORA-600 16513