ORA-600 16703 – Database SOS

最近由于安装介质问题,导致tab$被清空的case又多了起来,最近遇到几个客户咨询此类问题

以前写过一些类似的文档,参考:
tab$恢复错误汇总
ORA-600 kzrini:!uprofile处理
10g数据库遭遇ORA-600 16703
12C数据库遭遇ORA-600 16703
ORA-00600: internal error code, arguments: [16703], [1403], [32]
ORA-600 16703直接把orachk备份表插入到tab$恢复
ORA-600 16703故障解析—tab$表被清空
tab$异常被处理之后报ORA-600 13304故障处理

aix平台tab$被删除可能出现ORA-600 [16703], [1403], [28]错误
 ORA-00600: internal error code, arguments: [16703], [1403], [4] 故障处理
 ORA-600 16703故障,客户找人恢复数据库,数据库被进一步恶意破坏—ORA-00704 ORA-00922
警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703

近期又遇到ORA-600 16703和ORA-702故障

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：近期又遇到ORA-600 16703和ORA-702故障

比较常见的由于软件注入恶意代码或者被人恶意破坏的常见故障ORA-600 16703和ORA-702故障

上面的故障是由于bootstrap$表被删除导致，以前处理过类似case:
恶意删除bootstrap$导致数据库无法正常启动
恶意删除bootstrap$导致数据库无法正常启动
2019恢复第一单—ORA-704-ORA-702恢复
ORA-00702: bootstrap verison ” inconsistent with version ’8.0.0.0.0′

该故障的原因是一般是由于在互联网上下载了带恶意脚本的oracle安装介质,300天之后数据库重启会导致tab$被删除,参见:
ORA-600 16703故障解析—tab$表被清空
警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703
再次提醒oracle使用者:
1. 不要使用从第三方网站下载非官方的数据库访问软件（发生过被注入恶意脚本的plsql dev案例:plsql dev引起的数据库被黑勒索比特币实现原理分析和解决方案）
2. 从官方下载oracle 安装程序和补丁程序（防止被注入恶意脚本,参考:警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703）

ORA-600 16703故障,客户找人恢复数据库,数据库被进一步恶意破坏—ORA-00704 ORA-00922

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 16703故障,客户找人恢复数据库,数据库被进一步恶意破坏—ORA-00704 ORA-00922

有朋友找到我,数据库报ORA-600 16703错误,这个本来是一个比较常见的破坏故障(警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703数据库启动报错如下:

修复tab$启动库报ORA-00704 ORA-00922错误

SQL> alter database Open;
alter database Open
*
第 1 行出现错误:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00922: missing or invalid option
进程 ID: 1340
会话 ID: 191 序列号: 3

ORA-00704 ORA-00922是比较少见的错误,第一感觉bootstrap$损坏了,对数据库启动过程进行跟踪

PARSING IN CURSOR #11700472 len=600 dep=1 uid=0 oct=1 lid=0 tim=338738406773 hv=4034608779 
ad='7ffdef83f360' sqlid='asgjp8bs7qgnb'
CREATE TABLE UNDO$("US#" 
END OF STMT
PARSE #11700472:c=0,e=361,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=338738406773
EXEC #11700472:c=0,e=73,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=0,tim=338738406917
CLOSE #11700472:c=0,e=3,dep=1,type=0,tim=338738406997
=====================
PARSE ERROR #635423520:len=841 dep=1 uid=0 oct=1 lid=0 tim=338738407066 err=922
CREATE TABLE TS$<"TS#" NU ...
ORA-00704: 引导程序进程失败
ORA-00922: 选项缺失或无效
ORA-00704: 引导程序进程失败
ORA-00922: 选项缺失或无效

*** 2023-05-17 19:27:51.813
USER (ospid: 1340): terminating the instance due to error 704

*** 2023-05-17 19:27:54.050
EXEC #11710688:c=0,e=2481834,p=16,cr=62,cu=0,mis=0,r=0,dep=0,og=1,plh=0,tim=338740646732
ERROR #11710688:err=1092 tim=338740646777

进一步分析bootstrap$表记录

通过上述分析,可以确认原库的CREATE TABLE TS$(“TS#”被人修改为CREATE TABLE TS$<“TS#”,通过观察客户机器以及和客户确认,客户找的技术人员上传了bbed工具,并进行了一些操作.基于上述信息,大概率是被人通过bbed工具把TS$(修改为了TS$<,从而使得数据库修复tab$之后也无法正常启动.

tab$异常被处理之后报ORA-600 13304故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：tab$异常被处理之后报ORA-600 13304故障处理

又一例数据库启动报ORA-600 16703 1403 20错误故障

Sun Jun 13 14:00:56 2021
NOTE: dependency between database xff and diskgroup resource ora.DG_ARCH_xff.dg is established
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_56340.trc  (incident=348265):
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Incident details in: /opt/oracle/diag/rdbms/xff/xff1/incident/incdir_348265/xff1_ora_56340_i348265.trc
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_56340.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_56340.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 56340): terminating the instance due to error 704
Instance terminated by USER, pid = 56340

这个故障比较明显,根据我们之前的分析经验(警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703),应该是tab$被恶意破坏导致,通过分析安装程序,确认是该问题,客户通过互联网上的相关文章,dd方式进行处理,结果数据库报ORA-600 13304错误,无法继续,让我们提供技术支持

SMON: enabling tx recovery
Database Characterset is AL32UTF8
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_83843.trc  (incident=396265):
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /opt/oracle/diag/rdbms/xff/xff1/incident/incdir_396265/xff1_ora_83843_i396265.trc
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_83843.trc:
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Errors in file /opt/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_83843.trc:
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 83843): terminating the instance due to error 600
Instance terminated by USER, pid = 83843

通过我们的技术对数据库进行一系列恢复之后,open过程报错

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00904: "NAME": invalid identifier
Process ID: 23346
Session ID: 680 Serial number: 51933

通过跟踪启动过程分析

PARSE ERROR #140574232044112:len=45 dep=1 uid=0 oct=3 lid=0 tim=1623621695884944 err=904
select value$ from sys.props$ where name = :1
ORA-00604: error occurred at recursive SQL level 1
ORA-00904: "NAME": invalid identifier
ORA-00604: error occurred at recursive SQL level 1
ORA-00904: "NAME": invalid identifier

基本上可以确定是由于客户自行恢复导致props$表异常.通过进一步分析,确认是由于在对tab$处理不合适导致,进一步对tab$进行处理,数据库恢复正常,实现数据0丢失

ORA-00600: internal error code, arguments: [16703], [1403], [4] 故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-00600: internal error code, arguments: [16703], [1403], [4] 故障处理

有一个客户数据库遭遇ORA-600 16703错误,故障原因见:警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703,这种故障已经恢复比较多,在这次的恢复中遇到一个新错误,进行记录
接手给客户报错情况ORA-00600: internal error code, arguments: [16703], [1403], [20]

Thu May 13 22:36:11 2021
SMON: enabling cache recovery
Thu May 13 22:36:11 2021
NSA2 started with pid=61, OS id=6261 
Archived Log entry 90224 added for thread 1 sequence 50454 ID 0x19ae1c6c dest 1:
Errors in file /oracle/diag/rdbms/xff/xff/trace/xff_ora_5931.trc  (incident=741052):
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/diag/rdbms/xff/xff/incident/incdir_741052/xff_ora_5931_i741052.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/diag/rdbms/xff/xff/trace/xff_ora_5931.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Errors in file /oracle/diag/rdbms/xff/xff/trace/xff_ora_5931.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 5931): terminating the instance due to error 704
Instance terminated by USER, pid = 5931
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (5931) as a result of ORA-1092
Thu May 13 22:36:13 2021
ORA-1092 : opitsk aborting process

这种故障,由于是恶意脚本在数据库启动的时候清空tab$所致,使用bbed对tab$进行反向删除操作,实现数据库打开.
在这次的恢复过程中遇到ORA-600 16703 1403 4的错误

SQL> startup mount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area 7.0818E+10 bytes
Fixed Size                  2260448 bytes
Variable Size            1.3422E+10 bytes
Database Buffers         5.7177E+10 bytes
Redo Buffers              217030656 bytes
Database mounted.
SQL> alter database open ;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [4], [], [], [],
[], [], [], [], [], []
Process ID: 51886
Session ID: 3497 Serial number: 3

根据ora-600 16703 1403 4基本上可以判断是由于tab$这个表中缺少obj#=4的对象导致,通过查询正常库,确认是该对象为tab$,也就是说由于tab$对象中少了tab$记录.通过bbed分析确认

SQL> SELECT a.OBJ#
  2        ,TAB#
  3        ,a.DATAOBJ#
  4        ,BOBJ#
  5        ,NAME
  6        ,DBMS_ROWID.ROWID_RELATIVE_FNO (a.ROWID) FILE_ID
  7        ,DBMS_ROWID.ROWID_BLOCK_NUMBER (a.ROWID) BLOCK_ID
  8    FROM TAB$ a, obj$ b
  9   WHERE     a.obj# = b.obj#
 10         AND A.OBJ# IN (4);

      OBJ#       TAB#   DATAOBJ#      BOBJ# NAME
---------- ---------- ---------- ---------- ------------------------------
   FILE_ID   BLOCK_ID
---------- ----------
         4          1          2          2 TAB$
         1        147

BBED> set dba 1,147
        DBA             0x00400093 (4194451 1,147)

BBED> x /rnnnnnnnnnnnnncnnnnnnnntnnnnnnnnnncct  *kdbr[14]
rowdata[6848]                               @7349
-------------
flag@7349: 0x20 (KDRHFH)
lock@7350: 0x02
cols@7351:    0
nrid@7352:0x00407b09.1
BBED> set dba 0x00407b09
        DBA             0x00407b09 (4225801 1,31497)

BBED> p kdbt[1]
struct kdbt[1], 4 bytes                     @110
   sb2 kdbtoffs                             @110      10
   sb2 kdbtnrow                             @112      2

BBED> x /rnnnnnnnnnnnnncnnnnnnnntnnnnnnnnnncct  *kdbr[11]
rowdata[815]                                @4436
------------
flag@4436: 0x5c (KDRHFL, KDRHFF, KDRHFD, KDRHFC)
lock@4437: 0x02
cols@4438:    0
ckix@4439:    8

BBED> x /rn  *kdbr[8]
rowdata[950]                                @4571
------------
flag@4571: 0xac (KDRHFL, KDRHFF, KDRHFH, KDRHFK)
lock@4572: 0x00
cols@4573:    1
kref@4574:    1
hrid@4576:0x00400093.8
nrid@4582:0x00400094.0
col    0[2] @4590: 4

确认该记录发生了行迁移导致该问题,对其对应的block进行修复,数据库正常打开.

ORA-600 16703直接把orachk备份表插入到tab$恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 16703直接把orachk备份表插入到tab$恢复

有一个朋友和我说，他们数据库出现了以下错误ORA-600 16703 错误

他们是在虚拟化环境中,可以恢复到上一个快照点,但是主机启动之后,数据库依旧异常,让我们进行处理

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed Mar 24 17:04:01 2021

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ WRITE

SQL> select count(1) from tab$;

  COUNT(1)
----------
         0

很明显tab$已经被清空,数据库无法正常使用.因为库没有crash,尝试把备份的orachk表插入进来

SQL> insert into tab$ select * from ORACHKB514061BDCB10EBA9CF58F3;

6318 rows created.

SQL> commit;

Commit complete.

SQL> select 'DROP TRIGGER '||owner||'."'||TRIGGER_NAME||'";' from dba_triggers w
here TRIGGER_NAME like 'DBMS_%_INTERNAL% '
  2  union all
  3  select 'DROP PROCEDURE '||owner||'."'||a.object_name||'";' from dba_procedu
res a where a.object_name like 'DBMS_%_INTERNAL% '
  4  union all
  5  select 'drop '||object_type||' '||owner||'.'||object_name||';' from dba_obj
ects where object_name in('DBMS_SUPPORT_DBMONITOR','DBMS_SUPPORT_DBMONITORP');

'DROPTRIGGER'||OWNER||'."'||TRIGGER_NAME||'";'
--------------------------------------------------------------------------------

drop PROCEDURE SYS.DBMS_SUPPORT_DBMONITORP;
drop TRIGGER SYS.DBMS_SUPPORT_DBMONITOR;

SQL> drop PROCEDURE SYS.DBMS_SUPPORT_DBMONITORP;

Procedure dropped.

SQL> drop TRIGGER SYS.DBMS_SUPPORT_DBMONITOR;

Trigger dropped.

SQL> commit;

Commit complete.

SQL>

重启数据库,该故障恢复完成,数据完美恢复0丢失.

aix平台tab$被删除可能出现ORA-600 [16703], [1403], [28]错误

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：aix平台tab$被删除可能出现ORA-600 [16703], [1403], [28]错误

以前在恢复过程中遇到过ORA-00600: internal error code, arguments: [16703], [1403], [28]错误(10g数据库遭遇ORA-600 16703)以为是因为10g版本的tab$记录被删除的原因导致报错和最常见的ORA-00600: internal error code, arguments: [16703], [1403], [20]不完全一致(警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703),最近又遇到一个ORA-600 16703 1403 28错误的case,而且数据库版本是11.2.0.4 for aix平台,进一步说明该问题不是由于10g和11g的tab$被删除的区别导致,更多可能是由于操作系统不一样,数据库启动基表访问顺序不一致导致,特此进行说明.
数据库启动成功后报错

Wed Apr 01 22:36:19 2020
Completed: ALTER DATABASE OPEN /* db agent *//* {2:54387:2} */
Wed Apr 01 22:36:19 2020
Starting background process CJQ0
Wed Apr 01 22:36:19 2020
CJQ0 started with pid=53, OS id=7078224 
Wed Apr 01 22:36:21 2020
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_7668220.trc  (incident=40337):
ORA-00600: internal error code, arguments: [kzrini:!uprofile], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_40337/orcl2_ora_7668220_i40337.trc
Wed Apr 01 22:36:21 2020
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_6881718.trc  (incident=40369):
ORA-00600: internal error code, arguments: [kzrini:!uprofile], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_40369/orcl2_ora_6881718_i40369.trc
Setting Resource Manager plan SCHEDULER[0x32DB]:DEFAULT_MAINTENANCE_PLAN via scheduler window
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter

Wed Apr 01 22:51:16 2020
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_smon_7078802.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00957: duplicate column name
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_smon_7078802.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00957: duplicate column name
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_smon_7078802.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00957: duplicate column name

Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_m000_4850312.trc  (incident=48045):
ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_48045/orcl2_m000_4850312_i48045.trc
Thu Apr 02 00:59:35 2020
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_smon_7078802.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00957: duplicate column name
Thu Apr 02 00:59:36 2020
DDE: Problem Key 'ORA 600 [kzrini:!uprofile]' was completely flood controlled (0x6)
Further messages for this problem key will be suppressed for up to 10 minutes
Thu Apr 02 00:59:37 2020
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_m000_4850312.trc  (incident=48046):
ORA-00600: internal error code, arguments: [kewrose_1], [600], 
[ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280], [], [], [], []

数据库再次重启报错

Completed: ALTER DATABASE MOUNT /* db agent *//* {1:25340:2} */
ALTER DATABASE OPEN /* db agent *//* {1:25340:2} */
This instance was first to open
Picked broadcast on commit scheme to generate SCNs
Thu Apr 02 02:17:59 2020
Thread 2 opened at log sequence 13485
  Current log# 3 seq# 13485 mem# 0: +DATA/orcl/onlinelog/group_3.265.1003137665
  Current log# 3 seq# 13485 mem# 1: +FLASH/orcl/onlinelog/group_3.259.1003137677
Successful open of redo thread 2
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu Apr 02 02:17:59 2020
SMON: enabling cache recovery
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_7799020.trc  (incident=48395):
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_48395/orcl2_ora_7799020_i48395.trc
Thu Apr 02 02:18:01 2020
Thu Apr 02 02:18:01 2020
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_7799020.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_7799020.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 7799020): terminating the instance due to error 704
Instance terminated by USER, pid = 7799020
ORA-1092 signalled during: ALTER DATABASE OPEN /* db agent *//* {1:25340:2} */...

对于此类问题,通过分析,确定也是由于DBMS_SUPPORT_DBMONITORP恶意脚本导致tab$记录被删除,导致数据库启动异常,处理方法基本上就是对tab$进行恢复,然后open数据库.

ORA-00600: internal error code, arguments: [16703], [1403], [32]

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-00600: internal error code, arguments: [16703], [1403], [32]

有网友咨询数据库启动报ORA-00600: internal error code, arguments: [16703], [1403], [32]错误,这个错误和以往遇到的不太一样，以前恢复的一些案例
tab$恢复错误汇总
 10g数据库遭遇ORA-600 16703
12C数据库遭遇ORA-600 16703
tab$被恶意删除sys用户之外记录
 ORA-600 16703故障解析—tab$表被清空
 警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703

SQL> startup
ORACLE 例程已经启动。
Total System Global Area 1.3892E+10 bytes
Fixed Size                  5420776 bytes
Variable Size            2281703704 bytes
Database Buffers         1.1576E+10 bytes
Redo Buffers               28131328 bytes
数据库装载完毕。
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [32], [], [], [],[], [], [], [], [], []
进程 ID: 9512
会话 ID: 272 序列号: 22801

查看alert日志

Sun Jun 30 14:47:55 2019
SMON: enabling cache recovery
Errors in file D:\APP\SHORCL\diag\rdbms\orcl\orcl\trace\orcl_ora_7824.trc  (incident=177881) (PDBNAME=CDB$ROOT):
ORA-00600: 内部错误代码, 参数: [16703], [1403], [32], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\SHORCL\diag\rdbms\orcl\orcl\incident\incdir_177881\orcl_ora_7824_i177881.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Sun Jun 30 14:47:57 2019
Errors in file D:\APP\SHORCL\diag\rdbms\orcl\orcl\trace\orcl_ora_7824.trc:
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [16703], [1403], [32], [], [], [], [], [], [], [], [], []
Sun Jun 30 14:47:57 2019
Errors in file D:\APP\SHORCL\diag\rdbms\orcl\orcl\trace\orcl_ora_7824.trc:
ORA-00704: 引导程序进程失败
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [16703], [1403], [32], [], [], [], [], [], [], [], [], []
Sun Jun 30 14:47:57 2019
Errors in file D:\APP\SHORCL\diag\rdbms\orcl\orcl\trace\orcl_ora_7824.trc:
ORA-00704: 引导程序进程失败
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [16703], [1403], [32], [], [], [], [], [], [], [], [], []
Sun Jun 30 14:47:57 2019
Error 704 happened during db open, shutting down database
USER (ospid: 7824): terminating the instance due to error 704
Sun Jun 30 14:48:02 2019
Instance terminated by USER, pid = 7824
ORA-1092 signalled during: ALTER DATABASE OPEN...

根据以往经验,这个很可能也是tab$数据被删除导致。经过分析,该库的区别是由于该库是12C的pdb
12.1.0.2

pdb

通过分析确认,确实是tab$数据被删除,通过bbed反向删除处理,实现时间完美恢复,open之后删除恶意脚本,数据库直接使用,实现完美恢复

SQL> startup mount
ORACLE 例程已经启动。
Total System Global Area 1.3892E+10 bytes
Fixed Size                  5420776 bytes
Variable Size            2281703704 bytes
Database Buffers         1.1576E+10 bytes
Redo Buffers               28131328 bytes
数据库装载完毕。
SQL> alter database open;
数据库已更改。
SQL> select 'drop '||object_type||' '||owner||'.'||object_name||';' from dba_obj
ects where object_name in('DBMS_SUPPORT_DBMONITOR','DBMS_SUPPORT_DBMONITORP');
'DROP'||OBJECT_TYPE||''||OWNER||'.'||OBJECT_NAME||';'
--------------------------------------------------------------------------------
drop TRIGGER SYS.DBMS_SUPPORT_DBMONITOR;
drop PROCEDURE SYS.DBMS_SUPPORT_DBMONITORP;
SQL>
SQL> drop TRIGGER SYS.DBMS_SUPPORT_DBMONITOR;
触发器已删除。
SQL> drop PROCEDURE SYS.DBMS_SUPPORT_DBMONITORP;
过程已删除。
SQL> shutdown immediate;
数据库已经关闭。
已经卸载数据库。
ORACLE 例程已经关闭。
SQL> startup
ORACLE 例程已经启动。
Total System Global Area 1.3892E+10 bytes
Fixed Size                  5420776 bytes
Variable Size            2281703704 bytes
Database Buffers         1.1576E+10 bytes
Redo Buffers               28131328 bytes
数据库装载完毕。
数据库已经打开。
SQL> show pdbs;
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDBORCL                        MOUNTED
SQL> alter session set  container=pdborcl;
会话已更改。
SQL> alter database open;
数据库已更改。

12C数据库遭遇ORA-600 16703

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：12C数据库遭遇ORA-600 16703

关于tab$异常的问题,我们在10G(10g数据库遭遇ORA-600 16703),11G(警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703)中有发现,近期在12C版本中也发现该问题.再次提醒请注意检查安装介质的md5值和现在运行的库中是否有相关恶意程序
数据库版本

alert日志报错

************************************************************
Warning: The SCN headroom for this database is only 45 days!
************************************************************
************************************************************
Warning: The SCN headroom for this database is only 45 days!
************************************************************
Picked broadcast on commit scheme to generate SCNs
Fri Jan 25 11:34:55 2019
Starting background process TMON
Fri Jan 25 11:34:55 2019
TMON started with pid=104, OS id=47282
Fri Jan 25 11:34:55 2019
Thread 1 advanced to log sequence 11942 (thread open)
Thread 1 opened at log sequence 11942
Current log# 5 seq# 11942 mem# 0: +DATA/xifenfei/ONLINELOG/redo01_01.log
Successful open of redo thread 1
Fri Jan 25 11:34:55 2019
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan 25 11:34:55 2019
Network Resource Management enabled for Process LG00 (pid 451491) for Exadata I/O
Fri Jan 25 11:34:55 2019
SMON: enabling cache recovery
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_46443.trc (incident=394486):
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Fri Jan 25 11:34:56 2019
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_46443.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Fri Jan 25 11:34:56 2019
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_46443.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Fri Jan 25 11:34:56 2019
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_46443.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Fri Jan 25 11:34:56 2019
Error 704 happened during db open, shutting down database
USER (ospid: 46443): terminating the instance due to error 704
Fri Jan 25 11:34:57 2019
Instance terminated by USER, pid = 46443

处理思路基本上和其他两个版本一样,都是对tab$进行修复,然后直接open数据库,但是在这次的恢复中遇到ORA-00600 kdfReserveSingle_1错误
ORA-600 kdfReserveSingle_1

SQL> alter database open ;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280],
[], [], [], [], [], [], [], [], []
Process ID: 319852
Session ID: 2401 Serial number: 24555
---alert日志
Starting background process TMON
Fri Jan 25 20:51:41 2019
TMON started with pid=94, OS id=322554
Fri Jan 25 20:51:41 2019
Thread 1 advanced to log sequence 11943 (thread open)
Thread 1 opened at log sequence 11943
Current log# 6 seq# 11943 mem# 0: +DATA/xifenfei/ONLINELOG/redo02_01.log
Successful open of redo thread 1
Fri Jan 25 20:51:41 2019
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan 25 20:51:41 2019
Network Resource Management enabled for Process LG00 (pid 319754) for Exadata I/O
Fri Jan 25 20:51:41 2019
SMON: enabling cache recovery
Fri Jan 25 20:51:41 2019
[319852] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:88159754 end:88160264 diff:510 ms (0.5 seconds)
Verifying minimum file header compatibility (11g) for tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
Fri Jan 25 20:51:41 2019
SMON: enabling tx recovery
Fri Jan 25 20:51:41 2019
Database Characterset is ZHS16GBK
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_319852.trc (incident=466446):
ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Fri Jan 25 20:51:42 2019
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_319852.trc:
ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280], [], [], [], [], [], [], [], [], []
Fri Jan 25 20:51:42 2019
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_319852.trc:
ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 319852): terminating the instance due to error 600
Fri Jan 25 20:51:43 2019
Instance terminated by USER, pid = 319852
ORA-1092 signalled during: alter database open…
opiodr aborting process unknown ospid (319852) as a result of ORA-1092
Fri Jan 25 20:51:49 2019
ORA-1092 : opitsk aborting process

这个错误主要是由于12c的tab$和以前版本比有一些特殊,导致开始恢复不全,通过继续修复tab$之后数据库正常open.
相关文章汇总
tab$恢复错误汇总
 12C数据库遭遇ORA-600 16703
10G数据库遭遇ORA-600 16703
ORA-600 16703故障解析—tab$表被清空
 警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703

tab$恢复错误汇总

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：tab$恢复错误汇总

在以前多以前发现的tab$被恶意脚本删除的问题(ORA-600 16703故障解析—tab$表被清空,警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703),虽然多次强调注意Oracle安装介质安全,但是很不幸,还是大量客户中招.我们这一年多对于tab$的故障进行了大量case处理,拯救了大量客户的核心数据,也积累了一些常见的可能遭遇的错误.主要恢复思路是使用bbed处理异常block,让数据库open起来.
ORA-00704 ORA-39700
有核心基表处理异常导致

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Process ID: 1603
Session ID: 1 Serial number: 5
Sun Jan 06 21:30:14 2019
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1603.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1603.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Error 704 happened during db open, shutting down database
USER (ospid: 1603): terminating the instance due to error 704
Instance terminated by USER, pid = 1603
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (1603) as a result of ORA-1092
Sun Jan 06 21:30:14 2019
ORA-1092 : opitsk aborting process

ora-704 ora-604 ora-01555
由于scn异常导致

SQL> alter database open upgrade;
alter database open upgrade
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name
"_SYSSMU2_2996391332$" too small
Process ID: 26520
Session ID: 1 Serial number: 5
Sun Jan 06 19:49:12 2019
SMON: enabling cache recovery
ORA-01555 caused by SQL statement below (SQL ID: bqbdby3c400p7, SCN: 0x0022.1117ef75):
select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_26520.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name "_SYSSMU2_2996391332$" too small
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_26520.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name "_SYSSMU2_2996391332$" too small
Error 704 happened during db open, shutting down database
USER (ospid: 26520): terminating the instance due to error 704
Instance terminated by USER, pid = 26520
ORA-1092 signalled during: alter database open upgrade...
opiodr aborting process unknown ospid (26520) as a result of ORA-1092

ORA-600 13304
有核心基表处理异常导致

SQL> startup mount;
ORACLE instance started.
Total System Global Area  521936896 bytes
Fixed Size                  2254824 bytes
Variable Size             352323608 bytes
Database Buffers          163577856 bytes
Redo Buffers                3780608 bytes
Database mounted.
SQL> alter database open upgrade;
alter database open upgrade
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [],
[], [], [], []
Process ID: 1724
Session ID: 1 Serial number: 5
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sun Jan 06 21:31:04 2019
SMON: enabling cache recovery
[1724] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:2239884804 end:2239884864 diff:60 (0 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc  (incident=61755):
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_61755/orcl_ora_1724_i61755.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc:
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc:
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 1724): terminating the instance due to error 600
Instance terminated by USER, pid = 1724
ORA-1092 signalled during: alter database open upgrade...
opiodr aborting process unknown ospid (1724) as a result of ORA-1092
Sun Jan 06 21:31:06 2019
ORA-1092 : opitsk aborting process

ORA-00704 ORA-600 kdBlkCheckError
恢复的block有逻辑坏块

SQL> startup
ORACLE instance started.
Total System Global Area 3056513024 bytes
Fixed Size                  2257152 bytes
Variable Size             704646912 bytes
Database Buffers         2332033024 bytes
Redo Buffers               17575936 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497],
[6121], [], [], [], [], [], [], [], []
Process ID: 76932
Session ID: 191 Serial number: 3
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc  (incident=6153):
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_6153/xifenfei_ora_76932_i6153.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 76932): terminating the instance due to error 704
Instance terminated by USER, pid = 76932
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (76932) as a result of ORA-1092
Sat Feb 22 11:04:19 2014
ORA-1092 : opitsk aborting process

标签归档：ORA-600 16703