2025年第一起ORA-600 16703故障恢复

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：2025年第一起ORA-600 16703故障恢复

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

又有一个客户数据库启动报ORA-600 16703错误
ora-600-16703

查看alert日志信息

Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_ora_8784.trc  (incident=20617):
ORA-00600: 内部错误代码, 参数: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\incident\incdir_20617\xff_ora_8784_i20617.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_ora_8784.trc:
ORA-00704: 引导程序进程失败
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_ora_8784.trc:
ORA-00704: 引导程序进程失败
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 8784): terminating the instance due to error 704
Instance terminated by USER, pid = 8784
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (8784) as a result of ORA-1092

这个故障一般是由于oracle安装介质被注入了恶意脚本,在数据库运行300天之后重启数据库就会遭遇该问题:
警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703
对于这个问题,我们这边经过多年的恢复经验积累和研究,可以实现直接open库并且不用做逻辑迁移,完美恢复之后业务直接使用,最大限度恢复数据和最小限度减少停机时间
以往类似恢复case汇总:
tab$恢复错误汇总
ORA-600 16703故障再现
tab$异常被处理之后报ORA-600 13304故障处理
ORA-600 16703直接把orachk备份表插入到tab$恢复
最近遇到几起ORA-600 16703故障(tab$被清空),请引起重视
ORA-00600: internal error code, arguments: [16703], [1403], [32]
aix平台tab$被删除可能出现ORA-600 [16703], [1403], [28]错误
ORA-00600: internal error code, arguments: [16703], [1403], [4] 原因
ORA-00600: internal error code, arguments: [16703], [1403], [4] 故障处理

ORA-600 16703故障,客户找人恢复数据库,数据库被进一步恶意破坏—ORA-00704 ORA-00922

ORA-600 kzrini:!uprofile处理

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：ORA-600 kzrini:!uprofile处理

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

朋友反馈,oracle expdp导出数据hang住,然后重启数据库,启动成功,但是执行导出的时候报ORA-600 kzrini:!uprofile错误
kzrini uprofile

因为有过大量类似故障的恢复case,所以第一反应这种故障很可能就是tab$被清空的故障,参见:警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703,对于这类故障,我们是国内第一个发现并且进行预警(在blog,qq/微信群,oracle技术大会等),但是还有很多朋友中招.今天这个发现故障之后没有重启,比较顺利，进行山删除插入即可,参考以前的处理:ORA-600 16703直接把orachk备份表插入到tab$恢复

tab$恢复错误汇总

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：tab$恢复错误汇总

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

在以前多以前发现的tab$被恶意脚本删除的问题(ORA-600 16703故障解析—tab$表被清空,警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703),虽然多次强调注意Oracle安装介质安全,但是很不幸,还是大量客户中招.我们这一年多对于tab$的故障进行了大量case处理,拯救了大量客户的核心数据,也积累了一些常见的可能遭遇的错误.主要恢复思路是使用bbed处理异常block,让数据库open起来.
ORA-00704 ORA-39700
有核心基表处理异常导致

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Process ID: 1603
Session ID: 1 Serial number: 5
Sun Jan 06 21:30:14 2019
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1603.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1603.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Error 704 happened during db open, shutting down database
USER (ospid: 1603): terminating the instance due to error 704
Instance terminated by USER, pid = 1603
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (1603) as a result of ORA-1092
Sun Jan 06 21:30:14 2019
ORA-1092 : opitsk aborting process

ora-704 ora-604 ora-01555
由于scn异常导致

SQL> alter database open upgrade;
alter database open upgrade
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name
"_SYSSMU2_2996391332$" too small
Process ID: 26520
Session ID: 1 Serial number: 5
Sun Jan 06 19:49:12 2019
SMON: enabling cache recovery
ORA-01555 caused by SQL statement below (SQL ID: bqbdby3c400p7, SCN: 0x0022.1117ef75):
select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_26520.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name "_SYSSMU2_2996391332$" too small
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_26520.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name "_SYSSMU2_2996391332$" too small
Error 704 happened during db open, shutting down database
USER (ospid: 26520): terminating the instance due to error 704
Instance terminated by USER, pid = 26520
ORA-1092 signalled during: alter database open upgrade...
opiodr aborting process unknown ospid (26520) as a result of ORA-1092

ORA-600 13304
有核心基表处理异常导致

SQL> startup mount;
ORACLE instance started.
Total System Global Area  521936896 bytes
Fixed Size                  2254824 bytes
Variable Size             352323608 bytes
Database Buffers          163577856 bytes
Redo Buffers                3780608 bytes
Database mounted.
SQL> alter database open upgrade;
alter database open upgrade
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [],
[], [], [], []
Process ID: 1724
Session ID: 1 Serial number: 5
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sun Jan 06 21:31:04 2019
SMON: enabling cache recovery
[1724] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:2239884804 end:2239884864 diff:60 (0 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc  (incident=61755):
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_61755/orcl_ora_1724_i61755.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc:
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc:
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 1724): terminating the instance due to error 600
Instance terminated by USER, pid = 1724
ORA-1092 signalled during: alter database open upgrade...
opiodr aborting process unknown ospid (1724) as a result of ORA-1092
Sun Jan 06 21:31:06 2019
ORA-1092 : opitsk aborting process

ORA-00704 ORA-600 kdBlkCheckError
恢复的block有逻辑坏块

SQL> startup
ORACLE instance started.
Total System Global Area 3056513024 bytes
Fixed Size                  2257152 bytes
Variable Size             704646912 bytes
Database Buffers         2332033024 bytes
Redo Buffers               17575936 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497],
[6121], [], [], [], [], [], [], [], []
Process ID: 76932
Session ID: 191 Serial number: 3
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc  (incident=6153):
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_6153/xifenfei_ora_76932_i6153.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 76932): terminating the instance due to error 704
Instance terminated by USER, pid = 76932
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (76932) as a result of ORA-1092
Sat Feb 22 11:04:19 2014
ORA-1092 : opitsk aborting process