12.1人工修改操作系统时间导致数据库异常

有客户数据库版本为12.1.0.1 版本RAC,突发发生重启,让协助分析原因
数据库alert日志报ORA-15064错误

Mon Apr 15 15:06:26 2019
WARNING: inbound connection timed out (ORA-3136)
Mon Apr 15 15:41:26 2019
NOTE: ASMB terminating
Mon Apr 15 15:41:26 2019
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_asmb_61426.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 1892 Serial number: 29
Mon Apr 15 15:41:26 2019
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_asmb_61426.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 1892 Serial number: 29
Mon Apr 15 15:41:26 2019
System state dump requested by (instance=1, osid=61426 (ASMB)), summary=[abnormal instance termination].
Mon Apr 15 15:41:26 2019
USER (ospid: 61426): terminating the instance due to error 15064
Mon Apr 15 15:41:26 2019
System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_diag_61287.trc
Mon Apr 15 15:41:27 2019
opiodr aborting process unknown ospid (1171) as a result of ORA-1092
Mon Apr 15 15:41:27 2019
ORA-1092 : opitsk aborting process

这里看,明显asmb异常导致数据库无法正常访问asm从而出现数据库crash的问题.

分析asm日志

Mon Apr 15 15:41:26 2019
WARNING: client [+ASM1:+ASM] not responsive for 2069s; state=0x1. pid 23155
NOTE: umbilicus traces dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_gen0_23050.trc
WARNING: client [orcl1:orcl] not responsive for 2069s; state=0x1. killing pid 61436
NOTE: umbilicus traces dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_gen0_23050.trc
WARNING: fencing client [orcl1:orcl] after 2069 seconds (mbr 2)
WARNING: client [-MGMTDB:_mgmtdb] not responsive for 2070s; state=0x1. killing pid 24026
NOTE: umbilicus traces dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_gen0_23050.trc
WARNING: fencing client [-MGMTDB:_mgmtdb] after 2070 seconds (mbr 1)
Mon Apr 15 15:41:26 2019
NOTE: cleaned up ASM client -MGMTDB:_mgmtdb
NOTE: cleaned up ASM client orcl1:orcl
Mon Apr 15 15:41:43 2019
NOTE: Standard client -MGMTDB:_mgmtdb registered, osid 183707, mbr 0x1 (reg:1371965153)
Mon Apr 15 15:42:16 2019
NOTE: Standard client orcl1:orcl registered, osid 184063, mbr 0x2 (reg:2088418628)
Mon Apr 15 15:44:30 2019
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.

asm日志中和mos中的GEN0 terminating the ASM instance due to error 15082 (文档 ID 2096988.1)描述比较匹配.根据客户反馈,他们使用ntp进行修改了时间,基本上可以确定是由于oracle的Bug 19032250(在12.1.0.2中修复)在ntp修改时间跨度过大触发的相关问题(人工直接修改时间也可能出现类似问题)

对于rac修改时间建议
1. 如果时间慢了,关闭数据库和集群直接把时间向前调整,启动集群和数据库
2. 如果时间快了,关闭数据库和集群等实际时间过关闭集群和库的时间之后,再往回调整时间,启动集群和数据库

tab$被恶意删除sys用户之外记录

数据库open成功,但是alert日志报大量ORA-00600错误

Sun Apr 14 14:30:46 2019
SMCO started with pid=53, OS id=6761
Completed: ALTER DATABASE OPEN /* db agent *//* {1:65047:2} */
Sun Apr 14 14:30:49 2019
Starting background process CJQ0
Sun Apr 14 14:30:49 2019
CJQ0 started with pid=54, OS id=6776
Setting Resource Manager plan SCHEDULER[0x32DF]:DEFAULT_MAINTENANCE_PLAN via scheduler window
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Sun Apr 14 14:30:52 2019
Starting background process VKRM
Sun Apr 14 14:30:52 2019
VKRM started with pid=37, OS id=6809
Sun Apr 14 14:30:54 2019
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_6811.trc  (incident=288633):
ORA-00600: internal error code, arguments: [kkpo_rcinfo_defstg:delseg], [84638], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288633/xifenfei1_j000_6811_i288633.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_6811.trc:
ORA-00600: internal error code, arguments: [kkpo_rcinfo_defstg:delseg], [84638], [], [], [], [], [], [], [], [], [], []
ORA-06512: at "APEX_030200.WWV_FLOW_MAIL", line 695
ORA-06512: at line 1
Sun Apr 14 14:30:57 2019
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_7491.trc  (incident=288658):
ORA-00600: 内部错误代码, 参数: [16659], [kqldtu], [INS], [0], [206196], [], [], [], [], [], [], []
Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288658/xifenfei1_j000_7491_i288658.trc
Sun Apr 14 14:34:10 2019
Dumping diagnostic data in directory=[cdmp_20190414143410], requested by (instance=1, osid=7491 (J000)), summary=[incident=288658].
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_7491.trc:
ORA-00600: 内部错误代码, 参数: [16659], [kqldtu], [INS], [0], [206196], [], [], [], [], [], [], []
ORA-06512: 在 "WEBCSMS.P_YGERROR", line 3
Sun Apr 14 14:39:08 2019
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_8515.trc  (incident=288593):
ORA-00600: 内部错误代码, 参数: [kdfReserveSingle_1], [0], [65280], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288593/xifenfei1_j000_8515_i288593.trc
ORA-06512: 在 line 1
Sun Apr 14 14:52:14 2019
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j001_11062.trc  (incident=288705):
ORA-00600: internal error code, arguments: [16607], [0x3CFB04C90], [257], [9], [0x000000000], [], [], [], [], [], [], []
Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288705/xifenfei1_j001_11062_i288705.trc
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_16712.trc  (incident=288597):
ORA-00600: 内部错误代码, 参数: [16607], [0x3C7CEA678], [1281], [9], [0x000000000], [], [], [], [], [], [], []
Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288597/xifenfei1_ora_16712_i288597.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

报错比较多,客户还反馈登录数据库之后,发现所有的表都丢失。第一反应可能数据字典损坏了,然后让客户查看备库,现在dg的备库也一样表都丢失了,进一步确认字典可能异常,让客户提供system文件进行本地分析.发现DBMS_SUPPORT_DBMONITOR触发器调用DBMS_SUPPORT_DBMONITORP存储过程,和警告:互联网中有oracle介质被注入恶意程序导致—ORA-600 16703中的名称非常类似,但是有点不一样,以前的恶意脚本中都是被注入并且触发之后,数据库无法正常启动,这次数据库能够正常open成功.分析恶意脚本,确认原因
1
2
3
确实这次的恶意脚本是在2016年8月份被创建在库中,在600天之后重启被触发,而且是删除非sys的tab$中记录.知道了恶意脚本的源头,那恢复就比较容易,直接通过批量bbed程序对tab$反删除可以实现比较完美恢复.原则上这样的故障可以实现数据库完美恢复,原库继续使用.

ORACLE Instance XFF (pid = 18) – Error 600 encountered while recovering transaction

分享一次由于一个表异常导致数据库报类似:ORACLE Instance XFF (pid = 18) – Error 600 encountered while recovering transaction故障的案例
一个10.2.0.4的数据库,正常运行的库突然出现如下错误

Sun Apr 07 11:07:12 2019
Thread 1 advanced to log sequence 602883 (LGWR switch)
  Current log# 3 seq# 602883 mem# 0: L:\ORADATA\XFF\REDO03.LOG
Sun Apr 07 11:10:38 2019
Thread 1 advanced to log sequence 602884 (LGWR switch)
  Current log# 1 seq# 602884 mem# 0: L:\ORADATA\XFF\REDO01.LOG
Sun Apr 07 11:11:56 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\udump\XFF_ora_22956.trc:
ORA-00600: 内部错误代码, 参数: [ktspgfb-1], [], [], [], [], [], [], []
Sun Apr 07 11:12:46 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\udump\XFF_ora_27408.trc:
ORA-00600: 内部错误代码, 参数: [kcbnew_3], [0], [1], [168354056], [], [], [], []
Sun Apr 07 11:13:57 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\udump\XFF_ora_6632.trc:
ORA-00600: 内部错误代码, 参数: [ktspgfb-1], [], [], [], [], [], [], []

过一段时间报,然后实例直接crash

Tue Apr 09 07:47:35 2019
ORACLE Instance XFF (pid = 18) - Error 600 encountered while recovering transaction (1, 1) on object 113718002.
Tue Apr 09 07:47:35 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_smon_12948.trc:
ORA-00600: internal error code, arguments: [kcbgcur_3], [168454497], [8], [4], [0], [], [], []
Tue Apr 09 07:55:23 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_pmon_22652.trc:
ORA-00474: SMON process terminated with error
Tue Apr 09 07:55:24 2019
PMON: terminating instance due to error 474
Tue Apr 09 07:55:24 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_lgwr_28608.trc:
ORA-00474: SMON process terminated with error
Tue Apr 09 07:55:34 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_psp0_12544.trc:
ORA-00474: SMON process terminated with error
Tue Apr 09 07:55:34 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_j000_5216.trc:
ORA-00474: SMON process terminated with error
Tue Apr 09 07:55:35 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_ckpt_28204.trc:
ORA-00474: SMON process terminated with error
Tue Apr 09 07:55:36 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_mman_9320.trc:
ORA-00474: SMON process terminated with error
Tue Apr 09 07:55:44 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_q002_24384.trc:
ORA-00474: SMON process terminated with error
Tue Apr 09 07:55:53 2019
Errors in file c:\oracle\product\10.2.0\admin\XFF\bdump\XFF_reco_24124.trc:
ORA-00474: SMON process terminated with error

根据以上报错,数据库crash的原因是由于undo异常导致,通过对undo进行重建,解决掉异常undo,但是业务运行之后,一样的问题又重现,最后通过分析确认是对象异常导致

SQL> create table XFF.T_XIFENFEI_xff as select * from XFF.T_XIFENFEI;
create table XFF.T_XIFENFEI_xff as select * from XFF.T_XIFENFEI
                                                           *
ERROR at line 1:
ORA-00600: internal error code, arguments: [kcbz_check_objd_typ], [0], [0], [1], [], [], [], []
屏蔽相关block obj的check之后
SQL> create table XFF.T_XIFENFEI_xff as select * from XFF.T_XIFENFEI;
create table XFF.T_XIFENFEI_xff as select * from XFF.T_XIFENFEI
                                                           *
ERROR at line 1:
ORA-00600: internal error code, arguments: [ktspScanInit-l1], [], [], [], [],[], [], []

比较明显该表对象出现逻辑异常,通过基于rowid的方式对该表数据进行抽取

SQL> create table XFF.T_XIFENFEI_new
  2  as
  3  select * from XFF.T_XIFENFEI where 1=0;
Table created.
SQL> set serveroutput on
SQL> set concat off
SQL> DECLARE
  2   nrows number;
  3   rid rowid;
  4   dobj number;
  5   ROWSPERBLOCK number;
  6  BEGIN
  7   ROWSPERBLOCK:=1000;
  8   nrows:=0;
  9   select data_object_id  into dobj
 10   from dba_objects
 11   where owner = 'XFF'
 12   and object_name = 'T_XIFENFEI'
 13   ;
 14   for i in (select relative_fno, block_id, block_id+blocks-1 totblocks
 15             from dba_extents
 16             where owner = 'XFF'
 17               and segment_name = 'T_XIFENFEI'
 18            order by extent_id)
 19   loop
 20     for br in i.block_id..i.totblocks loop
 21      for j in 1..ROWSPERBLOCK loop
 22      begin
 23        rid := dbms_rowid.ROWID_CREATE(1,dobj,i.relative_fno, br , j-1);
 24        insert into XFF.T_XIFENFEI_NEW
 25        select /*+ ROWID(A) */ *
 26        from XFF.T_XIFENFEI A
 27        where rowid = rid;
 28        if sql%rowcount = 1 then nrows:=nrows+1; end if;
 29        if (mod(nrows,10000)=0) then commit; end if;
 30      exception when others then null;
 31      end;
 32      end loop;
 33    end loop;
 34   end loop;
 35   COMMIT;
 36   dbms_output.put_line('Total rows: '||to_char(nrows));
 37  END;
 38  /
Total rows: 227000
PL/SQL procedure successfully completed.

再次观察数据库恢复正常,也不再crash和报错,恢复完成

linux资源限制导致数据库异常

一起由于liunx系统资源限制导致数据库无法启动案例分享
数据库启动报ORA-01157错

SQL> startup
ORACLE instance started.
Total System Global Area 3340451840 bytes
Fixed Size		    2217952 bytes
Variable Size		 1862273056 bytes
Database Buffers	 1459617792 bytes
Redo Buffers		   16343040 bytes
Database mounted.
ORA-01157: cannot identify/lock data file 5 - see DBWR trace file
ORA-01110: data file 5: '/home/oracle/oradata/XIFENFEI.dbf'

该错误一般是由于文件丢失或者路径错误导致

alert日志显示

Sun Apr 07 20:57:03 2019
ALTER DATABASE OPEN
Sun Apr 07 20:57:03 2019
Errors in file /dbdata/oracle/diag/rdbms/orcl/orcl/trace/orcl_dbw0_2681.trc:
ORA-01157: cannot identify/lock data file 5 - see DBWR trace file
ORA-01110: data file 5: '/home/oracle/oradata/XIFENFEI.dbf'
ORA-27092: size of file exceeds file size limit of the process
Additional information: 262144
Additional information: 262145
Errors in file /dbdata/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2802.trc:
ORA-01157: cannot identify/lock data file 5 - see DBWR trace file
ORA-01110: data file 5: '/home/oracle/oradata/XIFENFEI.dbf'
ORA-1157 signalled during: ALTER DATABASE OPEN...
Sun Apr 07 20:57:04 2019
Errors in file /dbdata/oracle/diag/rdbms/orcl/orcl/trace/orcl_m000_2804.trc  (incident=38578):
ORA-00600: internal error code, arguments: [kcidr_io_check_common_6], [10],
     [/home/oracle/oradata/XIFENFEI.dbf], [8192], [2], [5], [], [], [], [], [], []
ORA-27092: size of file exceeds file size limit of the process

这里看到提示ORA-27092: size of file exceeds file size limit of the process
查看系统limit配置

[oracle@XFF ~]$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) 640000
scheduling priority             (-e) 0
file size               (blocks, -f) 2097152
pending signals                 (-i) 128489
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 131072
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 131072
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

一般操作系统block size为1k,这里限制文件大小为2097152=(2G)
查看文件

[oracle@XFF ~]$ ls -l /home/oracle/oradata/XIFENFEI.dbf
-rw-r-----. 1 oracle oinstall 2147491840 Apr  7 19:04 /home/oracle/oradata/XIFENFEI.dbf

文件大小为2097160>2097152,导致异常

设置系统对文件大小限制2097152kb

[root@XFF ~]# ulimit -f 102400000
[root@XFF ~]# su - oracle
[oracle@XFF ~]$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) 640000
scheduling priority             (-e) 0
file size               (blocks, -f) 102400000
pending signals                 (-i) 128489
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 131072
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 131072
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

重启数据库,open成功

SQL> shutdown immediate;
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area 3340451840 bytes
Fixed Size		    2217952 bytes
Variable Size		 1862273056 bytes
Database Buffers	 1459617792 bytes
Redo Buffers		   16343040 bytes
Database mounted.
Database opened.

sql plan baseline简单介绍

Oracle 11g开始,提供了一种新的固定执行计划的方法,即SQL plan baseline,中文名SQL执行计划基线(简称基线),可以认为是OUTLINE(大纲)或者SQL PROFILE的改进版本,基本上它的主要作用可以归纳为如下两个:
1、稳定给定SQL语句的执行计划,防止执行环境或对象统计信息等因子的改变对SQL语句的执行计划产生影响
2、减少数据库中出现SQL语句性能退化的概率,理论上不允许一条语句切换到一个比已经执行过的执行计划慢很多的新的执行计划上(可以通过OPTIMIZER_USE_SQL_PLAN_BASELINE实现)
3、sql baseline对于sql 大小写,sql空格可以生效,但是对于非绑定变量sql,如果使用不同变量无法生效(无force_matching功能)
确认当前无sql baseline启用

SQL> select sql_handle,plan_name,dbms_lob.substr(sql_text,60,1) sql_text,ACCEPTED from dba_sql_plan_baselines;
no rows selected

创建场景
模拟一个表有index,如果再不强制的情况下,查询直接使用index,但是我这边要通过sql baseline模拟使用走全表扫描,实现不修改sql的情况下直接修改执行计划

SQL> create table t_xifenfei tablespace users as select * from dba_objects;
Table created.
SQL> create index i_xifenfei on t_xifenfei(object_id) tablespace users;
Index created.
SQL> execute dbms_stats.gather_table_stats('SYS','T_XIFENFEI',CASCADE=>TRUE);
PL/SQL procedure successfully completed.
SQL> select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100;
OBJECT_NAME
--------------------------------------------------------------------------------------------------------------
ORA$BASE
SQL> SELECT * FROM table(dbms_xplan.display_cursor);
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------
SQL_ID  b9hj14ntjgmtr, child number 0
-------------------------------------
select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100
Plan hash value: 1926396081
------------------------------------------------------------------------------------------
| Id  | Operation                   | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |            |       |       |     2 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T_XIFENFEI |     1 |    30 |     2   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | I_XIFENFEI |     1 |       |     1   (0)| 00:00:01 |
------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("OBJECT_ID"=100)
19 rows selected.
SQL> select /*+FULL(T_XIFENFEI)*/OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100;
OBJECT_NAME
--------------------------------------------------------------------------------------------------------------
ORA$BASE
SQL> SELECT * FROM table(dbms_xplan.display_cursor);
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------
SQL_ID  aqgv7stwu6w5t, child number 0
-------------------------------------
select /*+FULL(T_XIFENFEI)*/OBJECT_NAME from T_XIFENFEI where
OBJECT_ID=100
Plan hash value: 548923532
--------------------------------------------------------------------------------
| Id  | Operation         | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |            |       |       |   349 (100)|          |
|*  1 |  TABLE ACCESS FULL| T_XIFENFEI |     1 |    30 |   349   (1)| 00:00:05 |
--------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("OBJECT_ID"=100)
19 rows selected.

从shared pool->library cache中直接加载sql plan baseline

SQL> set serveroutput on
SQL>  DECLARE
  2      ret PLS_INTEGER;
  3    BEGIN
  4      ret := dbms_spm.load_plans_from_cursor_cache(sql_id          => 'b9hj14ntjgmtr',
  5                                                   plan_hash_value => null);
  6      dbms_output.put_line(ret || ' SQL plan baseline(s) created');
  7    END;
  8    /
1 SQL plan baseline(s) created
PL/SQL procedure successfully completed.
SQL> select sql_handle,plan_name,dbms_lob.substr(sql_text,60,1) sql_text,ACCEPTED from dba_sql_plan_baselines;
SQL_HANDLE                     PLAN_NAME
------------------------------ ------------------------------
SQL_TEXT
--------------------------------------------------------------------------------------------------------------
ACC
---
SQL_ed6b78bdb7b643ad           SQL_PLAN_fuuvsrqvvchxd04acd9ab
select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100
YES
SQL> select * from table(dbms_xplan.display_cursor('b9hj14ntjgmtr','',''));
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------
SQL_ID  b9hj14ntjgmtr, child number 1
-------------------------------------
select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100
Plan hash value: 1926396081
------------------------------------------------------------------------------------------
| Id  | Operation                   | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |            |       |       |     2 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T_XIFENFEI |     1 |    30 |     2   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | I_XIFENFEI |     1 |       |     1   (0)| 00:00:01 |
------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("OBJECT_ID"=100)
Note
-----
   - SQL plan baseline SQL_PLAN_fuuvsrqvvchxd04acd9ab used for this statement
23 rows selected.

利用第一个baseline的sql_handle创建新执行计划的baseline

SQL> set serveroutput on
SQL> DECLARE
  2    ret pls_integer;
  3   begin
  4   ret := DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE (
  5   sql_id=>'aqgv7stwu6w5t',
  6   plan_hash_value=>548923532,sql_handle=>'SQL_ed6b78bdb7b643ad'
  7   );
  8  dbms_output.put_line(ret || ' SQL plan baseline(s) created');
  9   end;
 10   /
1 SQL plan baseline(s) created
PL/SQL procedure successfully completed.
SQL>  select sql_handle,plan_name,dbms_lob.substr(sql_text,60,1) sql_text,ACCEPTED from dba_sql_plan_baselines;
SQL_HANDLE                     PLAN_NAME
------------------------------ ------------------------------
SQL_TEXT
--------------------------------------------------------------------------------------------------------------
ACC
---
SQL_ed6b78bdb7b643ad           SQL_PLAN_fuuvsrqvvchxd04acd9ab
select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100
YES
SQL_ed6b78bdb7b643ad           SQL_PLAN_fuuvsrqvvchxdf0c521d1
select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100
YES

删除第一个baseline

SQL> set serveroutput on
SQL> DECLARE
  2    ret pls_integer;
  3   begin
  4   ret := DBMS_SPM.drop_sql_plan_baseline (  sql_handle=>'SQL_ed6b78bdb7b643ad'
     ,plan_name=>'SQL_PLAN_fuuvsrqvvchxd04acd9ab');
  5  dbms_output.put_line(ret || ' SQL plan baseline(s) created');
  6   end;
  7   /
1 SQL plan baseline(s) created
PL/SQL procedure successfully completed.
SQL> select sql_handle,plan_name,dbms_lob.substr(sql_text,60,1) sql_text,ACCEPTED from dba_sql_plan_baselines;
SQL_HANDLE                     PLAN_NAME
------------------------------ ------------------------------
SQL_TEXT
--------------------------------------------------------------------------------------------------------------
ACC
---
SQL_ed6b78bdb7b643ad           SQL_PLAN_fuuvsrqvvchxdf0c521d1
select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100
YES

验证baseline生效,实现sql语句执行计划的改变

SQL> alter system flush shared_pool;
System altered.
SQL> select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100;
OBJECT_NAME
--------------------------------------------------------------------------------------------------------------
ORA$BASE
SQL>  select * from table(dbms_xplan.display_cursor('b9hj14ntjgmtr','',''));
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------
SQL_ID  b9hj14ntjgmtr, child number 1
-------------------------------------
select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100
Plan hash value: 548923532
--------------------------------------------------------------------------------
| Id  | Operation         | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |            |       |       |   349 (100)|          |
|*  1 |  TABLE ACCESS FULL| T_XIFENFEI |     1 |    30 |   349   (1)| 00:00:05 |
--------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("OBJECT_ID"=100)
Note
-----
   - SQL plan baseline SQL_PLAN_fuuvsrqvvchxdf0c521d1 used for this statement
22 rows selected.

利用coe脚本利用baseline快速绑定sql执行计划

SQL> set serveroutput on
SQL> DECLARE
  2    ret pls_integer;
  3   begin
  4   ret := DBMS_SPM.drop_sql_plan_baseline (  sql_handle=>'SQL_ed6b78bdb7b643ad',
      plan_name=>'SQL_PLAN_fuuvsrqvvchxdf0c521d1');
  5  dbms_output.put_line(ret || ' SQL plan baseline(s) created');
  6   end;
  7   /
1 SQL plan baseline(s) created
PL/SQL procedure successfully completed.
SQL> select sql_handle,plan_name,dbms_lob.substr(sql_text,60,1) sql_text,ACCEPTED from dba_sql_plan_baselines;
no rows selected
SQL> alter system flush shared_pool;
System altered.
SQL> select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100;
OBJECT_NAME
--------------------------------------------------------------------------------------------------------------
ORA$BASE
SQL> select /*+FULL(T_XIFENFEI)*/OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100;
OBJECT_NAME
--------------------------------------------------------------------------------------------------------------
ORA$BASE
SQL> select sql_id,sql_text from v$sql where sql_text like '%from T_XIFENFEI where OBJECT_ID=100%';
SQL_ID
-------------
SQL_TEXT
--------------------------------------------------------------------------------------------------------------
aqgv7stwu6w5t
select /*+FULL(T_XIFENFEI)*/OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100
7jdqvvnpxb9z5
select sql_id,sql_text from v$sql where sql_text like '%from T_XIFENFEI where OBJECT_ID=100%'
b9hj14ntjgmtr
select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100
SQL> select sql_id,PLAN_HASH_VALUE from v$sql where sql_id in('b9hj14ntjgmtr','aqgv7stwu6w5t');
SQL_ID        PLAN_HASH_VALUE
------------- ---------------
aqgv7stwu6w5t       548923532
b9hj14ntjgmtr      1926396081
SQL>  select * from table(dbms_xplan.display_cursor('aqgv7stwu6w5t','',''));
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------
SQL_ID  aqgv7stwu6w5t, child number 0
-------------------------------------
select /*+FULL(T_XIFENFEI)*/OBJECT_NAME from T_XIFENFEI where
OBJECT_ID=100
Plan hash value: 548923532
--------------------------------------------------------------------------------
| Id  | Operation         | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |            |       |       |   349 (100)|          |
|*  1 |  TABLE ACCESS FULL| T_XIFENFEI |     1 |    30 |   349   (1)| 00:00:05 |
--------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("OBJECT_ID"=100)
19 rows selected.
SQL>  select * from table(dbms_xplan.display_cursor('b9hj14ntjgmtr','',''));
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------
SQL_ID  b9hj14ntjgmtr, child number 0
-------------------------------------
select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100
Plan hash value: 1926396081
------------------------------------------------------------------------------------------
| Id  | Operation                   | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |            |       |       |     2 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T_XIFENFEI |     1 |    30 |     2   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | I_XIFENFEI |     1 |       |     1   (0)| 00:00:01 |
------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("OBJECT_ID"=100)
19 rows selected.
SQL> @coe_load_sql_baseline.sql
Parameter 1:
ORIGINAL_SQL_ID (required)
Enter value for 1: b9hj14ntjgmtr
Parameter 2:
MODIFIED_SQL_ID (required)
Enter value for 2: aqgv7stwu6w5t
     PLAN_HASH_VALUE          AVG_ET_SECS
-------------------- --------------------
           548923532                 .003
Parameter 3:
PLAN_HASH_VALUE (required)
Enter value for 3: 548923532
SQL> select sql_handle,plan_name,dbms_lob.substr(sql_text,60,1) sql_text,ACCEPTED from dba_sql_plan_baselines;
SQL_HANDLE                     PLAN_NAME
------------------------------ ------------------------------
SQL_TEXT
--------------------------------------------------------------------------------
ACC
---
SQL_ed6b78bdb7b643ad           SQL_PLAN_fuuvsrqvvchxdf0c521d1
select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100
YES
SQL> alter system flush shared_pool ;
System altered.
SQL> select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100;
OBJECT_NAME
--------------------------------------------------------------------------------
ORA$BASE
SQL>  select * from table(dbms_xplan.display_cursor('b9hj14ntjgmtr','',''));
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
SQL_ID  b9hj14ntjgmtr, child number 1
-------------------------------------
select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100
Plan hash value: 548923532
--------------------------------------------------------------------------------
| Id  | Operation         | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |            |       |       |   349 (100)|          |
|*  1 |  TABLE ACCESS FULL| T_XIFENFEI |     1 |    30 |   349   (1)| 00:00:05 |
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("OBJECT_ID"=100)
Note
-----
   - SQL plan baseline SQL_PLAN_fuuvsrqvvchxdf0c521d1 used for this statement
22 rows selected.

sql空格和大小写改变不影响baseline效果

SQL>  select * from table(dbms_xplan.display_cursor('dwfxd7x6kwx6u','',''));
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
SQL_ID  dwfxd7x6kwx6u, child number 1
-------------------------------------
select     OBJECT_NAME from T_XIFENFEI where OBJECT_ID=100
Plan hash value: 548923532
--------------------------------------------------------------------------------
| Id  | Operation         | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |            |       |       |   349 (100)|          |
|*  1 |  TABLE ACCESS FULL| T_XIFENFEI |     1 |    30 |   349   (1)| 00:00:05 |
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("OBJECT_ID"=100)
Note
-----
   - SQL plan baseline SQL_PLAN_fuuvsrqvvchxdf0c521d1 used for this statement
22 rows selected.
SQL> select     OBJECT_NAME from T_xifenfei where OBJECT_ID=100;
OBJECT_NAME
--------------------------------------------------------------------------------
ORA$BASE
SQL> select sql_id,sql_text from v$sql where sql_text like '%from T_xifenfei where OBJECT_ID=100%';
SQL_ID
-------------
SQL_TEXT
--------------------------------------------------------------------------------
5spn2x6ac44af
select sql_id,sql_text from v$sql where sql_text like '%from T_xifenfei where OB
JECT_ID=100%'
8tytmh8r6w80n
select     OBJECT_NAME from T_xifenfei where OBJECT_ID=100
SQL>  select * from table(dbms_xplan.display_cursor('8tytmh8r6w80n','',''));
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
SQL_ID  8tytmh8r6w80n, child number 1
-------------------------------------
select     OBJECT_NAME from T_xifenfei where OBJECT_ID=100
Plan hash value: 548923532
--------------------------------------------------------------------------------
| Id  | Operation         | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |            |       |       |   349 (100)|          |
|*  1 |  TABLE ACCESS FULL| T_XIFENFEI |     1 |    30 |   349   (1)| 00:00:05 |
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("OBJECT_ID"=100)
Note
-----
   - SQL plan baseline SQL_PLAN_fuuvsrqvvchxdf0c521d1 used for this statement
22 rows selected.

但是sql变量不一样导致baseline失效

SQL>  select * from table(dbms_xplan.display_cursor('fp9u8wkp5cuw1','',''));
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
SQL_ID  fp9u8wkp5cuw1, child number 0
-------------------------------------
select OBJECT_NAME from T_XIFENFEI where OBJECT_ID=101
Plan hash value: 1926396081
------------------------------------------------------------------------------------------
| Id  | Operation                   | Name       | Rows  | Bytes | Cost (%CPU)|
Time     |
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |            |       |       |     2 (100)|         |
|   1 |  TABLE ACCESS BY INDEX ROWID| T_XIFENFEI |     1 |    30 |     2   (0)|00:00:01 |
|*  2 |   INDEX RANGE SCAN          | I_XIFENFEI |     1 |       |     1   (0)|
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
00:00:01 |
------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("OBJECT_ID"=101)
19 rows selected.

非归档数据库异常恢复一例

由于存储故障,数据库为非归档模式,通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)收集信息确认数据库redo异常
2


Thu Mar 28 11:36:13 2019
ALTER DATABASE RECOVER    CONTINUE DEFAULT
Media Recovery Log /u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397869_%u_.arc
Thu Mar 28 11:36:13 2019
Errors with log /u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397869_%u_.arc
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr00_17611.trc:
ORA-00308:cannot open archived log
    '/u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397869_%u_.arc'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER    CONTINUE DEFAULT
Media Recovery Log /u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397869_%u_.arc
Errors with log /u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397869_%u_.arc
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr00_17611.trc:
ORA-00308:cannot open archived log
   '/u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397869_%u_.arc'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
Thu Mar 28 11:38:44 2019
ALTER DATABASE RECOVER  datafile 5,6
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 3 Seq 5397870 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/orcl/redo03.log
ORA-279 signalled during: ALTER DATABASE RECOVER  datafile 5,6  ...
Thu Mar 28 11:39:08 2019
ALTER DATABASE RECOVER    CONTINUE DEFAULT
Media Recovery Log /u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397870_%u_.arc
Errors with log /u01/app/oracle/flash_recovery_area/ORCL/archivelog/2019_03_28/o1_mf_1_5397870_%u_.arc
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER CANCEL
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_17605.trc  (incident=365041):
ORA-00600: internal error code, arguments: [3051], [82], [], [], [], [], [], [], [], [], [], []
ORA-600 signalled during: ALTER DATABASE RECOVER CANCEL ...

通过屏蔽一致性,强制打开库,报kgegpa错误,实例启动失败

Database Characterset is ZHS16GBK
No Resource Manager plan active
Exception[type:SIGSEGV, Address not mapped to object][ADDR:0x319C0CF3] [PC:0x2297740, kgegpa()+40] [flags: 0x0, count:1]
Exception[type:SIGSEGV, Address not mapped to object][ADDR:0x319C0CF3] [PC:0x229596B, kgebse()+279][flags: 0x2, count:2]
Exception[type:SIGSEGV, Address not mapped to object][ADDR:0x319C0CF3] [PC:0x229596B, kgebse()+279][flags: 0x2, count:2]
Thu Mar 28 11:43:15 2019
PMON (ospid: 17939): terminating the instance due to error 397
Instance terminated by PMON, pid = 17939

处理上述错误相关undo,启动数据库报ORA-00600 4193,ORA-00600 4137, ORA-00600 6006

Thu Mar 28 11:50:37 2019
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p001_18267.trc  (incident=373059):
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_373059/orcl_p001_18267_i373059.trc
Stopping background process MMON
Trace dumping is performing id=[cdmp_20190328115038]
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_18247.trc  (incident=372995):
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_372995/orcl_smon_18247_i372995.trc
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_18247.trc  (incident=372996):
ORA-00600: internal error code, arguments: [4137], [34.22.4206895], [0], [0], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_372996/orcl_smon_18247_i372996.trc
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_18263.trc  (incident=373044):
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_373044/orcl_ora_18263_i373044.trc
ORACLE Instance orcl (pid = 16) - Error 600 encountered while recovering transaction (34, 22).
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_18247.trc:
ORA-00600: internal error code, arguments: [4137], [34.22.4206895], [0], [0], [], [], [], [], [], [], [], []

通过重建undo,相关报错消失,安排数据导出重建库

ORA-00322 ORA-00312恢复

数据库启动ORA-00322 ORA-00312错误,无法正常启动

Fri Mar 29 17:44:20 2019
ALTER DATABASE RECOVER  datafile 1
Media Recovery Start
Serial Media Recovery started
Media Recovery failed with error 19909
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 1  ...
Fri Mar 29 17:44:20 2019
Errors in file c:\app\administrator\diag\rdbms\xff\xff\trace\xff_m000_5392.trc:
ORA-00322: log 1 of thread 1 is not current copy
ORA-00312: online log 1 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\xff\REDO01.LOG'
Errors in file c:\app\administrator\diag\rdbms\xff\xff\trace\xff_m000_5392.trc:
ORA-00322: log 2 of thread 1 is not current copy
ORA-00312: online log 2 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\xff\REDO02.LOG'
Errors in file c:\app\administrator\diag\rdbms\xff\xff\trace\xff_m000_5392.trc:
ORA-00322: log 3 of thread 1 is not current copy
ORA-00312: online log 3 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\xff\REDO03.LOG'

人工指定redo应用,报ORA-00600 3051错误

Fri Mar 29 17:56:33 2019
ALTER DATABASE RECOVER  datafile 1
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 2 Seq 27542 Reading mem 0
  Mem# 0: D:\XIFENFEI\REDO02.LOG
ORA-279 signalled during: ALTER DATABASE RECOVER  datafile 1  ...
Fri Mar 29 17:56:49 2019
ALTER DATABASE RECOVER    LOGFILE 'D:\xifenfei\REDO02.log'
Media Recovery Log D:\xifenfei\REDO02.log
Errors with log D:\xifenfei\REDO02.log
ORA-363 signalled during: ALTER DATABASE RECOVER    LOGFILE 'D:\xifenfei\REDO02.log'  ...
ALTER DATABASE RECOVER CANCEL
Errors in file c:\app\administrator\diag\rdbms\xff\xff\trace\xff_ora_8532.trc  (incident=147928):
ORA-00600: ??????, ??: [3051], [82], [], [], [], [], [], [], [], [], [], []
Incident details in: c:\app\administrator\diag\rdbms\xff\xff\incident\incdir_147928\xff_ora_8532_i147928.trc

比较明显redo无法正常应用,通过屏蔽数据库一致性,强制拉库

Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0
  Mem# 0: D:\XIFENFEI\REDO01.LOG
Block recovery stopped at EOT rba 1.76.16
Block recovery completed at rba 1.76.16, scn 0.1073742057
Doing block recovery for file 3 block 272
Resuming block recovery (PMON) for file 3 block 272
Block recovery from logseq 1, block 72 to scn 1073742051
Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0
  Mem# 0: D:\XIFENFEI\REDO01.LOG
Block recovery completed at rba 1.72.16, scn 0.1073742052
Errors in file c:\app\administrator\diag\rdbms\xff\xff\trace\xff_smon_5144.trc:
ORA-01595: error freeing extent (16) of rollback segment (10))
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Fri Mar 29 17:59:12 2019
Errors in file c:\app\administrator\diag\rdbms\xff\xff\trace\xff_mmon_13928.trc  (incident=149097):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: c:\app\administrator\diag\rdbms\xff\xff\incident\incdir_149097\xff_mmon_13928_i149097.trc
Fri Mar 29 17:59:12 2019
Trace dumping is performing id=[cdmp_20190329175912]
Completed: alter database open resetlogs

通过重建undo,数据库open正常,安排导出数据导入数据,恢复完成

利用VNCR修复CVE-2012-1675漏洞

随着对安全的重视,TNS Listener远程数据投毒漏洞(CVE-2012-1675)被很多单位要求进行整改,而且级别是高危
1


如果是11.2.0.4之前版本,特别是在rac环境中,如果要整改该问题相对比较麻烦,需要通过配置Oracle wallet来实现,配置比较复杂,而且还要重启实例,影响比较大,具体参考:Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle RAC (Doc ID 1340831.1),对于单机环境直接参考以前文章:Oracle Database Server ‘TNS Listener’远程数据投毒漏洞(CVE-2012-1675)的解决方案.从11.2.0.4开始Oracle引入了Valid Node Checking For Registration (VNCR)新特性,可以通过简单的配置即可完成该漏洞修复
在listener.ora文件中增加(grid/oracle用户)

VALID_NODE_CHECKING_REGISTRATION_LISTENER=ON
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN1=ON
REGISTRATION_INVITED_NODES_LISTENER_SCAN1=(hisdb1,hisdb2)

重启监听

LSNRCTL>set current_listener listener_name
LSNRCTL>stop
LSNRCTL>start

验证是否生效
1.在远程机器尝试远程注册

alter system set remote_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.115.15)(PORT=1521))' scope=memory;

2.观察监听日志
VNCR


类似这样证明已经生效.


补充说明

VALID_NODE_CHECKING_REGISTRATION_listener_name
Values:
OFF/0 - Disable VNCR//禁用VNCR,此功能不会对注册过来的service进行check;
ON/1/LOCAL - The default. Enable VNCR. All local machine IPs can register.
启用VNCR,默认只允许本机的所有IP的服务注册到本监听,可通过REGISTRATION_INVITED_NODES参数添加其他有必要的服务器;
SUBNET/2 - All machines in the subnet are allowed registration.//指定子网内的服务器可以注册过来
REGISTRATION_INVITED_NODES_listener-name
控制允许链接过来的节点,可以通过IP地址/主机名/网段来指定
Values are valid IPs, valid hosts, a subnet using CIDR notation (for ip4/6), or wildcard (*) for ipv4.
For example:REGISTRATION_INVITED_NODES_Listener=(net-vm1, 127.98.45.209, 127.42.5.*)
Note
that when an INVITED list is set, it will automatically include the
machine's local IP in the list. There is no need to include it.
--11.2.0.4和12c不一样之处
在12.1 RAC数据库上,listener的参数VALID_NODE_CHECKING_REGISTRATION_listener_name默认设置为SUBNET / 2,
即子网中的所有计算机都允许注册.所以12c默认不能解决CVE-2012-1675漏洞

参考文档
Valid Node Checking For Registration (VNCR) (Doc ID 1600630.1)
How to Enable VNCR on RAC Database to Register only Local Instances (Doc ID 1914282.1)

12C数据库遭遇ORA-600 16703

关于tab$异常的问题,我们在10G(10g数据库遭遇ORA-600 16703),11G(警告:互联网中有oracle介质被注入恶意程序导致—ORA-600 16703)中有发现,近期在12C版本中也发现该问题.再次提醒请注意检查安装介质的md5值和现在运行的库中是否有相关恶意程序
数据库版本
12c-tab-version


alert日志报错

************************************************************
Warning: The SCN headroom for this database is only 45 days!
************************************************************
************************************************************
Warning: The SCN headroom for this database is only 45 days!
************************************************************
Picked broadcast on commit scheme to generate SCNs
Fri Jan 25 11:34:55 2019
Starting background process TMON
Fri Jan 25 11:34:55 2019
TMON started with pid=104, OS id=47282
Fri Jan 25 11:34:55 2019
Thread 1 advanced to log sequence 11942 (thread open)
Thread 1 opened at log sequence 11942
Current log# 5 seq# 11942 mem# 0: +DATA/xifenfei/ONLINELOG/redo01_01.log
Successful open of redo thread 1
Fri Jan 25 11:34:55 2019
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan 25 11:34:55 2019
Network Resource Management enabled for Process LG00 (pid 451491) for Exadata I/O
Fri Jan 25 11:34:55 2019
SMON: enabling cache recovery
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_46443.trc (incident=394486):
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Fri Jan 25 11:34:56 2019
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_46443.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Fri Jan 25 11:34:56 2019
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_46443.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Fri Jan 25 11:34:56 2019
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_46443.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Fri Jan 25 11:34:56 2019
Error 704 happened during db open, shutting down database
USER (ospid: 46443): terminating the instance due to error 704
Fri Jan 25 11:34:57 2019
Instance terminated by USER, pid = 46443

处理思路基本上和其他两个版本一样,都是对tab$进行修复,然后直接open数据库,但是在这次的恢复中遇到ORA-00600 kdfReserveSingle_1错误
ORA-600 kdfReserveSingle_1

SQL> alter database open ;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280],
[], [], [], [], [], [], [], [], []
Process ID: 319852
Session ID: 2401 Serial number: 24555
---alert日志
Starting background process TMON
Fri Jan 25 20:51:41 2019
TMON started with pid=94, OS id=322554
Fri Jan 25 20:51:41 2019
Thread 1 advanced to log sequence 11943 (thread open)
Thread 1 opened at log sequence 11943
Current log# 6 seq# 11943 mem# 0: +DATA/xifenfei/ONLINELOG/redo02_01.log
Successful open of redo thread 1
Fri Jan 25 20:51:41 2019
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan 25 20:51:41 2019
Network Resource Management enabled for Process LG00 (pid 319754) for Exadata I/O
Fri Jan 25 20:51:41 2019
SMON: enabling cache recovery
Fri Jan 25 20:51:41 2019
[319852] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:88159754 end:88160264 diff:510 ms (0.5 seconds)
Verifying minimum file header compatibility (11g) for tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
Fri Jan 25 20:51:41 2019
SMON: enabling tx recovery
Fri Jan 25 20:51:41 2019
Database Characterset is ZHS16GBK
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_319852.trc (incident=466446):
ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Fri Jan 25 20:51:42 2019
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_319852.trc:
ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280], [], [], [], [], [], [], [], [], []
Fri Jan 25 20:51:42 2019
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_319852.trc:
ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 319852): terminating the instance due to error 600
Fri Jan 25 20:51:43 2019
Instance terminated by USER, pid = 319852
ORA-1092 signalled during: alter database open…
opiodr aborting process unknown ospid (319852) as a result of ORA-1092
Fri Jan 25 20:51:49 2019
ORA-1092 : opitsk aborting process

这个错误主要是由于12c的tab$和以前版本比有一些特殊,导致开始恢复不全,通过继续修复tab$之后数据库正常open.
相关文章汇总
tab$恢复错误汇总
12C数据库遭遇ORA-600 16703
10G数据库遭遇ORA-600 16703
ORA-600 16703故障解析—tab$表被清空
警告:互联网中有oracle介质被注入恶意程序导致—ORA-600 16703

SCN Compatibility问题汇总-2019年6月23日

最近有不少朋友咨询关于2019年6月23日之前数据库scn,数据库是否要升级或者打补丁等问题,这里对相关问题做一个汇总。
1. 是否一定要升级或者打补丁
不一定,以下几种情况可以不用处理
1) 数据库版本11.1.0.7.20+ /11.2.0.3.9+ /11.2.0.4+版本已经包含了该补丁,无需处理
2) 数据库版本全部低于1)中数据库版本
3) 你的数据库环境中不存在1和2中数据库dblink访问
4) 你的数据库环境中有1和2中的数据库dblink访问,但是通过通过dbms_scn设置,不让1中的数据库版本scn compatibility改变(都保持为1)

2. 到了2019年6月23日之后,数据库的scn发生什么改变
简单的说就是数据库每秒可以使用的scn变大了,距离天花板的scn更加大,出现数据库scn用完的概率大大降低

--compat            -- SCN compatibility value
--headroom_in_scn   -- Difference between current SCN and RSL
--headroom_in_sec   -- number of seconds it would take to reachRSL
SQL> set serveroutput on ;
SQL>  declare
  2   rsl number;
  3   headroom_in_scn number;
  4   headroom_in_sec number;
  5   cur_scn_compat number;
  6   max_scn_compat number;
  7   begin
  8   dbms_scn.GetCurrentSCNParams(rsl,headroom_in_scn,headroom_in_sec,cur_scn_compat,max_scn_compat);
  9   dbms_output.put_line('rsl=' || rsl);
 10   dbms_output.put_line('headroom_in_scn=' || headroom_in_scn);
 11   dbms_output.put_line('headroom_in_sec=' || headroom_in_sec);
 12   dbms_output.put_line('cur_scn_compat=' || cur_scn_compat);
 13   dbms_output.put_line('max_scn_compat=' || max_scn_compat);
 14   end;
 15   /
rsl=16424869609472
headroom_in_scn=16424867538319
headroom_in_sec=1002494356
cur_scn_compat=1
max_scn_compat=3
PL/SQL procedure successfully completed.
SQL> Alter Database Set SCN Compatibility 2;
Database altered.
SQL>  declare
  2   rsl number;
  3   headroom_in_scn number;
  4   headroom_in_sec number;
  5   cur_scn_compat number;
  6   max_scn_compat number;
  7   begin
  8   dbms_scn.GetCurrentSCNParams(rsl,headroom_in_scn,headroom_in_sec,cur_scn_compat,max_scn_compat);
  9   dbms_output.put_line('rsl=' || rsl);
 10   dbms_output.put_line('headroom_in_scn=' || headroom_in_scn);
 11   dbms_output.put_line('headroom_in_sec=' || headroom_in_sec);
 12   dbms_output.put_line('cur_scn_compat=' || cur_scn_compat);
 13   dbms_output.put_line('max_scn_compat=' || max_scn_compat);
 14   end;
 15   /
rsl=21792299122688
headroom_in_scn=21792297051479
headroom_in_sec=665048127
cur_scn_compat=2
max_scn_compat=3
PL/SQL procedure successfully completed.
SQL> Alter Database Set SCN Compatibility 3;
Database altered.
SQL>  declare
  2   rsl number;
  3   headroom_in_scn number;
  4   headroom_in_sec number;
  5   cur_scn_compat number;
  6   max_scn_compat number;
  7   begin
  8   dbms_scn.GetCurrentSCNParams(rsl,headroom_in_scn,headroom_in_sec,cur_scn_compat,max_scn_compat);
  9   dbms_output.put_line('rsl=' || rsl);
 10   dbms_output.put_line('headroom_in_scn=' || headroom_in_scn);
 11   dbms_output.put_line('headroom_in_sec=' || headroom_in_sec);
 12   dbms_output.put_line('cur_scn_compat=' || cur_scn_compat);
 13   dbms_output.put_line('max_scn_compat=' || max_scn_compat);
 14   end;
 15   /
rsl=34585263898624
headroom_in_scn=34585261822622
headroom_in_sec=351819476
cur_scn_compat=3
max_scn_compat=3
PL/SQL procedure successfully completed.

这里大概演示了数据库scn compatibility变化带来的相关变化,这里可以看出来每秒

3. 如何禁用/启用scn compatibility自动升级

SQL>  begin dbms_scn.DisableAutoRollover; end;
  2   /
PL/SQL procedure successfully completed.
SQL> declare
  2   EFFECTIVE_AUTO_ROLLOVER_TS date;
  3   TARGET_COMPAT number;
  4   IS_ENABLED boolean;
  5   begin
  6    dbms_scn.GETSCNAUTOROLLOVERPARAMS(EFFECTIVE_AUTO_ROLLOVER_TS,TARGET_COMPAT,IS_ENABLED);
  7    dbms_output.put_line('EFFECTIVE_AUTO_ROLLOVER_TS='||to_char(EFFECTIVE_AUTO_ROLLOVER_TS,'yyyy-mm-dd hh24:mi:ss'));
  8    dbms_output.put_line('TARGET_COMPAT=' || TARGET_COMPAT);
  9   if(IS_ENABLED)then
 10    dbms_output.put_line('IS_ENABLED IS TURE');
 11   else
 12    dbms_output.put_line('IS_ENABLED IS FALSE');
 13   end if;
 14   end;
 15   /
EFFECTIVE_AUTO_ROLLOVER_TS=2019-06-23 00:00:00
TARGET_COMPAT=3
IS_ENABLED IS FALSE
PL/SQL procedure successfully completed.
SQL>

4. scn compatibility手工调整

SQL> Alter Database Set SCN Compatibility 2;
Database altered.
SQL> Alter Database Set SCN Compatibility 3;
Database altered.
SQL> Alter Database Set SCN Compatibility 1;
Alter Database Set SCN Compatibility 1
*
ERROR at line 1:
ORA-01126: database must be mounted in this instance and not open in any instance
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.
Total System Global Area 4999609080 bytes
Fixed Size                  9145080 bytes
Variable Size            1040187392 bytes
Database Buffers         3942645760 bytes
Redo Buffers                7630848 bytes
Database mounted.
SQL>  Alter Database Set SCN Compatibility 1;
Database altered.
SQL> alter database open;
Database altered.
SQL>  set serveroutput on ;
SQL>  declare
  2   rsl number;
  3   headroom_in_scn number;
  4   headroom_in_sec number;
  5   cur_scn_compat number;
 max_scn_compat number;
  6    7   begin
  8   dbms_scn.GetCurrentSCNParams(rsl,headroom_in_scn,headroom_in_sec,cur_scn_compat,max_scn_compat);
  9   dbms_output.put_line('rsl=' || rsl);
 dbms_output.put_line('headroom_in_scn=' || headroom_in_scn);
 10   11   dbms_output.put_line('headroom_in_sec=' || headroom_in_sec);
 12   dbms_output.put_line('cur_scn_compat=' || cur_scn_compat);
 13   dbms_output.put_line('max_scn_compat=' || max_scn_compat);
 end;
 / 14   15
rsl=16425127591936
headroom_in_scn=16425125502261
headroom_in_sec=1002510101
cur_scn_compat=1
max_scn_compat=3
PL/SQL procedure successfully completed.

参考文档:
Recommended patching and actions for Oracle database versions 12.1.0.1, 11.2.0.3 and earlier-before June 2019(Doc ID 2335265.1)
Recommended patches and actions for Oracle databases versions 12.1.0.1, 11.2.0.3 and earlier–before June 2019(Doc ID 2361478.1)