ORA-27069: skgfdisp: 尝试在文件范围外执行 I/O

接到网友技术支持请求,win 2003 ntfs格式文件系统,Oracle 8.1.7版本,主机重启后,数据库无法正常启动,offline datafile 15,数据库open成功,但是datafile 无法正常online,报错为:ORA-27069: skgfdisp: attempt to do I/O beyond the range of the file,请求协助处理

SQL> recover datafile 'D:\ORACLE\ORADATA\ORCL\ZSF_DATA.DBF';
ORA-00283: 恢复会话因错误而取消
ORA-01115: 从文件 15 读取块时出现 IO 错误 (块 # 1030071)
ORA-01110: 数据文件 15: 'D:\ORACLE\ORADATA\ORCL\ZSF_DATA.DBF'
ORA-27069: skgfdisp: 尝试在文件范围外执行 I/O
OSD-04026: 无效的参数经过. (OS 1030071)

使用bbed,成功online datafile 15

Tue Oct 28 16:30:35 2014
ALTER DATABASE RECOVER  datafile 15
Tue Oct 28 16:30:35 2014
Media Recovery Datafile: 15
Media Recovery Start
Media Recovery Log
Recovery of Online Redo Log: Thread 1 Group 1 Seq 245110 Reading mem 0
  Mem# 0 errs 0: D:\ORACLE\ORADATA\ORCL\REDO03.LOG
Media Recovery failed with error 1115
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 15  ...
Tue Oct 28 16:32:53 2014
Shutting down instance (abort)
License high water mark = 6
Instance terminated by USER, pid = 1548
Starting up ORACLE RDBMS Version: 8.1.7.0.0.
System parameters with non-default values:
  processes                = 600
  shared_pool_size         = 52428800
  large_pool_size          = 20971520
  java_pool_size           = 20971520
  control_files            = D:\oracle\oradata\ORCL\control01.ctl, D:\oracle\oradata\ORCL\control02.ctl
  db_block_buffers         = 19200
  db_block_size            = 8192
  compatible               = 8.1.0
  log_buffer               = 32768
  log_checkpoint_interval  = 10000
  log_checkpoint_timeout   = 1800
  db_files                 = 1024
  db_file_multiblock_read_count= 8
  max_enabled_roles        = 30
  remote_login_passwordfile= EXCLUSIVE
  global_names             = TRUE
  distributed_transactions = 500
  instance_name            = ORCL
  service_names            = ORCL
  mts_dispatchers          = (PROTOCOL=TCP)(PRE=oracle.aurora.server.SGiopServer)
  open_links               = 4
  sort_area_size           = 65536
  sort_area_retained_size  = 65536
  db_name                  = ORCL
  open_cursors             = 500
  ifile                    = D:\oracle\admin\ORCL\pfile\init.ora
  os_authent_prefix        =
  job_queue_processes      = 4
  job_queue_interval       = 10
  parallel_max_servers     = 5
  background_dump_dest     = D:\oracle\admin\ORCL\bdump
  user_dump_dest           = D:\oracle\admin\ORCL\udump
  max_dump_file_size       = 10240
  oracle_trace_collection_name=
PMON started with pid=2
DBW0 started with pid=3
LGWR started with pid=4
CKPT started with pid=5
SMON started with pid=6
RECO started with pid=7
SNP0 started with pid=8
SNP1 started with pid=9
SNP2 started with pid=10
SNP3 started with pid=11
Tue Oct 28 16:33:01 2014
starting up 1 shared server(s) ...
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
Tue Oct 28 16:33:02 2014
ALTER DATABASE   MOUNT
Tue Oct 28 16:33:06 2014
Successful mount of redo thread 1, with mount id 1389958722.
Tue Oct 28 16:33:06 2014
Database mounted in Exclusive Mode.
Completed: ALTER DATABASE   MOUNT
Tue Oct 28 16:33:49 2014
ALTER DATABASE RECOVER  database until cancel
Tue Oct 28 16:33:49 2014
Media Recovery Start
Media Recovery Log
kcrrga: Warning.  Log sequence in archive filename wrapped
to fix length as indicated by %S in LOG_ARCHIVE_FORMAT.
Old log archive with same name might be overwritten.
ORA-279 signalled during: ALTER DATABASE RECOVER  database until cancel  ...
Tue Oct 28 16:34:03 2014
ALTER DATABASE RECOVER    LOGFILE 'D:\ORACLE\ORADATA\ORCL\REDO02.LOG'
Tue Oct 28 16:34:03 2014
Media Recovery Log D:\ORACLE\ORADATA\ORCL\REDO02.LOG
Incomplete recovery applied all redo ever generated.
Recovery completed through change %s139866389
Media Recovery Complete
Completed: ALTER DATABASE RECOVER    LOGFILE 'D:\ORACLE\ORADA
Tue Oct 28 16:34:29 2014
alter database datafile 15 online
Tue Oct 28 16:34:29 2014
Completed: alter database datafile 15 online
Tue Oct 28 16:34:36 2014
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 139866389
Tue Oct 28 16:34:38 2014
Thread 1 opened at log sequence 1
  Current log# 2 seq# 1 mem# 0: D:\ORACLE\ORADATA\ORCL\REDO02.LOG
Successful open of redo thread 1.
Tue Oct 28 16:34:38 2014
SMON: enabling cache recovery
Tue Oct 28 16:34:38 2014
Dictionary check beginning
Dictionary check complete
Tue Oct 28 16:34:39 2014
SMON: enabling tx recovery
Tue Oct 28 16:34:44 2014
Completed: alter database open resetlogs

数据库datafile 15 online成功后,客户操作业务继续发生ORA-600[ktsxs_add2]错误

Tue Oct 28 17:07:42 2014
Errors in file D:\oracle\admin\ORCL\udump\ORA02340.TRC:
ORA-00600: 内部错误代码,自变量: [ktsxs_add2], [14], [15], [42534], [5732], [5733], [], []
Tue Oct 28 17:07:53 2014
Errors in file D:\oracle\admin\ORCL\udump\ORA02340.TRC:
ORA-00600: 内部错误代码,自变量: [ktsxs_add2], [14], [15], [42534], [5732], [5733], [], []
Tue Oct 28 17:08:03 2014
Errors in file D:\oracle\admin\ORCL\udump\ORA02340.TRC:
ORA-00600: 内部错误代码,自变量: [ktsxs_add2], [14], [15], [42534], [5732], [5733], [], []
Tue Oct 28 17:08:16 2014
Errors in file D:\oracle\admin\ORCL\udump\ORA02340.TRC:
ORA-00600: 内部错误代码,自变量: [ktsxs_add2], [14], [15], [42534], [5732], [5733], [], []
Tue Oct 28 17:08:23 2014
Errors in file D:\oracle\admin\ORCL\udump\ORA02308.TRC:
ORA-00600: 内部错误代码,自变量: [ktsxs_add2], [14], [15], [42534], [5732], [5733], [], []
Tue Oct 28 17:08:31 2014
Errors in file D:\oracle\admin\ORCL\udump\ORA02340.TRC:
ORA-00600: 内部错误代码,自变量: [ktsxs_add2], [14], [15], [42534], [5732], [5733], [], []
Tue Oct 28 17:08:38 2014
Errors in file D:\oracle\admin\ORCL\udump\ORA02308.TRC:
ORA-00600: 内部错误代码,自变量: [ktsxs_add2], [14], [15], [42534], [5732], [5733], [], []

通过分析相关日志发现是insert插入表报错,很好理解,该库的datafile 15已经超过了系统的限制,现在继续插入数据,因此报错,查询可能异常对象

SQL> col segment_name for a20
SQL> SELECT distinct OWNER, SEGMENT_NAME, SEGMENT_TYPE, A.PARTITION_NAME
  2    FROM DBA_EXTENTS A
  3   WHERE FILE_ID = 15
  4     AND 1030071 <= BLOCK_ID;
OWNER                          SEGMENT_NAME         SEGMENT_TYPE
------------------------------ -------------------- ------------------
PARTITION_NAME
------------------------------
ZSF                            DETAIL               TABLE
ZSF                            DETAIL1              INDEX
ZSF                            DETAIL2              INDEX
OWNER                          SEGMENT_NAME         SEGMENT_TYPE
------------------------------ -------------------- ------------------
PARTITION_NAME
------------------------------
ZSF                            DETAIL3              INDEX
ZSF                            DETAIL4              INDEX
ZSF                            FK_RECI_ORD          INDEX
OWNER                          SEGMENT_NAME         SEGMENT_TYPE
------------------------------ -------------------- ------------------
PARTITION_NAME
------------------------------
ZSF                            PREPAY1              INDEX
ZSF                            RECEDETAIL1          INDEX

创建新表空间

Create tablespace zsf_new datafile  'D:\ORACLE\ORADATA\ORCL\ZSF_DATA_new01.dbf' size 4096m;
alter tablespace zsf_new add datafile 'D:\ORACLE\ORADATA\ORCL\ZSF_DATA_new02.dbf'
size 128m autoextend on next 128M maxsize 4096m;

迁移异常对象到新表空间

alter table ZSF.DETAIL move tablespace ZSF_new;
alter index ZSF.DETAIL1 rebuild tablespace ZSF_new;
alter index ZSF.DETAIL2 rebuild tablespace ZSF_new;
alter index ZSF.DETAIL3 rebuild tablespace ZSF_new;
alter index ZSF.DETAIL4 rebuild tablespace ZSF_new;
alter index ZSF.FK_RECI_ORD rebuild tablespace ZSF_new;
alter index ZSF.PREPAY1 rebuild tablespace ZSF_new;
alter index ZSF.RECEDETAIL1 rebuild tablespace ZSF_new;

然后对于datafile 15所在表空间增加新文件,因为已经迁移了异常对象,然后resize datafile 15小于8G,关闭自扩展,至此该数据库恢复完成

11g中 connect by 语句执行计划改变

从10.2.0.3升级到11.2.0.4的朋友,如果细心会发现,以下sql在11.2.0.4中执行效率变低(该sql主要是获取连接用户获取权限信息)
select privilege#,level from sysauth$ connect by grantee#=prior privilege# and privilege#>0 start with grantee#=:1 and privilege#>0
如果你接触是Oracle版本比较多,而且还比较细心,你可能会进一步发现在11.2.0.2中该条sql是:select /*+ connect_by_filtering */ privilege#, level from sysauth$ connect by grantee#=prior privilege# and privilege#>0 start with grantee#=:1 and privilege#>0 也就是说使用了/*+ connect_by_filtering */提示.我这里通过简单测试说明问题.

在11.2.0.4环境中

14:16:19 SQL> set autot trace exp stat
14:16:20 SQL> set time on
14:16:20 SQL> set timing on
14:16:20 SQL> var a1 number;
14:16:20 SQL> exec :a1:=6;
PL/SQL 过程已成功完成。
已用时间:  00: 00: 00.00
14:16:20 SQL> select privilege#,level from sysauth$ connect by grantee#=prior
14:16:20 SQL> privilege# and privilege#>0 start with grantee#=:a1  and privilege#>0
14:16:22 SQL> /
已用时间:  00: 00: 00.01
执行计划
----------------------------------------------------------
Plan hash value: 2624122540
------------------------------------------------------------------------------------------------------
| Id  | Operation                               | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                        |            |     7 |   182 |     3  (34)| 00:00:01 |
|*  1 |  CONNECT BY NO FILTERING WITH START-WITH|            |       |       |            |          |
|   2 |   INDEX FAST FULL SCAN                  | I_SYSAUTH1 |   618 |  4944 |     2   (0)| 00:00:01 |
------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - access("GRANTEE#"=PRIOR "PRIVILEGE#")
       filter("PRIVILEGE#">0 AND "GRANTEE#"=TO_NUMBER(:A1) AND "PRIVILEGE#">0)
统计信息
----------------------------------------------------------
          1  recursive calls
          0  db block gets
          7  consistent gets
          0  physical reads
          0  redo size
        599  bytes sent via SQL*Net to client
        520  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          2  sorts (memory)
          0  sorts (disk)
          1  rows processed

这里可以看出来这里使用的执行计划使用了CONNECT BY NO FILTERING WITH START-WITH,逻辑读为7.

10.2.0.3环境中

14:32:57 SQL> set lines 150
14:33:00 SQL> set autot trace exp stat
14:33:01 SQL> set time on
14:33:01 SQL> set timing on
14:33:01 SQL> var a1 number;
14:33:01 SQL> exec :a1:=6;
PL/SQL procedure successfully completed.
Elapsed: 00:00:00.00
14:33:01 SQL> select privilege#,level from sysauth$ connect by grantee#=prior
14:33:01 SQL> privilege# and privilege#>0 start with grantee#=:a1  and privilege#>0  ;
Elapsed: 00:00:00.00
Execution Plan
----------------------------------------------------------
Plan hash value: 2620769641
----------------------------------------------------------------------------------------
| Id  | Operation                 | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT          |            |     3 |    24 |     2   (0)| 00:00:01 |
|*  1 |  CONNECT BY WITH FILTERING|            |       |       |            |          |
|*  2 |   INDEX RANGE SCAN        | I_SYSAUTH1 |     3 |    24 |     2   (0)| 00:00:01 |
|   3 |   NESTED LOOPS            |            |       |       |            |          |
|   4 |    CONNECT BY PUMP        |            |       |       |            |          |
|*  5 |    INDEX RANGE SCAN       | I_SYSAUTH1 |     3 |    24 |     2   (0)| 00:00:01 |
----------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - access("GRANTEE#"=PRIOR "PRIVILEGE#")
       filter("PRIVILEGE#">0)
   2 - access("GRANTEE#"=TO_NUMBER(:A1) AND "PRIVILEGE#">0)
   5 - access("GRANTEE#"=PRIOR "PRIVILEGE#" AND "PRIVILEGE#">0)
Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
          4  consistent gets
          0  physical reads
          0  redo size
        583  bytes sent via SQL*Net to client
        492  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          3  sorts (memory)
          0  sorts (disk)
          1  rows processed

这里执行计划使用的为CONNECT BY WITH FILTERING,而且逻辑读为4,对于这个sql来说,使用CONNECT BY WITH FILTERING执行效率更高.

这里可以很明显的看到:connect by查询的执行计划从10g的CONNECT BY WITH FILTERING变为了11g中的CONNECT BY NO FILTERING WITH SW (UNIQUE),从而使得执行计划发生改变。但是Oracle一般有个特性,就是当引入新特性之时,一般都会伴随隐含参数或者event来屏蔽新特性.这里也例外,我们可以通过”_optimizer_connect_by_elim_dups” = false和”_connect_by_use_union_all” = “old_plan_mode”来屏蔽11g中关于connect by执行计划的改变,使得执行计划恢复到10G的CONNECT BY WITH FILTERING方式

14:30:45 SQL> alter session set "_optimizer_connect_by_elim_dups" = false;
会话已更改。
已用时间:  00: 00: 00.00
14:30:46 SQL> alter session set "_connect_by_use_union_all" = "old_plan_mode";
会话已更改。
已用时间:  00: 00: 00.00
14:30:46 SQL> set autot trace exp stat
14:30:46 SQL> set time on
14:30:46 SQL> set timing on
14:30:46 SQL> var a1 number;
14:30:46 SQL> exec :a1:=6;
PL/SQL 过程已成功完成。
已用时间:  00: 00: 00.00
14:30:46 SQL> select privilege#,level from sysauth$ connect by grantee#=prior
14:30:46 SQL> privilege# and privilege#>0 start with grantee#=:a1  and privilege#>0 ;
已用时间:  00: 00: 00.01
执行计划
----------------------------------------------------------
Plan hash value: 2620769641
----------------------------------------------------------------------------------------
| Id  | Operation                 | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT          |            |     2 |    16 |     2   (0)| 00:00:01 |
|*  1 |  CONNECT BY WITH FILTERING|            |       |       |            |          |
|*  2 |   INDEX RANGE SCAN        | I_SYSAUTH1 |     2 |    16 |     2   (0)| 00:00:01 |
|   3 |   NESTED LOOPS            |            |       |       |            |          |
|   4 |    CONNECT BY PUMP        |            |       |       |            |          |
|*  5 |    INDEX RANGE SCAN       | I_SYSAUTH1 |     2 |    16 |     2   (0)| 00:00:01 |
----------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - access("GRANTEE#"=PRIOR "PRIVILEGE#")
       filter("PRIVILEGE#">0)
   2 - access("GRANTEE#"=TO_NUMBER(:A1) AND "PRIVILEGE#">0)
   5 - access("GRANTEE#"=PRIOR "PRIVILEGE#" AND "PRIVILEGE#">0)
统计信息
----------------------------------------------------------
          1  recursive calls
          0  db block gets
          4  consistent gets
          0  physical reads
          0  redo size
        599  bytes sent via SQL*Net to client
        520  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          3  sorts (memory)
          0  sorts (disk)
          1  rows processed

11.2.0.2中也许是考虑到connect by 不够成熟,因此使用了hint /*+ connect_by_filtering */ 来固定执行计划

14:22:09 SQL> select /*+ connect_by_filtering */ privilege#,level from sysauth$ connect by grantee#=prior
14:22:09 SQL>  privilege# and privilege#>0 start with grantee#=:a1  and privilege#>0
14:22:10 SQL> /
已用时间:  00: 00: 00.00
执行计划
----------------------------------------------------------
Plan hash value: 2620769641
----------------------------------------------------------------------------------------
| Id  | Operation                 | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT          |            |     7 |   182 |     8  (25)| 00:00:01 |
|*  1 |  CONNECT BY WITH FILTERING|            |       |       |            |          |
|*  2 |   INDEX RANGE SCAN        | I_SYSAUTH1 |     2 |    16 |     2   (0)| 00:00:01 |
|   3 |   NESTED LOOPS            |            |     5 |   105 |     4   (0)| 00:00:01 |
|   4 |    CONNECT BY PUMP        |            |       |       |            |          |
|*  5 |    INDEX RANGE SCAN       | I_SYSAUTH1 |     2 |    16 |     1   (0)| 00:00:01 |
----------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - access("GRANTEE#"=PRIOR "PRIVILEGE#")
       filter("PRIVILEGE#">0)
   2 - access("GRANTEE#"=TO_NUMBER(:A1) AND "PRIVILEGE#">0)
   5 - access("GRANTEE#"="connect$_by$_pump$_002"."prior privilege# " AND
              "PRIVILEGE#">0)
统计信息
----------------------------------------------------------
          1  recursive calls
          0  db block gets
          4  consistent gets
          0  physical reads
          0  redo size
        599  bytes sent via SQL*Net to client
        520  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          3  sorts (memory)
          0  sorts (disk)
          1  rows processed

CONNECT BY NO FILTERING WITH SW (UNIQUE)和CONNECT BY WITH FILTERING,没有明显的优劣之分,只有在特定的情况下,进行了实际的测试,选择合适自己的sql的执行计划

数据库启动报ORA-00704 ORA-39714错误解决

数据库启动失败,报ORA-00704、ORA-39714错误

[oracle@www.xifenfei.com ~]$ sqlplus / as sysdba
SQL*Plus: Release 12.1.0.1.0 Production on Thu Aug 7 08:15:35 2014
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
SQL> startup
ORACLE instance started.
Total System Global Area  663945216 bytes
Fixed Size                  2291808 bytes
Variable Size             369100704 bytes
Database Buffers          289406976 bytes
Redo Buffers                3145728 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-39714: upgrade script utlmmig.sql failed
Process ID: 11592
Session ID: 1 Serial number: 5
SQL> startup upgrade
SP2-0642: SQL*Plus internal error state 2133, context 3114:0:0
Unsafe to proceed
ORA-03114: not connected to ORACLE
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

alert日志报错

Thu Aug 07 07:42:25 2014
SMON: enabling cache recovery
Thu Aug 07 07:42:25 2014
Errors in file /u01/app/oracle/diag/rdbms/orcl/ORCL/trace/ORCL_ora_11592.trc:
ORA-39714: upgrade script utlmmig.sql failed
Thu Aug 07 07:42:25 2014
Errors in file /u01/app/oracle/diag/rdbms/orcl/ORCL/trace/ORCL_ora_11592.trc:
ORA-00704: bootstrap process failure
ORA-39714: upgrade script utlmmig.sql failed
Thu Aug 07 07:42:25 2014
Errors in file /u01/app/oracle/diag/rdbms/orcl/ORCL/trace/ORCL_ora_11592.trc:
ORA-00704: bootstrap process failure
ORA-39714: upgrade script utlmmig.sql failed
Thu Aug 07 07:42:25 2014
Error 704 happened during db open, shutting down database
USER (ospid: 11592): terminating the instance due to error 704

通过分析utlmmig.sql脚本知道,数据库在升级bootstrap$之前会先在props$表中插入BOOTSTRAP_UPGRADE_ERROR相关记录,数据库在启动之时会检测该值,如果发现该值存在,数据库只能以upgrade模式启动,清理掉相关记录,数据库即可正常启动

[oracle@www.xifenfei.com ~]$ sqlplus / as sysdba
SQL*Plus: Release 12.1.0.1.0 Production on Thu Aug 7 07:42:44 2014
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup upgrade
ORACLE instance started.
Total System Global Area  663945216 bytes
Fixed Size                  2291808 bytes
Variable Size             369100704 bytes
Database Buffers          289406976 bytes
Redo Buffers                3145728 bytes
Database mounted.
Database opened.
SQL>   delete from props$ where name = 'BOOTSTRAP_UPGRADE_ERROR';
1 row deleted.
SQL>   delete from props$ where name = 'LOGMNR_BOOTSTRAP_UPGRADE_ERROR';
0 rows deleted.
SQL>   commit;
Commit complete.
SQL>
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area  663945216 bytes
Fixed Size                  2291808 bytes
Variable Size             369100704 bytes
Database Buffers          289406976 bytes
Redo Buffers                3145728 bytes
Database mounted.
Database opened.
SQL>

数据库虽然正常启动成功,但是由于bootstrap$对象升级失败,后续还是有很大风险,建议分析报错原因,解决原因然后继续升级bootstrap$基表

bootstrap$核心index(I_OBJ1,I_USER1,I_FILE#_BLOCK#,I_IND1,I_TS#,I_CDEF1等)异常恢复—ORA-00701错误解决

在Oracle使用的过程中,经常遭遇一些核心index出现异常,导致数据库无法正常使用,特别是在数据库open的情况下,因为出现这些bootstrap$中的部分index是无法通过设置event38003进行重建,从而导致数据库无法正常使用,最常见异常index 有:I_ICOL1, I_TS1, I_CDEF1, I_CDEF2, I_CDEF3, I_CDEF4, I_PROXY_DATA$, I_IND1, I_TS#, I_UNDO1, I_UNDO2, I_COBJ#, I_USER1, I_USER2, I_CON1, I_CON2, I_FILE1, I_FILE2, I_FILE#_BLOCK#, I_USER#, I_OBJ#, I_PROXY_ROLE_DATA$_1, I_PROXY_ROLE_DATA$_2, I_CCOL1, I_CCOL2, I_TAB1, I_COL1, I_COL2, I_COL3, I_OBJ1, I_OBJ2, I_OBJ3, I_OBJ4, I_OBJ5
重建的index异常报错有:ORA-00701,ORA-01410,ORA-08102,ORA-08103,ORA-01578,ORA-01499,ORA-00600等各类错误
因为处理细节复杂,无法一一描述出来.这里假设Oracle数据库一个非常核心的表OBJ$中的I_OBJ1异常,现在准备重建它,大概步骤如下:
1. 假设异常index信息(I_OBJ1)

SQL> alter index sys.i_obj1 rebuild;
alter index sys.i_obj1 rebuild
*
ERROR at line 1:
ORA-00701: object necessary for warmstarting database cannot be altered
SQL> SELECT SQL_TEXT FROM BOOTSTRAP$ WHERE SQL_TEXT LIKE '%obj%';
SQL_TEXT
--------------------------------------------------------------------------------
CREATE UNIQUE INDEX I_OBJ1 ON OBJ$(OBJ#,OWNER#,TYPE#) PCTFREE 10 INITRANS 2 MAXT
RANS 255 STORAGE (  INITIAL 64K NEXT 1024K MINEXTENTS 1 MAXEXTENTS 2147483645 PC
TINCREASE 0 OBJNO 36 EXTENTS (FILE 1 BLOCK 336))

2. 创建获取异常index所属表

SQL>  select table_name from dba_indexes where owner='SYS' AND INDEX_NAME='I_OBJ1';
TABLE_NAME
--------------------------------------------------------------------------------
OBJ$

3. 获取obj$ ddl 语句

SQL> set long 10000
SQL> set pages 1000
SQL>  SELECT DBMS_METADATA.GET_DDL('TABLE','OBJ$','SYS') FROM DUAL;
DBMS_METADATA.GET_DDL('TABLE','OBJ$','SYS')
--------------------------------------------------------------------------------
  CREATE TABLE "SYS"."OBJ$"
   (    "OBJ#" NUMBER NOT NULL ENABLE,
        "DATAOBJ#" NUMBER,
        "OWNER#" NUMBER NOT NULL ENABLE,
        "NAME" VARCHAR2(128) NOT NULL ENABLE,
        "NAMESPACE" NUMBER NOT NULL ENABLE,
        "SUBNAME" VARCHAR2(128),
        "TYPE#" NUMBER NOT NULL ENABLE,
        "CTIME" DATE NOT NULL ENABLE,
        "MTIME" DATE NOT NULL ENABLE,
        "STIME" DATE NOT NULL ENABLE,
        "STATUS" NUMBER NOT NULL ENABLE,
        "REMOTEOWNER" VARCHAR2(128),
        "LINKNAME" VARCHAR2(128),
        "FLAGS" NUMBER,
        "OID$" RAW(16),
        "SPARE1" NUMBER,
        "SPARE2" NUMBER,
        "SPARE3" NUMBER,
        "SPARE4" VARCHAR2(1000),
        "SPARE5" VARCHAR2(1000),
        "SPARE6" DATE,
        "SIGNATURE" RAW(16),
        "SPARE7" NUMBER,
        "SPARE8" NUMBER,
        "SPARE9" NUMBER
   ) PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(INITIAL 16384 NEXT 106496 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "SYSTEM"

4. 获取obj$相关index 语句

SQL>  SELECT DBMS_METADATA.GET_DDL('INDEX',INDEX_NAME,'SYS') FROM dba_indexes where owner='SYS' AND TABLE_NAME='OBJ$';
DBMS_METADATA.GET_DDL('INDEX',INDEX_NAME,'SYS')
--------------------------------------------------------------------------------
  CREATE UNIQUE INDEX "SYS"."I_OBJ5" ON "SYS"."OBJ$" ("SPARE3", "NAME", "NAMESPA
CE", "TYPE#", "OWNER#", "REMOTEOWNER", "LINKNAME", "SUBNAME", "OBJ#")
  PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS
  STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "SYSTEM"
  CREATE INDEX "SYS"."I_OBJ4" ON "SYS"."OBJ$" ("DATAOBJ#", "TYPE#", "OWNER#")
  PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS
  STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "SYSTEM"
  CREATE INDEX "SYS"."I_OBJ3" ON "SYS"."OBJ$" ("OID$")
  PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS
  STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "SYSTEM"
  CREATE UNIQUE INDEX "SYS"."I_OBJ2" ON "SYS"."OBJ$" ("OWNER#", "NAME", "NAMESPA
CE", "REMOTEOWNER", "LINKNAME", "SUBNAME", "TYPE#", "SPARE3", "OBJ#")
  PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS
  STORAGE(INITIAL 16384 NEXT 106496 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "SYSTEM"
  CREATE UNIQUE INDEX "SYS"."I_OBJ1" ON "SYS"."OBJ$" ("OBJ#", "OWNER#", "TYPE#")
  PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS
  STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "SYSTEM"

5. 创建obj$img表和相关index

SQL>   CREATE TABLE "SYS"."OBJ$IMG"
  2     (    "OBJ#" NUMBER NOT NULL ENABLE,
  3          "DATAOBJ#" NUMBER,
  4          "OWNER#" NUMBER NOT NULL ENABLE,
  5          "NAME" VARCHAR2(128) NOT NULL ENABLE,
  6          "NAMESPACE" NUMBER NOT NULL ENABLE,
  7          "SUBNAME" VARCHAR2(128),
  8          "TYPE#" NUMBER NOT NULL ENABLE,
  9          "CTIME" DATE NOT NULL ENABLE,
 10          "MTIME" DATE NOT NULL ENABLE,
 11          "STIME" DATE NOT NULL ENABLE,
 12          "STATUS" NUMBER NOT NULL ENABLE,
 13          "REMOTEOWNER" VARCHAR2(128),
 14          "LINKNAME" VARCHAR2(128),
 15          "FLAGS" NUMBER,
 16          "OID$" RAW(16),
 17          "SPARE1" NUMBER,
 18          "SPARE2" NUMBER,
 19          "SPARE3" NUMBER,
 20          "SPARE4" VARCHAR2(1000),
 21          "SPARE5" VARCHAR2(1000),
 22          "SPARE6" DATE,
 23          "SIGNATURE" RAW(16),
 24          "SPARE7" NUMBER,
 25          "SPARE8" NUMBER,
 26          "SPARE9" NUMBER
 27     ) PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 28   NOCOMPRESS LOGGING
 29    STORAGE(INITIAL 16384 NEXT 106496 MINEXTENTS 1 MAXEXTENTS 2147483645
 30    PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
 31    BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
 32    TABLESPACE "SYSTEM"
 33  ;
Table created.
SQL>
SQL>   CREATE UNIQUE INDEX "SYS"."I_OBJ_5IMG" ON "SYS"."OBJ$IMG" ("SPARE3", "NAME", "NAMESPACE", "TYPE#", "OWNER#", "REMOTEOWNER", "LINKNAME", "SUBNAME", "OBJ#")
  2    PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS
  3    STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  4    PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  5    BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  6    TABLESPACE "SYSTEM";
Index created.
SQL>   CREATE INDEX "SYS"."I_OBJ4IMG" ON "SYS"."OBJ$IMG" ("DATAOBJ#", "TYPE#", "OWNER#")
  2    PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS
  3    STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  4    PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  5    BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  6    TABLESPACE "SYSTEM";
Index created.
SQL>   CREATE INDEX "SYS"."I_OBJ3IMG" ON "SYS"."OBJ$IMG" ("OID$")
  2    PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS
  3    STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  4    PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  5    BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  6    TABLESPACE "SYSTEM";
Index created.
SQL>   CREATE UNIQUE INDEX "SYS"."I_OBJ2IMG" ON "SYS"."OBJ$IMG" ("OWNER#", "NAME", "NAMESPACE", "REMOTEOWNER", "LINKNAME", "SUBNAME", "TYPE#", "SPARE3", "OBJ#")
  2    PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS
  3    STORAGE(INITIAL 16384 NEXT 106496 MINEXTENTS 1 MAXEXTENTS 2147483645
  4    PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  5    BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  6    TABLESPACE "SYSTEM";
Index created.
SQL>   CREATE UNIQUE INDEX "SYS"."I_OBJ1IMG" ON "SYS"."OBJ$IMG" ("OBJ#", "OWNER#", "TYPE#")
  2    PCTFREE 10 INITRANS 2 MAXTRANS 255 COMPUTE STATISTICS
  3    STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  4    PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  5    BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  6    TABLESPACE "SYSTEM"
  7  ;
Index created.

6. 获取bootstrap$语句

SQL> SELECT DBMS_METADATA.GET_DDL('TABLE','BOOTSTRAP$','SYS') FROM DUAL;
DBMS_METADATA.GET_DDL('TABLE','BOOTSTRAP$','SYS')
--------------------------------------------------------------------------------
  CREATE TABLE "SYS"."BOOTSTRAP$"
   (    "LINE#" NUMBER NOT NULL ENABLE,
        "OBJ#" NUMBER NOT NULL ENABLE,
        "SQL_TEXT" VARCHAR2(4000) NOT NULL ENABLE
   ) PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
 NOCOMPRESS LOGGING
  STORAGE(INITIAL 57344 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
  TABLESPACE "SYSTEM"

7. 创建bootstrap$img表

SQL>   CREATE TABLE "SYS"."BOOTSTRAP$IMG"
  2     (    "LINE#" NUMBER NOT NULL ENABLE,
  3          "OBJ#" NUMBER NOT NULL ENABLE,
  4          "SQL_TEXT" VARCHAR2(4000) NOT NULL ENABLE
  5     ) PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255
  6   NOCOMPRESS LOGGING
  7    STORAGE(INITIAL 57344 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
  8    PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1
  9    BUFFER_POOL DEFAULT FLASH_CACHE DEFAULT CELL_FLASH_CACHE DEFAULT)
 10    TABLESPACE "SYSTEM"
 11  /
Table created.

8.插入数据

insert into obj$img select * from obj$;
insert into bootstrap$img select * from bootstrap$;
commit;

9. 删除bootstrap$img对象名

delete from bootstrap$img where obj# in
 (select obj# from obj$
  where name in ('OBJ$',  'I_OBJ1',  'I_OBJ2', 'I_OBJ3', 'I_OBJ4', 'I_OBJ5',
                 'BOOTSTRAP$'));
commit;

10. 插入新创建对象

insert into bootstrap$img select * from bootstrap$tmpstr;
commit;

11. 关闭数据库

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.

12. bbed修改相关block值

[oracle@oel6 ~]$ bbed
Password:
BBED: Release 2.0.0.0.0 - Limited Production on Thu Aug 7 04:54:50 2014
Copyright (c) 1982, 2013, Oracle and/or its affiliates.  All rights reserved.
************* !!! For Oracle Internal Use only !!! ***************
BBED> set filename '/u01/app/oracle/oradata/XIFENFEI/system01.dbf'
        FILENAME        /u01/app/oracle/oradata/XIFENFEI/system01.dbf
BBED> set blocksize 8192
        BLOCKSIZE       8192
BBED> set block 1
        BLOCK#          1
BBED> set mode edit
        MODE            Edit
BBED> set count 32
        COUNT           32
BBED> m /x e81d
Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) y
 File: /u01/app/oracle/oradata/XIFENFEI/system01.dbf (0)
 Block: 1                Offsets:   196 to  227           Dba:0x00000000
------------------------------------------------------------------------
 e81d4000 12000000 00000000 41bad632 15bad632 01000000 00000000 00000000
 <32 bytes per line>
BBED> m /x 5200 offset +2
 File: /u01/app/oracle/oradata/XIFENFEI/system01.dbf (0)
 Block: 1                Offsets:   198 to  229           Dba:0x00000000
------------------------------------------------------------------------
 52001200 00000000 000041ba d63215ba d6320100 00000000 00000000 00000000
 <32 bytes per line>
BBED> d offset 96
 File: /u01/app/oracle/oradata/XIFENFEI/system01.dbf (0)
 Block: 1                Offsets:   196 to  227           Dba:0x00000000
------------------------------------------------------------------------
 e81d5200 12000000 00000000 41bad632 15bad632 01000000 00000000 00000000
 <32 bytes per line>
BBED> sum apply
Check value for File 0, Block 1:
current = 0x5fbf, required = 0x5fbf
BBED> verify
DBVERIFY - Verification starting
FILE = /u01/app/oracle/oradata/XIFENFEI/system01.dbf
BLOCK = 1
DBVERIFY - Verification complete
Total Blocks Examined         : 1
Total Blocks Processed (Data) : 0
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0
Message 531 not found;  product=RDBMS; facility=BBED
BBED> exit

13. 启动数据库

SQL> startup
ORACLE instance started.
Total System Global Area  663945216 bytes
Fixed Size                  2291808 bytes
Variable Size             369100704 bytes
Database Buffers          289406976 bytes
Redo Buffers                3145728 bytes
Database mounted.
Database opened.

14. 验证I_OBJ1 index重建情况

SQL> SELECT OBJECT_ID,DATA_OBJECT_ID,CREATED FROM DBA_OBJECTS WHERE OBJECT_NAME='I_OBJ1';
 OBJECT_ID DATA_OBJECT_ID CREATED
---------- -------------- ---------
     77120          77120 06-AUG-14
SQL> !date
Thu Aug  6 05:29:25 CST 2014
SQL> SELECT HEADER_FILE,HEADER_BLOCK FROM DBA_SEGMENTS WHERE SEGMENT_NAME='I_OBJ1';
HEADER_FILE HEADER_BLOCK
----------- ------------
          1        77296
SQL> SELECT SQL_TEXT FROM BOOTSTRAP$ WHERE SQL_TEXT LIKE '%obj%';
SQL_TEXT
-----------------------------------------------------------------------------------------------------------------------------------------------
create unique index i_obj1 on obj$(obj#, owner#, type#) PCTFREE 10 INITRANS 2 MAXTRANS 255 STORAGE (  INITIAL 65536 NEXT 1048576 MINEXTENTS 1 M
AXEXTENTS 2147483645 PCTINCREASE 0 OBJNO 77120 EXTENTS (FILE 1 BLOCK 77296))
create unique index i_obj2 on obj$(owner#, name, namespace,remoteowner, linkname, subname, type#, spare3, obj#) PCTFREE 10 INITRANS 2 MAXTRANS
255 STORAGE (  INITIAL 16384 NEXT 106496 MINEXTENTS 1 MAXEXTENTS 2147483645 PCTINCREASE 0 OBJNO 77119 EXTENTS (FILE 1 BLOCK 77288))
create index i_obj3 on obj$(oid$) PCTFREE 10 INITRANS 2 MAXTRANS 255 STORAGE (  INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645 P
CTINCREASE 0 OBJNO 77118 EXTENTS (FILE 1 BLOCK 77280))
create index i_obj4 on obj$(dataobj#, type#, owner#) PCTFREE 10 INITRANS 2 MAXTRANS 255 STORAGE (  INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXE
XTENTS 2147483645 PCTINCREASE 0 OBJNO 77117 EXTENTS (FILE 1 BLOCK 77272))
create unique index i_obj5 on obj$(spare3, name, namespace, type#, owner#, remoteowner, linkname, subname, obj#) PCTFREE 10 INITRANS 2 MAXTRANS
 255 STORAGE (  INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645 PCTINCREASE 0 OBJNO 77116 EXTENTS (FILE 1 BLOCK 77264))

不同版本对应的基表和index结构不一样(本篇blog使用的是12c做试验),但是本方法支持数据库从Oracle 7到Oracle 12c对于bootstrap$中核心index处理.
因为重建bootstrap$中的核心index是一个复杂而且风险非常大的事情,在你无法确定风险或者无法正常完成此类操作之时,如有需要请联系我
Phone:17813235971    Q Q:107644445    E-Mail:dba@xifenfei.com

打patch出现Copy failed—tfa服务导致部分lib未释放

在aix的11.2.0.4打psu之时,可能遇到由于tfa服务未停止导致部分lib未能够随着crs关闭而释放,从而导致升级升级失败,遇到这样的情况auto方式或者人工方式打patch都会失败,需要关闭tfa,并且使用slibclean释放lib对象,然后重新打patch
打11.2.0.4.3 psu中的18522515 patch失败

xifenf01:/home/grid> cle/app/grid -local /oradata/sys/soft/18706472/18522515                                                     <
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation.  All rights reserved.
Oracle Home       : /u01/oracle/app/grid
Central Inventory : /u01/oracle/app/oraInventory
   from           : /u01/oracle/app/grid/oraInst.loc
OPatch version    : 11.2.0.3.6
OUI version       : 11.2.0.4.0
Log file location : /u01/oracle/app/grid/cfgtoollogs/opatch/opatch2014-10-17_11-56-27AM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   18522515
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/oracle/app/grid')
Is the local system ready for patching? [y|n]
Y
User Responded with: Y
Backing up files...
Applying interim patch '18522515' to OH '/u01/oracle/app/grid'
Patching component oracle.crs, 11.2.0.4.0...
Copy failed from '/oradata/sys/soft/18706472/18522515/files/lib/libhasgen11.so' to '/u01/oracle/app/grid/lib/libhasgen11.so'...
Please verify all applications associated with the Oracle Home '/u01/oracle/app/grid' are shut down. If this is AIX,
please perform solution documented in Note 739963.1 on https://myoraclesupport.oracle.com.
Do you want to retry copying the file? [y|n]
y
User Responded with: Y
Retry copying the file to '/u01/oracle/app/grid/lib/libhasgen11.so'.
Copy failed from '/oradata/sys/soft/18706472/18522515/files/lib/libocr11.so' to '/u01/oracle/app/grid/lib/libocr11.so'...
Please verify all applications associated with the Oracle Home '/u01/oracle/app/grid' are shut down. If this is AIX,
please perform solution documented in Note 739963.1 on https://myoraclesupport.oracle.com.
Do you want to retry copying the file? [y|n]
y
User Responded with: Y
Retry copying the file to '/u01/oracle/app/grid/lib/libocr11.so'.
Copy failed from '/oradata/sys/soft/18706472/18522515/files/lib/libocrb11.so' to '/u01/oracle/app/grid/lib/libocrb11.so'...
Please verify all applications associated with the Oracle Home '/u01/oracle/app/grid' are shut down. If this is AIX,
please perform solution documented in Note 739963.1 on https://myoraclesupport.oracle.com.
Do you want to retry copying the file? [y|n]
y
User Responded with: Y
Retry copying the file to '/u01/oracle/app/grid/lib/libocrb11.so'.
The following actions have failed:
Copy failed from '/oradata/sys/soft/18706472/18522515/files/lib/libhasgen11.so' to '/u01/oracle/app/grid/lib/libhasgen11.so'...
Copy failed from '/oradata/sys/soft/18706472/18522515/files/lib/libocr11.so' to '/u01/oracle/app/grid/lib/libocr11.so'...
Do you want to proceed? [y|n]
y
User Responded with: Y
Verifying the update...
There are 2 copy files under ORACLE_HOME that are not patched.
Files check failed: Some files under ORACLE_HOME are not patched. Please see log file for details.
ApplySession failed in system modification phase... 'Verification of patch failed: Files are not updated completely.'
Restoring "/u01/oracle/app/grid" to the state prior to running NApply...
OPatch failed to restore the files from backup area. Not running "make".
NApply was not able to restore the home.  Please invoke the following scripts:
  - restore.[sh,bat]
  - make.txt (Unix only)
to restore the ORACLE_HOME.  They are located under
"/u01/oracle/app/grid/.patch_storage/NApply/2014-10-17_11-56-27AM"
UtilSession failed: ApplySession failed in system modification phase... 'Verification of patch failed: Files are not updated completely.'
Log file location: /u01/oracle/app/grid/cfgtoollogs/opatch/opatch2014-10-17_11-56-27AM_1.log
OPatch failed with error code 73

关闭tfa服务

xifenf01:/home/grid> ps -ef|grep -v grep|grep -E 'init|d.bin|ocls|UID|ora|crs|sql|tns|grid'
     UID      PID     PPID   C    STIME    TTY  TIME CMD
    root        1        0   0 11:16:01      -  0:00 /etc/init
    root 32243912        1   0 11:19:15      -  0:00 /usr/sbin/vxnotify -g oradata_dg -icfspdvACLMSk
    root 32374816        1   0 11:17:46      -  0:00 /bin/sh /etc/init.ohasd run
    root 32505956        1   0 11:17:56      -  0:09 /u01/oracle/app/grid/jdk/jre/bin/java -Xms64m -Xmx256m -classpath
/u01/oracle/app/grid/tfa/xifenf01/tfa_home/jar/RATFA.jar:/u01/oracle/app/grid/tfa/xifenf01/tfa_home/jar/je-4.0.103.jar:
/u01/oracle/app/grid/tfa/xifenf01/tfa_home/jar/ojdbc6.jar oracle.rat.tfa.TFAMain /u01/oracle/app/grid/tfa/xifenf01/tfa_home
    root 33095694        1   0 11:19:15      -  0:00 /usr/sbin/vxnotify -g crsdata_dg -icfspdvACLMSk
    grid 34013244  5505292   0 11:56:25  pts/0  0:00 -sh
    root  3211548        1   0 11:17:34      -  0:00 /opt/VRTSdbed/bin/vxdbd
    root  5112296        1   0 11:17:46      -  0:00 /bin/sh /etc/init.tfa run
    grid  6422822 34013244   0 13:03:33  pts/0  0:00 ps -ef
xifenf01:/home/grid> exit
xifenf01/#/etc/init.tfa stop
Stopping TFA
TFA is running  - Will wait 5 seconds (up to 3 times)
TFA is running  - Will wait 5 seconds (up to 3 times)
TFA is running  - Will wait 5 seconds (up to 3 times)
TFAmain Force Stopped Successfully
Killing TFA running with pid 32505956
. . .
Successfully stopped TFA..
xifenf01/#ps -ef|grep -v grep|grep -E 'init|d.bin|ocls|UID|ora|crs|sql|tns|grid'
     UID      PID     PPID   C    STIME    TTY  TIME CMD
    root        1        0   0 11:16:01      -  0:00 /etc/init
    root 32243912        1   0 11:19:15      -  0:00 /usr/sbin/vxnotify -g oradata_dg -icfspdvACLMSk
    root 32374816        1   0 11:17:46      -  0:00 /bin/sh /etc/init.ohasd run
    root 33095694        1   0 11:19:15      -  0:00 /usr/sbin/vxnotify -g crsdata_dg -icfspdvACLMSk
    root  3211548        1   0 11:17:34      -  0:00 /opt/VRTSdbed/bin/vxdbd
    root  5112296        1   0 11:17:46      -  0:00 /bin/sh /etc/init.tfa run

清理未释放lib对象

xifenf01/#genkld | grep /u01/oracle/app/grid
 900000004523000  252429f /u01/oracle/app/grid/lib/libclntsh.so
 9000000012e7000  2ddd80d /u01/oracle/app/grid/lib/libttsh11.so
 9000000040c5000     6c16 /u01/oracle/app/grid/lib/libskgxn2.so
 9000000012da000     cb95 /u01/oracle/app/grid/lib/libocrutl11.so
 9000000010cb000   20efa3 /u01/oracle/app/grid/lib/libocrb11.so
 90000000102a000    a0f25 /u01/oracle/app/grid/lib/libocr11.so
 900000000989000   6a0dc1 /u01/oracle/app/grid/lib/libhasgen11.so
 9000000075ee000     b1cf /u01/oracle/app/grid/jdk/jre/bin/libnio.a
 9000000075cd000    20c36 /u01/oracle/app/grid/jdk/jre/bin/libnet.a
 9000000075b5000    17840 /u01/oracle/app/grid/jdk/jre/bin/libzip.a
 900000007592000    21708 /u01/oracle/app/grid/jdk/jre/bin/libj9ute23.so
 90000000757d000    14a1f /u01/oracle/app/grid/jdk/jre/bin/libiverel23.so
 900000007513000    69b64 /u01/oracle/app/grid/jdk/jre/bin/libjclscar_23.so
 9000000074ee000    243de /u01/oracle/app/grid/jdk/jre/bin/libj9vrb23.so
 9000000074b9000    3404a /u01/oracle/app/grid/jdk/jre/bin/libj9jvmti23.so
 90000000748a000    2ed44 /u01/oracle/app/grid/jdk/jre/bin/libj9dyn23.so
 9000000073f0000    99873 /u01/oracle/app/grid/jdk/jre/bin/libj9gc23.so
 9000000073d8000    17a91 /u01/oracle/app/grid/jdk/jre/bin/libj9bcv23.so
 900000006f20000   4b7279 /u01/oracle/app/grid/jdk/jre/bin/libj9jit23.so
 900000006f0f000    10f07 /u01/oracle/app/grid/jdk/jre/bin/libj9trc23.so
 900000006ed0000    2cd1f /u01/oracle/app/grid/jdk/jre/bin/libj9dmp23.so
 900000006efd000    11bb3 /u01/oracle/app/grid/jdk/jre/bin/libj9zlib23.so
 900000006e96000    37b96 /u01/oracle/app/grid/jdk/jre/bin/libjava.a
 900000006ece000     1d41 /u01/oracle/app/grid/jdk/jre/bin/libwrappers.a
 900000006dcf000    70780 /u01/oracle/app/grid/jdk/jre/bin/libj9vm23.so
 900000006e42000    534c0 /u01/oracle/app/grid/jdk/jre/bin/libj9prt23.so
 900000006e40000     1d37 /u01/oracle/app/grid/jdk/jre/bin/libj9hookable23.so
 900000006da6000    1cde3 /u01/oracle/app/grid/jdk/jre/bin/j9vm/libjvm.so
 900000006dc3000     bf62 /u01/oracle/app/grid/jdk/jre/bin/libj9thr23.so
 900000000464000     214b /u01/oracle/app/grid/jdk/jre/bin/libjsig.so
 900000006d96000     f802 /u01/oracle/app/grid/jdk/jre/bin/classic/libjvm.so
xifenf01/#
xifenf01/#/usr/sbin/slibclean
xifenf01/#/usr/sbin/slibclean
xifenf01/#/usr/sbin/slibclean
xifenf01/#/usr/sbin/slibclean
xifenf01/#genkld | grep /u01/oracle/app/grid
xifenf01/#genkld | grep /u01/oracle/app/grid
xifenf01/#

继续打patch

xifenf01/#su - grid
xifenf01:/home/grid> cle/app/grid -local /oradata/sys/soft/18706472/18522515                                                     <
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation.  All rights reserved.
Oracle Home       : /u01/oracle/app/grid
Central Inventory : /u01/oracle/app/oraInventory
   from           : /u01/oracle/app/grid/oraInst.loc
OPatch version    : 11.2.0.3.6
OUI version       : 11.2.0.4.0
Log file location : /u01/oracle/app/grid/cfgtoollogs/opatch/opatch2014-10-17_13-07-20PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   18522515
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/oracle/app/grid')
Is the local system ready for patching? [y|n]
Y
User Responded with: Y
Backing up files...
Applying interim patch '18522515' to OH '/u01/oracle/app/grid'
Patching component oracle.crs, 11.2.0.4.0...
Verifying the update...
Patch 18522515 successfully applied.
Log file location: /u01/oracle/app/grid/cfgtoollogs/opatch/opatch2014-10-17_13-07-20PM_1.log
OPatch succeeded.
xifenf01:/home/grid>

Quick Reference to Patch Numbers for Database PSU, SPU(CPU), Bundle Patches and Patchsets—201410

Patchsets

 l12.1.0.2 (12.1.0.2.0 PATCH SET FOR ORACLE DATABASE SERVER)  17694377
 11.2.0.4 (11.2.0.4.0 PATCH SET FOR ORACLE DATABASE SERVER)  13390677
 11.2.0.3 (11.2.0.3.0 PATCH SET FOR ORACLE DATABASE SERVER)  10404530
 11.2.0.2 (11.2.0.2.0 PATCH SET FOR ORACLE DATABASE SERVER)  10098816
 11.1.0.7 (11.1.0.7.0 PATCH SET FOR ORACLE DATABASE SERVER)  6890831
 10.2.0.5 (10.2.0.5 PATCH SET FOR ORACLE DATABASE SERVER)  8202632
 d10.2.0.4 (10.2.0.4.0 PATCH SET FOR ORACLE DATABASE SERVER)  6810189
 e10.2.0.3 (10.2.0.3 PATCH SET FOR ORACLE DATABASE SERVER)  5337014
 10.2.0.2 (10.2.0.2 PATCH SET FOR ORACLE DATABASE SERVER)  4547817
 10.1.0.5 (10.1.0.5 PATCH SET FOR ORACLE DATABASE SERVER)  4505133
 10.1.0.4 (10.1.0.4 PATCH SET FOR ORACLE DATABASE SERVER)  4163362
 10.1.0.3 (10.1.0.3 PATCH SET FOR ORACLE DATABASE SERVER)  3761843
 9.2.0.8 (9.2.0.8 PATCH SET FOR ORACLE DATABASE SERVER)  4547809
 9.2.0.7 (9.2.0.7 PATCH SET FOR ORACLE DATABASE SERVER)  4163445
 9.2.0.6 (9.2.0.6 PATCH SET FOR ORACLE DATABASE SERVER)  3948480
 9.2.0.5 (ORACLE 9I DATABASE SERVER RELEASE 2 – PATCH SET 4 VERSION 9.2.0.5.0)  3501955
 9.2.0.4 (9.2.0.4 PATCH SET FOR ORACLE DATABASE SERVER)  3095277
 9.2.0.3 (9.2.0.3 PATCH SET FOR ORACLE DATABASE SERVER)  2761332
 9.2.0.2 (9.2.0.2 PATCH SET FOR ORACLE DATABASE SERVER)  2632931
 9.0.1.5 (9.0.1.5 PATCHSET)  3301544
 9.0.1.4 (9.0.1.4 PATCH SET FOR ORACLE DATABASE SERVER)  2517300
 9.0.1.3 (9.0.1.3. PATCH SET FOR ORACLE DATA SERVER)  2271678
 8.1.7.4 (8.1.7.4 PATCH SET FOR ORACLE DATA SERVER)  2376472
 8.1.7.3 (8.1.7.3 PATCH SET FOR ORACLE DATA SERVER)  2189751
 8.1.7.2 (8.1.7.2.1 PATCH SET FOR ORACLE DATA SERVER)  1909158

PSU, SPU(CPU), Bundle Patches

12.1.0.2
 Description  PSU   GI PSU
 OCT2014  19303936 (12.1.0.2.1)  19392646 (12.1.0.2.1)
12.1.0.1
 Description  PSU  GI PSU   Bundle Patch (Windows64bit)  Bundle Patch (Windows32bit)
 OCT2014  19121550 (12.1.0.1.5)  j19392372 / k19392451(12.1.0.1.5) 19542943 (12.1.0.1.14)
 JUL2014  18522516 (12.1.0.1.4)  j18705901 / k18705972(12.1.0.1.4) 19062327 (12.1.0.1.11)
 APR2014  18031528 (12.1.0.1.3)  j18139660 / k18413105  (12.1.0.1.3) 18448604 (12.1.0.1.7)
 JAN2014  17552800 (12.1.0.1.2)  17735306 (12.1.0.1.2) 17977915 (12.1.0.1.3)
 OCT2013  17027533 (12.1.0.1.1)  17272829 (12.1.0.1.1)  17363796 (12.1.0.1.1)  17363795 (12.1.0.1.1)
11.2.0.4
 Description  PSU  SPU(CPU)  GI PSU  Bundle Patch (Windows 32bit & 64bit)
 OCT2014  19121551 (11.2.0.4.4)  19271443  19380115  19651773
 JUL2014  18522509 (11.2.0.4.3)  18681862  18706472 (11.2.0.4.3)  18842982
 APR2014  18031668 (11.2.0.4.2)  18139690  18139609 (11.2.0.4.2)  18296644
 JAN2014  17478514 (11.2.0.4.1)  17551709  N/A  17987366
11.2.0.3
 Description  PSU  SPU(CPU)  GI PSU  Bundle Patch(Windows64bit)  Bundle Patch(Windows32bit)
 OCT2014  19121548 (11.2.0.3.12)  19271438  19440385 (11.2.0.3.12)  19618575  19618574
 JUL2014  18522512 (11.2.0.3.11)  18681866  18706488 (11.2.0.3.11)  18940194  18940193
 APR2014  18031683 (11.2.0.3.10)  18139695  18139678 (11.2.0.3.10)  18372244  18372243
 JAN2014  17540582 (11.2.0.3.9)  17478415  17735354 (11.2.0.3.9)  18075406  17906981
 OCT2013  16902043 (11.2.0.3.8)  17082364  17272731 (11.2.0.3.8)  17363850  17363844
 JUL2013  16619892 (11.2.0.3.7)  16742095  16742216 (11.2.0.3.7)  16803775  16803774
 APR2013  16056266 (11.2.0.3.6)  16294378  16083653 (11.2.0.3.6)  16345834  16345833
 JAN2013  14727310 (11.2.0.3.5)  14841409  14727347 (11.2.0.3.5)  16042648  16042647
 OCT2012  14275605 (11.2.0.3.4)  14390252  14275572 (11.2.0.3.4)  14613223  14613222
 JUL2012  13923374 (11.2.0.3.3)  14038787  13919095 (11.2.0.3.3)  14223718  14223717
 APR2012  13696216 (11.2.0.3.2)  13632717  13696251 (11.2.0.3.2)  13885389  13885388
 JAN2012  13343438 (11.2.0.3.1)  13466801  13348650 (11.2.0.3.1)  13413168  13413167
11.2.0.2
 Description  PSU   SPU(CPU)  GI PSU  Bundle Patch(Windows64bit)  Bundle Patch(Windows32bit)
 aOCT2013  17082367 (11.2.0.2.12)  17082375  17272753 (11.2.0.2.12)  17363838  17363837
 JUL2013  16619893 (11.2.0.2.11)  16742100  16742320 (11.2.0.2.11)  16345852  16345851
 APR2013  16056267 (11.2.0.2.10)  16294412  16166868 (11.2.0.2.10)  16345846  16345845
 JAN2013  14727315 (11.2.0.2.9)  14841437  14841385 (11.2.0.2.9)  16100399  16100398
 OCT2012  14275621 (11.2.0.2.8)  14390377  14390437 (11.2.0.2.8)  14672268  14672267
 JUL2012  13923804 (11.2.0.2.7)  14038791  14192201 (11.2.0.2.7)  14134043  14134042
 APR2012  13696224 (11.2.0.2.6)  13632725  13696242 (11.2.0.2.6)  13697074  13697073
 JAN2012  13343424 (11.2.0.2.5)  13343244  13653086 (11.2.0.2.5)  13413155  13413154
 OCT2011  12827726 (11.2.0.2.4)  12828071  12827731 (11.2.0.2.4)  13038788  13038787
 JUL2011  12419331 (11.2.0.2.3)  12419321  12419353 (11.2.0.2.3)  12714463  12714462
 APR2011  11724916 (11.2.0.2.2)  11724984  12311357 (11.2.0.2.2)  11896292  11896290
 JAN2011  10248523 (11.2.0.2.1)  N/A  N/A  10432053  10432052
11.2.0.1
 Description  PSU  CPU  Bundle Patch (Windows64bit)  Bundle Patch (Windows32bit)
 aJUL2011  12419378 (11.2.0.1.6)  12419278  12429529  12429528
 APR2011  11724930 (11.2.0.1.5)  11724991  11731176  11883240
 JAN2011  10248516 (11.2.0.1.4)  10249532  10432045  10432044
 OCT2010  9952216 (11.2.0.1.3)  9952260  10100101  10100100
 JUL2010  9654983 (11.2.0.1.2)  9655013  9736865  9736864
 APR2010  9352237 (11.2.0.1.1)  9369797  N/A  N/A
11.1.0.7
 Description  PSU  SPU(CPU)  Bundle Patch (Windows64bit)  Bundle Patch (Windows32bit)
bOCT2014  19152553 (11.1.0.7.21)  19274522  19609034  19609032
bJUL2014  18522513 (11.1.0.7.20)  18681875  18944208  18944207
bAPR2014  18031726 (11.1.0.7.19)  18139703  18372258  18372257
bJAN2014  17465583 (11.1.0.7.18)  17551415  17906936  17906935
bOCT2013  17082366 (11.1.0.7.17)  17082374  17363760  17363759
bJUL2013  16619896 (11.1.0.7.16)  16742110  16803788  16803787
bAPR2013  16056268 (11.1.0.7.15)  16308394  16345862  16345861
bJAN2013  14739378 (11.1.0.7.14)  14841452  15848067  15848066
 bOCT2012  14275623 (11.1.0.7.13)  14390384  14672313  14672312
 JUL2012  13923474 (11.1.0.7.12)  14038803  14109868  14109867
 APR2012  13621679 (11.1.0.7.11)  13632731  13715810  13715809
 JAN2012  13343461 (11.1.0.7.10)  13343453  13460956  13460955
 OCT2011  12827740 (11.1.0.7.9)  12828097  12914916  12914915
 JUL2011  12419384 (11.1.0.7.8)  12419265  12695278  12695277
 APR2011  11724936 (11.1.0.7.7)  11724999  11741170  11741169
 JAN2011  10248531 (11.1.0.7.6)  10249534  10350788  10350787
 OCT2010  9952228  (11.1.0.7.5)  9952269  9773825  9773817
 JUL2010  9654987 (11.1.0.7.4)  9655014  9869912  9869911
 APR2010  9352179 (11.1.0.7.3)  9369783  9392335  9392331
 JAN2010  9209238 (11.1.0.7.2)  9114072  9166861  9166858
 OCT2009  8833297 (11.1.0.7.1)  8836375  8928977  8928976
 JUL2009  N/A  8534338  8553515  8553512
 APR2009  N/A  8290478  8343070  8343061
11.1.0.6
 Description  CPU  Bundle Patch (Windows64bit)  Bundle Patch (Windows32bit)
 aJUL2009  8534378  8563155  8563154
 APR2009  8290402  8333657  8333655
 JAN2009  7592335  7631981  7631980
 OCT2008  7375639  7378393  7378392
 JUL2008  7150417  7210197  7210195
 APR2008  6864063  6867180  6867178
10.2.0.5
 Description  PSU  SPU(CPU)  Bundle Patch(Windows64bit)  Bundle Patch(Windows32bit)  Bundle Patch(WindowsItanium)
abJUL2013  16619894 (10.2.0.5.12)  16742123  16803782  16803780  16803781
 bAPR2013  16056270 (10.2.0.5.11)  16270946  16345857  16345855  16345856
 bJAN2013  14727319 (10.2.0.5.10)  14841459  15848062  15848060  15848061
 bOCT2012  14275629 (10.2.0.5.9)  14390396  14553358  14553356  14553357
 bJUL2012  13923855 (10.2.0.5.8)  14038805  14134053  14134051  14134052
 bAPR2012  13632743 (10.2.0.5.7)  13632738  13654815  13654814  13870404
 JAN2012  13343471 (10.2.0.5.6)  13343467  b13460968 b13460967  N/A
 bOCT2011  12827745 (10.2.0.5.5)  12828105  c12914913  12914911  N/A
 JUL2011  12419392 (10.2.0.5.4)  12419258  12429524  12429523  N/A
 APR2011  11724962 (10.2.0.5.3)  11725006  12328269  12328268  N/A
 JAN2011  10248542 (10.2.0.5.2)  10249537  10352673  10352672  N/A
 OCT2010  9952230 (10.2.0.5.1)  9952270  10099855  10058290  N/A
10.2.0.4
 Description  PSU  SPU(CPU)  Bundle Patch(Windows32bit)  Bundle Patch(Windows64bit)  Bundle Patch(WindowsItanium)
 bgJUL2013  16619897 (10.2.0.4.17)  16742253  N/A  N/A  N/A
 bgAPR2013  16056269 (10.2.0.4.16)  16270931  N/A  N/A  N/A
 bgJAN2013  14736542 (10.2.0.4.15)  14841471  N/A  N/A  N/A
bgOCT2012  14275630 (10.2.0.4.14)  14390410  N/A  N/A  N/A
bgJUL2012  13923851 (10.2.0.4.13)  14038814  N/A  N/A  N/A
 abAPR2012  12879933 (10.2.0.4.12)  12879926  13928775  13928776  N/A
 JAN2012  12879929 (10.2.0.4.11)  12879912  b13654060  N/A  N/A
 bOCT2011  12827778 (10.2.0.4.10)  12828112  12914908  12914910  12914909
 JUL2011  12419397 (10.2.0.4.9)  12419249  12429519  12429521  12429520
 APR2011  11724977 (10.2.0.4.8)  11725015  12328501  12328503  12328502
 JAN2011  10248636 (10.2.0.4.7)  10249540  10349197  10349200  10349198
 OCT2010  9952234 (10.2.0.4.6)  9952272  10084980  10084982  10084981
 JUL2010  9654991 (10.2.0.4.5)  9655017  9777076  9777078  9777077
 APR2010  9352164 (10.2.0.4.4)  9352191  9393548  9393550  9393549
 JAN2010  9119284 (10.2.0.4.3)  9119226  9169457  9169460  9169458
 OCT2009  8833280 (10.2.0.4.2)  8836308  8880857  8880861  8880858
 JUL2009  8576156 (10.2.0.4.1)  8534387  8559466  8559467  8541782
 APR2009  N/A  8290506  8307237  8307238  8333678
 JAN2009  N/A  7592346  7584866  7584867  N/A
 OCT2008  N/A  7375644  7386320  7386321  N/A
 JUL2008  N/A  7150470  7218676  7218677  N/A
10.2.0.3
 Description  CPU (Unix/Linux)  Bundle Patch (Windows32bit)  Bundle Patch (WindowsItanium)  Bundle Patch (Windows64bit)
 aJAN2009  7592354  7631956  7631958  7631957
 OCT2008  7369190  7353782  7353784  7353785
 JUL2008  7150622  7252496  7252497  7252498
 APR2008  6864068  6867054  6867055  6867056
 JAN2008  6646853  6637237  6637238  6637239
 OCT2007  6394981  6430171  6430173  6430174
 JUL2007  6079591  6116131  6038242  6116139
 APR2007  5901891  5948242  5916262  5948243
 JAN2007  5881721  5846376  5846377  5846378
10.2.0.2
 Description  CPU (Unix/Linux)  Bundle Patch (Windows32bit)   Bundle Patch (Windows64bit)  Bundle Patch (WindowsItanium)
 iJAN2009  7592355  N/A  N/A  N/A
 hOCT2008  7375660  N/A  N/A  N/A
 hJUL2008  7154083  N/A  N/A  N/A
 hAPR2008  6864071  N/A  N/A  N/A
 aJAN2008  6646850  N/A  N/A  N/A
 fOCT2007  6394997  6397028  6397030  6397029
 JUL2007  6079588  6013105  6013121  6013118
 APR2007  5901881  5912173  5912179  5912176
 JAN2007  5689957  5716143  5699839  5699824
 OCT2006  5490848  5502226  5500921  5500894
 JUL2006  5225799  5251025  5251028  5251026
 APR2006  5079037  5140461  5140567  5140508
10.2.0.1
 Description  CPU (Unix/Linux)  Bundle Patch (Windows32bit)  Bundle Patch (Windows64bit)  Bundle Patch (WindowsItanium)
 APR2007  5901880  N/A  N/A  N/A
 JAN2007  5689937  5695784  5695786  5695785
 OCT2006  5490846  5500927  5500954  5500951
 JUL2006  5225798  5239698  5239701  5239699
 APR2006  5049080  5059238  5059261  5059251
 JAN2006  4751931  4751539  4770480  4751549
10.1.0.5
 Description  CPU (Unix/Linux)  Bundle Patch (Windows32bit)   Bundle Patch (WindowsItanium)
 JAN2012  13343482  13413002  13413003
 OCT2011  12828135  12914905  12914906
 JUL2011  12419228  12429517  12429518
 APR2011  11725035  11731119  11731120
 JAN2011  N/A  N/A  N/A
 OCT2010  9952279  10089559  10089560
 JUL2010  9655023  9683651  9683652
 APR2010  9352208  9390288  9390289
 JAN2010  9119261  9187104  9187105
 OCT2009  8836540  8785211  8785212
 JUL2009  8534394  8656224  8656226
 APR2009  8290534  8300356  8300360
 JAN2009  7592360  7486619  7586049
 OCT2008  7375686  7367493  7367494
 JUL2008  7154097  7047034  7047037
 APR2008  6864078  6867107  6867108
 JAN2008  6647005  6637274  6637275
 OCT2007  6395024  6408393  6408394
 JUL2007  6079585  6115804  6115818
 APR2007  5901877  5907304  5907305
 JAN2007  5689908  5716295  5634747
 OCT2006  5490845  5500883  5500885
 JUL2006  5225797  5251148  5251140
 APR2006  5049074  5057606  5057609
 JAN2006  4751932  4882231  4882236
10.1.0.4
 Description  CPU (Unix/Linux)  Bundle Patch (Windows32bit)  Bundle Patch (WindowsItanium)
 APR2007  5901876  5909871  5909879
 JAN2007  5689894  5695771  5695772
 OCT2006  5490844  5500878  5500880
 JUL2006  5225796  5239736  5239737
 APR2006  5049067  5059200  5059227
 JAN2006  4751928  4751259  4745040
 OCT2005  4567866  4579182  4579188
 JUL2005  4392423  4440706  4404600
 APR2005  4210374  4287619  4287611
10.1.0.3
 Description  CPU (Unix/Linux)  Bundle Patch (Windows32bit)  Bundle Patch (WindowsItanium)
 JAN2007  5923277  N/A  N/A
 OCT2006  5566825  N/A  N/A
 JUL2006  5435164  N/A  N/A
 APR2006  5158022  N/A  N/A
 JAN2006  4751926  4741077  4741084
 OCT2005  4567863  4567518  4567523
 JUL2005  4392409  4389012  4389014
 APR2005  4193286  4269715  4158888
 JAN2005  4003062  4074232  3990812
10.1.0.2
 Description  CPU (Unix/Linux)  Bundle Patch (Windows32bit)  Bundle Patch (WindowsItanium)
 APR2005  4193293  4181849  4213305
 JUL2005  4400766  4388944  4388948
 JAN2005  4003051  4104364  4083038
9.2.0.8
 Description  CPU (Unix/Linux)  Bundle Patch (Windows32bit)  Bundle Patch (WindowsItanium)
 JUL2010  9655027  9683644  9683645
 APR2010  9352224  9390286  N/A
 JAN2010  9119275  9187106  N/A
 OCT2009  8836758  8785185  8785186
 JUL2009  8534403  8427417  8427418
 APR2009  8290549  8300340  8300346
 JAN2009  7592365  7703210  7703212
 OCT2008  7375695  7394394  7394402
 JUL2008  7154111  7047026  7047029
 APR2008  6864082  6867138  6867139
 JAN2008  6646842  6637265  6637266
 OCT2007  6395038  6417013  6417014
 JUL2007  6079582  6130293  6130295
 APR2007  5901875  5916268  5916275
 JAN2007  N/A  N/A  N/A
 OCT2006  5490859  5652380  5639519
9.2.0.7
 Description  CPU (Unix/Linux)  Bundle Patch (Windows32bit)  Bundle Patch (WindowsItanium)
 JUL2007  6079579  6146759  6146748
 APR2007  5901872  5907274  5907275
 JAN2007  5689875  5654905  5654909
 OCT2006  5490841  5500873  5500874
 JUL2006  5225794  5250980  5250981
 APR2006  5049060  5064365  5064364
 JAN2006  4751923  4751528  4741074
 OCT2005  4567854  4579590  4579599
 JUL2005  4547566  N/A  N/A
9.2.0.6
 Description  CPU (Unix/Linux)  Bundle Patch (Windows32bit)  Bundle Patch (WindowsItanium)
 OCT2006  5490840  5500865  5500871
 JUL2006  5225793  5239794  5239793
 APR2006  5049051  5059614  5059615
 JAN2006  4751921  4751261  4751262
 OCT2005  4567846  4579093  4579097
 JUL2005  4392392  4445852  4401917
 APR2005  4193295  4269928  4213298
9.2.0.5
 Description  CPU (Unix/Linux)  Bundle Patch (Windows32bit)  Bundle Patch (WindowsItanium)
 OCT2006  5689708  N/A  N/A
 JUL2006  5435138  N/A  N/A
 APR2006  5219762  N/A  N/A
 OCT2005  4560421  N/A  N/A
 JUL2005  4392256  4387563  4391819
 APR2005  4193299  4195791  4214192
 JAN2005  4003006  4104374  3990809
9.2.0.4
 Description  CPU (Unix/Linux)  Bundle Patch (Windows32bit)  Bundle Patch (WindowsItanium)
 JAN2005  4002994  4104369  4083202
8.1.7.4
 Description  CPU (Unix/Linux)  Bundle Patch (Windows32bit)
 JAN2007  5689799  5686514
 OCT2006  5490835  5496067
 JUL2006  5225788  5236412
 APR2006  5045247  5057601
 JAN2006  4751906  4751570
 OCT2005  4560405  4554818
 JUL2005  4392446  4437058
 APR2005  4193312  4180163
 JAN2005  4002909  3921893

参考:Quick Reference to Patch Numbers for Database PSU, SPU(CPU), Bundle Patches and Patchsets (文档 ID 1454618.1)

11.2.0.4 GI单独安装tfa

在11.2.0.4安装rac执行root.sh之时需要在root的环境变量中指定可以直接执行unzip命令(在非Linux,Win环境下root用户默认环境变量无unzip命令),如果忘记执行配置相应的PATH环境变量,将导致tfa不被安装,事后安装步骤
未安装tfa现状

--/etc/inittab中只有这一条和gi相关
h1:2:respawn:/etc/init.ohasd run >/dev/null 2>&1 </dev/null
--只启动了init.ohasd
xifenf01/oradata/sys/soft#ps -ef|grep init
    root        1        0   0 13:12:11      -  0:00 /etc/init
    root 30539998        1   0 18:58:17      -  0:00 /bin/sh /etc/init.ohasd run
    root 31391906  5177692   0 19:44:15  pts/1  0:00 grep init

使用tfa_setup.sh脚本安装

xifenf01/#export PATH=$PATH:/u01/oracle/app/grid/bin
xifenf01/#/u01/oracle/app/grid/crs/install/tfa_setup.sh -silent -crshome /u01/oracle/app/grid
Starting TFA installation
TFA requires BASH shell. Please install bash and try again.

提示缺少bash shell,下载相关包安装安装(系统AIX 7.1)

xifenf01/oradata/sys/soft#rpm -ivh bash-4.2-3.aix6.1.ppc.rpm
bash                        ##################################################

重新安装tfa

xifenf01/oradata/sys/soft#/u01/oracle/app/grid/crs/install/tfa_setup.sh -silent -crshome /u01/oracle/app/grid
Starting TFA installation
Using JAVA_HOME : /u01/oracle/app/grid/jdk/jre
Running Auto Setup for TFA as user root...
The following installation requires temporary use of SSH.
If SSH is not configured already then we will remove SSH
when complete.
Installing TFA now...
TFA Will be Installed on xifenf01...
TFA will scan the following Directories
++++++++++++++++++++++++++++++++++++++++++++
.-----------------------------------------------------.
|                       xifenf01                      |
+------------------------------------------+----------+
| Trace Directory                          | Resource |
+------------------------------------------+----------+
| /u01/oracle/app/grid/OPatch/crs/log      | CRS      |
| /u01/oracle/app/grid/cfgtoollogs         | INSTALL  |
| /u01/oracle/app/grid/crs/log             | CRS      |
| /u01/oracle/app/grid/cv/log              | CRS      |
| /u01/oracle/app/grid/evm/admin/log       | CRS      |
| /u01/oracle/app/grid/evm/admin/logger    | CRS      |
| /u01/oracle/app/grid/evm/log             | CRS      |
| /u01/oracle/app/grid/install             | INSTALL  |
| /u01/oracle/app/grid/log                 | CRS      |
| /u01/oracle/app/grid/log/                | CRS      |
| /u01/oracle/app/grid/network/log         | CRS      |
| /u01/oracle/app/grid/oc4j/j2ee/home/log  | CRSOC4J  |
| /u01/oracle/app/grid/opmn/logs           | CRS      |
| /u01/oracle/app/grid/racg/log            | CRS      |
| /u01/oracle/app/grid/rdbms/log           | ASM      |
| /u01/oracle/app/grid/scheduler/log       | CRS      |
| /u01/oracle/app/grid/srvm/log            | CRS      |
| /u01/oracle/app/oraInventory/ContentsXML | INSTALL  |
| /u01/oracle/app/oraInventory/logs        | INSTALL  |
'------------------------------------------+----------'
Installing TFA on xifenf01
HOST: xifenf01  TFA_HOME: /u01/oracle/app/grid/tfa/xifenf01/tfa_home
.-----------------------------------------------------.
| Host     | Status of TFA | PID     | Port | Version |
+----------+---------------+---------+------+---------+
| xifenf01 | RUNNING       | 7536914 | 5000 | 2.5.1.5 |
'----------+---------------+---------+------+---------'
Summary of TFA Installation:
.------------------------------------------------------------------.
|                             xifenf01                             |
+---------------------+--------------------------------------------+
| Parameter           | Value                                      |
+---------------------+--------------------------------------------+
| Install location    | /u01/oracle/app/grid/tfa/xifenf01/tfa_home |
| Repository location | /u01/oracle/app/oracle/tfa/repository      |
| Repository usage    | 0 MB out of 10240 MB                       |
'---------------------+--------------------------------------------'
TFA is successfully installed..
Usage : /u01/oracle/app/grid/tfa/bin/tfactl <command> [options]
<command> =
         print        Print requested details
         purge        Delete collections from TFA repository
         directory    Add or Remove or Modify directory in TFA
         host         Add or Remove host in TFA
         set          Turn ON/OFF or Modify various TFA features
         diagcollect  Collect logs from across nodes in cluster
For help with a command: /u01/oracle/app/grid/tfa/bin/tfactl <command> -help

安装tfa成功后

--/etc/inittab
h1:2:respawn:/etc/init.ohasd run >/dev/null 2>&1 </dev/null
htfa:2:respawn:/etc/init.tfa run >/dev/null 2>&1 </dev/null
--init.tfa进程存在
xifenf01/oradata/sys/soft#ps -ef|grep init
    root        1        0   0 13:12:11      -  0:00 /etc/init
    root 30277638        1   0 19:26:37      -  0:00 /bin/sh /etc/init.tfa run
    root 30539998        1   0 18:58:17      -  0:00 /bin/sh /etc/init.ohasd run
    root 31391906  5177692   0 19:44:15  pts/1  0:00 grep init

另外一个节点安装tfa

xifenf02/#export PATH=$PATH:/u01/oracle/app/grid/bin
xifenf02/oradata/sys/soft#rpm -ivh bash-4.2-3.aix6.1.ppc.rpm.rpm
bash                        ##################################################
xifenf02/#/u01/oracle/app/grid/crs/install/tfa_setup.sh -silent -crshome /u01/oracle/app/grid
Starting TFA installation
Using JAVA_HOME : /u01/oracle/app/grid/jdk/jre
Running Auto Setup for TFA as user root...
The following installation requires temporary use of SSH.
If SSH is not configured already then we will remove SSH
when complete.
Installing TFA now...
TFA Will be Installed on xifenf02...
TFA will scan the following Directories
++++++++++++++++++++++++++++++++++++++++++++
.-----------------------------------------------------.
|                       xifenf02                      |
+------------------------------------------+----------+
| Trace Directory                          | Resource |
+------------------------------------------+----------+
| /u01/oracle/app/grid/OPatch/crs/log      | CRS      |
| /u01/oracle/app/grid/cfgtoollogs         | INSTALL  |
| /u01/oracle/app/grid/crs/log             | CRS      |
| /u01/oracle/app/grid/cv/log              | CRS      |
| /u01/oracle/app/grid/evm/admin/log       | CRS      |
| /u01/oracle/app/grid/evm/admin/logger    | CRS      |
| /u01/oracle/app/grid/evm/log             | CRS      |
| /u01/oracle/app/grid/install             | INSTALL  |
| /u01/oracle/app/grid/log                 | CRS      |
| /u01/oracle/app/grid/log/                | CRS      |
| /u01/oracle/app/grid/network/log         | CRS      |
| /u01/oracle/app/grid/oc4j/j2ee/home/log  | CRSOC4J  |
| /u01/oracle/app/grid/opmn/logs           | CRS      |
| /u01/oracle/app/grid/racg/log            | CRS      |
| /u01/oracle/app/grid/rdbms/log           | ASM      |
| /u01/oracle/app/grid/scheduler/log       | CRS      |
| /u01/oracle/app/grid/srvm/log            | CRS      |
| /u01/oracle/app/oraInventory/ContentsXML | INSTALL  |
| /u01/oracle/app/oraInventory/logs        | INSTALL  |
'------------------------------------------+----------'
Installing TFA on xifenf02
HOST: xifenf02  TFA_HOME: /u01/oracle/app/grid/tfa/xifenf02/tfa_home
.-----------------------------------------------------.
| Host     | Status of TFA | PID     | Port | Version |
+----------+---------------+---------+------+---------+
| xifenf02 | RUNNING       | 5898636 | 5000 | 2.5.1.5 |
| xifenf01 | RUNNING       | 7536914 | 5000 | 2.5.1.5 |
'----------+---------------+---------+------+---------'
Summary of TFA Installation:
.------------------------------------------------------------------.
|                             xifenf02                             |
+---------------------+--------------------------------------------+
| Parameter           | Value                                      |
+---------------------+--------------------------------------------+
| Install location    | /u01/oracle/app/grid/tfa/xifenf02/tfa_home |
| Repository location | /u01/oracle/app/oracle/tfa/repository      |
| Repository usage    | 0 MB out of 10240 MB                       |
'---------------------+--------------------------------------------'
TFA is successfully installed..
Usage : /u01/oracle/app/grid/tfa/bin/tfactl <command> [options]
<command> =
         print        Print requested details
         purge        Delete collections from TFA repository
         directory    Add or Remove or Modify directory in TFA
         host         Add or Remove host in TFA
         set          Turn ON/OFF or Modify various TFA features
         diagcollect  Collect logs from across nodes in cluster
For help with a command: /u01/oracle/app/grid/tfa/bin/tfactl <command> -help

11gR2 GI打PSU ORACLE_HOME空间不足案例—需要20G以上空闲空间

GI打auto打patch命令

#/u01/oracle/app/grid/OPatch/opatch auto /u01/soft/18706472 -ocmrf /u01/soft/ocm.rsp
Executing /u01/oracle/app/grid/perl/bin/perl /u01/oracle/app/grid/OPatch/crs/patch11203.pl
-patchdir /u01/soft -patchn 18706472 -ocmrf /u01/soft/ocm.rsp -paramfile /u01/oracle/app/grid/crs/install/crsconfig_params
This is the main log file: /u01/oracle/app/grid/cfgtoollogs/opatchauto2014-10-15_01-15-28.log
This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/oracle/app/grid/cfgtoollogs/opatchauto2014-10-15_01-15-28.report.log
2014-10-15 01:15:28: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/oracle/app/grid/crs/install/crsconfig_params
Stopping CRS...
Stopped CRS successfully
patch /u01/soft/18706472/18522509  apply successful for home  /u01/oracle/app/grid
patch /u01/soft/18706472/18522515  apply failed  for home  /u01/oracle/app/grid
Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.
opatch auto succeeded.

这里可以看到,打patch 18522509成功,后续patch都失败,而不是仅仅18522515失败

分析错误日志

Composite patch 18522509 successfully applied.
 OPatch Session completed with warnings.
 Log file location: /u01/oracle/app/grid/cfgtoollogs/opatch/opatch2014-10-15_01-19-00AM_1.log
 OPatch completed with warnings.
2014-10-15 01:21:44: patch /u01/soft/18706472/18522509  apply successful for home  /u01/oracle/app/grid
2014-10-15 01:21:44: Executing command /u01/oracle/app/grid/OPatch/opatch napply /u01/soft/18706472/18522515 -local -silent -ocmrf /
u01/soft/ocm.rsp -oh /u01/oracle/app/grid -invPtrLoc /u01/oracle/app/grid/oraInst.loc as grid
2014-10-15 01:21:44: Running as user grid: /u01/oracle/app/grid/OPatch/opatch napply /u01/soft/18706472/18522515 -local -silent -ocm
rf /u01/soft/ocm.rsp -oh /u01/oracle/app/grid -invPtrLoc /u01/oracle/app/grid/oraInst.loc
2014-10-15 01:21:44: s_run_as_user2: Running /bin/su grid -c ' /u01/oracle/app/grid/OPatch/opatch napply /u01/soft/18706472/18522515
 -local -silent -ocmrf /u01/soft/ocm.rsp -oh /u01/oracle/app/grid -invPtrLoc /u01/oracle/app/grid/oraInst.loc '
2014-10-15 01:21:49: Removing file /tmp/uaa7jC7eu
2014-10-15 01:21:49: Successfully removed file: /tmp/uaa7jC7eu
2014-10-15 01:21:49: /bin/su exited with rc=73
2014-10-15 01:21:49: status of apply patch is 18688
2014-10-15 01:21:49: The apply patch output is Oracle Interim Patch Installer version 11.2.0.3.6
 Copyright (c) 2013, Oracle Corporation.  All rights reserved.
 Oracle Home       : /u01/oracle/app/grid
 Central Inventory : /u01/oracle/app/oraInventory
    from           : /u01/oracle/app/grid/oraInst.loc
 OPatch version    : 11.2.0.3.6
 OUI version       : 11.2.0.4.0
 Log file location : /u01/oracle/app/grid/cfgtoollogs/opatch/opatch2014-10-15_01-21-44AM_1.log
 Verifying environment and performing prerequisite checks...
 Prerequisite check "CheckSystemSpace" failed.
 The details are:
 Required amount of space(24021.839MB) is not available.
 UtilSession failed:
 Prerequisite check "CheckSystemSpace" failed.
 Log file location: /u01/oracle/app/grid/cfgtoollogs/opatch/opatch2014-10-15_01-21-44AM_1.log
 OPatch failed with error code 73

这里可以看出来,patch 18522509 成功,但是18522515在check的时候失败,由于$ORACLE_HOME(GI)需要可用空间为24021.839MB,现在空闲空间不足

查看空间并腾空间

# df -g
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4          10.00      9.56    5%    12443     1% /
/dev/hd2          15.00     11.42   24%    60339     3% /usr
/dev/hd9var       10.00      9.91    1%     3310     1% /var
/dev/hd3          10.00      9.65    4%      364     1% /tmp
/dev/hd1          10.00     10.00    1%       32     1% /home
/dev/hd11admin      0.25      0.25    1%        5     1% /admin
/proc                 -         -    -         -     -  /proc
/dev/hd10opt      10.00      8.98   11%    12401     1% /opt
/dev/fslv00       50.00     12.94   75%    59137     2% /u01
/dev/odm           0.00      0.00   -1%        6   100% /dev/odm
/dev/vx/dsk/crs_dg/crs_vol      1.96      1.75   12%        5     1% /crsdata
/dev/vx/dsk/ora_dg/data01_vol   2048.00   2031.32    1%        4     1% /oradata/data01
/dev/vx/dsk/ora_dg/sys_vol    400.00    350.29   13%     3702     1% /oradata/sys
# cd /u01
# ls
lost+found  oracle      soft
# cd soft
# du -sg
17.34   .
# mv /u01/soft /oradata/sys/
# df -g
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4          10.00      9.56    5%    12443     1% /
/dev/hd2          15.00     11.42   24%    60339     3% /usr
/dev/hd9var       10.00      9.91    1%     3310     1% /var
/dev/hd3          10.00      9.65    4%      364     1% /tmp
/dev/hd1          10.00     10.00    1%       32     1% /home
/dev/hd11admin      0.25      0.25    1%        5     1% /admin
/proc                 -         -    -         -     -  /proc
/dev/hd10opt      10.00      8.98   11%    12401     1% /opt
/dev/fslv00       50.00     27.56   45%    59137     2% /u01
/dev/odm           0.00      0.00   -1%        6   100% /dev/odm
/dev/vx/dsk/crs_dg/crs_vol      1.96      1.75   12%        5     1% /crsdata
/dev/vx/dsk/ora_dg/data01_vol   2048.00   2031.32    1%        4     1% /oradata/data01
/dev/vx/dsk/ora_dg/sys_vol    400.00    333.54   14%     3702     1% /oradata/sys
# cd soft
# ls -ltr
total 12464202
drwxr-xr-x    5 grid     dba            1024 Jul 07 21:06 18706472
-rw-rw-r--    1 grid     dba            2123 Jul 15 20:06 PatchSearch.xml
-rw-r--r--    1 root     system   1801653734 Oct 14 14:39 p13390677_112040_aix64-5l_1of7.zip
-rw-r--r--    1 root     system   1170882875 Oct 14 14:40 p13390677_112040_aix64-5l_2of7.zip
-rw-r--r--    1 root     system   2127071138 Oct 14 14:41 p13390677_112040_aix64-5l_3of7.zip
-rw-r--r--    1 root     system   1242090126 Oct 14 14:43 p18706472_112040_AIX64-5L.zip
-rw-r--r--    1 root     system     34964928 Oct 14 14:43 p6880880_112000_AIX64-5L.zip
-rwxr-xr-x    1 root     system       127965 Oct 14 14:48 unzip_aix
-rw-r--r--    1 root     system      4871110 Oct 14 14:50 ONEOFF_112043_AIX64.zip
drwxr-xr-x    8 grid     dba            1024 Oct 14 14:55 grid
drwxr-xr-x    8 oracle   dba            1024 Oct 14 14:57 database
-rw-r--r--    1 grid     dba             621 Oct 15 00:11 ocm.rsp

确实/u01空间不足,通过把RAC安装文件迁移到其他目录,确保/u01有足够空间用来打patch

回退命令

# /u01/oracle/app/grid/OPatch/opatch auto /u01/soft/18706472 -rollback -ocmrf /u01/soft/ocm.rsp

重新打patch

#/u01/oracle/app/grid/OPatch/opatch auto /u01/soft/18706472 -ocmrf /u01/soft/ocm.rsp                 <
Executing /u01/oracle/app/grid/perl/bin/perl /u01/oracle/app/grid/OPatch/crs/patch11203.pl -patchdir /oradata/sys/soft
-patchn 18706472 -ocmrf /oradata/sys/soft/ocm.rsp -paramfile /u01/oracle/app/grid/crs/install/crsconfig_params
This is the main log file: /u01/oracle/app/grid/cfgtoollogs/opatchauto2014-10-15_01-57-42.log
This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/oracle/app/grid/cfgtoollogs/opatchauto2014-10-15_01-57-42.report.log
2014-10-15 01:57:42: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/oracle/app/grid/crs/install/crsconfig_params
Stopping CRS...
Stopped CRS successfully
patch /oradata/sys/soft/18706472/18522509  apply successful for home  /u01/oracle/app/grid
patch /oradata/sys/soft/18706472/18522515  apply successful for home  /u01/oracle/app/grid
patch /oradata/sys/soft/18706472/18522514  apply successful for home  /u01/oracle/app/grid
Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.
opatch auto succeeded.

检查确认patch成功

# su - grid
xifenfei1:/home/grid> opatch lspatches
18522514;ACFS PATCH SET UPDATE : 11.2.0.4.3 (18522514)
18522515;OCW Patch Set Update : 11.2.0.4.3 (18522515)
18522509;Database Patch Set Update : 11.2.0.4.3 (18522509)
xifenfei1:/home/grid> crsctl status res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       xifenfei1
               ONLINE  ONLINE       xifenfei2
ora.asm
               OFFLINE OFFLINE      xifenfei1
               OFFLINE OFFLINE      xifenfei2
ora.gsd
               OFFLINE OFFLINE      xifenfei1
               OFFLINE OFFLINE      xifenfei2
ora.net1.network
               ONLINE  ONLINE       xifenfei1
               ONLINE  ONLINE       xifenfei2
ora.ons
               ONLINE  ONLINE       xifenfei1
               ONLINE  ONLINE       xifenfei2
ora.registry.acfs
               OFFLINE OFFLINE      xifenfei1
               OFFLINE OFFLINE      xifenfei2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       xifenfei2
ora.cvu
      1        ONLINE  ONLINE       xifenfei2
ora.xifenfei1.vip
      1        ONLINE  ONLINE       xifenfei1
ora.xifenfei2.vip
      1        ONLINE  ONLINE       xifenfei2
ora.oc4j
      1        ONLINE  ONLINE       xifenfei2
ora.scan1.vip
      1        ONLINE  ONLINE       xifenfei2

通过多次resetlogs规避类似ORA-01248: file N was created in the future of incomplete recovery错误

数据库现状
控制文件
recover_xifenfei0
控制文件中数据文件信息
recover_xifenfei1
数据文件头信息
recover_xifenfei2
redo信息
recover_xifenfei3
根据当前数据库恢复检查脚本(Oracle Database Recovery Check)收集的信息,数据库的是非归档状态,而且redo已经覆盖,数据库datafile 5 无法直接online.遇到这样情况,可以使用bbed修改文件头scn实现online(使用bbed让rac中的sysaux数据文件online),也可以通过使用_allow_resetlogs_corruption等隐含参数实现online.本恢复案例中有180个数据文件,160个offline,然后open数据库,所以大量数据文件无法正常online,bbed工作量太大.在恢复过程中不幸遇到ORA-01248

数据库resetlogs出现ORA-01248错误

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01248: file 5 was created in the future of incomplete recovery
ORA-01110: data file 5: 'F:\TTDATA\PUBRTS.DAT'

alert日志记录

Fri Oct 10 15:09:26 2014
alter database open resetlogs
Fri Oct 10 15:09:26 2014
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
ORA-1248 signalled during: alter database open resetlogs...
Fri Oct 10 15:15:22 2014
alter database open
Fri Oct 10 15:15:22 2014
ORA-1589 signalled during: alter database open...
Fri Oct 10 15:15:30 2014
alter database  open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
ORA-1248 signalled during: alter database  open resetlogs...

尝试offline文件然后resetlogs

SQL>ALTER DATABASE DATAFILE 5  OFFLINE;
Database altered.
sql>ALTER DATABASE OPEN RESETLOGS;
ERROR at line 1:
ORA-01245: ffline file 5 will be lost if resetlogs is done
ORA-01110: data file 5: 'F:\TTDATA\PUBRTS.DAT'

alert日志

Fri Oct 10 15:19:37 2014
ALTER DATABASE DATAFILE 5 offline
Fri Oct 10 15:19:37 2014
Completed: ALTER DATABASE DATAFILE 5 offline
Fri Oct 10 15:19:40 2014
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
ORA-1245 signalled during: alter database open resetlogs...

出现该错误原因是由于数据库是非归档模式,offline数据文件需要使用offline drop

Fri Oct 10 15:22:16 2014
alter database datafile 5 offline drop
Fri Oct 10 15:22:17 2014
Completed: alter database datafile 5 offline drop
Fri Oct 10 15:23:13 2014
alter database open resetlogs
Fri Oct 10 15:23:14 2014
Fri Oct 10 15:23:49 2014
RESETLOGS after complete recovery through change 1422423346
Resetting resetlogs activation ID 3503292347 (0xd0cfffbb)
Fri Oct 10 15:24:01 2014
Setting recovery target incarnation to 3
Fri Oct 10 15:24:04 2014
Assigning activation ID 3649065262 (0xd980512e)
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=23, OS id=3772
Fri Oct 10 15:24:04 2014
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=24, OS id=3668
Fri Oct 10 15:24:05 2014
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\ORACLE\PRODUCT\10.2.0\ORADATA\CLTTDB\REDO01.LOG
Successful open of redo thread 1
Fri Oct 10 15:24:05 2014
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Oct 10 15:24:05 2014
ARC0: STARTING ARCH PROCESSES
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the 'no FAL' ARCH
ARC2 started with pid=25, OS id=636
Fri Oct 10 15:24:06 2014
ARC0: Becoming the 'no SRL' ARCH
Fri Oct 10 15:24:06 2014
ARC1: Becoming the heartbeat ARCH
Fri Oct 10 15:24:06 2014
SMON: enabling cache recovery
Fri Oct 10 15:24:07 2014
Successfully onlined Undo Tablespace 1.
Dictionary check beginning
File #5 is offline, but is part of an online tablespace.
data file 5: 'F:\TTDATA\PUBRTS.DAT'
Dictionary check complete
Fri Oct 10 15:24:19 2014
SMON: enabling tx recovery
Fri Oct 10 15:24:19 2014
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=26, OS id=868
Fri Oct 10 15:24:21 2014
LOGSTDBY: Validating controlfile with logical metadata
Fri Oct 10 15:24:21 2014
LOGSTDBY: Validation complete
Completed: alter database open resetlogs

open成功后,再次resetlogs库,实现数据文件online

Fri Oct 10 15:28:44 2014
ALTER DATABASE DATAFILE 5 online
Fri Oct 10 15:28:44 2014
Completed: ALTER DATABASE DATAFILE 5 online
Fri Oct 10 15:31:46 2014
alter database open resetlogs
Fri Oct 10 15:31:46 2014
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
Setting recovery target incarnation to 4
Fri Oct 10 15:32:00 2014
Assigning activation ID 3649091231 (0xd980b69f)
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=23, OS id=700
Fri Oct 10 15:32:00 2014
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=24, OS id=3360
Fri Oct 10 15:32:01 2014
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\ORACLE\PRODUCT\10.2.0\ORADATA\CLTTDB\REDO01.LOG
Successful open of redo thread 1
Fri Oct 10 15:32:01 2014
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Oct 10 15:32:01 2014
ARC0: STARTING ARCH PROCESSES
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the 'no FAL' ARCH
ARC2 started with pid=25, OS id=2016
Fri Oct 10 15:32:02 2014
ARC0: Becoming the 'no SRL' ARCH
Fri Oct 10 15:32:02 2014
ARC1: Becoming the heartbeat ARCH
Fri Oct 10 15:32:02 2014
SMON: enabling cache recovery
Fri Oct 10 15:32:03 2014
Successfully onlined Undo Tablespace 1.
Dictionary check beginning
Fri Oct 10 15:32:15 2014
Dictionary check complete
Fri Oct 10 15:32:15 2014
SMON: enabling tx recovery
Fri Oct 10 15:32:15 2014
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=26, OS id=256
Fri Oct 10 15:32:17 2014
LOGSTDBY: Validating controlfile with logical metadata
Fri Oct 10 15:32:17 2014
LOGSTDBY: Validation complete
Completed: alter database open resetlogs

MOS又一次不靠谱—ORA-27163: out of memory

数据库版本

oracle -> 11g @xifenfei:/home/oracle$sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Tue Sep 30 10:28:30 2014
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
SQL> select * from v$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL Release 11.2.0.4.0 - Production
CORE    11.2.0.4.0      Production
TNS for HPUX: Version 11.2.0.4.0 - Production
NLSRTL Version 11.2.0.4.0 - Production

数据库补丁信息

oracle -> 11g @xifenfei:/home/oracle$opatch lspatches
18973907;
18795105;
18701707;
18284357;
18020394;
17564992;
17308789;
17306264;
17259786;
17079301;
16477664;
16188701;
14368995;
14245531;
11744544;
18522515;OCW Patch Set Update : 11.2.0.4.3 (18522515)
18522509;Database Patch Set Update : 11.2.0.4.3 (18522509)

收集spa报告报ORA-27163: out of memory错误

SQL> SELECT dbms_sqlpa.report_analysis_task('SPA_TEST', 'HTML', 'errors','ALL') FROM dual;
ERROR:
ORA-27163: out of memory
ORA-06512: at "SYS.DBMS_SQLTUNE_INTERNAL", line 8211
ORA-06512: at "SYS.DBMS_SQLPA", line 515
ORA-06512: at line 1
no rows selected

设置event后收集

Connected.
SQL> alter session set events '31156 trace name context forever, level 0x400';
Session altered.
SQL> SELECT dbms_sqlpa.report_analysis_task('SPA_TEST', 'HTML', 'errors','ALL') FROM dual;
DBMS_SQLPA.REPORT_ANALYSIS_TASK('SPA_TEST','HTML','ERRORS','ALL')
--------------------------------------------------------------------------------
<html>
    <head>
        <title>
   SQL Performance Impact Analyzer Report

查询MOS发现XML Parser Fails With ORA-27163 (Out Of Memory) (Doc ID 1599434.1),相关描述:
bug-xml
按照文档描述,该问题在11.2.0.4中已经修复,可是在hp unix中依然存在该问题,经验告诉我们,现在的MOS不能完全相信