ORA-00001: unique constraint (PERFSTAT.STATS$SQL_SUMMARY_PK) violated

出现如下错误(ORA-00001: unique constraint (PERFSTAT.STATS$SQL_SUMMARY_PK) violated)

Oracle9i Enterprise Edition Release 9.2.0.6.0 - 64bit Production
With the Partitioning option
JServer Release 9.2.0.6.0 - Production
ORACLE_HOME = /oracle9/app/product/9.2.0
System name:    AIX
Node name:      zwq_bill_1
Release:        1
Version:        6
Machine:        00F64FF34C00
Instance name: bill1
Redo thread mounted by this instance: 1
Oracle process number: 30
Unix process pid: 46531060, image: oracle@zwq_bill_1 (J000)
*** SESSION ID:(218.47085) 2012-04-02 19:30:45.561
*** 2012-04-02 19:30:45.561
ORA-12012: error on auto execute of job 1
ORA-00001: unique constraint (PERFSTAT.STATS$SQL_SUMMARY_PK) violated
ORA-06512: at "PERFSTAT.STATSPACK", line 1361
ORA-06512: at "PERFSTAT.STATSPACK", line 2471
ORA-06512: at "PERFSTAT.STATSPACK", line 91
ORA-06512: at line 1

这个是oracle的一个Bug 2784796,提供解决方法有
1.run the statspack at level 0
2.restart the instance
3.set cursor sharing to exact (probably not feasible)
4.禁用主键,创建合适非唯一index

ALTER TABLE PERFSTAT.STATS$SQL_SUMMARY MODIFY
CONSTRAINT STATS$SQL_SUMMARY_PK DISABLE NOVALIDATE;

5.修改STATS$V_$SQLXS视图
分析思路如下:
1)根据主键冲突找到主键包含列(spctab.sql)

create table          STATS$SQL_SUMMARY
(snap_id              number(6)        not null
,dbid                 number           not null
,instance_number      number           not null
,text_subset          varchar2(31)     not null
,sql_text             varchar2(1000)
,sharable_mem         number
,sorts                number
,module               varchar2(64)
,loaded_versions      number
,fetches              number
,executions           number
,loads                number
,invalidations        number
,parse_calls          number
,disk_reads           number
,buffer_gets          number
,rows_processed       number
,command_type         number
,address              raw(8)
,hash_value           number
,version_count        number
,cpu_time             number
,elapsed_time         number
,outline_sid          number
,outline_category     varchar2(64)
,child_latch          number
--注意下面5列构成主键
,constraint STATS$SQL_SUMMARY_PK primary key
    (snap_id, dbid, instance_number, hash_value, text_subset)
 using index tablespace &&tablespace_name
   storage (initial 1m next 1m pctincrease 0)
,constraint STATS$SQL_SUMMARY_FK foreign key (snap_id, dbid, instance_number)
                references STATS$SNAPSHOT on delete cascade
)tablespace &&tablespace_name
storage (initial 1m next 1m pctincrease 0) pctfree 5 pctused 40;

2)找到该表插入数据(spcpkg.sql)

insert into stats$sql_summary
            ( snap_id
            , dbid
            , instance_number
            , text_subset
            , sharable_mem
            , sorts
            , module
            , loaded_versions
            , fetches
            , executions
            , loads
            , invalidations
            , parse_calls
            , disk_reads
            , buffer_gets
            , rows_processed
            , command_type
            , address
            , hash_value
            , version_count
            , cpu_time
            , elapsed_time
            , outline_sid
            , outline_category
            , child_latch
            )
       select l_snap_id
            , p_dbid
            , p_instance_number
            , substrb(sql_text,1,31)
            , sharable_mem
            , sorts
            , module
            , loaded_versions
            , fetches
            , executions
            , loads
            , invalidations
            , parse_calls
            , disk_reads
            , buffer_gets
            , rows_processed
            , command_type
            , address
            , hash_value
            , version_count
            , cpu_time
            , elapsed_time
            , outline_sid
            , outline_category
            , child_latch
         from stats$v$sqlxs
        where is_obsolete = 'N'
          and (   buffer_gets   > l_buffer_gets_th
               or disk_reads    > l_disk_reads_th
               or parse_calls   > l_parse_calls_th
               or executions    > l_executions_th
               or sharable_mem  > l_sharable_mem_th
               or version_count > l_version_count_th
              );

3)找出stats$v$sqlxs对象(spcusr.sql)

create or replace view STATS$V_$SQLXS as
select max(sql_text)        sql_text
     , sum(sharable_mem)    sharable_mem
     , sum(sorts)           sorts
     , min(module)          module
     , sum(loaded_versions) loaded_versions
     , sum(fetches)         fetches
     , sum(executions)      executions
     , sum(loads)           loads
     , sum(invalidations)   invalidations
     , sum(parse_calls)     parse_calls
     , sum(disk_reads)      disk_reads
     , sum(buffer_gets)     buffer_gets
     , sum(rows_processed)  rows_processed
     , max(command_type)    command_type
     , address              address
     , hash_value           hash_value
     , count(1)             version_count
     , sum(cpu_time)        cpu_time
     , sum(elapsed_time)    elapsed_time
     , max(outline_sid)     outline_sid
     , max(outline_category) outline_category
     , max(is_obsolete)     is_obsolete
     , max(child_latch)     child_latch
  from v$sql
 group by hash_value, address;
create or replace public synonym STATS$V$SQLXS for STATS$V_$SQLXS;

4)通过这里可以看出,要是的STATS$SQL_SUMMARY主键不重复,只要是的STATS$V_$SQLXS查询出来的记录唯一,所以解决方案就是在STATS$V_$SQLXS视图中增加下列条件,确保查询出来的记录唯一,从而不会发生主键冲突

where
( plan_hash_value > 0
or executions > 0
or parse_calls > 0
or disk_reads > 0
or buffer_gets > 0
)

该bug在10g中修复,对于不能及时升级的数据库,建议采用第五种方法解决问题,比较治标治本,对业务基本上无影响

DB2数据库激活

db2当前激活状态

[db2inst1@xifenfei ~]$ db2licm -l
Product name:                     "DB2 Enterprise Server Edition"
License type:                     "Trial"
Expiry date:                      "06/22/2012"
Product identifier:               "db2ese"
Version information:              "9.5"
Product name:                     "DB2 Connect Server"
License type:                     "Trial"
Expiry date:                      "06/22/2012"
Product identifier:               "db2consv"
Version information:              "9.5"

查看license文件

[db2inst1@xifenfei ~]$ db2licm -l db2ese_cV9.5CPU.lic
Product name:                     "DB2 Enterprise Server Edition"
License type:                     "CPU Option"
Expiry date:                      "Permanent"
Product identifier:               "db2ese"
Version information:              "9.5"
Features:
DB2 Database Partitioning:        "Not licensed"
DB2 Performance Optimization ESE: "Not licensed"
DB2 Storage Optimization:         "Not licensed"
DB2 Advanced Access Control:      "Not licensed"
DB2 Geodetic Data Management:     "Not licensed"
DB2 pureXML ESE:                  "Not licensed"
IBM Homogeneous Federation ESE:   "Not licensed"
IBM Homogeneous Replication ESE:  "Not licensed"
Product name:                     "DB2 Connect Server"
License type:                     "Trial"
Expiry date:                      "06/22/2012"
Product identifier:               "db2consv"
Version information:              "9.5"

导入license文件

[db2inst1@xifenfei ~]$ db2licm -a db2ese_cV9.5CPU.lic
LIC1402I  License added successfully.
LIC1426I   This product is now licensed for use as outlined in your License Agreement.
USE OF THE PRODUCT CONSTITUTES ACCEPTANCE OF THE TERMS OF THE IBM LICENSE AGREEMENT,
LOCATED IN THE FOLLOWING DIRECTORY: "/opt/db2/V9.5/license/en_US.iso88591"

再次查看db2激活状态

[db2inst1@xifenfei ~]$ db2licm -l
Product name:                     "DB2 Enterprise Server Edition"
License type:                     "CPU Option"
Expiry date:                      "Permanent"
Product identifier:               "db2ese"
Version information:              "9.5"
Features:
DB2 Database Partitioning:        "Not licensed"
DB2 Performance Optimization ESE: "Not licensed"
DB2 Storage Optimization:         "Not licensed"
DB2 Advanced Access Control:      "Not licensed"
DB2 Geodetic Data Management:     "Not licensed"
DB2 pureXML ESE:                  "Not licensed"
IBM Homogeneous Federation ESE:   "Not licensed"
IBM Homogeneous Replication ESE:  "Not licensed"
Product name:                     "DB2 Connect Server"
License type:                     "Trial"
Expiry date:                      "06/22/2012"
Product identifier:               "db2consv"
Version information:              "9.5"

各种数据库的激活方式不同(只讨论激活,不涉及授权问题):
1.oracle/mysql不需要激活,可以直接使用
2.sql server 需要序列号激活
3.db2需要license文件激活

linux中不能ping通hostname可能存在问题

不能ping通hostname

[oracle@xifenfei ~]$ hostname
xifenfei
[oracle@xifenfei ~]$ ping xifenfei
ping: unknown host xifenfei
[oracle@xifenfei ~]$ nslookup
> xifenfei
;; connection timed out; no servers could be reached
> exit
[oracle@xifenfei ~]$ more /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1    xifenfei1 localhost.localdomain localhost

10G中存在问题

[oracle@xifenfei ~]$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.1.0 - Production on Tue Apr 3 01:54:22 2012
Copyright (c) 1982, 2005, Oracle.  All rights reserved.
Connected to an idle instance.
--sqlplus连接非常慢
SQL> startup
ORA-00600: internal error code, arguments: [keltnfy-ldmInit], [46], [1], [], [], [], [], []
--ORA-00600[keltnfy-ldmInit]错误

11G中存在问题

[oracle@xifenfei ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Tue Apr 3 02:02:29 2012
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
Connected.
SQL> startup
ORA-00119: invalid specification for system parameter LOCAL_LISTENER
ORA-00130: invalid listener address '(ADDRESS=(PROTOCOL=TCP)(HOST=xifenfei)(PORT=1521))'
--因为不能ping通xifenfei,提示LOCAL_LISTENER错误
--修改pfile文件
local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=1521))'
--启动数据库
SQL> startup pfile='/tmp/pfile'
ORACLE instance started.
Total System Global Area  368263168 bytes
Fixed Size                  1345016 bytes
Variable Size             306186760 bytes
Database Buffers           54525952 bytes
Redo Buffers                6205440 bytes
Database mounted.
Database opened.
[oracle@xifenfei ~]$ ps -ef|grep ora_
oracle    5960     1  0 02:09 ?        00:00:00 ora_pmon_ora11g
oracle    5964     1  0 02:09 ?        00:00:00 ora_psp0_ora11g
oracle    5970     1  1 02:09 ?        00:00:00 ora_vktm_ora11g
oracle    5976     1  0 02:09 ?        00:00:00 ora_gen0_ora11g
oracle    5980     1  0 02:09 ?        00:00:00 ora_diag_ora11g
oracle    5984     1  0 02:09 ?        00:00:00 ora_dbrm_ora11g
oracle    5988     1  0 02:09 ?        00:00:00 ora_dia0_ora11g
oracle    5992     1  0 02:09 ?        00:00:00 ora_mman_ora11g
oracle    5996     1  0 02:09 ?        00:00:00 ora_dbw0_ora11g
oracle    6000     1  0 02:09 ?        00:00:00 ora_lgwr_ora11g
oracle    6004     1  0 02:09 ?        00:00:00 ora_ckpt_ora11g
oracle    6008     1  0 02:09 ?        00:00:00 ora_smon_ora11g
oracle    6012     1  0 02:09 ?        00:00:00 ora_reco_ora11g
oracle    6016     1  0 02:09 ?        00:00:00 ora_mmon_ora11g
oracle    6020     1  0 02:09 ?        00:00:00 ora_mmnl_ora11g
oracle    6028     1  0 02:09 ?        00:00:00 ora_s000_ora11g
oracle    6055     1  0 02:10 ?        00:00:00 ora_p000_ora11g
oracle    6059     1  0 02:10 ?        00:00:00 ora_p001_ora11g
oracle    6063     1  0 02:10 ?        00:00:00 ora_arc0_ora11g
oracle    6069     1  0 02:10 ?        00:00:00 ora_arc1_ora11g
oracle    6073     1  0 02:10 ?        00:00:00 ora_arc2_ora11g
oracle    6077     1  0 02:10 ?        00:00:00 ora_arc3_ora11g
oracle    6081     1  0 02:10 ?        00:00:00 ora_qmnc_ora11g
oracle    6089     1  0 02:10 ?        00:00:00 ora_q000_ora11g
oracle    6093     1  0 02:10 ?        00:00:00 ora_q001_ora11g
oracle    6141     1  0 02:11 ?        00:00:00 ora_d000_ora11g
oracle    6145     1  2 02:11 ?        00:00:00 ora_cjq0_ora11g
--数据库启动正常
[oracle@xifenfei ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Tue Apr 3 02:10:37 2012
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
Connected.
SQL> show parameter local_listener;
ORA-01012: not logged on
Process ID: 0
Session ID: 0 Serial number: 0
--sqlplus不能操作,而且sqlplus登录非常慢

监听异常

--监听配置文件
LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.10)(PORT = 1522))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1522))
    )
  )
SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
     (GLOBAL_DBNAME = ora11g)
     (ORACLE_HOME = /u01/oracle/oracle/product/11.2.0/db_1)
     (SID_NAME = ora11g)
    )
  )
--启动监听
[oracle@xifenfei ~]$ lsnrctl start
LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 03-APR-2012 02:19:52
Copyright (c) 1991, 2011, Oracle.  All rights reserved.
Starting /u01/oracle/oracle/product/11.2.0/db_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 11.2.0.3.0 - Production
System parameter file is /u01/oracle/oracle/product/11.2.0/db_1/network/admin/listener.ora
Log messages written to /u01/oracle/diag/tnslsnr/xifenfei/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.10)(PORT=1522)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1522)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.10)(PORT=1522)))
--一直处于等待状态
--客户端登录
C:\Users\XIFENFEI>sqlplus sys/xifenfei@192.168.1.10:1522/ora11g
SQL*Plus: Release 11.2.0.3.0 Production on 星期二 4月 3 12:48:15 2012
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
--一直hang住
[root@xifenfei admin]# netstat -an|grep 1522
tcp        0      0 192.168.1.10:1522           0.0.0.0:*                   LISTEN
tcp      260      0 192.168.1.10:1522           192.168.1.1:51977           ESTABLISHED
tcp        0      0 192.168.1.10:24317          192.168.1.10:1522           ESTABLISHED
tcp      198      0 192.168.1.10:1522           192.168.1.10:24317          ESTABLISHED
--这里显示已经连接

建议:在修改主机名时要慎重,修改的时候要确保/etc/hosts和/etc/sysconfig/network文件都被正确修改

使用wrap加密pl/sql代码

1.测试脚本

[oracle@bas ~]$ more pkg_wrap_xff.sql
create or replace package pkg_wrap_xff is
  procedure welcome_in(abc in varchar2);
END pkg_wrap_xff;
/
create or replace package body pkg_wrap_xff is
  procedure welcome_in(abc in varchar2) as
  begin
    dbms_output.put_line('welcome ' || abc);
  end;
END pkg_wrap_xff;
/

2.创建并测试包

SQL> @pkg_wrap_xff
Package created.
Package body created.
SQL> set serveroutput on
SQL> exec pkg_wrap_xff.welcome_in('xifenfei');
welcome xifenfei
PL/SQL procedure successfully completed.

3.查看包内容

SQL> SELECT TEXT FROM dba_source a WHERE a.name='PKG_WRAP_XFF';
TEXT
--------------------------------------------------------------
package pkg_wrap_xff is
  procedure welcome_in(abc in varchar2);
END pkg_wrap_xff;
package body pkg_wrap_xff is
  procedure welcome_in(abc in varchar2) as
  begin
    dbms_output.put_line('welcome ' || abc);
  end;
END pkg_wrap_xff;
9 rows selected.

4.使用wrap加密

[oracle@bas ~]$ wrap iname=pkg_wrap_xff.sql oname=pkg_wrap_xff.plb
PL/SQL Wrapper: Release 10.2.0.1.0- 64bit Production on Tue Apr 03 11:16:52 2012
Copyright (c) 1993, 2004, Oracle.  All rights reserved.
Processing pkg_wrap_xff.sql to pkg_wrap_xff.plb
[oracle@bas ~]$ ll pkg_wrap_xff*
-rw-r--r--  1 oracle oinstall 634 Apr  3 11:16 pkg_wrap_xff.plb
-rw-r--r--  1 oracle oinstall 273 Apr  3 10:58 pkg_wrap_xff.sql
[oracle@bas ~]$ more pkg_wrap_xff.plb
create or replace package pkg_wrap_xff wrapped
a000000
1
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
9
53 85
W36vGRTVGRHdbwYRR6PgEKn/uJgwg1zZf9OpynQ2Z/aHUmNhYcN/NpFphdvMis61lthVP41T
adMRoYz9KTALorx2DjxUFXms0VvEXmDignlfcQjICNxh0Rmhsp2KsCjohpTO
/
create or replace package body pkg_wrap_xff wrapped
a000000
1
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
b
96 ce
QzGnt9RcmxQyUfes+xtqkeZypV0wg1zwqNPWfI6iCq2Ve93Dd2BzFE8hHRRYZ/LM86Rx2o0E
y67F1TM7QgP9WoGRBCUelGt4QvSOmwMecLCe57PVPD5lbxyJLK26scjDS8soGPzcCDysP+WR
C0zeZ9lSlyLXqNex8XpUxi7tILux/gNr1FIOWaBRhYqgTZ754pVDNlG4SXE=
/

4.测试加密包

SQL> drop package PKG_WRAP_XFF;
Package dropped.
SQL> @pkg_wrap_xff.plb
Package created.
Package body created.
SQL> set serveroutput on
PL/SQL procedure successfully completed.
SQL> exec pkg_wrap_xff.welcome_in('www.xifenfei.com');
welcome www.xifenfei.com
PL/SQL procedure successfully completed.

查看加密包内容

SQL> set pagesize 1000
SQL> SELECT TEXT FROM dba_source a WHERE a.name='PKG_WRAP_XFF';
TEXT
--------------------------------------------------------------------------------
package pkg_wrap_xff wrapped
a000000
1
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
9
53 85
W36vGRTVGRHdbwYRR6PgEKn/uJgwg1zZf9OpynQ2Z/aHUmNhYcN/NpFphdvMis61lthVP41T
adMRoYz9KTALorx2DjxUFXms0VvEXmDignlfcQjICNxh0Rmhsp2KsCjohpTO
package body pkg_wrap_xff wrapped
a000000
1
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
b
96 ce
QzGnt9RcmxQyUfes+xtqkeZypV0wg1zwqNPWfI6iCq2Ve93Dd2BzFE8hHRRYZ/LM86Rx2o0E
y67F1TM7QgP9WoGRBCUelGt4QvSOmwMecLCe57PVPD5lbxyJLK26scjDS8soGPzcCDysP+WR
C0zeZ9lSlyLXqNex8XpUxi7tILux/gNr1FIOWaBRhYqgTZ754pVDNlG4SXE=

原则上来说wrap加密是不可逆的过程,所以可以通过使用wrap加密,实现了屏蔽代码的作用,确保了自己的知识产权。其实oracle本身很多的系统包也是通过这种方法进行加密处理,以保护oracle的产权。

网关不通致使vip/lsnr资源异常

crs_stat显示节点1的listener和vip时断时续(一会online,一会offline)

rac1-> crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.devdb.db   application    ONLINE    ONLINE    rac1
ora....b1.inst application    ONLINE    ONLINE    rac1
ora....b2.inst application    ONLINE    ONLINE    rac2
ora....SM1.asm application    ONLINE    ONLINE    rac1
ora....C1.lsnr application    ONLINE    OFFLINE
ora.rac1.gsd   application    ONLINE    ONLINE    rac1
ora.rac1.ons   application    ONLINE    ONLINE    rac1
ora.rac1.vip   application    ONLINE    ONLINE    rac2
ora....SM2.asm application    ONLINE    ONLINE    rac2
ora....C2.lsnr application    ONLINE    OFFLINE
ora.rac2.gsd   application    ONLINE    ONLINE    rac2
ora.rac2.ons   application    ONLINE    ONLINE    rac2
ora.rac2.vip   application    ONLINE    ONLINE    rac1
rac1-> crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.devdb.db   application    ONLINE    ONLINE    rac1
ora....b1.inst application    ONLINE    ONLINE    rac1
ora....b2.inst application    ONLINE    ONLINE    rac2
ora....SM1.asm application    ONLINE    ONLINE    rac1
ora....C1.lsnr application    ONLINE    OFFLINE
ora.rac1.gsd   application    ONLINE    ONLINE    rac1
ora.rac1.ons   application    ONLINE    ONLINE    rac1
ora.rac1.vip   application    ONLINE    ONLINE    rac2
ora....SM2.asm application    ONLINE    ONLINE    rac2
ora....C2.lsnr application    ONLINE    ONLINE    rac2
ora.rac2.gsd   application    ONLINE    ONLINE    rac2
ora.rac2.ons   application    ONLINE    ONLINE    rac2
ora.rac2.vip   application    ONLINE    ONLINE    rac2

查看crsd.log日志

0Attempting to start `ora.rac1.vip` on member `rac2`
0Start of `ora.rac1.vip` on member `rac2` failed.
0startRunnable: setting CLI values
0Attempting to start `ora.rac1.vip` on member `rac1`
0Start of `ora.rac1.vip` on member `rac1` succeeded.
0startRunnable: setting CLI values
0Attempting to start `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
0Start of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded.
u_freem: mem passed is null
0CheckResource error for ora.rac1.vip error code = 1
0In stateChanged, ora.rac1.vip target is ONLINE
0ora.rac1.vip on rac1 went OFFLINE unexpectedly
0StopResource: setting CLI values
0Attempting to stop `ora.rac1.vip` on member `rac1`
0Stop of `ora.rac1.vip` on member `rac1` succeeded.
0ora.rac1.vip RESTART_COUNT=0 RESTART_ATTEMPTS=0
0ora.rac1.vip failed on rac1 relocating.
0StopResource: setting CLI values
0Attempting to stop `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
0Stop of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded.
0Attempting to start `ora.rac1.vip` on member `rac2`
0Start of `ora.rac1.vip` on member `rac2` failed.
0Attempting to start `ora.rac1.vip` on member `rac2`
0Start of `ora.rac1.vip` on member `rac2` succeeded.
0CRS-1002: Resource 'ora.rac1.vip' is already running on member 'rac2'

这里可以看出由于vip资源失败,致使lsnr资源也出现失败,紧接着又是启动vip,再启动lsnr。所以使得我们通过crs_stat -t观察资源情况时,看到这两个进程一直处于波动状态

分析ora.rac1.vip.log日志

[ora.rac1.vip]: clsrcexecut:env ORACLE_CONFIG_HOME=/u01/app/oracle/product/10.2.0/crs_1
[ora.rac1.vip]: clsrcexecut:cmd=/u01/app/oracle/product/10.2.0/crs_1/bin/racgeut -e
_USR_ORA_DEBUG=0 54 /u01/app/oracle/product/10.2.0/crs_1/bin/racgvip check rac1
[ora.rac1.vip]: clsrcexecut: rc = 1, time = 6.430s
[ora.rac1.vip]: end for resource = ora.rac1.vip, action=check,status=1,time=6.450s
[ora.rac1.vip]: ping to 192.168.1.1 via eth0 failed, rc = 1 (host=rac1)
ping to 192.168.1.1 via eth0 failed, rc = 1 (host=rac1)
[ora.rac1.vip]: clsrcstartorp: Error with malloc
[ora.rac1.vip]: ping to 192.168.1.1 via eth0 failed, rc = 1 (host=rac1)
ping to 192.168.1.1 via eth0 failed, rc = 1 (host=rac1)
Interface eth0 checked failed (host=rac1)
Invalid parameters, or failed to bring up VIP (host=rac1)

通过这里发现:从eth0网卡ping192.168.1.1(网关)不通,导致VIP资源不能正常工作

核实问题原因/解决
我们人工从节点1上ping 网关(192.168.1.1),果真不通.继续检查发现,网关服务器上意外的开启了防火墙,对部分进来的包进行了过滤,恰好节点1在被禁止之列,使得节点1 ping 网关不成功,从而出现该了该错误.关闭防火墙或者重新设置规则后,rac工作正常,未出现vip和lsnr资源出现波动情况.

OCR/Vote disk 维护操作

数据库版本

SQL>  select * from v$version;
BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - Prod
PL/SQL Release 10.2.0.5.0 - Production
CORE    10.2.0.5.0      Production
TNS for Linux: Version 10.2.0.5.0 - Production
NLSRTL Version 10.2.0.5.0 - Production

ocr测试(可以online处理)

rac2-> ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     160396
         Used space (kbytes)      :       4376
         Available space (kbytes) :     156020
         ID                       : 1302494786
         Device/File Name         : /dev/raw/raw11
                                    Device/File integrity check succeeded
                                    Device/File not configured
         Cluster registry integrity check succeeded
rac2-> more /etc/oracle/ocr.loc
ocrconfig_loc=/dev/raw/raw11
local_only=false
--增加ocr镜像
[root@rac2 bin]# ./ocrconfig -replace ocrmirror /dev/raw/raw12
rac2-> ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     160396
         Used space (kbytes)      :       4376
         Available space (kbytes) :     156020
         ID                       : 1302494786
         Device/File Name         : /dev/raw/raw11
                                    Device/File integrity check succeeded
         Device/File Name         : /dev/raw/raw12
                                    Device/File integrity check succeeded
         Cluster registry integrity check succeeded
rac2-> more /etc/oracle/ocr.loc
#Device/file  getting replaced by device /dev/raw/raw12
ocrconfig_loc=/dev/raw/raw11
ocrmirrorconfig_loc=/dev/raw/raw12
local_only=false
--删除ocr
[root@rac2 bin]# ./ocrconfig -replace ocr
rac2-> more /etc/oracle/ocr.loc
#Device/file /dev/raw/raw11 being deleted
ocrconfig_loc=/dev/raw/raw12
local_only=false
rac2-> ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     160396
         Used space (kbytes)      :       4376
         Available space (kbytes) :     156020
         ID                       : 1302494786
         Device/File Name         : /dev/raw/raw12
                                    Device/File integrity check succeeded
                                    Device/File not configured
         Cluster registry integrity check succeeded
--补充删除ocr镜像
[root@rac2 bin]# ./ocrconfig -replace ocrmirror

Vote disk测试(10g offline/11g online)

--关闭crs
[root@rac2 bin]# ./crsctl stop crs
[root@rac1 bin]# ./crsctl stop crs
--查询vote disk
rac2-> crsctl query css votedisk
 0.     0    /dev/raw/raw31
--增加vote disk
[root@rac2 bin]# ./crsctl add css votedisk /dev/raw/raw23 -force
Now formatting voting disk: /dev/raw/raw23
successful addition of votedisk /dev/raw/raw23.
[root@rac2 bin]# ./crsctl add css votedisk /dev/raw/raw33 -force
Now formatting voting disk: /dev/raw/raw33
successful addition of votedisk /dev/raw/raw33.
[root@rac2 bin]# ./crsctl add css votedisk /dev/raw/raw32 -force
Now formatting voting disk: /dev/raw/raw32
successful addition of votedisk /dev/raw/raw32.
rac2-> crsctl query css votedisk
 0.     0    /dev/raw/raw31
 1.     0    /dev/raw/raw23
 2.     0    /dev/raw/raw33
 3.     0    /dev/raw/raw32
located 4 votedisk(s).
--删除vote disk
[root@rac2 bin]# ./crsctl delete css votedisk /dev/raw/raw33 -force
successful deletion of votedisk /dev/raw/raw33.
--启动crs
[root@rac2 bin]# ./crsctl start crs
[root@rac1 bin]# ./crsctl start crs

补充官方操作说明[ID 428681.1]
http://www.xifenfei.com/wp-content/uploads/2012/04/OCR_Vote_disk_Maintenance_Operations.pdf

RAC 10g升级到10.2.0.5

1.Back Up database
一般情况下rman备份

2.备份ocr和vote disk

[root@rac2 bin]# ./ocrconfig -export /tmp/ocr_export.bak
[root@rac2 bin]# more /etc/oracle/ocr.loc
ocrconfig_loc=/dev/raw/raw11
local_only=FALSE
[root@rac2 bin]# dd if=/dev/raw/raw11 of=/tmp/ocr_dd.bak
[root@rac2 bin]# dd if=/dev/raw/raw31 of=/tmp/vote_dd.bak

3.Update Oracle Time Zone Definitions
Actions for the DSTv4 update in the 10.2.0.5 patchset [ID 1086400.1]

4.Stopping All Processes
滚动升级关闭一个节点所有进程,非滚动升级关闭所有进程

$ isqlplusctl stop
$ emctl stop dbconsole
$ srvctl stop service -d db_name [-s service_name_list [-i inst_name]]
$ srvctl stop instance -d db_name -i inst_name
$ srvctl stop asm -n node
$ srvctl stop listener -n node [-l listenername]
$ srvctl stop nodeapps -n node
# CRS_home/bin/crsctl stop crs(root执行,滚动升级不需要关闭)

5.Back Up the System
$ORACLE_BASE中文件,主要包括(db和crs安装文件/oraInventory文件)

6.升级crs软件
执行./runInstaller选择crs目录

执行下面命令
# CRS_home/bin/crsctl stop crs
# CRS_home/install/root102.sh

7.升级db软件
关闭crs和db所有进程(步骤同4)
执行./runInstaller选择db目录

执行下面命令
# ORACLE_HOME/root.sh

8.升级数据库
8.1)检查数据库升级需要满足条件,对存在不合适之处,进行修正
How to Download and Run Oracle’s Database Pre-Upgrade Utility [ID 884522.1]

SQL> STARTUP UPGRADE
SQL> SPOOL upgrade_info.log
SQL> @/rdbms/admin/utlu102i.sql
SQL> SPOOL OFF
SQL> ALTER SYSTEM SET CLUSTER_DATABASE=FALSE SCOPE=spfile;
--其他根据upgrade_info.log中提示修改
SQL> SHUTDOWN IMMEDIATE
SQL> STARTUP UPGRADE

8.2)启动监听
srvctl start listener -n node

8.3)升级数据库

SQL> SPOOL patch.log
SQL> @?/rdbms/admin/catupgrd.sql
--检查patch.log,发现有错误查找原因,重新执行catupgrd.sql脚本
SQL> SPOOL OFF
SQL> SHUTDOWN IMMEDIATE
SQL> STARTUP
SQL> @?/rdbms/admin/utlrp.sql
SQL> ALTER SYSTEM SET CLUSTER_DATABASE=TRUE SCOPE=spfile;
--包括其他修改调整参数
SQL> SHUTDOWN IMMEDIATE
--使用rac管理相关命令,启动需要启动资源

9.修改相关目录权限
# ORACLE_HOME/install/changePerm.sh

具体操作步骤请阅读README.html

永久表空间出现临时段不能扩展原因探讨

数据库版本

SQL> select * from v$version;
BANNER
-----------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production
PL/SQL Release 11.2.0.3.0 - Production
CORE    11.2.0.3.0      Production
TNS for Linux: Version 11.2.0.3.0 - Production
NLSRTL Version 11.2.0.3.0 - Production

创建5M测试表空间

SQL> CREATE TABLESPACE T_1652 DATAFILE '/tmp/t_1652_01.dbf' size 5M
  2  AUTOEXTEND OFF LOGGING PERMANENT EXTENT MANAGEMENT LOCAL AUTOALLOCATE
  3  SEGMENT SPACE MANAGEMENT AUTO blocksize 8192;
Tablespace created.

测试CTAS

SQL> create table CHF.T_XIFENFEI TABLESPACE T_1652 as
  2  select LPAD('XIFENFEI',1024,'F') "C_XFF" from  dual connect by level <=3500;
create table CHF.T_XIFENFEI TABLESPACE T_1652 as
*
ERROR at line 1:
ORA-01652: unable to extend temp segment by 128 in tablespace T_1652
SQL> create table CHF.T_XIFENFEI TABLESPACE T_1652 as
  2  select LPAD('XIFENFEI',1024,'F') "C_XFF" from  dual connect by level <=3000;
Table created.

测试CREATE INDEX

SQL> create index chf.i_xifenfei on chf.t_xifenfei(c_xff)
  2  tablespace t_1652;
create index chf.i_xifenfei on chf.t_xifenfei(c_xff)
                                   *
ERROR at line 1:
ORA-01658: unable to create INITIAL extent for segment in tablespace T_1652
SQL> Select MAX(d.bytes) total_bytes,
  2         nvl(SUM(f.Bytes), 0) free_bytes,
  3         d.file_name,
  4         MAX(d.bytes) - nvl(SUM(f.bytes), 0) used_bytes,
  5  from   DBA_FREE_SPACE f , DBA_DATA_FILES d
  6  where  f.tablespace_name(+) = d.tablespace_name
  7  and    f.file_id(+) = d.file_id
  8  and    d.tablespace_name = 'T_1652'
  9  group by d.file_name;
TOTAL_BYTES FREE_BYTES FILE_NAME                 USED_BYTES
----------- ---------- ------------------------- ----------
    5242880          0 /tmp/t_1652_01.dbf           5242880
SQL> drop table chf.t_xifenfei purge;
Table dropped.
SQL> create table CHF.T_XIFENFEI TABLESPACE T_1652 as
  2  select LPAD('XIFENFEI',1024,'F') "C_XFF" from  dual connect by level <=2000;
Table created.
SQL> Select MAX(d.bytes) total_bytes,
  2         nvl(SUM(f.Bytes), 0) free_bytes,
  3         d.file_name,
  4         MAX(d.bytes) - nvl(SUM(f.bytes), 0) used_bytes,
  5  from   DBA_FREE_SPACE f , DBA_DATA_FILES d
  6  where  f.tablespace_name(+) = d.tablespace_name
  7  and    f.file_id(+) = d.file_id
  8  and    d.tablespace_name = 'T_1652'
  9  group by d.file_name;
TOTAL_BYTES FREE_BYTES FILE_NAME                 USED_BYTES
----------- ---------- ------------------------- ----------
    5242880    1048576 /tmp/t_1652_01.dbf           4194304
SQL> create index chf.i_xifenfei on chf.t_xifenfei(c_xff)
  2  tablespace t_1652;
create index chf.i_xifenfei on chf.t_xifenfei(c_xff)
                                   *
ERROR at line 1:
ORA-01652: unable to extend temp segment by 128 in tablespace T_1652
SQL> ALTER DATABASE DATAFILE '/tmp/t_1652_01.dbf' RESIZE 10M;
Database altered.
SQL> create index chf.i_xifenfei on chf.t_xifenfei(c_xff)
  2  tablespace t_1652;
Index created.

测试MOVE

SQL> drop table chf.t_xifenfei purge;
Table dropped.
SQL> create table CHF.T_XIFENFEI TABLESPACE T_1652 as
  2  select LPAD('XIFENFEI',1024,'F') "C_XFF" from  dual connect by level <=3500;
Table created.
SQL> alter table chf.t_xifenfei move;
alter table chf.t_xifenfei move
*
ERROR at line 1:
ORA-01652: unable to extend temp segment by 128 in tablespace T_1652
SQL> Select MAX(d.bytes) total_bytes,
  2         nvl(SUM(f.Bytes), 0) free_bytes,
  3         d.file_name,
  4         MAX(d.bytes) - nvl(SUM(f.bytes), 0) used_bytes,
  5  from   DBA_FREE_SPACE f , DBA_DATA_FILES d
  6  where  f.tablespace_name(+) = d.tablespace_name
  7  and    f.file_id(+) = d.file_id
  8  and    d.tablespace_name = 'T_1652'
  9  group by d.file_name;
TOTAL_BYTES FREE_BYTES FILE_NAME                 USED_BYTES
----------- ---------- ------------------------- ----------
   10485760    4194304 /tmp/t_1652_01.dbf           6291456
SQL>  ALTER DATABASE DATAFILE '/tmp/t_1652_01.dbf' RESIZE 15M;
Database altered.
SQL> alter table chf.t_xifenfei move;
Table altered.

这里可以发现CTAS,CREATE INDEX,MOVE操作都有个共同点:需要一次性创建一个较大SEGMENT,但是这个SEGMENT的创建过程是在数据库中逐渐实现(非初始化指定大小)。
也就是说,ORACLE对这些对象的处理方法是:对于这样的segment先当作临时段处理,当处理完成后,再把这些在永久表空间中的临时段转换为永久段;所以当这些永久表空间中的临时段在扩展的时候,遇到该永久表空间不足,而该段目前还是临时段(在永久表空间中的临时段),就出现了ORA-01652提示一个永久表空间unable to extend temp segment

表空间online出现ORA-00600[kcbz_check_objd_typ]处理过程

online表空间出现ORA-00600[kcbz_check_objd_typ]

Fri Mar 30 14:09:24 2012
alter tablespace xff offline
Fri Mar 30 14:09:28 2012
Completed: alter tablespace xff offline
Fri Mar 30 17:49:59 2012
alter tablespace xff rename datafile '/oradataa/xifenfei.dbf' to '/oradatab/xifenfei_bak.dbf'
Fri Mar 30 17:50:03 2012
Completed: alter tablespace xff rename datafile '/oradataa/xifenfei.dbf' to '/oradatab/xifenfei_bak.dbf'
Fri Mar 30 17:50:03 2012
alter tablespace coweb_bak_new online
Fri Mar 30 17:53:08 2012
Errors in file /oracle/ora10/admin/ora10g/udump/ora10g_ora_21275.trc:
ORA-00600: internal error code, arguments: [kcbz_check_objd_typ], [0], [0], [1], [], [], [], []

分析trace文件

Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /oracle/ora10/product
System name:	Linux
Node name:	FH-DB-01
Release:	2.6.18-92.el5
Version:	#1 SMP Tue Apr 29 13:16:15 EDT 2008
Machine:	x86_64
Instance name: ora10g
Redo thread mounted by this instance: 1
Oracle process number: 182
Unix process pid: 21275, image: oracle@FH-DB-01 (TNS V1-V3)
*** 2012-03-30 17:50:16.469
*** ACTION NAME:() 2012-03-30 17:50:15.939
*** MODULE NAME:(sqlplus@FH-DB-01 (TNS V1-V3)) 2012-03-30 17:50:15.939
*** SERVICE NAME:(SYS$USERS) 2012-03-30 17:50:15.939
*** SESSION ID:(921.23041) 2012-03-30 17:50:15.939
*** SESSION ID:(921.23041) 2012-03-30 17:50:15.939
OBJD MISMATCH typ=6, seg.obj=-2, diskobj=294051, dsflg=0, dsobj=294044, tid=294044, cls=1
Input data (nil), 0, 0
Formatted dump of block:
buffer tsn: 10 rdba: 0x0435c094 (1024/70631572)
scn: 0x0b2d.2b4f8874 seq: 0x01 flg: 0x04 tail: 0x88740601
frmt: 0x02 chkval: 0x5626 type: 0x06=trans data
Hex dump of block: st=0, typ_found=1
Dump of memory from 0x000000006DBB2000 to 0x000000006DBB4000
06DBB2000 0000A206 0435C094 2B4F8874 04010B2D  [......5.t.O+-...]
06DBB2010 00005626 00000002 00047CA3 2B4F8874  [&V.......|..t.O+]
06DBB2020 00000B2D 00320002 0435C091 00000000  [-.....2...5.....]
06DBB2030 00000000 00000000 00000000 00000000  [................]
        Repeat 2 times
06DBB2060 00000000 02800000 00000000 00240000  [..............$.]
06DBB2070 1F3C1F60 00000000 00000000 00000000  [`.<.............]
06DBB2080 00000000 00001F60 00001F5C 00000000  [....`...\.......]
06DBB2090 00000000 00000000 00000000 00000000  [................]
        Repeat 501 times
06DBB3FF0 00000000 00000000 00000000 88740601  [..............t.]
Block header dump:  0x0435c094
 Object id on Block? Y
 seg/obj: 0x47ca3  csc: 0xb2d.2b4f8874  itc: 2  flg: E  typ: 2 - INDEX
     brn: 0  bdba: 0x435c091 ver: 0x01 opc: 0
     inc: 0  exflg: 0
 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x0000.000.00000000  0x00000000.0000.00  ----    0  fsc 0x0000.00000000
0x02   0x0000.000.00000000  0x00000000.0000.00  ----    0  fsc 0x0000.00000000

这里可以得出几个结论:
1)版本平台信息
2)通过下面信息都是知道,data_object_id=294044的对象出现异常(seg.obj=-2)

OBJD MISMATCH typ=6, seg.obj=-2, diskobj=294051, dsflg=0, dsobj=294044, tid=294044, cls=1
Block header dump:  0x0435c094
 Object id on Block? Y
 seg/obj: 0x47ca3  csc: 0xb2d.2b4f8874  itc: 2  flg: E  typ: 2 - INDEX
     brn: 0  bdba: 0x435c091 ver: 0x01 opc: 0
     inc: 0  exflg: 0

3)结合dba_objects视图查询出来是xifenfei_index 索引出现异常

问题原因猜测
因为没有找到权威解释,对于这里在online表空间的时候出现ORA-00600[kcbz_check_objd_typ]错误,个人的猜测可能是在online的时候,验证数据字典中关于该表空间中对象的相关记录和该表空间总的存储数据比较,然后发现不匹配,所以出现了该错误。

处理方法
虽然该表空间离线,但是我们可以使用drop操作直接删除数据字典中index的记录,然后再online表空间,这样可以绕过数据字典和表空间中的存储比较。当表空间online成功后,然后再创建index

ASMM表空间强制终止DML操作导致ORA-600 [ktspfupdst-1]

发现错误ORA-00600 [ktspfupdst-1],trace关键内容如下

Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Oracle Data Mining options
JServer Release 9.2.0.8.0 - Production
ORACLE_HOME = /oracle9/app/product/9.2.0
System name:    AIX
Node name:      zwq_crm1
Release:        3
Version:        5
Machine:        00C420B44C00
Instance name: crm1
Redo thread mounted by this instance: 1
Oracle process number: 389
Unix process pid: 1896900, image: oracle@zwq_crm1 (TNS V1-V3)
----------------------------------------------
*** 2012-03-31 02:50:48.509
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [ktspfupdst-1], [], [], [], [], [], [], []
ORA-00604: error occurred at recursive SQL level 1
ORA-01013: user requested cancel of current operation
Current SQL statement for this session:
INSERT INTO XIFENFEI
SELECT * FROM T_XFF
----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp+0148          bl       ksedst               1029746FC ?
ksfdmp+0018          bl       01FD4014
kgerinv+00e8         bl       _ptrgl
kgeasnmierr+004c     bl       kgerinv              000000001 ? 000000000 ?
                                                   000000005 ? 000000001 ?
                                                   000000001 ?
ktspfupdst+0540      bl       kgeasnmierr          110006308 ? 1103994E8 ?
                                                   102A9239C ? 000000000 ?
                                                   000000005 ? 000000010 ?
                                                   000000020 ? 000000006 ?
ktspstchg+00e4       bl       ktspfupdst           000000060 ? 300000004 ?
                                                   FFFFFFFFFFF6E48 ?
                                                   50601CE000000ED ?
                                                   3B401B34C5D02F2A ?
                                                   B92000004000020 ?
kdoiur+062c          bl       ktspstchg            000000000 ? 700000C39D779E8 ?
                                                   000000000 ?
kcoubk+00e4          bl       _ptrgl
ktundo+0988          bl       kcoubk               1010CCD80 ? FFFFFFFFFFF76C0 ?
                                                   100ED51C0 ? FFFFFFFFFFF7150 ?
                                                   1101FAF78 ? 1102567C0 ?
                                                   700000C396A1300 ? 000000002 ?
ktubko+03bc          bl       ktundo               1840DFB30 ?
                                                   3B401B3400000002 ?
                                                   000000000 ? 000000000 ?
                                                   FFFFFFFFFFF85D8 ?
                                                   700000C80A1E880 ? 2FFFF8540 ?
                                                   FFFFFFFFFFF8780 ?
ktuabt+0638          bl       ktubko               DF000000DF ?
                                                   FFFFFFFFFFF8690 ? 000000000 ?
                                                   FFFFFFFFFFF85D8 ? 102973880 ?
                                                   700000C844FA418 ?
ktcrab+02b4          bl       ktuabt               700000C80A1E840 ? 200017CD8 ?
ktcrsp+026c          bl       ktcrab               100F698E4 ? 000000001 ?
ksures+0074          bl       ktcrsp               700000C844FA448 ?
opiexe+3380          bl       01FD4138
opiall0+102c         bl       opiexe               400000000 ? 110002A48 ?
                                                   FFFFFFFFFFFA0A0 ?
kpoal8+0a78          bl       opiall0              5EFFFFBED4 ? 22103A43F8 ?
                                                   FFFFFFFFFFFA5B8 ? 000000000 ?
                                                   FFFFFFFFFFFA508 ? 1103A4B00 ?
                                                   6FF00000738 ?
                                                   24000000007FFF ?
opiodr+08cc          bl       _ptrgl
ttcpip+0cc4          bl       _ptrgl
opitsk+0d60          bl       ttcpip               11000CF90 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
opiino+0758          bl       opitsk               000000000 ? 000000000 ?
opiodr+08cc          bl       _ptrgl
opidrv+032c          bl       opiodr               3C00000018 ? 4101FAF78 ?
                                                   FFFFFFFFFFFF7B0 ? 0A000F350 ?
sou2o+0028           bl       opidrv               3C0C000000 ? 4A00E8B50 ?
                                                   FFFFFFFFFFFF7B0 ?
main+0138            bl       01FD3A28
__start+0098         bl       main                 000000000 ? 000000000 ?
--------------------- Binary Stack Dump ---------------------

这里可以得到信息如下:
1)系统平台aix 5.3,数据库版本9.2.0.8 rac
2)这个错误可能和一个insert select操作,然后取消有关系

表空间管理方式

SQL> SELECT tablespace_name, SEGMENT_SPACE_MANAGEMENT from dba_tablespaces
  2  where tablespace_name=
  3  (select tablespace_name from dba_tables where table_name='XIFENFEI');
TABLESPACE_NAME                SEGMEN
------------------------------ ------
CUSTSERV                       AUTO

查看MOS发现和[ID 388599.1]相符
错误原因:

1. An insert or update on a table causes the addition of a new extent
   and the operation is cancelled.
2. The segment uses Automatic Segment Space Management (ASSM).
3. The call stack in the associated trace file resembles:
    ktspfupdst ktspstchg kdoiur kcoubk ktundo ktubko ktuabt ktcrab ktcrsp

解决方法:
官方给出方案就是升级到新版本,不过可以采取一个比较折中的处理方案,先想办法导出或者备份该对象数据,然后trunate或者重建表和相关index操作,解决该问题