sqlplus版本不正确,导致ORA-12560

sqlplus登录数据库报ORA-12560

C:\Users\XIFENFEI>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.2.0 Production on Tue Feb 14 23:33:31 2012
Copyright (c) 1982, 2010, Oracle.  All rights reserved.
ERROR:
ORA-12560: TNS:protocol adapter error
Enter user-name:
ERROR:
ORA-12560: TNS:protocol adapter error
Enter user-name:
ERROR:
ORA-12560: TNS:protocol adapter error
SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus
C:\Users\XIFENFEI>sqlplus -v
SQL*Plus: Release 11.2.0.2.0 Production

在这里发现sqlplus的版本信息不正确,我的数据库安装的11.2.0.1,这么怎么显示sqlplus是11.2.0.2,然后进入%ORACLE_HOME%/bin目录下面执行sqlplus,登录成功

C:\Users\XIFENFEI>cd e:\oracle\11_2_0\BIN
e:\oracle\11_2_0\BIN>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on 星期二 2月 14 23:44:38 2012
Copyright (c) 1982, 2010, Oracle.  All rights reserved.
连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, Oracle Label Security, OLAP, Data Mining,
Oracle Database Vault and Real Application Testing options
SQL>

出现了这样的问题,考虑应该是环境变量设置不当导致,查看环境变量发现下面一部分

E:\oracle\timesten\bin;E:\oracle\timesten\ttoracle_home\instantclient_11_2;e:\oracle\11_2_0\bin;

问题的答案就是在这里,因为我的电脑是先安装Oracle,前几天安装了TimesTen,导致环境变量发生了改变,使得我在默认情况下使用sqlplus的时候,自动调用了TimesTen中的sqlplus导致

C:\Users\XIFENFEI>cd E:\oracle\timesten\ttoracle_home\instantclient_11_2
E:\oracle\timesten\ttoracle_home\instantclient_11_2>dir sqlplus*
 驱动器 E 中的卷没有标签。
 卷的序列号是 38D0-2A35
 E:\oracle\timesten\ttoracle_home\instantclient_11_2 的目录
2010/11/19  00:50           655,360 sqlplus.exe
2011/11/08  04:12            84,524 sqlplus.sym
2011/11/08  04:12               554 SQLPLUS_README
               3 个文件        740,438 字节
               0 个目录  9,825,832,960 可用字节

清除掉shared pool中某条sql语句方法

在论坛中看到一个帖子,如何清除掉shared pool中某条sql语句,如果是在10g以前的版本,那估计会比较麻烦,为了删除一条sql语句记录,需要清空整个shared pool,在10g中提供了新的dbms_shared_pool包可以实现该功能(如果该包没有安装,可以通过?/rdbms/admin/dbmspool.sql安装),下面我通过在11g中试验证明该问题
1.数据库版本信息

SQL> select * from v$version;
BANNER
----------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
PL/SQL Release 11.2.0.1.0 - Production
CORE    11.2.0.1.0      Production
TNS for 32-bit Windows: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production

2.模拟一条sql语句

SQL> create table xff as
  2  select * from dba_tables
  3  where rownum<10;
表已创建。
SQL> select count(*) from xff;
  COUNT(*)
----------
         9
SQL> col sql_text for a30
SQL> SELECT ADDRESS,HASH_VALUE,SQL_TEXT FROM V$SQLAREA
  2  WHERE SQL_TEXT LIKE 'select % xff';
ADDRESS  HASH_VALUE SQL_TEXT
-------- ---------- ------------------------------
1EFB91B8 3642190903 select count(*) from xff

3.打破神话一:权限操作
对表进行权限操作,可以清除该表在shared pool中关于该表记录

SQL> grant select on xff to chf;
授权成功。
SQL> col sql_text for a30
SQL> SELECT ADDRESS,HASH_VALUE,SQL_TEXT FROM V$SQLAREA
  2  WHERE SQL_TEXT LIKE 'select % xff';
ADDRESS  HASH_VALUE SQL_TEXT
-------- ---------- ------------------------------
1EFB91B8 3642190903 select count(*) from xff
SQL> revoke select on xff from chf;
撤销成功。
SQL> col sql_text for a30
SQL> SELECT ADDRESS,HASH_VALUE,SQL_TEXT FROM V$SQLAREA
  2  WHERE SQL_TEXT LIKE 'select % xff';
ADDRESS  HASH_VALUE SQL_TEXT
-------- ---------- ------------------------------
1EFB91B8 3642190903 select count(*) from xff

证明grant和revoke操作没有清除shared pool中关于该表的sql语句

4.打破神话二:ddl操作
对表进行ddl操作,可以清除该表在shared pool中关于该表记录

SQL> alter table xff add  owner1 varchar2(100);
表已更改。
SQL> SELECT ADDRESS,HASH_VALUE,SQL_TEXT FROM V$SQLAREA
  2  WHERE SQL_TEXT LIKE 'select % xff';
ADDRESS  HASH_VALUE SQL_TEXT
-------- ---------- ------------------------------
1EFB91B8 3642190903 select count(*) from xff

事实证明ddl操作不能达到预期效果,没有清除掉这条sql语句

5.刷新shared pool

SQL> alter system flush shared_pool
  2  ;
系统已更改。
SQL> SELECT ADDRESS,HASH_VALUE,SQL_TEXT FROM V$SQLAREA
  2  WHERE SQL_TEXT LIKE 'select % xff';
未选定行

把整个shared pool都刷新了,自然其中的一条sql语句也没有了,在10g前只能通过这种方法实现

6.使用dbms_shared_pool.purge

SQL> select count(*) from xff;
  COUNT(*)
----------
         9
SQL> SELECT ADDRESS,HASH_VALUE,SQL_TEXT FROM V$SQLAREA
  2  WHERE SQL_TEXT LIKE 'select % xff';
ADDRESS  HASH_VALUE SQL_TEXT
-------- ---------- ------------------------------
1EFB91B8 3642190903 select count(*) from xff
SQL> exec dbms_shared_pool.purge('1EFB91B8, 3642190903','C');
PL/SQL 过程已成功完成。
SQL> SELECT ADDRESS,HASH_VALUE,SQL_TEXT FROM V$SQLAREA
  2  WHERE SQL_TEXT LIKE 'select % xff';
未选定行

试验证明在shared pool中清除了一条sql记录

7.关于dbms_shared_pool.purge参数说明

purge(name varchar2, flag char DEFAULT 'P', heaps number DEFAULT 1);
--    name
--      The name of the object to keep.  There are two kinds of objects:
--      PL/SQL objects, triggers, sequences, types and Java objects,
--      which are specified by name, and
--      SQL cursor objects which are specified by a two-part number
--      (indicating a location in the shared pool).  For example:
--        dbms_shared_pool.keep('scott.hispackage')
--      will keep package HISPACKAGE, owned by SCOTT.  The names for
--      PL/SQL objects follows SQL rules for naming objects (i.e.,
--      delimited identifiers, multi-byte names, etc. are allowed).
--      A cursor can be keeped by
--        dbms_shared_pool.keep('0034CDFF, 20348871', 'C')
--      The complete hexadecimal address must be in the first 8 characters.
--      The value for this identifier is the concatenation of the
--      'address' and 'hash_value' columns from the v$sqlarea view.  This
--      is displayed by the 'sizes' call above.
--      Currently 'TABLE' and 'VIEW' objects may not be keeped.
--
--    flag
--      This is an optional parameter.  If the parameter is not specified,
--        the package assumes that the first parameter is the name of a
--        package/procedure/function and will resolve the name.  Otherwise,
--        the parameter is a character string indicating what kind of object
--        to keep the name identifies.  The string is case insensitive.
--        The possible values and the kinds of objects they indicate are
--        given in the following table:
--          Value        Kind of Object to keep
--          -----      ----------------------
--	      P          package/procedure/function
--	      Q          sequence
--	      R          trigger
--	      T          type
--            JS         java source
--            JC         java class
--	      JR         java resource
--	      JD         java shared data
--	      C          cursor
--      If and only if the first argument is a cursor address and hash-value,
--        the flag parameter should be set to 'C' (or 'c').
--
--    heaps
--      heaps to purge. e.g if heap 0 and heap 6 are to be purged.
--      1<<0 | 1<<6 => hex 0x41 => decimal 65. so specify heaps=>65.
--      Default is 1 i.e heap 0 which means the whole object will be purged.

ORA-600[4194]/[4193]解决

朋友的库启动的时候出现ORA-600[4194]/[4193]错误

Tue Feb 14 09:34:11 2012
Errors in file d:\oracle\product\10.2.0\admin\interlib\bdump\interlib_smon_2784.trc:
ORA-01595: error freeing extent (2) of rollback segment (3))
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4194], [6], [30], [], [], [], [], []
Tue Feb 14 09:35:34 2012
Errors in file d:\oracle\product\10.2.0\admin\interlib\udump\interlib_ora_2824.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4193], [2005], [2008], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [2005], [2008], [], [], [], [], []
Tue Feb 14 09:36:30 2012
DEBUG: Replaying xcb 0x1fa24174, pmd 0x1fba06d4 for failed op 8
Doing block recovery for file 2 block 177
No block recovery was needed
Tue Feb 14 09:37:30 2012
Errors in file d:\oracle\product\10.2.0\admin\interlib\bdump\interlib_pmon_2732.trc:
ORA-00600: internal error code, arguments: [4193], [2005], [2008], [], [], [], [], []
Tue Feb 14 09:37:31 2012
Errors in file d:\oracle\product\10.2.0\admin\interlib\bdump\interlib_pmon_2732.trc:
ORA-00600: internal error code, arguments: [4193], [2005], [2008], [], [], [], [], []

从这里可以看到出现了ORA-600[4194]/[4193],第一感觉就是undo出现问题。
4193:表示undo和redo不一致(Arg [a] Undo record seq number,Arg [b] Redo record seq number );
4194:表示也是undo和redo不一致(Arg [a] Maximum Undo record number in Undo block,Arg [b] Undo record number from Redo block)
至于为什么有时候会只出现其中一个,我不太清楚,求答案

直接设置了下面参数,数据库就意外的open成功,这位朋友比较幸运

undo_tablespace=SYSTEM
undo_management=MANUAL

既然库已经open,然后新建undo空间,删除出问题的undo,做如下修改,数据库恢复完成

undo_tablespace=新undo
undo_management=AUTO

如果出现极端的情况可能需要做如下处理:
1.使用_offline_rollback_segments和_corrupted_rollback_segments屏蔽掉有问题的undo segment
2.继续可能出现ora-600[2662],需要推进scn

Query Duration=0与ORA-01555

1.ALERT日志错误
奇怪之处:Query Duration=0 sec,竟然出现了ORA-01555

Tue Feb  7 02:41:34 2012
ORA-01555 caused by SQL statement below (Query Duration=0 sec, SCN: 0x0b2e.efcd78a9):
Tue Feb  7 02:41:34 2012
SELECT "ID_NO","CUST_ID" FROM "DBACCADM"."DCUSTMSG" "C" WHERE "ID_NO"=:1

2.ORA-01555解释
超过了undo_retention时间,undo被覆盖导致ORA-01555

[zwq_acc1:/home/oraeye/check]oerr ora 1555
01555, 00000, "snapshot too old: rollback segment number %s with name \"%s\" too small"
// *Cause: rollback records needed by a reader for consistent read are
//         overwritten by other writers
// *Action: If in Automatic Undo Management mode, increase undo_retention
//          setting. Otherwise, use larger rollback segments

3.数据库版本

SQL> select * from v$version;
BANNER
----------------------------------------------------------------
Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production
PL/SQL Release 9.2.0.8.0 - Production
CORE    9.2.0.8.0       Production
TNS for IBM/AIX RISC System/6000: Version 9.2.0.8.0 - Production
NLSRTL Version 9.2.0.8.0 - Production

4.undo基本信息
从这里可以发现,两个节点的undo表空间还有很多剩余,缺发生了undo被覆盖从而出现了ORA-01555

SQL> col name for a20
SQL> col value for a15
SQL> SELECT INST_ID, NAME, VALUE
  2    FROM GV$PARAMETER
  3   WHERE UPPER (Name) LIKE '%UNDO%';
   INST_ID NAME                 VALUE
---------- -------------------- ---------------
         1 undo_management      AUTO
         1 undo_tablespace      UNDOTBS1
         1 undo_suppress_errors FALSE
         1 undo_retention       1800
         2 undo_management      AUTO
         2 undo_tablespace      UNDOTBS2
         2 undo_suppress_errors FALSE
         2 undo_retention       1800
8 rows selected.
TABLESPACE_NAME                CURRENT_TOTAL(MB)   USED(MB)   FREE(MB)      FREE% AUT MAX_TOTAL(MB)
------------------------------ ----------------- ---------- ---------- ---------- --- -------------
UNDOTBS1                                   40950    1587.94 39362.0625      96.12 NO          40950
UNDOTBS2                                   57330    1926.31 55403.6875      96.64 NO          57330
SQL> SELECT DISTINCT STATUS ,
  2                  COUNT(*) "EXTENT_NUM",
  3                  SUM(BYTES) / 1024 / 1024 / 1024 "UNDO(G)"
  4    FROM DBA_UNDO_EXTENTS
  5   GROUP BY STATUS;
STATUS    EXTENT_NUM    UNDO(G)
--------- ---------- ----------
ACTIVE           208 .273658752
EXPIRED         7651 2.42865753
UNEXPIRED        941 .752548218

查询MOS[ID 761128.1],发现可能是Oracle bug导致(BUG:6799685 – ORA-1555 ERROR WITH QUERY DURATION=0 AND UNDO_RETENTION=1800和BUG:5475085 – V$UNDOSTAT.EXPBLKREUCNT IS NEVER INCREMENTED)

5.解决方法
Increase the size of the UNDO tablespace and increase the UNDO_RETENTION parameter value to try to prevent required undo expiring too quickly.
基于本库,因为undo空间还有很大剩余,直接设置UNDO_RETENTION=3600即可(可以从一定程度上缓解整个问题,但是要从根本上解决整个问题,需要升级到10.2.0.4及其以上版本)

StatSpack报告中redo size为负数

在一份statspack报告中发现redo size 为负数

DB Name         DB Id    Instance     Inst Num Release     Cluster Host
------------ ----------- ------------ -------- ----------- ------- ------------
CRM           3413823439 crm2                2 9.2.0.8.0   YES     zwq_crm2
              Snap Id     Snap Time      Sessions Curs/Sess Comment
            --------- ------------------ -------- --------- -------------------
Begin Snap:     47654 05-Feb-12 11:00:04    2,301      20.0
  End Snap:     47655 05-Feb-12 12:00:02    2,298      20.3
   Elapsed:               59.97 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
               Buffer Cache:    36,448M      Std Block Size:          8K
           Shared Pool Size:    10,240M          Log Buffer:     20,480K
Load Profile
~~~~~~~~~~~~                            Per Second       Per Transaction
                                   ---------------       ---------------
  负数=====>      Redo size:           -359,121.01             -7,828.69
              Logical reads:            349,787.58              7,625.22
              Block changes:              5,656.13                123.30
             Physical reads:             12,521.51                272.96
            Physical writes:                427.44                  9.32
                 User calls:             22,223.63                484.47
                     Parses:              4,673.27                101.88
                Hard parses:                 46.78                  1.02
                      Sorts:              4,027.70                 87.80
                     Logons:                 10.57                  0.23
                   Executes:             10,480.35                228.47
               Transactions:                 45.87

一时之间感觉很奇怪,在运行的数据库redo size不可能为负数(只要数据库在运行redo size都应该大于0).查询MOS[ID 308414.1]发现原来是一个bug引起(Bug:1713403 NEGATIVE VALUE IN V$SESSTAT FOR STATISTIC REDO SIZE),sp中的redo size其本质还是来源于V$SESSTAT.

Applies to:
Oracle Server - Enterprise Edition - Version: 8.1.7.4 to 9.2.0.8
This problem can occur on any platform.
Symptoms
Redo Size is displayed as a Negative number in a Statspack report.
For example:
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
             --------------- ---------------
Redo size:        -17,931.33      -55,277.73
Logical reads:     31,095.80       95,860.43
...
Cause
Bug:1713403 NEGATIVE VALUE IN V$SESSTAT FOR STATISTIC REDO SIZE
Overflow of 'redo size' statistic. This is fixed in Oracle10g and above.
Patches do not exist for earlier releases.
Solution
Use 'redo blocks written' instead to measure the amount of redo.
Ignore negative 'redo size'.

shell处理alert日志

每天都检查oracle日志,所以写了一个比较完善的shell,让其自动处理,在运行程序之前,需要在该脚本目录下新建tmp目录

#!/usr/bin/ksh
export LANG=en
#临时目录
tmp_dest=$HOME/check/tmp
#用户名
username=username
#密码
password=password
cd $tmp_dest
sqlplus -s $username/$password<<XFF>/dev/null
set echo on
spool sqlplus.txt
col name format a20
col value format a55
select name,value from v\$parameter where name='background_dump_dest';
spool off
exit
XFF
alert_path_num=$(grep -n "background_dump_dest" $tmp_dest/sqlplus.txt |awk -F":" '{print $1}')
alert_path=$(cat $tmp_dest/sqlplus.txt |sed -n "${alert_path_num}p" | awk -F" " '{print $2}')
first_day=`cat $tmp_dest/first_day.tmp`
d_day=$(date +%e)
if [ $d_day -lt 10 ]
then
###########注意:以下两种方式选择其一###################
#部分系统出现alert日志例如:Tue Aug  7 07:44:59 2012
 last_day=$(date +%a)' '$(date +%b)'.*'$d_day'.*'$(date +%Y)
#部分系统出现alert日志例如:Thu Jun 07 13:56:18 2012
 n_day=`echo $d_day | awk 'gsub(/^ *| *$/,"")'`
 last_day=$(date +%a)' '$(date +%b)'.*0'$n_day'.*'$(date +%Y)
else
 last_day=$(date|cut -c 1-10).*$(date +%Y)
fi
echo $last_day > $tmp_dest/first_day.tmp
first_num=$(grep -n "$first_day" $alert_path/alert_$ORACLE_SID.log |head -1|awk -F":" '{print $1}')
if [ -z "$first_num" ]
then
   first_num=1
fi
#修改$last_day为'.*',表示alert日志结尾
last_num=$(grep -n ".*" $alert_path/alert_$ORACLE_SID.log |tail -1|awk -F":" '{print $1}')
point=1
export=$point
echo "########################## checking alert_log start $first_day ########################## "
sed -n "${first_num},${last_num}p" $alert_path/alert_$ORACLE_SID.log > $tmp_dest/trunc_alert
cat $tmp_dest/trunc_alert|grep ORA-|while read line
do
   line=$(echo "$line"|sed -e 's/\[/\\[/g;s/\]/\\]/g')
   time=$(grep -n "$line" $tmp_dest/trunc_alert | awk -F':' '{print $1}'|wc -l)
if [ "$time" -ge 1 ]
then
   num=$(grep -n "$line" $tmp_dest/trunc_alert|awk -F':' '{print $1}'|tail -1)
   #echo $num
   front_num=$((num-1))
   back_num=$((num+9))
   echo "++++++++++++++++++$point++++++++++++++++++++"
    sed -n "${front_num},${back_num}p" $tmp_dest/trunc_alert
   echo "++++++++++++++++++$point-End++++++++++++++++"
   point=$((point+1))
   flag=1
else
     if [ -z "$time" ]
     then
     flag=0
     fi
fi
done
#rm $tmp_dest/trunc_alert
#rm $tmp_dest/sqlplus.txt
if [ " $flag " -eq 0 ]
   then
   echo "No errors in $first_day !"
fi
echo "########################## checking alert_log end $last_day ########################## "

检查结果

########################## checking alert_log start Fri Jan 27.*2012 ##########################
++++++++++++++++++1++++++++++++++++++++
  Current log# 3 seq# 918 mem# 0: /opt/oracle/oradata/orcl/mcrm/redo03.log
Tue Jan 31 22:00:22 2012
Errors in file /opt/oracle/admin/mcrm/bdump/mcrm_j001_23329.trc:
ORA-01114: IO error writing block to file 201 (block # 550944)
ORA-27072: File I/O error
Linux-x86_64 Error: 28: No space left on device
Additional information: 4
Additional information: 550944
Additional information: -1
ORA-01114: IO error writing block to file 201 (block # 550922)
ORA-27072: File I/O error
Additional information: 4
Additional information: 550922
++++++++++++++++++1-End++++++++++++++++
++++++++++++++++++2++++++++++++++++++++
Additional information: 550944
Additional information: -1
ORA-01114: IO error writing block to file 201 (block # 550922)
ORA-27072: File I/O error
Additional information: 4
Additional information: 550922
Additional information: 114688
Wed Feb  1 07:24:02 2012
Thread 1 advanced to log sequence 919 (LGWR switch)
  Current log# 1 seq# 919 mem# 0: /opt/oracle/oradata/orcl/mcrm/redo01.log
Wed Feb  1 07:24:02 2012
Errors in file /opt/oracle/admin/mcrm/bdump/mcrm_arc1_3624.trc:
ORA-19504: failed to create file "/opt/oracle/oradata/archivelog/1_918_741262746.dbf"
++++++++++++++++++2-End++++++++++++++++
########################## checking alert_log end Sun Feb 12.*2012##########################

Timesten安装与试用

1.安装timesten

[oracle@bas linux8664]$ ./setup.sh
NOTE: Each TimesTen installation is identified by a unique instance name.
      The instance name must be a non-null alphanumeric string, not longer
      than 255 characters.
Please choose an instance name for this installation? [ tt1122 ] xifenfei
Instance name will be 'xifenfei'.
Is this correct? [ yes ] yes
Of the three components:
  [1] Client/Server and Data Manager
  [2] Data Manager Only
  [3] Client Only
Which would you like to install? [ 1 ] 1
Of the following options :
  [1] /home/oracle
  [2] /home/oracle/timesten
  [3] Specify a location
  [q] Quit the installation
Where would you like to install the xifenfei instance of TimesTen? [ 1 ] 2
Where would you like to create the daemon home directory? [ /home/oracle/timesten/TimesTen/xifenfei/info ]
The daemon logs will be located in /home/oracle/timesten/TimesTen/xifenfei/info
Would you like to specify a different location for the daemon logs? [ no ]
Installing into /home/oracle/timesten/TimesTen/xifenfei ...
Uncompressing ...
NOTE: If you are configuring TimesTen for use with Oracle Clusterware, the
      daemon port number must be the same across all TimesTen installations
      managed within the same Oracle Clusterware cluster.
NOTE: All installations that replicate to each other must use the same daemon
      port number that is set at installation time. The daemon port number can
      be verified by running 'ttVersion'.
The default port number is 53396.
Do you want to use the default port number for the TimesTen daemon? [ yes ]
The daemon will run on the default port number (53396).
NOTE: For security, we recommend that you restrict access to the
      TimesTen installation to members of a single OS group. Only members of
      that OS group will be allowed to perform direct mode connections to
      TimesTen, and only members of that OS group will be allowed to perform
      operations that access TimesTen data stores, TimesTen files and shared
      memory. The OS group defaults to the primary group of the instance
      administrator. You can default to this group, choose another OS group
      or you can make this instance world-accessible. If you choose to make
      this instance world-accessible, all database files and shared memory
      are readable and writable by all users.
Restrict access to the the TimesTen installation to the group 'oinstall'? [ yes ] yes
NOTE: Enabling PL/SQL will increase the size of some TimesTen libraries.
Would you like to enable PL/SQL for this instance? [ yes ] yes
      TNS_ADMIN was not set in your environment but there is a tnsnames.ora
      file in /opt/app/oracle/product/10.2.0/db_1/network/admin.
Would you like to use this TNS_ADMIN setting for the In-Memory Database Cache? [ yes ] yes
TNS_ADMIN will be set to /opt/app/oracle/product/10.2.0/db_1/network/admin
You can change TNS_ADMIN later by running <install_dir>/bin/ttmodinstall.
NOTE: It appears that you are running version 3.4 or higher of the g++
      compiler. TimesTen ships with multiple sets of client libraries and server
      binaries : one built for compatibility with g++ 3.4.6 and one with
      g++ 4.1.0. The installer has created links to the 3.4.6 library in the
      <install_dir>/lib directory and to the 3.4.6 server binary in the
      <install_dir>/bin directory. If you want to use a different compiler,
      please modify the links to point to the desired library and server binary.
Installing server components ...
What is the TCP/IP port number that you want the TimesTen Server to listen on? [ 53397 ]
Do you want to install QuickStart and the TimesTen Documentation? [ no ] no
Would you like to install the documentation (without QuickStart)? [ yes ]
Where would you like to create the doc directory (s=skip)? [ /home/oracle/timesten/TimesTen/xifenfei/doc ] s
Installing client components ...
Would you like to use TimesTen Replication with Oracle Clusterware? [ no ] no
NOTE: The TimesTen daemon startup/shutdown scripts have not been installed.
Run the 'setuproot' script :
        cd /home/oracle/timesten/TimesTen/xifenfei/bin
        ./setuproot -install
This will move the TimesTen startup script into its appropriate location.
The startup script is currently located here :
  '/home/oracle/timesten/TimesTen/xifenfei/startup/tt_xifenfei'.
The documentation was not installed.
To manually install the documentation, run the command 'setup.sh -installDoc'
The 11.2.2.2 Release Notes are located here :
  '/home/oracle/timesten/TimesTen/xifenfei/README.html'
Starting the daemon ...
TimesTen Daemon startup OK.
End of TimesTen installation.
--设置开机启动
[root@bas linux8664]# cd /home/oracle/timesten/TimesTen/xifenfei/bin/
[root@bas bin]# ./setuproot -install
Would you like to install the TimesTen daemon startup scripts into /etc/init.d? [ yes ] yes
Copying /home/oracle/timesten/TimesTen/xifenfei/startup/tt_xifenfei to /etc/init.d
Successfully installed the following scripts :
/etc/init.d/tt_xifenfei
/etc/rc.d/rc0.d/K45tt_xifenfei
/etc/rc.d/rc1.d/K45tt_xifenfei
/etc/rc.d/rc2.d/S90tt_xifenfei
/etc/rc.d/rc3.d/S90tt_xifenfei
/etc/rc.d/rc5.d/S90tt_xifenfei
/etc/rc.d/rc6.d/K45tt_xifenfei

2.配置DSN

[root@bas info]# pwd
/home/oracle/timesten/TimesTen/xifenfei/info
[root@bas info]# more sys.odbc.ini
[ODBC Data Sources]
TT_1122=TimesTen 11.2.2 Driver
[TT_1122]
Driver=/home/oracle/timesten/TimesTen/xifenfei/lib/libtten.so
DataStore=/home/oracle/timesten/TimesTen/xifenfei/info/TT_1122
DatabaseCharacterSet=ZHS16GBK
PermSize=300
TempSize=64
OracleNetServiceName=orcl_tt

Driver = the TimesTen Direct Linked ODBC Driver
DataStore = the location and the name of the database files
LogDir = the directory for the transaction logs
PermSize = the size of the permanent region of the database. In the above example, this is configured for 40MB
TempSize = the size of the temporary region of the database. In the above example, this is configured for 32MB
DatabaseCharacterSet = the character set used by the database
OracleNetServiceName = the TNS service name to the Oracle database. This attribute is required for In-Memory Database Cache only.

3.查看相关文件和进程

[root@bas info]# ps -ef|grep timesten|grep -v grep
oracle   30391     1  0 00:01 ?        00:00:00 /home/oracle/timesten/TimesTen/xifenfei/bin/timestend -initfd 13
oracle   30395 30391  0 00:01 ?        00:00:00 /home/oracle/timesten/TimesTen/xifenfei/bin/timestensubd -verbose -userlog tterrors.log -supportlog ttmesg.log -id 1000000 -facility user
oracle   30396 30391  0 00:01 ?        00:00:00 /home/oracle/timesten/TimesTen/xifenfei/bin/timestensubd -verbose -userlog tterrors.log -supportlog ttmesg.log -id 1000001 -facility user
oracle   30397 30391  0 00:01 ?        00:00:00 /home/oracle/timesten/TimesTen/xifenfei/bin/timestensubd -verbose -userlog tterrors.log -supportlog ttmesg.log -id 1000002 -facility user
oracle   30398 30391  0 00:01 ?        00:00:00 /home/oracle/timesten/TimesTen/xifenfei/bin/timestensubd -verbose -userlog tterrors.log -supportlog ttmesg.log -id 1000003 -facility user
oracle   30400 30391  0 00:01 ?        00:00:00 /home/oracle/timesten/TimesTen/xifenfei/bin/ttcserver -verbose -userlog tterrors.log -supportlog ttmesg.log -id 1000004 -p 53397 -facility user -group oinstall
[root@bas TimesTen]# pwd
/home/oracle/timesten/TimesTen
[root@bas TimesTen]# ls -l
total 4
drwxr-x---  17 oracle oinstall 4096 Feb 12 00:01 xifenfei
[root@bas TimesTen]# cd xifenfei
[root@bas xifenfei]# ls -l
total 140
drwxr-x---  4 oracle oinstall  4096 Feb 12 00:01 3rdparty
drwxr-x---  2 oracle oinstall  4096 Feb 12 00:43 bin
drwxr-x---  3 oracle oinstall  4096 Dec 24 00:23 include
drwxr-x---  2 oracle oinstall  4096 Feb 12 00:18 info
drwxr-x---  2 oracle oinstall  4096 Feb 12 00:00 lib
drwxr-x---  2 oracle oinstall  4096 Dec 24 00:23 mibs
drwxr-x---  3 oracle oinstall  4096 Dec 24 00:23 network
drwxr-x---  3 oracle oinstall  4096 Dec 24 00:23 nls
drwxr-x---  2 oracle oinstall  4096 Dec 24 00:23 oraclescripts
drwxr-x---  4 oracle oinstall  4096 Feb 11 23:59 PERL
drwxr-x---  8 oracle oinstall  4096 Dec 24 00:23 plsql
-r--r-----  1 oracle oinstall 74764 Feb 12 00:00 README.html
drwxr-x---  2 oracle oinstall  4096 Feb 11 23:59 startup
drwxr-x---  2 oracle oinstall  4096 Dec 24 00:23 support
drwxrw----  5 oracle oinstall  4096 Feb 12 00:00 ttclasses
drwxr-x---  3 oracle oinstall  4096 Feb 11 23:59 ttoracle_home

4.试用Timesten

[oracle@bas bin]$ ttisql -version
TimesTen Release 11.2.2.2.0
[oracle@bas info]$ ttisql TT_1122
Copyright (c) 1996-2011, Oracle.  All rights reserved.
Type ? or "help" for help, type "exit" to quit ttIsql.
connect "DSN=TT_1122";
Warning 01S00: Undefined connection attribute "ConnectionCharacte" on line 144 of file /home/oracle/timesten/TimesTen/xifenfei/info/sys.odbc.ini
Connection successful: DSN=TT_1122;UID=oracle;DataStore=/home/oracle/timesten/TimesTen/xifenfei/info/TT_1122;DatabaseCharacterSet=ZHS16GBK;ConnectionCharacterSet=US7ASCII;DRIVER=/home/oracle/timesten/TimesTen/xifenfei/lib/libtten.so;PermSize=300;TempSize=64;TypeMode=0;OracleNetServiceName=orcl_tt;
(Default setting AutoCommit=1)
Command> tables;
0 tables found.
Command> create table t_xff(id number,name varchar2(100));
Command> tables;
  ORACLE.T_XFF
1 table found.
Command> desc t_xff;
Table ORACLE.T_XFF:
  Columns:
    ID                              NUMBER
    NAME                            VARCHAR2 (100) INLINE
1 table found.
Command> insert into t_xff values(1,'xifenfei');
1 row inserted.
Command> insert into t_xff values(2,'www.xifenfei.com');
1 row inserted.
Command> commit;
Command> select * from t_xff;
< 1, xifenfei >
< 2, www.xifenfei.com >
2 rows found.
Command> update t_xff set name='xff' where id=1;
1 row updated.
Command> select * from t_xff;
< 1, xff >
< 2, www.xifenfei.com >
2 rows found.
Command> delete from t_xff where id=1;
1 row deleted.
Command> commit;
Command> select * from t_xff;
< 2, www.xifenfei.com >
1 row found.
Command> help
Use "help all" to get a description of all commands or use "help <cmd>" to
limit it to that command.
?                         free                      sqlcolumns
!                         functions                 sqlgetinfo
@@                        globalprocessing          sqlquerytimeout
accept                    help                      sqlstatistics
allfunctions              history                   sqltables
allindexes                host                      statsclear
allpackages               if                        statsestimate
allprocedures             indexes                   statsupdate
allsequences              isolation                 synonyms
allsynonyms               monitor                   tables
alltables                 tblsize                   timing
allviews                  multipleconnections       tryglobalprocessing
autocommit                ncharencoding             tryhash
builtins                  optfirstrow               trymaterialize
cachegroups               optprofile                trymergejoin
cachesqlget               packages                  trynestedloopjoin
clearhistory              passthrough               tryrowid
clienttimeout             prefetchcount             tryrowlocks
close                     prepare                   tryserial
closeall                  print                     trytbllocks
cmdcache                  procedures                trytmphash
columnlabels              quantify                  trytmptable
commit                    prompt                    trytmprange
commitdurable             remark                    tryrange
compact                   repschemes                undefine
compare                   retryconnect              unsetjoinorder
connect                   rollback                  unsetuseindex
define                    run                       use
describe                  savehistory               variable
disconnect                sequences                 verbosity
dssize                    set                       version
e:                        setjoinorder              vertical
exec                      setuseindex               views
execandfetch              setvariable               waitfor
exit                      show                      whenever
explain                   showjoinorder             xlabookmarkdelete
fetchall                  showplan                  <sql_statement>
fetchone                  sleep
Command> exit
Disconnecting...
Done.

nmon使用说明

Nmon 工具是 IBM 提供的免费的监控 AIX 系统与 Linux 系统资源的工具。该工具可实时监控系统性能,也可以将服务器的系统资源耗用情况收集起来并输出一个特定的文件,并可利用 excel 分析工具进行数据的统计分析,非常利用 UNIX 或者 Linux 系统的性能数据分析。

1.下载地址
nmon官网
NMON_Analyser官网
本地下载nmon
本地下载nmon_analyser

2.安装nmon
在压缩包中找到相应的版本,上传至服务器,然后授予执行权限

3.主要操作说明

+-HELP---------most-keys-toggle-on/off------------------------------------------+
|h = Help information     q = Quit nmon             0 = reset peak counts       |
|+ = double refresh time  - = half refresh          r = ResourcesCPU/HW/MHz/AIX |
|c = CPU by processor     C=upto 128 CPUs           p = LPAR Stats (if LPAR)    |
|l = CPU avg longer term  k = Kernel Internal       # = PhysicalCPU if SPLPAR   |
|m = Memory & Paging      M = Multiple Page Sizes   P = Paging Space            |
|d = DiskI/O Graphs       D = DiskIO +Service times o = Disks %Busy Map         |
|a = Disk Adapter         e = ESS vpath stats       V = Volume Group stats      |
|^ = FC Adapter (fcstat)  O = VIOS SEA (entstat)    v = Verbose=OK/Warn/Danger  |
|n = Network stats        N=NFS stats (NN for v4)   j = JFS Usage stats         |
|A = Async I/O Servers    w = see AIX wait procs   "="= Net/Disk KB<-->MB       |
|b = black&white mode     g = User-Defined-Disk-Groups (see cmdline -g)         |
|t = Top-Process --->     1=basic 2=CPU-Use 3=CPU(default) 4=Size 5=Disk-I/O    |
|u = Top+cmd arguments    U = Top+WLM Classes       . = only busy disks & procs |
|W = WLM Section          S = WLM SubClasses)                                   |

4.实时监控结果
1)监控内存使用情况

| Memory -----------------------------------------------------------------------|
|          Physical  PageSpace |        pages/sec  In     Out | FileSystemCache |
|% Used       93.8%     34.3%  | to Paging Space   0.0    0.0 | (numperm) 44.3% |
|% Free        6.2%     65.7%  | to File System    0.0  257.9 | Process   18.2% |
|MB Used    1786.0MB   175.8MB | Page Scans        0.0        | System    31.4% |
|MB Free     118.0MB   336.2MB | Page Cycles       0.0        | Free       6.2% |
|Total(MB)  1904.0MB   512.0MB | Page Steals       0.0        |           ------|
|                              | Page Faults     279.9        | Total    100.0% |
|------------------------------------------------------------ | numclient 44.3% |
|Min/Maxperm     361MB( 19%)  1443MB( 76%) <--% of RAM        | maxclient 75.8% |
|Min/Maxfree     960   1088       Total Virtual    2.4GB      | User      58.4% |
|Min/Maxpgahead    2      8    Accessed Virtual    0.9GB 40.1%| Pinned    28.6% |
|-------------------------------------------------------------------------------|

2)监控cpu使用情况

|                           0----------25-----------50----------75----------100
|CPU User%  Sys% Wait% Idle%|           |            |           |            |
|  0   0.0   0.0   0.0 100.0|>                                                |
|  1   0.0   0.0   0.0 100.0|>          |
|  2   0.0   0.0   0.0 100.0|>                                                |
|  3   0.0   0.0   0.0 100.0|>                                                |
|Physical Averages          +-----------|------------|-----------|------------+
|All   0.2   2.5   0.7  96.6|>                                                |
|                           +-----------|------------|-----------|------------+

3)监控进程状态

| Top-Processes-(147) -----Mode=3  [1=Basic 2=CPU 3=Perf 4=Size 5=I/O 6=Cmds]-----------------------------|
|  PID       %CPU     Size      Res     Res      Res     Char    RAM      Paging         Command          |
|            Used       KB      Set     Text     Data     I/O     Use   io   other repage                 |
| 1908868     0.8    30508    29764      132    29632        2    2%      0      3      0 secldapclntd    |
| 2306196     0.7      512      512        0      512        0    0%      1      8      0 trclogio        |
| 2732116     0.6     2520        0        0        0        0    0%      0     33      0 <defunct Zombie>|
|  340036     0.2     1416      296       72      224        0    0%      0      0      0 dtgreet         |

5.监控一段时间性能

-f            spreadsheet output format [note: default -s300 -c288]
optional
 -s <seconds>  between refreshing the screen [default 2]
 -c <number>   of refreshes [default millions]
 -t            spreadsheet includes top processes
具体信息nmon -h

例如:nmon -f -t -s 30 -c 120
-s 30:每30秒进行一次数据采集
-c 120:一共采集120次

6.分析数据
打开nmon analyser,设置宏的安全级别是低 ,之后点击 Analyser NMON data 按钮 输入文件 保存成excel格式即可。

Easy Connect Naming Method与EZCONNECT关系

Easy Connect Naming Method这个东西是Oracle 10g推出的东东,我想不用我解释它的好,dba和开发人员都喜欢它,以前一直都用它,从没有关注到它和sqlnet.ora中的NAMES.DIRECTORY_PATH的关系,昨天一朋友和我说到了EZCONNECT,今天查询了一些资料和做了一些实验,使得自己对NAMES.DIRECTORY_PATH和EZCONNECT有了新的认识,也怪自己一致忽略了这个知识点。

1.NAMES.DIRECTORY_PATH= (TNSNAMES)

[oracle@node1 samples]$ more $ORACLE_HOME/network/admin/sqlnet.ora
#NAMES.DIRECTORY_PATH= (EZCONNECT)
NAMES.DIRECTORY_PATH= (TNSNAMES)
[oracle@node1 samples]$ sqlplus hr/xifenfei@127.0.0.1/ecp
SQL*Plus: Release 10.2.0.5.0 - Production on Wed Feb 8 23:12:12 2012
Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.
ERROR:
ORA-12154: TNS:could not resolve the connect identifier specified
Enter user-name:
ERROR:
ORA-01017: invalid username/password; logon denied
Enter user-name:
ERROR:
ORA-01017: invalid username/password; logon denied
SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus

证明sqlnet.ora中的NAMES.DIRECTORY_PATH设置为TNSNAMES时,Easy Connect Naming Method不能工作。

2.NAMES.DIRECTORY_PATH= (EZCONNECT)

[oracle@node1 samples]$ more $ORACLE_HOME/network/admin/sqlnet.ora
NAMES.DIRECTORY_PATH= (EZCONNECT)
#NAMES.DIRECTORY_PATH= (TNSNAMES)
[oracle@node1 samples]$ sqlplus hr/xifenfei@127.0.0.1/ecp
SQL*Plus: Release 10.2.0.5.0 - Production on Wed Feb 8 23:13:21 2012
Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL>

登录成功,其实这里生效是因为设置了NAMES.DIRECTORY_PATH为EZCONNECT的功效,从EZCONNECT的上面也可以看出是

3.NAMES.DIRECTORY_PATH为默认值

[oracle@node1 samples]$ more $ORACLE_HOME/network/admin/sqlnet.ora
#NAMES.DIRECTORY_PATH= (EZCONNECT)
#NAMES.DIRECTORY_PATH= (TNSNAMES)
[oracle@node1 samples]$ sqlplus hr/xifenfei@127.0.0.1/ecp
SQL*Plus: Release 10.2.0.5.0 - Production on Wed Feb 8 23:13:49 2012
Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL>

当NAMES.DIRECTORY_PATH为默认值时,使用Easy Connect Naming Method也可以登录成功(正是因为这个原因,导致我忽略了EZCONNECT的存在).

4.sqlnet.ora中关于NAMES.DIRECTORY_PATH说明

#names.directory_path
#
#Syntax:  <adapter-name>
#Default: TNSNAMES,ONAMES,HOSTNAME
#
# Sets the (ordered) list of naming adaptors to use in resolving a name.
# The default is as shown for 3.0.2 of sqlnet onwards. The default was
# (TNSNAMES, ONAMES) before that. The value can be presented without
# parentheses if only a single entry is being specified. The parameter is
# recognized from version 2.3.2 of sqlnet onward. Acceptable values include:
#  TNSNAMES -- tnsnames.ora lookup
#  ONAMES   -- Oracle Names
#  HOSTNAME -- use the hostname (or an alias of the hostname)
#  NIS      -- NIS (also known as "yp")
#  CDS      -- OSF DCE's Cell Directory Service
#  NDS      -- Novell's Netware Directory Service

5.关于NAMES.DIRECTORY_PATH参数的补充说明

tnsnames:local naming naming method
Set to resolve a net service name through the tnsnames.ora file on the client.
hostname:host naming method
Set to resolve a host name alias through an existing names resolution service or a centrally-maintained set of /etc/hosts files.
onames:Oracle Names method
Set to resolve database objects through a Oracle Names server.
ldap:directory naming naming method
Set to resolve a database service name, net service name, or net service alias through a directory server.
cds:Cell Directory Services (CDS) external naming method
Set to resolve an Oracle database name in a Distributed Computing Environment (DCE) environment.
nis:Network Information Service (NIS) external naming method
Set to resolve service information through an existing NIS.
Ezconnect:The easy connect naming method eliminates the need for service name lookup in the tnsnames.ora files for TCP/IP environments; in fact, no naming or directory system is required if you use this method.

ORA-00600[4454]

数据库版本信息

SQL> select * from gv$version;
   INST_ID BANNER
---------- ----------------------------------------------------------------
         1 Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production
         1 PL/SQL Release 9.2.0.8.0 - Production
         1 CORE 9.2.0.8.0       Production
         1 TNS for IBM/AIX RISC System/6000: Version 9.2.0.8.0 - Production
         1 NLSRTL Version 9.2.0.8.0 - Production
         2 Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production
         2 PL/SQL Release 9.2.0.8.0 - Production
         2 CORE 9.2.0.8.0       Production
         2 TNS for IBM/AIX RISC System/6000: Version 9.2.0.8.0 - Production
         2 NLSRTL Version 9.2.0.8.0 - Production
10 rows selected.

早上检查某运营商的结算数据库时发现

Mon Feb  6 16:03:23 2012
Errors in file /oracle9/app/admin/settl/udump/settl1_ora_1355948.trc:
ORA-00600: internal error code, arguments: [4454], [], [], [], [], [], [], []
Mon Feb  6 16:03:23 2012
Errors in file /oracle9/app/admin/settl/udump/settl1_ora_1355948.trc:
ORA-00600: internal error code, arguments: [4454], [], [], [], [], [], [], []

trace文件信息

*** SESSION ID:(100.35758) 2012-02-06 16:03:23.223
*** 2012-02-06 16:03:23.223
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [4454], [], [], [], [], [], [], []
Current SQL statement for this session:
select trim(operator_id) into :b1  from b_sys_proc where program_name=:b2
----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp+0148          bl       ksedst               1029746CC ?
ksfdmp+0018          bl       01FD4010
kgeriv+0118          bl       _ptrgl
kgeasi+00cc          bl       kgeriv               1101FAF48 ? 110248038 ?
                                                   00000000C ? 110235978 ?
                                                   000000040 ?
ktcsptg+00c4         bl       kgeasi               1100062D8 ? 110389E88 ?
                                                   116600001166 ? 200000002 ?
                                                   000000000 ? 70000010738F0E8 ?
                                                   0FFFFFFFF ? 0FFFFFFFF ?
opiexe+0524          bl       ktcsptg              000000000 ? 000000000 ?
opiefn0+01c0         bl       opiexe               49FFFFA640 ? 4900000001 ?
                                                   FFFFFFFFFFFA5C8 ?
opiefn+0100          bl       opiefn0              01000CF60 ? 1029C61B0 ?
                                                   000000002 ? FFFFFFFFFFFC1BC ?
                                                   000000001 ? 00000000C ?
                                                   00000000B ? 110061F50 ?
opiodr+08cc          bl       _ptrgl
ttcpip+0cc4          bl       _ptrgl
opitsk+0d60          bl       ttcpip               11000CF60 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
opiino+0758          bl       opitsk               000000000 ? 000000000 ?
opiodr+08cc          bl       _ptrgl
opidrv+032c          bl       opiodr               3C00000018 ? 4101FAF48 ?
                                                   FFFFFFFFFFFF790 ? 0A000EEA8 ?
sou2o+0028           bl       opidrv               3C0C000000 ? 4A0147B50 ?
                                                   FFFFFFFFFFFF790 ?
main+0138            bl       01FD3A28
__start+0098         bl       main                 000000000 ? 000000000 ?
………………
SO: 70000010738f0e8, type: 4, owner: 7000001043acd90, flag: INIT/-/-/0x00
    (session) trans: 70000010dde1dd8, creator: 7000001043acd90, flag: (100041) USR/- BSY/-/-/-/-/-
              DID: 0001-0064-7625733A, short-term DID: 0000-0000-00000000
              txn branch: 7000001147dbed8
              oct: 0, prv: 0, sql: 700000117b7f778, psql: 0, user: 24/SETTLE
    O/S info: user: settle, term: , ospid: 1400836, machine: zwq_jies2
              program: /settlement/pkg01/bin/long/long_app@zwq_jies2 (T
    application name: /settlement/pkg01/bin/long/long_app@zwq_jies2 (T, hash value=0
    last wait for 'SQL*Net message from client' blocking sess=0x0 seq=483 wait_time=240
                driver id=54435000, #bytes=1, =0

查询MOS,发现是Bug# 1402161(虽然在mos中声明该bug是在9.2.0.1中已经被修复,但是这里的trace文件中的Call Stack Trace和mos中记录一致,而且网络上也存在很多9.2.0.8中关于该bug的情况),trace文件最后一段可以看出是在节点2上(zwq_jies2)运行/settlement/pkg01/bin/long/long_app程序导致出现该错误
产生该错误原因:

This exception is signalled because the savepoint number is not what was
  expected.
The current transaction savepoint is less than the beginning savepoint of
  the transaction.

解决方法:
把/settlement/pkg01/bin/long/long_app中的需要运行的程序分割成几个小程序运行。

Break the job into smaller chunks and reconnect for each part of the job
  to reset the savepoint number.