跟踪ORACLE非当前会话

使用oradebug

session 1
--查询出需要跟踪会话v$process.pid
session 2
SQL> oradebug SETORAPID 15
Unix process pid: 14851, image: oracle@xifenfei (TNS V1-V3)
SQL> oradebug EVENT 10046 TRACE NAME CONTEXT FOREVER, LEVEL 4
Statement processed.
--执行session 1
session 1
SQL> oradebug EVENT 10046 trace name context off
Statement processed.
SQL> oradebug TRACEFILE_NAME
/u01/oracle/admin/xifenfei/udump/xff_ora_14851.trc

使用dbms_system.set_sql_trace_in_session

session 1
--查询出需要跟踪会话的sid,SERIAL#
session 2
SQL> exec dbms_system.set_sql_trace_in_session(12,130,true);
PL/SQL procedure successfully completed.
--session 1执行sql
session2关闭跟踪
SQL> exec dbms_system.set_sql_trace_in_session(12,130,false);
PL/SQL procedure successfully completed.
--通过session 1找出trace文件

ORA-01052: required destination LOG_ARCHIVE_DUPLEX_DEST is not specified

SCN说明
1、Oracle的SCN在每秒16384次commit的情况下可以维持534年,每秒16384次commit是Oracle早先认为的任何系统的极限commit强度;
2、Oracle里SCN的起点是1988年1月1日;
3、_minimum_giga_scn=n的含义是把SCN往前推进到nG,但请注意,只有在SCN小于nG的时候才会用到这个隐含参数,反之则Oracle会置这个隐含参数于不顾。

求模拟_minimum_giga_scn值
这里通过时间差,大概的模拟_minimum_giga_scn小于当前时间和1988年1月1日的scn最大值(300>290)

SQL> select to_char(sysdate,'yyyy-mm-dd hh24:mi:ss')from dual;
TO_CHAR(SYSDATE,'YYYY-MM-DDHH24:MI:SS'
--------------------------------------
2012-03-18 04:27:50
SQL> select months_between(sysdate,to_date('19880101','yyyymmdd')) from dual;
MONTHS_BETWEEN(SYSDATE,TO_DATE('19880101','YYYYMMDD'))
------------------------------------------------------
                                             290.55443
SQL> select 16384*60*60*24*31*300/(1024*1024*1024) SCN from dual;
       SCN
----------
12260.7422

启动数据库测试

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
--------------------------------
*._minimum_giga_scn=12260
--------------------------------
SQL> startup pfile='/tmp/pfile'
ORACLE instance started.
Total System Global Area  236000356 bytes
Fixed Size                   451684 bytes
Variable Size             201326592 bytes
Database Buffers           33554432 bytes
Redo Buffers                 667648 bytes
Database mounted.
ORA-01052: required destination LOG_ARCHIVE_DUPLEX_DEST is not specified

分析ORA-01052

SQL> !oerr ora 1052
01052, 00000, "required destination LOG_ARCHIVE_DUPLEX_DEST is not specified"
// *Cause:  A valid destination for parameter LOG_ARCHIVE_DUPLEX_DEST was not
//          specified when parameter LOG_ARCHIVE_MIN_SUCCEED_DEST was set to
//          two.
// *Action: Either specify a value for parameter LOG_ARCHIVE_DUPLEX_DEST, or
//          reduce the value for parameter LOG_ARCHIVE_MIN_SUCCEED_DEST to one.
SQL> show parameter LOG_ARCHIVE_DUPLEX_DEST;
NAME                                 TYPE       VALUE
------------------------------------ ---------- ------------------------------
log_archive_duplex_dest              string
SQL> show parameter LOG_ARCHIVE_MIN_SUCCEED_DEST
NAME                                 TYPE       VALUE
------------------------------------ ---------- ------------------------------
log_archive_min_succeed_dest         integer    1

这里可以看出,不是以为我上面的两个参数设置错误导致ORA-01052,而是因为推进scn过大导致ORA-01052的错误

减小_minimum_giga_scn测试
这里选择_minimum_giga_scn小于当前时间和1988年1月1日的scn最大值(280<290) [sql] SQL> select 16384*60*60*24*31*280/(1024*1024*1024) SCN from dual; SCN ---------- 11443.3594 SQL> shutdown immediate ORA-01109: database not open Database dismounted. ORACLE instance shut down. ----------------------------- *._minimum_giga_scn=11443 ----------------------------- SQL> startup pfile='/tmp/pfile' ORACLE instance started. Total System Global Area 236000356 bytes Fixed Size 451684 bytes Variable Size 201326592 bytes Database Buffers 33554432 bytes Redo Buffers 667648 bytes Database mounted. Database opened. SQL> select to_char(dbms_flashback.get_system_change_number(), 2 '9999999999999999') from dual; TO_CHAR(DBMS_FLASHBACK.GET_SYSTEM_ ---------------------------------- 12286827692251 SQL> select dbms_flashback.get_system_change_number()/(1024*1024*1024) from dual; DBMS_FLASHBACK.GET_SYSTEM_CHANGE_NUMBER()/(1024*1024*1024) ---------------------------------------------------------- 11443 [/sql] 通过试验可以发现,在我们的数据库遇到异常,需要恢复通过推进scn来恢复的时候,不是推进的越大越好;如果推进的太大可能导致ORA-01052错误,一般建议是比当前不一致的scn稍微大一点 参考:http://www.dbsnake.net/two-scn-internal-points.html

收集统计信息出现ORA-00600[ksxprqfre3]

alert日志出现ORA-00600[ksxprqfre3]错误

Sun Mar 18 22:11:44 2012
Errors in file /oracle9/app/admin/offon/bdump/offon2_p001_8220790.trc:
ORA-00600: internal error code, arguments: [ksxprqfre3], [0x11033D5A8], [], [], [], [], [], []
Sun Mar 18 22:11:44 2012
Errors in file /oracle9/app/admin/offon/bdump/offon2_p001_8220790.trc:
ORA-00600: internal error code, arguments: [ksxprqfre3], [0x11033E178], [], [], [], [], [], []

通过这里可以发现,是由于并发进程导致异常

trace文件内容

Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Oracle Data Mining options
JServer Release 9.2.0.8.0 - Production
ORACLE_HOME = /oracle9/app/product/9.2.0
System name:    AIX
Node name:      zwq_offon2
Release:        3
Version:        5
Machine:        00CA56A44C00
Instance name: offon2
Redo thread mounted by this instance: 2
Oracle process number: 328
Unix process pid: 8220790, image: oracle@zwq_offon2 (P001)
*** 2012-03-18 22:11:43.994
*** SESSION ID:(381.10338) 2012-03-18 22:11:43.989
krqh = 11033d5a8
rqh trace:
  Slot: 6    Op: 114  ksxpirqh: irqh requested
  Slot: 7    Op: 130  ksxpirqh: rqh SUCC
  Slot: 8    Op: 123  ksxpwait: receive mv to doneq
  Slot: 9    Op: 121  ksxprcv: rqh found on doneq
  Slot: 10   Op: 102  ksxprqfre: freed to queue
  Slot: 11   Op: 101  ksxprqalo: allocated
  Slot: 12   Op: 105  ksxpdteq: enqueue to delta q
  Slot: 13   Op: 117  ksxpvsnd: send via ksxpisnd
  Slot: 14   Op: 106  ksxpisnd: send requested
  Slot: 0    Op: 107  ksxpisnd: send submit OK
  Slot: 1    Op: 153  pre send completion cbk
  Slot: 2    Op: 102  ksxprqfre: freed to queue
  Slot: 3    Op: 114  ksxpirqh: irqh requested
  Slot: 4    Op: 130  ksxpirqh: rqh SUCC
  Slot: 5    Op: 153  pre send completion cbk
Dump of memory from 0x000000011033D5A8 to 0x000000011033D660
11033D5A0                   00000001 1033D5A8          [.....3..]
11033D5B0 00000001 1033D5A8 00000001 1033D5B8  [.....3.......3..]
11033D5C0 00000001 1033D5B8 00000006 00000001  [.....3..........]
11033D5D0 00000000 00000000 00000000 00000000  [................]
11033D5E0 00000572 00000000 07000000 EE473B28  [...r.........G;(]
11033D5F0 00010000 00000000 00000001 1029CEA8  [.............)..]
11033D600 00000001 104382A0 07000000 F279B5A8  [.....C.......y..]
11033D610 00000000 00000572 00000000 00000000  [.......r........]
11033D620 00000000 00000000 00000000 13DCD225  [...............%]
11033D630 13DA7A25 00010001 26750000 00000000  [..z%....&u......]
11033D640 00000001 10339C70 6B996672 82997282  [.....3.pk.fr..r.]
11033D650 7B796665 69756A06 00000001 1033E228  [{yfeiuj......3.(]
*** 2012-03-18 22:11:43.996
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [ksxprqfre3], [0x11033D5A8], [], [], [], [], [], []
Current SQL statement for this session:
SELECT /*+ PIV_SSF */ SYS_OP_MSR(SYS_OP_COUNTCHG(SUBSTRB(ROWIDTOCHAR(A1.C0),1,15),1),COUNT(DISTINCT A1.C3),COUNT(*)) C0 FROM (SELECT /*+ INDEX_RRS(A2 "PK_TINEXWORK_DATA
") */ A2.ROWID C0,A2."WORKSHTSN" C1,A2."EXWORKSN" C2,SYS_OP_LBID(8574,'L',A2.ROWID) C3 FROM "OFFON"."TINEXWORK_DATA"  PX_GRANULE(0, BLOCK_RANGE, DYNAMIC) A2) A1
----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp+0148          bl       ksedst               102973B94 ?
ksfdmp+0018          bl       01FD34D8
kgerinv+00e8         bl       _ptrgl
kgeasnmierr+004c     bl       kgerinv              127C7F9D7226029B ?
                                                   9770A0AFAF4BDC0 ?
                                                   12E0BE826D694B2F ?
                                                   000000000 ? 000000010 ?
ksxprqfre+0054       bl       kgeasnmierr          110006728 ? 110338C88 ?
                                                   1030069D4 ? 100000001 ?
                                                   000000002 ? 11033D5A8 ?
                                                   080000000 ?
                                                   7F7F7F7F7F7F7F7F ?
ksxpwait+06bc        bl       ksxprqfre            7000000D6D793D8 ?
ksliwat+02e0         bl       ksxpwait             1014F4508 ? 1101FB1C8 ?
kslwaitns+001c       bl       ksliwat              FFFFFFFFFFFC4B0 ? 000000004 ?
                                                   FFFFFFFFFFFC590 ? 11030BC60 ?
                                                   FFFFFFFFFFFC530 ?
                                                   7000000DDF1FBE0 ? 110006868 ?
kskthbwt+01b4        bl       kslwaitns            000000002 ? 7000000EAEEB7A0 ?
                                                   000000000 ? 000000000 ?
                                                   000000868 ? 7000000DAFEFB30 ?
                                                   7000000F4979B10 ?
kslwait+005c         bl       kskthbwt             FFFFFFFFFFFC6E0 ?
                                                   2422424800000000 ?
                                                   1015EDD34 ? 110305194 ?
                                                   FFFF0000000003 ? 000000000 ?
                                                   000000001 ? FFFFFFFFFFFF920 ?
ksxprcv+0184         bl       kslwait              102970928 ? 000000006 ?
                                                   000000000 ? 000000000 ?
                                                   110305148 ? 110006728 ?
kxfpqidqr+0600       bl       ksxprcv              11029CEA8 ? 000000008 ?
                                                   FFFFFFFFFFFD020 ?
                                                   FFFFFFFFFFFCFC0 ?
                                                   162DFEF4180 ? 000000000 ?
                                                   000000000 ? 000000000 ?
kxfpqdqr+02c0        bl       kxfpqidqr            11029CF80 ? 11029CF20 ?
                                                   FFFFFFFFFFFD7C0 ?
                                                   7000000EAE2DA30 ?
kxfxgs+0050          bl       kxfpqdqr             101518AF0 ? 000002000 ?
                                                   FFFFFFFFFFFD9F0 ? 000000000 ?
kxfxmai+0100         bl       kxfxgs               102970928 ? 000000060 ?
                                                   7000000EEF059F0 ?
                                                   70000000006D670 ?
kxfprdp+05b0         bl       _ptrgl
opirip+0390          bl       kxfprdp
opidrv+0300          bl       opirip               000000018 ? 0101FB1C8 ?
                                                   000000000 ?
sou2o+0028           bl       opidrv               320C000000 ? 0A00E8B50 ?
                                                   000000000 ?
main+01a4            bl       01FD2EF0
__start+0098         bl       main                 000000000 ? 000000000 ?
--------------------- Binary Stack Dump ---------------------
    SO: 7000000cf267198, type: 4, owner: 7000000cd2a53e0, flag: INIT/-/-/0x00
    (session) trans: 0, creator: 7000000cd2a53e0, flag: (48000041) USR/- -/-/-/-/-/-
              DID: 0002-0148-003F8809, short-term DID: 0000-0000-00000000
              txn branch: 0
              oct: 0, prv: 0, sql: 0, psql: 0, user: 25/MONITOR
    O/S info: user: maint1, term: , ospid: 8220790, machine: zwq_acc2
              program: oracle@zwq_offon2 (P001)
    application name: SQL*Plus, hash value=3669949024
    waiting for 'PX Deq: reap credit' blocking sess=0x0 seq=182 wait_time=0
                =0, =0, =0
    temporary object counter: 0

这里可以得出信息如下:
1)系统环境AIX 5.3 9.2.0.8.0 RAC
2)报错的语句是收集TINEXWORK_DATA表相关统计信息
3)是由于SQL*Plus收集统计信息使用并发导致该错误

查询MOS发现
Bug 5887047: ORA-00600 [KSXPRQFRE3] TRACE WHEN GATHERING OPTIMIZER STATISTICS描述相符:
1)数据库版本9.2.0.8
2)收集统计信息操作导致
3)STACK TRACE一致

处理建议
针对官方没有给出明确的解决方案,因为该问题出现概率比较低,如果经常出现需要考虑升级数据库版本

Oracle Database常用补丁集Patch号

Patchset / PSU Patch Number

11.2.0.4	13390677      11.2.0.4.0 PATCH SET FOR ORACLE DATABASE SERVER
11.2.0.3.8	16902043      DATABASE PATCH SET UPDATE 11.2.0.3.8 (INCLUDES CPUOCT2013)
11.2.0.3.7	16619892      DATABASE PATCH SET UPDATE 11.2.0.3.7 (INCLUDES CPUJUL2013)
11.2.0.3.6	16056266      DATABASE PATCH SET UPDATE 11.2.0.3.6 (INCLUDES CPUAPR2013)
11.2.0.3.5	14727310      DATABASE PATCH SET UPDATE 11.2.0.3.5 (INCLUDES CPUJAN2013):
11.2.0.3.4	14275605      DATABASE PATCH SET UPDATE 11.2.0.3.4 (INCLUDES CPUOCT2012)
11.2.0.3.3	13923374      DATABASE PATCH SET UPDATE 11.2.0.3.3 (INCLUDES CPU JUL2012):
11.2.0.3.2	13696216      DATABASE PATCH SET UPDATE 11.2.0.3.2 (INCLUDES CPU APR2012)
11.2.0.3.1	13343438      DATABASE PATCH SET UPDATE 11.2.0.3.1 (INCLUDES CPU JAN2012)
11.2.0.3	10404530      11.2.0.3.0 PATCH SET FOR ORACLE DATABASE SERVER
11.2.0.2.11	16619893      DATABASE PATCH SET UPDATE 11.2.0.2.11 (INCLUDES CPUJUL2013)
11.2.0.2.10	16056267      DATABASE PATCH SET UPDATE 11.2.0.2.10 (INCLUDES CPUAPR2013)
11.2.0.2.9	14727315      DATABASE PATCH SET UPDATE 11.2.0.2.9 (INCLUDES CPUJAN2013):
11.2.0.2.8	14275621      DATABASE PATCH SET UPDATE 11.2.0.2.8 (INCLUDES CPUOCT2012)
11.2.0.2.7	13923804      DATABASE PATCH SET UPDATE 11.2.0.2.7 (INCLUDES CPU JUL2012)
11.2.0.2.6	13696224      DATABASE PATCH SET UPDATE 11.2.0.2.6 (INCLUDES CPU APR2012)
11.2.0.2.5	13343424      DATABASE PATCH SET UPDATE 11.2.0.2.5 (INCLUDES CPU JAN2012)
11.2.0.2.4	12827726      DATABASE PSU 11.2.0.2.4 (INCLUDES CPUOCT2011)
11.2.0.2.3	12419331      DATABASE PSU 11.2.0.2.3 (INCLUDES CPUJUL2011)
11.2.0.2.2	11724916      DATABASE PSU 11.2.0.2.2 (INCLUDES CPUAPR2011)
11.2.0.2.1	10248523      DATABASE PSU 11.2.0.2.1
11.2.0.2	10098816      11.2.0.2.0 PATCH SET FOR ORACLE DATABASE SERVER
11.2.0.1.6	12419378       DATABASE PSU 11.2.0.1.6 (INCLUDES CPUJUL2011)
11.2.0.1.5	11724930      DATABASE PSU 11.2.0.1.5 (INCLUDES CPUAPR2011)
11.2.0.1.4	10248516      DATABASE PSU 11.2.0.1.4 (INCLUDES CPUJAN2011)
11.2.0.1.3	9952216      DATABASE PSU 11.2.0.1.3 (INCLUDES CPUOCT2010)
11.2.0.1.2	9654983      DATABASE PSU 11.2.0.1.2 (INCLUDES CPUJUL2010)
11.2.0.1.1	9352237      DATABASE PSU 11.2.0.1.1
11.1.0.7.16	16619896 [*]      DATABASE PATCH SET UPDATE 11.1.0.7.16 (INCLUDES CPUJUL2013)
11.1.0.7.15	16056268  [*]      DATABASE PATCH SET UPDATE 11.1.0.7.15 (INCLUDES CPUAPR2013)
11.1.0.7.14	14739378 [*]      DATABASE PATCH SET UPDATE 11.1.0.7.14 (INCLUDES CPUJAN2013)
11.1.0.7.13	14275623 [*]      DATABASE PATCH SET UPDATE 11.1.0.7.13 (INCLUDES CPUOCT2012)
11.1.0.7.12	13923474      DATABASE PATCH SET UPDATE 11.1.0.7.12 (INCLUDES CPU JUL2012)
11.1.0.7.11	13621679      DATABASE PATCH SET UPDATE 11.1.0.7.11 (INCLUDES CPU APR2012)
11.1.0.7.10	13343461      DATABASE PATCH SET UPDATE 11.1.0.7.10 (INCLUDES CPU JAN2012)
11.1.0.7.9	12827740      DATABASE PSU 11.1.0.7.9 (INCLUDES CPUOCT2011)
11.1.0.7.8	12419384      DATABASE PSU 11.1.0.7.8 (INCLUDES CPUJUL2011)
11.1.0.7.7	11724936      DATABASE PSU 11.1.0.7.7 (INCLUDES CPUAPR2011)
11.1.0.7.6	10248531      DATABASE PSU 11.1.0.7.6 (INCLUDES CPUJAN2011)
11.1.0.7.5	9952228      DATABASE PSU 11.1.0.7.5 (INCLUDES CPUOCT2010)
11.1.0.7.4	9654987      DATABASE PSU 11.1.0.7.4 (INCLUDES CPUJUL2010)
11.1.0.7.3	9352179      DATABASE PSU 11.1.0.7.3 (INCLUDES CPUAPR2010)
11.1.0.7.2	9209238      DATABASE PSU 11.1.0.7.2 (INCLUDES CPUJAN2010)
11.1.0.7.1	8833297      DATABASE PSU 11.1.0.7.1 (INCLUDES CPUOCT2009)
11.1.0.7	6890831      11.1.0.7.0 PATCH SET FOR ORACLE DATABASE SERVER
10.2.0.5.12	16619894 [*]      DATABASE PATCH SET UPDATE 10.2.0.5.12 (INCLUDES CPUJUL2013):
10.2.0.5.11	16056270  [*]      DATABASE PATCH SET UPDATE 10.2.0.5.11 (INCLUDES CPUAPR2013)
10.2.0.5.10	14727319 [*]      DATABASE PATCH SET UPDATE 10.2.0.5.10 (INCLUDES CPUJAN2013):
10.2.0.5.9	14275629 [*]      DATABASE PATCH SET UPDATE 10.2.0.5.9 (INCLUDES CPUOCT2012)
10.2.0.5.8	13923855 [*]      DATABASE PATCH SET UPDATE 10.2.0.5.8 (INCLUDES CPU JUL2012)
10.2.0.5.7	13632743 [*]      DATABASE PATCH SET UPDATE 10.2.0.5.7 (INCLUDES CPU APR2012)
10.2.0.5.6	13343471 [*]      DATABASE PATCH SET UPDATE 10.2.0.5.6 (INCLUDES CPU JAN2012)
10.2.0.5.5	12827745 [*]      DATABASE PSU 10.2.0.5.5 (INCLUDES CPUOCT2011)
10.2.0.5.4	12419392      DATABASE PSU 10.2.0.5.4 (INCLUDES CPUJUL2011)
10.2.0.5.3	11724962      DATABASE PSU 10.2.0.5.3 (INCLUDES CPUAPR2011)
10.2.0.5.2	10248542      DATABASE PSU 10.2.0.5.2 (INCLUDES CPUJAN2011)
10.2.0.5.1	9952230      DATABASE PSU 10.2.0.5.1 (INCLUDES CPUOCT2010)
10.2.0.5	8202632      10.2.0.5.0 PATCH SET FOR ORACLE DATABASE SERVER
10.2.0.4.17	16619897 [*]      DATABASE PSU 10.2.0.4.17 (REQUIRES PRE-REQUISITE 10.2.0.4.4 | INCLUDES CPUJUL2013):
10.2.0.4.16	16056269 [*]      DATABASE PSU 10.2.0.4.16 (REQUIRES PRE-REQUISITE 10.2.0.4.4 | INCLUDES CPUAPR2013)
10.2.0.4.15	14736542 [*]      DATABASE PSU 10.2.0.4.15 (REQUIRES PRE-REQUISITE 10.2.0.4.4 | INCLUDES CPUJAN2013):
10.2.0.4.14	14275630 [**]      DATABASE PSU 10.2.0.4.14 (REQUIRES PRE-REQUISITE 10.2.0.4.4 | INCLUDES CPUOCT2012)
0.2.0.4.13	13923851 [*]      DATABASE PSU 10.2.0.4.13 (REQUIRES PRE-REQUISITE 10.2.0.4.4 | INCLUDES CPUJUL2012)
10.2.0.4.12	12879933 [*]
DATABASE PSU 10.2.0.4.12 (REQUIRES PRE-REQUISITE 10.2.0.4.4 | INCLUDES CPUAPR2012)
10.2.0.4.11	12879929 [*]      DATABASE PATCH SET UPDATE 10.2.0.4.11 (PRE-REQ 10.2.0.4.4 | INCLUDES CPUJAN2012)
10.2.0.4.10	12827778      DATABASE PSU 10.2.0.4.10 (REQUIRES PRE-REQUISITE 10.2.0.4.4 | INCLUDES CPUOCT2011)
10.2.0.4.9	12419397      DATABASE PSU 10.2.0.4.9 (REQUIRES PRE-REQUISITE 10.2.0.4.4 | INCLUDES CPUJUL2011)
10.2.0.4.8	11724977      DATABASE PSU 10.2.0.4.8 (REQUIRES PRE-REQUISITE 10.2.0.4.4 | INCLUDES CPUAPR2011)
10.2.0.4.7	10248636      DATABASE PSU 10.2.0.4.7 (REQUIRES PRE-REQUISITE 10.2.0.4.4 | INCLUDES CPUJAN2011)
10.2.0.4.6	9952234      DATABASE PSU 10.2.0.4.6 (REQUIRES PRE-REQUISITE 10.2.0.4.4 | INCLUDES CPUOCT2010)
10.2.0.4.5	9654991      DATABASE PSU 10.2.0.4.5 (REQUIRES PRE-REQUISITE 10.2.0.4.4 | INCLUDES CPUJUL2010)    [overlay PSU]
10.2.0.4.4	9352164      DATABASE PSU 10.2.0.4.4 (INCLUDES CPUAPR2010)
10.2.0.4.3	9119284      DATABASE PSU 10.2.0.4.3 (INCLUDES CPUJAN2010)
10.2.0.4.2	8833280      DATABASE PSU 10.2.0.4.2 (INCLUDES CPUOCT2009)
10.2.0.4.1	8576156      DATABASE PSU 10.2.0.4.1 (INCLUDES CPUJUL2009)
10.2.0.4	6810189      10.2.0.4.0 PATCH SET FOR ORACLE DATABASE SERVER
10.2.0.3	5337014      10.2.0.3 PATCH SET FOR ORACLE DATABASE SERVER
10.2.0.2	4547817      10.2.0.2 PATCH SET FOR ORACLE DATABASE SERVER
10.1.0.5	4505133      10.1.0.5 PATCH SET FOR ORACLE DATABASE SERVER
10.1.0.4	4163362      10.1.0.4 PATCH SET FOR ORACLE DATABASE SERVER
10.1.0.3	3761843      10.1.0.3 PATCH SET FOR ORACLE DATABASE SERVER
9.2.0.8	4547809      9.2.0.8 PATCH SET FOR ORACLE DATABASE SERVER
9.2.0.7	4163445      9.2.0.7 PATCH SET FOR ORACLE DATABASE SERVER
9.2.0.6	3948480      9.2.0.6 PATCH SET FOR ORACLE DATABASE SERVER
9.2.0.5	3501955      ORACLE 9I DATABASE SERVER RELEASE 2 - PATCH SET 4 VERSION 9.2.0.5.0
9.2.0.4	3095277      9.2.0.4 PATCH SET FOR ORACLE DATABASE SERVER
9.2.0.3	2761332      9.2.0.3 PATCH SET FOR ORACLE DATABASE SERVER
9.2.0.2	2632931      9.2.0.2 PATCH SET FOR ORACLE DATABASE SERVER
9.0.1.5	3301544      9.0.1.5 PATCHSET
9.0.1.4	2517300      9.0.1.4 PATCH SET FOR ORACLE DATABASE SERVER
9.0.1.3	2271678      9.0.1.3. PATCH SET FOR ORACLE DATA SERVER
8.1.7.4	2376472      8.1.7.4 PATCH SET FOR ORACLE DATA SERVER
8.1.7.3	2189751      8.1.7.3 PATCH SET FOR ORACLE DATA SERVER
8.1.7.2	1909158      8.1.7.2.1 PATCH SET FOR ORACLE DATA SERVER

NOTE:
[*] 10.2.0.4 and 10.2.0.5 are now in extended support mode and PSU’s released after Aug 01,2011 will need ES License to download them.
参考:Quick Reference to Patchset Patch Numbers [ID 753736.1]

10.2.0.5出现ORA-00600[kcblasm_1]

今天朋友执行一条如下sql

SELECT YF,
       KHBH,
       MAX(KHXM),
       MAX(YYBDM),
       MAX(YYBMC),
       MIN(ZJZH),
       SUM(CASE
             WHEN RQ = 20120321 THEN
              NVL(ZJYE, 0)
             ELSE
              0
           END),
       SUM(CASE
             WHEN RQ = 20120301 THEN
              NVL(ZJYE, 0)
             ELSE
              0
           END),
       SUM(CASE
             WHEN RQ = 20120321 THEN
              NVL(ZQSZ, 0)
             ELSE
              0
           END),
       SUM(CASE
             WHEN RQ = 20120301 THEN
              NVL(ZQSZ, 0)
             ELSE
              0
           END),
       SUM(CASE
             WHEN RQ = 20120321 THEN
              NVL(ZC, 0)
             ELSE
              0
           END),
       SUM(CASE
             WHEN RQ = 20120301 THEN
              NVL(ZC, 0)
             ELSE
              0
           END),
       SUM(CASE
             WHEN RQ = 20120321 THEN
              NVL(DWJZ, 1)
             ELSE
              0
           END),
       SUM(CASE
             WHEN RQ = 20120301 THEN
              NVL(DWJZ, 1)
             ELSE
              0
           END),
       MAX(NVL(ZC, 0)) ZGZC,
       SUM(NVL(ZRZC, 0)) ZRZC,
       SUM(NVL(ZCZC, 0)) ZCZC,
       SUM(NVL(FDYK, 0)) FDYK,
       SUM(NVL(ZRGF, 0)) ZRGF,
       SUM(NVL(ZCGF, 0)) ZCGF,
       AVG(NVL(ZC, 0)) PJZC,
       SUM(NVL(CJJE, 0)) CJJE,
       SUM(NVL(YJ, 0)) YJ,
       SUM(NVL(ZJFSS, 0)) ZJFSS,
       SUM(NVL(ZYJ, 0)) ZYJ,
       SUM(NVL(CJCS, 0)) CJCS
  FROM CRM_FX2_KHZJHZ P
 WHERE YF = 201203
 GROUP BY YF, KHBH;

出现ORA-00600: internal error code, arguments: [kcblasm_1], [103], [], [], [], [], []错误

数据库版本

Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bi
PL/SQL Release 10.2.0.5.0 - Production
CORE 10.2.0.5.0 Production
TNS for Linux: Version 10.2.0.5.0 - Production
NLSRTL Version 10.2.0.5.0 - Production

通过MOS发现该错误在10.2.0.5上有相关描述

This problem is hit only in 10.2.0.5, up to PSU 10.2.0.5.4 in which problem is fixed.
Problem is affecting 10.2.0.5 to 10.2.0.5.3.
Cause
This is is a known and common problem hit in 10.2.0.5, investigated and corrected in unpublished Bug 7612454.
The problem was introduced in 10.2.0.5, by the fix of Bug:7523755.
This has been corrected in PSU 10.2.0.5.4 and 11.2.

解决方法

To quickest way to the solution of this problem is to apply PSU 10.2.0.5.4, Patch 12419392 or later.
All alternative solutions for this problem are listed below:
•	Upgrade the database to 11.2.
OR
•	Apply 10.2.0.5.4 Patch Set Update (Patch 12419392) or later PSUs where bug is fixed.
The available PSUs are mentioned in "10.2.0.5 Patch Set Updates - List of Fixes in each PSU" (Document 1337394.1)
OR
•	Apply interim Patch 7612454 on top of 10.2.0.5 (10.2.0.5.0-10.2.0.5.3):
o	For UNIX / Linux platforms apply Patch 7612454 available for download on MOS.
o	For Windows platforms apply Patch 3 or higher.
Please check Document 342443.1 for latest patches available for Windows on top of 10.2.0.5.

这个错误在10.2.0.5上很容易发生,官方也没有给出发生这个错误的原因,以及如何改写相关sql,给出的方案是升级到10.2.0.5.4或者更高。这里可以看出升级需要慎重,可能导致原库有小部分sql不能正常执行。

lob类型数据转换为系统文件

1.插入一条blob数据

SQL> create directory ULTLOBDIR as '/home/oracle';
Directory created.
SQL> create table blobtest(col1 BLOB);
Table created.
SQL> declare
a_blob BLOB;
  2    3  bfile_name BFILE := BFILENAME('ULTLOBDIR','tt.txt.bak');
  4  begin
  5  insert into blobtest values (empty_blob())
  6  returning col1 into a_blob;
  7  dbms_lob.fileopen(bfile_name);
  8  dbms_lob.loadfromfile(a_blob, bfile_name, dbms_lob.getlength(bfile_name));
  9  dbms_lob.fileclose(bfile_name);
 10  commit;
 11  end;
 12  /
PL/SQL procedure successfully completed.
SQL> select dbms_lob.getlength(col1) from blobtest;
DBMS_LOB.GETLENGTH(COL1)
------------------------
                    4829
SQL> !pwd
/home/oracle
SQL> !ls -l tt.txt.bak
-rw-r--r-- 1 oracle oinstall 4829 03-19 17:26 tt.txt.bak

2.创建存储过程

CREATE OR REPLACE PROCEDURE RETRIEVE_LOB_TO_FILE(TEMP_BLOB IN BLOB,
                                                 FILE_PATH IN VARCHAR2,
                                                 FILE_NAME IN VARCHAR2) IS
  DATA_BUFFER   RAW(32767);
  POSITION      INTEGER := 1;
  FILEHANDLE    UTL_FILE.FILE_TYPE;
  ERROR_NUMBER  NUMBER;
  ERROR_MESSAGE VARCHAR2(100);
  BLOB_LENGTH   INTEGER;
  CHUNK_SIZE    BINARY_INTEGER := 32767;
BEGIN
  BLOB_LENGTH := DBMS_LOB.GETLENGTH(TEMP_BLOB);
  FILEHANDLE  := UTL_FILE.FOPEN(FILE_PATH, FILE_NAME, 'wb', 1024);
  WHILE POSITION < BLOB_LENGTH LOOP
    DBMS_LOB.READ(TEMP_BLOB, CHUNK_SIZE, POSITION, DATA_BUFFER);
    UTL_FILE.PUT_RAW(FILEHANDLE, DATA_BUFFER);
    POSITION    := POSITION + CHUNK_SIZE;
    DATA_BUFFER := NULL;
  END LOOP;
  UTL_FILE.FCLOSE(FILEHANDLE);
EXCEPTION
  WHEN OTHERS THEN
    BEGIN
      ERROR_NUMBER  := SQLCODE;
      ERROR_MESSAGE := SUBSTR(SQLERRM, 1, 100);
      DBMS_OUTPUT.PUT_LINE('Error #: ' || ERROR_NUMBER);
      DBMS_OUTPUT.PUT_LINE('Error Message: ' || ERROR_MESSAGE);
      UTL_FILE.FCLOSE_ALL;
    END;
END;
/

3.测试读取blob到系统

SQL> declare
  2  tmp_blob blob default empty_blob();
  3  begin
  4  dbms_lob.createtemporary(tmp_blob, true);
  5  select col1 into tmp_blob from blobtest;
  6  retrieve_lob_to_file (tmp_blob, 'ULTLOBDIR','xifenfei.txt');
  7  end;
  8  /
PL/SQL procedure successfully completed.
SQL> !pwd
/home/oracle
SQL> !ls -l xifenfei.txt
-rw-r--r-- 1 oracle oinstall 4829 03-20 23:44 xifenfei.txt

坏块分类与检测

很多人都对oracle都知道oracle坏块有逻辑坏块和物理坏块之分,其实根据物理和逻辑坏块的分类,还可以继续分下去;对于坏块的检查,很多人知道物理和逻辑坏块使用dbv和rman检测,那其他种类的坏块该怎么办呢?这篇文章整理自MOS,给大家整理个分析坏块的思路,在后续文章中,会给出各种坏块的解决思路
http://www.xifenfei.com/wp-content/uploads/2012/03/Corruption_block_type_and_analysis.pdf
附件:hout和hcheck

IP=FIRST作用说明

我相信细心的朋友,估计都会发现我们在使用netca创建rac(10g/11g)的监听的时候,会发现 IP=FIRST,如下面展示

LISTENER_VENUS =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS_LIST =
        (ADDRESS = (PROTOCOL = TCP)(HOST = racnode1-vip)(PORT = 1521)(IP = FIRST))
        (ADDRESS = (PROTOCOL = TCP)(HOST = racnode1)(PORT = 1521)(IP = FIRST))
      )
    )
  )

那么 IP=FIRST表示什么含义呢?我通过下面的试验证明

1. 主机相关配置

[oracle@localhost ~]$ hostname
localhost.localdomain
[oracle@localhost ~]$ more /etc/hosts
127.0.0.1               localhost.localdomain localhost
[oracle@localhost ~]$ /sbin/ifconfig
eth0      Link encap:Ethernet  HWaddr 00:15:17:67:9C:39
          inet addr:192.168.8.121  Bcast:192.168.15.255  Mask:255.255.248.0
          inet6 addr: fe80::215:17ff:fe67:9c39/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:378351781 errors:0 dropped:0 overruns:0 frame:0
          TX packets:357773718 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:65412948319 (60.9 GiB)  TX bytes:92608894986 (86.2 GiB)
          Base address:0x1100 Memory:88020000-88040000
eth0:0    Link encap:Ethernet  HWaddr 00:15:17:67:9C:39
          inet addr:202.91.244.3  Bcast:202.91.247.255  Mask:255.255.248.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Base address:0x1100 Memory:88020000-88040000
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:40636368 errors:0 dropped:0 overruns:0 frame:0
          TX packets:40636368 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3385490475 (3.1 GiB)  TX bytes:3385490475 (3.1 GiB)

2. 当前监听配置

SID_LIST_LISTENER =
  (SID_LIST =
      (SID_DESC =
      (SID_NAME = mcrm)
      (ORACLE_HOME = /opt/oracle/product/10.2.0/db_1)
      (GLOBAL_DBNAME =mcrm)
    )
  )
LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = localhost.localdomain)(PORT = 1521)  )
)
  )

3. 监听状态

 [oracle@localhost ~]$ lsnrctl status
LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 20-MAR-2012 00:17:04
Copyright (c) 1991, 2007, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost.localdomain)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 10.2.0.4.0 - Production
Start Date                03-MAR-2012 23:56:18
Uptime                    16 days 0 hr. 20 min. 45 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/oracle/product/10.2.0/db_1/network/admin/listener.ora
Listener Log File         /opt/oracle/product/10.2.0/db_1/network/log/listener.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=localhost.localdomain)(PORT=1521)))
Services Summary...
Service "mcrm" has 2 instance(s).
  Instance "mcrm", status UNKNOWN, has 1 handler(s) for this service...
  Instance "mcrm", status READY, has 1 handler(s) for this service...
Service "mcrmXDB" has 1 instance(s).
  Instance "mcrm", status READY, has 1 handler(s) for this service...
Service "mcrm_XPT" has 1 instance(s).
  Instance "mcrm", status READY, has 1 handler(s) for this service...
The command completed successfully

这里可以看出来监听的还是(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=localhost.localdomain)(PORT=1521))),也就是说监听的是主机名

4. 当前监听IP和端口

[oracle@localhost ~]$ netstat -an |grep 1521|grep LISTEN
tcp        0      0 0.0.0.0:1521                0.0.0.0:*                   LISTEN

这里看出来,监听所有网卡

5. 修改监听文件

SID_LIST_LISTENER =
  (SID_LIST =
      (SID_DESC =
      (SID_NAME = mcrm)
      (ORACLE_HOME = /opt/oracle/product/10.2.0/db_1)
      (GLOBAL_DBNAME =mcrm)
    )
  )
LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = localhost.localdomain)(PORT = 1521) (IP=FIRST) )
)
  )

6. 重启监听

[oracle@localhost ~]$ lsnrctl stop
LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 20-MAR-2012 00:18:42
Copyright (c) 1991, 2007, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost.localdomain)(PORT=1521)(IP=FIRST)))
The command completed successfully
[oracle@localhost ~]$ lsnrctl start
LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 20-MAR-2012 00:18:47
Copyright (c) 1991, 2007, Oracle.  All rights reserved.
Starting /opt/oracle/product/10.2.0/db_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 10.2.0.4.0 - Production
System parameter file is /opt/oracle/product/10.2.0/db_1/network/admin/listener.ora
Log messages written to /opt/oracle/product/10.2.0/db_1/network/log/listener.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost.localdomain)(PORT=1521)(IP=FIRST)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 10.2.0.4.0 - Production
Start Date                20-MAR-2012 00:18:48
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/oracle/product/10.2.0/db_1/network/admin/listener.ora
Listener Log File         /opt/oracle/product/10.2.0/db_1/network/log/listener.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
  Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "mcrm" has 1 instance(s).
  Instance "mcrm", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

通过(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521))),已经发现,这里只监听127.0.0.1

7. 查看监听IP和端口

[oracle@localhost ~]$ netstat -an |grep 1521|grep LISTEN
tcp        0      0 127.0.0.1:1521              0.0.0.0:*                   LISTEN

这里进一步验证监听的ip地址已经只有了127.0.0.1而没有了其他网卡的地址

总结说明
通过这里的试验证明IP = FIRST的作用使得当我们使用主机名为监听中的host配置的时候,它只会监听hostname解析出来的ip地址,而不是默认情况下所有网卡地址。

ORA-7445[__milli_memcpy]分析

帮朋友分析日志的过程中发现在两个节点上多次出现类此错误
节点1日志分析

Mon Mar  5 08:19:20 2012
Errors in file /u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc:
ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []
Mon Mar  5 08:19:21 2012
Errors in file /u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc:
ORA-07445: exception encountered: core dump [kghalf()+961] [SIGSEGV] [Invalid permissions for mapped object] [0x000000000] [] []
ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []
Mon Mar  5 08:19:22 2012
Errors in file /u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc:
ORA-07445: exception encountered: core dump [kghalf()+464] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
ORA-07445: exception encountered: core dump [kghalf()+961] [SIGSEGV] [Invalid permissions for mapped object] [0x000000000] [] []
ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []

对应trace文件

/u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc
Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Oracle Data Mining options
JServer Release 9.2.0.8.0 - Production
ORACLE_HOME = /u01/app/oracle/product/9.2.0
System name:	HP-UX
Node name:	HDDB1
Release:	B.11.23
Version:	U
Machine:	ia64
Instance name: hddms1
Redo thread mounted by this instance: 1
Oracle process number: 42
Unix process pid: 18556, image: oracle@HDDB1 (TNS V1-V3)
*** 2012-03-05 08:19:20.675
*** SESSION ID:(100.40701) 2012-03-05 08:19:20.602
Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0x9fffffffbf580000, PC: [0x400000000182a9f0, __milli_memcpy()+2448]
  r1: 6000000000524900       r20:                8       br5:                0
  r2:                0       r21: 9fffffffbf4c21ec       br6: 40000000018ad000
  r3:               37       r22:                8       br7: e000000120001400
  r4:                0       r23: c00000005e3bac98        ip: 400000000182a9f0
  r5: 60000000005311d8       r24: 9fffffffbf4c2268      iipa:                0
  r6:         40000004       r25: 9fffffffbf4c2268       cfm:       2a54295428
  r7: 9fffffffffff1ba8       r26: 9fffffffbf580000        um:               1a
  r8: 9fffffffbf4c2268       r27: 9fffffffbf57ff00       rsc:               1f
  r9:             8000       r28: 9fffffffbf580008       bsp: 9fffffffbf802150
 r10: 9fffffffbf4ba260       r29: 9fffffffbf57ff08  bspstore: 9fffffffbf802150
 r11:                0       r30:                0      rnat:                0
 r12: 9ffffffffffe5240       r31:                0       ccv:         20000000
 r13: 9fffffffbf5a1420      NaTs:                0      unat:                0
 r14: 9fffffffbf4c23e8       PRs: ffffffffffff4837      fpsr:    9804c8274433f
 r15: 9fffffffbf580028       br0: 400000000202a7e0       pfs: c000000000001b3e
 r16:  3fffffffffffdff       br1:                0        lc:  3ffffffffffce89
 r17: 9fffffffbf4c2258       br2:                0        ec:                5
 r18:                0       br3:                0       isr: 9fffffffbf802150
 r19:               80       br4:                0       ifa:                0
Reason code: 0008
*** 2012-03-05 08:19:20.735
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []
Current SQL statement for this session:
update pmis.PM_BZ_DC_DWRFHSJ set ZGFDFH=:v_result where PARINST_ID=:v_pid and T_ID>:v_mdtbegin and T_ID<=:v_mdtend
----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
c00000008975ebc0        49  procedure PMIS.PM_DC_SJJGGC_EXECINF
c0000000971ef530         2  anonymous block
----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp()+528         call     ksedst()             000000001 ?
                                                   C050000000000996 ?
                                                   4000000002A53E60 ?
ssexhd()+1184        call     ksedmp()             000000003 ?
                                                   C0500000000010A9 ?
                                                   40000000025E1420 ?
                                                   FFFFFFFFFFFF9075 ?
                                                   9FFFFFFFFFFDF2E0 ?
                                                   6000000000514428 ?
                                                   6000000000514420 ?
                                                   60000000005178E0 ?
<kernel>             call     ssexhd()             400000000086EFB0 ?
                                                   000000015 ? 000000007 ?
                                                   C050000000000085 ?
__milli_memcpy()+24  call     <kernel>
48
$cold_updgrh()+1088  call     __milli_memcpy()     000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   9FFFFFFFBF564348 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ?
upduaw()+224         call     $cold_updgrh()       C0000000A2665550 ?
                                                   9FFFFFFFBF558FA0 ?
                                                   C000000000000C1E ?
                                                   4000000001916600 ?
                                                   000000000 ? 00000E137 ?
kdusru()+720         call     upduaw()             C0000000A2665550 ?
                                                   C000000000005BBF ?
                                                   4000000001912AC0 ?
                                                   00000E135 ?
                                                   9FFFFFFFFFFE52B0 ?
                                                   C0000000A2665660 ?
                                                   9FFFFFFFFFFE63F0 ?
                                                   9FFFFFFFBF5590D8 ?
kauupd()+544         call     kdusru()             9FFFFFFFBF4C5D24 ?
                                                   000000000 ?
                                                   9FFFFFFFBF558FA0 ?
                                                   000000000 ?
                                                   C00000008B5D1108 ?
                                                   C00000008B5D1138 ?
                                                   000000018 ?
                                                   9FFFFFFFBF5591E0 ?
updrow()+3024        call     kauupd()             9FFFFFFFFFFE65D0 ?
                                                   000000000 ?
                                                   6000000000530010 ?
                                                   000000000 ?
                                                   C00000009202CF38 ?
                                                   00000010B ? 00001AA7A ?
                                                   C000000086F2B778 ?
qerupFetch()+608     call     updrow()             9FFFFFFFBF4C89F0 ?
                                                   C0000000A2665550 ?
                                                   C000000000000C1D ?
                                                   40000000018F5BF0 ?
                                                   000000000 ?
updaul()+1008        call     qerupFetch()         C0000000A0DB24D0 ?
                                                   000000000 ?
                                                   C0000000A2665550 ?
                                                   000007FFF ?
updThreePhaseExe()+  call     updaul()             C000000091A57FA0 ?
432                                                000000006 ? 000000000 ?
                                                   C0000000000019BB ?
                                                   40000000019DF9B0 ?
                                                   000008E23 ?
                                                   60000000001D30F0 ?
                                                   0000000BD ?
updexe()+624         call     updThreePhaseExe()   C000000091A57FA0 ?
                                                   000000000 ?
                                                   9FFFFFFFBF558FA0 ?
                                                   9FFFFFFFFFFF1AF0 ?
                                                   C000000000000FA6 ?
                                                   40000000019DFCA0 ?
                                                   000000000 ?
opiexe()+6944        call     updexe()             C000000091A57FA0 ?
                                                   9FFFFFFFBF4C8A00 ?
                                                   C000000000002858 ?
                                                   4000000001857620 ?
                                                   000008E25 ?
                                                   9FFFFFFFFFFF1AD0 ?
                                                   9FFFFFFFBF558FA0 ?
                                                   9FFFFFFFBF4C88D4 ?
opipls()+3488        call     opiexe()             000000004 ?
                                                   9FFFFFFFFFFF2F50 ?
                                                   9FFFFFFFFFFF1D30 ?
                                                   C000000000002C60 ?
                                                   40000000019852D0 ?
                                                   000000000 ? 000007123 ?
                                                   9FFFFFFFFFFF1B40 ?
opiodr()+3088        call     opipls()             9FFFFFFFFFFF2F50 ?
                                                   9FFFFFFFFFFF2F58 ?
                                                   000000004 ?
                                                   C00000000000214A ?
                                                   40000000018747B0 ?
                                                   00000E123 ?
                                                   9FFFFFFFFFFF2E70 ?
                                                   9FFFFFFFFFFF2DE0 ?
rpidrus()+304        call     opiodr()             9FFFFFFFFFFF3810 ?
                                                   9FFFFFFFFFFF37E0 ?
                                                   9FFFFFFFFFFF36F8 ?
                                                   9FFFFFFFFFFF36F4 ?
skgmstack()+288      call     rpidrus()            9FFFFFFFFFFF4DA0 ?
                                                   C000000000000716 ?
                                                   4000000001862FE0 ?
                                                   9FFFFFFFFFFF47F0 ?
                                                   600000000004F3C0 ?
                                                   9FFFFFFFFFFF4DB8 ?
                                                   6000000000524900 ?
                                                   9FFFFFFFFFFF4DA0 ?
rpidru()+256         call     skgmstack()          9FFFFFFFFFFF4DC0 ?
                                                   600000000004EDF0 ?
                                                   00000F618 ?
                                                   400000000129E400 ?
                                                   9FFFFFFFFFFF4DA0 ?
rpiswu2()+784        call     rpidru()             00000057B ?
                                                   C000000000001329 ?
                                                   4000000001870CE0 ?
rpidrv()+2096        call     rpiswu2()            C00000007D7B3708 ?
                                                   9FFFFFFFFFFF5510 ?
                                                   6000000000515CE8 ?
                                                   60000000000502E8 ?
                                                   9FFFFFFFFFFF5520 ?
                                                   000000205 ?
                                                   6000000000530010 ?
                                                   6000000000531090 ?
psddr0()+256         call     rpidrv()             000000000 ? 000000066 ?
                                                   9FFFFFFFFFFF6630 ?
                                                   00000003A ?
psdnal()+672         call     psddr0()             000000000 ? 000000066 ?
                                                   9FFFFFFFFFFF6630 ?
                                                   000000030 ?
pevm_EXIM()+400      call     psdnal()             9FFFFFFFFFFF84A0 ?
                                                   9FFFFFFFFFFF8440 ?
                                                   6000000000531A04 ?
                                                   600000000064BAA0 ?
                                                   C000000090880CC8 ?
                                                   600000000063EA80 ?
pfrrun()+37120       call     pevm_EXIM()          9FFFFFFFBF567AD8 ?
                                                   C000000090880CC8 ?
                                                   C000000000001532 ?
                                                   400000000173DAD0 ?
                                                   000006269 ?
                                                   4000000002190B40 ?
                                                   9FFFFFFFBF4BA068 ?
                                                   9FFFFFFFBF5C08A0 ?
peicnt()+544         call     pfrrun()             9FFFFFFFBF567CDC ?
                                                   C000000000000D1D ?
                                                   4000000001676FE0 ?
kkxexe()+832         call     peicnt()             9FFFFFFFFFFF84A0 ?
                                                   9FFFFFFFBF567AD8 ?
                                                   C000000000000818 ?
                                                   40000000021A3A80 ?
                                                   00000FEAB ?
                                                   9FFFFFFFFFFF7EB0 ?
                                                   9FFFFFFFBF567CDC ?
                                                   000000000 ?
opiexe()+12592       call     kkxexe()             9FFFFFFFFFFF8430 ?
                                                   C000000000002858 ?
                                                   4000000001858C30 ?
                                                   0000080AB ?
                                                   9FFFFFFFFFFF8420 ?
                                                   9FFFFFFFBF4F3A6C ?
                                                   9FFFFFFFBF4F3A68 ?
                                                   9FFFFFFFFFFF84A0 ?
opiall0()+3456       call     opiexe()             000000004 ?
                                                   9FFFFFFFFFFF9A80 ?
                                                   9FFFFFFFFFFF8830 ?
                                                   C0000000000024D1 ?
                                                   40000000017A3350 ?
                                                   000000000 ? 00000C82B ?
                                                   9FFFFFFFFFFF8640 ?
kpoal8()+2272        call     opiall0()            000000002 ?
                                                   9FFFFFFFFFFF9A10 ?
                                                   9FFFFFFFFFFFA2B4 ?
                                                   000000000 ?
                                                   9FFFFFFFFFFF9954 ?
                                                   9FFFFFFFFFFFA2BC ?
                                                   000000000 ?
                                                   FFFFFFFFFFFFDFFF ?
opiodr()+3088        call     kpoal8()             9FFFFFFFFFFFA0B0 ?
                                                   000000000 ? 000000000 ?
                                                   C00000000000214A ?
                                                   40000000018747B0 ?
                                                   00000F82F ?
                                                   9FFFFFFFFFFFA130 ?
                                                   9FFFFFFFFFFFA1D0 ?
ttcpip()+1888        call     opiodr()             9FFFFFFFFFFFA460 ?
                                                   9FFFFFFFFFFFA430 ?
                                                   9FFFFFFFFFFFA348 ?
                                                   9FFFFFFFFFFFA344 ?
                                                   C0000000000018B9 ?
                                                   400000000172CB70 ?
                                                   000000000 ? 00000C0AB ?
opitsk()+1920        call     ttcpip()             6000000000052C40 ?
                                                   000000002 ?
                                                   9FFFFFFFFFFFB950 ?
                                                   6000000000273AA0 ?
                                                   9FFFFFFFFFFFBAB0 ?
                                                   9FFFFFFFFFFFB8C4 ?
                                                   9FFFFFFFBF75A660 ?
                                                   4000000000D2C7AA ?
opiino()+2656        call     opitsk()             000000000 ? 000000000 ?
                                                   C000000000000D1F ?
                                                   400000000236CE90 ?
                                                   000000000 ?
opiodr()+3088        call     opiino()             60000000005DEC18 ?
                                                   6000000000531170 ?
                                                   9FFFFFFFFFFFF4C0 ?
                                                   C00000000000214A ?
                                                   40000000018747B0 ?
                                                   00000E825 ?
                                                   9FFFFFFFFFFFD7C0 ?
                                                   9FFFFFFFFFFFF4D0 ?
opidrv()+1088        call     opiodr()             9FFFFFFFFFFFDF10 ?
                                                   9FFFFFFFFFFFDEE0 ?
                                                   9FFFFFFFFFFFDDF8 ?
                                                   9FFFFFFFFFFFDDF4 ?
sou2o()+48           call     opidrv()             9FFFFFFFFFFFEF60 ?
                                                   000000004 ?
                                                   9FFFFFFFFFFFF4C0 ?
main()+352           call     sou2o()              9FFFFFFFFFFFF4E0 ?
                                                   00000003C ? 000000004 ?
                                                   9FFFFFFFFFFFF4C0 ?
main_opd_entry()+80  call     main()               000000000 ?
                                                   9FFFFFFFFFFFF9B0 ?
                                                   C000000000000004 ?
                                                   C00000000002FA60 ?
--------------------- Binary Stack Dump ---------------------

观察节点2,发现类此错误不同之处有

Wed Mar  7 08:19:11 2012
Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc:
ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []
Wed Mar  7 08:19:12 2012
Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc:
ORA-00600: internal error code, arguments: [kghfrempty:ds], [0x9FFFFFFFBF55A550], [], [], [], [], [], []
ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []
Wed Mar  7 08:19:13 2012
Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc:
ORA-07445: exception encountered: core dump [kgidmp()+1568] [SIGSEGV] [Address not mapped to object] [0xB38F00000000D9] [] []
ORA-00600: internal error code, arguments: [kghfrempty:ds], [0x9FFFFFFFBF55A550], [], [], [], [], [], []
ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []
Wed Mar  7 08:19:14 2012
Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc:
ORA-07445: exception encountered: core dump [kghalf()+993] [SIGSEGV] [Address not mapped to object] [0xC0000000A8B01DA0] [] []
ORA-07445: exception encountered: core dump [kgidmp()+1568] [SIGSEGV] [Address not mapped to object] [0xB38F00000000D9] [] []
ORA-00600: internal error code, arguments: [kghfrempty:ds], [0x9FFFFFFFBF55A550], [], [], [], [], [], []
ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []

对应trace文件不同之处

*** 2012-03-07 08:19:11.582
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []
No current SQL statement being executed.
----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
c000000090c4a818      1356  procedure PMIS.PM_DC_SJJGGC_QDX
c0000000a25512c0        43  procedure PMIS.PM_DC_SJJGGC_EXECINF
c00000009a8306e8         2  anonymous block

通过这两个节点的错误记录,都包含PMIS.PM_DC_SJJGGC_EXECINF和anonymous,所以初步怀疑是某个程序在这个时间定运行某个任务(包含PMIS.PM_DC_SJJGGC_EXECINF过程)导致。
继续观察trace日志发现两个节点都是类此情况相同的程序,相同的主机

  SO: c00000007d7b3708, type: 4, owner: c00000007d6d0190, flag: INIT/-/-/0x00
    (session) trans: c000000082974470, creator: c00000007d6d0190, flag: (100041) USR/- BSY/-/-/-/-/-
              DID: 0001-002A-0000693D, short-term DID: 0000-0000-00000000
              txn branch: 0000000000000000
              oct: 6, prv: 0, sql: c0000000a229ab38, psql: c000000084e6b510, user: 517/PMIS
    O/S info: user: Administrator, term: DMIS_SWITCH2, ospid: 5600:3020, machine: WORKGROUP\DMIS_SWITCH2
              program: PMTDCtrl.exe
    application name: PMTDCtrl.exe, hash value=0
    last wait for 'enqueue' blocking sess=0x0 seq=32849 wait_time=127
                name|mode=54540004, id1=1, id2=10
    temporary object counter: 0

现在更加怀疑是PMTDCtrl.exe运行某过程导致数据库出现这些错误

错误总结
查看MOS发现该问题是一个bug导致(Bug 6166690:ORA-7445 [MILLI_MEMCPY] WHEN REF CURSOR AS OUT PARAMETER)
错误原因:It is caused by a Dynamic SQL used to obtain a REF CURSOR from a procedure (that opens the cursor).
处理建议:通过个bug,我们可以分析PMIS.PM_DC_SJJGGC_EXECINF过程,看看是否符合bug描述,如果符合建议改写过程或者打上补丁(Patch 2709343)
补丁:p2709343_92080_HPUX-IA64

weblogic相关日志介绍

对于一个系统,数据库维护人员来说,对系统/数据库的运行日志都会比较感兴趣。很多情况下在出现故障前,我们可以通过日志发现预兆,把故障扼杀在摇篮中;有些问题发生我们不能及时发现或者当时分析问题不够全面,我们需要结合日志进一步分析问题,寻找原因,提出解决方案,避免类此问题再次出现。对于weblogic系统的维护也不例外,我们也需要关注起日志情况。
一、日志汇总

[oracle@xifenfei logs]$ pwd
/u01/weblogic/user_projects/xff/xifenfei/servers/AdminServer/logs
--其中user_projects/xff/xifenfei需要根据你的实际情况决定
[oracle@xifenfei logs]$ ll *.log
-rw-r-----  1 oracle oinstall  32557 Mar 18 00:01 access.log
-rw-r-----  1 oracle oinstall  49055 Mar 17 19:22 AdminServer-diagnostic.log
-rw-r-----  1 oracle oinstall 741595 Mar 18 00:22 AdminServer.log
-rw-r-----  1 oracle oinstall 306831 Mar 18 00:01 xifenfei.log
--access.log HTTP访问日志
--AdminServer-diagnostic.log weblogic的诊断日志(开发模式下)
--AdminServer.log WebLogic SERVER运行日志
--xifenfei.log DOMAIN运行日志(名称和你的域相同)

二、日志详细介绍
1.access.log
在WebLogic中可以对用HTTP,HTTPS协议访问的服务器上的文件都做记录,该LOG文件默认的名字为Access.log,内容如下,该文件具体记录在某个时间,某个IP地址的客户端访问了服务器上的那个文件。
格式如:

192.168.1.1 - - [17/Mar/2012:16:23:05 +0800] "GET /css/fmw.css HTTP/1.1" 200 3250
192.168.1.1 - - [17/Mar/2012:16:23:05 +0800] "GET /css/fmw_bottom_area.css HTTP/1.1" 200 2689
192.168.1.1 - - [17/Mar/2012:16:23:05 +0800] "GET /JSLibrary/educational_text.js HTTP/1.1" 200 39124
192.168.1.1 - - [17/Mar/2012:16:23:05 +0800] "GET /JSLibrary/coneEventHandeler.js HTTP/1.1" 200 7689
192.168.1.1 - - [17/Mar/2012:16:23:05 +0800] "GET /JSLibrary/getIllustration.js HTTP/1.1" 200 1802
192.168.1.1 - - [17/Mar/2012:16:23:05 +0800] "GET /JSLibrary/GeneratePopup.js HTTP/1.1" 200 32682
192.168.1.1 - - [17/Mar/2012:16:23:05 +0800] "GET /images/oracle_logo_red.png HTTP/1.1" 200 7202

2.AdminServer-diagnostic.log
该日志在生产环境中没有,主要是记录程序运行中的一些错误的详细信息(程序员最关注的东西)
格式如:

MDS-01329: unable to load element "persistence-config"
MDS-01370: MetadataStore configuration for metadata-store-usage "OWSM_TargetRepos" is invalid.
ORA-06550: line 1, column 12:
PLS-00201: identifier 'MDS_INTERNAL_SHREDDED.GETREPOSITORYVERSION' must be declared
ORA-06550: line 1, column 7:
PL/SQL: Statement ignored

3.AdminServer.log
假如WebLogic SERVER在启动或运行过程中有错误发生,错误信息会显示在屏幕上,并且会记录在一个LOG文件中,该文件默认名为AdminServer.log。该文件也记录WebLogic的启动及关闭等其他运行信息。可在Gernal属性页中设置该文件的路径及名字,错误的输出的等级等

4.xifenfei.log
记录一个DOMIAN的运行情况,一个DOMAIN中的各个WebLogic SERVER可以把它们的一些运行信息(比如:很严重的错误)发送给一个DOMAIN的ADMINISTRATOR SERVER上,ADMINISTRATOR SERVER把这些信息些到DOMAIN 日志中。默认名为:domain_name.log

三、图形界面操作
登录weblogic后台进行相关操作
1.日志相关参数设置
左侧菜单:环境–>服务器
右侧菜单:服务器–>日志记录
只找到AdminServer.log、access.log,进行一些操作
如:设置日志文件的回滚,设置日志文件名等

2.日志查看
左侧菜单:环境–>服务器
右侧菜单:点击服务器名称
左侧菜单:诊断–>日志文件
右侧菜单:选中相关日志点击查看