联系：手机/微信(+86 17813235971) QQ(107644445)
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
帮朋友分析日志的过程中发现在两个节点上多次出现类此错误
节点1日志分析
Mon Mar 5 08:19:20 2012 Errors in file /u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc: ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] [] Mon Mar 5 08:19:21 2012 Errors in file /u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc: ORA-07445: exception encountered: core dump [kghalf()+961] [SIGSEGV] [Invalid permissions for mapped object] [0x000000000] [] [] ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] [] Mon Mar 5 08:19:22 2012 Errors in file /u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc: ORA-07445: exception encountered: core dump [kghalf()+464] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] [] ORA-07445: exception encountered: core dump [kghalf()+961] [SIGSEGV] [Invalid permissions for mapped object] [0x000000000] [] [] ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []
对应trace文件
/u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc
Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Oracle Data Mining options
JServer Release 9.2.0.8.0 - Production
ORACLE_HOME = /u01/app/oracle/product/9.2.0
System name: HP-UX
Node name: HDDB1
Release: B.11.23
Version: U
Machine: ia64
Instance name: hddms1
Redo thread mounted by this instance: 1
Oracle process number: 42
Unix process pid: 18556, image: oracle@HDDB1 (TNS V1-V3)
*** 2012-03-05 08:19:20.675
*** SESSION ID:(100.40701) 2012-03-05 08:19:20.602
Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0x9fffffffbf580000, PC: [0x400000000182a9f0, __milli_memcpy()+2448]
r1: 6000000000524900 r20: 8 br5: 0
r2: 0 r21: 9fffffffbf4c21ec br6: 40000000018ad000
r3: 37 r22: 8 br7: e000000120001400
r4: 0 r23: c00000005e3bac98 ip: 400000000182a9f0
r5: 60000000005311d8 r24: 9fffffffbf4c2268 iipa: 0
r6: 40000004 r25: 9fffffffbf4c2268 cfm: 2a54295428
r7: 9fffffffffff1ba8 r26: 9fffffffbf580000 um: 1a
r8: 9fffffffbf4c2268 r27: 9fffffffbf57ff00 rsc: 1f
r9: 8000 r28: 9fffffffbf580008 bsp: 9fffffffbf802150
r10: 9fffffffbf4ba260 r29: 9fffffffbf57ff08 bspstore: 9fffffffbf802150
r11: 0 r30: 0 rnat: 0
r12: 9ffffffffffe5240 r31: 0 ccv: 20000000
r13: 9fffffffbf5a1420 NaTs: 0 unat: 0
r14: 9fffffffbf4c23e8 PRs: ffffffffffff4837 fpsr: 9804c8274433f
r15: 9fffffffbf580028 br0: 400000000202a7e0 pfs: c000000000001b3e
r16: 3fffffffffffdff br1: 0 lc: 3ffffffffffce89
r17: 9fffffffbf4c2258 br2: 0 ec: 5
r18: 0 br3: 0 isr: 9fffffffbf802150
r19: 80 br4: 0 ifa: 0
Reason code: 0008
*** 2012-03-05 08:19:20.735
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []
Current SQL statement for this session:
update pmis.PM_BZ_DC_DWRFHSJ set ZGFDFH=:v_result where PARINST_ID=:v_pid and T_ID>:v_mdtbegin and T_ID<=:v_mdtend
----- PL/SQL Call Stack -----
object line object
handle number name
c00000008975ebc0 49 procedure PMIS.PM_DC_SJJGGC_EXECINF
c0000000971ef530 2 anonymous block
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp()+528 call ksedst() 000000001 ?
C050000000000996 ?
4000000002A53E60 ?
ssexhd()+1184 call ksedmp() 000000003 ?
C0500000000010A9 ?
40000000025E1420 ?
FFFFFFFFFFFF9075 ?
9FFFFFFFFFFDF2E0 ?
6000000000514428 ?
6000000000514420 ?
60000000005178E0 ?
<kernel> call ssexhd() 400000000086EFB0 ?
000000015 ? 000000007 ?
C050000000000085 ?
__milli_memcpy()+24 call <kernel>
48
$cold_updgrh()+1088 call __milli_memcpy() 000000000 ? 000000000 ?
000000000 ? 000000000 ?
9FFFFFFFBF564348 ?
000000000 ? 000000000 ?
000000000 ?
upduaw()+224 call $cold_updgrh() C0000000A2665550 ?
9FFFFFFFBF558FA0 ?
C000000000000C1E ?
4000000001916600 ?
000000000 ? 00000E137 ?
kdusru()+720 call upduaw() C0000000A2665550 ?
C000000000005BBF ?
4000000001912AC0 ?
00000E135 ?
9FFFFFFFFFFE52B0 ?
C0000000A2665660 ?
9FFFFFFFFFFE63F0 ?
9FFFFFFFBF5590D8 ?
kauupd()+544 call kdusru() 9FFFFFFFBF4C5D24 ?
000000000 ?
9FFFFFFFBF558FA0 ?
000000000 ?
C00000008B5D1108 ?
C00000008B5D1138 ?
000000018 ?
9FFFFFFFBF5591E0 ?
updrow()+3024 call kauupd() 9FFFFFFFFFFE65D0 ?
000000000 ?
6000000000530010 ?
000000000 ?
C00000009202CF38 ?
00000010B ? 00001AA7A ?
C000000086F2B778 ?
qerupFetch()+608 call updrow() 9FFFFFFFBF4C89F0 ?
C0000000A2665550 ?
C000000000000C1D ?
40000000018F5BF0 ?
000000000 ?
updaul()+1008 call qerupFetch() C0000000A0DB24D0 ?
000000000 ?
C0000000A2665550 ?
000007FFF ?
updThreePhaseExe()+ call updaul() C000000091A57FA0 ?
432 000000006 ? 000000000 ?
C0000000000019BB ?
40000000019DF9B0 ?
000008E23 ?
60000000001D30F0 ?
0000000BD ?
updexe()+624 call updThreePhaseExe() C000000091A57FA0 ?
000000000 ?
9FFFFFFFBF558FA0 ?
9FFFFFFFFFFF1AF0 ?
C000000000000FA6 ?
40000000019DFCA0 ?
000000000 ?
opiexe()+6944 call updexe() C000000091A57FA0 ?
9FFFFFFFBF4C8A00 ?
C000000000002858 ?
4000000001857620 ?
000008E25 ?
9FFFFFFFFFFF1AD0 ?
9FFFFFFFBF558FA0 ?
9FFFFFFFBF4C88D4 ?
opipls()+3488 call opiexe() 000000004 ?
9FFFFFFFFFFF2F50 ?
9FFFFFFFFFFF1D30 ?
C000000000002C60 ?
40000000019852D0 ?
000000000 ? 000007123 ?
9FFFFFFFFFFF1B40 ?
opiodr()+3088 call opipls() 9FFFFFFFFFFF2F50 ?
9FFFFFFFFFFF2F58 ?
000000004 ?
C00000000000214A ?
40000000018747B0 ?
00000E123 ?
9FFFFFFFFFFF2E70 ?
9FFFFFFFFFFF2DE0 ?
rpidrus()+304 call opiodr() 9FFFFFFFFFFF3810 ?
9FFFFFFFFFFF37E0 ?
9FFFFFFFFFFF36F8 ?
9FFFFFFFFFFF36F4 ?
skgmstack()+288 call rpidrus() 9FFFFFFFFFFF4DA0 ?
C000000000000716 ?
4000000001862FE0 ?
9FFFFFFFFFFF47F0 ?
600000000004F3C0 ?
9FFFFFFFFFFF4DB8 ?
6000000000524900 ?
9FFFFFFFFFFF4DA0 ?
rpidru()+256 call skgmstack() 9FFFFFFFFFFF4DC0 ?
600000000004EDF0 ?
00000F618 ?
400000000129E400 ?
9FFFFFFFFFFF4DA0 ?
rpiswu2()+784 call rpidru() 00000057B ?
C000000000001329 ?
4000000001870CE0 ?
rpidrv()+2096 call rpiswu2() C00000007D7B3708 ?
9FFFFFFFFFFF5510 ?
6000000000515CE8 ?
60000000000502E8 ?
9FFFFFFFFFFF5520 ?
000000205 ?
6000000000530010 ?
6000000000531090 ?
psddr0()+256 call rpidrv() 000000000 ? 000000066 ?
9FFFFFFFFFFF6630 ?
00000003A ?
psdnal()+672 call psddr0() 000000000 ? 000000066 ?
9FFFFFFFFFFF6630 ?
000000030 ?
pevm_EXIM()+400 call psdnal() 9FFFFFFFFFFF84A0 ?
9FFFFFFFFFFF8440 ?
6000000000531A04 ?
600000000064BAA0 ?
C000000090880CC8 ?
600000000063EA80 ?
pfrrun()+37120 call pevm_EXIM() 9FFFFFFFBF567AD8 ?
C000000090880CC8 ?
C000000000001532 ?
400000000173DAD0 ?
000006269 ?
4000000002190B40 ?
9FFFFFFFBF4BA068 ?
9FFFFFFFBF5C08A0 ?
peicnt()+544 call pfrrun() 9FFFFFFFBF567CDC ?
C000000000000D1D ?
4000000001676FE0 ?
kkxexe()+832 call peicnt() 9FFFFFFFFFFF84A0 ?
9FFFFFFFBF567AD8 ?
C000000000000818 ?
40000000021A3A80 ?
00000FEAB ?
9FFFFFFFFFFF7EB0 ?
9FFFFFFFBF567CDC ?
000000000 ?
opiexe()+12592 call kkxexe() 9FFFFFFFFFFF8430 ?
C000000000002858 ?
4000000001858C30 ?
0000080AB ?
9FFFFFFFFFFF8420 ?
9FFFFFFFBF4F3A6C ?
9FFFFFFFBF4F3A68 ?
9FFFFFFFFFFF84A0 ?
opiall0()+3456 call opiexe() 000000004 ?
9FFFFFFFFFFF9A80 ?
9FFFFFFFFFFF8830 ?
C0000000000024D1 ?
40000000017A3350 ?
000000000 ? 00000C82B ?
9FFFFFFFFFFF8640 ?
kpoal8()+2272 call opiall0() 000000002 ?
9FFFFFFFFFFF9A10 ?
9FFFFFFFFFFFA2B4 ?
000000000 ?
9FFFFFFFFFFF9954 ?
9FFFFFFFFFFFA2BC ?
000000000 ?
FFFFFFFFFFFFDFFF ?
opiodr()+3088 call kpoal8() 9FFFFFFFFFFFA0B0 ?
000000000 ? 000000000 ?
C00000000000214A ?
40000000018747B0 ?
00000F82F ?
9FFFFFFFFFFFA130 ?
9FFFFFFFFFFFA1D0 ?
ttcpip()+1888 call opiodr() 9FFFFFFFFFFFA460 ?
9FFFFFFFFFFFA430 ?
9FFFFFFFFFFFA348 ?
9FFFFFFFFFFFA344 ?
C0000000000018B9 ?
400000000172CB70 ?
000000000 ? 00000C0AB ?
opitsk()+1920 call ttcpip() 6000000000052C40 ?
000000002 ?
9FFFFFFFFFFFB950 ?
6000000000273AA0 ?
9FFFFFFFFFFFBAB0 ?
9FFFFFFFFFFFB8C4 ?
9FFFFFFFBF75A660 ?
4000000000D2C7AA ?
opiino()+2656 call opitsk() 000000000 ? 000000000 ?
C000000000000D1F ?
400000000236CE90 ?
000000000 ?
opiodr()+3088 call opiino() 60000000005DEC18 ?
6000000000531170 ?
9FFFFFFFFFFFF4C0 ?
C00000000000214A ?
40000000018747B0 ?
00000E825 ?
9FFFFFFFFFFFD7C0 ?
9FFFFFFFFFFFF4D0 ?
opidrv()+1088 call opiodr() 9FFFFFFFFFFFDF10 ?
9FFFFFFFFFFFDEE0 ?
9FFFFFFFFFFFDDF8 ?
9FFFFFFFFFFFDDF4 ?
sou2o()+48 call opidrv() 9FFFFFFFFFFFEF60 ?
000000004 ?
9FFFFFFFFFFFF4C0 ?
main()+352 call sou2o() 9FFFFFFFFFFFF4E0 ?
00000003C ? 000000004 ?
9FFFFFFFFFFFF4C0 ?
main_opd_entry()+80 call main() 000000000 ?
9FFFFFFFFFFFF9B0 ?
C000000000000004 ?
C00000000002FA60 ?
--------------------- Binary Stack Dump ---------------------
观察节点2，发现类此错误不同之处有
Wed Mar 7 08:19:11 2012 Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc: ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] [] Wed Mar 7 08:19:12 2012 Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc: ORA-00600: internal error code, arguments: [kghfrempty:ds], [0x9FFFFFFFBF55A550], [], [], [], [], [], [] ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] [] Wed Mar 7 08:19:13 2012 Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc: ORA-07445: exception encountered: core dump [kgidmp()+1568] [SIGSEGV] [Address not mapped to object] [0xB38F00000000D9] [] [] ORA-00600: internal error code, arguments: [kghfrempty:ds], [0x9FFFFFFFBF55A550], [], [], [], [], [], [] ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] [] Wed Mar 7 08:19:14 2012 Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc: ORA-07445: exception encountered: core dump [kghalf()+993] [SIGSEGV] [Address not mapped to object] [0xC0000000A8B01DA0] [] [] ORA-07445: exception encountered: core dump [kgidmp()+1568] [SIGSEGV] [Address not mapped to object] [0xB38F00000000D9] [] [] ORA-00600: internal error code, arguments: [kghfrempty:ds], [0x9FFFFFFFBF55A550], [], [], [], [], [], [] ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []
对应trace文件不同之处
*** 2012-03-07 08:19:11.582 ksedmp: internal or fatal error ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] [] No current SQL statement being executed. ----- PL/SQL Call Stack ----- object line object handle number name c000000090c4a818 1356 procedure PMIS.PM_DC_SJJGGC_QDX c0000000a25512c0 43 procedure PMIS.PM_DC_SJJGGC_EXECINF c00000009a8306e8 2 anonymous block
通过这两个节点的错误记录，都包含PMIS.PM_DC_SJJGGC_EXECINF和anonymous，所以初步怀疑是某个程序在这个时间定运行某个任务(包含PMIS.PM_DC_SJJGGC_EXECINF过程)导致。
继续观察trace日志发现两个节点都是类此情况相同的程序，相同的主机
SO: c00000007d7b3708, type: 4, owner: c00000007d6d0190, flag: INIT/-/-/0x00
(session) trans: c000000082974470, creator: c00000007d6d0190, flag: (100041) USR/- BSY/-/-/-/-/-
DID: 0001-002A-0000693D, short-term DID: 0000-0000-00000000
txn branch: 0000000000000000
oct: 6, prv: 0, sql: c0000000a229ab38, psql: c000000084e6b510, user: 517/PMIS
O/S info: user: Administrator, term: DMIS_SWITCH2, ospid: 5600:3020, machine: WORKGROUP\DMIS_SWITCH2
program: PMTDCtrl.exe
application name: PMTDCtrl.exe, hash value=0
last wait for 'enqueue' blocking sess=0x0 seq=32849 wait_time=127
name|mode=54540004, id1=1, id2=10
temporary object counter: 0
现在更加怀疑是PMTDCtrl.exe运行某过程导致数据库出现这些错误
错误总结
查看MOS发现该问题是一个bug导致(Bug 6166690:ORA-7445 [MILLI_MEMCPY] WHEN REF CURSOR AS OUT PARAMETER)
错误原因:It is caused by a Dynamic SQL used to obtain a REF CURSOR from a procedure (that opens the cursor).
处理建议:通过个bug，我们可以分析PMIS.PM_DC_SJJGGC_EXECINF过程，看看是否符合bug描述,如果符合建议改写过程或者打上补丁(Patch 2709343)
补丁:p2709343_92080_HPUX-IA64
Bug 6166690: ORA-7445 [MILLI_MEMCPY] WHEN REF CURSOR AS OUT PARAMETER