ORA-00600[3689]引起MRP进程异常

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-00600[3689]引起MRP进程异常

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

今天检查数据库发现一数据库因为添加数据文件导致ORA-00600[3689]错误，进入引起MRP进程异常,从而使得数据库可以正常接收归档日志,但是不能在备库应用归档日志
数据库版本

SQL> select * from v$version;
BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bi
PL/SQL Release 10.2.0.1.0 - Production
CORE    10.2.0.1.0      Production
TNS for Solaris: Version 10.2.0.1.0 - Production
NLSRTL Version 10.2.0.1.0 – Production

检查日志

Fri Oct 19 12:00:22 2012
RFS[1]: No standby redo logfiles created
RFS[1]: Archived Log: '/baceldba/archive/1_30374_730819916.dbf'
Fri Oct 19 12:00:28 2012
Media Recovery Log /baceldba/archive/1_30374_730819916.dbf
WARNING: File being created with same name as in Primary
Existing file may be overwritten
Fri Oct 19 12:00:50 2012
Recovery created file /data/oracle/oradata/vodapp/NETTVCLPS02.dbf
Successfully added datafile 32 to media recovery
Datafile #32: '/data/oracle/oradata/vodapp/NETTVCLPS02.dbf '
WARNING: File being created with same name as in Primary
Existing file may be overwritten
Fri Oct 19 12:01:15 2012
Recovery created file /data/oracle/oradata/vodapp/UCICMS01_DATA04.dbf
Successfully added datafile 33 to media recovery
Datafile #33: '/data/oracle/oradata/vodapp/UCICMS01_DATA04.dbf'
Fri Oct 19 12:01:16 2012
Errors in file /data/oracle/admin/vodapp/bdump/vodapp_mrp0_21304.trc:
ORA-00600: internal error code, arguments: [3689], [33], [], [], [], [], [], []
Errors with log /baceldba/archive/1_30374_730819916.dbf
MRP0: Background Media Recovery terminated with error 600  <--mrp进程异常
Fri Oct 19 12:01:16 2012
Errors in file /data/oracle/admin/vodapp/bdump/vodapp_mrp0_21304.trc:
ORA-00600: internal error code, arguments: [3689], [33], [], [], [], [], [], []
Some recovered datafiles maybe left media fuzzy
Media recovery may continue but open resetlogs may fail
Fri Oct 19 12:01:18 2012
Errors in file /data/oracle/admin/vodapp/bdump/vodapp_mrp0_21304.trc:
ORA-00600: internal error code, arguments: [3689], [33], [], [], [], [], [], []
Fri Oct 19 12:01:18 2012
Errors in file /data/oracle/admin/vodapp/bdump/vodapp_mrp0_21304.trc:
ORA-00600: internal error code, arguments: [3689], [33], [], [], [], [], [], []

通过观察日志发现，数据库的mrp0进程因为ora-600的错误导致异常终止，从而使得dg备库无法正常使用归档日志

分析trace文件得出

*** 2012-10-19 12:01:16.327
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [3689], [33], [], [], [], [], [], []
----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp()+744         CALL     ksedst()             000000840 ?
                                                   FFFFFFFF7FFFA08C ?
                                                   000000000 ?
                                                   FFFFFFFF7FFF6B80 ?
                                                   FFFFFFFF7FFF58E8 ?
                                                   FFFFFFFF7FFF62E8 ?
kgeriv()+220         PTR_CALL 0000000000000000     000106000 ? 106323304 ?
                                                   106323000 ? 000106323 ?
                                                   000106000 ? 106323304 ?
kgesiv()+112         CALL     kgeriv()             10631DCD0 ? 000000000 ?
                                                   000000E69 ? 000000001 ?
                                                   FFFFFFFF7FFFA5E8 ?
                                                   000001430 ?
ksesic1()+96         CALL     kgesiv()             10631DCD0 ? 10646AD00 ?
                                                   000000E69 ? 000000001 ?
                                                   FFFFFFFF7FFFA5E8 ?
                                                   000001420 ?
kcvadc1_10gr2()+265  CALL     ksesic1()            000000E69 ? 00010631D ?
2                                                  1063232F8 ? 000106000 ?
                                                   106323000 ? 000106323 ?
kcoapl()+5664        PTR_CALL 0000000000000000     FFFFFFFF7A5570F8 ?
                                                   FFFFFFFF7FFFB320 ?
                                                   FFFFFFFF76667E18 ?
                                                   000000000 ? 000000021 ?
                                                   000000020 ?
kcramr()+5352        CALL     kcoapl()             FFFFFFFF76667E00 ?
                                                   000000003 ? 1055136B0 ?
                                                   105513340 ? 00000001B ?
                                                   000000017 ?
krddmr()+2688        CALL     kcramr()             000000000 ?
                                                   FFFFFFFF7A55FF78 ?
                                                   000002000 ? 000106000 ?
                                                   0000001F6 ? 464897868 ?
krsmdp()+2332        CALL     krddmr()             105502000 ?
                                                   FFFFFFFF7A5570F8 ?
                                                   000000001 ? 10646C760 ?
                                                   464897868 ? 000000000 ?
ksbrdp()+896         PTR_CALL 0000000000000000     000380000 ? 000380000 ?
                                                   105517000 ? 000106328 ?
                                                   000106000 ? 00000FFFF ?
opirip()+824         CALL     ksbrdp()             000106320 ? 380007734 ?
                                                   000380000 ? 00038000E ?
                                                   000106000 ? 1005DCF80 ?
opidrv()+1200        CALL     opirip()             10632A000 ? 000106000 ?
                                                   000106332 ? 380007000 ?
                                                   00010632A ? 1063D73A0 ?
sou2o()+80           CALL     opidrv()             10632D460 ? 000000001 ?
                                                   000000032 ? 000000000 ?
                                                   000000032 ? 000106000 ?
opimai_real()+268    CALL     sou2o()              FFFFFFFF7FFFF968 ?
                                                   000000032 ? 000000004 ?
                                                   FFFFFFFF7FFFF990 ?
                                                   105C1F000 ? 000105C1F ?
main()+152           CALL     opimai_real()        000000003 ?
                                                   FFFFFFFF7FFFFA68 ?
                                                   000000000 ? 000000000 ?
                                                   0023F9764 ? 000014400 ?
_start()+380         CALL     main()               000000001 ?
                                                   FFFFFFFF7FFFFB78 ?
                                                   000000000 ?
                                                   FFFFFFFF7FFFFA70 ?
                                                   FFFFFFFF7FFFFB80 ?
                                                   FFFFFFFF7CF00200 ?
--------------------- Binary Stack Dump ---------------------

查询mos发现
Bug 5230162 : ORA-600[3689][42] ON PHYSICAL STANDBY PREVENT MEDIA RECOVERY描述相符

处理建议
1.10.2.0.1数据库不稳定，bug较多，建议升级数据库到10.2.0.4或者10.2.0.5
2.当前standby_file_management为true，建议修改为false,每次手工两边增加数据文件
3.主库添加数据文件后，观察备库的mrp进程是否正常，如果异常
1)尝试启动mrp进程，如果因为新增加的数据文件导致异常，那先使用rman或者其他方法拷贝主库新建立数据文件到备库，然后启动mrp进程
2)如果因为某种原因导致归档日志出现gap，可以从备份中找出归档日志，如果备份中没有，那可以使用基于scn的备份方式来修复该故障。
3)如果scn相差较大，建议直接重建备库

One thought on “ORA-00600[3689]引起MRP进程异常”

  1. Bug 5230162 : ORA-600[3689][42] ON PHYSICAL STANDBY PREVENT MEDIA RECOVERY

    Hdr: 5230162 10.2.0.1 RDBMS 10.2.0.1 DICTIONARY PRODID-5 PORTID-226 ORA-600
Abstract: ORA-600[3689][42] ON PHYSICAL STANDBY PREVENT MEDIA RECOVERY
*** 05/16/06 07:17 pm ***
TAR:
----
PROBLEM:
--------
Physical standby media recovery mrp0 failed with
ORA-600[3689][42]
This cause physical standby stop recovery.
Ct performed add 3 datafiles to tablespace bilanpuissance around May 15
21:41:15. Also did resize of the datafiles. 13min later at May 15 21:58:14,
they drop those 3 datafiles and create another tablespace
BILANPUISSANCE_2006.
After that they did data load to above tablespaces.
But at May 16 08:15:57, both tablespaces are dropped.
On standby site, we can see datafiles are added, straight after that
ora-600[3689][42] is complained:
Mon May 15 22:02:09 2006
Successfully added datafile 42 to media recovery
Datafile #42: '+DATA/pp_stby/datafile/bilanpuissance.461.590536923'
Successfully added datafile 46 to media recovery
Datafile #46: '+DATA/pp_stby/datafile/bilanpuissance.462.590536929'
on May 15 22:05:06 2006
Successfully added datafile 47 to media recovery
Datafile #47: '+DATA/pp_stby/datafile/bilanpuissance.463.590537069'
Mon May 15 22:05:39 2006
Errors in file /opt/oracle/admin/coparppr/bdump/pp_stby_mrp0_3600.trc:
ORA-600: internal error code, arguments: [3689], [42], [], [], [], [], [],
[]
Primary Database are 2 nodes RAC, standby is single node. All are 10.2
It is currently happening on ct site.
they
DIAGNOSTIC ANALYSIS:
--------------------
ora-600[3689][42] is complaining about file #42 is already part of media
recovery, but we are trying to add a file with #42 again.
Try to offline drop file #42, #46 and #46 on standby. Not help.
Trying to recreate a standby controlfile from primary. But all datafile names
are different (OMF), too troublesome to alter all file name (on ASM), so this
approach is not tried.
WORKAROUND:
-----------
None
RELATED BUGS:
-------------
None. Only bug 2802219 has similar error but fixed in 10.1
REPRODUCIBILITY:
----------------
currently on ct site
TEST CASE:
----------
STACK TRACE:
------------
ksedst ksedmp ksfdmp kgeriv kgesiv ksesic1 kcvadc1_10gr2 kcvafc_10gr2 kcoapl
kcramr krddmr krsmdp ksbrdp opirip opidrv sou2o
SUPPORTING INFORMATION:
-----------------------
alert log - both primary and standby
pp_stby_mrp0_30732.trc - mrp0 trace. more mrp0 trace available in RDA
RDA.RDA_coprac4_20060516.zip - RDA of standby
24 HOUR CONTACT INFORMATION FOR P1 BUGS:
----------------------------------------
support contact for next 2 hour - Joei Xu +61 3 86164121 AIM: jwxuau
DIAL-IN INFORMATION:
--------------------
IMPACT DATE:
------------
now
    回复

发表评论

邮箱地址不会被公开。 必填项已用*标注

15 + 9 =