今天检查数据库发现一数据库因为添加数据文件导致ORA-00600[3689]错误，进入引起MRP进程异常,从而使得数据库可以正常接收归档日志,但是不能在备库应用归档日志
数据库版本
SQL> select * from v$version;
BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bi
PL/SQL Release 10.2.0.1.0 - Production
CORE 10.2.0.1.0 Production
TNS for Solaris: Version 10.2.0.1.0 - Production
NLSRTL Version 10.2.0.1.0 – Production
检查日志
Fri Oct 19 12:00:22 2012
RFS[1]: No standby redo logfiles created
RFS[1]: Archived Log: '/baceldba/archive/1_30374_730819916.dbf'
Fri Oct 19 12:00:28 2012
Media Recovery Log /baceldba/archive/1_30374_730819916.dbf
WARNING: File being created with same name as in Primary
Existing file may be overwritten
Fri Oct 19 12:00:50 2012
Recovery created file /data/oracle/oradata/vodapp/NETTVCLPS02.dbf
Successfully added datafile 32 to media recovery
Datafile #32: '/data/oracle/oradata/vodapp/NETTVCLPS02.dbf '
WARNING: File being created with same name as in Primary
Existing file may be overwritten
Fri Oct 19 12:01:15 2012
Recovery created file /data/oracle/oradata/vodapp/UCICMS01_DATA04.dbf
Successfully added datafile 33 to media recovery
Datafile #33: '/data/oracle/oradata/vodapp/UCICMS01_DATA04.dbf'
Fri Oct 19 12:01:16 2012
Errors in file /data/oracle/admin/vodapp/bdump/vodapp_mrp0_21304.trc:
ORA-00600: internal error code, arguments: [3689], [33], [], [], [], [], [], []
Errors with log /baceldba/archive/1_30374_730819916.dbf
MRP0: Background Media Recovery terminated with error 600 <--mrp进程异常
Fri Oct 19 12:01:16 2012
Errors in file /data/oracle/admin/vodapp/bdump/vodapp_mrp0_21304.trc:
ORA-00600: internal error code, arguments: [3689], [33], [], [], [], [], [], []
Some recovered datafiles maybe left media fuzzy
Media recovery may continue but open resetlogs may fail
Fri Oct 19 12:01:18 2012
Errors in file /data/oracle/admin/vodapp/bdump/vodapp_mrp0_21304.trc:
ORA-00600: internal error code, arguments: [3689], [33], [], [], [], [], [], []
Fri Oct 19 12:01:18 2012
Errors in file /data/oracle/admin/vodapp/bdump/vodapp_mrp0_21304.trc:
ORA-00600: internal error code, arguments: [3689], [33], [], [], [], [], [], []
通过观察日志发现，数据库的mrp0进程因为ora-600的错误导致异常终止，从而使得dg备库无法正常使用归档日志
分析trace文件得出
*** 2012-10-19 12:01:16.327
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [3689], [33], [], [], [], [], [], []
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp()+744 CALL ksedst() 000000840 ?
FFFFFFFF7FFFA08C ?
000000000 ?
FFFFFFFF7FFF6B80 ?
FFFFFFFF7FFF58E8 ?
FFFFFFFF7FFF62E8 ?
kgeriv()+220 PTR_CALL 0000000000000000 000106000 ? 106323304 ?
106323000 ? 000106323 ?
000106000 ? 106323304 ?
kgesiv()+112 CALL kgeriv() 10631DCD0 ? 000000000 ?
000000E69 ? 000000001 ?
FFFFFFFF7FFFA5E8 ?
000001430 ?
ksesic1()+96 CALL kgesiv() 10631DCD0 ? 10646AD00 ?
000000E69 ? 000000001 ?
FFFFFFFF7FFFA5E8 ?
000001420 ?
kcvadc1_10gr2()+265 CALL ksesic1() 000000E69 ? 00010631D ?
2 1063232F8 ? 000106000 ?
106323000 ? 000106323 ?
kcoapl()+5664 PTR_CALL 0000000000000000 FFFFFFFF7A5570F8 ?
FFFFFFFF7FFFB320 ?
FFFFFFFF76667E18 ?
000000000 ? 000000021 ?
000000020 ?
kcramr()+5352 CALL kcoapl() FFFFFFFF76667E00 ?
000000003 ? 1055136B0 ?
105513340 ? 00000001B ?
000000017 ?
krddmr()+2688 CALL kcramr() 000000000 ?
FFFFFFFF7A55FF78 ?
000002000 ? 000106000 ?
0000001F6 ? 464897868 ?
krsmdp()+2332 CALL krddmr() 105502000 ?
FFFFFFFF7A5570F8 ?
000000001 ? 10646C760 ?
464897868 ? 000000000 ?
ksbrdp()+896 PTR_CALL 0000000000000000 000380000 ? 000380000 ?
105517000 ? 000106328 ?
000106000 ? 00000FFFF ?
opirip()+824 CALL ksbrdp() 000106320 ? 380007734 ?
000380000 ? 00038000E ?
000106000 ? 1005DCF80 ?
opidrv()+1200 CALL opirip() 10632A000 ? 000106000 ?
000106332 ? 380007000 ?
00010632A ? 1063D73A0 ?
sou2o()+80 CALL opidrv() 10632D460 ? 000000001 ?
000000032 ? 000000000 ?
000000032 ? 000106000 ?
opimai_real()+268 CALL sou2o() FFFFFFFF7FFFF968 ?
000000032 ? 000000004 ?
FFFFFFFF7FFFF990 ?
105C1F000 ? 000105C1F ?
main()+152 CALL opimai_real() 000000003 ?
FFFFFFFF7FFFFA68 ?
000000000 ? 000000000 ?
0023F9764 ? 000014400 ?
_start()+380 CALL main() 000000001 ?
FFFFFFFF7FFFFB78 ?
000000000 ?
FFFFFFFF7FFFFA70 ?
FFFFFFFF7FFFFB80 ?
FFFFFFFF7CF00200 ?
--------------------- Binary Stack Dump ---------------------
查询mos发现
Bug 5230162 : ORA-600[3689][42] ON PHYSICAL STANDBY PREVENT MEDIA RECOVERY描述相符
处理建议
1.10.2.0.1数据库不稳定，bug较多，建议升级数据库到10.2.0.4或者10.2.0.5
2.当前standby_file_management为true，建议修改为false,每次手工两边增加数据文件
3.主库添加数据文件后，观察备库的mrp进程是否正常，如果异常
1)尝试启动mrp进程，如果因为新增加的数据文件导致异常，那先使用rman或者其他方法拷贝主库新建立数据文件到备库，然后启动mrp进程
2)如果因为某种原因导致归档日志出现gap，可以从备份中找出归档日志，如果备份中没有，那可以使用基于scn的备份方式来修复该故障。
3)如果scn相差较大，建议直接重建备库
