2018年9月 – Database SOS

ORA-00600 dbkif_find_next_record_1

数据库版本信息

SQL> select * from v$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
PL/SQL Release 11.2.0.1.0 - Production
CORE	11.2.0.1.0	Production
TNS for 64-bit Windows: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production

数据库启动报ORA-00600 dbkif_find_next_record_1错误

alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 3 processes
Started redo scan
Completed redo scan
 read 133 KB redo, 72 data blocks need recovery
Started redo application at
 Thread 1: logseq 49070, block 14720
Recovery of Online Redo Log: Thread 1 Group 2 Seq 49070 Reading mem 0
  Mem# 0: D:\APP\xff\ORADATA\ORCL\REDO02.LOG
Completed redo application of 0.09MB
Errors in file d:\app\xff\diag\rdbms\orcl\orcl\trace\orcl_ora_2340.trc  (incident=477756):
ORA-00600: ??????, ??: [dbkif_find_next_record_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\app\xff\diag\rdbms\orcl\orcl\incident\incdir_477756\orcl_ora_2340_i477756.trc
Wed Sep 26 10:52:07 2018
Trace dumping is performing id=[cdmp_20180926105207]
Wed Sep 26 10:52:08 2018
Aborting crash recovery due to error 600
Errors in file d:\app\xff\diag\rdbms\orcl\orcl\trace\orcl_ora_2340.trc:
ORA-00600: ??????, ??: [dbkif_find_next_record_1], [], [], [], [], [], [], [], [], [], [], []
Errors in file d:\app\xff\diag\rdbms\orcl\orcl\trace\orcl_ora_2340.trc:
ORA-00600: ??????, ??: [dbkif_find_next_record_1], [], [], [], [], [], [], [], [], [], [], []
ORA-600 signalled during: alter database open...

这里比较明显，数据库是在做实例恢复的时候遭遇到ORA-600 dbkif_find_next_record_1错误,无法正常应用日志导致该错误,具体原因由于:the NAB and finds it is less than the block# from the on-disk-RBA recorded，对于这种问题，通过人工修改next available block# in the Online Redo Log的相关记录,即可正常open数据库,而且理论上数据0丢失

ORA-07445 qcdlgcd

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-07445 qcdlgcd

SQL执行报ORA-03113错误

SQL> SELECT *
  FROM USER.TABLE_NAME t
 WHERE t.YEAR = '2019'
   AND t.UPPCODE = '51010000'
   AND  t.MONTH = '01';
  2    3    4    5
SELECT *
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 68389
Session ID: 2419 Serial number: 34370

数据库版本等信息

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
ORACLE_HOME = /oracle/product/db12cr1
System name:	Linux
Node name:	Tkfcsdb
Release:	2.6.32-431.el6.x86_64
Version:	#1 SMP Sun Nov 10 22:19:54 EST 2013
Machine:	x86_64

alert日志报错

Thu Sep 27 17:41:15 2018
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x4] [PC:0xCDFB0C6, qcdlgcd()+70] [flags: 0x0, count: 1]
Errors in file /oracle/diag/rdbms/tst12uf/tst12uf/trace/tst12uf_ora_234433.trc  (incident=126129) (PDBNAME=PTST12UF):
ORA-07445:exception encountered:core dump[qcdlgcd()+70][SIGSEGV][ADDR:0x4][PC:0xCDFB0C6][Address not mapped to object][]
Incident details in: /oracle/diag/rdbms/tst12uf/tst12uf/incident/incdir_126129/tst12uf_ora_234433_i126129.trc
Use ADRCI or Support Workbench to package the incident.

trace文件

----- Beginning of Customized Incident Dump(s) -----
Dumping swap information
Memory (Avail / Total) = 948.73M / 63736.63M
Swap (Avail / Total) = 30025.52M /  32255.99M
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x4] [PC:0xCDFB0C6, qcdlgcd()+70] [flags: 0x0, count: 1]
Registers:
%rax: 0x0000000000000000 %rbx: 0x000000020c3da3f0 %rcx: 0x00007fc492268a40
%rdx: 0x00007fc49239eab0 %rdi: 0x000000022cd5da20 %rsi: 0x00007fc492f2bc80
%rsp: 0x00007fffbd9c9430 %rbp: 0x00007fffbd9c94a0  %r8: 0x00000002372efcb8
 %r9: 0x000000022cd5da20 %r10: 0x00007fc49239eab0 %r11: 0x00007fc492268a40
%r12: 0x000000022cd5da20 %r13: 0x00007fc492f2bc80 %r14: 0x00007fc49239eab0
%r15: 0x00007fc492268a40 %rip: 0x000000000cdfb0c6 %efl: 0x0000000000010206
  qcdlgcd()+53 (0xcdfb0b5) mov %rdi,%r12
  qcdlgcd()+56 (0xcdfb0b8) mov 0x10(%rax),%rbx
  qcdlgcd()+60 (0xcdfb0bc) jz 0xcdfb29e
  qcdlgcd()+66 (0xcdfb0c2) mov 0x60(%r15),%rax
> qcdlgcd()+70 (0xcdfb0c6) movzwl 0x4(%rax),%r8d
  qcdlgcd()+75 (0xcdfb0cb) cmp $30,%r8d
  qcdlgcd()+79 (0xcdfb0cf) jnle 0xcdfb277
  qcdlgcd()+85 (0xcdfb0d5) pxor %xmm0,%xmm0
  qcdlgcd()+89 (0xcdfb0d9) movaps %xmm0,-0x40(%rbp)
*** 2018-09-27 17:41:15.110
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x3, level=3, mask=0x0)
[TOC00004]
----- Current SQL Statement for this session (sql_id=52chysuyh36t4) -----
SELECT /* DS_SVC */ /*+ dynamic_sampling(0) no_sql_tune
no_monitoring optimizer_features_enable(default)
no_parallel result_cache(snapshot=3600) */
SUM(C1) FROM (SELECT /*+ qb_name("innerQuery")
NO_INDEX_FFS( "T")  */ 1 AS C1 FROM "USER"."TABLE_NAME" "T"
WHERE ("T"."MONTH"='01') AND ("T"."UPPCODE"='51010000') AND ("T"."YEAR"='2019')) innerQuery
[TOC00004-END]
[TOC00005]
----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
skdstdst()+45        call     kgdsdst()            7FC492BAB678 000000003
                                                   7FC492B8D0B0 ? 7FC492B8D1C8 ?
                                                   7FC492BAAEA8 ? 000000083 ?
ksedst()+119         call     skdstdst()           7FC492BAB678 000000001
                                                   000000001 7FC492B8D1C8 ?
                                                   7FC492BAAEA8 ? 000000083 ?
dbkedDefDump()+1119  call     ksedst()             000000001 000000001 ?
                                                   000000001 ? 7FC492B8D1C8 ?
                                                   7FC492BAAEA8 ? 000000083 ?
ksedmp()+261         call     dbkedDefDump()       000000003 000000003
                                                   000000001 ? 7FC492B8D1C8 ?
                                                   7FC492BAAEA8 ? 000000083 ?
ssexhd()+2650        call     ksedmp()             00000044F 000000003 ?
                                                   000000001 ? 000000003
                                                   7FC492BAAEA8 ? 000000083 ?
sslsshandler()+456   call     ssexhd()             202F206C69617641
                                                   3D20296C61746F54
                                                   4D33372E38343920
                                                   3633373336202F20
                                                   6177530A4D33362E 000000083 ?
__sighandler()       call     sslsshandler()       000002000 000000000 000000000
                                                   3633373336202F20 ?
                                                   6177530A4D33362E ?
                                                   000000083 ?
qcdlgcd()+70         signal   __sighandler()       22CD5DA20 7FC492F2BC80
                                                   7FC49239EAB0 7FC492268A40 ?
                                                   2372EFCB8 ? 22CD5DA20 ?
kkdlgcd()+118        call     qcdlgcd()            22CD5DA20 ? 7FC492F2BC80 ?
                                                   7FC49239EAB0 ? 7FC492268A40 ?
                                                   2372EFCB8 ? 22CD5DA20 ?
kkmfbtic()+17        call     kkdlgcd()            7FC49239EAB0 ? 7FC492268A40 ?
                                                   7FC49239EAB0 ? 7FC492268A40 ?
                                                   2372EFCB8 ? 22CD5DA20 ?
qcsgcic()+163        call     kkmfbtic()           7FC49239EAB0 ? 7FC492268A40 ?
                                                   7FC49239EAB0 ? 7FC492268A40 ?
                                                   2372EFCB8 ? 22CD5DA20 ?
kkmgkc()+147         call     qcsgcic()            22CD5DA20 ? 7FC492F2BC80 ?
                                                   7FC49239EAB0 ? 7FC492268A40 ?
                                                   000000007 22CD5DA20 ?
kokscit()+65         call     kkmgkc()             22CD5DA20 ? 7FC492F2BC80 ?
                                                   7FC49239EAB0 ? 7FC492268A40 ?
                                                   000000007 ? 22CD5DA20 ?
qkebCreateColById()  call     kokscit()            22CD5DA20 ? 7FC492F2BC80 ?
+298                                               7FC49239EAB0 ? 7FC492268A40 ?
                                                   000000007 ? 22CD5DA20 ?
qksvcGetGuardCol()+  call     qkebCreateColById()  000000002 7FC4923A4078
161                                                7FC49239EAB0 ? 000000007 ?
                                                   7FC492268A40 ? 22CD5DA20 ?
qksvcProcessVCColum  call     qksvcGetGuardCol()   7FC49239D9D8 ? 7FC4923A4078 ?
ns()+1625                                          7FC49239EAB0 ? 000000007 ?
                                                   7FC492268A40 ? 22CD5DA20 ?
qkacol()+405         call     qksvcProcessVCColum  000000000 ? 000000000 ?
                              ns()                 7FC49239EAB0 ? 000000007 ?
                                                   7FC492268A40 ? 22CD5DA20 ?
qkadrv()+933         call     qkacol()             7FC49239EAB0 ? 000000000 ?
                                                   7FC49239EAB0 ? 000000007 ?
                                                   7FC492268A40 ? 22CD5DA20 ?
opitca()+2417        call     qkadrv()             7FC4923A4078 ? 000000001 ?
                                                   7FC49239EAB0 ? 000000007 ?
                                                   7FC492268A40 ? 22CD5DA20 ?
kksFullTypeCheck()+  call     opitca()             7FC4922F4BD0 22CD5DC40
79                                                 7FFFBD9CF3B0 000000007 ?
                                                   7FC492268A40 ? 22CD5DA20 ?
rpiswu2()+2235       call     kksFullTypeCheck()   7FFFBD9CDF08 ? 22CD5DC40 ?
                                                   7FFFBD9CF3B0 ? 000000007 ?
                                                   7FC492268A40 ? 22CD5DA20 ?
kksLoadChild()+7590  call     rpiswu2()            7FFFBD9CDF08 ? 22CD5DC40 ?
                                                   7FFFBD9CF3B0 ? 000000008 ?
                                                   7FC492F5D260 ? 22CD5DA20 ?
kxsGetRuntimeLock()  call     kksLoadChild()       7FFFBD9CDF08 ? 22CD5DC40 ?
+2155                                              000000000 000000008 ?
                                                   7FC492F5D260 ? 22CD5DA20 ?
kksfbc()+14306       call     kxsGetRuntimeLock()  7FC492F2BC80 7FC4922F4BD0
                                                   7FFFBD9CF330 22D1E7C28 ?
                                                   7FC492F5D260 ? 22D1E7C28
kkspsc0()+3146       call     kksfbc()             7FC4922F4BD0 7FC4922F4BD0 ?
                                                   7FFFBD9CF330 ? 22D1E7C28 ?
                                                   7FC492F5D260 ? 22D1E7C28 ?
kksParseCursor()+11  call     kkspsc0()            7FC4928321A0 7FC49235D198
8                                                  00000015F 000000003 000000006
                                                   000000020
opiosq0()+2210       call     kksParseCursor()     7FFFBD9D0178 ? 7FC49235D198 ?
                                                   00000015F ? 000000003 ?
                                                   000000006 ? 000000020 ?
opiall0()+4530       call     opiosq0()            000000003 7FC49235D198 ?
                                                   7FC492F5D260 ? 000000020
                                                   000000000 000000020 ?
opikpr()+567         call     opiall0()            000000003 ? 000000022
                                                   7FFFBD9D0A50 000000000
                                                   000000000 ? 000000020 ?
opiodr()+1165        call     opikpr()             000000065 ? 000000022 ?
                                                   7FFFBD9D22F0 000000000 ?
                                                   000000000 ? 000000020 ?
rpidrus()+206        call     opiodr()             000000065 00000001F
                                                   7FFFBD9D22F0 ? 000000000 ?
                                                   000000000 ? 100000001
skgmstack()+144      call     rpidrus()            7FFFBD9D1B70 00000001F ?
                                                   7FC492F2BE78 000000000 ?
                                                   000000000 ? 100000001 ?
rpiswu2()+723        call     skgmstack()          7FFFBD9D1B48 ? 7FC492F2B7A0 ?
                                                   00000F618 ? 00CBFE570 ?
                                                   7FFFBD9D1B70 ? 100000001 ?
kprball()+1163       call     rpiswu2()            7FFFBD9D1B48 ? 7FC492F2B7A0 ?
                                                   00000F618 ? 000000002 ?
                                                   7FC492F5D260 ? 100000001 ?
qksdsExeStmt()+2411  call     kprball()            7FFFBD9D22F0 004000180
                                                   00000F618 ? 000000002 ?
                                                   7FC492F5D260 ? 100000001 ?
qksdsExecute()+959   call     qksdsExeStmt()       7FFFBD9D2B98 7FC49235D150
                                                   000000001 000000008
                                                   7FFFBD9D28C8 100000001 ?
kkoatVerifyEst()+30  call     qksdsExecute()       7FFFBD9D2B98 7FC49235D150 ?
85                                                 000000001 ? 000000008 ?
                                                   7FFFBD9D28C8 ? 100000001 ?
kkeAdjSingTabCard()  call     kkoatVerifyEst()     300000001 7FC4923E9DA8
+714                                               1EE909D75846 7FC49235D108
                                                   000000000 100000001 ?
kkecdn()+3803        call     kkeAdjSingTabCard()  7FC4923E9DA8 7FC4923EA328
                                                   7FFFBD9D2E30 ? 7FC49235D108 ?
                                                   000000000 ? 100000001 ?
kkotap()+13019       call     kkecdn()             7FC4923ECF08 000000003 ?
                                                   7FFFBD9D2E30 ? 7FC4923EA328 ?
                                                   000000000 ? 100000001 ?
kkoiqb()+8331        call     kkotap()             7FC492F5D260 ? 000000000 ?
                                                   000000011 000000000 000000000
                                                   100000001 ?
kkooqb()+532         call     kkoiqb()             000000000 ? 000000000
                                                   000000000 000000000 ?
                                                   000000000 ? 100000001 ?
kkoqbc()+2385        call     kkooqb()             7FC4927F8BC0 ? 000000006
                                                   000000000 ? 000000000 ?
                                                   000000000 ? 100000001 ?
apakkoqb()+182       call     kkoqbc()             7FC4927F8BC0 ? 7FC4927F8BC0 ?
                                                   000000001 000000000 ?
                                                   000000000 ? 100000001 ?
apaqbdDescendents()  call     apakkoqb()           000000000 ? 7FC4927F8BC0 ?
+488                                               21EE306B8 ? 000000000 ?
                                                   000000000 ? 100000001 ?
apadrv()+5383        call     apaqbdDescendents()  7FFFBD9DC2B0 ? 7FC4927F8BC0 ?
                                                   21EE306B8 ? 000000000 ?
                                                   000000000 ? 100000001 ?
opitca()+2106        call     apadrv()             21EE306B8 ? 7FC4927F8BC0 ?
                                                   21EE306B8 ? 000000000 ?
                                                   000000000 ? 100000001 ?
kksLoadChild()+7318  call     opitca()             7FC492845648 21EE306B8
                                                   7FFFBD9DE6F0 000000000 ?
                                                   000000000 ? 100000001 ?
kxsGetRuntimeLock()  call     kksLoadChild()       7FC492845648 ? 21EE306B8 ?
+2155                                              000000000 000000000 ?
                                                   000000000 ? 100000001 ?
kksfbc()+14306       call     kxsGetRuntimeLock()  7FC492F2BC80 7FC492845648
                                                   7FFFBD9DE670 232AE3E98 ?
                                                   000000000 ? 232AE3E98
kkspsc0()+3146       call     kksfbc()             7FC492845648 7FC492845648 ?
                                                   7FFFBD9DE670 ? 232AE3E98 ?
                                                   000000000 ? 232AE3E98 ?
kksParseCursor()+11  call     kkspsc0()            7FC492832108 7FFFBD9E0A28
8                                                  000000081 000000003 000000006
                                                   0000000A4
opiosq0()+2210       call     kksParseCursor()     7FFFBD9DF4B8 ? 7FFFBD9E0A28 ?
                                                   000000081 ? 000000003 ?
                                                   000000006 ? 0000000A4 ?
kpoal8()+1223        call     opiosq0()            000000003 7FFFBD9E0A28 ?
                                                   7FC492F5D260 ? 0000000A4
                                                   000000000 0000000A4 ?
opiodr()+1165        call     kpoal8()             00000005E 00000001F
                                                   7FFFBD9E3278 0000000A4 ?
                                                   000000000 ? 0000000A4 ?
ttcpip()+2699        call     opiodr()             00000005E 00000001F
                                                   7FFFBD9E3278 ? 0000000A4 ?
                                                   000000000 ? 100000000
opitsk()+1734        call     ttcpip()             7FC492F41070 ? 00000005E ?
                                                   7FFFBD9E3278 000000000 ?
                                                   7FFFBD9E2CD8 7FFFBD9E3484
opiino()+945         call     opitsk()             000000400 000000000
                                                   7FFFBD9E3278 ? 000000000 ?
                                                   7FFFBD9E2CD8 ? 7FFFBD9E3484 ?
opiodr()+1165        call     opiino()             00000003C 000000004
                                                   7FFFBD9E4918 000000000 ?
                                                   7FFFBD9E2CD8 ? 7FFFBD9E3484 ?
opidrv()+587         call     opiodr()             00000003C 000000004
                                                   7FFFBD9E4918 ? 000000000 ?
                                                   7FFFBD9E2CD8 ? 000000000
sou2o()+145          call     opidrv()             00000003C 000000004
                                                   7FFFBD9E4918 000000000 ?
                                                   7FFFBD9E2CD8 ? 000000000 ?
opimai_real()+154    call     sou2o()              7FFFBD9E48F0 00000003C
                                                   000000004 7FFFBD9E4918
                                                   7FFFBD9E2CD8 ? 000000000 ?
ssthrdmain()+412     call     opimai_real()        000000000 7FFFBD9E4C00
                                                   000000004 ? 7FFFBD9E4918 ?
                                                   7FFFBD9E2CD8 ? 000000000 ?
main()+236           call     ssthrdmain()         000000000 000000002
                                                   7FFFBD9E4C00 000000001
                                                   000000000 000000000 ?
__libc_start_main()  call     main()               7FFFBD9E559F 7FFFBD9E55AD
+253                                               7FFFBD9E4C00 ? 000000001 ?
                                                   000000000 ? 000000000 ?
_start()+41          call     __libc_start_main()  000BBD640 000000002
                                                   7FFFBD9E4E48 000000000 ?
                                                   000000000 ? 000000000 ?
[TOC00005-END]
[TOC00006]
--------------------- Binary Stack Dump ---------------------

解决方法
这里比较明显,我们可以发现这个sql执行报错，其实本质是由于递归执行动态采样报错而引起异常的.通过设置_fix_control来规避这个问题

SQL> alter session set "_fix_control"='14191778:0';
Session altered.
SQL> SELECT *
  FROM USER.TABLE_NAME t
 WHERE t.YEAR = '2019'
   AND t.UPPCODE = '51010000'
   AND  t.MONTH = '01';
  2    3    4    5
	ID EMPCODE		COMCODE 	     YEAR	QUARTER
---------- -------------------- -------------------- ---------- ----------
PUSHRATIO  D A CREATECODE	    UPDATECODE		 CREATEDATE
---------- - - -------------------- -------------------- -------------------
UPDATEDATE	    UPPCODE		 CLAIM		      MONTH
------------------- -------------------- -------------------- ----------
     77813 5101000166		51010702	     2019	1
50	   0 0 1777					 2018-07-18 17:44:19
		    51010000		 50		      01
     77909 8000545245		51010702	     2019	1
0	   0 0 1777		    1777		 2018-07-18 17:35:12
2018-07-18 17:41:20 51010000		 100		      01
	ID EMPCODE		COMCODE 	     YEAR	QUARTER
---------- -------------------- -------------------- ---------- ----------
PUSHRATIO  D A CREATECODE	    UPDATECODE		 CREATEDATE
---------- - - -------------------- -------------------- -------------------
UPDATEDATE	    UPPCODE		 CLAIM		      MONTH
------------------- -------------------- -------------------- ----------
     77912 5101000047		51010702	     2019	1
0	   0 0 1777		    1777		 2018-07-18 17:41:03
2018-07-18 17:44:28 51010000		 100		      01
SQL> SQL>

ORA-600 kddummy_blkchk导致数据库无法open

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 kddummy_blkchk导致数据库无法open

数据库启动报ORA-600 kddummy_blkchk

Sun Sep  9 21:45:28 2018
SMON: enabling tx recovery
Sun Sep  9 21:45:28 2018
Database Characterset is ZHS16GBK
Opening with internal Resource Manager plan
where NUMA PG = 1, CPUs = 24
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=35, OS id=53346452
Sun Sep  9 21:45:28 2018
Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_smon_39714888.trc:
ORA-00600: internal error code, arguments: [kddummy_blkchk], [609], [6], [18018], [], [], [], []
Sun Sep  9 21:45:28 2018
Completed: ALTER DATABASE OPEN
Sun Sep  9 21:45:28 2018
db_recovery_file_dest_size of 89900 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Sun Sep  9 21:45:29 2018
Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_smon_39714888.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kddummy_blkchk], [609], [6], [18018], [], [], [], []
Sun Sep  9 21:45:32 2018
Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_smon_39714888.trc:
ORA-00600: internal error code, arguments: [kddummy_blkchk], [609], [6], [18018], [], [], [], []
Sun Sep  9 21:46:20 2018
Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_pmon_11403488.trc:
ORA-00474: SMON process terminated with error
Sun Sep  9 21:46:20 2018
PMON: terminating instance due to error 474
Instance terminated by PMON, pid = 11403488

kddummy_blkchk错误mos记录

  Format: ORA-600 [kddummy_blkchk] [a] [b] 
VERSIONS:
  versions 9.2 to 10.2
DESCRIPTION:
  --what condition caused the error to be reported
ARGUMENTS:
  Arg [a] Absolute file number
  Arg [b] Bock number
  Arg  Internal error code returned from kcbchk() which indicates the problem encountered.

trace文件信息

Block after image:
buffer tsn: 14 rdba: 0x98400006 (609/6)
scn: 0x0007.9e675f6d seq: 0x01 flg: 0x04 tail: 0x5f6d1e01
frmt: 0x02 chkval: 0x61ae type: 0x1e=KTFB Bitmapped File Space Bitmap
Hex dump of corrupt header 3 = CHKVAL
Dump of memory from 0x070000043F034000 to 0x070000043F034014
*** 2018-09-09 21:45:28.531
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [kddummy_blkchk], [609], [6], [18018], [], [], [], []
----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedst+001c          bl       ksedst1              90000000032FBDC ? 000000000 ?
ksedmp+0290          bl       ksedst               104A2CDD0 ?
ksfdmp+0018          bl       03F26B04
kgerinv+00dc         bl       _ptrgl
kseinpre+0040        bl       kgerinv              110123BE0 ? 000000001 ?
                                                   104BD0420 ? 07FFFFFFF ?
                                                   000000000 ?
ksesin+0048          bl       kseinpre             104BD0420 ? 07FFFFFFF ?
                                                   000000000 ?
kco_blkchk+0798      bl       ksesin               104BD0888 ? 300000003 ?
                                                   000000000 ? 000000261 ?
                                                   000000000 ? 000000006 ?
                                                   000000000 ? 000004662 ?
kcoapl+0d24          bl       kco_blkchk           000000000 ? 70000046D293500 ?
                                                   FFFFFFFFFFF9BA0 ? 1101954D0 ?
                                                   FFFFFFFFFFF9AC0 ?
kcbapl+0174          bl       kcoapl               FFFFFFFFFFFAFD0 ?
                                                   70000043F034000 ? 1104C2884 ?
                                                   EDEADBEEF ? 200000000000 ?
                                                   110000790 ? 000000000 ?
kcrfw_redo_gen+2988  bl       kcbapl               FFFFFFFFFFFA040 ?
                                                   70000046D293500 ? 100FAF764 ?
                                                   700000010008000 ? 000000000 ?
kcbchg1_main+25d4    bl       kcrfw_redo_gen       000000000 ? 000000000 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 000000002 ?
kcbchg1+038c         bl       kcbchg1_main         000000000 ? 000000000 ?
                                                   000000018 ? 000000000 ?
                                                   000000000 ? 000000228 ?
ktbchgro+0380        bl       kcbchg1              062886020 ? 1601C0F50 ?
                                                   FFFFFFFFFFFAE80 ?
                                                   FFFFFFFFFFFAEB8 ? 000000000 ?
                                                   000000000 ?
ktfbbset+01e0        bl       ktbchgro             000000000 ? 100000001 ?
                                                   FFFFFFFFFFFB110 ?
                                                   FFFFFFFFFFFAFD0 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   000000001 ?
ktfbfset+039c        bl       ktfbbset             000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   700000010018078 ? 000000000 ?
                                                   110000790 ?
ktfbfundo+0208       bl       ktfbfset             00000000B ? 00000000B ?
ktfbhget+0760        bl       ktfbfundo            FFFFFFFFFFFBFF8 ? 200000000 ?
ktfbffpre+00d0       bl       ktfbhget             FFFFFFFFFFFBFF8 ? 200000002 ?
                                                   1100000000 ? 2033FBE8014 ?
kteopdelete+12dc     bl       ktfbffpre            FFFFFFFFFFFC678 ? 204BCDD80 ?
                                                   1CE04BCEEE8 ? 0FFFFEFFF ?
ktsxfastdele+0124    bl       kteopdelete          700000010018078 ?
                                                   700000010008000 ? 1104772A8 ?
                                                   700000464E35278 ?
                                                   700000464E35278 ? 000000000 ?
                                                   FFFFFFFFFFFBDA0 ?
kteopshrink+0308     bl       _ptrgl
ktssdrbm_segment+0a  bl       kteopshrink          E0000000E ? FFFFFFFFFFFC608 ?
b0                                                 000000001 ? 000000001 ?
                                                   110477768 ? 000000002 ?
                                                   1100EA4E0 ?
ktssdro_segment+06d  bl       ktssdrbm_segment     FFFFFFFFFFFCFC8 ?
c                                                  FFFFFFFFFFFD090 ? 100000010 ?
                                                   000000000 ?
ktssdt_segs+03f4     bl       ktssdro_segment      70000046D28EA18 ? 6FFFFD3A0 ?
                                                   0FFFFD320 ?
ktmmon+135c          bl       ktssdt_segs          000000000 ?
                                                   7FFFFFFF7FFFFFFF ?
                                                   7FFFFFFF7FFFFFFF ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ?
                                                   7FFFFFFC7FFFFFFC ?
                                                   05B9523F8 ?
ktmSmonMain+0030     bl       ktmmon               110000790 ?
ksbrdp+04b4          bl       _ptrgl
opirip+03fc          bl       03F26C94
opidrv+0458          bl       opirip               11029F970 ? 4102A12B0 ?
                                                   FFFFFFFFFFFF6B0 ?
sou2o+0090           bl       opidrv               3202D99A1C ? 4A004A628 ?
                                                   FFFFFFFFFFFF6B0 ?
opimai_real+0150     bl       01F93134
main+0098            bl       opimai_real          4500000000000000 ?
                                                   800200140000400 ?
__start+0070         bl       main                 000000000 ? 000000000 ?
--------------------- Binary Stack Dump ---------------------

错误比较明显由于file 609 block 6异常，导致smon无法进行回滚,从使得数据库open之后,立马crash.这个问题可以通过使用bbed修复坏块解决,也可以通过设置合适的参数和事件,禁止smon的一些操作来规避
bbed修复block之后dbv检查

oracle@p740a:/tmp]$ dbv file=/u01/app/oracle/oradata/orcl/xifenfei38.dbf
DBVERIFY: Release 10.2.0.4.0 - Production on Sun Sep 16 11:01:54 2018
Copyright (c) 1982, 2007, Oracle.  All rights reserved.
DBVERIFY - Verification starting : FILE = /u01/app/oracle/oradata/orcl/xifenfei38.dbf
DBVERIFY - Verification complete
Total Pages Examined         : 1856000
Total Pages Processed (Data) : 1142647
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 710956
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 2277
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 120
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Highest block SCN            : 2658994127 (7.2658994127)

ORA 600 kddummy_blkchk相关bug列表

私网直连后遗症:一节点无法启动导致另外节点haip无法启动

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：私网直连后遗症:一节点无法启动导致另外节点haip无法启动

该案例为两节点rac(11.2.0.4)，private 网络使用直连方式，其中一个节点主机异常无法启动，另外一个节点集群启动发现haip无法正常启动

# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
     1        ONLINE  ONLINE       xifenfei1                  Started
ora.cluster_interconnect.haip                                                      >>>>  OFFLINE
     1        ONLINE  OFFLINE
ora.crf
     1        ONLINE  ONLINE       xifenfei1
ora.crsd
     1        ONLINE  OFFLINE                                                      >>>>  OFFLINE
ora.cssd
     1        ONLINE  ONLINE       xifenfei1
ora.cssdmonitor
     1        ONLINE  ONLINE       xifenfei1
ora.ctssd
     1        ONLINE  ONLINE       xifenfei1                  OBSERVER
ora.diskmon
     1        OFFLINE OFFLINE
ora.drivers.acfs
     1        ONLINE  ONLINE       xifenfei1
ora.evmd
     1        ONLINE  INTERMEDIATE xifenfei1
ora.gipcd
     1        ONLINE  ONLINE       xifenfei1
ora.gpnpd
     1        ONLINE  ONLINE       xifenfei1
ora.mdnsd
     1        ONLINE  ONLINE       xifenfei1

alerthostname日志

2018-09-02 10:38:56.767:
[/u01/app/11.2.0/grid/bin/orarootagent.bin(7866)]CRS-5818:Aborted command 'start' for resource 'ora.cluster_interconnect.haip'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/11.2.0/grid/log/xifenfei1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2018-09-02 10:39:00.771:
[ohasd(7495)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cluster_interconnect.haip'. Details at (:CRSPE00111:) {0:0:2} in /u01/app/11.2.0/grid/log/xifenfei1/ohasd/ohasd.log.
2018-09-02 10:40:00.802:
[/u01/app/11.2.0/grid/bin/orarootagent.bin(7866)]CRS-5818:Aborted command 'start' for resource 'ora.cluster_interconnect.haip'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/11.2.0/grid/log/xifenfei1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2018-09-02 10:40:04.806:
[ohasd(7495)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.cluster_interconnect.haip'. Details at (:CRSPE00111:) {0:0:2} in /u01/app/11.2.0/grid/log/xifenfei1/ohasd/ohasd.log.

orarootagent_root日志

2018-09-02 10:37:56.805: [ USRTHRD][3650455296]{0:0:2} No HAIP info configured in GPNP, using defaults
2018-09-02 10:37:56.805: [ USRTHRD][3650455296]{0:0:2} The final CIDR subnet 169.254/16
2018-09-02 10:37:56.805: [ default][3650455296]clsvactversion:4: Retrieving Active Version from local storage.
2018-09-02 10:37:56.809: [ USRTHRD][3650455296]{0:0:2} HAIP: mbr num is 0.
[   CLWAL][3650455296]clsw_Initialize: OLR initlevel [70000]
2018-09-02 10:37:56.843: [ USRTHRD][3650455296]{0:0:2} HAIP: initializing to 1 interfaces
2018-09-02 10:37:56.844: [ USRTHRD][3650455296]{0:0:2} HAIP: configured to use 1 interfaces

gipcd.log日志

2018-09-02 10:38:56.787: [ CLSINET][2477147904] Returning NETDATA: 0 interfaces
2018-09-02 10:38:56.988: [GIPCDCLT][2477147904] gipcdClientInterfaceRequest: sent local interface list back to client
2018-09-02 10:38:56.822: [GIPCHDEM][2468742912] gipchaDaemonInfRequest: sent local interfaceRequest,  hctx 0x1369730 [0000000000000010] { gipchaContext : host 'xifenfei1', name 'gipcd_ha_name', luid '184dd356-00000000', numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2018-09-02 10:38:56.822: [GIPCDCLT][2477147904] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceRequest, endp 00000000000002cb
2018-09-02 10:38:56.822: [GIPCDCLT][2477147904] gipcdClientInterfaceRequest: Received type(gipcdmsgtypeInterfaceRequest), endp(00000000000002cb), len(1032), buf(0x7fab858b7a78):[hostname(xifenfei1), retStatus(gipcretSuccess)]
2018-09-02 10:38:56.822: [GIPCDCLT][2477147904] gipcdClientInterfaceQueryToMonitor: enqueue local interface query (2) to worklist
2018-09-02 10:38:56.823: [GIPCDCLT][2477147904] gipcdClientInterfaceRequest: sent local interface query
2018-09-02 10:38:56.823: [GIPCDMON][2472945408] gipcdMonitorCheckXfer: set new infQuery
2018-09-02 10:38:56.831: [ GIPCLIB][2477147904] gipclibSetTraceLevel: to set level to 0

ohasd.log日志

2018-09-02 10:38:52.494: [GIPCHDEM][1878710016]gipchaDaemonInfRequest: sent local interfaceRequest,  hctx 0x2749eb0 [0000000000000010] { gipchaContext : host 'xifenfei1', name 'CLSFRAME_oracler-cluster', luid '47624c02-00000000', numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2018-09-02 10:38:57.255: [    AGFW][3305629440]{0:0:2} Received the reply to the message: RESOURCE_START[ora.cluster_interconnect.haip 1 1] ID 4098:502 from the agent /u01/app/11.2.0/grid/bin/orarootagent_root
2018-09-02 10:38:57.255: [    AGFW][3305629440]{0:0:2} Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.cluster_interconnect.haip 1 1] ID 4098:500
2018-09-02 10:38:57.255: [   CRSPE][3295123200]{0:0:2} Received reply to action [Start] message ID: 500
2018-09-02 10:38:57.256: [   CRSPE][3295123200]{0:0:2} Got agent-specific msg: CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the following error:
Start action for HAIP aborted. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/xifenfei1/agent/ohasd/orarootagent_root/orarootagent_root.log".
2018-09-02 10:38:57.500: [GIPCHDEM][1878710016]gipchaDaemonInfRequest: sent local interfaceRequest,  hctx 0x2749eb0 [0000000000000010] { gipchaContext : host 'xifenfei1', name 'CLSFRAME_oracler-cluster', luid '47624c02-00000000', numNode 0, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd

检查私网状态,发现eth2网络链路状态为down,由于网络直连，而另外一台机器无法启动

[root@xifenfei1 rules.d]# ethtool eth1
Settings for eth1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Speed: Unknown!
        Duplex: Unknown! (255)
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: no   ====>网卡链路状态异常
[root@xifenfei1 rules.d]# ifconfig
eth0      Link encap:Ethernet  HWaddr 6C:92:BF:2B:7B:36
          inet addr:10.10.17.42  Bcast:172.17.17.255  Mask:255.255.255.0
          inet6 addr: fe80::6e92:bfff:fe2b:7b36/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1     --------->注意
          RX packets:234424 errors:0 dropped:0 overruns:0 frame:0
          TX packets:160916 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:16926236 (16.1 MiB)  TX bytes:24269882 (23.1 MiB)
          Memory:91160000-91180000
eth1      Link encap:Ethernet  HWaddr 6C:92:BF:2B:7B:37
          inet addr:11.1.1.2  Bcast:11.1.1.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1      --------->注意少了RUNNING
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Memory:91140000-91160000

关于网卡链路异常导致haip无法启动的mos描述请参考:CRSD & HAIP Resources Remain In OFFLINE as Private Network Interface is Partially Up (Doc ID 1529721.1).该案例是11.2集群私网使用直连引起的直接后遗症(非常不建议集群私网使用直连方式)

月度归档：9月 2018