kgebse – Database SOS

raid故障恢复之后,数据库recover成功,但是open报ORA-03113: end-of-file on communication channel错误

SQL> recover database;
Media recovery complete.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 20394
Session ID: 191 Serial number: 3

对应的alert日志错误为ORA-600 [kkkicreatecgmap:!efn3]错误

ALTER DATABASE RECOVER  database
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 4 slaves
Mon Oct 20 18:51:06 2025
Recovery of Online Redo Log: Thread 1 Group 1 Seq 32119 Reading mem 0
  Mem# 0: /u01/oradata/redo01.log
Media Recovery Complete (orcl)
Completed: ALTER DATABASE RECOVER  database
Mon Oct 20 18:51:16 2025
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 3 processes
Started redo scan
Completed redo scan
 read 41 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 32119, block 34978
Recovery of Online Redo Log: Thread 1 Group 1 Seq 32119 Reading mem 0
  Mem# 0: /u01/oradata/redo01.log
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 32119, block 35061, scn 17375938230308
 0 data blocks read, 0 data blocks written, 41 redo k-bytes read
Mon Oct 20 18:51:16 2025
Thread 1 advanced to log sequence 32120 (thread open)
Thread 1 opened at log sequence 32120
  Current log# 2 seq# 32120 mem# 0: /u01/oradata/redo02.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Oct 20 18:51:16 2025
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Re-creating tempfile /u01/oradata/temp01.dbf
Database Characterset is ZHS16GBK
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x3999DC33][PC:0x2297750, kgegpa()+40][flags: 0x0, count: 1]
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x3999DC33][PC:0x229597B, kgebse()+279][flags: 0x2, count: 2]
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x3999DC33][PC:0x229597B, kgebse()+279][flags: 0x2, count: 2]
No Resource Manager plan active
Mon Oct 20 18:51:16 2025
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20286.trc  (incident=3649):
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_3649/orcl_dbrm_20286_i3649.trc
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20286.trc:
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
DBRM (ospid: 20286): terminating the instance due to error 56710
Instance terminated by DBRM, pid = 20286

对应的trace文件内容


----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
skdstdst()+36        call     kgdsdst()            000000000 ? 000000000 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   7FFD08773E78 ? 000000000 ?
ksedst1()+98         call     skdstdst()           000000000 ? 000000000 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
ksedst()+34          call     ksedst1()            000000000 ? 000000001 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbkedDefDump()+2736  call     ksedst()             000000000 ? 000000001 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
ksedmp()+36          call     dbkedDefDump()       000000003 ? 000000002 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
ksfdmp()+64          call     ksedmp()             000000003 ? 000000002 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbgexPhaseII()+1764  call     ksfdmp()             000000003 ? 000000002 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbgexProcessError()  call     dbgexPhaseII()       7F16CCF3E6F0 ? 7F16CA6F2598 ?
+2279                                              7FFD0877BC68 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbgeExecuteForError  call     dbgexProcessError()  7F16CCF3E6F0 ? 7F16CA6F2598 ?
()+83                                              000000001 ? 000000000 ?
                                                   7FFD00000000 ? 000000000 ?
dbgePostErrorKGE()+  call     dbgeExecuteForError  7F16CCF3E6F0 ? 7F16CA6F2598 ?
1615                          ()                   000000001 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbkePostKGE_kgsf()+  call     dbgePostErrorKGE()   000000000 ? 7F16CA560040 ?
63                                                 000000258 ? 7F16CA6F2598 ?
                                                   000000000 ? 000000000 ?
kgeadse()+383        call     dbkePostKGE_kgsf()   00A99D360 ? 7F16CA560040 ?
                                                   000000258 ? 7F16CA6F2598 ?
                                                   000000000 ? 000000000 ?
kgerinv_internal()+  call     kgeadse()            00A99D360 ? 7F16CA560040 ?
45                                                 000000258 ? 000000000 ?
                                                   000000000 ? 000000000 ?
kgerinv()+33         call     kgerinv_internal()   00A99D360 ? 7F16CA560040 ?
                                                   877420000000000 ? 000000258 ?
                                                   000000000 ? 000000000 ?
kgeasnmierr()+143    call     kgerinv()            00A99D360 ? 7F16CA560040 ?
                                                   877420000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
kkkicreatecgmap()+3  call     kgeasnmierr()        00A99D360 ? 7F16CA560040 ?
125                                                877420000000000 ? 000000000 ?
                                                   000000000 ? 00000057B ?
kskirefreshcgmap()+  call     kkkicreatecgmap()    121DC3050 ? 7F16CA560040 ?
104                                                877420000000000 ? 000000000 ?
                                                   000000000 ? 00000057B ?
kskreload()+1150     call     kskirefreshcgmap()   121DC3050 ? 7F16CA560040 ?
                                                   877420000000000 ? 000000000 ?
                                                   000000000 ? 00000057B ?
kskdbrmpa()+378      call     kskreload()          7FFD0877D428 ? 000000001 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
ksbabs()+465         call     kskdbrmpa()          7FFD0877D418 ? 000000058 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
ksbrdp()+923         call     ksbabs()             7FFD0877D418 ? 000000058 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
opirip()+618         call     ksbrdp()             7FFD0877D418 ? 000000058 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
opidrv()+598         call     opirip()             000000032 ? 000000004 ?
                                                   7FFD0877E598 ? 000000001 ?
                                                   000000000 ? 00000020A ?
sou2o()+98           call     opidrv()             000000032 ? 000000004 ?
                                                   7FFD0877E598 ? 000000001 ?
                                                   000000000 ? 00000020A ?
opimai_real()+261    call     sou2o()              7FFD0877E570 ? 000000032 ?
                                                   000000004 ? 7FFD0877E598 ?
                                                   000000000 ? 00000020A ?
ssthrdmain()+209     call     opimai_real()        000000000 ? 7FFD0877E760 ?
                                                   000000004 ? 7FFD0877E598 ?
                                                   000000000 ? 00000020A ?
main()+196           call     ssthrdmain()         000000003 ? 7FFD0877E760 ?
                                                   000000001 ? 000000000 ?
                                                   000000000 ? 00000020A ?
__libc_start_main()  call     main()               000000003 ? 7FFD0877E900 ?
+253                                               000000001 ? 000000000 ?
                                                   000000000 ? 00000020A ?
_start()+36          call     __libc_start_main()  0009D3D94 ? 000000001 ?
                                                   7FFD0877E8F8 ? 000000000 ?
                                                   000000000 ? 00000020A ?

虽然ORA-600 kkkicreatecgmap:!efn3没有见过但是数据库open过程中kgegpa、kgebse的故障还是遇到不少,大部分可能和undo有一定关系,处理undo问题之后,继续尝试open库,依旧报ORA-03113: end-of-file on communication channel

SQL> alter database Open;
ERROR:
ORA-03113: end-of-file on communication channel
Process ID: 20586
Session ID: 191 Serial number: 3

但是后台的alert日志已经改变ORA-600 4193,ORA-600 kkkicreatecgmap:!efn3错误

Mon Oct 20 18:54:39 2025
Thread 1 advanced to log sequence 32121 (thread open)
Thread 1 opened at log sequence 32121
  Current log# 3 seq# 32121 mem# 0: /u01/oradata/redo03.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Oct 20 18:54:39 2025
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_smon_20492.trc  (incident=4905):
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_4905/orcl_smon_20492_i4905.trc
No Resource Manager plan active
Mon Oct 20 18:54:39 2025
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20478.trc  (incident=4849):
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_4849/orcl_dbrm_20478_i4849.trc
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20478.trc:
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
DBRM (ospid: 20478): terminating the instance due to error 56710
Some DDE async actions failed or were cancelled
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_smon_20492.trc  (incident=4906):
ORA-00353: log corruption near block 8 change 17375938190767 time 10/03/2025 00:20:34
ORA-00312: online log 1 thread 1: '/u01/oradata/redo01.log'
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_4906/orcl_smon_20492_i4906.trc
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_smon_20492.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 8 change 17375938190767 time 10/03/2025 00:20:34
ORA-00312: online log 1 thread 1: '/u01/oradata/redo01.log'
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Instance terminated by DBRM, pid = 20478

虽然ORA-600 kkkicreatecgmap:!efn3还在,但是看到了比较熟悉的ORA-600 4193错误,处理undo异常回滚段,数据库open成功,重建undo,尝试导出数据,完成数据恢复任务.

补充说明,对于ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403]网络上没有任何资料,查询了mos发现一个有一点类似的报错信息
Bug 28167557 – bigscn_dbim_tm_def – trc – kkkdchkcriticalobj – ORA-700 [kkkdchkcriticalob (Doc ID 28167557.8)

Description
Information about new symptoms:
 
- Signaling function: kkkdchkcriticalobj (kkkd.c)
- Symptom: ORA-700 [kkkdchkcriticalobj:fail]
- Owner: SUELEE
- Special Run:  BIGSCN_DBIM_TM_DEF (owner: WAI-SZE.TAM, sr_id: 2709)
- Release: 19.1
- Job id: 22488605
- Base label:  RDBMS_MAIN_LINUX.X64_180607
 
ORA-600 [KKKICHKRMAPPRI:0=NR PEND] 
 ORA-600 kkkicreatecgmap:!group
 ORA-700 kkkdchkcriticalobj
  
 ORA-600 [KKKICHKRMAPPRI:0=NR PEND] 
 ORA-600 kkkicreatecgmap:!group
 ORA-700 kkkdchkcriticalobj
  
 
 REDISCOVERY INFORMATION:
 Symptoms are any of the following:
 ORA-600 [KKKICHKRMAPPRI:0=NR PEND] 
 ORA-600 kkkicreatecgmap:!group
 ORA-700 kkkdchkcriticalobj
  
 .
 WORKAROUND:
 None

由于这个是硬件故障恢复出来的数据文件（涉及磁盘坏道，磁盘顺序，磁盘在raid中均衡,raid的cache等因素可能会一起Oracle各种非常规问题）,分析原因意义不大,重点是快速解决问题,不做过多分析

Sat Sep 20 20:31:14 2025 QMNC started with pid=39, OS id=10637 Completed: ALTER DATABASE OPEN Starting background process CJQ0 Sat Sep 20 20:31:14 2025 CJQ0 started with pid=44, OS id=10654 Setting Resource Manager plan SCHEDULER[0x318E]:DEFAULT_MAINTENANCE_PLAN via scheduler window Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter Starting background process VKRM Sat Sep 20 20:31:17 2025 VKRM started with pid=40, OS id=10680 Sat Sep 20 20:38:01 2025 Starting background process SMCO Sat Sep 20 20:38:01 2025 SMCO started with pid=38, OS id=10955 Sat Sep 20 20:56:54 2025 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_j000_11564.trc (incident=148368): ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Block recovery from logseq 1082886, block 29263 to scn 74449804596 Recovery of Online Redo Log: Thread 1 Group 14 Seq 1082886 Reading mem 0 Mem# 0: /oracledb/oradata/orcl/redo14.log Block recovery stopped at EOT rba 1082886.29264.16 Block recovery completed at rba 1082886.29264.16, scn 17.1435360559 Block recovery from logseq 1082886, block 29263 to scn 74449804590 Recovery of Online Redo Log: Thread 1 Group 14 Seq 1082886 Reading mem 0 Mem# 0: /oracledb/oradata/orcl/redo14.log Block recovery completed at rba 1082886.29264.16, scn 17.1435360559 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_j000_11564.trc: ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] Sat Sep 20 20:56:58 2025 Sweep [inc][148368]: completed Sweep [inc2][148368]: completed Sat Sep 20 21:00:20 2025 Exception[type:SIGSEGV,Address not mapped to object][ADDR:0xBC44AC1][PC:0x932F8EA,kgegpa()+40][flags:0x0,count:1] Exception[type:SIGSEGV,Address not mapped to object][ADDR:0xBC44AC1][PC:0x932DEF3,kgebse()+771][flags:0x2,count:2] Exception[type:SIGSEGV,Address not mapped to object][ADDR:0xBC44AC1][PC:0x932DEF3,kgebse()+771][flags:0x2,count:2] Sat Sep 20 21:00:21 2025 Block recovery from logseq 1082886, block 29263 to scn 74449804596 Recovery of Online Redo Log: Thread 1 Group 14 Seq 1082886 Reading mem 0 Mem# 0: /oracledb/oradata/orcl/redo14.log …………………… Sat Sep 20 21:05:00 2025 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_10516.trc(incident=148296): ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Block recovery from logseq 1082886, block 32045 to scn 74449805729 Recovery of Online Redo Log: Thread 1 Group 14 Seq 1082886 Reading mem 0 Mem# 0: /oracledb/oradata/orcl/redo14.log Block recovery completed at rba 1082886.32056.16, scn 17.1435361698 Block recovery from logseq 1082886, block 32045 to scn 74449806046 Recovery of Online Redo Log: Thread 1 Group 14 Seq 1082886 Reading mem 0 Mem# 0: /oracledb/oradata/orcl/redo14.log Block recovery completed at rba 1082886.32321.16, scn 17.1435362015 Non-fatal internal error happenned while SMON was doing flushing of monitored table stats. SMON encountered 1 out of maximum 100 non-fatal internal errors. ******* Block recovery completed at rba 1082898.52054.16, scn 17.1444838013 Non-fatal internal error happenned while SMON was doing flushing of monitored table stats. SMON encountered 99 out of maximum 100 non-fatal internal errors. Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_10516.trc(incident=164458): ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Block recovery from logseq 1082898, block 52038 to scn 74459282045 Recovery of Online Redo Log: Thread 1 Group 8 Seq 1082898 Reading mem 0 Mem# 0: /oracledb/oradata/orcl/redo08.log Block recovery completed at rba 1082898.52054.16, scn 17.1444838014 Block recovery from logseq 1082898, block 52038 to scn 74459282088 Recovery of Online Redo Log: Thread 1 Group 8 Seq 1082898 Reading mem 0 Mem# 0: /oracledb/oradata/orcl/redo08.log Block recovery completed at rba 1082898.52104.16, scn 17.1444838057 Non-fatal internal error happenned while SMON was doing flushing of monitored table stats. SMON encountered 100 out of maximum 100 non-fatal internal errors. Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_10516.trc (incident=164459): ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Block recovery from logseq 1082898, block 52038 to scn 74459282045 Recovery of Online Redo Log: Thread 1 Group 8 Seq 1082898 Reading mem 0 Mem# 0: /oracledb/oradata/orcl/redo08.log Block recovery completed at rba 1082898.52054.16, scn 17.1444838014 Block recovery from logseq 1082898, block 52038 to scn 74459282101 Recovery of Online Redo Log: Thread 1 Group 8 Seq 1082898 Reading mem 0 Mem# 0: /oracledb/oradata/orcl/redo08.log Block recovery completed at rba 1082898.52130.16, scn 17.1444838070 Non-fatal internal error happenned while SMON was doing flushing of monitored table stats. SMON exceeded the maximum limit of 100 internal error(s). Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_10516.trc: ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] SMON (ospid: 10516): terminating the instance due to error 474 Mon Sep 22 04:05:28 2025 System state dump requested by(instance=1,osid=10516 (SMON)),summary=[abnormal instance termination]. System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_diag_10463.trc Instance terminated by SMON, pid = 10516

Mon Sep 22 09:00:03 2025 ALTER DATABASE OPEN Beginning crash recovery of 1 threads parallel recovery started with 32 processes Started redo scan Completed redo scan read 1360 KB redo, 405 data blocks need recovery Started redo application at Thread 1: logseq 1082898, block 49410 Recovery of Online Redo Log: Thread 1 Group 8 Seq 1082898 Reading mem 0 Mem# 0: /oracledb/oradata/orcl/redo08.log Completed redo application of 0.60MB Completed crash recovery at Thread 1: logseq 1082898, block 52130, scn 74459302102 405 data blocks read, 405 data blocks written, 1360 redo k-bytes read Thread 1 advanced to log sequence 1082899 (thread open) Thread 1 opened at log sequence 1082899 Current log# 9 seq# 1082899 mem# 0: /oracledb/oradata/orcl/redo09.log Successful open of redo thread 1 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set SMON: enabling cache recovery Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_78465.trc (incident=164779): ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] Block recovery from logseq 1082899, block 3 to scn 74459302111 Recovery of Online Redo Log: Thread 1 Group 9 Seq 1082899 Reading mem 0 Mem# 0: /oracledb/oradata/orcl/redo09.log Block recovery stopped at EOT rba 1082899.5.16 Block recovery completed at rba 1082899.5.16, scn 17.1444858077 Block recovery from logseq 1082899, block 3 to scn 74459302108 Recovery of Online Redo Log: Thread 1 Group 9 Seq 1082899 Reading mem 0 Mem# 0: /oracledb/oradata/orcl/redo09.log Block recovery completed at rba 1082899.5.16, scn 17.1444858077 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_78465.trc: ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_78465.trc: ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], [] Error 600 happened during db open, shutting down database USER (ospid: 78465): terminating the instance due to error 600 Instance terminated by USER, pid = 78465 ORA-1092 signalled during: ALTER DATABASE OPEN... opiodr aborting process unknown ospid (78465) as a result of ORA-1092 Mon Sep 22 09:00:08 2025 ORA-1092 : opitsk aborting process

Mon Sep 22 19:14:14 2025 ALTER DATABASE MOUNT USER (ospid: 11679): terminating the instance System state dump requested by (instance=1, osid=11679), summary=[abnormal instance termination]. System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_diag_11615.trc Dumping diagnostic data in directory=[cdmp_20250922191419],requested by(instance=1,osid=11679) ,summary=[abnormal instance termination]. Instance terminated by USER, pid = 11679

分类目录归档：kgebse