Oracle备份恢复 – Database SOS

obet快速修改scn/resetlogs恢复数据库(缺少归档,ORA-00308）

标题：obet快速修改scn/resetlogs恢复数据库(缺少归档,ORA-00308）

最初是由于业务系统报ORA-01578导致

然后相关人员上来进行了一系列操作,包括但不限于offline文件,create datafile 44,recover datafile,还尝试做resetlogs,使用备份控制文件还原等等.我接手该故障的时候状态

44号文件状态是12月1日的,而且resetlogs信息也不对.通过和客户沟通,确认是他们在没有备份44号文件的前提下直接执行了类似alter database create datafile 44的命令,但是在应用了写归档之后,发现提示有归档不存在

ALTER DATABASE RECOVER    CONTINUE DEFAULT
Media Recovery Log /data/app/oracle/archive/orcl/1_181285_1103629544.dbf
ORA-279 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER    CONTINUE DEFAULT
Media Recovery Log /data/app/oracle/archive/orcl/1_181286_1103629544.dbf
Tue Jan 06 12:48:12 2026
Errors with log /data/app/oracle/archive/orcl/1_181286_1103629544.dbf
Errors in file /data/app/oracle/diag/rdbms/orcl8/orcl/trace/orcl_pr00_8338.trc:
ORA-00308: cannot open archived log '/data/app/oracle/archive/orcl/1_181286_1103629544.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

分析现存归档,确实发现12月1日之后丢失了部分归档日志(归档不连续)

基于这种情况(故障数据文件直接被覆盖,归档又出现了多天的连续断档,而且没有有效备份),只能先打开数据库,然后根据情况导出数据,然后导入到新库中.这里我直接使用obet来修改resetlogs和checkpoint 信息,实现数据库快速打开

OBET> help
OBET (Oracle Block Editor Tool) commands:
  open <config_file>    - Load file list from config file (format: <num> <path>)
  info                  - Show loaded file list (from open command)
  set filename <path>   - Set target file path (required)
  set file <num>        - Set filename using loaded file number (from open list)
  set blocksize <size>  - Set block size (2048,4096,[8192],16384,32768)
  set block <num>       - Set block number (starts from 0, default: 1)
  set offset <offset>   - Set offset within block (< blocksize, default: 0)
  set count <bytes>     - Set number of bytes to read (default: 32)
  set mode edit/browse  - Enable edit/browse mode
  d/dump [options]      - Display data (options: block X, offset Y, count N)
  m/modify <hex> [opts] - Modify data with hex (opts: block X, offset Y)
  undo                  - Undo last modification
  sum [block X]         - Calculate checksum for block (default: current block)
  sum apply [block X]   - Apply checksum: write calculated value to block
  tailchk [block X]     - Calculate tailchk for block (default: current block)
  tailchk apply [block X] - Apply tailchk: write calculated value to block
  repair [block X]      - Repair bad block (fix seq_kcbh, tailchk, checksum)
  copy data <src> to <dest>  - Copy data between files
    <src> format: file,block,offset,count (e.g., 1,1,10,64)
    <dest> format: file[,block][,offset] (e.g., 3 or 3,1 or 3,1,128)
  copy block file#,block# to file#,block#     - Copy entire data block
  copy chkscn file n to file m         - Copy datafile header checkpoint SCN info
  copy resetlogscn file n to file m    - Copy datafile header resetlogs info
  backup [file n] [block m] - Backup data block (optional file and block parameters)
  corrupt [block X]     - Mark block as corrupted (default: current block)
  show                  - Display current settings (filename, blocksize, block, offset, count, mode)
  license               - Show/manage software license (registration code required)
  version               - Show software version and developer information
  p/print <param>       - Print Oracle structure,Use the 'p/print' command to see details
  undo                  - Undo the last copy chkscn or copy resetlogscn operation
  spool <file>          - Start logging to file (spool off to stop)
  help                  - Show this help message
  quit/exit             - Exit OBET

OBET> open 1.txt                 
Loaded 2 files from config file '1.txt'.

OBET> info

Loaded files (2 total):
----------------------------------------
Number  Path
----------------------------------------
     1  /data/app/oracle/oradata/orcl/system01.dbf
    44  /data/app/oracle/oradata/orcl/xff26.dbf
----------------------------------------

OBET> set file 44
filename set to: /data/app/oracle/oradata/orcl/xff26.dbf (file#44)

OBET> d

File: /data/app/oracle/oradata/orcl/xff26.dbf
Block: 1                Offsets:     0 to    31
--------------------------------------------------------------------------------
0xifenfei000 0BA20000 0100000B 00000000 00000104 B05F0000 00000000 0004200B 65583D61 

<32 bytes read>


OBET> backup file 44 block 1
Created backup directory: backup_blk
Successfully backed up block 1 from file#44 to /tmp/backup_blk/xff26.dbf_1.20260106182405

OBET> copy resetlogscn file 1 to file 44
Error: Edit mode not enabled. Use 'set mode edit' first.

OBET> set mode edit     
mode set to: edit

OBET> copy resetlogscn file 1 to file 44

Confirm Modify resetlogscn:
Source: file#1 (/data/app/oracle/oradata/orcl/system01.dbf)
Target: file#44 (/data/app/oracle/oradata/orcl/xff26.dbf)
Proceed? (Y/YES to confirm): y
Successfully copied resetlog SCN information from file#1 to file#44.

OBET> copy chkscn file 1 to file 44

Confirm Modify chkscn:
Source: file#1 (/data/app/oracle/oradata/orcl/system01.dbf)
Target: file#44 (/data/app/oracle/oradata/orcl/xff26.dbf)
Proceed? (Y/YES to confirm): y
Successfully copied checkpoint SCN information from file#1 to file#44.

OBET> tailchk
Check tailchk for File /data/app/oracle/oradata/orcl/xff26.dbf, Block 1:
current = 0x010B0000, required = 0x010B0000

OBET> sum
Check value for File /data/app/oracle/oradata/orcl/xff26.dbf, Block 1:
current = 0xB05F, required = 0x6DA3

OBET> sum apply

Confirm applying checksum:
File: /data/app/oracle/oradata/orcl/xff26.dbf
Block: 1
Offset in block: 16 (file offset: 0x0xifenfei010)
Original value: 0xB05F
New value:      0x6DA3
Confirm? (Y/YES to proceed): y
Verification successful: Stored checksum matches calculated value (0x6DA3).
Checksum applied successfully.

然后重建ctl正常打开数据库

[oracle@xifenfei check_db]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Tue Jan 6 18:29:22 2026

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area 6.8413E+10 bytes
Fixed Size		    2270360 bytes
Variable Size		 3.3554E+10 bytes
Database Buffers	 3.4762E+10 bytes
Redo Buffers		   93585408 bytes
Database mounted.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01190: control file or data file 1 is from before the last RESETLOGS
ORA-01110: data file 1: '/data/app/oracle/oradata/orcl/system01.dbf'


SQL> shutdown immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> 
SQL> 
SQL> startup nomount pfile='/tmp/pfile
';
ORACLE instance started.

Total System Global Area 6.8413E+10 bytes
Fixed Size		    2270360 bytes
Variable Size		 3.3554E+10 bytes
Database Buffers	 3.4762E+10 bytes
Redo Buffers		   93585408 bytes
SQL> @ctl.sql

Control file created.

SQL> SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-00264: no recovery required


SQL> alter database open;

Database altered.

后续根据情况选择导出表,对于导出报错的表再进行二次处理.

使用DBMS_PDB.RECOVER抢救单个pdb

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：使用DBMS_PDB.RECOVER抢救单个pdb

由于存储异常导致数据库启动报ORA-1172错误

尝试recover datafile 1 报ORA-600 kdBlkCheckError错误

Wed Jan 07 14:31:17 2026
Decreasing number of real time LMS from 5 to 0
ALTER DATABASE RECOVER  datafile 1
Wed Jan 07 14:31:21 2026
Media Recovery Start
Wed Jan 07 14:31:21 2026
Serial Media Recovery started
Wed Jan 07 14:31:21 2026
Recovery of Online Redo Log: Thread 2 Group 24 Seq 2584 Reading mem 0
  Mem# 0: +DATA/CDBDG/ONLINELOG/group_24.393.1122613933
  Mem# 1: +ARCH/CDBDG/ONLINELOG/group_24.276.1122613933
Wed Jan 07 14:31:21 2026
Recovery of Online Redo Log: Thread 1 Group 18 Seq 13823 Reading mem 0
  Mem# 0: +DATA/CDBDG/ONLINELOG/group_18.387.1122613923
  Mem# 1: +ARCH/CDBDG/ONLINELOG/group_18.270.1122613925
Wed Jan 07 14:31:22 2026
Recovery of Online Redo Log: Thread 1 Group 19 Seq 13824 Reading mem 0
  Mem# 0: +DATA/CDBDG/ONLINELOG/group_19.388.1122613925
  Mem# 1: +ARCH/CDBDG/ONLINELOG/group_19.271.1122613925
Errors in file /u01/app/oracle/diag/rdbms/cdbdg/cdb1/trace/cdb1_ora_2753010.trc (incident=806358) (PDBNAME=CDB$ROOT):
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [105538], [6401], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/cdbdg/cdb1/incident/incdir_806358/cdb1_ora_2753010_i806358.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Wed Jan 07 14:31:25 2026
Media Recovery failed with error 10562
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 1  ...

通过dbv检查file 1（system）数据文件,发现大量坏块

bash-4.2$ dbv file=system.271.1122584815

DBVERIFY: Release 12.1.0.2.0 - Production on Wed Jan 7 20:15:26 2026

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /data/bak/system.271.1122584815

DBV-00200: Block, DBA 4195214, already marked corrupt

DBV-00200: Block, DBA 4210332, already marked corrupt

DBV-00200: Block, DBA 4210333, already marked corrupt
……………………
DBV-00200: Block, DBA 4312889, already marked corrupt

DBV-00200: Block, DBA 4313119, already marked corrupt

DBV-00200: Block, DBA 4313122, already marked corrupt


DBVERIFY - Verification complete

Total Pages Examined         : 524288
Total Pages Processed (Data) : 79989
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 20781
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 408840
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 14678
Total Pages Marked Corrupt   : 33
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3135037281 (4374.3135037281)

分析发现这个库是包含了多个pdb,客户的业务主要在con_id=3的pdb中

bash-4.2$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Wed Jan 7 20:27:42 2026

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MOUNTED
         3 XFF                            MOUNTED
         4 DYCW                           MOUNTED
SQL>

尝试对除cdb的system/sysaux之外的文件进行recover操作均成功

对于这种情况,比较好的处理思路就是通过对单独的pdb文件使用DBMS_PDB.RECOVER处理,然后插入到一个新的cdb中,第一步使用dbca创建一个和当前库字符集一样的cdb

dbca -silent -ignorePreReqs -createDatabase -templateName $ORACLE_HOME/assistants/dbca/templates/General_Purpose.dbc \
-responseFile NO_VALUE \
-gdbname oratest  -sid oratest \
-createAsContainerDatabase TRUE \
-sysPassword oracle -systemPassword oracle -pdbAdminPassword oracle -dbsnmpPassword oracle \
-datafileDestination '/data/oradata' \
-redoLogFileSize 200 \
-characterset ZHS16GBK -nationalCharacterSet AL16UTF16 \
-totalMemory 4096 \
-databaseType MULTIPURPOSE \
-emConfiguration NONE

然后利用老的pdb生成需要的xml文件

BEGIN
DBMS_PDB.RECOVER(PDB_DESCR_FILE=> '/tmp/XFF.xml', PDB_NAME=>'XFF', 
FILENAMES=> '/data/bak/system.268.1122584817,
/data/bak/system.441.1125224053,
/data/bak/sysaux.269.1122584817,
…………
/data/bak/odin.312.1122593153,
/data/bak/dsg.314.1122593223');
END; 
/

利用该xml文件,直接把这个pdb插入到新cdb中

Wed Jan 07 14:40:52 2026
create pluggable database XFF using '/tmp/XFF.xml' nocopy
Wed Jan 07 14:41:29 2026
Session (221,18860): EXTERNAL SCN ALERT: Advanced SCN by 3185580 minutes worth to 0x1116.bade672c,
    by Pluggable Database plug-in
Session (221,18860): EXTERNAL SCN SOURCE: Inbound connection from client
Session (221,18860): EXTERNAL SCN SOURCE: DB Logon User: SYS, Remote Machine: p770a,
  Program: sqlplus@p770a (TNS V1-V3),OS User: oracle
****************************************************************
Pluggable Database XFF with pdb id - 3 is created as UNUSABLE.
If any errors are encountered before the pdb is marked as NEW,
then the pdb must be dropped
****************************************************************
Database Characterset for XFF is ZHS16GBK
Deleting old file#8 from file$ 
Deleting old file#9 from file$ 
Deleting old file#10 from file$ 
…………
Deleting old file#148 from file$ 
Deleting old file#199 from file$ 
Deleting old file#200 from file$ 
Adding new file#9 to file$(old file#8) 
Adding new file#8 to file$(old file#145) 
Adding new file#10 to file$(old file#9) 
…………
Adding new file#19 to file$(old file#148) 
Adding new file#18 to file$(old file#199) 
Adding new file#17 to file$(old file#200) 
Marking tablespace #2 invalid since it is not present   in the describe file 
Successfully created internal service XFF at open
ALTER SYSTEM: Flushing buffer cache inst=0 container=3 local
Wed Jan 07 14:41:49 2026
****************************************************************
Post plug operations are now complete.
Pluggable database XFF with pdb id - 3 is now marked as NEW.
****************************************************************
Completed: create pluggable database XFF using '/tmp/XFF.xml' nocopy
alter database open
Wed Jan 07 14:43:21 2026
Pluggable database XFF dictionary check beginning
Tablespace 'TEMP' #2 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Pluggable Database XFF Dictionary check complete
Database Characterset for XFF is ZHS16GBK
*********************************************************************
WARNING: The following temporary tablespaces in container(XFF)
         contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
***************************************************************
WARNING: Pluggable Database XFF with pdb id - 3 is
         altered with errors or warnings. Please look into
         PDB_PLUG_IN_VIOLATIONS view for more details.
***************************************************************
Opening pdb XFF (3) with no Resource Manager plan active
Pluggable database XFF opened read write
Completed: alter database open
alter tablespace temp add tempfile '/data/bak/temp01.dbf' size 16m autoextend on
Completed: alter tablespace temp add tempfile '/data/bak/temp01.dbf' size 16m autoextend on
ALTER PLUGGABLE DATABASE CLOSE IMMEDIATE
Wed Jan 07 14:46:08 2026
Immediate Kill Session#: 366, Serial#: 2995
Immediate Kill Session: sess: 7000100bafb5f58  OS pid: 5637924
ALTER SYSTEM: Flushing buffer cache inst=0 container=3 local
Wed Jan 07 14:46:22 2026
Pluggable database XFF closed
Completed: ALTER PLUGGABLE DATABASE CLOSE IMMEDIATE
alter database open
Wed Jan 07 14:46:48 2026
Database Characterset for XFF is ZHS16GBK
***************************************************************
WARNING: Pluggable Database XFF with pdb id - 3 is
         altered with errors or warnings. Please look into
         PDB_PLUG_IN_VIOLATIONS view for more details.
***************************************************************
Opening pdb XFF (3) with no Resource Manager plan active
Pluggable database XFF opened read write
Completed: alter database open

这次4T多的aix环境rac数据库的单个pdb通过DBMS_PDB.RECOVER方法非常完美的恢复,DBMS_PDB.RECOVER是拯救单个pdb的神兵利器.

数据库启动报 maximum number of processes () exceeded分析

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：数据库启动报 maximum number of processes () exceeded分析

比较少遇到打开一个数据库报ORA-00020: maximum number of processes () exceeded,导致open数据库失败

SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00450: background process 'QMNC' did not start
ORA-00444: background process "QMNC" failed while starting
ORA-00020: maximum number of processes () exceeded
进程 ID: 6340
会话 ID: 109 序列号: 3

看这个报错应该是由于进程数达到了限制，导致在open数据库过程中启动后台进程QMNC失败,导致open过程失败,分析进程数不足的原因,查看启动时候的参数配置

Fri Dec 26 16:48:27 2025
Adjusting the default value of parameter parallel_max_servers
from 2560 to 135 due to the value of parameter processes (150)
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Fri Dec 26 16:48:38 2025
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on. 
IMODE=BR
ILAT =28
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options.
Windows NT Version V6.2  
CPU                 : 64 - type 8664, 32 Physical Cores
Process Affinity    : 0x0x0000000000000000
Memory (Avail/Total): Ph:82501M/130752M, Ph+PgF:103658M/150208M 
Using parameter settings in client-side pfile D:\PFILE.TXT on machine HISSERVER
System parameters with non-default values:
  processes                = 150
  sga_target               = 39296M
  control_files            = "D:\ORACLE64\ORADATA\ORA817\CONTROL01.CTL"
  control_files            = "D:\ORACLE64\RECOVERY_AREA\ORA817\CONTROL02.CTL"
  db_block_size            = 8192
  compatible               = "11.2.0.0.0"
  db_recovery_file_dest    = "D:\ORACLE64\recovery_area"
  db_recovery_file_dest_size= 4122M
  undo_tablespace          = "UNDOTBS1"
  remote_login_passwordfile= "EXCLUSIVE"
  db_domain                = ""
  dispatchers              = "(PROTOCOL=TCP) (SERVICE=ora817XDB)"
  audit_file_dest          = "D:\ORACLE64\ADMIN\ORA817\ADUMP"
  audit_trail              = "DB"
  db_name                  = "ora817"
  open_cursors             = 300
  pga_aggregate_target     = 13075M
  diagnostic_dest          = "D:\ORACLE64"
Fri Dec 26 16:49:10 2025
PMON started with pid=2, OS id=1992 
Fri Dec 26 16:49:10 2025
PSP0 started with pid=3, OS id=6744 
Fri Dec 26 16:49:12 2025
VKTM started with pid=4, OS id=7592 at elevated priority
Fri Dec 26 16:49:12 2025
GEN0 started with pid=5, OS id=7780 
Fri Dec 26 16:49:12 2025
DBRM started with pid=7, OS id=5372 
Fri Dec 26 16:49:12 2025
MMAN started with pid=9, OS id=6700 
Fri Dec 26 16:49:12 2025
DBW0 started with pid=10, OS id=1920 
Fri Dec 26 16:49:12 2025
DBW1 started with pid=11, OS id=4476 
Fri Dec 26 16:49:12 2025
DBW2 started with pid=12, OS id=6788 
Fri Dec 26 16:49:12 2025
DBW3 started with pid=13, OS id=4608 
Fri Dec 26 16:49:12 2025
DBW4 started with pid=14, OS id=6716 
Fri Dec 26 16:49:12 2025
DBW5 started with pid=15, OS id=6252 
Fri Dec 26 16:49:12 2025
DBW6 started with pid=16, OS id=7948 
Fri Dec 26 16:49:12 2025
DBW7 started with pid=17, OS id=5868 
Fri Dec 26 16:49:12 2025
LGWR started with pid=18, OS id=1644 
Fri Dec 26 16:49:12 2025
CKPT started with pid=19, OS id=5704 
Fri Dec 26 16:49:12 2025
SMON started with pid=20, OS id=4532 
Fri Dec 26 16:49:12 2025
RECO started with pid=21, OS id=380 
Fri Dec 26 16:49:12 2025
MMON started with pid=22, OS id=1316 
Fri Dec 26 16:49:12 2025
MMNL started with pid=23, OS id=6108 
Fri Dec 26 16:49:12 2025
DIAG started with pid=6, OS id=7904 
Fri Dec 26 16:49:12 2025
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
starting up 1 shared server(s) ...
ORACLE_BASE from environment = D:\ORACLE64

这里可以看出来数据库版本是11.2.0.3,processes配置为默认值150.进一步查看启动报错时候的alert日志

ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Fri Dec 26 17:03:44 2025
Media Recovery failed with error 264
Slave exiting with ORA-283 exception
Errors in file D:\ORACLE64\diag\rdbms\ora817\ora817\trace\ora817_pr00_344.trc:
ORA-00283: recovery session canceled due to errors
ORA-00264: no recovery required
Recovery Slave PR00 previously exited with exception 283
ORA-283 signalled during: ALTER DATABASE RECOVER  database  ...
alter database open
Fri Dec 26 17:03:48 2025
Thread 1 opened at log sequence 37301
  Current log# 2 seq# 37301 mem# 0: D:\ORACLE64\ORADATA\ORA817\REDO02.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Dec 26 17:03:49 2025
SMON: enabling cache recovery
[6340] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:5143437 end:5143718 diff:281 (2 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Fri Dec 26 17:03:49 2025
Database Characterset is ZHS16GBK
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
ORA-00020: maximum number of processes (150) exceeded
 ORA-20 errors will not be written to the alert log for
 the next minute. Please look at trace files to see all
 the ORA-20 errors.
Process P123 submission failed with error = 20
Starting background process QMNC
Process QMNC submission failed with error = 20
Errors in file D:\ORACLE64\diag\rdbms\ora817\ora817\trace\ora817_ora_6340.trc:
ORA-00444: 后台进程 "QMNC" 启动失败
ORA-00020: 超出最大进程数 ()
Errors in file D:\ORACLE64\diag\rdbms\ora817\ora817\trace\ora817_ora_6340.trc:
ORA-00450: 后台进程 'QMNC' 未启动
ORA-00444: 后台进程 "QMNC" 启动失败
ORA-00020: 超出最大进程数 ()
Errors in file D:\ORACLE64\diag\rdbms\ora817\ora817\trace\ora817_ora_6340.trc:
ORA-00450: 后台进程 'QMNC' 未启动
ORA-00444: 后台进程 "QMNC" 启动失败
ORA-00020: 超出最大进程数 ()
Error 450 happened during db open, shutting down database
USER (ospid: 6340): terminating the instance due to error 450
Fri Dec 26 17:03:50 2025
Errors in file D:\ORACLE64\diag\rdbms\ora817\ora817\trace\ora817_p036_8044.trc  (incident=4105):
ORA-01578: ORACLE data block corrupted (file # 9, block # 504643)
ORA-01110: data file 9: 'D:\ORACLE64\ORADATA\ORA817\HRPRUN_DATA05.DBF'
Incident details in: D:\ORACLE64\diag\rdbms\ora817\ora817\incident\incdir_4105\ora817_p036_8044_i4105.trc
Fri Dec 26 17:03:50 2025
Process m000 submission failed with error = 20
Fri Dec 26 17:04:07 2025
Instance terminated by USER, pid = 6340
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (6340) as a result of ORA-1092
Fri Dec 26 17:04:09 2025
ORA-1092 : opitsk aborting process

看alert日志比较明显有Process P123 submission failed with error = 20提示,这里表示启动P123进程时候也遭遇到了进程数不足的限制导致该进程无法启动,而Pxxx进程是oracle的并行进程(而且在后面还报了由于坏块导致P036进程报错的信息),也就是说这个数据库在启动过程中启动了大量的并行进程(至少122个已经启动),这也就是说明了为什么150个进程不足的原因.在数据库完成实例恢复,完全open之前启动并行进程进行恢复的,很可能就是并行回滚操作.为了进一步验证猜想,把processes调整到1500,然后尝试启动库

Fri Dec 26 17:06:44 2025
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 32 processes
Started redo scan
Completed redo scan
 read 20 KB redo, 5 data blocks need recovery
Started redo application at
 Thread 1: logseq 37301, block 18209
Recovery of Online Redo Log: Thread 1 Group 2 Seq 37301 Reading mem 0
  Mem# 0: D:\ORACLE64\ORADATA\ORA817\REDO02.LOG
Completed redo application of 0.02MB
Completed crash recovery at
 Thread 1: logseq 37301, block 18249, scn 327081807
 5 data blocks read, 5 data blocks written, 20 redo k-bytes read
Fri Dec 26 17:06:46 2025
Thread 1 advanced to log sequence 37302 (thread open)
Thread 1 opened at log sequence 37302
  Current log# 3 seq# 37302 mem# 0: D:\ORACLE64\ORADATA\ORA817\REDO03.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Dec 26 17:06:47 2025
SMON: enabling cache recovery
[7104] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:5321812 end:5322031 diff:219 (2 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Fri Dec 26 17:06:49 2025
Errors in file D:\ORACLE64\diag\rdbms\ora817\ora817\trace\ora817_p091_5924.trc  (incident=5745):
ORA-01578: ORACLE data block corrupted (file # 9, block # 504643)
ORA-01110: data file 9: 'D:\ORACLE64\ORADATA\ORA817\HRPRUN_DATA05.DBF'
Incident details in: D:\ORACLE64\diag\rdbms\ora817\ora817\incident\incdir_5745\ora817_p091_5924_i5745.trc
Starting background process QMNC
Fri Dec 26 17:06:49 2025
QMNC started with pid=156, OS id=8012 
Errors in file D:\ORACLE64\diag\rdbms\ora817\ora817\trace\ora817_p091_5924.trc  (incident=5746):
ORA-01578: ORACLE data block corrupted (file # , block # )
Incident details in: D:\ORACLE64\diag\rdbms\ora817\ora817\incident\incdir_5746\ora817_p091_5924_i5746.trc
Errors in file D:\ORACLE64\diag\rdbms\ora817\ora817\trace\ora817_smon_7424.trc  (incident=4961):
ORA-01578: ORACLE data block corrupted (file # 9, block # 504643)
ORA-01110: data file 9: 'D:\ORACLE64\ORADATA\ORA817\HRPRUN_DATA05.DBF'
Incident details in: D:\ORACLE64\diag\rdbms\ora817\ora817\incident\incdir_4961\ora817_smon_7424_i4961.trc
Completed: alter database open
SMON: Restarting fast_start parallel rollback
Fri Dec 26 17:06:51 2025
Errors in file D:\ORACLE64\diag\rdbms\ora817\ora817\trace\ora817_p000_5472.trc  (incident=5009):
ORA-01578: ORACLE data block corrupted (file # 9, block # 504643)
ORA-01110: data file 9: 'D:\ORACLE64\ORADATA\ORA817\HRPRUN_DATA05.DBF'
Incident details in: D:\ORACLE64\diag\rdbms\ora817\ora817\incident\incdir_5009\ora817_p000_5472_i5009.trc
Errors in file D:\ORACLE64\diag\rdbms\ora817\ora817\trace\ora817_p000_5472.trc  (incident=5010):
ORA-01578: ORACLE data block corrupted (file # , block # )
Errors in file D:\ORACLE64\diag\rdbms\ora817\ora817\trace\ora817_smon_7424.trc  (incident=4962):
ORA-01578: ORACLE data block corrupted (file # 9, block # 504643)
ORA-01110: data file 9: 'D:\ORACLE64\ORADATA\ORA817\HRPRUN_DATA05.DBF'
Incident details in: D:\ORACLE64\diag\rdbms\ora817\ora817\incident\incdir_5010\ora817_p000_5472_i5010.trc

这次就比较清晰了,数据库在open过程中由于smon启动了并行回滚并遇到了坏块,导致P091进程和smon进程错误。因为数据库已经open,这个坏块导致无法回滚,直接对这个对象进行处理即可解决掉这个回滚报错问题.

ORA-600 ktbair2: illegal inheritance恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 ktbair2: illegal inheritance恢复

接到客户一个恢复case咨询,数据库open过程报ORA-00600: 内部错误代码, 参数: [ktbair2: illegal inheritance]错误

查看alert日志

2025-12-21T10:38:49.346922+08:00
alter database open
2025-12-21T10:38:50.286621+08:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
2025-12-21T10:38:50.300632+08:00
Beginning crash recovery of 1 threads
2025-12-21T10:38:50.378690+08:00
 parallel recovery started with 32 processes
 Thread 1: Recovery starting at checkpoint rba (logseq 47910 block 201391), scn 0
2025-12-21T10:38:50.446742+08:00
Started redo scan
2025-12-21T10:38:50.597853+08:00
Completed redo scan
 read 57808 KB redo, 409 data blocks need recovery
2025-12-21T10:38:50.827024+08:00
Started redo application at
 Thread 1: logseq 47910, block 201391, offset 0
2025-12-21T10:38:50.845043+08:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 47910 Reading mem 0
  Mem# 0: F:\APP\ORADATA\SRMDEV\REDO03.LOG
2025-12-21T10:38:50.929100+08:00
Completed redo application of 3.15MB
2025-12-21T10:38:50.975134+08:00
Errors in file F:\APP\diag\rdbms\srmdev\srmdev\trace\srmdev_p00e_11192.trc  (incident=1382836):
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
Incident details in: F:\APP\diag\rdbms\srmdev\srmdev\incident\incdir_1382836\srmdev_p00e_11192_i1382836.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-12-21T10:38:52.033922+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
…………
2025-12-21T10:39:02.224702+08:00
Errors in file F:\APP\diag\rdbms\srmdev\srmdev\trace\srmdev_ora_9960.trc:
ORA-01578: ORACLE 数据块损坏 (文件号 7, 块号 611)
ORA-01110: 数据文件 7: 'F:\APP\ORADATA\SRMDEV\USERS01.DBF'
ORA-10564: tablespace USERS
ORA-01110: 数据文件 7: 'F:\APP\ORADATA\SRMDEV\USERS01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 74736
ORA-00600: 内部错误代码, 参数: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
ORA-1578 signalled during: alter database open...

这个错误以前有过类似处理,主要是由于redo无法正确应用,数据库在open过程无法正常完成实例恢复所以出现该问题.以前处理过类似案例:
Oracle Recovery Tools修复ORA-00742、ORA-600 ktbair2: illegal inheritance故障
这次的故障相对简单一些,通过一些简单尝试之后数据库正常应用成功

2025-12-21T21:43:00.689653+08:00
ALTER DATABASE RECOVER  datafile 1  
2025-12-21T21:43:00.691654+08:00
Media Recovery Start
2025-12-21T21:43:00.693656+08:00
Serial Media Recovery started
2025-12-21T21:43:00.721676+08:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 47910 Reading mem 0
  Mem# 0: F:\APP\ORADATA\SRMDEV\REDO03.LOG
Completed: ALTER DATABASE RECOVER  datafile 1  
2025-12-21T21:43:14.109099+08:00
ALTER DATABASE RECOVER  datafile 3  
2025-12-21T21:43:14.111100+08:00
Media Recovery Start
2025-12-21T21:43:14.113102+08:00
Serial Media Recovery started
2025-12-21T21:43:14.139121+08:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 47910 Reading mem 0
  Mem# 0: F:\APP\ORADATA\SRMDEV\REDO03.LOG
Completed: ALTER DATABASE RECOVER  datafile 3  
2025-12-21T21:43:21.361498+08:00
ALTER DATABASE RECOVER  datafile 4  
2025-12-21T21:43:21.363499+08:00
Media Recovery Start
2025-12-21T21:43:21.365500+08:00
Serial Media Recovery started
2025-12-21T21:43:21.397524+08:00
Media Recovery failed with error 264
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 4  ...
2025-12-21T21:43:31.909350+08:00
ALTER DATABASE RECOVER  datafile 7  
2025-12-21T21:43:31.911351+08:00
Media Recovery Start
2025-12-21T21:43:31.913353+08:00
Serial Media Recovery started
2025-12-21T21:43:31.940373+08:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 47910 Reading mem 0
  Mem# 0: F:\APP\ORADATA\SRMDEV\REDO03.LOG
Completed: ALTER DATABASE RECOVER  datafile 7

然后比较幸运直接open数据库成功,但是报ORA-600 4194错误

2025-12-21T21:45:34.003260+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2025-12-21T21:45:34.262453+08:00
Undo initialization recovery: err:0 start: 45858156 end: 45858296 diff: 140 ms (0.1 seconds)
[13724] Successfully onlined Undo Tablespace 2.
Undo initialization online undo segments: err:0 start: 45858296 end: 45858359 diff: 63 ms (0.1 seconds)
Undo initialization finished serial:0 start:45858156 end:45858359 diff:203 ms (0.2 seconds)
Verifying minimum file header compatibility for tablespace encryption..
Verifying file header compatibility for tablespace encryption completed for pdb 0
Database Characterset is AL32UTF8
2025-12-21T21:45:34.584694+08:00
Errors in file F:\APP\diag\rdbms\srmdev\srmdev\trace\srmdev_smon_13616.trc  (incident=1510624):
ORA-00600: 内部错误代码, 参数: [4194], [58], [49], [], [], [], [], [], [], [], [], []
Incident details in:F:\APP\diag\rdbms\srmdev\srmdev\incident\incdir_1510624\srmdev_smon_13616_i1510624.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-12-21T21:45:34.598703+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2025-12-21T21:45:35.634474+08:00
Autotune of undo retention is turned off. 
2025-12-21T21:45:35.636476+08:00
Resource Manager disabled during database migration: plan '''' not set
Resource Manager disabled during database migration
2025-12-21T21:45:36.009753+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Doing block recovery for file 4 block 27127
Resuming block recovery (PMON) for file 4 block 27127
Block recovery from logseq 47911, block 78 to scn 0x0000000000000000
2025-12-21T21:45:36.026766+08:00
Recovery of Online Redo Log: Thread 1 Group 4 Seq 47911 Reading mem 0
  Mem# 0: F:\APP\ORADATA\SRMDEV\RED004.LOG
Block recovery completed at rba 0.0.0, scn 0x00000006a378f1bc
Doing block recovery for file 4 block 176
Resuming block recovery (PMON) for file 4 block 176
Block recovery from logseq 47911, block 78 to scn 0x00000006a378f23d
2025-12-21T21:45:36.045781+08:00
Recovery of Online Redo Log: Thread 1 Group 4 Seq 47911 Reading mem 0
  Mem# 0: F:\APP\ORADATA\SRMDEV\RED004.LOG
Block recovery completed at rba 47911.78.16, scn 0x00000006a378f23e
Non-fatal internal error happened while SMON was doing shrinking of rollback segments.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
replication_dependency_tracking turned off (no async multimaster replication found)
2025-12-21T21:45:36.493113+08:00
AQ Processes can not start in restrict mode
Starting background process CJQ0
2025-12-21T21:45:36.545152+08:00
CJQ0 started with pid=73, OS id=14120 
2025-12-21T21:45:37.546898+08:00
Completed: alter database open

这个处理起来比较简单,对于异常的undo进行处理之后,数据库正常导出dmp,完成本次恢复任务

一键恢复ORA-00704 ORA-00702故障—202512

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：一键恢复ORA-00704 ORA-00702故障—202512

今天又遇到一个客户遭遇ORA-00702错误

alert日志相关报错信息

Mon Dec 15 14:10:56 2025
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 5 processes
Started redo scan
Completed redo scan
 read 0 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 654, block 2, scn 16937055
Recovery of Online Redo Log: Thread 1 Group 3 Seq 654 Reading mem 0
  Mem# 0: E:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 654, block 3, scn 16957057
 0 data blocks read, 0 data blocks written, 0 redo k-bytes read
Thread 1 advanced to log sequence 655 (thread open)
Thread 1 opened at log sequence 655
  Current log# 1 seq# 655 mem# 0: E:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Dec 15 14:10:57 2025
SMON: enabling cache recovery
Errors in file E:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_7952.trc:
ORA-00704: 引导程序进程失败
ORA-00702: 引导程序版本 '' 与版本 '8.0.0.0.0' 不一致
Errors in file E:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_7952.trc:
ORA-00704: 引导程序进程失败
ORA-00702: 引导程序版本 '' 与版本 '8.0.0.0.0' 不一致
Error 704 happened during db open, shutting down database
USER (ospid: 7952): terminating the instance due to error 704
Instance terminated by USER, pid = 7952
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (7952) as a result of ORA-1092
Mon Dec 15 14:11:00 2025
ORA-1092 : opitsk aborting process
Mon Dec 15 14:39:18 2025

这种故障处理比较多，主要是bootstrap$基表损坏,后来为了解决这种问题方便,专门写了一个小工具（bootstrap$异常修复小工具）来一键修复该问题

通过bootstrap$异常修复小工具修复之后,然后直接启动数据库成功,完成本次恢复任务

open数据库报ora-600 kdsgrp1故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：open数据库报ora-600 kdsgrp1故障处理

客户服务器异常关机之后,数据库无法正常启动,查看alert日志发现如下报ORA-600 kdsgrp1报错信息导致数据库无法正常open

2025-12-05T16:17:23.315325+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
Undo initialization recovery: err:0 start: 7620046 end: 7620062 diff: 16 ms (0.0 seconds)
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_9672.trc  (incident=1243378):
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_1243378\his_ora_9672_i1243378.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-12-05T16:17:25.341421+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2025-12-05T16:17:25.533576+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_9672.trc  (incident=1243379):
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_1243379\his_ora_9672_i1243379.trc
2025-12-05T16:17:27.007101+08:00
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-12-05T16:17:27.008097+08:00
Undo initialization online undo segments: err:600 start: 7620062 end: 7623343 diff: 3281 ms (3.3 seconds)
2025-12-05T16:17:27.008097+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_9672.trc:
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
2025-12-05T16:17:27.008097+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_9672.trc:
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_9672.trc  (incident=1243380):
ORA-00603: ORACLE 服务器会话因致命错误而终止
ORA-01092: ORACLE 实例终止。强制断开连接
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\incident\incdir_1243380\his_ora_9672_i1243380.trc
2025-12-05T16:17:28.540363+08:00
opiodr aborting process unknown ospid (9672) as a result of ORA-603

尝试人工启动数据库,报错比较明显也是ORA-600 kdsgrp1错误

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Dec 5 16:45:48 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> recover datafile 1;
Media recovery complete.
SQL> recover database;
Media recovery complete.
SQL> alter database open ;
alter database open
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [],
[], [], [], [], []
Process ID: 10800
Session ID: 5025 Serial number: 34108

查看报错trace文件

*** CLIENT DRIVER:(SQL*PLUS) 2025-12-05T16:17:23.924647+08:00
 
[TOC00000]
Jump to table of contents
Dump continued from file: D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_9672.trc
[TOC00001]
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []

[TOC00001-END]
[TOC00002]
========= Dump for incident 1243378 (ORA 600 [kdsgrp1]) ========
[TOC00003]
----- Beginning of Customized Incident Dump(s) -----
kdsDumpState: cdb: 0 dspdb: 0 type: 1 info: 0x5 flag: 0x0
kdsDumpState: sample type: 0 layer: 0
* kdsgrp1-1: ***********************************************
            row 0x00c02754.24 continuation at: 0x00c02754.24 file# 3 block# 10068 slot 36 not found (dscnt: 0)
 state kdscurrid points to 0x00c28206.17
KDSTABN_GET: 0 ..... ntab: 1
curSlot: 36 ..... nrows: 60
Dumping kcb descriptor:
kcbds 0x1e08fe71510: pdb 0, tsn 1, rdba 0x00c02754, afn 3, objd 11331, cls 1, tidflg 0x8 0x80 0x0

在这里基本上可以确认报ORA-600 错误的是file 3 block 10068，dataobj 为11331,根据经验初步怀疑是sysaux中的wrh$_某个对象异常.对启动过程做10046进一步确认

EXEC #2063975471112:c=17341,e=35163,p=54,cr=283,cu=0,mis=1,r=0,dep=1,og=4,plh=794648223,tim=9717774784
WAIT #2063975471112: nam='db file sequential read' ela= 115 file#=3 block#=11434 blocks=1 obj#=11579 tim=9717774961
WAIT #2063975471112: nam='db file sequential read' ela= 93 file#=3 block#=11435 blocks=1 obj#=11579 tim=9717775098
WAIT #2063975471112: nam='db file sequential read' ela= 89 file#=3 block#=11436 blocks=1 obj#=11579 tim=9717775217
WAIT #2063975471112: nam='db file sequential read' ela= 98 file#=3 block#=11437 blocks=1 obj#=11579 tim=9717775361
WAIT #2063975471112: nam='db file sequential read' ela= 23087 file#=3 block#=11438 blocks=1 obj#=11579 tim=9717798471
WAIT #2063975471112: nam='db file sequential read' ela= 83 file#=3 block#=11439 blocks=1 obj#=11579 tim=9717798681
WAIT #2063975471112: nam='db file sequential read' ela= 183 file#=3 block#=10075 blocks=1 obj#=11332 tim=9717799100
WAIT #2063975471112: nam='db file sequential read' ela= 124 file#=3 block#=10078 blocks=1 obj#=11332 tim=9717799291
WAIT #2063975471112: nam='db file sequential read' ela= 115 file#=3 block#=165702 blocks=1 obj#=11332 tim=9717799460
WAIT #2063975471112: nam='db file sequential read' ela= 109 file#=3 block#=83355 blocks=1 obj#=11332 tim=9717799621
WAIT #2063975471112: nam='db file sequential read' ela= 139 file#=3 block#=165703 blocks=1 obj#=11332 tim=9717799816
WAIT #2063975471112: nam='db file sequential read' ela= 139 file#=3 block#=10079 blocks=1 obj#=11332 tim=9717800074
WAIT #2063975471112: nam='db file sequential read' ela= 112 file#=3 block#=165696 blocks=1 obj#=11332 tim=9717800248
WAIT #2063975471112: nam='db file sequential read' ela= 110 file#=3 block#=165701 blocks=1 obj#=11332 tim=9717800502
WAIT #2063975471112: nam='db file sequential read' ela= 107 file#=3 block#=83354 blocks=1 obj#=11332 tim=9717800665
WAIT #2063975471112: nam='db file sequential read' ela= 109 file#=3 block#=83356 blocks=1 obj#=11332 tim=9717800823
WAIT #2063975471112: nam='db file sequential read' ela= 107 file#=3 block#=83358 blocks=1 obj#=11332 tim=9717800978
WAIT #2063975471112: nam='db file sequential read' ela= 106 file#=3 block#=165698 blocks=1 obj#=11332 tim=9717801133
WAIT #2063975471112: nam='db file sequential read' ela= 106 file#=3 block#=164358 blocks=1 obj#=11331 tim=9717801298
WAIT #2063975471112: nam='db file sequential read' ela= 108 file#=3 block#=10068 blocks=1 obj#=11331 tim=9717801509
2025-12-05T16:52:21.382064+08:00
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []

FETCH #2063975471112:c=1599462,e=1655487,p=20,cr=26,cu=0,mis=0,r=0,dep=1,og=4,plh=794648223,tim=9719430286
STAT #2063975471112 id=1 cnt=0 pid=0 pos=1 obj=0 op='SORT AGGREGATE (cr=0 pr=0 pw=0 str=1 time=8 us)'
STAT #2063975471112 id=2 cnt=0 pid=1 pos=1 obj=0 op='HASH JOIN  (cr=0 pr=0 pw=0 str=1 time=5 us cost=9 size=34 card=1)'
STAT #2063975471112 id=3 cnt=1 pid=2 pos=1 obj=11579 op='TABLE ACCESS FULL WRM$_SNAPSHOT
 (cr=12 pr=6 pw=0 str=1 time=23966 us cost=4 size=17 card=1)'
STAT #2063975471112 id=4 cnt=1 pid=3 pos=1 obj=0 op='SORT AGGREGATE (cr=6 pr=3 pw=0 str=1 time=23513 us)'
STAT #2063975471112 id=5 cnt=1 pid=4 pos=1 obj=11579 op='TABLE ACCESS FULL WRM$_SNAPSHOT
(cr=6 pr=3 pw=0 str=1 time=23497 us cost=4 size=13 card=1)'
STAT #2063975471112 id=6 cnt=6 pid=2 pos=2 obj=11331 op='TABLE ACCESS BY INDEX ROWID BATCHED WRH$_UNDOSTAT
(cr=13 pr=13 pw=0 str=1 time=2455 us cost=4 size=102 card=6)'
STAT #2063975471112 id=7 cnt=7 pid=6 pos=1 obj=11332 op='INDEX SKIP SCAN WRH$_UNDOSTAT_PK
(cr=12 pr=12 pw=0 str=1 time=2304 us cost=2 size=0 card=6)'
2025-12-05T16:52:23.005928+08:00
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []

<error barrier> at 0x000000275BED14D0 placed dbsdrv.c@4959
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
<error barrier> at 0x000000275BED14D0 placed dbsdrv.c@4959
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []

到这里基本上可以确认是WRH$_UNDOSTAT_PK去找WRH$_UNDOSTAT中午对应记录,从而出现该问题,对应的具体sql为

PARSING IN CURSOR #2063975471112 len=233 dep=1 uid=0 oct=3 lid=0 tim=9717739563
  hv=517686153 ad='1e1efda1f20' sqlid='d4m7ss0gdqhw9'
select max(maxconcurrency) from sys.wrh$_undostat where instance_number = :1 and dbid = :2 and snap_id in  
(select snap_id from dba_hist_snapshot where end_interval_time >(select max(end_interval_time)-7 from dba_hist_snapshot))

BINDS #2063975471112:

 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=0000 frm=00 csi=00 siz=48 off=0
  kxsbbbfp=8e8646e8  bln=22  avl=02  flg=05
  value=1
 Bind#1
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=00 fl2=0000 frm=00 csi=00 siz=0 off=24
  kxsbbbfp=8e864700  bln=22  avl=06  flg=01
  value=971429521

对于这种问题,处理起来相对比较简单,绕过oracle open过程对该sql的执行,然后open库对该表的index进行rebuild

SQL> recover database;
Media recovery complete.
SQL> alter database open;

Database altered.

SQL> SELECT OWNER, SEGMENT_NAME, SEGMENT_TYPE, TABLESPACE_NAME, A.PARTITION_NAME
  2    FROM DBA_EXTENTS A
  3   WHERE FILE_ID = &FILE_ID
  4     AND &BLOCK_ID BETWEEN BLOCK_ID AND BLOCK_ID + BLOCKS - 1;
Enter value for file_id: 3
old   3:  WHERE FILE_ID = &FILE_ID
new   3:  WHERE FILE_ID = 3
Enter value for block_id: 10068
old   4:    AND &BLOCK_ID BETWEEN BLOCK_ID AND BLOCK_ID + BLOCKS - 1
new   4:    AND 10068 BETWEEN BLOCK_ID AND BLOCK_ID + BLOCKS - 1

OWNER
--------------------------------------------------------------------------------
SEGMENT_NAME
--------------------------------------------------------------------------------
SEGMENT_TYPE       TABLESPACE_NAME
------------------ ------------------------------
PARTITION_NAME
--------------------------------------------------------------------------------
SYS
WRH$_UNDOSTAT
TABLE              SYSAUX



SQL> select index_name from dba_indexes where table_name='WRH$_UNDOSTAT';

INDEX_NAME
--------------------------------------------------------------------------------
WRH$_UNDOSTAT_PK

SQL> alter index WRH$_UNDOSTAT_PK REBUILD ONLINE;

Index altered.

至此该数据库完成恢复任务,重启之后一切正常.

ORA-01172 ORA-01151故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-01172 ORA-01151故障处理

接手客户一个云平台硬件故障恢复之后,数据库无法启动的case,通过分析alert日志,发现数据库在open过程中报ORA-01172: recovery of thread 1 stuck at block 2220167 of file 262,ORA-01151: use media recovery to recover block, restore backup if needed等相关错误(其实也就是在做实例恢复的过程中报了logically corrupt导致无法完成实例恢复)

Sat Nov 01 14:29:10 2025
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 7034 KB redo, 937 data blocks need recovery
Started redo application at
 Thread 1: logseq 296553, block 389408
Recovery of Online Redo Log: Thread 1 Group 3 Seq 296553 Reading mem 0
  Mem# 0: /data/orcl/onlinelog/redo03a.log
Sat Nov 01 14:29:11 2025
Hex dump of (file 262, block 2220584) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p009_2648.trc
Sat Nov 01 14:29:11 2025
Hex dump of (file 262, block 2218886) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p007_2644.trc
Reading datafile '/data/orcl/datafile/xifenfei12.dbf' for corruption at rdba: 0x41a1db86 (file 262, block 2218886)
Reading datafile '/data/orcl/datafile/xifenfei12.dbf' for corruption at rdba: 0x41a1e228 (file 262, block 2220584)
Sat Nov 01 14:29:11 2025
Hex dump of (file 262, block 2219845) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p008_2646.trc
Reading datafile '/data/orcl/datafile/xifenfei12.dbf' for corruption at rdba: 0x41a1df45 (file 262, block 2219845)
Sat Nov 01 14:29:11 2025
Hex dump of (file 262, block 2220167) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p001_2632.trc
Reading datafile '/data/orcl/datafile/xifenfei12.dbf' for corruption at rdba: 0x41a1e087 (file 262, block 2220167)
Reread (file 262, block 2218886) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 2218886 OF FILE 262
Reread (file 262, block 2220584) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 2220584 OF FILE 262
Reread (file 262, block 2219845) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 2219845 OF FILE 262
Reread (file 262, block 2220167) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 2220167 OF FILE 262
Sat Nov 01 14:29:26 2025
Slave exiting with ORA-1172 exception
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p007_2644.trc:
ORA-01172: recovery of thread 1 stuck at block 2218886 of file 262
ORA-01151: use media recovery to recover block, restore backup if needed
Sat Nov 01 14:29:26 2025
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p008_2646.trc:
ORA-10388: parallel query server interrupt (failure)
Sat Nov 01 14:29:26 2025
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p009_2648.trc:
ORA-10388: parallel query server interrupt (failure)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p008_2646.trc:
ORA-10388: parallel query server interrupt (failure)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p009_2648.trc:
ORA-10388: parallel query server interrupt (failure)
Sat Nov 01 14:29:26 2025
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p001_2632.trc:
ORA-10388: parallel query server interrupt (failure)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p001_2632.trc:
ORA-10388: parallel query server interrupt (failure)
Sat Nov 01 14:29:26 2025
Aborting crash recovery due to slave death, attempting serial crash recovery
Beginning crash recovery of 1 threads
Started redo scan
Completed redo scan
 read 7034 KB redo, 937 data blocks need recovery
Started redo application at
 Thread 1: logseq 296553, block 389408
Recovery of Online Redo Log: Thread 1 Group 3 Seq 296553 Reading mem 0
  Mem# 0: /data/orcl/onlinelog/redo03a.log
Hex dump of (file 262,block 2220167) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2606.trc
Reading datafile '/data/orcl/datafile/xifenfei12.dbf'for corruption at rdba: 0x41a1e087 (file 262,block 2220167)
Reread (file 262, block 2220167) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 2220167 OF FILE 262
Aborting crash recovery due to error 1172
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2606.trc:
ORA-01172: recovery of thread 1 stuck at block 2220167 of file 262
ORA-01151: use media recovery to recover block, restore backup if needed
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2606.trc:
ORA-01172: recovery of thread 1 stuck at block 2220167 of file 262
ORA-01151: use media recovery to recover block, restore backup if needed
ORA-1172 signalled during: ALTER DATABASE OPEN...

接手故障之后,尝试recover database恢复,结果报ORA-600 4552错误

SQL> recover database;
ORA-10562: Error occurred while applying redo to data block (file# 262, block#
2222153)
ORA-10564: tablespace XIFENFEI
ORA-01110: data file 262: '/data/orcl/datafile/xifenfei12.dbf'
ORA-10560: block type '0'
ORA-00600: internal error code, arguments: [4552], [1], [0], [], [], [], [],
[], [], [], [], []

关于ORA-600 4552对应的alert日志信息

Sat Nov 01 17:49:58 2025
ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 16 slaves
Sat Nov 01 17:49:59 2025
Recovery of Online Redo Log: Thread 1 Group 3 Seq 296553 Reading mem 0
  Mem# 0: /data/orcl/onlinelog/redo03a.log
Sat Nov 01 17:49:59 2025
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr0c_28770.trc  (incident=1018821):
ORA-00600: internal error code, arguments: [4552], [1], [0], [], [], [], [], [], [], [], [], []
Incident details in:/u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_1018821/orcl_pr0c_28770_i1018821.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Sat Nov 01 17:50:03 2025
Sweep [inc][1018821]: completed
Sweep [inc2][1018821]: completed
Slave exiting with ORA-10562 exception
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr0c_28770.trc:
ORA-10562: Error occurred while applying redo to data block (file# 262, block# 2222153)
ORA-10564: tablespace xifenfei
ORA-01110: data file 262: '/data/orcl/datafile/xifenfei12.dbf'
ORA-10560: block type '0'
ORA-00600: internal error code, arguments: [4552], [1], [0], [], [], [], [], [], [], [], [], []
Recovery Slave PR0C previously exited with exception 10562
Media Recovery failed with error 448
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr00_28732.trc:
ORA-00283: recovery session canceled due to errors
ORA-00448: normal completion of background process
ORA-10562 signalled during: ALTER DATABASE RECOVER  database  ...

无法整个库级别recover,尝试数据文件recover操作

SQL> recover datafile 1;
Media recovery complete.
…………
SQL> recover datafile 22,23,24,26,25,27,28,29,30;
Media recovery complete.
…………
SQL>    recover datafile  251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261;
Media recovery complete.
SQL> recover datafile 262;
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [3020], [262], [2215808],
[1101123456], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 262, block# 2215808,
file offset is 972029952 bytes)
ORA-10564: tablespace XIFENFEI
ORA-01110: data file 262: '/data/orcl/datafile/xifenfei12.dbf'
ORA-10560: block type '0'

SQL> recover datafile 263;
Media recovery complete.

出file# 262数据文件之外,其他文件全部recover成功,对应的ORA-600 3020错误相关alert日志信息

Sat Nov 01 17:53:37 2025
ALTER DATABASE RECOVER  datafile 262  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 3 Seq 296553 Reading mem 0
  Mem# 0: /data/orcl/onlinelog/redo03a.log
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_28561.trc  (incident=1018717):
ORA-00600: internal error code, arguments: [3020], [262], [2215808], [1101123456], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 262, block# 2215808, file offset is 972029952 bytes)
ORA-10564: tablespace xifenfei
ORA-01110: data file 262: '/data/orcl/datafile/xifenfei12.dbf'
ORA-10560: block type '0'
Incident details in:/u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_1018717/orcl_ora_28561_i1018717.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Media Recovery failed with error 600
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 262  ...

对于这种情况,有两种处理方式:
1)在recover过程中对于报错的block标记为坏块,然后继续恢复,这样正常应用日志成功,再把标记的坏块修复好
2)直接修改该文件头跳过该文件跳过这些block的应用日志,直接骗过数据库
在本case中由于客户急着恢复业务,需要尽快处理,所以采用了第一个方案,这里我使用自研的m_scn(modify_scn)工具快速修改相关数据文件信息

[oracle@host-172-18-50-10 tmp]$ cat 1.txt
1@/data/orcl/datafile/system01.dbf
262@/data/orcl/datafile/xifenfei12.dbf
[oracle@host-172-18-50-10 tmp]$ ./m_scn 1.txt
Please Enter Password: 

===== Starting Datafile Header modification program =====
Datafile list file: 1.txt
Operation Mode: Only Modify Datafile Header CheckPoint
Block Size: 8192
Log Path: /tmp/modify_scn
---------------------------------------------------------
Preparing Datafile list file...
Verifying Datafile existence...
Datafile verification passed
Initializing working directory...
Recovery script created: /tmp/modify_scn/backup/recover_datafile.sh
---------------------------------------------------------
Starting Datafile Header processing (total 2 files)...
[1/2] Processing Datafile Header: /data/orcl/datafile/system01.dbf (File number: 1)
  - Skipping file number 1 (control file)
---------------------------------------------------------
[2/2] Processing Datafile Header: /data/orcl/datafile/xifenfei12.dbf (File number: 262)
  - Backing up Datafile header...
  - Executing Datafile Header modification with block size 8192...
  - Datafile Header processing completed
---------------------------------------------------------
Cleaning up temporary files...
================= All operations completed =================

Note: Execute /tmp/modify_scn/backup/recover_datafile.sh operation for rollback

然后查询相关scn信息,确认修改文件信息没有问题并尝试recover 262号文件

[oracle@host-172-18-50-10 tmp]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sat Nov 1 18:02:18 2025

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount;
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 4.1823E+10 bytes
Fixed Size                  2262368 bytes
Variable Size            4294970016 bytes
Database Buffers         3.7447E+10 bytes
Redo Buffers               78614528 bytes
Database mounted.
SQL> set pages 10000
SQL> set numw 16
SQL> SELECT status,
  2  checkpoint_change#,
  3  to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,
  4  last_change#,
  5  count(*) ROW_NUM
FROM v$datafile
  6    7  GROUP BY status, checkpoint_change#, checkpoint_time,last_change#
ORDER BY status, checkpoint_change#, checkpoint_time;
  8  

set numw 16
col CHECKPOINT_TIME for a40
set lines 150
set pages 1000
SELECT status,
to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,FUZZY,checkpoint_change#,
count(*) ROW_NUM
FROM v$datafile_header
GROUP BY status, checkpoint_change#, to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss'),fuzzy
ORDER BY status, checkpoint_change#, checkpoint_time;

SELECT dd.FILE#,
dd.NAME,
dd.STATUS,
dd.checkpoint_change# dfile_chkp_change,
dh.checkpoint_change# dfile_hed_chkp_change,
dh.recover,
dh.fuzzy
FROM v$datafile dd,
v$datafile_header dh
WHERE dd.FILE#=dh.FILE#
AND dd.checkpoint_change#<>dh.checkpoint_change#;


STATUS  CHECKPOINT_CHANGE# CHECKPOINT_TIME         LAST_CHANGE#          ROW_NUM
------- ------------------ ------------------- ---------------- ----------------
ONLINE      16816934458875 2025-11-01 17:58:28   16816934458875              258
RECOVER     16816934368799 2025-11-01 05:29:39   16816934456943                1
SYSTEM      16816934458875 2025-11-01 17:58:28   16816934458875                4

SQL> SQL> SQL> SQL> SQL> SQL> SQL>   2    3    4    5    6  
STATUS  CHECKPOINT_TIME                          FUZ CHECKPOINT_CHANGE#          ROW_NUM
------- ---------------------------------------- --- ------------------ ----------------
OFFLINE 2025-11-01 17:58:28                      NO      16816934458875                1
ONLINE  2025-11-01 17:58:28                      NO      16816934458875              262

SQL> SQL>   2    3    4    5    6    7    8    9   10   11  
           FILE#
----------------
NAME
----------------------------------------------------------------------------------
STATUS  DFILE_CHKP_CHANGE DFILE_HED_CHKP_CHANGE REC FUZ
------- ----------------- --------------------- --- ---
             262
/data/orcl/datafile/xifenfei12.dbf
RECOVER    16816934368799        16816934458875 YES NO

SQL> recover datafile 262;
Media recovery complete.

open数据库成功

SQL> alter database open;

Database altered.

Sat Nov 01 18:06:00 2025
ALTER DATABASE OPEN
Thread 1 opened at log sequence 296554
  Current log# 1 seq# 296554 mem# 0: /data/orcl/onlinelog/redo01a.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
[33941] Successfully onlined Undo Tablespace 143.
Undo initialization finished serial:0 start:12793234 end:12793304 diff:70 (0 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Sat Nov 01 18:06:01 2025
QMNC started with pid=20, OS id=33973 
Completed: ALTER DATABASE OPEN

至此基本上完成本次恢复任务,后续根据alert日志,有个别表可能由于在file# 262中丢失一些数据导致不一致的问题进行单独,其他没有太大问题,最快帮客户恢复了业务

C_OBJ#_INTCOL#坏块导致数据库无法open故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：C_OBJ#_INTCOL#坏块导致数据库无法open故障处理

客户通过硬件恢复出来数据文件之后,尝试启动数据库报错,他们经过多轮尝试依旧无法open数据库,还原到恢复出来文件的初始状态,通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)脚本检测,发现file# 3 需要sequence#为3200的日志,其他文件需要sequence#为3205的日志

检查发现数据库为非归档模式,而且sequence#为3200的redo已经被覆盖

基于上述情况,可以确认除file# 3之外,其他文件可以正常recover

[oracle@oracledb check_db]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Oct 15 16:08:33 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> recover datafile 1;
Media recovery complete.
SQL> recover datafile 2,4,5,7,8,9,10,11,12,13,14,15,16,17;
Media recovery complete.
SQL> recover datafile 18,19,20,21;
Media recovery complete.

对于缺少日志的数据文件,直接使用自研的m_scn(modify scn)工具进行修改

[oracle@oracledb check_db]$ chmod +x m_scn
[oracle@oracledb check_db]$ ./m_scn 1.txt
Please Enter Password: 

===== Starting Datafile Header modification program =====
Datafile list file: 1.txt
Operation Mode: Only Modify Datafile Header CheckPoint
Block Size: 8192
Log Path: /tmp/modify_scn
---------------------------------------------------------
Preparing Datafile list file...
Verifying Datafile existence...
ERROR: Datafile  does not exist!
Datafile verification passed
Initializing working directory...
Recovery script created: /tmp/modify_scn/backup/recover_datafile.sh
---------------------------------------------------------
Starting Datafile Header processing (total 3 files)...
[1/3] Processing Datafile Header: /data/20251014HF/oradata/system01.dbf (File number: 1)
  - Skipping file number 1 (control file)
---------------------------------------------------------
[2/3] Processing Datafile Header: /data/20251014HF/oradata/sysaux01.dbf (File number: 3)
  - Backing up Datafile header...
  - Executing Datafile Header modification with block size 8192...
  - Datafile Header processing completed
---------------------------------------------------------
Cleaning up temporary files...
================= All operations completed =================

Note: Execute /tmp/modify_scn/backup/recover_datafile.sh operation for rollback

[oracle@oracledb check_db]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Oct 15 16:15:12 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> recover datafile 3;
Media recovery complete.

然后尝试打开数据库报ORA-01092 ORA-01578等错误

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00604: error occurred at recursive SQL level 2
ORA-01578: ORACLE data block corrupted (file # 1, block # 132585)
ORA-01110: data file 1: '/data/app/oracle/oradata/ORCL/system01.dbf'
Process ID: 1617
Session ID: 1 Serial number: 5

报错比较明显是由于坏块导致数据库无法打开,进一步检查alert日志

2025-10-14T21:38:22.889985+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2025-10-14T21:38:22.909799+08:00
Starting background process SMCO
2025-10-14T21:38:22.930114+08:00
SMCO started with pid=57, OS id=1630 
2025-10-14T21:38:23.000833+08:00
ARC0 (PID:1616): Archived Log entry 3 added for T-1.S-3205 ID 0x6539a8b7 LAD:1
2025-10-14T21:38:23.524409+08:00
Undo initialization recovery: err:0 start: 19911054 end: 19911123 diff: 69 ms (0.1 seconds)
2025-10-14T21:38:23.826920+08:00
[26765] Successfully onlined Undo Tablespace 2.
Undo initialization online undo segments: err:0 start: 19911123 end: 19911426 diff: 303 ms (0.3 seconds)
Undo initialization finished serial:0 start:19911054 end:19911454 diff:400 ms (0.4 seconds)
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Verifying minimum file header compatibility for tablespace encryption..
Verifying file header compatibility for tablespace encryption completed for pdb 0
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is AL32UTF8
No Resource Manager plan active
2025-10-14T21:38:24.645423+08:00
Hex dump of (file 1, block 132585) in trace file /u01/app/diag/rdbms/orcl/ORCL/trace/ORCL_ora_26765.trc

Invalid temporary block relative dba: 0x004205e9 (file 1, block 132585)
Bad header found during buffer read
Data in bad block:
 type: 66 format: 2 rdba: 0x4c444242
 last change scn: 0x4c44.4242.4c444242 seq: 0x44 flg: 0x4c
 spare3: 0x4c44
 consistency value in tail: 0xfcc60601
 check value in block header: 0x4242
 computed block checksum: 0x9e0a

Reading datafile '/data/app/oracle/oradata/ORCL/system01.dbf' for corrupt data at rdba:0x004205e9(file 1,block 132585)
Reread (file 1, block 132585) found same corrupt data (no logical check)
2025-10-14T21:38:24.838291+08:00
Errors in file /u01/app/diag/rdbms/orcl/ORCL/trace/ORCL_ora_26765.trc  (incident=51502):
ORA-01578: ORACLE data block corrupted (file # 1, block # 132585)
ORA-01110: data file 1: '/data/app/oracle/oradata/ORCL/system01.dbf'
Incident details in: /u01/app/diag/rdbms/orcl/ORCL/incident/incdir_51502/ORCL_ora_26765_i51502.trc
2025-10-14T21:38:24.861113+08:00
Corrupt Block Found
         TIME STAMP (GMT) = 10/14/2025 21:38:23
         CONT = 0, TSN = 0, TSNAME = SYSTEM
         RFN = 1, BLK = 132585, RDBA = 4326889
         OBJN = 66, OBJD = 64, OBJECT = C_OBJ#_INTCOL#, SUBOBJECT = 
         SEGMENT OWNER = SYS, SEGMENT TYPE = Cluster Segment
2025-10-14T21:38:26.554220+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2025-10-14T21:38:26.769742+08:00
Errors in file /u01/app/diag/rdbms/orcl/ORCL/trace/ORCL_ora_26765.trc:
ORA-00604: error occurred at recursive SQL level 2
ORA-01578: ORACLE data block corrupted (file # 1, block # 132585)
ORA-01110: data file 1: '/data/app/oracle/oradata/ORCL/system01.dbf'
2025-10-14T21:38:26.769940+08:00
Errors in file /u01/app/diag/rdbms/orcl/ORCL/trace/ORCL_ora_26765.trc:
ORA-00604: error occurred at recursive SQL level 2
ORA-01578: ORACLE data block corrupted (file # 1, block # 132585)
ORA-01110: data file 1: '/data/app/oracle/oradata/ORCL/system01.dbf'
Error 604 happened during db open, shutting down database
Errors in file /u01/app/diag/rdbms/orcl/ORCL/trace/ORCL_ora_26765.trc  (incident=51503):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00604: error occurred at recursive SQL level 2
ORA-01578: ORACLE data block corrupted (file # 1, block # 132585)
ORA-01110: data file 1: '/data/app/oracle/oradata/ORCL/system01.dbf'
Incident details in: /u01/app/diag/rdbms/orcl/ORCL/incident/incdir_51503/ORCL_ora_26765_i51503.trc
Invalid temporary block relative dba: 0x004205e9 (file 1, block 132585)
Bad header found during validation
Data in bad block:
 type: 66 format: 2 rdba: 0x4c444242
 last change scn: 0x4c44.4242.4c444242 seq: 0x44 flg: 0x4c
 spare3: 0x4c44
 consistency value in tail: 0xfcc60601
 check value in block header: 0x4242
 computed block checksum: 0x9e0a

Reread of blocknum=132585, file=/data/app/oracle/oradata/ORCL/system01.dbf. found same corrupt data
Reread of blocknum=132585, file=/data/app/oracle/oradata/ORCL/system01.dbf. found same corrupt data
Reread of blocknum=132585, file=/data/app/oracle/oradata/ORCL/system01.dbf. found same corrupt data
Reread of blocknum=132585, file=/data/app/oracle/oradata/ORCL/system01.dbf. found same corrupt data
Reread of blocknum=132585, file=/data/app/oracle/oradata/ORCL/system01.dbf. found same corrupt data
Checker run found 1 new persistent data failures
2025-10-14T21:38:27.515191+08:00
opiodr aborting process unknown ospid (26765) as a result of ORA-603
2025-10-14T21:38:27.528063+08:00
ORA-603 : opitsk aborting process
License high water mark = 18
USER (ospid: (prelim)): terminating the instance due to ORA error 
2025-10-14T21:38:28.576271+08:00
Instance terminated by USER(prelim), pid = 26765

通过分析alert日志可以确定是由于file 1, block 132585损坏,对应的对象为C_OBJ#_INTCOL#,该对象是histgrm$表的cluster,非数据库必须核心对象,可以在启动的时候跳过该对象,启动数据库,然后设置event对该对象进行处理

SQL> alter database open;

Database altered.

SQL>  create table good_histgrm$ as select * from histgrm$;

Table created.

SQL> truncate cluster c_obj#_intcol#;

Cluster truncated.

SQL> insert into histgrm$ select * from good_histgrm$ ;

286071 rows created.

SQL> commit;

Commit complete.

SQL> select /*+ full(a) */ count(*) from histgrm$ a ;

  COUNT(*)
----------
    286071

SQL> select /*+ full(a) */ count(*) from histgrm$ a ;

  COUNT(*)
----------
    286071

至此整体上完成该库的恢复任务

ORA-600 kkkicreatecgmap:!efn3

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 kkkicreatecgmap:!efn3

raid故障恢复之后,数据库recover成功,但是open报ORA-03113: end-of-file on communication channel错误

SQL> recover database;
Media recovery complete.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 20394
Session ID: 191 Serial number: 3

对应的alert日志错误为ORA-600 [kkkicreatecgmap:!efn3]错误

ALTER DATABASE RECOVER  database
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 4 slaves
Mon Oct 20 18:51:06 2025
Recovery of Online Redo Log: Thread 1 Group 1 Seq 32119 Reading mem 0
  Mem# 0: /u01/oradata/redo01.log
Media Recovery Complete (orcl)
Completed: ALTER DATABASE RECOVER  database
Mon Oct 20 18:51:16 2025
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 3 processes
Started redo scan
Completed redo scan
 read 41 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 32119, block 34978
Recovery of Online Redo Log: Thread 1 Group 1 Seq 32119 Reading mem 0
  Mem# 0: /u01/oradata/redo01.log
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 32119, block 35061, scn 17375938230308
 0 data blocks read, 0 data blocks written, 41 redo k-bytes read
Mon Oct 20 18:51:16 2025
Thread 1 advanced to log sequence 32120 (thread open)
Thread 1 opened at log sequence 32120
  Current log# 2 seq# 32120 mem# 0: /u01/oradata/redo02.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Oct 20 18:51:16 2025
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Re-creating tempfile /u01/oradata/temp01.dbf
Database Characterset is ZHS16GBK
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x3999DC33][PC:0x2297750, kgegpa()+40][flags: 0x0, count: 1]
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x3999DC33][PC:0x229597B, kgebse()+279][flags: 0x2, count: 2]
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x3999DC33][PC:0x229597B, kgebse()+279][flags: 0x2, count: 2]
No Resource Manager plan active
Mon Oct 20 18:51:16 2025
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20286.trc  (incident=3649):
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_3649/orcl_dbrm_20286_i3649.trc
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20286.trc:
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
DBRM (ospid: 20286): terminating the instance due to error 56710
Instance terminated by DBRM, pid = 20286

对应的trace文件内容


----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
skdstdst()+36        call     kgdsdst()            000000000 ? 000000000 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   7FFD08773E78 ? 000000000 ?
ksedst1()+98         call     skdstdst()           000000000 ? 000000000 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
ksedst()+34          call     ksedst1()            000000000 ? 000000001 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbkedDefDump()+2736  call     ksedst()             000000000 ? 000000001 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
ksedmp()+36          call     dbkedDefDump()       000000003 ? 000000002 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
ksfdmp()+64          call     ksedmp()             000000003 ? 000000002 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbgexPhaseII()+1764  call     ksfdmp()             000000003 ? 000000002 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbgexProcessError()  call     dbgexPhaseII()       7F16CCF3E6F0 ? 7F16CA6F2598 ?
+2279                                              7FFD0877BC68 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbgeExecuteForError  call     dbgexProcessError()  7F16CCF3E6F0 ? 7F16CA6F2598 ?
()+83                                              000000001 ? 000000000 ?
                                                   7FFD00000000 ? 000000000 ?
dbgePostErrorKGE()+  call     dbgeExecuteForError  7F16CCF3E6F0 ? 7F16CA6F2598 ?
1615                          ()                   000000001 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbkePostKGE_kgsf()+  call     dbgePostErrorKGE()   000000000 ? 7F16CA560040 ?
63                                                 000000258 ? 7F16CA6F2598 ?
                                                   000000000 ? 000000000 ?
kgeadse()+383        call     dbkePostKGE_kgsf()   00A99D360 ? 7F16CA560040 ?
                                                   000000258 ? 7F16CA6F2598 ?
                                                   000000000 ? 000000000 ?
kgerinv_internal()+  call     kgeadse()            00A99D360 ? 7F16CA560040 ?
45                                                 000000258 ? 000000000 ?
                                                   000000000 ? 000000000 ?
kgerinv()+33         call     kgerinv_internal()   00A99D360 ? 7F16CA560040 ?
                                                   877420000000000 ? 000000258 ?
                                                   000000000 ? 000000000 ?
kgeasnmierr()+143    call     kgerinv()            00A99D360 ? 7F16CA560040 ?
                                                   877420000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
kkkicreatecgmap()+3  call     kgeasnmierr()        00A99D360 ? 7F16CA560040 ?
125                                                877420000000000 ? 000000000 ?
                                                   000000000 ? 00000057B ?
kskirefreshcgmap()+  call     kkkicreatecgmap()    121DC3050 ? 7F16CA560040 ?
104                                                877420000000000 ? 000000000 ?
                                                   000000000 ? 00000057B ?
kskreload()+1150     call     kskirefreshcgmap()   121DC3050 ? 7F16CA560040 ?
                                                   877420000000000 ? 000000000 ?
                                                   000000000 ? 00000057B ?
kskdbrmpa()+378      call     kskreload()          7FFD0877D428 ? 000000001 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
ksbabs()+465         call     kskdbrmpa()          7FFD0877D418 ? 000000058 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
ksbrdp()+923         call     ksbabs()             7FFD0877D418 ? 000000058 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
opirip()+618         call     ksbrdp()             7FFD0877D418 ? 000000058 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
opidrv()+598         call     opirip()             000000032 ? 000000004 ?
                                                   7FFD0877E598 ? 000000001 ?
                                                   000000000 ? 00000020A ?
sou2o()+98           call     opidrv()             000000032 ? 000000004 ?
                                                   7FFD0877E598 ? 000000001 ?
                                                   000000000 ? 00000020A ?
opimai_real()+261    call     sou2o()              7FFD0877E570 ? 000000032 ?
                                                   000000004 ? 7FFD0877E598 ?
                                                   000000000 ? 00000020A ?
ssthrdmain()+209     call     opimai_real()        000000000 ? 7FFD0877E760 ?
                                                   000000004 ? 7FFD0877E598 ?
                                                   000000000 ? 00000020A ?
main()+196           call     ssthrdmain()         000000003 ? 7FFD0877E760 ?
                                                   000000001 ? 000000000 ?
                                                   000000000 ? 00000020A ?
__libc_start_main()  call     main()               000000003 ? 7FFD0877E900 ?
+253                                               000000001 ? 000000000 ?
                                                   000000000 ? 00000020A ?
_start()+36          call     __libc_start_main()  0009D3D94 ? 000000001 ?
                                                   7FFD0877E8F8 ? 000000000 ?
                                                   000000000 ? 00000020A ?

虽然ORA-600 kkkicreatecgmap:!efn3没有见过但是数据库open过程中kgegpa、kgebse的故障还是遇到不少,大部分可能和undo有一定关系,处理undo问题之后,继续尝试open库,依旧报ORA-03113: end-of-file on communication channel

SQL> alter database Open;
ERROR:
ORA-03113: end-of-file on communication channel
Process ID: 20586
Session ID: 191 Serial number: 3

但是后台的alert日志已经改变ORA-600 4193,ORA-600 kkkicreatecgmap:!efn3错误

Mon Oct 20 18:54:39 2025
Thread 1 advanced to log sequence 32121 (thread open)
Thread 1 opened at log sequence 32121
  Current log# 3 seq# 32121 mem# 0: /u01/oradata/redo03.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Oct 20 18:54:39 2025
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_smon_20492.trc  (incident=4905):
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_4905/orcl_smon_20492_i4905.trc
No Resource Manager plan active
Mon Oct 20 18:54:39 2025
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20478.trc  (incident=4849):
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_4849/orcl_dbrm_20478_i4849.trc
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20478.trc:
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
DBRM (ospid: 20478): terminating the instance due to error 56710
Some DDE async actions failed or were cancelled
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_smon_20492.trc  (incident=4906):
ORA-00353: log corruption near block 8 change 17375938190767 time 10/03/2025 00:20:34
ORA-00312: online log 1 thread 1: '/u01/oradata/redo01.log'
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_4906/orcl_smon_20492_i4906.trc
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_smon_20492.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 8 change 17375938190767 time 10/03/2025 00:20:34
ORA-00312: online log 1 thread 1: '/u01/oradata/redo01.log'
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Instance terminated by DBRM, pid = 20478

虽然ORA-600 kkkicreatecgmap:!efn3还在,但是看到了比较熟悉的ORA-600 4193错误,处理undo异常回滚段,数据库open成功,重建undo,尝试导出数据,完成数据恢复任务.

补充说明,对于ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403]网络上没有任何资料,查询了mos发现一个有一点类似的报错信息
Bug 28167557 – bigscn_dbim_tm_def – trc – kkkdchkcriticalobj – ORA-700 [kkkdchkcriticalob (Doc ID 28167557.8)

Description
Information about new symptoms:
 
- Signaling function: kkkdchkcriticalobj (kkkd.c)
- Symptom: ORA-700 [kkkdchkcriticalobj:fail]
- Owner: SUELEE
- Special Run:  BIGSCN_DBIM_TM_DEF (owner: WAI-SZE.TAM, sr_id: 2709)
- Release: 19.1
- Job id: 22488605
- Base label:  RDBMS_MAIN_LINUX.X64_180607
 
ORA-600 [KKKICHKRMAPPRI:0=NR PEND] 
 ORA-600 kkkicreatecgmap:!group
 ORA-700 kkkdchkcriticalobj
  
 ORA-600 [KKKICHKRMAPPRI:0=NR PEND] 
 ORA-600 kkkicreatecgmap:!group
 ORA-700 kkkdchkcriticalobj
  
 
 REDISCOVERY INFORMATION:
 Symptoms are any of the following:
 ORA-600 [KKKICHKRMAPPRI:0=NR PEND] 
 ORA-600 kkkicreatecgmap:!group
 ORA-700 kkkdchkcriticalobj
  
 .
 WORKAROUND:
 None

由于这个是硬件故障恢复出来的数据文件（涉及磁盘坏道，磁盘顺序，磁盘在raid中均衡,raid的cache等因素可能会一起Oracle各种非常规问题）,分析原因意义不大,重点是快速解决问题,不做过多分析

分类目录归档：Oracle备份恢复