Oracle备份恢复 – 第4页

断电引起文件scn异常数据库恢复

由于异常断电，数据库最初启动报错

Fri Mar 01 08:41:17 2024
ALTER DATABASE   MOUNT
Successful mount of redo thread 1, with mount id 1865809648
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE   MOUNT
Fri Mar 01 08:41:24 2024
ALTER DATABASE OPEN
Errors in file /home/oracle/app/diag/rdbms/XFF/XFF/trace/XFF_ora_25243.trc:
ORA-01113: file 13 needs media recovery
ORA-01110: data file 13: '/data2/oracle/oradata/data/data00.dbf'
ORA-1113 signalled during: ALTER DATABASE OPEN...

经过应用厂商一系列操作，主要是如下操作

Fri Mar 01 11:10:56 2024
ALTER DATABASE RECOVER datafile 13
Media Recovery Start
Serial Media Recovery started
WARNING! Recovering data file 13 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
ORA-279 signalled during: ALTER DATABASE RECOVER datafile 13 ...
Fri Mar 01 11:11:09 2024
ALTER DATABASE RECOVER CONTINUE DEFAULT
Media Recovery Log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
Errors with log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
ORA-308 signalled during: ALTER DATABASE RECOVER CONTINUE DEFAULT ...
ALTER DATABASE RECOVER CONTINUE DEFAULT
Media Recovery Log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
Errors with log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
ORA-308 signalled during: ALTER DATABASE RECOVER CONTINUE DEFAULT ...
ALTER DATABASE RECOVER CANCEL
Media Recovery Canceled
Completed: ALTER DATABASE RECOVER CANCEL
Fri Mar 01 11:16:50 2024
db_recovery_file_dest_size of 10240 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Fri Mar 01 11:51:56 2024
Starting ORACLE instance (normal)
Fri Mar 01 12:11:35 2024
alter database datafile 13 offline
ORA-1145 signalled during: alter database datafile 13 offline...
Fri Mar 01 12:12:29 2024
alter database recover cancel
ORA-1112 signalled during: alter database recover cancel...
Fri Mar 01 12:13:24 2024
ALTER DATABASE RECOVER database until cancel
Media Recovery Start
started logmerger process
Fri Mar 01 12:13:24 2024
WARNING! Recovering data file 13 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 14 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 15 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 16 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 17 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 18 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 19 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 20 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 21 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
WARNING! Recovering data file 22 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
Parallel Media Recovery started with 48 slaves
ORA-279 signalled during: ALTER DATABASE RECOVER database until cancel ...
ALTER DATABASE RECOVER CONTINUE DEFAULT
Media Recovery Log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
Errors with log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
Errors in file /home/oracle/app/diag/rdbms/XFF/XFF/trace/XFF_pr00_38097.trc:
ORA-00308:cannot open archived log '/home/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-308 signalled during: ALTER DATABASE RECOVER CONTINUE DEFAULT ...
ALTER DATABASE RECOVER CONTINUE DEFAULT
Media Recovery Log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
Errors with log /home/oracle/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc
Errors in file /home/oracle/app/diag/rdbms/XFF/XFF/trace/XFF_pr00_38097.trc:
ORA-00308:cannot open archived log '/home/app/flash_recovery_area/XFF/archivelog/2024_03_01/o1_mf_1_153563_%u_.arc'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-308 signalled during: ALTER DATABASE RECOVER CONTINUE DEFAULT ...
ALTER DATABASE RECOVER CANCEL
Signalling error 1152 for datafile 1!
Errors in file /home/oracle/app/diag/rdbms/XFF/XFF/trace/XFF_pr00_38097.trc:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1: '/data1/oracle/oradata/XFF/system01.dbf'
Slave exiting with ORA-1547 exception
Errors in file /home/oracle/app/diag/rdbms/XFF/XFF/trace/XFF_pr00_38097.trc:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1: '/data1/oracle/oradata/XFF/system01.dbf'
ORA-10879 signalled during: ALTER DATABASE RECOVER CANCEL ...
Fri Mar 01 13:23:05 2024
ALTER DATABASE DATAFILE '/data2/oracle/oradata/data/data00.dbf' OFFLINE DROP
Completed: ALTER DATABASE DATAFILE '/data2/oracle/oradata/data/data00.dbf' OFFLINE DROP

接手现场之后,尝试单个文件recover操作

SQL> recover datafile 1;
ORA-00283: recovery session canceled due to errors
ORA-00264: no recovery required
SQL> recover datafile 2;
Media recovery complete.
SQL> recover datafile 3;
Media recovery complete.
SQL> recover datafile 4;
Media recovery complete.
SQL> recover datafile 5;
Media recovery complete.
SQL> recover datafile 6,7,8,9,10;
Media recovery complete.
SQL> recover datafile 11;
Media recovery complete.
SQL> recover datafile 12;
Media recovery complete.
SQL> recover datafile 13;
ORA-00279: change 1474236715 generated at 02/29/2024 17:13:00 needed for thread 1
ORA-00289: suggestion : /home/oracle/app/program/11g/dbs/arch1_153563_1136037378.dbf
ORA-00280: change 1474236715 for thread 1 is in sequence #153563


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}

ORA-00308: cannot open archived log '/home/oracle/app/program/11g/dbs/arch1_153563_1136037378.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3


SQL> recover datafile 14;
ORA-00279: change 1474236715 generated at 02/29/2024 17:13:00 needed for thread 1
ORA-00289: suggestion : /home/oracle/app/program/11g/dbs/arch1_153563_1136037378.dbf
ORA-00280: change 1474236715 for thread 1 is in sequence #153563


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}

ORA-00308: cannot open archived log '/home/oracle/app/program/11g/dbs/arch1_153563_1136037378.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

基于这样的情况,通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)检查发现/data2挂载点所有数据文件异常,由于以前的操作日志已经被清空无法判断原因,初步怀疑和这个挂载点本身有关系


这种情况直接使用bbed修改文件头,然后open库,再逻辑导出数据,完成本次数据恢复工作,参考类似文档
bbed 修改datafile header
使用bbed让rac中的sysaux数据文件online
当然这类故障也可以通过自研的Oracle Recovery Tools工具进行修复处理,类似文档:
Oracle Recovery Tools解决ORA-00279 ORA-00289 ORA-00280故障

又一例:ORA-600 kclchkblk_4和2662故障

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：又一例:ORA-600 kclchkblk_4和2662故障

有客户恢复请求:由于未知原因导致aix环境的rac两台主机同时重启之后数据库无法正常启动,初步判断是由于写丢失导致故障（ORA-00742 ORA-00353）

Wed Feb 21 09:23:06 2024
ALTER DATABASE OPEN
This instance was first to open
Abort recovery for domain 0
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_5767246.trc:
ORA-01113: file 32 needs media recovery
ORA-01110: data file 32: '+DATA/xff/datafile/x5sys_cs.dbf'
ORA-1113 signalled during: ALTER DATABASE OPEN...
Wed Feb 21 09:23:27 2024
ALTER DATABASE RECOVER  datafile '+DATA/xff/datafile/x5sys_cs.dbf'  
Media Recovery Start
Serial Media Recovery started
WARNING! Recovering data file 32 from a fuzzy backup. It might be an online
backup taken without entering the begin backup command.
Recovery of Online Redo Log: Thread 2 Group 14 Seq 48490 Reading mem 0
  Mem# 0: +DATA/xff/onlinelog/group_14.313.1060528521
Recovery of Online Redo Log: Thread 1 Group 7 Seq 64195 Reading mem 0
  Mem# 0: +DATA/xff/onlinelog/group_7.306.1060527979
Wed Feb 21 09:24:10 2024
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_5767246.trc:
ORA-00742: Log read detects lost write in thread %d sequence %d block %d
ORA-00334: archived log: '+DATA/xff/onlinelog/group_14.313.1060528521'
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_5767246.trc  (incident=336478):
ORA-00353: log corruption near block 139727 change 26346459680 time 02/20/2024 20:13:50
ORA-00312: online log 14 thread 2: '+DATA/xff/onlinelog/group_14.313.1060528521'

尝试屏蔽一致性强制拉库后数据库报ORA-600 kclchkblk_4
参考:ora-600 2662和ora-600 kclchkblk_4恢复、redo异常 ORA-600 kclchkblk_4 故障恢复

Wed Feb 21 09:55:26 2024
SMON: enabling cache recovery
Wed Feb 21 09:55:26 2024
Redo thread 2 internally disabled at seq 5 (CKPT)
Archived Log entry 112707 added for thread 2 sequence 4 ID 0xffffffffe144183b dest 1:
ARC0: Archiving disabled thread 2 sequence 5
Archived Log entry 112708 added for thread 2 sequence 5 ID 0xffffffffe144183b dest 1:
Wed Feb 21 09:55:28 2024
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6423264.trc  (incident=360479):
ORA-00600: internal error code, arguments: [kclchkblk_4], [6], [576721660], [6], [576702892]
Incident details in: /oracle/db/diag/rdbms/xff/xff1/incident/incdir_360479/xff1_ora_6423264_i360479.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Thread 1 advanced to log sequence 3 (LGWR switch)
  Current log# 7 seq# 3 mem# 0: +DATA/xff/onlinelog/group_7.306.1161510375
Archived Log entry 112709 added for thread 1 sequence 2 ID 0xffffffffe144183b dest 1:
Wed Feb 21 09:55:31 2024
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6423264.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kclchkblk_4], [6], [576721660], [6], [576702892]
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6423264.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kclchkblk_4], [6], [576721660], [6], [576702892]

后续处理中出现和这个错误类似的ORA-600 2662错误

Wed Feb 21 15:37:35 2024
SMON: enabling cache recovery
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6357664.trc  (incident=432423):
ORA-00600: internal error code, arguments: [2662], [6], [576742938], [6], [576834283], [12583104]
Incident details in: /oracle/db/diag/rdbms/xff/xff1/incident/incdir_432423/xff1_ora_6357664_i432423.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6357664.trc:
ORA-00600: internal error code, arguments: [2662], [6], [576742938], [6], [576834283], [12583104], [], [], [], [], [], []
Errors in file /oracle/db/diag/rdbms/xff/xff1/trace/xff1_ora_6357664.trc:
ORA-00600: internal error code, arguments: [2662], [6], [576742938], [6], [576834283], [12583104], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 6357664): terminating the instance due to error 600
Instance terminated by USER, pid = 6357664

通过对oracle scn进行修改,数据库open成功

SQL> recover database;
Media recovery complete.
SQL> 
SQL> 
SQL> 
SQL> oradebug setmypid
Statement processed.
SQL> oradebug DUMPvar SGA kcsgscn_
kcslf kcsgscn_ [700000000019B70, 700000000019BA0) = 00000000 00000000 00000000 00000000 
SQL> oradebug poke 
BEFORE: [700000000019B70, 700000000019B78) = 00000000 00000000
AFTER:  [700000000019B70, 700000000019B78) = 00000006 22710D2B
SQL> oradebug DUMPvar SGA kcsgscn_ 
kcslf kcsgscn_ [700000000019B70, 700000000019BA0) = 00000006 22710D2B 00000000 00000000 
SQL> alter database open;

Database altered.

后续检查发现obj$中的index异常（ORA-08102: index key not found, obj# 39）
类似文章:通过bbed修改obj$中dataobj$重现I_OBJ4索引报ORA-08102错误

SQL> truncate CLUSTER "SYS"."SMON_SCN_TO_TIME_AUX";

truncate CLUSTER "SYS"."SMON_SCN_TO_TIME_AUX"
                       *
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-08102: index key not found, obj# 39, file 1, block 967206 (2)

SQL> SQL> select object_name,object_type from dba_objects where object_id=39;

OBJECT_NAME                    OBJECT_TYPE
------------------------------ -------------------
I_OBJ4                         INDEX

对于此类问题使用非常规方法把obj$字典表进行重建（需要注意undo需要为自动管理方式,temp不能为空）,参考:
bootstrap$核心index(I_OBJ1,I_USER1,I_FILE#_BLOCK#,I_IND1,I_TS#,I_CDEF1等)异常恢复—ORA-00701错误解决

Oracle误删除数据文件恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle误删除数据文件恢复

有客户通过sftp误删除oracle数据文件,咨询我们是否可以恢复,通过远程上去检查,发现运气不错,数据库还没有crash,通过句柄找到被删除文件

oracle@cwgstestdb[testwctdb]/proc/20611/fd$ls -ltr
total 0
lr-x------ 1 oracle oinstall 64 Feb 20 14:03 9 -> /oracle/db19c/rdbms/mesg/oraus.msb
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 8 -> /oracle/db19c/dbs/lkTESTWCTDB
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 7 -> /oracle/db19c/dbs/hc_testwctdb.dat
lr-x------ 1 oracle oinstall 64 Feb 20 14:03 6 -> /var/lib/sss/mc/passwd
lr-x------ 1 oracle oinstall 64 Feb 20 14:03 5 -> /proc/20611/fd
lr-x------ 1 oracle oinstall 64 Feb 20 14:03 4 -> /oracle/db19c/rdbms/mesg/oraus.msb
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 305 -> /oradata/ftms_zx_test01_data8.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 304 -> /oradata/ftms_zx_test01_data7.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 303 -> /oradata/ftms_zx_test01_data6.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 302 -> '/oradata/ftms_zx_test01_data5.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 301 -> '/oradata/ftms_zx_test01_data4.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 300 -> '/oradata/ftms_zx_test01_data3.dbf (deleted)'
lr-x------ 1 oracle oinstall 64 Feb 20 14:03 3 -> /dev/null
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 299 -> '/oradata/ftms_zx_test01_data2.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 298 -> '/oradata/ftms_zx_test01_data1.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 297 -> '/oradata/ftms_zx_test01_data.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 296 -> /oradata/ftms_zx_test_data.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 295 -> '/oradata/TESTWCTDB/sd.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 294 -> /oradata/TESTWCTDB/ftms_cs3_jiamiceshi
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 293 -> /langchao/dumpdata/FTMS_CS_TDE.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 292 -> /oradata/ftms_zx_test01.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 291 -> /langchao/dumpdata/FTMS_CS_DATA4.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 290 -> '/oradata/ftms_zx_data5.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 289 -> /langchao/dumpdata/FTMS_CS_DATA3.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 288 -> /langchao/dumpdata/FTMS_CS_DATA2.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 287 -> /langchao/dumpdata/FTMS_JD_DATA2.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 286 -> '/oradata/LCBIPECDS _TEMP_DAT.DBF'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 285 -> '/oradata/rTB_MBFE_TEMP (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 284 -> '/oradata/TESTWCTDB/temp01.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 283 -> '/oradata/ftms_credit_data5.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 282 -> /oradata/ftmshtdata.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 281 -> '/oradata/dump_data/FTMS_CSBF_DATA.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 280 -> /langchao/dumpdata/FTMS_NEWBL2_DATA.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 279 -> /langchao/dumpdata/FTMS_CS_DATA.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 278 -> /oradata/LCBIPECDS_DAT.DBF
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 277 -> /oradata/rTB_MBFE
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 276 -> /oradata/udpcount_02.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 275 -> /oradata/udpcount_01.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 274 -> '/oradata/ftms_credit_data_6.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 273 -> /langchao/dumpdata/FTMS_JD_DATA.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 272 -> '/oradata/ftms_old.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 271 -> '/oradata/ftms_credit_data2.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 270 -> /langchao/dumpdata/PJDIP_DATA.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 269 -> '/oradata/ftms_credit_data.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 268 -> /langchao/dumpdata/FTMS_NEWBL_DATA.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 267 -> '/oradata/ftms_zx_data4.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 266 -> /langchao/dumpdata/QIANZHANG_DATA.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 265 -> '/oradata/ftms_zx_data3.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 264 -> '/oradata/ftms_zx_data2.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 263 -> '/oradata/ftms_zx_data.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 262 -> /langchao/dumpdata/FTMSDIP_DATA.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 261 -> /oradata/TESTWCTDB/users01.dbf
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 260 -> '/oradata/TESTWCTDB/undotbs01.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 259 -> '/oradata/TESTWCTDB/sysaux01.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 258 -> '/oradata/TESTWCTDB/system01.dbf (deleted)'
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 257 -> /oradata/TESTWCTDB/control02.ctl
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 256 -> /oradata/TESTWCTDB/control01.ctl
l-wx------ 1 oracle oinstall 64 Feb 20 14:03 2 -> /dev/null
lrwx------ 1 oracle oinstall 64 Feb 20 14:03 10 -> 'socket:[823411]'
l-wx------ 1 oracle oinstall 64 Feb 20 14:03 1 -> /dev/null
lr-x------ 1 oracle oinstall 64 Feb 20 14:03 0 -> /dev/null

查询数据文件大小（被删除的文件文件大小通过v$datafile查询为0）

SQL> select name,bytes/1024/1024/1024 from v$datafile;

NAME                                                                             BYTES/1024/1024/1024
-------------------------------------------------------------------------------- --------------------
/oradata/TESTWCTDB/system01.dbf                                                                     0
/oradata/TESTWCTDB/sysaux01.dbf                                                                     0
/oradata/TESTWCTDB/undotbs01.dbf                                                                    0
/oradata/TESTWCTDB/users01.dbf                                                             .004882813
/langchao/dumpdata/FTMSDIP_DATA.dbf                                                                 3
/oradata/ftms_zx_data.dbf                                                                           0
/oradata/ftms_zx_data2.dbf                                                                          0
/oradata/ftms_zx_data3.dbf                                                                          0
/langchao/dumpdata/QIANZHANG_DATA.dbf                                                               5
/oradata/ftms_zx_data4.dbf                                                                          0
/langchao/dumpdata/FTMS_NEWBL_DATA.dbf                                                             30
/oradata/ftms_credit_data.dbf                                                                       0
/langchao/dumpdata/PJDIP_DATA.dbf                                                                  20
/oradata/ftms_credit_data2.dbf                                                                      0
/oradata/ftms_old.dbf                                                                               0
/langchao/dumpdata/FTMS_JD_DATA.dbf                                                                15
/oradata/ftms_credit_data_6.dbf                                                                     0
/oradata/udpcount_01.dbf                                                                            5
/oradata/udpcount_02.dbf                                                                            5
/oradata/rTB_MBFE                                                                              .03125
/oradata/LCBIPECDS_DAT.DBF                                                                         .5
/langchao/dumpdata/FTMS_CS_DATA.dbf                                                                30
/langchao/dumpdata/FTMS_NEWBL2_DATA.dbf                                                            30
/oradata/dump_data/FTMS_CSBF_DATA.dbf                                                               0
/oradata/ftmshtdata.dbf                                                                    .087890625
/oradata/ftms_credit_data5.dbf                                                                      0
/langchao/dumpdata/FTMS_JD_DATA2.dbf                                                                3
/langchao/dumpdata/FTMS_CS_DATA2.dbf                                                       31.9999847
/langchao/dumpdata/FTMS_CS_DATA3.dbf                                                               10
/oradata/ftms_zx_data5.dbf                                                                          0
/langchao/dumpdata/FTMS_CS_DATA4.dbf                                                        12.109375
/oradata/ftms_zx_test01.dbf                                                                19.0527344
/langchao/dumpdata/FTMS_CS_TDE.dbf                                                                  1
/oradata/TESTWCTDB/ftms_cs3_jiamiceshi                                                     .029296875
/oradata/TESTWCTDB/sd.dbf                                                                           0
/oradata/ftms_zx_test_data.dbf                                                             .009765625
/oradata/ftms_zx_test01_data.dbf                                                                    0
/oradata/ftms_zx_test01_data1.dbf                                                                   0
/oradata/ftms_zx_test01_data2.dbf                                                                   0
/oradata/ftms_zx_test01_data3.dbf                                                                   0
/oradata/ftms_zx_test01_data4.dbf                                                                   0
/oradata/ftms_zx_test01_data5.dbf                                                                   0
/oradata/ftms_zx_test01_data6.dbf                                                          12.5976563
/oradata/ftms_zx_test01_data7.dbf                                                          9.08203125
/oradata/ftms_zx_test01_data8.dbf                                                                6.25

45 rows selected.

把数据文件拷贝回来

cp /proc/20611/fd/302   /langchao/orabak/
cp /proc/20611/fd/301   /langchao/orabak/
cp /proc/20611/fd/300   /langchao/orabak/
cp /proc/20611/fd/299   /langchao/orabak/
cp /proc/20611/fd/298   /langchao/orabak/
cp /proc/20611/fd/297   /langchao/orabak/
cp /proc/20611/fd/295   /langchao/orabak/
cp /proc/20611/fd/290   /langchao/orabak/
cp /proc/20611/fd/285   /langchao/orabak/
cp /proc/20611/fd/284   /langchao/orabak/
cp /proc/20611/fd/283   /langchao/orabak/
cp /proc/20611/fd/281   /langchao/orabak/
cp /proc/20611/fd/274   /langchao/orabak/
cp /proc/20611/fd/272   /langchao/orabak/
cp /proc/20611/fd/271   /langchao/orabak/
cp /proc/20611/fd/269   /langchao/orabak/
cp /proc/20611/fd/267   /langchao/orabak/
cp /proc/20611/fd/265   /langchao/orabak/
cp /proc/20611/fd/264   /langchao/orabak/
cp /proc/20611/fd/263   /langchao/orabak/
cp /proc/20611/fd/260   /langchao/orabak/
cp /proc/20611/fd/259   /langchao/orabak/
cp /proc/20611/fd/258   /langchao/orabak/

由于涉及system表空间数据文件被删除,无法在open情况下直接操作,直接关闭数据库,启动到mount状态,重命名数据文件路径,recover数据文件,open库,恢复完成
参考以前类似恢复:
Solaris rm datafile recovery—利用句柄误删除数据文件恢复
如果数据库已经关闭,需要考虑以下类似恢复方式:
dbca删除库和rm删库恢复
 记录一次rm -rf 删除数据文件异常恢复

ORA-01595/ORA-600 4194处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-01595/ORA-600 4194处理

又一个客户由于服务器断电，导致oracle无法正常启动报ORA-600 4194错误

Sat Jan 20 13:45:22 2024
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 32 processes
Started redo scan
Completed redo scan
 read 31 KB redo, 5 data blocks need recovery
Started redo application at
 Thread 1: logseq 9, block 3
Recovery of Online Redo Log: Thread 1 Group 3 Seq 9 Reading mem 0
  Mem# 0: D:\ORACLE\ORADATA\xff\REDO03.LOG
Completed redo application of 0.02MB
Completed crash recovery at
 Thread 1: logseq 9, block 66, scn 81924188
 5 data blocks read, 5 data blocks written, 31 redo k-bytes read
Thread 1 advanced to log sequence 10 (thread open)
Thread 1 opened at log sequence 10
  Current log# 1 seq# 10 mem# 0: D:\ORACLE\ORADATA\xff\REDO01.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is AL32UTF8
No Resource Manager plan active
Errors in file d:\oracle\diag\rdbms\xff\xff\trace\xff_smon_5164.trc  (incident=384222):
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\oracle\diag\rdbms\xff\xff\incident\incdir_384222\xff_smon_5164_i384222.trc
Errors in file d:\oracle\diag\rdbms\xff\xff\trace\xff_ora_6832.trc  (incident=384270):
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\oracle\diag\rdbms\xff\xff\incident\incdir_384270\xff_ora_6832_i384270.trc
Doing block recovery for file 3 block 207
Resuming block recovery (PMON) for file 3 block 207
Block recovery from logseq 10, block 64 to scn 81924393
Recovery of Online Redo Log: Thread 1 Group 1 Seq 10 Reading mem 0
  Mem# 0: D:\ORACLE\ORADATA\xff\REDO01.LOG
Block recovery stopped at EOT rba 10.66.16
Block recovery completed at rba 10.66.16, scn 0.81924391
Doing block recovery for file 3 block 160
Resuming block recovery (PMON) for file 3 block 160
Block recovery from logseq 10, block 64 to scn 81924389
Recovery of Online Redo Log: Thread 1 Group 1 Seq 10 Reading mem 0
  Mem# 0: D:\ORACLE\ORADATA\xff\REDO01.LOG
Block recovery completed at rba 10.64.16, scn 0.81924390
Errors in file d:\oracle\diag\rdbms\xff\xff\trace\xff_smon_5164.trc:
ORA-01595: 释放区 (2) 回退段 (3) 时出错
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
Trace dumping is performing id=[cdmp_20240120134525]
Doing block recovery for file 3 block 3857
Resuming block recovery (PMON) for file 3 block 3857
Block recovery from logseq 10, block 64 to scn 81924396
Recovery of Online Redo Log: Thread 1 Group 1 Seq 10 Reading mem 0
  Mem# 0: D:\ORACLE\ORADATA\xff\REDO01.LOG
Block recovery stopped at EOT rba 10.66.16
Block recovery completed at rba 10.66.16, scn 0.81924391
Doing block recovery for file 3 block 272
Resuming block recovery (PMON) for file 3 block 272
Block recovery from logseq 10, block 64 to scn 81924390
Recovery of Online Redo Log: Thread 1 Group 1 Seq 10 Reading mem 0
  Mem# 0: D:\ORACLE\ORADATA\xff\REDO01.LOG
Block recovery completed at rba 10.66.16, scn 0.81924391
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Sat Jan 20 13:45:26 2024
QMNC started with pid=56, OS id=8052 
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0x66F8B134] [PC:0x93C7D88, kgegpa()+38]
Dump file d:\oracle\diag\rdbms\xff\xff\trace\alert_xff.log
Doing block recovery for file 3 block 3857
Resuming block recovery (PMON) for file 3 block 3857
Block recovery from logseq 10, block 64 to scn 81924396
Recovery of Online Redo Log: Thread 1 Group 1 Seq 10 Reading mem 0
  Mem# 0: D:\ORACLE\ORADATA\xff\REDO01.LOG
Block recovery completed at rba 10.66.16, scn 0.81924399
Errors in file d:\oracle\diag\rdbms\xff\xff\trace\xff_mmon_7876.trc  (incident=396089):
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-06512: 在 "SYS.DBMS_HA_ALERTS_PRVT", line 310
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], 
Doing block recovery for file 3 block 3857
Resuming block recovery (PMON) for file 3 block 3857
Block recovery from logseq 10, block 64 to scn 81924396
Recovery of Online Redo Log: Thread 1 Group 1 Seq 10 Reading mem 0
  Mem# 0: D:\ORACLE\ORADATA\xff\REDO01.LOG
Block recovery completed at rba 10.66.16, scn 0.81924399
Errors in file d:\oracle\diag\rdbms\xff\xff\trace\xff_pmon_8876.trc  (incident=384102):
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
Sat Jan 20 13:45:39 2024
Errors in file d:\oracle\diag\rdbms\xff\xff\trace\xff_pmon_8876.trc:
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
PMON (ospid: 8876): terminating the instance due to error 472
Instance terminated by PMON, pid = 8876

这个故障比较简单,由于undo回滚异常导致,对异常的undo进行处理,数据库正常open,重建undo表空间导出数据,完成本次恢复工作

从ORA-00283 ORA-16433报错开始恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：从ORA-00283 ORA-16433报错开始恢复

接手一个客户无法正常启动的故障数据库，尝试recover 报ORA-00283 ORA-16433错误

[oracle@xff trace]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sat Jan 27 04:46:23 2024

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


???:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> show pdbs;
SQL> select open_mode from v$database;

OPEN_MODE
--------------------
MOUNTED

SQL>
SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-16433: The database must be opened in read/write mode

通过对控制文件进行处理,再次尝试recover库

SQL> recover database;
ORA-00399: corrupt change description in redo log
ORA-00353: log corruption near block 134877 change 3249721295 time 01/27/2024 00:21:05
ORA-00312: online log 1 thread 1:'/u01/app/oracle/oradata/xff/redo01.log'

由于redo和数据文件不匹配,无法正常recover库,尝试强制打开库报ORA-600 2662错误

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [0], [3249721308], [0],[3249730440], [16777344],[],[],[],[],[],[]
ORA-00600: internal error code, arguments: [2662], [0], [3249721307], [0],[3249730440], [16777344],[],[],[],[],[],[]
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [0], [3249721303], [0],[3249730440], [16777344],[],[],[],[],[],[]
Process ID: 117336
Session ID: 1146 Serial number: 11764

基于这种错误,尝试oradebug修改scn

SQL> oradebug setmypid
oradebug DUMPvar SGA kcsgscn_
Statement processed.
SQL> kcslf kcsgscn_ [06001FBB0, 06001FBE0) = 00000000 00000000 00000000 00000000 00000000 
SQL> oradebug poke 0x06001FBB0 4 0x10000000
oradebug DUMPvar SGA kcsgscn_
ORA-32521: error parsing ORADEBUG command:

发现报ORA-32521错误,证明常规的oradebug方法无法修改scn,参考相关文章:
oradebug poke ORA-32521/ORA-32519故障解决
第一次通过其他方法处理,由于计算失误导致数据库启动报ORA-600 2252错误

SQL> ALTER DATABASE OPEN RESETLOGS;
ALTER DATABASE OPEN RESETLOGS
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [2252], [45264], [0], [11641],[3340959744], [],[],[],[],[],[]

该错误是相关文章参考:
记录一次ORA-00600[2252]故障解决
 ORA-00600: internal error code, arguments: [2252], [3987]
主机断电系统回到N年前数据库报ORA-600 kcm_headroom_warn_1错误
处理正确的scn值之后,数据库open成功,然后逻辑方式导出数据,恢复工作完成

SQL> alter database open ;

Database altered.

近期又遇到ORA-600 16703和ORA-702故障

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：近期又遇到ORA-600 16703和ORA-702故障

比较常见的由于软件注入恶意代码或者被人恶意破坏的常见故障ORA-600 16703和ORA-702故障

上面的故障是由于bootstrap$表被删除导致，以前处理过类似case:
恶意删除bootstrap$导致数据库无法正常启动
恶意删除bootstrap$导致数据库无法正常启动
2019恢复第一单—ORA-704-ORA-702恢复
ORA-00702: bootstrap verison ” inconsistent with version ’8.0.0.0.0′

该故障的原因是一般是由于在互联网上下载了带恶意脚本的oracle安装介质,300天之后数据库重启会导致tab$被删除,参见:
ORA-600 16703故障解析—tab$表被清空
警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703
再次提醒oracle使用者:
1. 不要使用从第三方网站下载非官方的数据库访问软件（发生过被注入恶意脚本的plsql dev案例:plsql dev引起的数据库被黑勒索比特币实现原理分析和解决方案）
2. 从官方下载oracle 安装程序和补丁程序（防止被注入恶意脚本,参考:警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703）

ORA-01033: ORACLE initialization or shutdown in progress 故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-01033: ORACLE initialization or shutdown in progress 故障处理

客户反馈数据库使用plsql dev登录报ORA-01033: ORACLE initialization or shutdown in progress的错误

出现该错误一般是由于数据库没有正常open成功,查看oracle 告警日志发现

Mon Jan 22 16:55:50 2024
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: alter database mount exclusive
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 139 KB redo, 70 data blocks need recovery
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_7792.trc  (incident=20565):
ORA-00600: ??????, ??: [kcratr_nab_less_than_odr], [1], [1916], [28210], [28222], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_20565\orcl_ora_7792_i20565.trc
Mon Jan 22 16:55:57 2024
Trace dumping is performing id=[cdmp_20240122165557]
Aborting crash recovery due to error 600
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_7792.trc:
ORA-00600: ??????, ??: [kcratr_nab_less_than_odr], [1], [1916], [28210], [28222], [], [], [], [], [], [], []
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_7792.trc:
ORA-00600: ??????, ??: [kcratr_nab_less_than_odr], [1], [1916], [28210], [28222], [], [], [], [], [], [], []
ORA-600 signalled during: alter database open...

这个错误比较常见,解决起来比较简单,参考:
kcratr_nab_less_than_odr
12c启动报kcratr_nab_less_than_odr
又一例ORA-600 kcratr_nab_less_than_odr
在恢复过程中中还遇到了ORA-00700 kcrf_split_brain_error错误,但是没有影响数据库open

Mon Jan 22 20:13:55 2024
alter database open
Beginning crash recovery of 1 threads
Started redo application at
 Thread 1: logseq 1916, block 27931
Recovery of Online Redo Log: Thread 1 Group 2 Seq 1916 Reading mem 0
  Mem# 0: D:\TEMP\ORCL\REDO02.LOG
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 1916, block 28210, scn 43957072
 0 data blocks read, 0 data blocks written, 139 redo k-bytes read
Errors in file d:\app\xifenfei\diag\rdbms\orcl\orcl\trace\orcl_ora_6104.trc  (incident=15729):
ORA-00700: 软内部错误, 参数: [kcrf_split_brain_error], [1], [1916], [28222], [28209], [4], [], [], [], [], [], []
Incident details in: d:\app\xifenfei\diag\rdbms\orcl\orcl\incident\incdir_15729\orcl_ora_6104_i15729.trc
Mon Jan 22 20:13:56 2024
Trace dumping is performing id=[cdmp_20240122201356]
Mon Jan 22 20:13:56 2024
Thread 1 advanced to log sequence 1917 (thread open)
Thread 1 opened at log sequence 1917
  Current log# 3 seq# 1917 mem# 0: D:\TEMP\ORCL\REDO03.LOG
Successful open of redo thread 1
Mon Jan 22 20:13:56 2024
SMON: enabling cache recovery
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is AL32UTF8
replication_dependency_tracking turned off (no async multimaster replication found)
WARNING: AQ_TM_PROCESSES is set to 0. System operation                     might be adversely affected.
Completed: alter database open

至此数据库open成功但是dbv检测system有很多坏块需要分析处理

C:\Users\XIFENFEI>dbv file=d:/temp/orcl/system01.dbf

DBVERIFY: Release 11.2.0.1.0 - Production on 星期一 1月 22 21:07:18 2024

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = D:\TEMP\ORCL\SYSTEM01.DBF
页 106278 流入 - 很可能是介质损坏
Corrupt block relative dba: 0x00419f26 (file 1, block 106278)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x00419f26
 last change scn: 0x0000.01410f78 seq: 0x2 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x00000000
 check value in block header: 0xbf11
 computed block checksum: 0xaf18

页 106279 标记为损坏
Corrupt block relative dba: 0x00419f27 (file 1, block 106279)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x00000000
 check value in block header: 0xbbb2
 block checksum disabled

页 106280 标记为损坏
Corrupt block relative dba: 0x00419f28 (file 1, block 106280)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x00000000
 check value in block header: 0xbbb2
 block checksum disabled

页 106281 标记为损坏
Corrupt block relative dba: 0x00419f29 (file 1, block 106281)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x00000000
 check value in block header: 0xbbb2
 block checksum disabled

页 106282 标记为损坏
Corrupt block relative dba: 0x00419f2a (file 1, block 106282)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x00000000
 check value in block header: 0xbbb2
 block checksum disabled

页 106283 标记为损坏
Corrupt block relative dba: 0x00419f2b (file 1, block 106283)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x00000000
 check value in block header: 0xbbb2
 block checksum disabled

页 106284 标记为损坏
Corrupt block relative dba: 0x00419f2c (file 1, block 106284)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x00000000
 check value in block header: 0xbbb2
 block checksum disabled

页 106285 标记为损坏
Corrupt block relative dba: 0x00419f2d (file 1, block 106285)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x00000000
 check value in block header: 0xbbb2
 block checksum disabled

页 106286 标记为损坏
Corrupt block relative dba: 0x00419f2e (file 1, block 106286)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x2c310602
 check value in block header: 0xbbb2
 block checksum disabled

页 143094 流入 - 很可能是介质损坏
Corrupt block relative dba: 0x00422ef6 (file 1, block 143094)
Fractured block found during dbv:
Data in bad block:
 type: 6 format: 2 rdba: 0x00422ef6
 last change scn: 0x0000.028f863b seq: 0x2 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x00000000
 check value in block header: 0xda23
 computed block checksum: 0x4210

页 143095 标记为损坏
Corrupt block relative dba: 0x00422ef7 (file 1, block 143095)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x00000000
 check value in block header: 0xbbb2
 block checksum disabled

页 143096 标记为损坏
Corrupt block relative dba: 0x00422ef8 (file 1, block 143096)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x00000000
 check value in block header: 0xbbb2
 block checksum disabled

页 143097 标记为损坏
Corrupt block relative dba: 0x00422ef9 (file 1, block 143097)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x00000000
 check value in block header: 0xbbb2
 block checksum disabled

页 143098 标记为损坏
Corrupt block relative dba: 0x00422efa (file 1, block 143098)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x00000000
 check value in block header: 0xbbb2
 block checksum disabled

页 143099 标记为损坏
Corrupt block relative dba: 0x00422efb (file 1, block 143099)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x00000000
 check value in block header: 0xbbb2
 block checksum disabled

页 143100 标记为损坏
Corrupt block relative dba: 0x00422efc (file 1, block 143100)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x00000000
 check value in block header: 0xbbb2
 block checksum disabled

页 143101 标记为损坏
Corrupt block relative dba: 0x00422efd (file 1, block 143101)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x00000000
 check value in block header: 0xbbb2
 block checksum disabled

页 143102 标记为损坏
Corrupt block relative dba: 0x00422efe (file 1, block 143102)
Bad header found during dbv:
Data in bad block:
 type: 178 format: 3 rdba: 0xc8c9c1b6
 last change scn: 0x0000.0a0df8c7 seq: 0x0 flg: 0x00
 spare1: 0xbf spare2: 0xc9 spare3: 0xc9bf
 consistency value in tail: 0x8a780602
 check value in block header: 0xbbb2
 block checksum disabled



DBVERIFY - 验证完成

检查的页总数: 152320
处理的页总数 (数据): 115189
失败的页总数 (数据): 0
处理的页总数 (索引): 13086
失败的页总数 (索引): 0
处理的页总数 (其他): 9741
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 14286
标记为损坏的总页数: 18
流入的页总数: 2
加密的总页数        : 0
最高块 SCN            : 44036082 (0.44036082)

通过分析aud$的extent,确认这些坏块全部属于该对象

SQL> select block_id,blocks from dba_extents where segment_name='AUD$';

  BLOCK_ID     BLOCKS
---------- ----------
…………
    102016       1024
    103040       1024
    104064       1024
    105088       1024
    106112       1024
…………
    141056       1024
    142080       1024
    143104       1024

已选择124行。

处理方法比较简单,直接truncate aud$表即可

存储故障,强制拉库报ORA-600 kcbzib_kcrsds_1处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：存储故障,强制拉库报ORA-600 kcbzib_kcrsds_1处理

硬件故障,客户自行强制resetlogs库,报ORA-600 kcbzib_kcrsds_1错误

2024-01-17T17:30:33.094367+08:00
alter database open resetlogs
2024-01-17T17:30:33.105461+08:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 16456823130015 time 
.... (PID:1584): Clearing online redo logfile 1 /u01/app/oracle/oradata/XFF/redo01.log
.... (PID:1584): Clearing online redo logfile 2 /u01/app/oracle/oradata/XFF/redo02.log
.... (PID:1584): Clearing online redo logfile 3 /u01/app/oracle/oradata/XFF/redo03.log
Clearing online log 1 of thread 1 sequence number 1345
Clearing online log 2 of thread 1 sequence number 1346
Clearing online log 3 of thread 1 sequence number 1347
2024-01-17T17:30:43.054150+08:00
.... (PID:1584): Clearing online redo logfile 1 complete
.... (PID:1584): Clearing online redo logfile 2 complete
.... (PID:1584): Clearing online redo logfile 3 complete
Resetting resetlogs activation ID 259120030 (0xf71db9e)
Online log /u01/app/oracle/oradata/XFF/redo01.log: Thread 1 Group 1 was previously cleared
Online log /u01/app/oracle/oradata/XFF/redo02.log: Thread 1 Group 2 was previously cleared
Online log /u01/app/oracle/oradata/XFF/redo03.log: Thread 1 Group 3 was previously cleared
2024-01-17T17:30:43.201042+08:00
Setting recovery target incarnation to 3
2024-01-17T17:30:43.267669+08:00
Smart fusion block transfer is disabled:
  instance mounted in exclusive mode.
2024-01-17T17:30:43.282033+08:00
Crash Recovery excluding pdb 2 which was cleanly closed.
Endian type of dictionary set to little
2024-01-17T17:30:43.396061+08:00
Assigning activation ID 280673168 (0x10babb90)
Redo log for group 1, sequence 1 is not located on DAX storage
2024-01-17T17:30:43.433441+08:00
TT00 (PID:1652): Gap Manager starting
2024-01-17T17:30:43.526972+08:00
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /u01/app/oracle/oradata/XFF/redo01.log
Successful open of redo thread 1
2024-01-17T17:30:43.528058+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Stopping change tracking
2024-01-17T17:30:44.097557+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_1584.trc(incident=263984)(PDBNAME=CDB$ROOT):
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_263984/XFF_ora_1584_i263984.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2024-01-17T17:30:47.913013+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2024-01-17T17:30:48.228934+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_1584.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2024-01-17T17:30:48.229250+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_1584.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2024-01-17T17:30:48.229572+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_1584.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_1584.trc  (incident=263985) (PDBNAME=CDB$ROOT):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_263985/XFF_ora_1584_i263985.trc
2024-01-17T17:30:50.982998+08:00
opiodr aborting process unknown ospid (1584) as a result of ORA-603
2024-01-17T17:30:50.989089+08:00
ORA-603 : opitsk aborting process

这个错误处理过多次一般是由于scn异常导致,以前部分类似文章
ORA-600 kcbzib_kcrsds_1报错
 12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
 redo异常强制拉库报ORA-600 kcbzib_kcrsds_1修复
 Patch SCN工具一键恢复ORA-600 kcbzib_kcrsds_1
处理好该错误之后,数据库在open pdb的时候报ORA-600 4193错误

2024-01-18T09:59:18.211131+08:00
alter pluggable database all open
2024-01-18T09:59:18.213569+08:00
XFFPDB(4):Pluggable database XFFPDB opening in read write
2024-01-18T09:59:18.590332+08:00
QPI: opatch file present, opatch
QPI: qopiprep.bat file present
2024-01-18T09:59:18.701197+08:00
XFFPDB(4):SUPLOG: Initialize PDB SUPLOG SGA, old value 0x0, new value 0x18
XFFPDB(4):Autotune of undo retention is turned on. 
XFFPDB(4):Endian type of dictionary set to little
2024-01-18T09:59:18.842283+08:00
Buffer Cache Full DB Caching mode changing from FULL CACHING ENABLED to FULL CACHING DISABLED 
Full DB Caching disabled: DEFAULT_CACHE_SIZE should be at least 153531 MBs bigger than current size. 
2024-01-18T09:59:19.017859+08:00
XFFPDB(4):Undo initialization recovery: Parallel FPTR failed: start:233023254 end:233023260 diff:6 ms(0.0 seconds)
XFFPDB(4):Undo initialization recovery: err:0 start: 233023254 end: 233023286 diff: 32 ms (0.0 seconds)
2024-01-18T09:59:19.256942+08:00
XFFPDB(4):[2475] Successfully onlined Undo Tablespace 40.
XFFPDB(4):Undo initialization online undo segments: err:0 start: 233023286 end: 233023500 diff:214 ms(0.2 seconds)
XFFPDB(4):Undo initialization finished serial:0 start:233023228 end:233023508 diff:280 ms (0.3 seconds)
XFFPDB(4):Database Characterset for XFFPDB is ZHS16GBK
XFFPDB(4):*********************************************************************
XFFPDB(4):WARNING: The following temporary tablespaces in container(XFFPDB)
XFFPDB(4):         contain no files.
XFFPDB(4):         This condition can occur when a backup controlfile has
XFFPDB(4):         been restored.  It may be necessary to add files to these
XFFPDB(4):         tablespaces.  That can be done using the SQL statement:
XFFPDB(4): 
XFFPDB(4):         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
XFFPDB(4): 
XFFPDB(4):         Alternatively, if these temporary tablespaces are no longer
XFFPDB(4):         needed, then they can be dropped.
XFFPDB(4):           Empty temporary tablespace: TEMP
XFFPDB(4):*********************************************************************
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_2475.trc  (incident=392032) (PDBNAME=XFFPDB):
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
XFFPDB(4):Incident details in:/u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_392032/XFF_ora_2475_i392032.trc
XFFPDB(4):Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2024-01-18T09:59:20.079597+08:00
XFFPDB(4):*****************************************************************
XFFPDB(4):An internal routine has requested a dump of selected redo.
XFFPDB(4):This usually happens following a specific internal error, when
XFFPDB(4):analysis of the redo logs will help Oracle Support with the
XFFPDB(4):diagnosis.
XFFPDB(4):It is recommended that you retain all the redo logs generated (by
XFFPDB(4):all the instances) during the past 12 hours, in case additional
XFFPDB(4):redo dumps are required to help with the diagnosis.
XFFPDB(4):*****************************************************************
===========================================================
Dumping current patch information
===========================================================
Unable to obtain current patch information due to error: 20001
===========================================================
2024-01-18T09:59:31.793666+08:00
XFFPDB(4):Flush retried for xcb 0x179400620, pmd 0x174a2f6c0
XFFPDB(4):Doing block recovery for file 31 block 1239
2024-01-18T09:59:31.805351+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_2475.trc  (incident=392034)(PDBNAME=XFFPDB):
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
XFFPDB(4):Incident details in:/u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_392034/XFF_ora_2475_i392034.trc
XFFPDB(4):Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2024-01-18T09:59:32.009996+08:00
XFFPDB(4):*****************************************************************
XFFPDB(4):An internal routine has requested a dump of selected redo.
XFFPDB(4):This usually happens following a specific internal error, when
XFFPDB(4):analysis of the redo logs will help Oracle Support with the
XFFPDB(4):diagnosis.
XFFPDB(4):It is recommended that you retain all the redo logs generated (by
XFFPDB(4):all the instances) during the past 12 hours, in case additional
XFFPDB(4):redo dumps are required to help with the diagnosis.
XFFPDB(4):*****************************************************************
Incident details in: /u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_392384/XFF_mz00_2879_i392384.trc
2024-01-18T09:59:36.918296+08:00
Reread of blocknum=142165, file=/u01/app/oracle/oradata/XFF/system01.dbf. found same corrupt data
Reread of blocknum=142165, file=/u01/app/oracle/oradata/XFF/system01.dbf. found same corrupt data
Reread of blocknum=142165, file=/u01/app/oracle/oradata/XFF/system01.dbf. found same corrupt data
Reread of blocknum=142165, file=/u01/app/oracle/oradata/XFF/system01.dbf. found same corrupt data
Reread of blocknum=142165, file=/u01/app/oracle/oradata/XFF/system01.dbf. found same corrupt data
2024-01-18T09:59:41.146850+08:00
XFFPDB(4):Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_2475.trc:
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
2024-01-18T09:59:45.434605+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_2475.trc  (incident=407714):
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_407714/XFF_ora_2475_i407714.trc
2024-01-18T09:59:45.439083+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_407714/XFF_ora_2475_i407714.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
2024-01-18T09:59:48.575971+08:00
XFFPDB(4):pdb open recovery: pdbid=4 log=255 acquired_ip=1

这个相对比较简单,对异常undo进行处理,cdb和pdb都open成功,使用数据泵把数据迁移到新库,完成本次恢复(比较幸运,硬件故障的库直接迁移成功,没有报其他错误)

ORA-600 kcrf_resilver_log_1故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 kcrf_resilver_log_1故障处理

接手一个客户的数据库故障处理,最初数据库启动报ORA-600 kcrf_resilver_log_1错

Mon Jan 08 16:16:22 2024
ALTER DATABASE   MOUNT
Successful mount of redo thread 1, with mount id 2385308630
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE   MOUNT
Mon Jan 08 16:16:26 2024
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Started redo scan
Errors in file /u2/11G/base/diag/rdbms/xff/xff/trace/xff_ora_3350.trc  (incident=276167):
ORA-00600: internal error code, arguments: [kcrf_resilver_log_1], [0x1542C3A00], [2], , 
Incident details in:/u2/11G/base/diag/rdbms/xff/xff/incident/incdir_276167/xff_ora_3350_i276167.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Aborting crash recovery due to error 600
Errors in file /u2/11G/base/diag/rdbms/xff/xff/trace/xff_ora_3350.trc:
ORA-00600: internal error code, arguments: [kcrf_resilver_log_1], [0x1542C3A00], [2]
Errors in file /u2/11G/base/diag/rdbms/xff/xff/trace/xff_ora_3350.trc:
ORA-00600: internal error code, arguments: [kcrf_resilver_log_1], [0x1542C3A00], [2]
ORA-600 signalled during: ALTER DATABASE OPEN...

客户自行recover数据库之后报ORA-00283 ORA-00742 ORA-00312错

Mon Jan 08 17:05:34 2024
ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 16 slaves
Mon Jan 08 17:05:35 2024
Recovery of Online Redo Log: Thread 1 Group 2 Seq 63899 Reading mem 0
  Mem# 0: /u2/11G/data/xff/redo02.log
Media Recovery failed with error 742
Errors in file /u2/11G/base/diag/rdbms/xff/xff/trace/xff_pr00_3857.trc:
ORA-00283: recovery session canceled due to errors
ORA-00742: Log read detects lost write in thread %d sequence %d block %d
ORA-00312: online log 2 thread 1: '/u2/11G/data/xff/redo02.log'
ORA-283 signalled during: ALTER DATABASE RECOVER  database  ...

客户强制打开数据库

Tue Jan 09 17:37:51 2024
ALTER DATABASE OPEN
Errors in file /u2/11G/base/diag/rdbms/xff/xff/trace/xff_ora_3501.trc:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open
ORA-1589 signalled during: ALTER DATABASE OPEN...
Tue Jan 09 17:43:45 2024
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 4418119911
Resetting resetlogs activation ID 2289128497 (0x88715431)
Online log /u2/11G/data/xff/redo01.log: Thread 1 Group 1 was previously cleared
Online log /u2/11G/data/xff/redo02.log: Thread 1 Group 2 was previously cleared
Online log /u2/11G/data/xff/redo03.log: Thread 1 Group 3 was previously cleared
Tue Jan 09 17:43:46 2024
Setting recovery target incarnation to 3
Tue Jan 09 17:43:46 2024
Assigning activation ID 2385405291 (0x8e2e656b)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /u2/11G/data/xff/redo01.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Jan 09 17:43:46 2024
SMON: enabling cache recovery
Errors in file /u2/11G/base/diag/rdbms/xff/xff/trace/xff_ora_3501.trc  (incident=492171):
ORA-00600: internal error code, arguments: [2662], [1], [123152622], [1], [123176387], [12583040], []
Incident details in: /u2/11G/base/diag/rdbms/xff/xff/incident/incdir_492171/xff_ora_3501_i492171.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u2/11G/base/diag/rdbms/xff/xff/trace/xff_ora_3501.trc:
ORA-00600: internal error code, arguments: [2662], [1], [123152622], [1], [123176387], [12583040], []
Errors in file /u2/11G/base/diag/rdbms/xff/xff/trace/xff_ora_3501.trc:
ORA-00600: internal error code, arguments: [2662], [1], [123152622], [1], [123176387], [12583040], []
Error 600 happened during db open, shutting down database
USER (ospid: 3501): terminating the instance due to error 600
Instance terminated by USER, pid = 3501
ORA-1092 signalled during: alter database open resetlogs...

这个故障相对比较简单,修改数据库scn之后,即可open数据库,然后逻辑方式迁移数据到新库即可

ORA-00600: internal error code, arguments: [4193], [35191], [35263]

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-00600: internal error code, arguments: [4193], [35191], [35263]

有客户数据库由于磁盘空间满导致数据库异常,然后自行尝试强制拉库,结果数据库报ORA-00600: internal error code, arguments: [4193], [35191], [35263]错误,无法启动成功

[oracle@oracledb ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Jan 12 22:40:54 2024
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount;
ORACLE instance started.

Total System Global Area 1.0066E+10 bytes
Fixed Size		   12684768 bytes
Variable Size		 4261412864 bytes
Database Buffers	 5771362304 bytes
Redo Buffers		   20869120 bytes
Database mounted.
SQL> recover database;
Media recovery complete.
SQL> 
SQL> 
SQL> alter database open;      
alter database open 
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [4193], [35191], [35263], [], [],
[], [], [], [], [], [], []
Process ID: 2553131
Session ID: 862 Serial number: 11059

alert日志报错信息

2024-01-12T22:39:31.107781-05:00
Thread 1 advanced to log sequence 15 (thread open)
Redo log for group 3, sequence 15 is not located on DAX storage
2024-01-12T22:39:31.113072-05:00
TT00 (PID:2558545): Gap Manager starting
2024-01-12T22:39:31.140008-05:00
Thread 1 opened at log sequence 15
  Current log# 3 seq# 15 mem# 0: /opt/oracle/oradata/ORCLCDB/redo03.log
Successful open of redo thread 1
2024-01-12T22:39:31.140524-05:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
Undo initialization recovery: err:0 start: 33599386 end: 33599409 diff: 23 ms (0.0 seconds)
[2553131] Successfully onlined Undo Tablespace 2.
Undo initialization online undo segments: err:0 start: 33599409 end: 33599417 diff: 8 ms (0.0 seconds)
Undo initialization finished serial:0 start:33599386 end:33599418 diff:32 ms (0.0 seconds)
Verifying minimum file header compatibility for tablespace encryption for pdb 1..
Verifying file header compatibility for tablespace encryption completed for pdb 1
Database Characterset is AL32UTF8
2024-01-12T22:39:31.267662-05:00
ORA-00600: internal error code, arguments: [4193], [35191], [35263], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2024-01-12T22:39:31.271184-05:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Doing block recovery for file 4 block 234016
Resuming block recovery (PMON) for file 4 block 234016
Block recovery from logseq 15, block 66 to scn 0x0000000000000000
2024-01-12T22:39:31.895999-05:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 15 Reading mem 0
  Mem# 0: /opt/oracle/oradata/ORCLCDB/redo03.log
Block recovery completed at rba 0.0.0, scn 0x000000060fd94e6c
Doing block recovery for file 4 block 144
Resuming block recovery (PMON) for file 4 block 144
Block recovery from logseq 15, block 66 to scn 0x000000060fd94ed2
2024-01-12T22:39:31.899101-05:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 15 Reading mem 0
  Mem# 0: /opt/oracle/oradata/ORCLCDB/redo03.log
Block recovery completed at rba 15.68.16, scn 0x000000060fd94ed3
Non-fatal internal error happened while SMON was doing shrinking of rollback segments.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
2024-01-12T22:39:31.960874-05:00
Errors in file /opt/oracle/diag/rdbms/orclcdb/ORCLCDB/trace/ORCLCDB_ora_2553131.trc(incident=159324)(PDBNAME=CDB$ROOT)
ORA-00600: internal error code, arguments: [4193], [35191], [35263], [], [], [], [], [], [], [], [], []
Incident details in: /opt/oracle/diag/rdbms/orclcdb/ORCLCDB/incident/incdir_159324/ORCLCDB_ora_2553131_i159324.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2024-01-12T22:39:32.042150-05:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2024-01-12T22:39:32.985037-05:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Doing block recovery for file 4 block 234016
Resuming block recovery (PMON) for file 4 block 234016
Block recovery from logseq 15, block 66 to scn 0x000000060f9df0d0
2024-01-12T22:39:33.021406-05:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 15 Reading mem 0
  Mem# 0: /opt/oracle/oradata/ORCLCDB/redo03.log
Block recovery completed at rba 0.0.0, scn 0x000000060fd94e6c
Doing block recovery for file 4 block 144
Resuming block recovery (PMON) for file 4 block 144
Block recovery from logseq 15, block 66 to scn 0x000000060fd94ed3
2024-01-12T22:39:33.023883-05:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 15 Reading mem 0
  Mem# 0: /opt/oracle/oradata/ORCLCDB/redo03.log
Block recovery completed at rba 15.70.16, scn 0x000000060fd94ed4
2024-01-12T22:39:33.027444-05:00
Errors in file /opt/oracle/diag/rdbms/orclcdb/ORCLCDB/trace/ORCLCDB_ora_2553131.trc:
ORA-00600: internal error code, arguments: [4193], [35191], [35263], [], [], [], [], [], [], [], [], []
2024-01-12T22:39:33.027525-05:00
Errors in file /opt/oracle/diag/rdbms/orclcdb/ORCLCDB/trace/ORCLCDB_ora_2553131.trc:
ORA-00600: internal error code, arguments: [4193], [35191], [35263], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
2024-01-12T22:39:33.048575-05:00
Errors in file /opt/oracle/diag/rdbms/orclcdb/ORCLCDB/trace/ORCLCDB_ora_2553131.trc(incident=159325)(PDBNAME=CDB$ROOT)
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [4193], [35191], [35263], [], [], [], [], [], [], [], [], []
Incident details in: /opt/oracle/diag/rdbms/orclcdb/ORCLCDB/incident/incdir_159325/ORCLCDB_ora_2553131_i159325.trc
2024-01-12T22:39:33.694774-05:00
opiodr aborting process unknown ospid (2553131) as a result of ORA-603
2024-01-12T22:39:33.713866-05:00
ORA-603 : opitsk aborting process
License high water mark = 12
USER (ospid: (prelim)): terminating the instance due to ORA error 
2024-01-12T22:39:34.734019-05:00
Instance terminated by USER(prelim), pid = 2553131

这个错误相对比较简单,一般是由于undo回滚段异常,对其进行规避,数据库open成功,然后重建新库迁移数据,完成本次恢复

分类目录归档：Oracle备份恢复