Oracle 23ai rm redo*.log恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:Oracle 23ai rm redo*.log恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

在oracle 23ai的pdb中创建用户和表,并且插入数据(不提交),在另外一个会话中abort库,并从os层面rm删除掉redo文件,模拟数据库当前redo丢失,数据库恢复
创建用户和表并插入数据

[oracle@xifenfei ~]$ sqlplus sys/oracle@free as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri May 3 15:40:55 2024
Version 23.4.0.24.05

Copyright (c) 1982, 2024, Oracle.  All rights reserved.


Connected to:
Oracle Database 23ai Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
Version 23.4.0.24.05

SQL> 
SQL> 
SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 FREEPDB1                       READ WRITE NO

SQL> alter session set container=FREEPDB1;

Session altered.

SQL> create user xff identified by oracle;

User created.

SQL> grant dba to xff ;

Grant succeeded.

SQL> conn xff/oracle@FREEPDB1
Connected.
SQL> create table t1 as select * from dba_objects;

Table created.


SQL> insert into t1 select *from t1;

75877 rows created.

SQL> /

151754 rows created.

另外一个会话中abort库

[oracle@xifenfei ~]$ sqlplus sys/oracle@free as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri May 3 15:43:30 2024
Version 23.4.0.24.05

Copyright (c) 1982, 2024, Oracle.  All rights reserved.


Connected to:
Oracle Database 23ai Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
Version 23.4.0.24.05

SQL> shutdown abort;
ORACLE instance shut down.
SQL> 
SQL> 
SQL> 
SQL> exit
Disconnected from Oracle Database 23ai Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
Version 23.4.0.24.05

操作系统层面rm -rf 删除redo

[oracle@xifenfei ~]$ cd $ORACLE_BASE/oradata
[oracle@xifenfei oradata]$ ls
FREE
[oracle@xifenfei oradata]$ cd FREE/
[oracle@192 FREE]$ ls
control01.ctl  FREEPDB1  redo01.log  redo03.log    system01.dbf  undotbs2.dbf
control02.ctl  pdbseed   redo02.log  sysaux01.dbf  temp01.dbf    users01.dbf
[oracle@xifenfei FREE]$ ls -ltr
total 2441036
drwxr-x---. 2 oracle     1000         85 May  1 16:49 pdbseed
-rw-r-----. 1 oracle oinstall   20979712 May  1 16:51 temp01.dbf
drwxr-x---. 2 oracle     1000        104 May  1 16:55 FREEPDB1
-rw-r-----. 1 oracle oinstall  209715712 May  3 15:23 redo01.log
-rw-r-----. 1 oracle oinstall  209715712 May  3 15:23 redo02.log
-rw-r-----. 1 oracle oinstall    7348224 May  3 15:23 users01.dbf
-rw-r-----. 1 oracle oinstall 1080041472 May  3 15:43 system01.dbf
-rw-r-----. 1 oracle oinstall  692068352 May  3 15:43 sysaux01.dbf
-rw-rw----. 1 oracle oinstall   52436992 May  3 15:43 undotbs2.dbf
-rw-r-----. 1 oracle oinstall  209715712 May  3 15:43 redo03.log
-rw-r-----. 1 oracle oinstall   18759680 May  3 15:43 control01.ctl
-rw-r-----. 1 oracle oinstall   18759680 May  3 15:43 control02.ctl
[oracle@xifenfei FREE]$ rm -rf redo0*
[oracle@192 FREE]$ ls -l redo*
ls: cannot access 'redo*': No such file or directory
[oracle@xifenfei FREE]$ 

尝试启动数据库,报ora-00313,ora-00312,ora-27037等错误

[oracle@xifenfei ~]$ sqlplus sys/oracle@free as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri May 3 15:44:17 2024
Version 23.4.0.24.05

Copyright (c) 1982, 2024, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount;
ORACLE instance started.

Total System Global Area 1603726344 bytes
Fixed Size                  5360648 bytes
Variable Size             671088640 bytes
Database Buffers          922746880 bytes
Redo Buffers                4530176 bytes
Database mounted.
SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-00313: open failed for members of log group 3 of thread 1
ORA-00312: online log 3 thread 1: '/opt/oracle/oradata/FREE/redo03.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7

由于所有redo均被删除(包含当前redo),因此只能强制resetlogs方式打开库,尝试打开数据库

SQL> shutdown immediate;
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
SQL> startup mount pfile='/tmp/pfile'
ORACLE instance started.

Total System Global Area 1603726344 bytes
Fixed Size                  5360648 bytes
Variable Size             671088640 bytes
Database Buffers          922746880 bytes
Redo Buffers                4530176 bytes
Database mounted.
SQL> recover database until cancel;
ORA-00279: change 4244588 generated at 05/03/2024 15:23:22 needed for thread 1
ORA-00289: suggestion :
/opt/oracle/product/23ai/dbhomeFree/dbs/arch1_6_1167842962.dbf
ORA-00280: change 4244588 for thread 1 is in sequence #6


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/opt/oracle/oradata/FREE/system01.dbf'


ORA-01112: media recovery not started

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by irrecoverable error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [],
[], [], [], [], [], [], []
Process ID: 5596
Session ID: 29 Serial number: 63204

数据库遇到常见的ORA-600 kcbzib_kcrsds_1错误,这类错误类似oracle在12c之前版本中的ORA-600 2662错误
ORA-600 kcbzib_kcrsds_1报错
存储故障,强制拉库报ORA-600 kcbzib_kcrsds_1处理
redo异常强制拉库报ORA-600 kcbzib_kcrsds_1修复
ORA-00603 ORA-01092 ORA-600 kcbzib_kcrsds_1
这种一般就是scn问题,通过修改数据库scn,数据库启动报ORA-16433错误

[oracle@xifenfei ~]$ sqlplus sys/oracle@free as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri May 3 15:49:00 2024
Version 23.4.0.24.05

Copyright (c) 1982, 2024, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area 1603726344 bytes
Fixed Size                  5360648 bytes
Variable Size             671088640 bytes
Database Buffers          922746880 bytes
Redo Buffers                4530176 bytes
SQL> alter database mount;

Database altered.

SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-16433: The database or pluggable database must be opened in read/write mode.


SQL> shutdown abort;
ORACLE instance shut down.

处理ctl,open数据库成功

SQL> startup nomount pfile='/tmp/pfile'
ORACLE instance started.

Total System Global Area 1603726344 bytes
Fixed Size                  5360648 bytes
Variable Size             671088640 bytes
Database Buffers          922746880 bytes
Redo Buffers                4530176 bytes
SQL> CREATE CONTROLFILE REUSE DATABASE "FREE" NORESETLOGS  NOARCHIVELOG
  2      MAXLOGFILES 16
  3      MAXLOGMEMBERS 3
  4      MAXDATAFILES 1024
  5      MAXINSTANCES 8
  6      MAXLOGHISTORY 292
  7  LOGFILE
  8    GROUP 1 '/opt/oracle/oradata/FREE/redo01.log'  SIZE 200M BLOCKSIZE 512,
  9    GROUP 2 '/opt/oracle/oradata/FREE/redo02.log'  SIZE 200M BLOCKSIZE 512,
 10    GROUP 3 '/opt/oracle/oradata/FREE/redo03.log'  SIZE 200M BLOCKSIZE 512
 11  -- STANDBY LOGFILE
 12  DATAFILE
 13    '/opt/oracle/oradata/FREE/system01.dbf',
 14    '/opt/oracle/oradata/FREE/pdbseed/system01.dbf',
 15    '/opt/oracle/oradata/FREE/sysaux01.dbf',
 16    '/opt/oracle/oradata/FREE/pdbseed/sysaux01.dbf',
 17    '/opt/oracle/oradata/FREE/users01.dbf',
 18    '/opt/oracle/oradata/FREE/pdbseed/undotbs01.dbf',
 19    '/opt/oracle/oradata/FREE/FREEPDB1/system01.dbf',
 20    '/opt/oracle/oradata/FREE/FREEPDB1/sysaux01.dbf',
 21    '/opt/oracle/oradata/FREE/FREEPDB1/undotbs01.dbf',
 22    '/opt/oracle/oradata/FREE/FREEPDB1/users01.dbf',
 23    '/opt/oracle/oradata/FREE/undotbs2.dbf'
 24  CHARACTER SET AL32UTF8
 25  ;

Control file created.

SQL> recover database;
Media recovery complete.
SQL> alter database open;

Database altered.

SQL> show pdbs;

          CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------------- ------------------------------ ---------- ----------
               2 PDB$SEED                       READ ONLY  NO
               3 FREEPDB1                       READ WRITE NO

至此当前数据库redo故障模拟和恢复基本完成

存储故障,强制拉库报ORA-600 kcbzib_kcrsds_1处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:存储故障,强制拉库报ORA-600 kcbzib_kcrsds_1处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

硬件故障,客户自行强制resetlogs库,报ORA-600 kcbzib_kcrsds_1错误

2024-01-17T17:30:33.094367+08:00
alter database open resetlogs
2024-01-17T17:30:33.105461+08:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 16456823130015 time 
.... (PID:1584): Clearing online redo logfile 1 /u01/app/oracle/oradata/XFF/redo01.log
.... (PID:1584): Clearing online redo logfile 2 /u01/app/oracle/oradata/XFF/redo02.log
.... (PID:1584): Clearing online redo logfile 3 /u01/app/oracle/oradata/XFF/redo03.log
Clearing online log 1 of thread 1 sequence number 1345
Clearing online log 2 of thread 1 sequence number 1346
Clearing online log 3 of thread 1 sequence number 1347
2024-01-17T17:30:43.054150+08:00
.... (PID:1584): Clearing online redo logfile 1 complete
.... (PID:1584): Clearing online redo logfile 2 complete
.... (PID:1584): Clearing online redo logfile 3 complete
Resetting resetlogs activation ID 259120030 (0xf71db9e)
Online log /u01/app/oracle/oradata/XFF/redo01.log: Thread 1 Group 1 was previously cleared
Online log /u01/app/oracle/oradata/XFF/redo02.log: Thread 1 Group 2 was previously cleared
Online log /u01/app/oracle/oradata/XFF/redo03.log: Thread 1 Group 3 was previously cleared
2024-01-17T17:30:43.201042+08:00
Setting recovery target incarnation to 3
2024-01-17T17:30:43.267669+08:00
Smart fusion block transfer is disabled:
  instance mounted in exclusive mode.
2024-01-17T17:30:43.282033+08:00
Crash Recovery excluding pdb 2 which was cleanly closed.
Endian type of dictionary set to little
2024-01-17T17:30:43.396061+08:00
Assigning activation ID 280673168 (0x10babb90)
Redo log for group 1, sequence 1 is not located on DAX storage
2024-01-17T17:30:43.433441+08:00
TT00 (PID:1652): Gap Manager starting
2024-01-17T17:30:43.526972+08:00
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /u01/app/oracle/oradata/XFF/redo01.log
Successful open of redo thread 1
2024-01-17T17:30:43.528058+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Stopping change tracking
2024-01-17T17:30:44.097557+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_1584.trc(incident=263984)(PDBNAME=CDB$ROOT):
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_263984/XFF_ora_1584_i263984.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2024-01-17T17:30:47.913013+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2024-01-17T17:30:48.228934+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_1584.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2024-01-17T17:30:48.229250+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_1584.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2024-01-17T17:30:48.229572+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_1584.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_1584.trc  (incident=263985) (PDBNAME=CDB$ROOT):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_263985/XFF_ora_1584_i263985.trc
2024-01-17T17:30:50.982998+08:00
opiodr aborting process unknown ospid (1584) as a result of ORA-603
2024-01-17T17:30:50.989089+08:00
ORA-603 : opitsk aborting process

这个错误处理过多次一般是由于scn异常导致,以前部分类似文章
ORA-600 kcbzib_kcrsds_1报错
12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
redo异常强制拉库报ORA-600 kcbzib_kcrsds_1修复
Patch SCN工具一键恢复ORA-600 kcbzib_kcrsds_1
处理好该错误之后,数据库在open pdb的时候报ORA-600 4193错误

2024-01-18T09:59:18.211131+08:00
alter pluggable database all open
2024-01-18T09:59:18.213569+08:00
XFFPDB(4):Pluggable database XFFPDB opening in read write
2024-01-18T09:59:18.590332+08:00
QPI: opatch file present, opatch
QPI: qopiprep.bat file present
2024-01-18T09:59:18.701197+08:00
XFFPDB(4):SUPLOG: Initialize PDB SUPLOG SGA, old value 0x0, new value 0x18
XFFPDB(4):Autotune of undo retention is turned on. 
XFFPDB(4):Endian type of dictionary set to little
2024-01-18T09:59:18.842283+08:00
Buffer Cache Full DB Caching mode changing from FULL CACHING ENABLED to FULL CACHING DISABLED 
Full DB Caching disabled: DEFAULT_CACHE_SIZE should be at least 153531 MBs bigger than current size. 
2024-01-18T09:59:19.017859+08:00
XFFPDB(4):Undo initialization recovery: Parallel FPTR failed: start:233023254 end:233023260 diff:6 ms(0.0 seconds)
XFFPDB(4):Undo initialization recovery: err:0 start: 233023254 end: 233023286 diff: 32 ms (0.0 seconds)
2024-01-18T09:59:19.256942+08:00
XFFPDB(4):[2475] Successfully onlined Undo Tablespace 40.
XFFPDB(4):Undo initialization online undo segments: err:0 start: 233023286 end: 233023500 diff:214 ms(0.2 seconds)
XFFPDB(4):Undo initialization finished serial:0 start:233023228 end:233023508 diff:280 ms (0.3 seconds)
XFFPDB(4):Database Characterset for XFFPDB is ZHS16GBK
XFFPDB(4):*********************************************************************
XFFPDB(4):WARNING: The following temporary tablespaces in container(XFFPDB)
XFFPDB(4):         contain no files.
XFFPDB(4):         This condition can occur when a backup controlfile has
XFFPDB(4):         been restored.  It may be necessary to add files to these
XFFPDB(4):         tablespaces.  That can be done using the SQL statement:
XFFPDB(4): 
XFFPDB(4):         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
XFFPDB(4): 
XFFPDB(4):         Alternatively, if these temporary tablespaces are no longer
XFFPDB(4):         needed, then they can be dropped.
XFFPDB(4):           Empty temporary tablespace: TEMP
XFFPDB(4):*********************************************************************
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_2475.trc  (incident=392032) (PDBNAME=XFFPDB):
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
XFFPDB(4):Incident details in:/u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_392032/XFF_ora_2475_i392032.trc
XFFPDB(4):Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2024-01-18T09:59:20.079597+08:00
XFFPDB(4):*****************************************************************
XFFPDB(4):An internal routine has requested a dump of selected redo.
XFFPDB(4):This usually happens following a specific internal error, when
XFFPDB(4):analysis of the redo logs will help Oracle Support with the
XFFPDB(4):diagnosis.
XFFPDB(4):It is recommended that you retain all the redo logs generated (by
XFFPDB(4):all the instances) during the past 12 hours, in case additional
XFFPDB(4):redo dumps are required to help with the diagnosis.
XFFPDB(4):*****************************************************************
===========================================================
Dumping current patch information
===========================================================
Unable to obtain current patch information due to error: 20001
===========================================================
2024-01-18T09:59:31.793666+08:00
XFFPDB(4):Flush retried for xcb 0x179400620, pmd 0x174a2f6c0
XFFPDB(4):Doing block recovery for file 31 block 1239
2024-01-18T09:59:31.805351+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_2475.trc  (incident=392034)(PDBNAME=XFFPDB):
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
XFFPDB(4):Incident details in:/u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_392034/XFF_ora_2475_i392034.trc
XFFPDB(4):Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2024-01-18T09:59:32.009996+08:00
XFFPDB(4):*****************************************************************
XFFPDB(4):An internal routine has requested a dump of selected redo.
XFFPDB(4):This usually happens following a specific internal error, when
XFFPDB(4):analysis of the redo logs will help Oracle Support with the
XFFPDB(4):diagnosis.
XFFPDB(4):It is recommended that you retain all the redo logs generated (by
XFFPDB(4):all the instances) during the past 12 hours, in case additional
XFFPDB(4):redo dumps are required to help with the diagnosis.
XFFPDB(4):*****************************************************************
Incident details in: /u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_392384/XFF_mz00_2879_i392384.trc
2024-01-18T09:59:36.918296+08:00
Reread of blocknum=142165, file=/u01/app/oracle/oradata/XFF/system01.dbf. found same corrupt data
Reread of blocknum=142165, file=/u01/app/oracle/oradata/XFF/system01.dbf. found same corrupt data
Reread of blocknum=142165, file=/u01/app/oracle/oradata/XFF/system01.dbf. found same corrupt data
Reread of blocknum=142165, file=/u01/app/oracle/oradata/XFF/system01.dbf. found same corrupt data
Reread of blocknum=142165, file=/u01/app/oracle/oradata/XFF/system01.dbf. found same corrupt data
2024-01-18T09:59:41.146850+08:00
XFFPDB(4):Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_2475.trc:
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
2024-01-18T09:59:45.434605+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_2475.trc  (incident=407714):
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_407714/XFF_ora_2475_i407714.trc
2024-01-18T09:59:45.439083+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_407714/XFF_ora_2475_i407714.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
2024-01-18T09:59:48.575971+08:00
XFFPDB(4):pdb open recovery: pdbid=4 log=255 acquired_ip=1

这个相对比较简单,对异常undo进行处理,cdb和pdb都open成功,使用数据泵把数据迁移到新库,完成本次恢复(比较幸运,硬件故障的库直接迁移成功,没有报其他错误)

Oracle 19c 断电异常恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:Oracle 19c 断电异常恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

19.3数据库由于异常断电,导致数据库无法启动,报ORA-600 ktbair2: illegal inheritance,ORA-600 6101等错误

2023-01-02T22:01:37.310225+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_pr0l_3788.trc:
ORA-10562: Error occurred while applying redo to data block (file# 10, block# 399386)
ORA-10564: tablespace DATA
ORA-01110: data file 10: 'D:\ORADATA\DATA04.ORA'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 81882
ORA-00600: internal error code, arguments: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
2023-01-02T22:01:37.544630+08:00
Slave exiting with ORA-10562 exception
2023-01-02T22:01:37.560258+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_pr03_4364.trc:
ORA-10562: Error occurred while applying redo to data block (file# 2, block# 716430)
ORA-10564: tablespace DATA
ORA-01110: data file 2: 'D:\ORADATA\DATA01.ORA'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 81743
ORA-00600: internal error code, arguments: [6101], [0], [16], [0], [0], [0], [], [], [], [], [], []
2023-01-02T22:01:38.294726+08:00
Slave exiting with ORA-10562 exception
2023-01-02T22:01:38.310354+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_pr0e_5980.trc:
ORA-10562: Error occurred while applying redo to data block (file# 2, block# 714911)
ORA-10564: tablespace DATA
ORA-01110: data file 2: 'D:\ORADATA\DATA01.ORA'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 74014
ORA-00600: internal error code, arguments: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
2023-01-02T22:01:48.921092+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_pr00_4760.trc:
ORA-00283: recovery session canceled due to errors
ORA-00448: normal completion of background process
2023-01-02T22:01:49.171125+08:00
ORA-756 signalled during: ALTER DATABASE RECOVER  database  ...

报错比较明显由于redo和datafile不匹配导致recover 不成功,尝试强制拉库

SQL> alter database open resetlogs ;
alter database open resetlogs 
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [],
[], [], [], [], [], [], []
进程 ID: 6068
会话 ID: 3631 序列号: 54960
Undo initialization recovery: err:600 start: 20760593 end: 20762484 diff: 1891 ms (1.9 seconds)
2023-01-02T22:09:05.539709+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_6068.trc:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2023-01-02T22:09:05.555336+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_6068.trc:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_6068.trc  (incident=293955):
ORA-00603: ORACLE 服务器会话因致命错误而终止
ORA-01092: ORACLE 实例终止。强制断开连接
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_293955\orcl_ora_6068_i293955.trc
2023-01-02T22:09:06.805497+08:00
opiodr aborting process unknown ospid (6068) as a result of ORA-603
2023-01-02T22:09:06.961768+08:00
ORA-603 : opitsk aborting process

参考类似处理open数据库成功:
ORA-600 kcbzib_kcrsds_1报错
12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
redo异常强制拉库报ORA-600 kcbzib_kcrsds_1修复
ORA-00603 ORA-01092 ORA-600 kcbzib_kcrsds_1

redo异常强制拉库报ORA-600 kcbzib_kcrsds_1修复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:redo异常强制拉库报ORA-600 kcbzib_kcrsds_1修复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

节点2 asm dismount导致redo写报错(ORA-00340,ORA-00345),经过分析asm和系统日志,确认是由于多路径异常导致io异常

2022-01-24T23:44:39.966602+08:00
WARNING: group 4 is being dismounted.
WARNING: ASMB force dismounting group 4 (REDO) due to ASM server dismount
SUCCESS: diskgroup REDO was dismounted
2022-01-24T23:44:41.103783+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF2/trace/XFF2_lgwr_228507.trc:
ORA-00345: redo log write error block 11961764 count 6
ORA-00312: online log 10 thread 2: '+REDO/XFF/ONLINELOG/group_10.261.1074690685'
2022-01-24T23:44:41.156809+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF2/trace/XFF2_lgwr_228507.trc:
ORA-00340: IO error processing online log 10 of thread 2
ORA-00345: redo log write error block 11961764 count 6
ORA-00312: online log 10 thread 2: '+REDO/XFF/ONLINELOG/group_10.261.1074690685'
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF2/trace/XFF2_lgwr_228507.trc  (incident=1341402):
ORA-340 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /u01/app/oracle/diag/rdbms/xff/XFF2/incident/incdir_1341402/XFF2_lgwr_228507_i1341402.trc
2022-01-24T23:44:41.505251+08:00
USER (ospid: 133928): terminating the instance due to error 340

由于节点2是突然crash,节点1做实例恢复失败,由于节点2的redo发生了写丢失,导致节点1实例恢复后库crash,进而是的该集群的相关数据库节点全部crash

2022-01-24T23:46:08.440519+08:00
Slave encountered ORA-10388 exception during crash recovery
2022-01-24T23:46:08.442854+08:00
Slave encountered ORA-10388 exception during crash recovery
Abort recovery for domain 0, flags 4
2022-01-24T23:46:08.444531+08:00
Aborting crash recovery due to error 742
2022-01-24T23:46:08.444695+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_450481.trc:
ORA-00742: Log read detects lost write in thread 2 sequence 63507 block 11941482
ORA-00312: online log 10 thread 2: '+REDO/XFF/ONLINELOG/group_10.261.1074690685'
Abort recovery for domain 0, flags 4
2022-01-24T23:46:08.771108+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_450481.trc:
ORA-00742: Log read detects lost write inthread 2 sequence 63507 block 11941482
ORA-00312: online log 10 thread 2: '+REDO/XFF/ONLINELOG/group_10.261.1074690685'
ORA-742 signalled during: ALTER DATABASE OPEN /* db agent *//* {0:17:165} */...
2022-01-24T23:46:10.143155+08:00
License high water mark = 33
2022-01-24T23:46:10.143752+08:00
USER (ospid: 451049): terminating the instance
2022-01-24T23:46:11.167337+08:00
Instance terminated by USER, pid = 451049

经过第三方强制拉库之后,数据库报ORA-600 kcbzib_kcrsds_1

2022-01-25T10:13:37.922332+08:00
Completed crash recovery at
 Thread 2: RBA 5.3.16, nab 3, scn 0x00000a348a032122
 0 data blocks read, 0 data blocks written, 0 redo k-bytes read
2022-01-25T10:13:38.071326+08:00
Thread 2 advanced to log sequence 6 (thread recovery)
validate pdb 0, flags x4, valid 0, pdb flags x204 
* validated domain 0, flags = 0x200
CRASH recovery complete: pdb 0 valid 1 (flags x4, pdb flags x200) 
Picked broadcast on commit scheme to generate SCNs
Endian type of dictionary set to little
2022-01-25T10:13:38.389741+08:00
TT00: Gap Manager starting (PID:220505)
2022-01-25T10:13:38.646484+08:00
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: +DATA/XFF/ONLINELOG/group_1.536.1094875107
Successful open of redo thread 1
2022-01-25T10:13:38.647243+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_216590.trc  (incident=1879556):
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xff/XFF1/incident/incdir_1879556/XFF1_ora_216590_i1879556.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2022-01-25T10:13:39.734366+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_216590.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2022-01-25T10:13:39.734424+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_216590.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2022-01-25T10:13:39.734499+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_216590.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2022-01-25T10:13:39.734536+08:00
Error 704 happened during db open, shutting down database
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_216590.trc  (incident=1879557):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xff/XFF1/incident/incdir_1879557/XFF1_ora_216590_i1879557.trc
2022-01-25T10:13:39.894464+08:00
2022-01-25T10:13:40.446888+08:00
opiodr aborting process unknown ospid (216590) as a result of ORA-603
2022-01-25T10:13:40.470643+08:00
ORA-603 : opitsk aborting process
License high water mark = 36
2022-01-25T10:13:40.471453+08:00
USER (ospid: 216590): terminating the instance due to error 704
2022-01-25T10:13:41.436133+08:00
opiodr aborting process unknown ospid (189796) as a result of ORA-1092
2022-01-25T10:13:41.439011+08:00
ORA-1092 : opitsk aborting process
2022-01-25T10:13:41.472060+08:00
PMON (ospid: 189585): terminating the instance due to error 704

该错误是12c之后才有的报错,由于文件异常导致,通过以前的解决经验,接手这个问题之后快速调整数据库文件头信息,顺利open库
参考以前相关blog内容:
Oracle 12c redo 丢失恢复
模拟19c数据库redo异常恢复
ORA-600 kcbzib_kcrsds_1报错
12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
ORA-00603 ORA-01092 ORA-600 kcbzib_kcrsds_1

[oracle@xifenfei02 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Wed Jan 26 00:31:50 2022

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount pfile='/tmp/pfile'
ORACLE instance started.

Total System Global Area 3.2320E+11 bytes
Fixed Size                 29879248 bytes
Variable Size            4.5634E+10 bytes
Database Buffers         1.9059E+11 bytes
Redo Buffers             1043861504 bytes
In-Memory Area           8.5899E+10 bytes
Database mounted.
SQL> alter database open ;

Database altered.

ORA-00603 ORA-01092 ORA-600 kcbzib_kcrsds_1

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-00603 ORA-01092 ORA-600 kcbzib_kcrsds_1

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

接手客户故障数据库报错为:ORA-16433

[orauser@xifenfei check_db]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Tue Dec 14 02:36:24 2021

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> alter database backup controlfile to trace as '/tmp/ctl';
alter database backup controlfile to trace as '/tmp/ctl'
*
ERROR at line 1:
ORA-16433: The database or pluggable database must be opened in read/write mode.


SQL> recover datafile 1;
ORA-00283: recovery session canceled due to errors
ORA-16433: The database or pluggable database must be opened in read/write mode.

这个错误一般是由于之前resetlogs未能正常打开库导致,通过rectl之后,尝试重新打开库,报错为:ORA-600 kcbzib_kcrsds_1

SQL> ALTER DATABASE OPEN;
ALTER DATABASE OPEN
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Process ID: 49304
Session ID: 4698 Serial number: 14852

这个错误一般是scn问题,对于之前版本直接使用隐含参数,event,oradebug进行调整scn即可,但是在12.2版本之后无法使用这些方法修改,处理起来相对麻烦一些,参考以前处理的相关文章:
ORA-600 kcbzib_kcrsds_1报错
12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
oradebug poke ORA-32521/ORA-32519故障解决

12C数据库报ORA-600 kcbzib_kcrsds_1故障处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:12C数据库报ORA-600 kcbzib_kcrsds_1故障处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有朋友由于主机断电,导致数据库无法启动,需要我们提供恢复支持.通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)检查分析,发现存放在一个磁盘的数据文件scn过小,数据库无法正常open,需要历史redo(已经被覆盖,而且数据库未归档).
20201130202509
20201130202545


对于这种情况,选择强制拉库,然后数据库报ORA-600 kcbzib_kcrsds_1错误
20201130203023
20201130203204

对于此类故障参考以前处理过类似案例:ORA-600 kcbzib_kcrsds_1报错,对数据库scn进行修改,数据库顺利open
20201130203529

模拟19c数据库redo异常恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:模拟19c数据库redo异常恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

对于19c在pdb情况下三种常见故障进行了模拟测试:
模拟19c数据库redo异常恢复
模拟19c数据库pdb undo异常恢复
模拟19c数据库root pdb undo异常恢复
模拟oracle 19c数据库redo丢失的恢复操作,模拟数据库有事务,在没有提交的情况下redo丢失故障

[oracle@localhost oradata]$ ss

SQL*Plus: Release 19.0.0.0.0 - Production on Mon Nov 16 16:11:16 2020
Version 19.5.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.5.0.0.0

SQL> conn xff/oracle@127.0.0.1/pdb
Connected.
SQL> create table t_xifenfei as select * from dba_objects;

Table created.

SQL> insert into t_xifenfei select * from t_xifenfei;
insert into t_xifenfei select * from t_xifenfei;
insert into t_xifenfei select * from t_xifenfei;
insert into t_xifenfei select * from t_xifenfei;
insert into t_xifenfei select * from t_xifenfei;

72351 rows created.

SQL> 
144702 rows created.

SQL> 
289404 rows created.

SQL> 
578808 rows created.

SQL> 

1157616 rows created.

另外一个会话kill数据库并且删除redo

[root@localhost ~]# ps -ef|grep pmon
oracle    38500      1  0 16:08 ?        00:00:00 ora_pmon_ora19c
root      39030  39009  0 16:11 pts/2    00:00:00 grep --color=auto pmon
[root@localhost ~]# kill -9 38500
[root@localhost ~]# ps -ef|grep pmon
root      39042  39009  0 16:11 pts/2    00:00:00 grep --color=auto pmon
[root@localhost ~]# ls -l /u01/app/oracle/oradata/ORA19C/redo*.log
ls: cannot access /u01/app/oracle/oradata/ORA19C/redo*.log: No such file or directory

启动数据库报错ORA-00313 ORA-00312 ORA-27037

SQL> startup 
ORACLE instance started.

Total System Global Area 4999609088 bytes
Fixed Size                  9145088 bytes
Variable Size             905969664 bytes
Database Buffers         4076863488 bytes
Redo Buffers                7630848 bytes
Database mounted.
ORA-00313: open failed for members of log group 3 of thread 1
ORA-00312: online log 3 thread 1: '/u01/app/oracle/oradata/ORA19C/redo03.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7

因为redo全部丢失只能尝试强制拉库

SQL> startup mount pfile='/tmp/pfile'
ORACLE instance started.

Total System Global Area 4999609088 bytes
Fixed Size                  9145088 bytes
Variable Size             905969664 bytes
Database Buffers         4076863488 bytes
Redo Buffers                7630848 bytes
Database mounted.
SQL>  recover database until cancel;
ORA-00279: change 2335666 generated at 11/16/2020 16:08:42 needed for thread 1
ORA-00289: suggestion :
/u01/app/oracle/product/19.2/db_1/dbs/arch1_12_1056620100.dbf
ORA-00280: change 2335666 for thread 1 is in sequence #12


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/u01/app/oracle/oradata/ORA19C/system01.dbf'


ORA-01112: media recovery not started


SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [],
[], [], [], [], [], [], []
Process ID: 39588
Session ID: 9 Serial number: 32012

数据库报ORA-600 kcbzib_kcrsds_1错误是由于在强制拉库过程中文件异常导致,通过对异常文件进行处理数据库open成功

SQL> recover database;
Media recovery complete.
SQL> alter database open;

Database altered.

SQL> 

SQL> alter session set container=pdb;

Session altered.

SQL> alter database open;

SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB                            READ WRITE NO

这个是模拟redo丢失或者损坏故障,在实际的生产故障中可能要比这个复杂很多.

ORA-600 kcbzib_kcrsds_1报错

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:ORA-600 kcbzib_kcrsds_1报错

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

数据库版本
12.2
客户存储故障,修复之后,多套库增加_allow_resetlogs_corruption隐含参数强制拉库出现都类似错误ORA-600 kcbzib_kcrsds_1,出现这个错误,一般都是由于数据库不一致强制拉库导致

2019-02-23T01:25:43.125621+08:00
 alter database open resetlogs
2019-02-23T01:25:43.231990+08:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 149251865354 time
Clearing online redo logfile 1 +DG_XFF/xifenfei/ONLINELOG/group_1.258.983824407
Clearing online redo logfile 2 +DG_XFF/xifenfei/ONLINELOG/group_2.259.983824409
Clearing online redo logfile 3 +DG_XFF/xifenfei/ONLINELOG/group_3.266.983825461
Clearing online redo logfile 4 +DG_XFF/xifenfei/ONLINELOG/group_4.267.983825461
Clearing online log 1 of thread 1 sequence number 20749
Clearing online log 2 of thread 1 sequence number 20750
Clearing online log 3 of thread 2 sequence number 1371
Clearing online log 4 of thread 2 sequence number 1372
2019-02-23T01:25:44.669890+08:00
ALTER SYSTEM SET remote_listener=' xifenfeidb-cluster-scan:1521' SCOPE=MEMORY SID='xifenfei2';
2019-02-23T01:25:44.671436+08:00
ALTER SYSTEM SET listener_networks='' SCOPE=MEMORY SID='xifenfei2';
2019-02-23T01:25:46.990077+08:00
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 complete
Clearing online redo logfile 3 complete
Clearing online redo logfile 4 complete
Resetting resetlogs activation ID 3002369299 (0xb2f48513)
Online log +DG_XFF/xifenfei/ONLINELOG/group_1.258.983824407: Thread 1 Group 1 was previously cleared
Online log +DG_XFF/xifenfei/ONLINELOG/group_2.259.983824409: Thread 1 Group 2 was previously cleared
Online log +DG_XFF/xifenfei/ONLINELOG/group_3.266.983825461: Thread 2 Group 3 was previously cleared
Online log +DG_XFF/xifenfei/ONLINELOG/group_4.267.983825461: Thread 2 Group 4 was previously cleared
2019-02-23T01:25:47.137701+08:00
Setting recovery target incarnation to 2
2019-02-23T01:25:47.152393+08:00
This instance was first to open
Ping without log force is disabled:
  not an Exadata system.
Picked broadcast on commit scheme to generate SCNs
Endian type of dictionary set to little
2019-02-23T01:25:47.597502+08:00
Assigning activation ID 3019587675 (0xb3fb405b)
2019-02-23T01:25:47.625734+08:00
TT00: Gap Manager starting (PID:22467)
2019-02-23T01:25:47.910026+08:00
Thread 2 opened at log sequence 1
  Current log# 3 seq# 1 mem# 0: +DG_XFF/xifenfei/ONLINELOG/group_3.266.983825461
Successful open of redo thread 2
2019-02-23T01:25:47.911069+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
2019-02-23T01:25:47.971709+08:00
Sleep 5 seconds and then try to clear SRLs in 2 time(s)
2019-02-23T01:25:48.065008+08:00
start recovery: pdb 0, passed in flags x10 (domain enable 0)
2019-02-23T01:25:48.065177+08:00
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
validate pdb 0, flags x10, valid 0, pdb flags x84
* validated domain 0, flags = 0x80
Instance recovery complete: valid 1 (flags x10, recovery domain flags x80)
2019-02-23T01:25:48.803746+08:00
Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei2/trace/xifenfei2_ora_21292.trc  (incident=128552):
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2019-02-23T01:25:49.947062+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2019-02-23T01:25:50.334684+08:00
Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei2/trace/xifenfei2_ora_21292.trc:
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2019-02-23T01:25:50.334880+08:00
Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei2/trace/xifenfei2_ora_21292.trc:
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
2019-02-23T01:25:50.362808+08:00
Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei2/trace/xifenfei2_ora_21292.trc  (incident=128553):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2019-02-23T01:25:51.521133+08:00
opiodr aborting process unknown ospid (21292) as a result of ORA-603

另外出现该错误之后,数据库再次恢复会出现类似,这个是由于open库的过程中导致控制文件损坏而出现的错误.

2019-02-23T22:52:39.390966+08:00
ALTER DATABASE RECOVER  database
2019-02-23T22:52:39.391125+08:00
Media Recovery Start
 Started logmerger process
2019-02-23T22:52:39.471904+08:00
Media Recovery failed with error 16433
2019-02-23T22:52:39.996235+08:00
Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_m000_1593.trc:
ORA-01110: data file 11: '+DG_XFF/xifenfei/DATAFILE/ls_zh.274.984235699'
2019-02-23T22:52:40.224440+08:00
Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_m000_1593.trc:
ORA-01110: data file 12: '+DG_XFF/xifenfei/DATAFILE/sf_zh.275.984235715'
2019-02-23T22:52:40.459606+08:00
Errors in file /oracle/base/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_m000_1593.trc:
ORA-01110: data file 13: '+DG_XFF/xifenfei/DATAFILE/tj_gl.276.984235729'
2019-02-23T22:52:40.574563+08:00
Recovery Slave PR00 previously exited with exception 283
ORA-283 signalled during: ALTER DATABASE RECOVER  database  ...

Oracle 12c redo 丢失恢复

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:Oracle 12c redo 丢失恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

模拟redo丢失
对数据库的一个pdb模拟事务操作,然后abort库,并且删除所有redo,模拟生产环境redo丢失的case

[oracle@ora1221 oradata]$ ss
SQL*Plus: Release 12.2.0.0.3 Production on Wed Jun 15 10:13:20 2016
Copyright (c) 1982, 2016, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 2516582400 bytes
Fixed Size                  8260048 bytes
Variable Size             671090224 bytes
Database Buffers         1828716544 bytes
Redo Buffers                8515584 bytes
Database mounted.
Database opened.
SQL>
SQL>
SQL> set pages 100
SQL> show pdbs;
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           MOUNTED
         4 PDB2                           MOUNTED
SQL> select con_id,file#,checkpoint_change# from v$datafile_header order by 1;
    CON_ID      FILE# CHECKPOINT_CHANGE#
---------- ---------- ------------------
         1          1            1500157
         1          3            1500157
         1          4            1500157
         1          7            1500157
         2          5            1371280
         2          6            1371280
         2          8            1371280
         3          9            1499902
         3         12            1499902
         3         11            1499902
         3         10            1499902
         4         15            1499903
         4         14            1499903
         4         13            1499903
         4         16            1499903
15 rows selected.
SQL> alter PLUGGABLE database pdb1 open;
Pluggable database altered.
SQL> show pdbs;
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO
         4 PDB2                           MOUNTED
SQL> alter session set container=pdb1;
Session altered.
SQL> create user chf identified by oracle;
User created.
SQL> grant dba to chf;
Grant succeeded.
SQL> create table chf.t_pdb1_xifenfei as select * from dba_objects;
Table created.
SQL> delete from chf.t_pdb1_xifenfei;
72426 rows deleted.
--另外一个节点
[oracle@ora1221 ~]$ ss
SQL*Plus: Release 12.2.0.0.3 Production on Wed Jun 15 10:19:21 2016
Copyright (c) 1982, 2016, Oracle.  All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.0.3 - 64bit Production
SQL> shutdown abort
ORACLE instance shut down.
[oracle@ora1221 orcl12c2]$ ls redo*
redo01.log  redo02.log  redo03.log
[oracle@ora1221 orcl12c2]$ rm redo0*
[oracle@ora1221 orcl12c2]$ ls -l redo*
ls: cannot access redo*: No such file or directory

尝试启动数据库

[oracle@ora1221 orcl12c2]$ ss
SQL*Plus: Release 12.2.0.0.3 Production on Wed Jun 15 10:26:20 2016
Copyright (c) 1982, 2016, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup mount;
ORACLE instance started.
Total System Global Area 2516582400 bytes
Fixed Size                  8260048 bytes
Variable Size             671090224 bytes
Database Buffers         1828716544 bytes
Redo Buffers                8515584 bytes
Database mounted.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl12c2/redo01.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7

使用隐含参数启动

----pfile里面增加
_allow_error_simulation=TRUE
_allow_resetlogs_corruption=true
~
SQL> shutdown immediate;
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down
SQL> startup pfile='/tmp/pfile' mount
ORACLE instance started.
Total System Global Area 2516582400 bytes
Fixed Size                  8260048 bytes
Variable Size             671090224 bytes
Database Buffers         1828716544 bytes
Redo Buffers                8515584 bytes
Database mounted.
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [],
[], [], [], [], [], [], []
Process ID: 36797
Session ID: 16 Serial number: 24277

继续重启库
ORA-600 kcbzib_kcrsds_1错误尝试重启数据库,如果不行,考虑使用bbed修改文件头信息

SQL> startup mount pfile='/tmp/pfile'
ORACLE instance started.
Total System Global Area 2516582400 bytes
Fixed Size                  8260048 bytes
Variable Size             671090224 bytes
Database Buffers         1828716544 bytes
Redo Buffers                8515584 bytes
Database mounted.
SQL> recover database until cancel;
ORA-00283: recovery session canceled due to errors
ORA-16433: The database or pluggable database must be opened in read/write
mode.
SQL> alter database backup controlfile to trace as '/tmp/ctl';
alter database backup controlfile to trace as '/tmp/ctl'
*
ERROR at line 1:
ORA-16433: The database or pluggable database must be opened in read/write
mode.

重建控制文件

SQL> startup nomount pfile='/tmp/pfile'
ORACLE instance started.
Total System Global Area 2516582400 bytes
Fixed Size                  8260048 bytes
Variable Size             671090224 bytes
Database Buffers         1828716544 bytes
Redo Buffers                8515584 bytes
SQL> CREATE CONTROLFILE REUSE DATABASE "orcl12c2" RESETLOGS  NOARCHIVELOG
  2      MAXLOGFILES 50
  3      MAXLOGMEMBERS 5
  4      MAXDATAFILES 100
  5      MAXINSTANCES 1
  6      MAXLOGHISTORY 226
  7  LOGFILE
  8    GROUP 1 '/u01/app/oracle/oradata/orcl12c2/redo01.log'  SIZE 200M,
  9    GROUP 2 '/u01/app/oracle/oradata/orcl12c2/redo02.log'  SIZE 200M,
 10    GROUP 3 '/u01/app/oracle/oradata/orcl12c2/redo03.log'  SIZE 200M
 11  DATAFILE
 12  '/u01/app/oracle/oradata/orcl12c2/system01.dbf',
 13  '/u01/app/oracle/oradata/orcl12c2/sysaux01.dbf',
 14  '/u01/app/oracle/oradata/orcl12c2/undotbs01.dbf',
 15  '/u01/app/oracle/oradata/orcl12c2/pdbseed/system01.dbf',
 16  '/u01/app/oracle/oradata/orcl12c2/pdbseed/sysaux01.dbf',
 17  '/u01/app/oracle/oradata/orcl12c2/users01.dbf',
 18  '/u01/app/oracle/oradata/orcl12c2/pdbseed/undotbs01.dbf',
 19  '/u01/app/oracle/oradata/orcl12c2/pdb1/system01.dbf',
 20  '/u01/app/oracle/oradata/orcl12c2/pdb1/sysaux01.dbf',
 21  '/u01/app/oracle/oradata/orcl12c2/pdb1/undotbs01.dbf',
 22  '/u01/app/oracle/oradata/orcl12c2/pdb1/users01.dbf',
 23  '/u01/app/oracle/oradata/orcl12c2/pdb2/system01.dbf',
 24  '/u01/app/oracle/oradata/orcl12c2/pdb2/sysaux01.dbf',
 25  '/u01/app/oracle/oradata/orcl12c2/pdb2/undotbs01.dbf',
 26  '/u01/app/oracle/oradata/orcl12c2/pdb2/users01.dbf'
 27  CHARACTER SET AL32UTF8
 28  ;
Control file created.
SQL> recover database until cancel;
ORA-00283: recovery session canceled due to errors
ORA-01610: recovery using the BACKUP CONTROLFILE option must be done
SQL> recover database until cancel using backup controlfile;
ORA-00279: change 1500161 generated at 06/15/2016 10:40:42 needed for thread 1
ORA-00289: suggestion :
/u01/app/oracle/product/12.2.0/db_2/dbs/arch1_1_914582438.dbf
ORA-00280: change 1500161 for thread 1 is in sequence #1
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/u01/app/oracle/oradata/orcl12c2/redo01.log
Log applied.
Media recovery complete.
SQL> alter database open resetlogs;
Database altered.

<strong>open过程alert日志</strong>

<strong>open pdb1</strong>

SQL> alter PLUGGABLE database pdb1  open;
Pluggable database altered.

pdb1 open alert日志

2016-06-15T11:13:39.423057+08:00
alter PLUGGABLE database pdb1  open
PDB1(3):Autotune of undo retention is turned on.
2016-06-15T11:13:39.495559+08:00
PDB1(3):Endian type of dictionary set to little
PDB1(3):[40547] Successfully onlined Undo Tablespace 2.
PDB1(3):Undo initialization finished serial:0 start:371149831 end:371149872 diff:41 ms (0.0 seconds)
PDB1(3):Database Characterset for PDB1 is AL32UTF8
PDB1(3):*********************************************************************
PDB1(3):WARNING: The following temporary tablespaces in container(PDB1)
PDB1(3):         contain no files.
PDB1(3):         This condition can occur when a backup controlfile has
PDB1(3):         been restored.  It may be necessary to add files to these
PDB1(3):         tablespaces.  That can be done using the SQL statement:
PDB1(3):
PDB1(3):         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
PDB1(3):
PDB1(3):         Alternatively, if these temporary tablespaces are no longer
PDB1(3):         needed, then they can be dropped.
PDB1(3):           Empty temporary tablespace: TEMP
PDB1(3):*********************************************************************
PDB1(3):Opatch validation is skipped for PDB PDB1 (con_id=0)
PDB1(3):Opening pdb with no Resource Manager plan active
Pluggable database PDB1 opened read write
Completed: alter PLUGGABLE database pdb1  open

open pdb2

SQL> alter PLUGGABLE database pdb2 open;
alter PLUGGABLE database pdb2 open
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [],
[], [], [], []

分析alert日志和trace文件

--alert日志部分
PDB1(3):alter PLUGGABLE database pdb2 open
PDB1(3):ORA-65118 signalled during: alter PLUGGABLE database pdb2 open...
2016-06-15T11:28:57.439963+08:00
PDB1(3):Unified Audit: Audit record write to table failed due to ORA-25153.
Writing the audit record to OS spillover file. Please grep in the trace files for ORA-25153 for more diagnostic information.
Errors in file /u01/app/oracle/diag/rdbms/orcl12c2/orcl12c2/trace/orcl12c2_ora_40547.trc  (incident=29073) (PDBNAME=PDB1):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
PDB1(3):Incident details in: /u01/app/oracle/diag/rdbms/orcl12c2/orcl12c2/incident/incdir_29073/orcl12c2_ora_40547_i29073.trc
PDB1(3):*****************************************************************
PDB1(3):An internal routine has requested a dump of selected redo.
PDB1(3):This usually happens following a specific internal error, when
PDB1(3):analysis of the redo logs will help Oracle Support with the
PDB1(3):diagnosis.
PDB1(3):It is recommended that you retain all the redo logs generated (by
PDB1(3):all the instances) during the past 12 hours, in case additional
PDB1(3):redo dumps are required to help with the diagnosis.
PDB1(3):*****************************************************************
2016-06-15T11:28:59.123041+08:00
PDB1(3):Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2016-06-15T11:28:59.945667+08:00
Dumping diagnostic data in directory=[cdmp_20160615112859], requested by (instance=1, osid=40547), summary=[incident=29073].
2016-06-15T11:35:59.987419+08:00
PDB1(3): alter PLUGGABLE database pdb2 open
PDB1(3):ORA-65118 signalled during:  alter PLUGGABLE database pdb2 open...
--trace部分
PARSING IN CURSOR #0x7f051a3d7650 len=118 dep=1 uid=0 oct=3 lid=0 tim=372490287736 hv=1128335472 ad='0x6ca82f00' sqlid='gu930gd1n223h'
select tablespace_name, tablespace_size, allocated_space, free_space, con_id  from cdb_temp_free_space order by con_id
END OF STMT
EXEC #0x7f051a3d7650:c=0,e=144,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=2538033465,tim=372490287732
FETCH #0x7f051a3d7650:c=0,e=290,p=0,cr=14,cu=0,mis=0,r=0,dep=1,og=4,plh=2538033465,tim=372490288109
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 52 FileOperation=2 fileno=0 filetype=36 obj#=402 tim=372490288373
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 17 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490288577
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 3 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490288655
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 690 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490289365
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 6 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490289470
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 445 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490289934
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 3 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490289983
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 375 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490290374
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 6 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490290453
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 367 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490290839
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 3 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490290882
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 355 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490291252
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 4 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490291298
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 276 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490291590
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 1 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490291614
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 256 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490291879
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 2 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490291903
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 261 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490292172
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 30 FileOperation=3 fileno=0 filetype=36 obj#=402 tim=372490292225
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 934 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490293171
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 3 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490293208
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 245 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490293465
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 262 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490293755
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 1 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490293780
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 250 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490294039
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 256 FileOperation=8 fileno=1 filetype=36 obj#=402 tim=372490294323
WAIT #0x7f051bd5f870: nam='Disk file operations I/O' ela= 8 FileOperation=5 fileno=0 filetype=36 obj#=402 tim=372490294359
2016-06-15T11:36:00.055196+08:00
Incident 29074 created, dump file: /u01/app/oracle/diag/rdbms/orcl12c2/orcl12c2/incident/incdir_29074/orcl12c2_ora_40547_i29074.trc
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []

从中可以判断出来是由于CDB$ROOT的未增加tempfile导致

SQL> alter tablespace temp add tempfile '/u01/app/oracle/oradata/orcl12c2/temp01.dbf' reuse;
Tablespace altered.
SQL>  alter PLUGGABLE database pdb2 open;
Pluggable database altered.

查看数据库恢复情况

SQL> show pdbs;
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO
         4 PDB2                           READ WRITE NO
SQL> select con_id,file#,checkpoint_change#,resetlogs_change# from v$datafile_header;
    CON_ID      FILE# CHECKPOINT_CHANGE# RESETLOGS_CHANGE#
---------- ---------- ------------------ -----------------
         1          1            2500167           1500164
         1          3            2500167           1500164
         1          4            2500167           1500164
         2          5            1371280           1341067
         2          6            1371280           1341067
         1          7            2500167           1500164
         2          8            1371280           1341067
         3          9            2501017           1500164
         3         10            2501017           1500164
         3         11            2501017           1500164
         3         12            2501017           1500164
         4         13            2502748           1500164
         4         14            2502748           1500164
         4         15            2502748           1500164
         4         16            2502748           1500164
15 rows selected.

至此基本上测试完成在在cdb环境中丢失redo的恢复。在生产中,需要把temp加全,并且建议重建数据库