ORA-10485故障解决

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：ORA-10485故障解决

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

今天对主库进行打了psu和ojvm成功之后，对standby库打上了psu和ojvm，然后给备库启动mrp进程发现余下报错

Fri Apr 16 23:18:54 2021
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION
Attempt to start background Managed Standby Recovery process (orcl1)
Fri Apr 16 23:18:54 2021
MRP0 started with pid=32, OS id=15961 
MRP0: Background Managed Standby Recovery process started (orcl1)
 started logmerger process
Fri Apr 16 23:18:59 2021
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 32 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63308_j7m6domc_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41599_j7m70kyx_.arc
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63309_j7m6dop8_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63310_j7m70fmp_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63311_j7m70hlj_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63312_j7m75cnk_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41600_j7m75b77_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41601_j7m75bcx_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63313_j7m7vs0h_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41602_j7m7vn3g_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41603_j7m7vq2b_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41604_j7m81g5r_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63314_j7m81cz2_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63315_j7m81d31_.arc
Fri Apr 16 23:19:10 2021
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41605_j7m8vwz5_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41606_j7m8vzcr_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63316_j7m8vxgm_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41607_j7m8vzfk_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63317_j7m8w213_.arc
Errors with log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63317_j7m8w213_.arc
MRP0: Background Media Recovery terminated with error 10485
Errors in file /u01/app/oracle/diag/rdbms/orcldg/orcl1/trace/orcl1_pr00_15963.trc:
ORA-10485: Real-Time Query cannot be enabled while applying migration redo.
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Fri Apr 16 23:19:12 2021
MRP0: Background Media Recovery process shutdown (orcl1)

由于redo中有migration操作,导致备库mrp应用进程失败,解决此类问题最简单的方法,就是把备库重启到mount状态,然后使用rman进行恢复操作,然后再继续open库,启动mrp解决这类问题

oracle@localhost trace]$ ss

SQL*Plus: Release 11.2.0.4.0 Production on Fri Apr 16 23:24:55 2021

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount;
ORACLE instance started.

Total System Global Area 4.2758E+10 bytes
Fixed Size                  2262656 bytes
Variable Size            5905582464 bytes
Database Buffers         3.6776E+10 bytes
Redo Buffers               74420224 bytes
Database mounted.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@localhost trace]$ rman target /

Recovery Manager: Release 11.2.0.4.0 - Production on Fri Apr 16 23:25:12 2021

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL (DBID=1442395283, not open)

RMAN> recover database;

Starting recover at 16-APR-21
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=143 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=286 device type=DISK
allocated channel: ORA_DISK_3
channel ORA_DISK_3: SID=428 device type=DISK
allocated channel: ORA_DISK_4
channel ORA_DISK_4: SID=570 device type=DISK

starting media recovery

archived log for thread 1 with sequence 63317 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63317_j7m8w213_.arc
archived log for thread 1 with sequence 63318 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63318_j7m932ms_.arc
archived log for thread 1 with sequence 63319 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63319_j7m93719_.arc
archived log for thread 1 with sequence 63320 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63320_j7m96b3g_.arc
archived log for thread 2 with sequence 41607 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41607_j7m8vzfk_.arc
archived log for thread 2 with sequence 41608 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41608_j7m963jp_.arc
archived log for thread 2 with sequence 41609 is already on disk as file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41609_j7m967wm_.arc
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63317_j7m8w213_.arc thread=1 sequence=63317
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41607_j7m8vzfk_.arc thread=2 sequence=41607
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63318_j7m932ms_.arc thread=1 sequence=63318
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63319_j7m93719_.arc thread=1 sequence=63319
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63320_j7m96b3g_.arc thread=1 sequence=63320
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41608_j7m963jp_.arc thread=2 sequence=41608
archived log file name=/u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41609_j7m967wm_.arc thread=2 sequence=41609
unable to find archived log
archived log thread=2 sequence=41610
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 04/16/2021 23:25:25
RMAN-06054: media recovery requesting unknown archived log for thread 2 with sequence 41610 and starting SCN of 15880064787

SQL> alter database open;

Database altered.

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION;

Database altered.

Fri Apr 16 23:26:45 2021
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION
Fri Apr 16 23:26:47 2021
RFS[2]: Assigned to RFS process 16538
RFS[2]: Opened log for thread 2 sequence 41611 dbid 1442395283 branch 914614547
Archived Log entry 54459 added for thread 2 sequence 41611 rlc 914614547 ID 0x5b2c46ea dest 2:
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41611_j7mc5qcc_.arc
Media Recovery Waiting for thread 2 sequence 41612
Fetching gap sequence in thread 2, gap sequence 41612-41612
Fri Apr 16 23:26:50 2021
RFS[1]: Opened log for thread 2 sequence 41612 dbid 1442395283 branch 914614547
Archived Log entry 54460 added for thread 2 sequence 41612 rlc 914614547 ID 0x5b2c46ea dest 2:
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41612_j7mc5ttr_.arc
Media Recovery Waiting for thread 2 sequence 41613
Fetching gap sequence in thread 2, gap sequence 41613-41613
RFS[1]: Opened log for thread 2 sequence 41613 dbid 1442395283 branch 914614547
Archived Log entry 54461 added for thread 2 sequence 41613 rlc 914614547 ID 0x5b2c46ea dest 2:
Fri Apr 16 23:26:55 2021
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41613_j7mc5y6k_.arc
Media Recovery Waiting for thread 1 sequence 63323
Deleted Oracle managed file /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_03_25/o1_mf_2_40942_j5s61m0d_.arc
RFS[1]: Opened log for thread 1 sequence 63323 dbid 1442395283 branch 914614547
Archived Log entry 54462 added for thread 1 sequence 63323 rlc 914614547 ID 0x5b2c46ea dest 2:
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63323_j7mc61gg_.arc
Media Recovery Waiting for thread 1 sequence 63324
Fri Apr 16 23:28:20 2021
RFS[3]: Assigned to RFS process 16545
RFS[3]: Opened log for thread 2 sequence 41615 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:20 2021
RFS[4]: Assigned to RFS process 16543
RFS[4]: Opened log for thread 1 sequence 63326 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:20 2021
RFS[5]: Assigned to RFS process 16551
RFS[5]: Opened log for thread 1 sequence 63324 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:20 2021
RFS[6]: Assigned to RFS process 16549
RFS[6]: Opened log for thread 2 sequence 41614 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:20 2021
RFS[7]: Assigned to RFS process 16547
RFS[7]: Opened log for thread 1 sequence 63325 dbid 1442395283 branch 914614547
Archived Log entry 54463 added for thread 2 sequence 41615 rlc 914614547 ID 0x5b2c46ea dest 2:
Fri Apr 16 23:28:20 2021
RFS[8]: Assigned to RFS process 16553
RFS[8]: Opened log for thread 2 sequence 41616 dbid 1442395283 branch 914614547
Archived Log entry 54464 added for thread 1 sequence 63326 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54465 added for thread 2 sequence 41616 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54466 added for thread 1 sequence 63325 rlc 914614547 ID 0x5b2c46ea dest 2:
RFS[3]: Opened log for thread 2 sequence 41617 dbid 1442395283 branch 914614547
RFS[8]: Opened log for thread 2 sequence 41618 dbid 1442395283 branch 914614547
RFS[4]: Opened log for thread 1 sequence 63327 dbid 1442395283 branch 914614547
Archived Log entry 54467 added for thread 2 sequence 41617 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54468 added for thread 1 sequence 63327 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54469 added for thread 2 sequence 41618 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54470 added for thread 1 sequence 63324 rlc 914614547 ID 0x5b2c46ea dest 2:
Archived Log entry 54471 added for thread 2 sequence 41614 rlc 914614547 ID 0x5b2c46ea dest 2:
Fri Apr 16 23:28:20 2021
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63324_j7mc8n1n_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41614_j7mc8n1r_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41615_j7mc8n03_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41616_j7mc8n2n_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41617_j7mc8n6m_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41618_j7mc8n6p_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63325_j7mc8n21_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63326_j7mc8n11_.arc
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63327_j7mc8n6t_.arc
Media Recovery Waiting for thread 1 sequence 63328
RFS[5]: Selected log 20 for thread 1 sequence 63328 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:23 2021
Primary database is in MAXIMUM PERFORMANCE mode
Re-archiving standby log 20 thread 1 sequence 63328
Fri Apr 16 23:28:23 2021
Archived Log entry 54472 added for thread 1 sequence 63328 ID 0x5b2c46ea dest 1:
RFS[9]: Assigned to RFS process 16555
RFS[9]: Selected log 20 for thread 1 sequence 63329 dbid 1442395283 branch 914614547
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_1_63328_j7mc8q4n_.arc
Media Recovery Waiting for thread 2 sequence 41619
Fri Apr 16 23:28:24 2021
Primary database is in MAXIMUM PERFORMANCE mode
RFS[10]: Assigned to RFS process 16557
RFS[10]: Selected log 30 for thread 2 sequence 41620 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:24 2021
RFS[11]: Assigned to RFS process 16559
RFS[11]: Selected log 31 for thread 2 sequence 41619 dbid 1442395283 branch 914614547
Fri Apr 16 23:28:24 2021
Archived Log entry 54473 added for thread 2 sequence 41619 ID 0x5b2c46ea dest 1:
Media Recovery Log /u01/nfs/fast_recovery_area/ORCLDG/archivelog/2021_04_16/o1_mf_2_41619_j7mc8rdz_.arc
Media Recovery Waiting for thread 1 sequence 63329 (in transit)
Recovery of Online Redo Log: Thread 1 Group 20 Seq 63329 Reading mem 0
  Mem# 0: /u01/app/oradata/orcl/std_redo20.log
Media Recovery Waiting for thread 2 sequence 41620 (in transit)
Recovery of Online Redo Log: Thread 2 Group 30 Seq 41620 Reading mem 0
  Mem# 0: /u01/app/oradata/orcl/std_redo30.log

mos上有类似文章供参考:MRP process getting terminated with error ORA-10485 (Doc ID 1618485.1)

记录一次oracle现场故障处理经过

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：记录一次oracle现场故障处理经过

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

近期到现场进行了一个数据库恢复,我在恢复之前该库先由于硬件进行恢复,然后由其他人对其进行了一系列数据库恢复,但是未恢复成功,客户希望我们到现场进行处理(因为网络原因无法远程).接手库之后,处理第一个问题,是客户在进行现场备份的时候(把linux数据拷贝到win的过程中)发现有几个文件拷贝异常,这个错误很可能是由于当初的硬件故障修复之后留下的后遗症(由于io设备错误,无法运行此项请求),通过工具进行拷贝,恢复出来
20210403210131

DUL> copy file from  /oradata2/xifenfeidata.dbf to /oradata2/xifenfeidata.dbf

starting copy datafile '/oradata1/xifenfeidata.dbf' to '/oradata2/xifenfeidata.dbf'
read data error from file '/oradata1/xifenfeidata.dbf'.error message:Input/output error
read block# error: 560171
read data error from file '/oradata1/xifenfeidata.dbf'.error message:Input/output error
read block# error: 560179
datafile copy completed with 2 block error.

[oracle@localhost ~]$ dbv file=/oradata2/xifenfeidata.dbf blocksize=16384

DBVERIFY: Release 11.2.0.3.0 - Production on Mon Mar 29 17:28:17 2021

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /oradata2/xifenfeidata.dbf
Page 560171 is marked corrupt
Corrupt block relative dba: 0x3bc88c2b (file 239, block 560171)
Completely zero block found during dbv: 

Page 560179 is marked corrupt
Corrupt block relative dba: 0x3bc88c33 (file 239, block 560179)
Completely zero block found during dbv: 



DBVERIFY - Verification complete

Total Pages Examined         : 4194302
Total Pages Processed (Data) : 2230726
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 1936953
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 26618
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 3
Total Pages Marked Corrupt   : 2
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 304929867 (106.304929867)

修复完相关无法拷贝文件之后,启动数据库报控制文件异常

Mon Mar 29 15:03:38 2021
alter database mount
USER (ospid: 29044): terminating the instance
Mon Mar 29 15:03:42 2021
System state dump requested by (instance=1, osid=29044), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/xff/xff/trace/xff_diag_28961.trc
Instance terminated by USER, pid = 29044

尝试重建ctl

[oracle@localhost ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Mar 29 17:40:17 2021

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount pfile='/tmp/pfile'
ORACLE instance started.

Total System Global Area 1.7704E+10 bytes
Fixed Size                  2235568 bytes
Variable Size            2348811088 bytes
Database Buffers         1.5301E+10 bytes
Redo Buffers               52580352 bytes
SQL> @/tmp/ctl.sql
CREATE CONTROLFILE REUSE DATABASE xff NORESETLOGS  NOARCHIVELOG
*
ERROR at line 1:
ORA-01503: CREATE CONTROLFILE failed
ORA-01189: file is from a different RESETLOGS than previous files
ORA-01110: data file 249: '/oradata/xff/system03.dbf'

初步判断是由于对方之前恢复导致部分文件resetlogs scn异常,通过bbed进行判断确认

BBED> set file 1
        FILE#           1

BBED> p kcvfhrls
struct kcvfhrls, 8 bytes                    @116     
   ub4 kscnbas                              @116      0x00000001
   ub2 kscnwrp                              @120      0x0000

BBED> set file 249
        FILE#           249

BBED> p kcvfhrls
struct kcvfhrls, 8 bytes                    @116     
   ub4 kscnbas                              @116      0x00000001
   ub2 kscnwrp                              @120      0x0000

通过bbed修改相关值,然后重建控制文件成功,尝试resetlogs库，报ORA-01248错误

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01248: file 234 was created in the future of incomplete recovery
ORA-01110: data file 234: '/oradata1/xifenfeidata5.DBF'

关于ORA-01248的错误解释

01248, 00000, "file %s was created in the future of incomplete recovery"
// *Cause:  Attempting to do a RESETLOGS open with a file entry in the
//          control file that was originally created after the UNTIL time 
//          of the incomplete recovery.
//          Allowing such an entry may hide the version of the file that 
//          is needed at this time.  The file number may be in use for 
//          a different file which would be lost if the RESETLOGS was allowed.
// *Action: If more recovery is desired then apply redo until the creation
//          time of the file is reached. If the file is not wanted and the
//          same file number is not in use at the stop time of the recovery,
//          then the file can be taken offline with the FOR DROP option.
//          Otherwise a different control file is needed to allow the RESETLOGS.
//          Another backup can be restored and recovered, or a control file can
//          be created via CREATE CONTROLFILE.

大概的意思是文件的创建时间大于文件当前的scn,通过查询确实如此

SQL> select file#,CREATION_CHANGE#,CREATION_TIME from v$datafile_header where file#=234;

           FILE# CREATION_CHANGE# CREATION_
---------------- ---------------- ---------
             234     419298664864 02-AUG-19

SQL> SELECT status,  
  2  to_char(checkpoint_change#,'9999999999999999') "SCN",
  3  to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,FUZZY,
  4  count(*) ROW_NUM
  5  FROM v$datafile_header
  6  GROUP BY status, checkpoint_change#, to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss'),fuzzy
  7  ORDER BY status, checkpoint_change#, checkpoint_time;

STATUS  SCN               CHECKPOINT_TIME     FUZ          ROW_NUM
------- ----------------- ------------------- --- ----------------
ONLINE       417750848223 2021-02-23 23:50:46 YES                7
ONLINE       417750848223 2021-03-21 11:44:25 NO               396

通过对部分scn进行修改（比如减小创建时间的scn)，然后尝试resetlogs库

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 5 with name
"_SYSSMU5_2708889888$" too small
Process ID: 3182
Session ID: 1 Serial number: 3

这个错误比较简单,参考以前的部分文章:在数据库open过程中常遇到ORA-01555汇总数据库open过程遭遇ORA-1555对应sql语句补充,处理之后,数据库open成功

SQL> startup mount;
ORACLE instance started.

Total System Global Area 1.7704E+10 bytes
Fixed Size                  2235568 bytes
Variable Size            2348811088 bytes
Database Buffers         1.5301E+10 bytes
Redo Buffers               52580352 bytes
Database mounted.
SQL> alter database open;

Database altered.

本次数据库恢复基本上完成,已经最大限度恢复数据,导出数据到新库,完成恢复任务

.Globeimposter-Beta666qqz扩展名数据库加密恢复

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：.Globeimposter-Beta666qqz扩展名数据库加密恢复

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

又接一医院客户请求,多套win系统被勒索病毒加密,其中有几套是oracle数据库,请求我们进行分析,确认是否可以恢复.
HOW TO BACK YOUR FILES.txt文件信息

                   YOUR FILES ARE ENCRYPTED !!!

TO DECRYPT, FOLLOW THE INSTRUCTIONS:

To recover data you need decrypt tool.

To get the decrypt tool you should:

1.In the letter include your personal ID! Send me this ID in your first email to me!
2.We can give you free test for decrypt few files (NOT VALUE) and assign the price for decryption all files!
3.After we send you instruction how to pay for decrypt tool and after payment you will receive a decryption tool! 
4.We can decrypt few files in quality the evidence that we have the decoder.


 DO NOT TRY TO DO SOMETHING WITH YOUR FILES BY YOURSELF YOU WILL BRAKE YOUR DATA !!! ONLY WE ARE CAN HELP YOU! CONTACT US:

China.Helper@aol.com

                   ATTENTION !!! THIS IS YOUR PERSONAL ID WICH YOU HAVE TO SEND IN FIRST LETTER:

Tq rx zo f3 B1 Eg S/ m1 SI Yw KS av ip Js /5 oU
uk FL LY Wa pF P1 Dc ss 8l dU cl pE xe Sa Gw oC
Fq /+ rF dz D3 DU Pz S6 6e uB M5 Wx zD 3C DW EC
nk 1I V1 rf zK R4 36 tq 7o bJ rK Rq 81 ib hf lh
+8 Oz rR 4g VM rz FH ST rJ ve 1S K2 PN FL 7I Gg
yp Wq vv 1j V8 Fz vN 0x y9 l2 Ig Ql fD lK MJ +H
Vw WV 80 FY /s OE oG 9V nC TY Ys Zd nQ is T2 Bw
U4 cK yM km OB Ko 8p Yg g/ DA 5N S+ DX e5 /v 0s
A9 Ae B6 Q1 aO Q9 gN 5/ pg HA LS jD 50 1K p6 Jn
T0 g4 MR Gp 3L l4 GM Fv rD Pq gC pp Tf kz 4k vh
ZG rz SB CD 1f lh M5 UA QI mn ky CG es re GI qc
7s 7h aZ /B sR 6V yn /I xC h7 Xc oR 4G uQ ZC DU
Bs Ij AI 1f 0c w0 Y7 Vd xy FI R2 lz L1 8r dK lF
zS SM CK Mb Rm wo EQ ht ht zj 1m R0 NM 0W 0T lA
9A AP vl dA dB XA Fx cH iR ux C8 Hn uv B9 H0 tk
0J Ph Cn VZ S+ 6b NT BT YZ jC Wf ah Ml N5 q6 FS
uZ Tk 5o 0+ Sq 3c lZ 0a SH LR nW jn 1f A2 rg k6
jx qq eD T1 GT 6w cC 6C TP 3j 6Z KV 6D 1N tS Jo
p/ Sl DB J2 yD Q1 u5 Y7 GS E9 /c kh U6 r8 QP wy
jU Fa +Y Um TZ Mo PY gQ /L pj 5d QD EK A8 g2 qY
8Z 1d Np 3M qm Ri Sf Nc IT cN 2O Uj Ou Gw DZ H3
Wb Lo BV mE wZ 4=

被加密文件类似
20210403180555

通过底层分析,只是小部分数据被加密破坏
20210403180929
这个客户相对比较幸运,他们有3月19日的备份,通过结合备份,实现比较好的效果数据恢复
如果此类的数据库文件(oracle,mysql,sql server)等被加密,需要专业恢复技术支持，请联系我们:
电话/微信:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com

.makop病毒加密数据库恢复

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：.makop病毒加密数据库恢复

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

最近接到客户几套oracle数据库所在的机器文件被加密,readme-warning.txt内容如下

::: Greetings :::


Little FAQ:
.1. 
Q: Whats Happen?
A: Your files have been encrypted and now have the "makop" extension. The file structure was not damaged, we did everything possible so that this could not happen.

.2. 
Q: How to recover files?
A: If you wish to decrypt your files you will need to pay in bitcoins.

.3. 
Q: What about guarantees?
A: Its just a business. We absolutely do not care about you and your deals, except getting benefits. If we do not do our work and liabilities - nobody will cooperate with us. Its not in our interests.
To check the ability of returning files, you can send to us any 2 files with SIMPLE extensions(jpg,xls,doc, etc... not databases!) and low sizes(max 1 mb), we will decrypt them and send back to you. That is our guarantee.

.4.
Q: How to contact with you?
A: You can write us to our mailbox: Evilminded@privatemail.com

.5.
Q: How will the decryption process proceed after payment?
A: After payment we will send to you our scanner-decoder program and detailed instructions for use. With this program you will be able to decrypt all your encrypted files.

.6.
Q: If I don抰 want to pay bad people like you?
A: If you will not cooperate with our service - for us, its does not matter. But you will lose your time and data, cause only we have the private key. In practice - time is much more valuable than money.



:::BEWARE:::
DON'T try to change encrypted files by yourself! 
If you will try to use any third party software for restoring your data or antivirus solutions - please make a backup for all encrypted files!
Any changes in encrypted files may entail damage of the private key and, as result, the loss all data.

通过对数据库文件进行分析,可以恢复
20210327185837

通过恢复工具进行处理,直接open数据库,并导入新库
20210327190400
20210327190633
如果此类的数据库文件(oracle,mysql,sql server)等被加密,需要专业恢复技术支持，请联系我们:
电话/微信:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com

ORA-600 kcratr_scan_lastbwr 恢复

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：ORA-600 kcratr_scan_lastbwr 恢复

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有朋友找到我们,系统断电之后,数据库无法正常启动,报ora-600 kcratr_scan_lastbwr错误

Thu Mar 25 20:33:45 2021
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: alter database mount exclusive
alter database open
Ping without log force is disabled
.
Thu Mar 25 20:33:47 2021
Beginning crash recovery of 1 threads
 parallel recovery started with 32 processes
Thu Mar 25 20:33:47 2021
Started redo scan
Hex dump of (file 10, block 176517) in trace file C:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_4176.trc

Reading datafile 'C:\APP\ADMINISTRATOR\ORADATA\ORCL\XFF.DBF' for corruption at rdba: 0x0282b185 (file 10, block 176517)
Reread (file 10, block 176517) found same corrupt data (logically corrupt)
Write verification failed for File 10 Block 176517 (rdba 0x282b185)
Errors in file C:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_4176.trc  (incident=165355):
ORA-00600: ??????, ??: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
Incident details in: C:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_165355\orcl_ora_4176_i165355.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Thu Mar 25 20:33:50 2021
Slave encountered ORA-10388 exception during crash recovery
Thu Mar 25 20:33:50 2021
Slave encountered ORA-10388 exception during crash recovery
Thu Mar 25 20:33:50 2021
Aborting crash recovery due to error 600
Thu Mar 25 20:33:59 2021
Errors in file C:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_4176.trc:
ORA-00600: ??????, ??: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
Thu Mar 25 20:33:59 2021
Errors in file C:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_4176.trc:
ORA-00600: ??????, ??: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
ORA-600 signalled during: alter database open...

故障原因,写丢失导致

Crash or instance recovery may fail because of a lost write even
though one of the mirrors has a good copy.  Reading a file header 
can corrupt a good mirror copy with a bad one.
 
Rediscovery Notes:
 ORA-600 [kcratr_scan_lostwrt] or ORA-600 [kcratr_scan_lastbwr] are signaled 
 even though one of the mirrors has a good copy.

解决方案比较简单直接recover顺利open库
20210326113024

ORA-600 16703直接把orachk备份表插入到tab$恢复

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：ORA-600 16703直接把orachk备份表插入到tab$恢复

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有一个朋友和我说，他们数据库出现了以下错误ORA-600 16703 错误
20210324195416

他们是在虚拟化环境中,可以恢复到上一个快照点,但是主机启动之后,数据库依旧异常,让我们进行处理

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed Mar 24 17:04:01 2021

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ WRITE

SQL> select count(1) from tab$;

  COUNT(1)
----------
         0

很明显tab$已经被清空,数据库无法正常使用.因为库没有crash,尝试把备份的orachk表插入进来

SQL> insert into tab$ select * from ORACHKB514061BDCB10EBA9CF58F3;

6318 rows created.

SQL> commit;

Commit complete.

SQL> select 'DROP TRIGGER '||owner||'."'||TRIGGER_NAME||'";' from dba_triggers w
here TRIGGER_NAME like 'DBMS_%_INTERNAL% '
  2  union all
  3  select 'DROP PROCEDURE '||owner||'."'||a.object_name||'";' from dba_procedu
res a where a.object_name like 'DBMS_%_INTERNAL% '
  4  union all
  5  select 'drop '||object_type||' '||owner||'.'||object_name||';' from dba_obj
ects where object_name in('DBMS_SUPPORT_DBMONITOR','DBMS_SUPPORT_DBMONITORP');

'DROPTRIGGER'||OWNER||'."'||TRIGGER_NAME||'";'
--------------------------------------------------------------------------------

drop PROCEDURE SYS.DBMS_SUPPORT_DBMONITORP;
drop TRIGGER SYS.DBMS_SUPPORT_DBMONITOR;

SQL> drop PROCEDURE SYS.DBMS_SUPPORT_DBMONITORP;

Procedure dropped.

SQL> drop TRIGGER SYS.DBMS_SUPPORT_DBMONITOR;

Trigger dropped.

SQL> commit;

Commit complete.

SQL>

重启数据库,该故障恢复完成,数据完美恢复0丢失.

oracle dul 12.2正式版发布

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：oracle dul 12.2正式版发布

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

oracle官方dul 正式发布 12.2版本(在上次的测试中dul 12.2完美支持Oracle 19c恢复还是beta版本)

[root@iZbp1hx0enix3hix1kvyrxZ tmp]# ./dul      

Data UnLoader: 12.2.0.0.1 - Internal Only - on Sun Mar 21 13:55:39 2021
with 64-bit io functions and the decompression option

Copyright (c) 1994 2021 Bernard van Duijnen All rights reserved.

 Strictly Oracle Internal Use Only
DUL> show parameters;
_SLPE_DEBUG               = FALSE
ALLOW_CHECKSUM_MISMATCH   = FALSE
ALLOW_DBA_MISMATCH        = FALSE
ALLOW_OTHER_OBJNO         = FALSE
ALLOW_TRAILER_MISMATCH    = FALSE
ALLOW_ZERO_IN_DATE_COLUMNS = FALSE
ASM_DO_HARD_CHECKS        = TRUE
AUTO_UPDATE_CHECKSUM      = TRUE
AUTO_UPDATE_TRAILER       = TRUE
BUFFER                    = 104857600
CF_FILES                  = 1022
CF_TABLESPACES            = 64
COMPATIBLE                = 11
CONTROL_FILE              = control.txt
DB_BLOCK_SIZE             = 8192
DB_NAME                   = 
DB_ID                     = 0
DC_COLUMNS                = 2000000
DC_LOB_ENTRIES            = 327680
DC_EXTENTS                = 10000
DC_OBJECTS                = 1000000
DC_SEGMENTS               = 100000
DC_TABLES                 = 10000
DC_USERS                  = 400
DEFAULT_CHARACTER_SET     = 
DEFAULT_NATIONAL_CHARACTER_SET = 
EXPORT_MODE               = true
FEEDBACK                  = 10000
FILE                      = 
FILE_SIZE_IN_MB           = 0
LDR_ENCLOSE_CHAR          = |
LDR_OUTPUT_IN_UTF8        = FALSE
LDR_PHYS_REC_SIZE         = 0
LOGFILE                   = dul.log
MAX_OPEN_FILES            = 8
MAX_SCAN_ROWS             = 0
MAX_SAMPLE_ROWS           = 5
OSD_MAX_THREADS           = 1055
OSD_BIG_ENDIAN_FLAG       = false
OSD_DBA_FILE_BITS         = 10
OSD_FILE_LEADER_SIZE      = 0
OSD_C_STRUCT_ALIGNMENT    = 32
OSD_WORD_SIZE             = 32
PARSE_HEX_ESCAPES         = FALSE
RESET_LOGFILE             = FALSE
SCAN_DATABASE_SCANS_LOB_SEGMENTS = TRUE
SCAN_STEP_SIZE            = 512
TRACE_FLAGS               = 0
UNEXP_MAX_ERRORS          = 1000
UNEXP_VERBOSE             = FALSE
USE_LOB_FILES             = FALSE
USE_SCANNED_EXTENT_MAP    = FALSE
VERIFY_NUMBER_PRECISION   = TRUE
WARN_RECREATE_FILES       = TRUE
WRITABLE_DATAFILES        = FALSE
DUL> exit

Life is DUL without it
[root@iZbp1hx0enix3hix1kvyrxZ tmp]#

.eking扩展名数据库恢复

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：.eking扩展名数据库恢复

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

又一个朋友数据库文件被加密
20210319212054

通过底层分析发现损坏较少
20210319211553
通过自研的oracle数据库比特币加密文件恢复工具处理
20210319231718
实现数据库顺利open,并使用expdp导出数据
20210319231855
如果此类的数据库文件(oracle,mysql,sql server)等被加密,需要专业恢复技术支持，请联系我们:
电话/微信:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com

一次 CRS-1013: ASM 磁盘组中的 OCR 位置不可访问 故障分析

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：一次 CRS-1013: ASM 磁盘组中的 OCR 位置不可访问 故障分析

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有朋友告知我集群突然异常,让我给看看什么原因,集群alert日志

2021-03-14 21:02:15.517 [OHASD(31771)]CRS-8500: Oracle Clusterware OHASD 进程以操作系统进程 ID 31771 开头
2021-03-14 21:02:15.561 [OHASD(31771)]CRS-0714: Oracle Clusterware 发行版 12.1.0.2.0。
2021-03-14 21:02:15.619 [OHASD(31771)]CRS-2112: 已在节点 rac1 上启动 OLR 服务。
2021-03-14 21:02:15.791 [OHASD(31771)]CRS-1301: 已在节点 rac1 上启动 Oracle 高可用性服务。
2021-03-14 21:02:15.910 [OHASD(31771)]CRS-8017: 位置:/etc/oracle/lastgasp具有2个重新启动指导日志文件,0个已发布,0个出现错误
2021-03-14 21:02:16.789 [CSSDAGENT(32015)]CRS-8500: Oracle Clusterware CSSDAGENT 进程以操作系统进程 ID 32015 开头
2021-03-14 21:02:16.868 [CSSDMONITOR(32017)]CRS-8500: Oracle Clusterware CSSDMONITOR 进程以操作系统进程 ID 32017 开头
2021-03-14 21:02:17.751 [ORAROOTAGENT(32008)]CRS-8500: Oracle Clusterware ORAROOTAGENT 进程以操作系统进程 ID 32008 开头
2021-03-14 21:02:17.916 [ORAAGENT(32012)]CRS-8500: Oracle Clusterware ORAAGENT 进程以操作系统进程 ID 32012 开头
2021-03-14 21:02:18.604 [ORAAGENT(32012)]CRS-5011: 检查资源 "ora.asm" 失败: 详细资料见 "(:CLSN00006:)" 
(位于 "/u01/app/gridbase/diag/crs/rac1/crs/trace/ohasd_oraagent_grid.trc")
2021-03-14 21:02:18.969 [ORAAGENT(32117)]CRS-8500: Oracle Clusterware ORAAGENT 进程以操作系统进程 ID 32117 开头
2021-03-14 21:02:19.050 [MDNSD(32130)]CRS-8500: Oracle Clusterware MDNSD 进程以操作系统进程 ID 32130 开头
2021-03-14 21:02:19.117 [EVMD(32132)]CRS-8500: Oracle Clusterware EVMD 进程以操作系统进程 ID 32132 开头
2021-03-14 21:02:20.078 [GPNPD(32151)]CRS-8500: Oracle Clusterware GPNPD 进程以操作系统进程 ID 32151 开头
2021-03-14 21:02:21.145 [GIPCD(32172)]CRS-8500: Oracle Clusterware GIPCD 进程以操作系统进程 ID 32172 开头
2021-03-14 21:02:21.163 [GPNPD(32151)]CRS-2328: 已在节点 rac1 上启动 GPNPD。
2021-03-14 21:02:22.172 [ORAROOTAGENT(32181)]CRS-8500: Oracle Clusterware ORAROOTAGENT 进程以操作系统进程 ID 32181 开头
2021-03-14 21:02:22.339 [CLSECHO(32204)]CRS-10001: 14-Mar-21 21:02 ACFS-9391: 正在检查现有 ADVM/ACFS 安装。
2021-03-14 21:02:22.580 [CLSECHO(32209)]CRS-10001: 14-Mar-21 21:02 ACFS-9392: 正在验证操作系统的 ADVM/ACFS 安装文件。
2021-03-14 21:02:22.598 [CLSECHO(32211)]CRS-10001: 14-Mar-21 21:02 ACFS-9393: 正在验证 ASM 管理员设置。
2021-03-14 21:02:22.646 [CLSECHO(32216)]CRS-10001: 14-Mar-21 21:02 ACFS-9308: 正在加载已安装的 ADVM/ACFS 驱动程序。
2021-03-14 21:02:22.678 [CLSECHO(32219)]CRS-10001: 14-Mar-21 21:02 ACFS-9154: 正在加载 'oracleoks.ko' 驱动程序。
2021-03-14 21:02:22.809 [CLSECHO(32234)]CRS-10001: 14-Mar-21 21:02 ACFS-9154: 正在加载 'oracleadvm.ko' 驱动程序。
2021-03-14 21:02:22.892 [CLSECHO(32290)]CRS-10001: 14-Mar-21 21:02 ACFS-9154: 正在加载 'oracleacfs.ko' 驱动程序。
2021-03-14 21:02:23.054 [CLSECHO(32334)]CRS-10001: 14-Mar-21 21:02 ACFS-9327: 正在验证 ADVM/ACFS 设备。
2021-03-14 21:02:23.079 [CLSECHO(32336)]CRS-10001: 14-Mar-21 21:02 ACFS-9156: 正在检测控制设备 '/dev/asm/.asm_ctl_spec'。
2021-03-14 21:02:23.108 [CLSECHO(32340)]CRS-10001: 14-Mar-21 21:02 ACFS-9156: 正在检测控制设备 '/dev/ofsctl'。
2021-03-14 21:02:23.263 [CLSECHO(32346)]CRS-10001: 14-Mar-21 21:02 ACFS-9322: 已完成
2021-03-14 21:02:28.571 [CSSDMONITOR(32409)]CRS-8500: Oracle Clusterware CSSDMONITOR 进程以操作系统进程 ID 32409 开头
2021-03-14 21:02:28.756 [CSSDAGENT(32425)]CRS-8500: Oracle Clusterware CSSDAGENT 进程以操作系统进程 ID 32425 开头
2021-03-14 21:02:28.975 [OCSSD(32436)]CRS-8500: Oracle Clusterware OCSSD 进程以操作系统进程 ID 32436 开头
2021-03-14 21:02:30.072 [OCSSD(32436)]CRS-1713: CSSD 守护程序已在 hub 模式下启动
2021-03-14 21:02:46.185 [OCSSD(32436)]CRS-1707: 节点 rac1 (编号为 1) 的租约获取已完成
2021-03-14 21:02:47.337 [OCSSD(32436)]CRS-1605: CSSD 表决文件联机: ORCL:OCR3; 详细资料见
 /u01/app/gridbase/diag/crs/rac1/crs/trace/ocssd.trc。
2021-03-14 21:02:47.357 [OCSSD(32436)]CRS-1605: CSSD 表决文件联机: ORCL:OCR2; 详细资料见 
/u01/app/gridbase/diag/crs/rac1/crs/trace/ocssd.trc。
2021-03-14 21:02:47.365 [OCSSD(32436)]CRS-1605: CSSD 表决文件联机: ORCL:OCR1; 详细资料见 
/u01/app/gridbase/diag/crs/rac1/crs/trace/ocssd.trc。
2021-03-14 21:02:48.781 [OCSSD(32436)]CRS-1601: CSSD 重新配置完毕。活动节点为 rac1 rac2 。
2021-03-14 21:02:50.971 [OCTSSD(32591)]CRS-8500: Oracle Clusterware OCTSSD 进程以操作系统进程 ID 32591 开头
2021-03-14 21:02:51.938 [OCTSSD(32591)]CRS-2403: 主机 rac1 上的集群时间同步服务处于观察程序模式。
2021-03-14 21:02:52.140 [OCTSSD(32591)]CRS-2407: 新的集群时间同步服务引用节点为主机 rac2。
2021-03-14 21:02:52.140 [OCTSSD(32591)]CRS-2401: 已在主机 rac1 上启动了集群时间同步服务。
2021-03-14 21:02:52.167 [OCTSSD(32591)]CRS-2409: 主机 rac1 上的时钟与集群标准时间不同步。
由于集群时间同步服务正在以观察程序模式运行, 所以未采取任何操作。
2021-03-14 21:02:59.284 [ORAAGENT(32117)]CRS-5011: 检查资源 "ora.asm" 失败: 详细资料见 "(:CLSN00006:)" (
位于 "/u01/app/gridbase/diag/crs/rac1/crs/trace/ohasd_oraagent_grid.trc")
2021-03-14 21:03:01.486 [ORAAGENT(32117)]CRS-5011: 检查资源 "ora.asm" 失败: 详细资料见 "(:CLSN00006:)" (
位于 "/u01/app/gridbase/diag/crs/rac1/crs/trace/ohasd_oraagent_grid.trc")
2021-03-14 21:03:01.514 [ORAAGENT(32117)]CRS-5011: 检查资源 "ora.asm" 失败: 详细资料见 "(:CLSN00006:)" (
位于 "/u01/app/gridbase/diag/crs/rac1/crs/trace/ohasd_oraagent_grid.trc")
2021-03-14 21:03:18.163 [OCTSSD(32591)]CRS-2407: 新的集群时间同步服务引用节点为主机 rac1。
2021-03-14 21:03:19.406 [OCSSD(32436)]CRS-1625: 节点 rac2 (编号为 2) 已关闭
2021-03-14 21:03:19.419 [OCSSD(32436)]CRS-1601: CSSD 重新配置完毕。活动节点为 rac1 。
2021-03-14 21:03:24.916 [OSYSMOND(318)]CRS-8500: Oracle Clusterware OSYSMOND 进程以操作系统进程 ID 318 开头
2021-03-14 21:03:26.558 [CRSD(325)]CRS-8500: Oracle Clusterware CRSD 进程以操作系统进程 ID 325 开头
2021-03-14 21:03:27.750 [CRSD(325)]CRS-1012: 已在节点 rac1 上启动 OCR 服务。
2021-03-14 21:03:27.807 [CRSD(325)]CRS-1201: 已在节点 rac1 上启动 CRSD。
2021-03-14 21:03:28.470 [ORAAGENT(1027)]CRS-8500: Oracle Clusterware ORAAGENT 进程以操作系统进程 ID 1027 开头
2021-03-14 21:03:28.499 [ORAROOTAGENT(1031)]CRS-8500: Oracle Clusterware ORAROOTAGENT 进程以操作系统进程 ID 1031 开头
2021-03-14 21:03:28.515 [ORAAGENT(1036)]CRS-8500: Oracle Clusterware ORAAGENT 进程以操作系统进程 ID 1036 开头
2021-03-14 21:03:28.666 [ORAAGENT(1036)]CRS-5011: 检查资源 "oracledb" 失败: 详细资料见 "(:CLSN00007:)"
 (位于 "/u01/app/gridbase/diag/crs/rac1/crs/trace/crsd_oraagent_oracle.trc")
2021-03-14 21:03:30.649 [ORAAGENT(32117)]CRS-5011: 检查资源 "ora.asm" 失败: 详细资料见 "(:CLSN00006:)" 
(位于 "/u01/app/gridbase/diag/crs/rac1/crs/trace/ohasd_oraagent_grid.trc")
2021-03-14 21:03:30.718 [ORAAGENT(32117)]CRS-5011: 检查资源 "ora.asm" 失败: 详细资料见 "(:CLSN00006:)" 
(位于 "/u01/app/gridbase/diag/crs/rac1/crs/trace/ohasd_oraagent_grid.trc")
2021-03-14 21:03:30.722 [CRSD(325)]CRS-1024: 由于此节点上的 ASM 实例未处于活动状态, 此节点上的集群就绪服务终止。详细信息见 
(:PROCR00009:) (位于 /u01/app/gridbase/diag/crs/rac1/crs/trace/crsd.trc)。
2021-03-14 21:03:30.736 [ORAROOTAGENT(1031)]CRS-5822: 代理 '/u01/app/grid/12.1.0/bin/orarootagent_root' 已从服务器断开连接。
详细资料见 (:CRSAGF00117:) {0:3:3} (位于 /u01/app/gridbase/diag/crs/rac1/crs/trace/crsd_orarootagent_root.trc)。
2021-03-14 21:03:30.736 [ORAAGENT(1027)]CRS-5822: 代理 '/u01/app/grid/12.1.0/bin/oraagent_grid' 已从服务器断开连接。
详细资料见 (:CRSAGF00117:) {0:1:3} (位于 /u01/app/gridbase/diag/crs/rac1/crs/trace/crsd_oraagent_grid.trc)。
2021-03-14 21:03:30.793 [CRSD(1157)]CRS-8500: Oracle Clusterware CRSD 进程以操作系统进程 ID 1157 开头
2021-03-14 21:03:31.457 [OLOGGERD(1162)]CRS-8500: Oracle Clusterware OLOGGERD 进程以操作系统进程 ID 1162 开头
2021-03-14 21:03:31.798 [ORAAGENT(32117)]CRS-5011: 检查资源 "ora.asm" 失败: 详细资料见 "(:CLSN00006:)" 
(位于 "/u01/app/gridbase/diag/crs/rac1/crs/trace/ohasd_oraagent_grid.trc")
2021-03-14 21:03:31.823 [ORAAGENT(32117)]CRS-5011: 检查资源 "ora.asm" 失败: 详细资料见 "(:CLSN00006:)" 
(位于 "/u01/app/gridbase/diag/crs/rac1/crs/trace/ohasd_oraagent_grid.trc")
2021-03-14 21:03:40.234 [CRSD(1157)]CRS-1013: ASM 磁盘组中的 OCR 位置不可访问。
详细资料见 /u01/app/gridbase/diag/crs/rac1/crs/trace/crsd.trc。
2021-03-14 21:03:40.238 [CRSD(1157)]CRS-0804: 由于 Oracle 集群注册表错误 [PROC-26: 访问物理存储时出错
ORA-15077: 找不到提供所需磁盘组的 ASM 实例
], 集群就绪服务中止。详细资料见 (:CRSD00111:) (位于 /u01/app/gridbase/diag/crs/rac1/crs/trace/crsd.trc)。

从整个集群的启动过程看cssd,crs都起来了,然后等一会由于crs无法访问ocr磁盘组,导致异常.开始crs起来了,证明ocr磁盘组应该是mount成功过.后面看错误提示又无法访问了.根据经验以及ora.asm失败的提示,怀疑很可能是asm实例出现问题了.对于这样的情况,分析asm的alert日志是最好的方法.通过分析日志发现

Sun Mar 14 21:03:24 2021
NOTE: Instance updated compatible.asm to 12.1.0.0.0 for grp 1
Sun Mar 14 21:03:24 2021
SUCCESS: diskgroup ARCHLOG was mounted
Sun Mar 14 21:03:24 2021
NOTE: Instance updated compatible.asm to 12.1.0.0.0 for grp 2
Sun Mar 14 21:03:24 2021
SUCCESS: diskgroup DATA was mounted
Sun Mar 14 21:03:24 2021
NOTE: Instance updated compatible.asm to 12.1.0.0.0 for grp 3
Sun Mar 14 21:03:24 2021
SUCCESS: diskgroup OCR was mounted
Sun Mar 14 21:03:24 2021
NOTE: Instance updated compatible.asm to 12.1.0.0.0 for grp 5
Sun Mar 14 21:03:24 2021
SUCCESS: ALTER DISKGROUP ALL MOUNT /* asm agent call crs *//* {0:9:3} */
Sun Mar 14 21:03:24 2021
WARNING: failed to online diskgroup resource ora.ARCHLOG.dg (unable to communicate with CRSD/OHASD)
WARNING: failed to online diskgroup resource ora.DATA.dg (unable to communicate with CRSD/OHASD)
WARNING: failed to online diskgroup resource ora.OCR.dg (unable to communicate with CRSD/OHASD)
Errors in file /u01/app/gridbase/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_32721.trc  (incident=123423):
ORA-00600: internal error code, arguments: [kfdAuDealloc2], [ARCHLOG], [213], [410], [0], [], [], [], [], [], [], []
Incident details in: /u01/app/gridbase/diag/asm/+asm/+ASM1/incident/incdir_123423/+ASM1_rbal_32721_i123423.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Sun Mar 14 21:03:25 2021
ERROR: An unrecoverable error has been identified in ASM metadata.
Sun Mar 14 21:03:27 2021
NOTE: [crsd.bin@rac1.schic.org (TNS V1-V3) 325] opening OCR file +OCR.255.4294967295
Starting background process ASMB
Sun Mar 14 21:03:27 2021
ASMB started with pid=28, OS id=932 
Sun Mar 14 21:03:27 2021
NOTE: ASMB registering with ASM instance as Standard client 0xffffffffffffffff (reg:3401595347) (new connection)
Sun Mar 14 21:03:27 2021
NOTE: Standard client +ASM1:+ASM:racscan registered, osid 934, mbr 0x0, asmb 932 (reg:3401595347)
Sun Mar 14 21:03:27 2021
NOTE: ASMB connected to ASM instance +ASM1 osid: 934 (Flex mode; client id 0xffffffffffffffff)
Sun Mar 14 21:03:28 2021
NOTE: AMDU dump of disk group ARCHLOG initiated at /u01/app/gridbase/diag/asm/+asm/+ASM1/incident/incdir_123423
ERROR: ORA-600 in COD recovery for diskgroup 1/0x730955f2 (ARCHLOG)
ERROR: ORA-600 thrown in RBAL for group number 1
Sun Mar 14 21:03:30 2021
Errors in file /u01/app/gridbase/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_32721.trc:
ORA-00600: internal error code, arguments: [kfdAuDealloc2], [ARCHLOG], [213], [410], [0], [], [], [], [], [], [], []
Sun Mar 14 21:03:30 2021
Errors in file /u01/app/gridbase/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_32721.trc:
ORA-00600: internal error code, arguments: [kfdAuDealloc2], [ARCHLOG], [213], [410], [0], [], [], [], [], [], [], []
USER (ospid: 32721): terminating the instance due to error 488
Sun Mar 14 21:03:30 2021
System state dump requested by (instance=1, osid=32721 (RBAL)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/gridbase/diag/asm/+asm/+ASM1/trace/+ASM1_diag_32691_20210314210330.trc
Sun Mar 14 21:03:30 2021
Dumping diagnostic data in directory=[cdmp_20210314210330], requested by (instance=1, osid=32721 (RBAL)), s
ummary=[abnormal instance termination].
Sun Mar 14 21:03:30 2021
Instance terminated by USER, pid = 32721

通过上述日志,果然发现ocr磁盘组先mount成功,然后asm实例由于ARCHLOG磁盘组的ORA-00600 kfdAuDealloc2错误而导致整个实例crash,从而使得ocr磁盘组无法被crs访问,从而出现了”CRS-0804: 由于 Oracle 集群注册表错误 [PROC-26: 访问物理存储时出错 ORA-15077: 找不到提供所需磁盘组的 ASM 实例], 集群就绪服务中止”这样的错误提示.进一步分析为什么archlog进程会报这个错误.

SQL> /* ASMCMD */alter diskgroup /*ASMCMD*/ "DATA" drop file '+DATA/xff/XIFENFEI.270.1040985885' 
Sun Mar 14 20:46:46 2021
SUCCESS: /* ASMCMD */alter diskgroup /*ASMCMD*/ "DATA" drop file '+DATA/xff/XIFENFEI.270.1040985885'
Sun Mar 14 20:49:24 2021
NOTE: Dropping directory '+archlog/oracledb/archivelog/2021_03_11' recursively
Sun Mar 14 20:49:24 2021
Errors in file /u01/app/gridbase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_15281.trc  (incident=114015):
ORA-00600: internal error code, arguments: [kfdAuDealloc2], [ARCHLOG], [213], [410], [0], [], [], [], [], [], [], []
Incident details in: /u01/app/gridbase/diag/asm/+asm/+ASM1/incident/incdir_114015/+ASM1_ora_15281_i114015.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Sun Mar 14 20:49:24 2021
ERROR: An unrecoverable error has been identified in ASM metadata.
NOTE:AMDU dump of disk group ARCHLOG initiated at/u01/app/gridbase/diag/asm/+asm/+ASM1/incident/incdir_114015
Sun Mar 14 20:49:28 2021
Errors in file /u01/app/gridbase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_15281.trc  (incident=114016):
ORA-00600: internal error code, arguments: [kfdAuDealloc2], [ARCHLOG], [213], [410], [0], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kfdAuDealloc2], [ARCHLOG], [213], [410], [0], [], [], [], [], [], [], []

因为这个库有一个历史背景:几天前由于存储cache导致,数据库使用备份还原(还原到一个新磁盘组中,老磁盘组没有使用),今天估计是运维人员在清理老磁盘组中不要的文件,然后archlog中的归档日志的时候,清空了+archlog/oracledb/archivelog/2021_03_11中的文件,然后触发asm删除该目录的异常（异常原因估计和上次清理存储cache引起了该磁盘组的元数据异常有关）.该故障的基本思路原因已经清楚：由于archlog磁盘组本身元数据库有问题,清理该磁盘组文件之后,引起该磁盘组删除空目录出发问题,从而使得整个asm 实例crash.进而引起crs异常.解决方法比较简单,因为archlog磁盘组本身已经不需要,直接dd掉磁盘头,让其启动的时候不再mount,故障解决

[grid@rac1 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCHLOG.dg
               ONLINE  OFFLINE      rac1                     STABLE
               ONLINE  OFFLINE      rac2                     STABLE
ora.DATA.dg
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.LISTENER1.lsnr
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.NEWDATA.dg
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.OCR.dg
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.asm
               ONLINE  ONLINE       rac1                     Started,STABLE
               ONLINE  ONLINE       rac2                     Started,STABLE
ora.net1.network
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.ons
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2                     STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1                     STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1                     STABLE
ora.MGMTLSNR
      1        ONLINE  OFFLINE      rac2                     169.254.86.142 7.7.7
                                                             .1,STARTING
ora.cvu
      1        ONLINE  ONLINE       rac1                     STABLE
ora.mgmtdb
      1        ONLINE  OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.oc4j
      1        ONLINE  OFFLINE      rac1                     STARTING
ora.xff.db
      1        ONLINE  OFFLINE      rac1                     STARTING
      2        ONLINE  OFFLINE      rac2                     STARTING
ora.rac1.vip
      1        ONLINE  ONLINE       rac1                     STABLE
ora.rac2.vip
      1        ONLINE  ONLINE       rac2                     STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rac2                     STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rac1                     STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rac1                     STABLE
--------------------------------------------------------------------------------
[grid@rac1 ~]$

Oracle Recovery Tools恢复—ORA-00704 ORA-01555故障

联系：手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题：Oracle Recovery Tools恢复—ORA-00704 ORA-01555故障

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

由于虚拟化环境使用了精简模式(预分配),后面出现分布式存储空间不足,导致虚拟化环境中的数据库服务器异常,通过一系列操作恢复好系统,发现数据库无法open,请求我们给予解决
通过我们的Oracle Database Recovery Check脚本分析,分析文件的checkpoint scn 有部分3月2日,还有一些是2月28日,是严重不一致,而且对应的归档也丢失
20210314194011

基于这样的情况,试试看强制打开库

C:\Users\XIFENFEI>sqlplus  / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on 星期四 3月 11 23:51:39 2021

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> startup mount pfile='d:/pfile.txt'
ORACLE 例程已经启动。

Total System Global Area 1603411968 bytes
Fixed Size                  2281656 bytes
Variable Size             469765960 bytes
Database Buffers         1124073472 bytes
Redo Buffers                7290880 bytes
数据库装载完毕。
SQL> recover database until cancel;
ORA-00279: 更改 57834775 (在 02/28/2021 22:37:35 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\XIFENFEI\PRODUCT\11.2.0.4\DBHOME_1\RDBMS\ARC0000003072_1043082043.0001
ORA-00280: 更改 57834775 (用于线程 1) 在序列 #3072 中


指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: 警告: RECOVER 成功但 OPEN RESETLOGS 将出现如下错误
ORA-01194: 文件 1 需要更多的恢复来保持一致性
ORA-01110: 数据文件 1: 'D:\BAIDUNETDISKDOWNLOAD\DATA\PROD\SYSTEM01.DBF'


ORA-01112: 未启动介质恢复


SQL> alter database open resetlogs;
alter database open resetlogs
*
第 1 行出现错误:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 10 with name
"_SYSSMU10_1197734989$" too small
进程 ID: 7928
会话 ID: 96 序列号: 3

在数据库open的过程中,报ORA-01555错误,这类问题比较明显以前写过类似文章:
在数据库open过程中常遇到ORA-01555汇总
数据库open过程遭遇ORA-1555对应sql语句补充
使用_allow_resetlogs_corruption导致ORA-00704/ORA-01555故障
这次尝试使用自己开发的小程序:Oracle Recovery Tools进行恢复
20210311235641

然后直接尝试打开数据库成功

SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-01113: ?? 1 ??????
ORA-01110: ???? 1: 'D:\BAIDUNETDISKDOWNLOAD\DATA\XFF\SYSTEM01.DBF'


SQL> recover database;
完成介质恢复。
SQL> alter database open;

数据库已更改。

这次证明,对于数据库open过程汇总报ORA-00704 ORA-01555故障,可以通过Oracle Recovery Tools工具一键式open库。
后续安排数据导出,对于个别导出报错的表利用dul进行处理,完成本次恢复任务