ORA-10877 – Database SOS

Thu Nov 17 03:00:14 2022 Archived Log entry 23860 added for thread 2 sequence 1958 ID 0x6200e2f5 dest 1: Thu Nov 17 03:13:11 2022 Auto-tuning: Shutting down background process GTX1 Thu Nov 17 04:00:02 2022 Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl2\trace\orcl2_lgwr_1740.trc (incident=672186): ORA-00600: 内部错误代码, 参数: [3417], [2], [0], [0], [0], [1], [2], [], [], [], [], [] Thu Nov 17 04:00:04 2022 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl2\trace\orcl2_lgwr_1740.trc: ORA-00600: 内部错误代码, 参数: [3417], [2], [0], [0], [0], [1], [2], [], [], [], [], [] LGWR (ospid: 1740): terminating the instance due to error 470

Thu Nov 17 04:03:09 2022 Successful mount of redo thread 2, with mount id 1648753015 Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE) Lost write protection disabled Completed: ALTER DATABASE MOUNT /* db agent *//* {0:1:38} */ ALTER DATABASE OPEN /* db agent *//* {0:1:38} */ This instance was first to open Beginning crash recovery of 1 threads parallel recovery started with 31 processes Thu Nov 17 04:03:14 2022 Started redo scan ORA-00600: ??????, ??: [3005], [1], [706], [10374], [0], [0], [], [], [], [], [], [] Thu Nov 17 04:03:15 2022 Reconfiguration started (old inc 14, new inc 16) List of instances: 1 2 (myinst: 2) Global Resource Directory frozen Thu Nov 17 04:03:15 2022 Communication channels reestablished Thu Nov 17 04:03:16 2022 * domain 0 valid = 0 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Nov 17 04:03:16 2022 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Thu Nov 17 04:03:16 2022 Sweep [inc][688298]: completed Sweep [inc2][688298]: completed Thu Nov 17 04:03:16 2022 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Thu Nov 17 04:03:16 2022 LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Abort recovery for domain 0 Aborting crash recovery due to error 600 Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl2\trace\orcl2_ora_15352.trc: ORA-00600: ??????, ??: [3005], [1], [706], [10374], [0], [0], [], [], [], [], [], [] Abort recovery for domain 0 Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl2\trace\orcl2_ora_15352.trc: ORA-00600: ??????, ??: [3005], [1], [706], [10374], [0], [0], [], [], [], [], [], [] ORA-600 signalled during: ALTER DATABASE OPEN /* db agent *//* {0:1:38} */...

SQL> recover database; ORA-00279: 更改 310644203 (在 11/17/2022 01:00:05 生成) 对于线程 2 是必需的 ORA-00289: 建议: +DATA/orcl/archivelog/2022_11_17/thread_2_seq_1956.22763.1120960801 ORA-00280: 更改 310644203 (用于线程 2) 在序列 #1956 中指定日志: {<RET>=suggested | filename | AUTO | CANCEL} auto ORA-00279: 更改 310663747 (在 11/17/2022 02:00:01 生成) 对于线程 2 是必需的 ORA-00289: 建议: +DATA/orcl/archivelog/2022_11_17/thread_2_seq_1957.22764.1120962585 ORA-00280: 更改 310663747 (用于线程 2) 在序列 #1957 中 ORA-10877: error signaled in parallel recovery slave ORA-01112: 未启动介质恢复

Thu Nov 17 08:07:39 2022 ALTER DATABASE RECOVER database Media Recovery Start started logmerger process Parallel Media Recovery started with 32 slaves Thu Nov 17 08:07:41 2022 Recovery of Online Redo Log: Thread 1 Group 1 Seq 705 Reading mem 0 Mem# 0: +DATA/orcl/onlinelog/group_1.261.1116409583 ORA-279 signalled during: ALTER DATABASE RECOVER database ... Thu Nov 17 08:08:07 2022 ALTER DATABASE RECOVER CONTINUE DEFAULT Media Recovery Log +DATA/orcl/archivelog/2022_11_17/thread_2_seq_1956.22763.1120960801 ORA-279 signalled during: ALTER DATABASE RECOVER CONTINUE DEFAULT ... ALTER DATABASE RECOVER CONTINUE DEFAULT Media Recovery Log +DATA/orcl/archivelog/2022_11_17/thread_2_seq_1957.22764.1120962585 Thu Nov 17 08:08:14 2022 Recovery of Online Redo Log: Thread 2 Group 4 Seq 1958 Reading mem 0 Mem# 0: +DATA/orcl/onlinelog/group_4.266.1116409589 Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl2\trace\orcl2_pr00_7116.trc (incident=704315): ORA-00353: 日志损坏接近块 20866 更改 310761542 时间 11/17/2022 03:00:04 ORA-00312: 联机日志 1 线程 1: '+DATA/orcl/onlinelog/group_1.261.1116409583' Thu Nov 17 08:08:26 2022 Sweep [inc][704315]: completed Thu Nov 17 08:08:27 2022 Media Recovery failed with error 354 Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl2\trace\orcl2_pr00_7116.trc: ORA-00283: 恢复会话因错误而取消 ORA-00354: 损坏重做日志块标头 ORA-00353: 日志损坏接近块 20866 更改 310761542 时间 11/17/2022 03:00:04 ORA-00312: 联机日志 1 线程 1: '+DATA/orcl/onlinelog/group_1.261.1116409583' Thu Nov 17 08:08:27 2022 ORA-10877 signalled during: ALTER DATABASE RECOVER CONTINUE DEFAULT ... ALTER DATABASE RECOVER CANCEL ORA-1112 signalled during: ALTER DATABASE RECOVER CANCEL ...

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 3020错误引起ORA-600 2663

数据库recover异常ORA-600 3020

SQL> recover database using backup controlfile until cancel;
ORA-00279: change 5693717234723 generated at 01/19/2021 10:44:52 needed for
thread 1
ORA-00289: suggestion : +RECOVER/arch/1_294845_938895110.dbf
ORA-00280: change 5693717234723 for thread 1 is in sequence #294845


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
+BACKUP/xifenfei/onlinelog/group_5.258.973180257
ORA-00279: change 5693717234723 generated at 01/15/2021 11:41:15 needed for
thread 2
ORA-00289: suggestion : +RECOVER/arch/2_336576_938895110.dbf
ORA-00280: change 5693717234723 for thread 2 is in sequence #336576


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
+DATA1/xifenfei/onlinelog/group_8.298.962885887
ORA-00600: internal error code, arguments: [3020], [128], [248606],
[537119518], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 128, block# 248606, file
offset is 2036580352 bytes)
ORA-10564: tablespace UNDOTBS1
ORA-01110: data file 128: '+DATA1/xifenfei/datafile/undotbs1_02.dbf'
ORA-10560: block type 'KTU UNDO BLOCK'


ORA-01112: media recovery not started

这个错误比较简单,一般是允许坏块继续恢复

SQL> recover database using backup controlfile allow 1 corruption;
ORA-00279: change 5693717234839 generated at 01/19/2021 10:44:52 needed for
thread 1
ORA-00289: suggestion : +RECOVER/arch/1_294845_938895110.dbf
ORA-00280: change 5693717234839 for thread 1 is in sequence #294845


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
+BACKUP/xifenfei/onlinelog/group_5.258.973180257
ORA-00279: change 5693717234839 generated at 01/15/2021 11:41:15 needed for
thread 2
ORA-00289: suggestion : +RECOVER/arch/2_336576_938895110.dbf
ORA-00280: change 5693717234839 for thread 2 is in sequence #336576


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
+DATA1/xifenfei/onlinelog/group_8.298.962885887
ORA-00279: change 5693717637654 generated at 01/19/2021 10:47:25 needed for
thread 1
ORA-00289: suggestion : +RECOVER/arch/1_294846_938895110.dbf
ORA-00280: change 5693717637654 for thread 1 is in sequence #294846
ORA-00278: log file '+BACKUP/xifenfei/onlinelog/group_5.258.973180257' no longer
needed for this recovery


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
+RECOVER/xifenfei/onlinelog/group_3.258.973180321
ORA-00279: change 5693717705759 generated at 01/19/2021 10:48:07 needed for
thread 1
ORA-00289: suggestion : +RECOVER/arch/1_294847_938895110.dbf
ORA-00280: change 5693717705759 for thread 1 is in sequence #294847
ORA-00278: log file '+RECOVER/xifenfei/onlinelog/group_3.258.973180321' no
longer needed for this recovery


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
+BACKUP/xifenfei/onlinelog/group_7.265.973181365
Log applied.
Media recovery complete.

后续重建ctl,尝试recover库,报ORA-10877错误

SQL> startup mount pfile='/tmp/pfile'
ORACLE instance started.

Total System Global Area 1.0088E+10 bytes
Fixed Size		    2261928 bytes
Variable Size		 2181041240 bytes
Database Buffers	 7851737088 bytes
Redo Buffers		   53149696 bytes
Database mounted.
SQL> recover database;
ORA-10877: error signaled in parallel recovery slave


--对应的alert日志
Wed Jan 20 13:34:04 2021
ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 64 slaves
Wed Jan 20 13:34:06 2021
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_pr00_50593.trc:
ORA-00313: open failed for members of log group 7 of thread 1
Media Recovery failed with error 313
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_pr00_50593.trc:
ORA-00283: recovery session canceled due to errors
ORA-00313: open failed for members of log group 7 of thread 1
ORA-10877 signalled during: ALTER DATABASE RECOVER  database  ...

resetlogs失败open数据库失败,ORA-600 2663

Wed Jan 20 13:42:34 2021
Setting recovery target incarnation to 2
Initializing SCN for created control file
Database SCN compatibility initialized to 3
Warning - High Database SCN: Current SCN value is 5693718057561, threshold SCN value is 0
If you have not previously reported this warning on this database, please notify Oracle Support so that additional diagnosis can be performed.
Wed Jan 20 13:42:35 2021
Assigning activation ID 3801294256 (0xe29325b0)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: +RECOVER/xifenfei/onlinelog/group_1.260.973179783
  Current log# 1 seq# 1 mem# 1: +BACKUP/xifenfei/onlinelog/group_1.260.973179787
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Jan 20 13:42:35 2021
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_52800.trc  (incident=189187):
ORA-00600: internal error code, arguments: [2663], [1325], [2886390384], [1325], [2886403118], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_189187/xifenfei1_ora_52800_i189187.trc
Wed Jan 20 13:42:38 2021
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_52800.trc:
ORA-00600: internal error code, arguments: [2663], [1325], [2886390384], [1325], [2886403118], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_52800.trc:
ORA-00600: internal error code, arguments: [2663], [1325], [2886390384], [1325], [2886403118], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 52800): terminating the instance due to error 600

这个错误比较明显,由于scn的异常导致,通过调整scn,数据库正常open成功,然后使用hcheck检查数据库字典一致(运气不错),没有太大问题,后续建议客户进行逻辑迁移

分类目录归档：ORA-10877