联系：手机/微信(+86 17813235971) QQ(107644445)
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
有客户一套110T的数据库由于存储控制器故障导致数据库无法正常启动,启动报错如下：
Wed Feb 2 23:16:13 2022 Recovery of Online Redo Log: Thread 1 Group 7 Seq 1647469 Reading mem 0 Mem# 0: /dev/vgredo6/rredo7b Mem# 1: /dev/vgredo4/rredo7a Wed Feb 2 23:16:14 2022 Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc: ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] [] Wed Feb 2 23:16:15 2022 Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc: ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] [] ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] [] Wed Feb 2 23:16:16 2022 Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc: ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] [] ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] [] ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] [] Wed Feb 2 23:16:16 2022 Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc: ORA-00600: internal error code, arguments: [kghstack_free2], [], [], [], [], [], [], [] ORA-00607: Internal error occurred while making a change to a data block ORA-00602: internal programming exception ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] [] ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] [] ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] [] Wed Feb 2 23:16:26 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_pmon_26722.trc: ORA-07445: exception encountered: core dump [kcbzre1()+6593] [SIGSEGV] [Address not mapped to object] [] [] [] Wed Feb 2 23:16:27 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_pmon_26722.trc: ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV][] [] ORA-07445: exception encountered: core dump [kcbzre1()+6593] [SIGSEGV] [Address not mapped to object] [] [] [] Wed Feb 2 23:16:27 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_pmon_26722.trc: ORA-00602: internal programming exception ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV][] [] ORA-07445: exception encountered: core dump [kcbzre1()+6593] [SIGSEGV] [Address not mapped to object] [] [] [] Wed Feb 2 23:17:11 2022 PSP0: terminating instance due to error 472 Instance terminated by PSP0, pid = 26724
该错误原因是由于redo信息和数据文件block信息不匹配导致无法正常应用日志,从而出现异常,在后续的recover 中还出现以下错误
Fri Feb 18 16:09:59 2022 ALTER DATABASE RECOVER datafile 609,610,611,612,613,614,615,602,603,604,605,606,607,608 Fri Feb 18 16:09:59 2022 Media Recovery Start parallel recovery started with 16 processes Fri Feb 18 16:10:00 2022 Recovery of Online Redo Log: Thread 1 Group 7 Seq 1647469 Reading mem 0 Mem# 0: /dev/vgredo6/rredo7b Mem# 1: /dev/vgredo4/rredo7a Fri Feb 18 16:12:17 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_p000_22509.trc: ORA-00600: internal error code, arguments: [6101], [0], [42], [96], [], [], [], [] Fri Feb 18 16:18:51 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_p000_22509.trc: ORA-10562: Error occurred while applying redo to data block (file# 602, block# 1693691) ORA-10564: tablespace DBS_DCDL_PT ORA-01110: data file 602: '/dev/vgora12/rdbs_dcdl_pt0155' ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 33682645 ORA-00600: internal error code, arguments: [6101], [0], [42], [96], [], [], [], [] Fri Feb 18 16:18:55 2022 Media Recovery failed with error 12801
Fri Feb 18 18:23:59 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_dbw1_22483.trc: ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV][] [] ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] [] ORA-00472: PMON process terminated with error Fri Feb 18 18:24:04 2022 DBW3: terminating instance due to error 472 Fri Feb 18 18:24:04 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_dbw3_22487.trc: ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] [] ORA-00472: PMON process terminated with error Fri Feb 18 18:24:04 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_dbw3_22487.trc: ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV] [] [] ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] [] ORA-00472: PMON process terminated with error Fri Feb 18 18:24:09 2022 LGWR: terminating instance due to error 472 Fri Feb 18 18:24:09 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_lgwr_22489.trc: ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] [] ORA-00472: PMON process terminated with error
从sqlplus中看到类似一些报错
SQL> recover datafile 601; ORA-03113: end-of-file on communication channel SQL> recover datafile 1066; ORA-00283: recovery session canceled due to errors ORA-12801: error signaled in parallel query server P015 ORA-00600: internal error code, arguments: [2037], [207064103], [207064103], [162], [6], [1], [1833009883], [1130705717] SQL> recover datafile 1065; ORA-00283: recovery session canceled due to errors ORA-12801: error signaled in parallel query server P004 ORA-00600: internal error code, arguments: [kcbzpb_1], [142189139], [3], [0], [], [], [], [] SQL> recover datafile 2042; ORA-00283: recovery session canceled due to errors ORA-12801: error signaled in parallel query server P014 ORA-00600: internal error code, arguments: [3020], [627], [3234156], [2633062764], [], [], [], [] ORA-10567: Redo is inconsistent with data block
通过屏蔽一致性,强制open库成功
Sun Feb 20 21:20:06 2022 SMON: enabling tx recovery Sun Feb 20 21:20:06 2022 Database Characterset is ZHS16GBK Sun Feb 20 21:20:07 2022 ORACLE Instance xifenfei (pid = 38) - Error 376 encountered while recovering transaction (74, 17) on object 34131051. Sun Feb 20 21:20:07 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc: ORA-00376: file 1416 cannot be read at this time ORA-01110: data file 1416: '/dev/vgora14/rdbs_icdl_pt116' Sun Feb 20 21:20:08 2022 Stopping background process MMNL Sun Feb 20 21:20:09 2022 ORACLE Instance xifenfei (pid = 38) - Error 376 encountered while recovering transaction (88, 36) on object 33514955. Sun Feb 20 21:20:09 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc: ORA-00376: file 1264 cannot be read at this time ORA-01110: data file 1264: '/dev/vgora14/rdbs_icdl_pt102' Sun Feb 20 21:20:09 2022 Stopping background process MMON Starting background process MMON Starting background process MMNL MMON started with pid=46, OS id=1482 Sun Feb 20 21:20:10 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc: ORA-01578: ORACLE data block corrupted (file # 652, block # 3767844) ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205' Sun Feb 20 21:20:10 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc: ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661) ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205' replication_dependency_tracking turned off (no async multimaster replication found) Sun Feb 20 21:20:11 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc: ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661) ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205' Sun Feb 20 21:20:11 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc: ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661) ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205' Sun Feb 20 21:20:11 2022 LOGSTDBY: Validating controlfile with logical metadata Sun Feb 20 21:20:11 2022 LOGSTDBY: Validation complete Sun Feb 20 21:20:11 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc: ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661) ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205' Completed: alter database open
对于异常undo进行处理,数据库正常open
由于客户短期无法迁移数据,先对于一些坏块进行修复，暂时运行数据库后续有时间窗口进行迁移.