ORA-19821故障分析

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:ORA-19821故障分析

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

数据库报错
数据库启动报ORA-00283和ORA-19821错

SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-19821: an intentionally corrupt log file was found
SQL> recover datafile 1;
ORA-00283: recovery session canceled due to errors
ORA-19821: an intentionally corrupt log file was found
Thu May 03 12:06:51 2018
Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0
  Mem# 0: D:\APP\SOHTDB\ORADATA\xifenfei\REDO01.LOG
Media Recovery failed with error 19821
Errors in file d:\app\sohtdb\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_pr00_660.trc:
ORA-00283: recovery session canceled due to errors
ORA-19821: an intentionally corrupt log file was found
Slave exiting with ORA-283 exception
Errors in file d:\app\sohtdb\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_pr00_660.trc:
ORA-00283: recovery session canceled due to errors
ORA-19821: an intentionally corrupt log file was found
ORA-283 signalled during: ALTER DATABASE RECOVER  database  ...

ORA-19821报错原因
这个错误相对比较少见,查询mos,由于设置了_disable_logging = TRUE导致该问题
ORA-19821


检查alert日志,_disable_logging参数确实被认为设置为true了.
_disable_logging


解决方法
根据官方的描述,这样的情况无法常规恢复,但是我们知道设置这个参数是为了不产生日志,因此出现这种情况,只能通过隐含参数,禁止数据库进行实例恢复,强制打开数据库.在这样的过程中非常容易遭遇类似ORA-600 2662的错误.
_disable_logging_mos


设置这个参数是为了不产生日志,当数据库非干净关闭(主机断电,数据库crash,shutdown abort等),就非常可能导致数据库无法正常启动.***千不可万不能在生产环境中设置_disable_logging = TRUE***
参考文档:Ora-19821 during the recovery (Doc ID 1217143.1)
Init.ora Parameter “_DISABLE_LOGGING” [Hidden] Reference Note (Doc ID 29552.1)

ORA-00333 故障恢复

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:ORA-00333 故障恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

数据库启动报错

SQL> startup
ORACLE instance started.
Total System Global Area 1241513984 bytes
Fixed Size                  1219136 bytes
Variable Size             218105280 bytes
Database Buffers         1006632960 bytes
Redo Buffers               15556608 bytes
Database mounted.
ORA-00333: redo log read error block 48641 count 8192

数据库启动报ORA-00333错误,官方解释为读redo log发生错误.

00333, 00000, "redo log read error block %s count %s"
// *Cause:  An IO error occurred while reading the log described in the
//          accompanying error.
// *Action: Restore accessibility to file, or get another copy of the file.

alert日志

Sat Apr 14 00:39:13 2018
 alter database open
Sat Apr 14 00:39:13 2018
Beginning crash recovery of 1 threads
 parallel recovery started with 7 processes
Sat Apr 14 00:39:13 2018
Started redo scan
Sat Apr 14 00:39:14 2018
Errors in file /oracle/admin/oa/udump/oa_ora_5659.trc:
ORA-00333: redo log read error block 54785 count 2048
ORA-00312: online log 1 thread 1: '/oracle/oradata/oa/redo01.log'
ORA-27072: File I/O error
Linux Error: 5: Input/output error
Additional information: 4
Additional information: 54785
Additional information: 957952
Sat Apr 14 00:39:14 2018
Errors in file /oracle/admin/oa/udump/oa_ora_5659.trc:
ORA-00333: redo log read error block 48641 count 8192
ORA-00312: online log 1 thread 1: '/oracle/oradata/oa/redo01.log'
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
Linux Error: 5: Input/output error
Additional information: 4
Additional information: 54785
Additional information: 957952
Sat Apr 14 00:39:14 2018
Aborting crash recovery due to error 333
Sat Apr 14 00:39:14 2018
Errors in file /oracle/admin/oa/udump/oa_ora_5659.trc:
ORA-00333: redo log read error block 48641 count 8192
ORA-333 signalled during:  alter database open...

由于硬件异常,数据库在启动的时候读取redo异常,从而使得数据库无法正常启动

检查系统日志

Apr 14 11:14:58 oa kernel: Info fld=0x0, Current sda: sense key Hardware Error
Apr 14 11:14:59 oa kernel: Additional sense: Internal target failure
Apr 14 11:14:59 oa kernel: end_request: I/O error, dev sda, sector 190500041
Apr 14 11:14:59 oa kernel: SCSI error : <0 0 0 0> return code = 0x8000002
Apr 14 11:14:59 oa kernel: Info fld=0x0, Current sda: sense key Hardware Error
Apr 14 11:14:59 oa kernel: Additional sense: Internal target failure
Apr 14 11:14:59 oa kernel: end_request: I/O error, dev sda, sector 190500049
Apr 14 11:14:59 oa kernel: SCSI error : <0 0 0 0> return code = 0x8000002
Apr 14 11:14:59 oa kernel: Info fld=0x0, Current sda: sense key Hardware Error
Apr 14 11:14:59 oa kernel: Additional sense: Internal target failure
Apr 14 11:14:59 oa kernel: end_request: I/O error, dev sda, sector 190500057
Apr 14 11:14:59 oa kernel: SCSI error : <0 0 0 0> return code = 0x8000002
Apr 14 11:14:59 oa kernel: Info fld=0x0, Current sda: sense key Hardware Error
Apr 14 11:14:59 oa kernel: Additional sense: Internal target failure
Apr 14 11:14:59 oa kernel: end_request: I/O error, dev sda, sector 190500065
Apr 14 11:14:59 oa kernel: SCSI error : <0 0 0 0> return code = 0x8000002

大量类似I/O error, dev sda, sector错误,很可能是由于硬件方面异常导致.

损坏redo为当前redo
redo


针对这样的情况,由于是硬件故障,先要通过dbv或者rman检查其他数据文件是否正常,如果有数据文件不能读,那需要对数据文件进行特殊处理.本次恢复的中,客户相对比较幸运,所有数据文件全部可以正常访问,只是当前redo异常,通过隐含参数强制拉库,然后导出数据,重建库解决.类似文章:又一起存储故障导致ORA-00333 ORA-00312恢复

OSD-04016: 异步 I/O 请求排队时出错

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:OSD-04016: 异步 I/O 请求排队时出错

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有某客户由于硬件故障,导致数据库无法启动,让我们介入处理
数据库启动报错

Mon Feb 26 17:28:24 2018
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 3 processes
Started redo scan
Completed redo scan
 read 2054 KB redo, 509 data blocks need recovery
Started redo application at
 Thread 1: logseq 41341, block 54
Recovery of Online Redo Log: Thread 1 Group 1 Seq 41341 Reading mem 0
  Mem# 0: E:\ORADATA\ORCL\REDO01.LOG
Completed redo application of 1.77MB
KCF: read, write or open error, block=0x16439 online=1
        file=1 'E:\ORADATA\ORCL\SYSTEM01.DBF'
        error=27070 txt: 'OSD-04016: 异步 I/O 请求排队时出错。
O/S-Error: (OS 1) 函数不正确。'
Errors in file d:\oracle\diag\rdbms\orcl\orcl\trace\orcl_dbw0_4928.trc:
ORA-01243: system tablespace file suffered media failure
ORA-01114: IO error writing block to file 1 (block # 91193)
ORA-01110: data file 1: 'E:\ORADATA\ORCL\SYSTEM01.DBF'
ORA-27070: async read/write failed
OSD-04016: 异步 I/O 请求排队时出错。
O/S-Error: (OS 1) 函数不正确。
DBW0 (ospid: 4928): terminating the instance due to error 1243
Mon Feb 26 17:28:29 2018
Instance terminated by DBW0, pid = 4928

20180227223540


这里错误比较明显,由于io错误,在数据库实例恢复之时,写回block正好在该损坏位置从而使得数据库无法正常实例恢复,进而无法open.

dbv验证文件
通过专业工具对system文件进行了重构system文件,然后dbv检查结果如下

DBVERIFY: Release 11.2.0.1.0 - Production on 星期二 2月 27 17:20:14 2018
Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.
DBVERIFY - 开始验证: FILE = D:\OK\SYSTEM01.DBF
页 91156 标记为损坏
Corrupt block relative dba: 0x00416414 (file 1, block 91156)
Bad header found during dbv:
Data in bad block:
 type: 229 format: 6 rdba: 0xe1d9e3e7
 last change scn: 0xd682.ffc8c7cb seq: 0x8c flg: 0x8c
 spare1: 0xc0 spare2: 0xf6 spare3: 0x70b3
 consistency value in tail: 0x71f50602
 check value in block header: 0x8195
 computed block checksum: 0x8689
DBVERIFY - 验证完成
检查的页总数: 215040
处理的页总数 (数据): 178112
失败的页总数 (数据): 0
处理的页总数 (索引): 19070
失败的页总数 (索引): 0
处理的页总数 (其他): 3118
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 14739
标记为损坏的总页数: 1
流入的页总数: 0
加密的总页数        : 0
最高块 SCN            : 1638554501 (0.1638554501)

这里比较明显重构出来的system文件只有block 91156坏块,这里注意和没有处理之前的坏块不一样

通过dump分析坏块所属对象

Start dump data block from file D:\OK\SYSTEM01.DBF minblk 91156 maxblk 91156
 V10 STYLE FILE HEADER:
	Compatibility Vsn = 186646528=0xb200000
	Db ID=1383974140=0x527dc4fc, Db Name='ORCL'
	Activation ID=0=0x0
	Control Seq=806694=0xc4f26, File size=215040=0x34800
	File Number=1, Blksiz=8192, File Type=3 DATA
Dump all the blocks in range:
buffer tsn: 0 rdba: 0xe1d9e3e7 (903/1696743)
scn: 0xd682.ffc8c7cb seq: 0x8c flg: 0x8c tail: 0x71f50602
frmt: 0x06 chkval: 0x8195 type: 0xe5=unknown
Hex dump of corrupt header 4 = CORRUPT
Start dump data block from file D:\OK\SYSTEM01.DBF minblk 91155 maxblk 91155
 V10 STYLE FILE HEADER:
	Compatibility Vsn = 186646528=0xb200000
	Db ID=1383974140=0x527dc4fc, Db Name='ORCL'
	Activation ID=0=0x0
	Control Seq=806694=0xc4f26, File size=215040=0x34800
	File Number=1, Blksiz=8192, File Type=3 DATA
Dump all the blocks in range:
buffer tsn: 0 rdba: 0x00416413 (1/91155)
scn: 0x0000.613ad8d6 seq: 0x01 flg: 0x06 tail: 0xd8d60601
frmt: 0x02 chkval: 0xcc5f type: 0x06=trans data
Hex dump of block: st=0, typ_found=1
Block header dump:  0x00416413
 Object id on Block? Y
 seg/obj: 0x25  csc: 0x00.613ad8ce  itc: 2  flg: -  typ: 2 - INDEX
     fsl: 0  fnx: 0x0 ver: 0x01
 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x0009.000.0016fda0  0x00c0130e.6f38.01  CB--    0  scn 0x0000.3e6ed294
0x02   0x0009.008.0023e862  0x00c002cf.a217.13  --U-    1  fsc 0x0000.613ad8d6
Start dump data block from file D:\OK\SYSTEM01.DBF minblk 91157 maxblk 91157
 V10 STYLE FILE HEADER:
	Compatibility Vsn = 186646528=0xb200000
	Db ID=1383974140=0x527dc4fc, Db Name='ORCL'
	Activation ID=0=0x0
	Control Seq=806694=0xc4f26, File size=215040=0x34800
	File Number=1, Blksiz=8192, File Type=3 DATA
Dump all the blocks in range:
buffer tsn: 0 rdba: 0x00416415 (1/91157)
scn: 0x0000.6193dc0c seq: 0x01 flg: 0x06 tail: 0xdc0c0601
frmt: 0x02 chkval: 0x8c21 type: 0x06=trans data
Hex dump of block: st=0, typ_found=1
Block header dump:  0x00416415
 Object id on Block? Y
 seg/obj: 0x25  csc: 0x00.6193dc04  itc: 2  flg: -  typ: 2 - INDEX
     fsl: 0  fnx: 0x0 ver: 0x01
 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x0006.00c.001ee103  0x00c008a5.8dbd.02  C---    0  scn 0x0000.57303e03
0x02   0x000b.00f.00053100  0x00c008af.1563.09  --U-    1  fsc 0x0000.6193dc0c

这里比较明显,可以确定坏块为index,object_id=0x25=37,通过查询其他库,确定为i_obj2(obj$的index)

使用该文件启动数据库

SQL> alter database rename file 'E:\ORADATA\ORCL\SYSTEM01.DBF' to 'd:\orcl\SYSTEM01.DBF';
数据库已更改。
SQL> recover database;
完成介质恢复。
SQL> alter database open;
数据库已更改。

检查alert日志
发现smon进程由于坏块的存储,出现大量报错,需要处理,不然数据库一段时间后就会crash.

Tue Feb 27 20:31:23 2018
QMNC started with pid=27, OS id=4652
Completed: alter database open
Tue Feb 27 20:31:25 2018
Starting background process CJQ0
Tue Feb 27 20:31:25 2018
CJQ0 started with pid=30, OS id=2172
Tue Feb 27 20:31:25 2018
db_recovery_file_dest_size of 3912 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Hex dump of (file 1, block 91156) in trace file d:\oracle\diag\rdbms\orcl\orcl\trace\orcl_cjq0_2172.trc
Corrupt block relative dba: 0x00416414 (file 1, block 91156)
Bad header found during multiblock buffer read
Data in bad block:
 type: 229 format: 6 rdba: 0xe1d9e3e7
 last change scn: 0xd682.ffc8c7cb seq: 0x8c flg: 0x8c
 spare1: 0xc0 spare2: 0xf6 spare3: 0x70b3
 consistency value in tail: 0x71f50602
 check value in block header: 0x8195
 computed block checksum: 0x8689
Reading datafile 'D:\ORCL\SYSTEM01.DBF' for corruption at rdba: 0x00416414 (file 1, block 91156)
Reread (file 1, block 91156) found same corrupt data
Errors in file d:\oracle\diag\rdbms\orcl\orcl\trace\orcl_smon_3992.trc  (incident=77085):
ORA-01578: ORACLE data block corrupted (file # 1, block # 91156)
ORA-01110: data file 1: 'D:\ORCL\SYSTEM01.DBF'
Incident details in: d:\oracle\diag\rdbms\orcl\orcl\incident\incdir_77085\orcl_smon_3992_i77085.trc
Errors in file d:\oracle\diag\rdbms\orcl\orcl\trace\orcl_cjq0_2172.trc  (incident=77221):
ORA-01578: ORACLE data block corrupted (file # 1, block # 91156)
ORA-01110: data file 1: 'D:\ORCL\SYSTEM01.DBF'
Incident details in: d:\oracle\diag\rdbms\orcl\orcl\incident\incdir_77221\orcl_cjq0_2172_i77221.trc

重建i_obj2 index,参考:http://www.xifenfei.com/?p=5566

文件read only导致数据库无法open

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:文件read only导致数据库无法open

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

在一次客户的恢复请求中,系统被黑客入侵,数据库无法启动.虽然system文件可以拷贝,但是无论拷贝到哪里,都无法dbv和open数据库,报O/S-Error: (OS 5)和DBV-00600错误.最后检查发现是文件设置了只读
数据库启动报错

ORACLE 例程已经启动。
Total System Global Area 6881345536 bytes
Fixed Size                  2130160 bytes
Variable Size            4026535696 bytes
Database Buffers         2818572288 bytes
Redo Buffers               34107392 bytes
数据库装载完毕。
ORA-01113: 文件 1 需要介质恢复
ORA-01110: 数据文件 1: 'D:\APP\ADMINISTRATOR\ORADATA\xifenfei\SYSTEM01.DBF'
SQL> recover datafile 1;
ORA-00283: 恢复会话因错误而取消
ORA-01114: 将块写入文件 1 时出现 IO 错误 (块 # 1)
ORA-01110: 数据文件 1: 'D:\APP\ADMINISTRATOR\ORADATA\xifenfei\SYSTEM01.DBF'
ORA-27091: 无法将 I/O 排队
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 5) 拒绝访问。
SQL> recover database until cancel using backup controlfile;
ORA-00283: 恢复会话因错误而取消
ORA-01201: 文件 1 标头无法正确写入

alert日志报错

Completed: ALTER DATABASE   MOUNT
Sun Dec 10 12:03:20 2017
alter database open
Errors in file d:\app\administrator\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_ora_6860.trc:
ORA-01110: 数据文件 1: 'D:\APP\ADMINISTRATOR\ORADATA\xifenfei\SYSTEM01.DBF'
ORA-01114: 将块写入文件 1 时出现 IO 错误 (块 # 1)
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 5) 拒绝访问。
ORA-01201: 文件 1 标头无法正确写入
ORA-1113 signalled during: alter database open...
Sun Dec 10 12:03:33 2017
ALTER DATABASE RECOVER  datafile 1
Media Recovery Start
Fast Parallel Media Recovery NOT enabled
Media Recovery failed with error 1114
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 1  ...

dbv检查文件

D:\APP\ADMINISTRATOR\ORADATA>dbv file=system01.dbf
DBVERIFY: Release 11.1.0.7.0 - Production on 星期日 12月 10 19:06:55 2017
Copyright (c) 1982, 2007, Oracle.  All rights reserved.
DBV-00600: 致命错误 - [22] [27041] [0] [0]

win-readonly


这里很明显,文件不能写操作的原因,就是由于该属性导致

12c ORA-01113 ORA-01110 恢复

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:12c ORA-01113 ORA-01110 恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

数据库版本
VERSION


数据库启动报错

Completed: ALTER DATABASE   MOUNT
Thu Aug 17 12:34:52 2017
alter database open
Thu Aug 17 12:34:52 2017
Ping without log force is disabled
.
Thu Aug 17 12:34:52 2017
Errors in file D:\APP\ORACLE\diag\rdbms\XIFENFEI\XIFENFEI\trace\XIFENFEI_ora_5960.trc:
ORA-01113: 文件 5 需要介质恢复
ORA-01110: 数据文件 5: 'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_UNDOTBS1_DN9MQJFK_.DBF'
ORA-1113 signalled during: alter database open

客户尝试恢复

SQL> startup mount;
ORACLE 例程已经启动。
Total System Global Area 1.0301E+10 bytes
Fixed Size                  3842760 bytes
Variable Size            1778388280 bytes
Database Buffers         8489271296 bytes
Redo Buffers               29708288 bytes
数据库装载完毕。
SQL> alter  database open;
alter  database open
*
第 1 行出现错误:
ORA-01113: 文件 5 需要介质恢复
ORA-01110: 数据文件 5: 'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_UNDOTBS1_DN9MQJFK_.DBF'
SQL> alter database datafile 5 offline drop;
数据库已更改。
SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-01113: 文件 6 需要介质恢复
ORA-01110: 数据文件 6: 'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_USERS_DN9MQH75_.DBF'
SQL> recover datafile 6;
ORA-00283: 恢复会话因错误而取消
ORA-00322: 日志 3 (用于线程 1) 不是最新副本
ORA-00312: 联机日志 3 线程 1: 'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6DW_.LOG'
ORA-00322: 日志 3 (用于线程 1) 不是最新副本
ORA-00312: 联机日志 3 线程 1: 'D:\APP\ORACLE\ORADATA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6BX_.LOG'

使用Oracle Database Recovery Check检测结果
check_ctl
check_resulte


尝试恢复

SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-00322: log 3 of thread 1 is not current copy
ORA-00312: online log 3 thread 1:
'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6DW_.LOG'
ORA-00322: log 3 of thread 1 is not current copy
ORA-00312: online log 3 thread 1:
'D:\APP\ORACLE\ORADATA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6BX_.LOG'
SQL> recover database until cancel
ORA-00279: 更改 9843709 (在 08/17/2017 07:04:02 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ARCHIVELOG\2017_08_17\O1_MF_1_717_%U_.ARC
ORA-00280: 更改 9843709 (用于线程 1) 在序列 #717 中
指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_2_DN9MV69G_.LOG
ORA-00310: archived log contains sequence 716; sequence 717 required
ORA-00334: archived log:
'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_2_DN9MV69G_.LOG'
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1:
'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_SYSTEM_DN9MN5OT_.DBF'
SQL> recover database until cancel
ORA-00279: 更改 9843709 (在 08/17/2017 07:04:02 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ARCHIVELOG\2017_08_17\O1_MF_1_717_%U_.AR
C
ORA-00280: 更改 9843709 (用于线程 1) 在序列 #717 中
指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV661_.LOG
ORA-00310: archived log contains sequence 715; sequence 717 required
ORA-00334: archived log:
'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV661_.LOG'
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1:
'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_SYSTEM_DN9MN5OT_.DBF'
SQL> recover database until cancel
ORA-00279: 更改 9843709 (在 08/17/2017 07:04:02 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ARCHIVELOG\2017_08_17\O1_MF_1_717_%U_.ARC
ORA-00280: 更改 9843709 (用于线程 1) 在序列 #717 中
指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6DW_.LOG
已应用的日志。
完成介质恢复。
SQL> alter database datafile 5 online;
数据库已更改。
SQL> recover database until cancel;
ORA-00279: 更改 9843709 (在 08/17/2017 07:04:02 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ARCHIVELOG\2017_08_17\O1_MF_1_717_%U_.AR
C
ORA-00280: 更改 9843709 (用于线程 1) 在序列 #717 中
指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6DW_.LOG
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1:
'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_SYSTEM_DN9MN5OT_.DBF'
ORA-01112: 未启动介质恢复
SQL> recover datafile 5;
ORA-00283: 恢复会话因错误而取消
ORA-00264: 不要求恢复
SQL> SELECT status,
  2  checkpoint_change#,
  3  checkpoint_time,FUZZY,
  4  count(*) ROW_NUM
  5  FROM v$datafile_header
  6  GROUP BY status, checkpoint_change#, checkpoint_time,fuzzy
  7  ORDER BY status, checkpoint_change#, checkpoint_time;
STATUS  CHECKPOINT_CHANGE# CHECKPOINT_TIME     FUZ          ROW_NUM
------- ------------------ ------------------- --- ----------------
ONLINE             9850826 2017-08-17 08:15:45 NO                 1
ONLINE             9857411 2017-08-17 08:15:45 NO                 7
ONLINE             9857411 2017-08-17 09:00:48 NO                 2
SQL> set numw 16
SQL> SELECT status,
  2  checkpoint_change#,
  3  checkpoint_time,last_change#,
  4  count(*) ROW_NUM
  5  FROM v$datafile
  6  GROUP BY status, checkpoint_change#, checkpoint_time,last_change#
  7  ORDER BY status, checkpoint_change#, checkpoint_time;
STATUS  CHECKPOINT_CHANGE# CHECKPOINT_TIME         LAST_CHANGE#          ROW_NUM
------- ------------------ ------------------- ---------------- ----------------
ONLINE             9857411 2017-08-17 09:00:48          9850826                1
ONLINE             9857411 2017-08-17 09:00:48          9857411                8
SYSTEM             9857411 2017-08-17 09:00:48          9857411                1
SQL> alter database open resetlogs;
alter database open resetlogs
*
第 1 行出现错误:
ORA-01152: 文件 1 没有从过旧的备份中还原
ORA-01110: 数据文件 1:
'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_SYSTEM_DN9MN5OT_.DBF'
[/shell]
这里比较明显,由于controlfile的scn 大于db的scn,从而出现了ORA-01152的错误,重试重建控制文件
<br>
<strong>重建控制文件</strong>

SQL> alter database backup controlfile to trace as 'd:\app\ctl.txt';
数据库已更改。
SQL> startup nomount;
ORACLE 例程已经启动。
Total System Global Area      10301210624 bytes
Fixed Size                        3842760 bytes
Variable Size                  1778388280 bytes
Database Buffers               8489271296 bytes
Redo Buffers                     29708288 bytes
SQL> CREATE CONTROLFILE REUSE DATABASE "XIFENFEI" NORESETLOGS  NOARCHIVELOG
  2      MAXLOGFILES 16
  3      MAXLOGMEMBERS 3
  4      MAXDATAFILES 100
  5      MAXINSTANCES 8
  6      MAXLOGHISTORY 292
  7  LOGFILE
  8    GROUP 1 (
  9      'D:\APP\ORACLE\ORADATA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV652_.LOG',
 10      'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV661_.LO
G'
 11    ) SIZE 50M BLOCKSIZE 512,
 12    GROUP 2 (
 13      'D:\APP\ORACLE\ORADATA\XIFENFEI\ONLINELOG\O1_MF_2_DN9MV68H_.LOG',
 14      'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_2_DN9MV69G_.LO
G'
 15    ) SIZE 50M BLOCKSIZE 512,
 16    GROUP 3 (
 17      'D:\APP\ORACLE\ORADATA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6BX_.LOG',
 18      'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6DW_.LO
G'
 19    ) SIZE 50M BLOCKSIZE 512
 20  DATAFILE
 21    'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_SYSTEM_DN9MN5OT_.DBF',
 22    'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_SYSAUX_DN9MK6B3_.DBF',
 23    'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_UNDOTBS1_DN9MQJFK_.DBF',
 24    'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_USERS_DN9MQH75_.DBF',
 25    'D:\APP\ORACLE\PRODUCT\12.1.0\DBHOME_1\DATABASE\XIFENFEIFILE06',
 26    'D:\APP\ORACLE\PRODUCT\12.1.0\DBHOME_1\DATABASE\XIFENFEIFILE05',
 27    'D:\APP\ORACLE\PRODUCT\12.1.0\DBHOME_1\DATABASE\XIFENFEIFILE04',
 28    'D:\APP\ORACLE\PRODUCT\12.1.0\DBHOME_1\DATABASE\XIFENFEIFILE03',
 29    'D:\APP\ORACLE\PRODUCT\12.1.0\DBHOME_1\DATABASE\XIFENFEIFILE02',
 30    'D:\APP\ORACLE\PRODUCT\12.1.0\DBHOME_1\DATABASE\XIFENFEIFILE01'
 31  CHARACTER SET ZHS16GBK
 32  ;
控制文件已创建。

尝试恢复

SQL> recover database;
完成介质恢复。
SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [4194], [41], [36], [], [], [], [],
[], [], [], [], []
进程 ID: 2864
会话 ID: 62 序列号: 54236

本来到这一步,错误比较明显,undo异常,这类直接对undo进行处理即可,可是运气不太好

异常crash之后redo损坏

SQL> startup pfile='d:/app/pfile.txt'
ORACLE 例程已经启动。
Total System Global Area 1.0301E+10 bytes
Fixed Size                  3842760 bytes
Variable Size            1778388280 bytes
Database Buffers         8489271296 bytes
Redo Buffers               29708288 bytes
数据库装载完毕。
ORA-00354: 损坏重做日志块标头
ORA-00353: 日志损坏接近块 60 更改 23924938639111 时间 08/17/2017 21:36:16
ORA-00312: 联机日志 1 线程 1:
'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV661_.LOG'
ORA-00312: 联机日志 1 线程 1:
'D:\APP\ORACLE\ORADATA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV652_.LOG'
SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 60 change 23924938639111 time 08/17/201721:36:16
ORA-00312: online log 1 thread 1:
'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV661_.LOG'
ORA-00312: online log 1 thread 1:
'D:\APP\ORACLE\ORADATA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV652_.LOG'
SQL> recover database until cancel;
ORA-00279: 更改 10050832 (在 08/17/2017 21:36:13 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ARCHIVELOG\2017_08_17\O1_MF_1_718_%U_.ARC
ORA-00280: 更改 10050832 (用于线程 1) 在序列 #718 中
指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV661_.LOG
ORA-00283: recovery session canceled due to errors
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 60 change 23924938639111 time 08/17/2017 21:36:16
ORA-00334: archived log:
'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV661_.LOG'

到这一步,只能通过屏蔽oracle 事务前滚,强制拉库恢复。

SQL> startup pfile='d:/app/pfile.txt' mount;
ORACLE 例程已经启动。
Total System Global Area 1.0301E+10 bytes
Fixed Size                  3842760 bytes
Variable Size            1778388280 bytes
Database Buffers         8489271296 bytes
Redo Buffers               29708288 bytes
数据库装载完毕。
SQL> recover database until cancel;
ORA-00279: 更改 10050832 (在 08/17/2017 21:36:13 生成) 对于线程 1 是必需的
ORA-00289: 建议: D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ARCHIVELOG\2017_08_17\O
1_MF_1_718_%U_.ARC
ORA-00280: 更改 10050832 (用于线程 1) 在序列 #718 中
指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: 'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_SYSTEM_DN9M
N5OT_.DBF'
ORA-01112: 未启动介质恢复
SQL> alter database open resetlogs;
数据库已更改。

ORA-600 3051故障恢复

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:ORA-600 3051故障恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

数据库恢复报错ORA-600 3051

SQL> startup mount pfile=D:\OneDrive\ORACLE\recover\script\pfile.txt
ORACLE 例程已经启动。
Total System Global Area 2137886720 bytes
Fixed Size                  2177456 bytes
Variable Size            1224738384 bytes
Database Buffers          905969664 bytes
Redo Buffers                5001216 bytes
数据库装载完毕。
SQL> select group#,sequence#,status from v$log;
    GROUP#  SEQUENCE# STATUS
---------- ---------- ----------------
         1     743269 ACTIVE
         3     743268 ACTIVE
         2     743270 CURRENT
SQL> select member from v$logfile;
MEMBER
--------------------------------------------------------------------------------
H:\ORADATA\xifenfei\REDO03.LOG
H:\ORADATA\xifenfei\REDO02.LOG
H:\ORADATA\xifenfei\REDO01.LOG
SQL> recover database;
ORA-00279: 更改 5683516934 (在 04/22/2017 15:54:28 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\XIFENFEI\PRODUCT\11.2.0.1\DBHOME_2\RDBMS\ARC0000743268_0910390029.0001
ORA-00280: 更改 5683516934 (用于线程 1) 在序列 #743268 中
指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
H:\ORADATA\xifenfei\REDO03.LOG
ORA-00279: 更改 5683516934 (在 04/22/2017 15:54:28 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\XIFENFEI\PRODUCT\11.2.0.1\DBHOME_2\RDBMS\ARC0000743268_0910390029.0001
ORA-00280: 更改 5683516934 (用于线程 1) 在序列 #743268 中
ORA-00278: 此恢复不再需要日志文件 'H:\ORADATA\xifenfei\REDO03.LOG'
指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
H:\ORADATA\xifenfei\REDO01.LOG
ORA-00310: archived log contains sequence 743269; sequence 743268 required
ORA-00334: archived log: 'H:\ORADATA\xifenfei\REDO01.LOG'
ORA-10879: error signaled in parallel recovery slave
ORA-00600: internal error code, arguments: [3051], [82], [], [], [], [], [],
[], [], [], [], []
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave

从查询的信息看库异常的时候由于redo写比较频繁,后续启动上看,需要sequence 743268的日志,但是由于redo部分异常,导致无法进行完整恢复.

屏蔽一致性强制打开库

SQL> shutdown immediate;
ORA-01109: ??????
已经卸载数据库。
ORACLE 例程已经关闭。
SQL> startup mount pfile='D:\OneDrive\ORACLE\recover\script\pfile.txt'
ORACLE 例程已经启动。
Total System Global Area 2137886720 bytes
Fixed Size                  2177456 bytes
Variable Size            1224738384 bytes
Database Buffers          905969664 bytes
Redo Buffers                5001216 bytes
数据库装载完毕。
SQL> recover database until cancel;
ORA-00279: 更改 5683516935 (在 04/22/2017 15:54:28 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\XIFENFEI\PRODUCT\11.2.0.1\DBHOME_2\RDBMS\ARC0000743268_0910390029.0001
ORA-00280: 更改 5683516935 (用于线程 1) 在序列 #743268 中
指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-10879: error signaled in parallel recovery slave
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: 'H:\xifenfei\SYSTEM01.DBF'
SQL> alter database open resetlogs;
alter database open resetlogs
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [1], [1388549650], [1],
[1388577164], [12583040], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [2662], [1], [1388549649], [1],
[1388577164], [12583040], [], [], [], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [1], [1388549647], [1],
[1388577164], [12583040], [], [], [], [], [], []
进程 ID: 12120
会话 ID: 192 序列号: 3

数据库这里信息明确,由于ORA-600 2662导致数据库启动失败,可以通过推scn的方式解决

启动报ORA-600 4193错误

SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
Sat Apr 29 00:30:11 2017
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is ZHS16GBK
Errors in file d:\app\xifenfei\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_smon_9464.trc(incident=9739):
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Incident details in:
 d:\app\xifenfei\diag\rdbms\xifenfei\xifenfei\incident\incdir_9739\xifenfei_smon_9464_i9739.trc
No Resource Manager plan active
Sat Apr 29 00:30:12 2017
Trace dumping is performing id=[cdmp_20170429003012]
Doing block recovery for file 3 block 35588
Resuming block recovery (PMON) for file 3 block 35588
Block recovery from logseq 4, block 76 to scn 6442492674
Recovery of Online Redo Log: Thread 1 Group 1 Seq 4 Reading mem 0
  Mem# 0: H:\xifenfei\REDO01.LOG
Block recovery completed at rba 4.85.16, scn 1.2147525379
Doing block recovery for file 3 block 208
Resuming block recovery (PMON) for file 3 block 208
Block recovery from logseq 4, block 76 to scn 6442492664
Recovery of Online Redo Log: Thread 1 Group 1 Seq 4 Reading mem 0
  Mem# 0: H:\xifenfei\REDO01.LOG
Block recovery completed at rba 4.80.16, scn 1.2147525369
Errors in file d:\app\xifenfei\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_smon_9464.trc:
ORA-01595: error freeing extent (6) of rollback segment (6))
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Starting background process QMNC

错误比较明显,直接通过屏蔽undo,open数据库,导出来数据,恢复完工

ORA-00704 ORA-00604 ORA-01406故障分析

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:ORA-00704 ORA-00604 ORA-01406故障分析

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

根据客户反馈系统运行的是9.2.0.8版本,但是服务器上面安装有10.2.0.1服务端,由于使用过10.2启动过数据库导致现在9.2.0.8无法启动.
数据库启动报错

Starting up ORACLE RDBMS Version: 9.2.0.8.0.
System parameters with non-default values:
…………
  db_cache_size            = 209715200
  compatible               = 9.2.0.0.0
  db_file_multiblock_read_count= 16
  fast_start_mttr_target   = 300
…………
Thread 1 advanced to log sequence 7517
Thread 1 opened at log sequence 7517
  Current log# 1 seq# 7517 mem# 0: F:\ORACLE\ORADATA\ORACLE\REDO01.LOG
Successful open of redo thread 1
Tue Apr 25 21:16:20 2017
SMON: enabling cache recovery
Tue Apr 25 21:16:20 2017
Errors in file f:\oracle\admin\oracle\udump\oracle_ora_3908.trc:
ORA-00704: 引导程序进程失败
ORA-00604: 递归 SQL 层 1 出现错误
ORA-01406: 读取的列值被截断
Tue Apr 25 21:16:20 2017
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Tue Apr 25 21:16:20 2017
Errors in file f:\oracle\admin\oracle\bdump\oracle_pmon_2124.trc:
ORA-00704: bootstrap process failure
Tue Apr 25 21:16:20 2017
Errors in file f:\oracle\admin\oracle\bdump\oracle_reco_2556.trc:
ORA-00704: bootstrap process failure
Tue Apr 25 21:16:20 2017
Errors in file f:\oracle\admin\oracle\bdump\oracle_smon_628.trc:
ORA-00704: bootstrap process failure
Tue Apr 25 21:16:21 2017
Errors in file f:\oracle\admin\oracle\bdump\oracle_ckpt_2212.trc:
ORA-00704: bootstrap process failure
Tue Apr 25 21:16:21 2017
Errors in file f:\oracle\admin\oracle\bdump\oracle_lgwr_2756.trc:
ORA-00704: bootstrap process failure
Tue Apr 25 21:16:21 2017
Errors in file f:\oracle\admin\oracle\bdump\oracle_dbw0_1756.trc:
ORA-00704: bootstrap process failure
Instance terminated by USER, pid = 3908
ORA-1092 signalled during: ALTER DATABASE OPEN...

错误比较明显bootstrap$的相关sql在递归的时候报错(ORA-01406)

我们分析alert日志

---9.2.0.1版本运行了很长时间
Mon Apr 13 20:44:29 2009
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 2
Using log_archive_dest parameter default value
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.1.0.
--然后升级到9.2.0.8
Thu Jun 18 17:32:09 2015
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 2
Using log_archive_dest parameter default value
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.8.0.
System parameters with non-default values:
…………
Thu Jun 18 17:32:18 2015
SMON: enabling cache recovery
Thu Jun 18 17:32:19 2015
Successfully onlined Undo Tablespace 1.
Thu Jun 18 17:32:19 2015
SMON: enabling tx recovery
Thu Jun 18 17:32:19 2015
Database Characterset is ZHS16GBK
Updating 9.2.0.1.0 NLS parameters in sys.props$
-- adding 9.2.0.8.0 NLS parameters.
replication_dependency_tracking turned off (no async multimaster replication found)
Completed: alter database open
…………
Thu Jun 18 17:38:32 2015
Database Characterset is ZHS16GBK
Thu Jun 18 17:38:33 2015
ALTER SYSTEM SET _system_trig_enabled=FALSE SCOPE=MEMORY;
Thu Jun 18 17:38:33 2015
ALTER SYSTEM SET job_queue_processes=0 SCOPE=MEMORY;
Thu Jun 18 17:38:33 2015
ALTER SYSTEM SET aq_tm_processes=0 SCOPE=MEMORY;
replication_dependency_tracking turned off (no async multimaster replication found)
Completed: ALTER DATABASE OPEN MIGRATE
--再升级到10.2.0.1
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
  processes                = 150
  timed_statistics         = TRUE
  sga_max_size             = 1610612736
  shared_pool_size         = 209715200
  large_pool_size          = 8388608
  java_pool_size           = 159383552
  streams_pool_size        = 50331648
  control_files            = F:\ORACLE\ORADATA\ORACLE\CONTROL01.CTL
  db_block_size            = 8192
  db_cache_size            = 209715200
  compatible               = 9.2.0.0.0
…………
Thu Jun 18 19:43:30 2015
Database Characterset is ZHS16GBK
Updating 9.2.0.8.0 NLS parameters in sys.props$
-- adding 10.2.0.1.0 NLS parameters.
…………
Thu Jun 18 19:43:44 2015
ALTER SYSTEM enable restricted session;
MMNL started with pid=12, OS id=5212
Thu Jun 18 19:43:44 2015
ALTER SYSTEM SET _system_trig_enabled=FALSE SCOPE=MEMORY;
Thu Jun 18 19:43:44 2015
ALTER SYSTEM SET aq_tm_processes=0 SCOPE=MEMORY;
Thu Jun 18 19:43:44 2015
ALTER SYSTEM SET resource_manager_plan='' SCOPE=MEMORY;
Thu Jun 18 19:43:44 2015
replication_dependency_tracking turned off (no async multimaster replication found)
Completed: ALTER DATABASE OPEN MIGRATE

这里很明显数据库从2009年4月开始9.2.0.1版本开始运行,然后到2015年6月18日升级到9.2.0.8版本,紧接着升级到10.2.0.1(升级8.2.0.8是为升级10.2.0.1的中间过度操作).然后这个库一直使用10.2.0.1版本运行,这次重启不知道什么原因客户以为是9.2.0.8的数据库版本,然后不管怎么样也无法启动成功(这里不知道什么原因win 服务中使用的9.2.0.8的软件,估计被人误操作了).解决该问题,就是把服务切换成10.2.0.1版本数据库正常启动.
再次提醒大家,在oracle恢复的过程中,需要仔细分析日志,日志不会骗人,不要轻信客户的现场描述

ORA 600 3005恢复

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:ORA 600 3005恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

数据库打开报ora-600 3005错误

D:\>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on 星期二 3月 7 23:04:25 2017
Copyright (c) 1982, 2010, Oracle.  All rights reserved.
连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> recover datafile 1;
完成介质恢复。
SQL> recover datafile 2;
完成介质恢复。
SQL> recover datafile 3;
完成介质恢复。
SQL> recover datafile 4;
完成介质恢复。
SQL> recover datafile 5;
完成介质恢复。
SQL> recover datafile 6;
完成介质恢复。
SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-00600: 内部错误代码, 参数: [3005], [1], [8242], [29937], [0], [0], [], [],
[], [], [], []

查询数据库信息

SQL> SELECT status,
  2  checkpoint_change#,
  3  checkpoint_time,FUZZY,
  4  count(*) ROW_NUM
  5  FROM v$datafile_header
  6  GROUP BY status, checkpoint_change#, checkpoint_time,fuzzy
  7  ORDER BY status, checkpoint_change#, checkpoint_time;
STATUS         CHECKPOINT_CHANGE# CHECKPOINT_TIM FUZZY     ROW_NUM
-------------- ------------------ -------------- ------ ----------
ONLINE                  227036249 06-3月 -17     NO              5
ONLINE                  227036252 06-3月 -17     NO              1
SQL> set numw 16
SQL> SELECT status,
  2  checkpoint_change#,
  3  checkpoint_time,last_change#,
  4  count(*) ROW_NUM
  5  FROM v$datafile
  6  GROUP BY status, checkpoint_change#, checkpoint_time,last_change#
  7  ORDER BY status, checkpoint_change#, checkpoint_time;
STATUS         CHECKPOINT_CHANGE# CHECKPOINT_TIM     LAST_CHANGE#
-------------- ------------------ -------------- ----------------
         ROW_NUM
----------------
ONLINE                  227036249 06-3月 -17
               4
ONLINE                  227036252 06-3月 -17
               1
SYSTEM                  227036249 06-3月 -17
               1

mos上关于ora-600 3005描述

VERSIONS:
versions 10.2 and later
DESCRIPTION:
Raised during pass one of the two pass recovery processing, which
reads and merges open redo threads into a hash table of blocks
that need recovery.
During examination of the the change vectors of online redologs, this
error is raised if no online redo log could be opened to cover the start RBA.
ARGUMENTS:
Arg [a] Thread
Arg [b] Redo Log File Sequence
Arg {c} Redo Log File Block Number
Arg [d] SCN Wrap
Arg [e] SCN Base

根据官方描述,出现该错误的原因是由于在数据库启动的过程中,通过控制文件读取的redo信息不匹配,从而出现该问题,通过重建控制文件可以绕过去该问题

SQL> shutdown immediate;
ORA-01109: 数据库未打开
已经卸载数据库。
ORACLE 例程已经关闭。
SQL> startup nomount pfile='d:/pfile.txt'
ORACLE 例程已经启动。
Total System Global Area      10288615424 bytes
Fixed Size                        2184672 bytes
Variable Size                  7482640928 bytes
Database Buffers               2785017856 bytes
Redo Buffers                     18771968 bytes
SQL> CREATE CONTROLFILE REUSE DATABASE "ORACLEDO" NORESETLOGS  NOARCHIVELOG
  2      MAXLOGFILES 16
  3      MAXLOGMEMBERS 3
  4      MAXDATAFILES 100
  5      MAXINSTANCES 8
  6      MAXLOGHISTORY 2336
  7  LOGFILE
  8    GROUP 1 'D:\APP\ADMINISTRATOR\ORADATA\ORACLEDOC\REDO01.LOG'  SIZE 50M BLOCKSIZE 512,
  9    GROUP 2 'D:\APP\ADMINISTRATOR\ORADATA\ORACLEDOC\REDO02.LOG'  SIZE 50M BLOCKSIZE 512,
 10    GROUP 3 'D:\APP\ADMINISTRATOR\ORADATA\ORACLEDOC\REDO03.LOG'  SIZE 50M BLOCKSIZE 512
 11  DATAFILE
 12    'D:\APP\ADMINISTRATOR\ORADATA\ORACLEDOC\SYSTEM01.DBF',
 13    'D:\APP\ADMINISTRATOR\ORADATA\ORACLEDOC\SYSAUX01.DBF',
 14    'D:\APP\ADMINISTRATOR\ORADATA\ORACLEDOC\UNDOTBS01.DBF',
 15    'D:\APP\ADMINISTRATOR\ORADATA\ORACLEDOC\USERS01.DBF',
 16    'D:\APP\ADMINISTRATOR\ORADATA\ORACLEDOC\XIFENFEI01.DBF',
 17    'D:\APP\ADMINISTRATOR\ORADATA\ORACLEDOC\XIFENFEI0102.DBF'
 18  CHARACTER SET AL32UTF8
 19  ;
控制文件已创建。
SQL> recover database;
完成介质恢复。
SQL> alter database open;
alter database open
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
进程 ID: 4036
会话 ID: 96 序列号: 1

这个错误就比较熟悉了,按照undo异常方案处理即可
补充说明
ora-600 3005的错误可能需要internal 帐号才能够查询到准确描述和处理方法,其实在这个库的运行最后crash之前,就已经报了控制文件异常,然后库crash掉了.

Mon Mar 06 10:16:37 2017
Thread 1 advanced to log sequence 8242 (LGWR switch)
  Current log# 1 seq# 8242 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORACLEDOC\REDO01.LOG
Mon Mar 06 11:06:31 2017
********************* ATTENTION: ********************
 The controlfile header block returned by the OS
 has a sequence number that is too old.
 The controlfile might be corrupted.
 PLEASE DO NOT ATTEMPT TO START UP THE INSTANCE
 without following the steps below.
 RE-STARTING THE INSTANCE CAN CAUSE SERIOUS DAMAGE
 TO THE DATABASE, if the controlfile is truly corrupted.
 In order to re-start the instance safely,
 please do the following:
 (1) Save all copies of the controlfile for later
     analysis and contact your OS vendor and Oracle support.
 (2) Mount the instance and issue:
     ALTER DATABASE BACKUP CONTROLFILE TO TRACE;
 (3) Unmount the instance.
 (4) Use the script in the trace file to
     RE-CREATE THE CONTROLFILE and open the database.
*****************************************************
MMON (ospid: 3320): terminating the instance
Mon Mar 06 11:06:32 2017
opiodr aborting process unknown ospid (1528) as a result of ORA-1092
Mon Mar 06 11:06:32 2017
ORA-1092 : opitsk aborting process
Mon Mar 06 11:06:32 2017
opiodr aborting process unknown ospid (2852) as a result of ORA-1092
Mon Mar 06 11:06:32 2017
ORA-1092 : opitsk aborting process
Mon Mar 06 11:06:33 2017
opiodr aborting process unknown ospid (3836) as a result of ORA-1092
Mon Mar 06 11:06:33 2017
ORA-1092 : opitsk aborting process
Instance terminated by MMON, pid = 3320

KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type]

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type]

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

在oracle asm的使用过程中由于操作系统层面的错误操作导致asm disk 被破坏,这里列举了几种破坏之后的kfed报错现象(KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type])
asm mount 磁盘组报错(ORA-15040 ORA-15042)

SQL> alter diskgroup DATA mount;
alter diskgroup DATA mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "2" is missing from group number "2"

asm alert日志报错(ORA-15335 ORA-15066 ORA-15196等)

ORA-15335: ASM metadata corruption detected in disk group 'DATA'
ORA-15130: diskgroup "DATA" is being dismounted
ORA-15066: offlining disk "DATA_0002" in group "DATA" may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483651] [48] [0 != 1]

kfed查看磁盘头报错
文件文件头(不光是disk header的4k,可能是连续的几个au,甚至更多)可能彻底损坏,一般kfed 读取都会看到KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type]之类错误

[oracle@fcomtaep2 disks]$ kfed read ASMRECO03
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
7FC18D899400 00000000 00000000 00000000 00000000 [................]
  Repeat 27 times
7FC18D8995C0 FEEE0001 0001FFFF FFFF0000 00000FFF [................]
7FC18D8995D0 00000000 00000000 00000000 00000000 [................]
  Repeat 1 times
7FC18D8995F0 00000000 00000000 00000000 AA550000 [..............U.]
7FC18D899600 20494645 54524150 00010000 0000005C [EFI PART....\...] <==== **** Here ******
7FC18D899610 BD82BBB3 00000000 00000001 00000000 [................]
7FC18D899620 0FFFFFFF 00000000 00000022 00000000 [........".......]
7FC18D899630 0FFFFFDE 00000000 FD8857E5 42D7B49B [.........W.....B]
7FC18D899640 0901FA87 6B3DB5AA 00000002 00000000 [......=k........]
7FC18D899650 00000080 00000080 FE48EB77 00000000 [........w.H.....]
7FC18D899660 00000000 00000000 00000000 00000000 [................]
  Repeat 25 times
7FC18D899800 EBD0A0A2 4433B9E5 B668C087 C79926B7 [......3D..h..&..]
7FC18D899810 5381F6DF 4626F988 0E4F468D D78D3B28 [...S..&F.FO.(;..]
7FC18D899820 000007A1 00000000 0FFFF85F 00000000 [........_.......]
7FC18D899830 00000000 00000000 00720070 006D0069 [........p.r.i.m.]
7FC18D899840 00720061 00000079 00000000 00000000 [a.r.y...........]
7FC18D899850 00000000 00000000 00000000 00000000 [................]
 Repeat 186 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

“EFI PART”是分区的元数据,一般是被分区导致asm disk损坏.

[ebernal@dbaasm new2]$ kfed read emcpowerl | head -25
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
2ABD671E9400 00000000 00000000 00000000 00000000 [................]
  Repeat 31 times
2ABD671E9600 4542414C 454E4F4C 00000001 00000000 [LABELONE........]
2ABD671E9610 E4E1DDB1 00000020 324D564C 31303020 [.... ...LVM2 001] <==== **** Here ******
2ABD671E9620 50365A77 71327874 34303156 4B4E6136 [wZ6Ptx2qV1046aNK]
2ABD671E9630 35395159 5147634C 487A5A38 63575A37 [YQ95LcGQ8ZzH7ZWc]
2ABD671E9640 00000000 00000019 00030000 00000000 [................]
2ABD671E9650 00000000 00000000 00000000 00000000 [................]
2ABD671E9660 00000000 00000000 00001000 00000000 [................]
2ABD671E9670 0002F000 00000000 00000000 00000000 [................]
2ABD671E9680 00000000 00000000 00000000 00000000 [................]
  Repeat 215 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

“LVM2 001” 是逻辑卷的名字,该asm disk很可能被做为lvm管理而被破坏

[ebernal@dbaasm tars]$ kfed read rhdisk16
kfbh.endian:                         65 ; 0x000: 0x41
kfbh.hard:                           73 ; 0x001: 0x49
kfbh.type:                           88 ; 0x002: *** Unknown Enum ***
kfbh.datfmt:                         32 ; 0x003: 0x20
kfbh.block.blk:              1111709260 ; 0x004: blk=1111709260
kfbh.block.obj:              1634861056 ; 0x008: file=131072
kfbh.check:                         119 ; 0x00c: 0x00000077
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
2B6FE2AC1400 20584941 4243564C 61720000 00000077  [AIX LVCB..raw...] <==== **** Here ******
2B6FE2AC1410 00000000 00000000 00000000 00000000  [................]
2B6FE2AC1420 00000000 00000000 30300000 38306430  [..........000d08]
2B6FE2AC1430 30306131 34643030 30303030 31303030  [1a0000d400000001]
2B6FE2AC1440 61006533 766C6D73 7461645F 00003161  [3e.asmlv_data1..]
2B6FE2AC1450 00000000 00000000 00000000 00000000  [................]
        Repeat 2 times
2B6FE2AC1480 54000000 4D206575 20207961 31312037  [...Tue May  7 11]
2B6FE2AC1490 3A33343A 32203633 0A333130 00000000  [:43:36 2013.....]
2B6FE2AC14A0 65755400 79614D20 20372020 343A3131  [.Tue May  7 11:4]
2B6FE2AC14B0 34323A38 31303220 00000A33 44000000  [8:24 2013......D]
2B6FE2AC14C0 41313830 30303444 6D6D7900 02007900  [081AD400.ymm.y..]
2B6FE2AC14D0 0100E40C 656E6F4E 00000000 00000000  [....None........]
2B6FE2AC14E0 00000000 00000000 00000000 00000000  [................]
        Repeat 14 times
2B6FE2AC15D0 00000000 00000000 65310000 61653934  [..........1e49ea]
2B6FE2AC15E0 342E3862 00000000 00000000 00000000  [b8.4............]
2B6FE2AC15F0 00000000 00000000 00000000 00000000  [................]
  Repeat 224 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][88]

这里的“AIX LVCB..raw” 是AIX OS volume 的元数据库,也就是说,asm disk 被作为了aix os层面破坏

[oracle@dbep2 disks]$ kfed read asm-disk3
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
06000000 00000000 00000000 00000000 00000000 [................]
  Repeat 25 times
0602100 51e2b7f6 00ed4e00 00000000 00000001  [...Q.N..........]
0602120 00000000 0000000b 00000100 0000003c [............<...]
0602140 00000242 0000007b 5d8468e7 6147782a [B...{....h.]*xGa]
0602160 d17851a2 327552e2 00000000 00000000 [.Qx..Ru2........]
0602200 00000000 00000000 3130752f 91a4f000 [......../u01....] <==== **** Here ******
0602220 ff8808e4 d5104cff 000000ac 00000100 [.....L..........]
0602240 00000000 00000000 00000000 09d18000 [................]
  Repeat 254 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][88]

这里的/u01很可能表明该asm disk被文件系统覆盖

对于asm disk的各种破坏情况,如果是normal/high冗余,那么asm dg没有问题,可以考虑通过删除异常盘,然后重新加入;如果是外部冗余遭遇到asm disk 被破坏,一般asm disk 会dismount,而且无法正常mount,如果有备份的磁盘头,可以尝试还原磁盘头,mount 磁盘组,然后只读方式迁移数据;如果没有备份磁盘头或者还原之后也无法mount,可能需要通过一些额外的方式处理比如通过工具在asm dismount状态下恢复数据文件,甚至通过对asm block/oracle block碎片重组的方式恢复数据.参考相关文章:
oracle asm系列文章汇总
pvid=yes导致asm无法mount
asm disk header 彻底损坏恢复
分区无法识别导致asm diskgroup无法mount
oracle asm disk格式化恢复—格式化为ext4文件系统
oracle asm disk格式化恢复—格式化为ntfs文件系统
asm disk误设置pvid导致asm diskgroup无法mount恢复
分享oracleasm createdisk重新创建asm disk后数据0丢失恢复案例
ORA-15042: ASM disk “N” is missing from group number “M” 故障恢复
如果您遇到此类情况,无法解决请联系我们,提供专业ORACLE数据库恢复技术支持
Phone:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com

解决CON$ ORA-600 kdsgrp1错误

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:解决CON$ ORA-600 kdsgrp1错误

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

数据库报ORA 600 kdsgrp1错误
数据库报ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []错

Thread 1 advanced to log sequence 23861 (LGWR switch)
  Current log# 7 seq# 23861 mem# 0: /oradata/easdb/redo07.log
Tue Nov 15 10:00:42 2016
Errors in file /u01/oracle/diag/rdbms/easdb/easdb/trace/easdb_dw00_3165.trc  (incident=908262):
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle/diag/rdbms/easdb/easdb/incident/incdir_908262/easdb_dw00_3165_i908262.trc
Tue Nov 15 10:00:55 2016
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Tue Nov 15 10:00:56 2016
Errors in file /u01/oracle/diag/rdbms/easdb/easdb/trace/easdb_dw00_3165.trc  (incident=908263):
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPW$WORKER", line 1751
ORA-06512: at line 2
Incident details in: /u01/oracle/diag/rdbms/easdb/easdb/incident/incdir_908263/easdb_dw00_3165_i908263.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
DW00 terminating with fatal err=600, pid=40, wid=1, job SYSTEM.
Tue Nov 15 10:01:01 2016
Thread 1 advanced to log sequence 23862 (LGWR switch)
  Current log# 2 seq# 23862 mem# 0: /oradata/easdb/redo02.log
Tue Nov 15 10:01:23 2016
Errors in file /u01/oracle/diag/rdbms/easdb/easdb/trace/easdb_dm00_3163.trc  (incident=908254):
ORA-31671: Worker process DW00 had an unhandled exception.
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPW$WORKER", line 1751
ORA-06512: at line 2
Incident details in: /u01/oracle/diag/rdbms/easdb/easdb/incident/incdir_908254/easdb_dm00_3163_i908254.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Tue Nov 15 10:01:26 2016
Tue Nov 15 10:01:28 2016
Thread 1 advanced to log sequence 23863 (LGWR switch)
  Current log# 4 seq# 23863 mem# 0: /oradata/easdb/redo04.log

trace文件中信息

*** 2016-11-15 10:00:35.977
* kdsgrp1-1: *************************************************
            row 0x004459e6.26 continuation at
            0x004459e6.26 file# 1 block# 285158 slot 38 not found
KDSTABN_GET: 0 ..... ntab: 1
curSlot: 38 ..... nrows: 208
kdsgrp - dump CR block dba=0x004459e6
Block header dump:  0x004459e6
 Object id on Block? Y
 seg/obj: 0x1c  csc: 0x01.c712f743  itc: 3  flg: -  typ: 1 - DATA
     fsl: 0  fnx: 0x0 ver: 0x01
 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x000b.015.0036d715  0x00c01bba.0fbd.02  C---    0  scn 0x0001.c6b4cb1a
0x02   0x000c.004.00044d36  0x04c0dd93.3eec.33  C---    0  scn 0x0001.c6d2c65b
0x03   0x000d.008.00008eb9  0x04c0777a.10e3.02  --U-    2  fsc 0x0056.c7346f21

确定报错对象和确认异常

SQL> select object_name from dba_objects where object_id=28;
OBJECT_NAME
---------------------------------------------------------
CON$
SQL> ANALYZE TABLE sys.CON$ VALIDATE STRUCTURE CASCADE online;
ANALYZE TABLE sys.CON$ VALIDATE STRUCTURE CASCADE online
*
ERROR at line 1:
ORA-01499: table/index cross reference failure - see trace file
SQL> SET LINES 122
SQL> COL INDEX_OWNER FOR A20
SQL> COL INDEX_NAME FOR A30
SQL> COL TABLE_OWNER FOR A20
SQL> COL COLUMN_NAME FOR A25
SQL> SELECT TABLE_OWNER,INDEX_NAME,COLUMN_NAME,COLUMN_POSITION
2  FROM Dba_Ind_Columns
3  WHERE table_name = upper('&TABLE_NAME') order by TABLE_OWNER,INDEX_OWNER,INDEX_NAME,COLUMN_POSITION;
Enter value for table_name: CON$
old   3:  WHERE table_name = upper('&TABLE_NAME') order by TABLE_OWNER,INDEX_OWNER,INDEX_NAME,COLUMN_POSITION
new   3:  WHERE table_name = upper('CON$') order by TABLE_OWNER,INDEX_OWNER,INDEX_NAME,COLUMN_POSITION
TABLE_OWNER	     INDEX_NAME 		    COLUMN_NAME 	      COLUMN_POSITION
-------------------- ------------------------------ ------------------------- ---------------
SYS		     I_CON1			    OWNER#				    1
SYS		     I_CON1			    NAME				    2
SYS		     I_CON2			    CON#				    1
SQL> select owner#,name from con$
2    minus
3   select /*+ full(t) */owner#,name from con$ t;
no rows selected
SQL> select /*+ full(t) */owner#,name from con$ t
2    minus
3   select owner#,name from con$  ;
no rows selected
SQL> select /*+ full(t) */ con# from con$ t
2    minus
3   select con# from con$ ;
no rows selected
SQL> select con# from con$
2    minus
3   select /*+ full(t) */ con# from con$ t   ;
      CON#
----------
   1037224
   1037225
   1037386
   1037387
   1037388
   ……
   1037846
62 rows selected.

通过上述分析,可以确定是由于CON$和I_CON2数据不一致,而且是index的数据比表中多了62条.针对这样情况,考虑通过重建index来解决.

尝试rebuild index

SQL> alter index I_CON2 rebuild online;
alter index I_CON2 rebuild online
*
ERROR at line 1:
ORA-00701: object necessary for warmstarting database cannot be altered
SQL>
SQL>
SQL>
SQL>
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup upgrade
ORACLE instance started.
Total System Global Area 2421825536 bytes
Fixed Size                  2215744 bytes
Variable Size            1828716736 bytes
Database Buffers          570425344 bytes
Redo Buffers               20467712 bytes
Database mounted.
Database opened.
SQL> alter index I_CON2 rebuild;
alter index I_CON2 rebuild
*
ERROR at line 1:
ORA-00701: object necessary for warmstarting database cannot be altered

因为是数据库核心index,无法直接rebuild解决,只能通过bootstrap$核心index(I_OBJ1,I_USER1,I_FILE#_BLOCK#,I_IND1,I_TS#,I_CDEF1等)异常恢复—ORA-00701错误解决 方式解决