alert文件出现如下日志
Fri Sep 16 02:58:25 2011
Hex dump of (file 19, block 1444767) in trace file /opt/oracle/admin/ora9i/udump/ora9i_ora_24702.trc
Corrupt block relative dba: 0x04d60b9f (file 19, block 1444767)
Fractured block found during backing up datafile
Data in bad block:
type: 6 format: 2 rdba: 0x04d60b9f
last change scn: 0x0abf.56961827 seq: 0x1 flg: 0x04
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x7c340601
check value in block header: 0xba17
computed block checksum: 0x6413
Reread of blocknum=1444767, file=/opt/oracle/oradata/ora9i/TS_INDX_Base.005.dbf. found valid data
因为这个是我们比较重要的业务服务器，如果出现坏块，不能及时处理，后果将不堪设想，因此我马上检查
先查看是什么对象：(结果是一个分区表的index，担心减少一半，index大不了重建，无大事)

SELECT OWNER, SEGMENT_NAME, SEGMENT_TYPE, TABLESPACE_NAME, A.PARTITION_NAME
  FROM DBA_EXTENTS A
 WHERE FILE_ID = &FILE_ID
   AND &BLOCK_ID BETWEEN BLOCK_ID AND BLOCK_ID + BLOCKS - 1;

然后检查是否真的坏块
方法一：dbv

[oracle@DB1 bdump]$ dbv file=/opt/oracle/oradata/ora9i/TS_INDX_Base.005.dbf blocksize=8192
DBVERIFY: Release 10.2.0.4.0 - Production on Fri Sep 16 09:15:11 2011
Copyright (c) 1982, 2007, Oracle.  All rights reserved.
DBVERIFY - Verification starting : FILE = /opt/oracle/oradata/ora9i/TS_INDX_Base.005.dbf
DBVERIFY - Verification complete
Total Pages Examined         : 2207744
Total Pages Processed (Data) : 0
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 2201581
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 3271
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 2892
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Highest block SCN            : 1454468581 (2751.1454468581)

结论无坏块
方法二：bbed

[oracle@DB1 lib]$ bbed
Password:
BBED: Release 2.0.0.0.0 - Limited Production on Fri Sep 16 10:01:26 2011
Copyright (c) 1982, 2007, Oracle.  All rights reserved.
************* !!! For Oracle Internal Use only !!! ***************
BBED> set filename '/opt/oracle/oradata/ora9i/TS_INDX_Base.005.dbf'
        FILENAME        /opt/oracle/oradata/ora9i/TS_INDX_Base.005.dbf
BBED> set block 1444767
        BLOCK#          1444767
BBED> VERIFY
DBVERIFY - Verification starting
FILE = /opt/oracle/oradata/ora9i/TS_INDX_Base.005.dbf
BLOCK = 1444767
DBVERIFY - Verification complete
Total Blocks Examined         : 1
Total Blocks Processed (Data) : 0
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 1
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0

再次证明无坏块
方法三：rman
RMAN> backup validate datafile 19;
结果查询V$BACKUP_CORRUPTION也无坏块
方法四：使用index查询
结果也正常，没报错，alert中无错误记录
发现alert报错，但是实际没有错误，查询相关资料发现
从日志中可以看到，提示Corrupt的block对应的dba为0x04d60b9f (file 19, block 1444767)，data block的类型为6（6为trans data，所有的data和index blocks都是该类型）。Oracle发现block有可能corrupt后，进行了reread，结果为found valid data，说明数据块未损坏。Fractured block found，表示rman发现这个数据块正在被使用，这时rman会进行重新读取，如果再次失败，才认为是坏块。如果第二次尝试读取时成功，则表示数据完好，不会产生影响。此类信息在IO负载较高的情况下进行rman备份时比较容易出现。
查询系统发现，正好该时间段是系统io负载最高的时候

1、启动数据库，发现错误
startup
2、查看是否是undo文件损坏引起，并查看是否是当前undo，不是当前undo直接offline，然后open数据库，再删除掉该数据该undo即可
select a.ts#,a.name,b.name from v$datafile a,v$tablespace b where a.ts#=b.ts#;
show parameter undo_tablespace;
3、损坏undo离线，创建pfile文件
alter database datafile n offline drop;
create pfile=’/tmp/pfile’ from spfile;
4、打开数据库,如果打开失败，请继续5，如果成功按照undo异常处理步骤（10g）方法处理
alter database open;
5、如果数据库不能正常打开，而是提示，如下错误：
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
6、检查日志文件，发现如下：
SMON: about to recover undo segment 21
SMON: mark undo segment 21 as needs recovery
SMON: about to recover undo segment 22
SMON: mark undo segment 22 as needs recovery
SMON: about to recover undo segment 23
SMON: mark undo segment 23 as needs recovery
SMON: about to recover undo segment 24
SMON: mark undo segment 24 as needs recovery
SMON: about to recover undo segment 25
SMON: mark undo segment 25 as needs recovery
SMON: about to recover undo segment 26
SMON: mark undo segment 26 as needs recovery
SMON: about to recover undo segment 27
SMON: mark undo segment 27 as needs recovery
SMON: about to recover undo segment 28
SMON: mark undo segment 28 as needs recovery
SMON: about to recover undo segment 29
SMON: mark undo segment 29 as needs recovery
SMON: about to recover undo segment 30
SMON: mark undo segment 30 as needs recovery
7、编辑pfile文件，内容为
*.undo_management=’MANUAL’
*._allow_resetlogs_corruption=true
*._corrupted_rollback_segments=(_SYSSMU21$,_SYSSMU22$,_SYSSMU23$,_SYSSMU24$,
_SYSSMU25$,_SYSSMU26$,_SYSSMU27$,_SYSSMU28$,_SYSSMU29$,_SYSSMU30$)
*.undo_tablespace=’SYSTEM’
8、退出当前sqlplus，重新登录，利用pfile启动数据库
startup
9、创建新undo表空间
CREATE UNDO TABLESPACE UNDOTBSNEW DATAFILE
‘/opt/oracle/oradata/xifenfei/UNDOTBS01.dbf
SIZE 50M autoextend on next 10m maxsize 30G;
10、查询回滚段信息，为下面删除损坏undo的回滚段做到心中有底
select segment_name,status from dba_rollback_segs;
11、删除损坏undo
drop tablespace UNDOTBSOLD including contents and datafiles;
12、查看回滚段状态
select segment_name,status from dba_rollback_segs;
13、如果有损坏表空间回滚段还存在，手工删除
drop rollback segment “_SYSSMUx$”;
14、修改pfile内容
*.undo_management=’AUTO’
#*._allow_resetlogs_corruption=true
#*._corrupted_rollback_segments=(_SYSSMU21$,_SYSSMU22$,_SYSSMU23$,_SYSSMU24$,
_SYSSMU25$,_SYSSMU26$,_SYSSMU27$,_SYSSMU28$,_SYSSMU29$,_SYSSMU30$)
*.undo_tablespace=’UNDOTBSNEW’
15、重启数据库
shutdown immediate
startup
说明：可以先删除需要恢复的回滚段，再删除损坏的undo表空间

1、启动数据库，发现错误
startup
2、查看是否是undo文件损坏引起
select a.ts#,a.name,b.name from v$datafile a,v$tablespace b where a.ts#=b.ts#;
3、损坏undo离线
alter database datafile n offline drop;
4、打开数据库
alter database open;
5、查看当前undo表空间，并确认是否是当前undo损坏，如果不是当前undo，直接执行8，否则继续
show parameter undo_tablespace;
6、创建新undo表空间
CREATE UNDO TABLESPACE UNDOTBSNEW DATAFILE
‘/opt/oracle/oradata/xifenfei/UNDOTBS01.dbf
SIZE 50M autoextend on next 10m maxsize 30G;
7、修改默认undo为新创建表空间
alter system set undo_tablespace=’UNDOTBSNEW’;
8、查询回滚段信息，为下面删除损坏undo的回滚段做到心中有底
select segment_name,status from dba_rollback_segs;
9、删除损坏undo，如果报ORA-01548错误，继续下面操作，否则跳到14
drop tablespace UNDOTBSOLD including contents and datafiles;
10、创建pfile文件
create pfile=’/tmp/pfile’ from spfile;
11、修改pfile文件，添加隐含参数
*._allow_resetlogs_corruption=true
*._corrupted_rollback_segments=(_SYSSMU1$,_SYSSMU2$,_SYSSMU3$,_SYSSMU4$,_SYSSMU5$,
_SYSSMU6$,_SYSSMU7$,_SYSSMU8$,_SYSSMU9$,_SYSSMU10$)
12、使用pfile启动数据库
startup pfile=’/tmp/pfile’
13、删除损坏undo
drop tablespace UNDOTBSOLD including contents and datafiles;
14、查看回滚段状态
select segment_name,status from dba_rollback_segs;
15、如果有损坏表空间回滚段还存在，手工删除
drop rollback segment “_SYSSMUx$”;
16、重启数据库
shutdown immediate
startup

环境模拟
删除状态为active的联机日志，然后强行关闭数据库
处理过程
SQL> startup
ORACLE instance started.
Total System Global Area 167772160 bytes
Fixed Size 1260720 bytes
Variable Size 142607184 bytes
Database Buffers 16777216 bytes
Redo Buffers 7127040 bytes
Database mounted.
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: ‘/u01/oradata/xienfei/redo01.log’
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3
Alert.log 日志中错误
Wed Aug 24 00:26:33 2011
Errors in file /u01/admin/xienfei/udump/xff_ora_9186.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: ‘/u01/oradata/xienfei/redo01.log’
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3
xff_ora_9186.trc文件中错误
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: ‘/u01/oradata/xienfei/redo01.log’
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3
查询当前日志组状态
SQL> select a.group#,a.status,b.member from v$log a,v$logfile b where a.group#=b.group#;
GROUP# STATUS MEMBER
———- —————- ———————————————
1 CURRENT /u01/oradata/xienfei/redo01.log
3 INACTIVE /u01/oradata/xienfei/redo03.log
2 ACTIVE /u01/oradata/xienfei/redo02.log
尝试删除redo日志
SQL> alter database drop logfile group 1;
alter database drop logfile group 1
*
ERROR at line 1:
ORA-01623: log 1 is current log for instance xff (thread 1) – cannot drop
ORA-00312: online log 1 thread 1: ‘/u01/oradata/xienfei/redo01.log’
发现是当前日志不能被删除，尝试切换日志
SQL> alter system switch logfile;
alter system switch logfile
*
ERROR at line 1:
ORA-01109: database not open
在数据库未打开状态，不能切换日志，只能尝试清空日志
SQL> alter database clear unarchived logfile group 1;
alter database clear unarchived logfile group 1
*
ERROR at line 1:
ORA-01624: log 1 needed for crash recovery of instance xff (thread 1)
ORA-00312: online log 1 thread 1: ‘/u01/oradata/xienfei/redo01.log’
因为数据库恢复需要使用，不能被清空，尝试不完成恢复
SQL> recover database until cancel;
ORA-00279: change 1272687 generated at 08/24/2011 00:20:05 needed for thread 1
ORA-00289: suggestion : /u01/archive/1_27_756841839.arc
ORA-00280: change 1272687 for thread 1 is in sequence #27
Specify log: {=suggested | filename | AUTO | CANCEL}
auto
ORA-00279: change 1272903 generated at 08/24/2011 00:25:17 needed for thread 1
ORA-00289: suggestion : /u01/archive/1_28_756841839.arc
ORA-00280: change 1272903 for thread 1 is in sequence #28
ORA-00278: log file ‘/u01/archive/1_27_756841839.arc’ no longer needed for this
recovery
ORA-00308: cannot open archived log ‘/u01/archive/1_28_756841839.arc’
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: ‘/u01/oradata/xienfei/system01.dbf’
已经提示数据不一致，尝试着打开数据库
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: ‘/u01/oradata/xienfei/system01.dbf’
创建pfile文件，添加隐含参数，使之不进行检查点一致性校验
SQL> create pfile=’/tmp/pfile’ from spfile;
File created.
修改pfile ，添加以下参数
*._allow_resetlogs_corruption=TRUE
*._allow_error_simulation=TRUE
SQL> shutdown abort
ORACLE instance shut down.
使用pfile打开数据库
SQL> startup pfile=’/tmp/pfile’
ORACLE instance started.
Total System Global Area 167772160 bytes
Fixed Size 1260720 bytes
Variable Size 150995792 bytes
Database Buffers 8388608 bytes
Redo Buffers 7127040 bytes
Database mounted.
ORA-38760: This database instance failed to turn on flashback database
发现flashback导致数据库不能被正常打开，尝试关闭它
SQL> alter database flashback off;
Database altered.
尝试直接open数据库
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open
根据提示，使用resetlogs打开数据库
SQL> alter database open resetlogs;
Database altered.
查询日志状态
SQL> select a.group#,a.status,b.member from v$log a,v$logfile b where a.group#=b.group#;
GROUP# STATUS MEMBER
———- —————- ———————————————
3 UNUSED /u01/oradata/xienfei/redo03.log
2 UNUSED /u01/oradata/xienfei/redo02.log
1 CURRENT /u01/oradata/xienfei/redo01.log
因为group1错误，而当前日志组在group 1上，所以切换日志组
SQL> alter system switch logfile;
System altered.
SQL> select a.group#,a.status,b.member from v$log a,v$logfile b where a.group#=b.group#;
GROUP# STATUS MEMBER
———- —————- ———————————————
3 UNUSED /u01/oradata/xienfei/redo03.log
2 CURRENT /u01/oradata/xienfei/redo02.log
1 ACTIVE /u01/oradata/xienfei/redo01.log
SQL> alter system checkpoint;
System altered.
SQL> select a.group#,a.status,b.member from v$log a,v$logfile b where a.group#=b.group#;
GROUP# STATUS MEMBER
———- —————- ———————————————
3 UNUSED /u01/oradata/xienfei/redo03.log
2 CURRENT /u01/oradata/xienfei/redo02.log
1 INACTIVE /u01/oradata/xienfei/redo01.log
删除有问题的group 1日志组
SQL> alter database drop logfile group 1;
Database altered.
SQL> alter system switch logfile;
System altered.
添加日志组并检查是否正确
SQL> select a.group#,a.status,b.member from v$log a,v$logfile b where a.group#=b.group#;
GROUP# STATUS MEMBER
———- —————- ———————————————
3 CURRENT /u01/oradata/xienfei/redo03.log
2 ACTIVE /u01/oradata/xienfei/redo02.log
SQL> alter database add logfile group 1 ‘/u01/oradata/xienfei/redo01.log’ size 50m reuse;
Database altered.
SQL> alter system switch logfile;
System altered.
SQL> select a.group#,a.status,b.member from v$log a,v$logfile b where a.group#=b.group#;
GROUP# STATUS MEMBER
———- —————- ———————————————
3 ACTIVE /u01/oradata/xienfei/redo03.log
2 ACTIVE /u01/oradata/xienfei/redo02.log
1 CURRENT /u01/oradata/xienfei/redo01.log
注意：根据oracle官方建议，使用oracle隐含参数运行数据库可能存在很多不稳定因素，建议立即导出数据库数据，然后新建库，重新导入数据

#  使用alter database backup controlfile to trace;产生控制文件的trace file
-- Below are two sets of SQL statements, each of which creates a new
-- control file and uses it to open the database. The first set opens
-- the database with the NORESETLOGS option and should be used only if
-- the current versions of all online logs are available. The second
-- set opens the database with the RESETLOGS option and should be used
-- if online logs are unavailable.
-- The appropriate set of statements can be copied from the trace into
-- a script file, edited as necessary, and executed when there is a
-- need to re-create the control file.
#  需要重建控制文件的话， 如果online redo 是可用的, 使用NORESETLOGS 创建新控制文件打开数据库
#  如果online redo是不可用的, 那么需要使用RESETLOGS 创建新控制文件打开数据库,
#  联机日志被重置，数据有可能丢失
--
--     Set #1. NORESETLOGS case
--
-- The following commands will create a new control file and use it
-- to open the database.
-- Data used by Recovery Manager will be lost.
-- Additional logs may be required for media recovery of offline
-- Use this only if the current versions of all online logs are
-- available.
-- After mounting the created controlfile, the following SQL
-- statement will place the database in the appropriate
-- protection mode:
--  ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE PERFORMANCE
#   下面的命令用来创建一个新的控制文件来开启数据库。 用于RMAN的数据将会丢失。
#   对于offline datafiles 可能需要额外的归档日志用来做介质恢复，
#   当当前版本的所有online redo log都没有问题的情况下使用noresetlogs。
STARTUP NOMOUNT
CREATE CONTROLFILE REUSE DATABASE "XIENFEI" NORESETLOGS FORCE LOGGING ARCHIVELOG
    MAXLOGFILES 16
    MAXLOGMEMBERS 3
    MAXDATAFILES 100
    MAXINSTANCES 8
    MAXLOGHISTORY 2921
LOGFILE
  GROUP 1 '/u01/oradata/xienfei/redo01.log'  SIZE 5M,
  GROUP 2 '/u01/oradata/xienfei/redo02.log'  SIZE 5M,
  GROUP 3 '/u01/oradata/xienfei/redo03.log'  SIZE 5M
-- STANDBY LOGFILE
DATAFILE
  '/u01/oradata/xienfei/system01.dbf',
  '/u01/oradata/xienfei/undotbs01.dbf',
  '/u01/oradata/xienfei/sysaux01.dbf',
  '/u01/oradata/xienfei/users01.dbf',
  '/u01/oradata/xienfei/example01.dbf',
  '/u01/oradata/xienfei/xff01.dbf',
  '/u01/oradata/xienfei/cf01.dbf',
  '/u01/oradata/xienfei/cfxff01.dbf'
CHARACTER SET ZHS16GBK
;
-- Commands to re-create incarnation table
-- Below log names MUST be changed to existing filenames on
-- disk. Any one log file from each branch can be used to
-- re-create incarnation records.
-- ALTER DATABASE REGISTER LOGFILE '/u01/archive/1_1_756837539.arc';
#  有些时候可能需要先注册归档日志，再才能够恢复
-- Recovery is required if any of the datafiles are restored backups,
-- or if the last shutdown was not normal or immediate.
#  恢复数据库
RECOVER DATABASE
-- All logs need archiving and a log switch is needed.
#  归档所有日志
ALTER SYSTEM ARCHIVE LOG ALL;
-- Database can now be opened normally.
#  打开数据库
ALTER DATABASE OPEN;
-- Commands to add tempfiles to temporary tablespaces.
-- Online tempfiles have complete space information.
-- Other tempfiles may require adjustment.
#  添加临时表空间
ALTER TABLESPACE TEMP ADD TEMPFILE '/u01/oradata/xienfei/temp01.dbf'
     SIZE 32505856  REUSE AUTOEXTEND ON NEXT 655360  MAXSIZE 32767M;
-- End of tempfile additions.
##    下面的RESETLOGS 重建语句， online redo log 中的内容都会丢失，所有的备份将无效
##    仅仅在online log 损坏的情况下使用 resetlogs 创建控制文件开启数据库。
##    resetlog创建控制文件后的开启步骤与noresetlog有一些不一样，请注意下面步骤
--     Set #2. RESETLOGS case
--
-- The following commands will create a new control file and use it
-- to open the database.
-- Data used by Recovery Manager will be lost.
-- The contents of online logs will be lost and all backups will
-- be invalidated. Use this only if online logs are damaged.
-- After mounting the created controlfile, the following SQL
-- statement will place the database in the appropriate
-- protection mode:
--  ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE PERFORMANCE
STARTUP NOMOUNT
CREATE CONTROLFILE REUSE DATABASE "XIENFEI" RESETLOGS FORCE LOGGING ARCHIVELOG
    MAXLOGFILES 16
    MAXLOGMEMBERS 3
    MAXDATAFILES 100
    MAXINSTANCES 8
    MAXLOGHISTORY 2921
LOGFILE
  GROUP 1 '/u01/oradata/xienfei/redo01.log'  SIZE 5M,
  GROUP 2 '/u01/oradata/xienfei/redo02.log'  SIZE 5M,
  GROUP 3 '/u01/oradata/xienfei/redo03.log'  SIZE 5M
-- STANDBY LOGFILE
DATAFILE
  '/u01/oradata/xienfei/system01.dbf',
  '/u01/oradata/xienfei/undotbs01.dbf',
  '/u01/oradata/xienfei/sysaux01.dbf',
  '/u01/oradata/xienfei/users01.dbf',
  '/u01/oradata/xienfei/example01.dbf',
  '/u01/oradata/xienfei/xff01.dbf',
  '/u01/oradata/xienfei/cf01.dbf',
  '/u01/oradata/xienfei/cfxff01.dbf'
CHARACTER SET ZHS16GBK
;
-- Commands to re-create incarnation table
-- Below log names MUST be changed to existing filenames on
-- disk. Any one log file from each branch can be used to
-- re-create incarnation records.
#  注册归档日志
-- ALTER DATABASE REGISTER LOGFILE '/u01/archive/1_1_756837539.arc';
-- Recovery is required if any of the datafiles are restored backups,
-- or if the last shutdown was not normal or immediate.
#  使用历史控制文件恢复数据库（因为redo log是新建，所以对于redo 而言，控制文件是新建的）
RECOVER DATABASE USING BACKUP CONTROLFILE
-- Database can now be opened zeroing the online logs.
#  使用RESETLOGS方式打开数据库
ALTER DATABASE OPEN RESETLOGS;
-- Commands to add tempfiles to temporary tablespaces.
-- Online tempfiles have complete space information.
-- Other tempfiles may require adjustment.
#  添加临时表空间
ALTER TABLESPACE TEMP ADD TEMPFILE '/u01/oradata/xienfei/temp01.dbf'
     SIZE 32505856  REUSE AUTOEXTEND ON NEXT 655360  MAXSIZE 32767M;
-- End of tempfile additions.