感谢中亦安图提供机会和大家分享了《恶意破坏的Oracle数据库恢复案例分享》主题ppt分享如下:
PPT下载:恶意破坏的Oracle数据库恢复案例分享
12c ORA-01113 ORA-01110 恢复
数据库版本
数据库启动报错
Completed: ALTER DATABASE MOUNT Thu Aug 17 12:34:52 2017 alter database open Thu Aug 17 12:34:52 2017 Ping without log force is disabled . Thu Aug 17 12:34:52 2017 Errors in file D:\APP\ORACLE\diag\rdbms\XIFENFEI\XIFENFEI\trace\XIFENFEI_ora_5960.trc: ORA-01113: 文件 5 需要介质恢复 ORA-01110: 数据文件 5: 'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_UNDOTBS1_DN9MQJFK_.DBF' ORA-1113 signalled during: alter database open
客户尝试恢复
SQL> startup mount; ORACLE 例程已经启动。 Total System Global Area 1.0301E+10 bytes Fixed Size 3842760 bytes Variable Size 1778388280 bytes Database Buffers 8489271296 bytes Redo Buffers 29708288 bytes 数据库装载完毕。 SQL> alter database open; alter database open * 第 1 行出现错误: ORA-01113: 文件 5 需要介质恢复 ORA-01110: 数据文件 5: 'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_UNDOTBS1_DN9MQJFK_.DBF' SQL> alter database datafile 5 offline drop; 数据库已更改。 SQL> alter database open; alter database open * 第 1 行出现错误: ORA-01113: 文件 6 需要介质恢复 ORA-01110: 数据文件 6: 'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_USERS_DN9MQH75_.DBF' SQL> recover datafile 6; ORA-00283: 恢复会话因错误而取消 ORA-00322: 日志 3 (用于线程 1) 不是最新副本 ORA-00312: 联机日志 3 线程 1: 'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6DW_.LOG' ORA-00322: 日志 3 (用于线程 1) 不是最新副本 ORA-00312: 联机日志 3 线程 1: 'D:\APP\ORACLE\ORADATA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6BX_.LOG'
使用Oracle Database Recovery Check检测结果
尝试恢复
SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-00322: log 3 of thread 1 is not current copy
ORA-00312: online log 3 thread 1:
'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6DW_.LOG'
ORA-00322: log 3 of thread 1 is not current copy
ORA-00312: online log 3 thread 1:
'D:\APP\ORACLE\ORADATA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6BX_.LOG'
SQL> recover database until cancel
ORA-00279: 更改 9843709 (在 08/17/2017 07:04:02 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ARCHIVELOG\2017_08_17\O1_MF_1_717_%U_.ARC
ORA-00280: 更改 9843709 (用于线程 1) 在序列 #717 中
指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_2_DN9MV69G_.LOG
ORA-00310: archived log contains sequence 716; sequence 717 required
ORA-00334: archived log:
'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_2_DN9MV69G_.LOG'
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1:
'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_SYSTEM_DN9MN5OT_.DBF'
SQL> recover database until cancel
ORA-00279: 更改 9843709 (在 08/17/2017 07:04:02 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ARCHIVELOG\2017_08_17\O1_MF_1_717_%U_.AR
C
ORA-00280: 更改 9843709 (用于线程 1) 在序列 #717 中
指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV661_.LOG
ORA-00310: archived log contains sequence 715; sequence 717 required
ORA-00334: archived log:
'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV661_.LOG'
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1:
'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_SYSTEM_DN9MN5OT_.DBF'
SQL> recover database until cancel
ORA-00279: 更改 9843709 (在 08/17/2017 07:04:02 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ARCHIVELOG\2017_08_17\O1_MF_1_717_%U_.ARC
ORA-00280: 更改 9843709 (用于线程 1) 在序列 #717 中
指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6DW_.LOG
已应用的日志。
完成介质恢复。
SQL> alter database datafile 5 online;
数据库已更改。
SQL> recover database until cancel;
ORA-00279: 更改 9843709 (在 08/17/2017 07:04:02 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ARCHIVELOG\2017_08_17\O1_MF_1_717_%U_.AR
C
ORA-00280: 更改 9843709 (用于线程 1) 在序列 #717 中
指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6DW_.LOG
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1:
'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_SYSTEM_DN9MN5OT_.DBF'
ORA-01112: 未启动介质恢复
SQL> recover datafile 5;
ORA-00283: 恢复会话因错误而取消
ORA-00264: 不要求恢复
SQL> SELECT status,
2 checkpoint_change#,
3 checkpoint_time,FUZZY,
4 count(*) ROW_NUM
5 FROM v$datafile_header
6 GROUP BY status, checkpoint_change#, checkpoint_time,fuzzy
7 ORDER BY status, checkpoint_change#, checkpoint_time;
STATUS CHECKPOINT_CHANGE# CHECKPOINT_TIME FUZ ROW_NUM
------- ------------------ ------------------- --- ----------------
ONLINE 9850826 2017-08-17 08:15:45 NO 1
ONLINE 9857411 2017-08-17 08:15:45 NO 7
ONLINE 9857411 2017-08-17 09:00:48 NO 2
SQL> set numw 16
SQL> SELECT status,
2 checkpoint_change#,
3 checkpoint_time,last_change#,
4 count(*) ROW_NUM
5 FROM v$datafile
6 GROUP BY status, checkpoint_change#, checkpoint_time,last_change#
7 ORDER BY status, checkpoint_change#, checkpoint_time;
STATUS CHECKPOINT_CHANGE# CHECKPOINT_TIME LAST_CHANGE# ROW_NUM
------- ------------------ ------------------- ---------------- ----------------
ONLINE 9857411 2017-08-17 09:00:48 9850826 1
ONLINE 9857411 2017-08-17 09:00:48 9857411 8
SYSTEM 9857411 2017-08-17 09:00:48 9857411 1
SQL> alter database open resetlogs;
alter database open resetlogs
*
第 1 行出现错误:
ORA-01152: 文件 1 没有从过旧的备份中还原
ORA-01110: 数据文件 1:
'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_SYSTEM_DN9MN5OT_.DBF'
[/shell]
这里比较明显，由于controlfile的scn 大于db的scn，从而出现了ORA-01152的错误,重试重建控制文件
<br>
<strong>重建控制文件</strong>
SQL> alter database backup controlfile to trace as 'd:\app\ctl.txt';
数据库已更改。
SQL> startup nomount;
ORACLE 例程已经启动。
Total System Global Area 10301210624 bytes
Fixed Size 3842760 bytes
Variable Size 1778388280 bytes
Database Buffers 8489271296 bytes
Redo Buffers 29708288 bytes
SQL> CREATE CONTROLFILE REUSE DATABASE "XIFENFEI" NORESETLOGS NOARCHIVELOG
2 MAXLOGFILES 16
3 MAXLOGMEMBERS 3
4 MAXDATAFILES 100
5 MAXINSTANCES 8
6 MAXLOGHISTORY 292
7 LOGFILE
8 GROUP 1 (
9 'D:\APP\ORACLE\ORADATA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV652_.LOG',
10 'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV661_.LO
G'
11 ) SIZE 50M BLOCKSIZE 512,
12 GROUP 2 (
13 'D:\APP\ORACLE\ORADATA\XIFENFEI\ONLINELOG\O1_MF_2_DN9MV68H_.LOG',
14 'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_2_DN9MV69G_.LO
G'
15 ) SIZE 50M BLOCKSIZE 512,
16 GROUP 3 (
17 'D:\APP\ORACLE\ORADATA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6BX_.LOG',
18 'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_3_DN9MV6DW_.LO
G'
19 ) SIZE 50M BLOCKSIZE 512
20 DATAFILE
21 'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_SYSTEM_DN9MN5OT_.DBF',
22 'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_SYSAUX_DN9MK6B3_.DBF',
23 'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_UNDOTBS1_DN9MQJFK_.DBF',
24 'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_USERS_DN9MQH75_.DBF',
25 'D:\APP\ORACLE\PRODUCT\12.1.0\DBHOME_1\DATABASE\XIFENFEIFILE06',
26 'D:\APP\ORACLE\PRODUCT\12.1.0\DBHOME_1\DATABASE\XIFENFEIFILE05',
27 'D:\APP\ORACLE\PRODUCT\12.1.0\DBHOME_1\DATABASE\XIFENFEIFILE04',
28 'D:\APP\ORACLE\PRODUCT\12.1.0\DBHOME_1\DATABASE\XIFENFEIFILE03',
29 'D:\APP\ORACLE\PRODUCT\12.1.0\DBHOME_1\DATABASE\XIFENFEIFILE02',
30 'D:\APP\ORACLE\PRODUCT\12.1.0\DBHOME_1\DATABASE\XIFENFEIFILE01'
31 CHARACTER SET ZHS16GBK
32 ;
控制文件已创建。
尝试恢复
SQL> recover database; 完成介质恢复。 SQL> alter database open; alter database open * 第 1 行出现错误: ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00600: internal error code, arguments: [4194], [41], [36], [], [], [], [], [], [], [], [], [] 进程 ID: 2864 会话 ID: 62 序列号: 54236
本来到这一步，错误比较明显，undo异常，这类直接对undo进行处理即可，可是运气不太好
异常crash之后redo损坏
SQL> startup pfile='d:/app/pfile.txt'
ORACLE 例程已经启动。
Total System Global Area 1.0301E+10 bytes
Fixed Size 3842760 bytes
Variable Size 1778388280 bytes
Database Buffers 8489271296 bytes
Redo Buffers 29708288 bytes
数据库装载完毕。
ORA-00354: 损坏重做日志块标头
ORA-00353: 日志损坏接近块 60 更改 23924938639111 时间 08/17/2017 21:36:16
ORA-00312: 联机日志 1 线程 1:
'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV661_.LOG'
ORA-00312: 联机日志 1 线程 1:
'D:\APP\ORACLE\ORADATA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV652_.LOG'
SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 60 change 23924938639111 time 08/17/201721:36:16
ORA-00312: online log 1 thread 1:
'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV661_.LOG'
ORA-00312: online log 1 thread 1:
'D:\APP\ORACLE\ORADATA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV652_.LOG'
SQL> recover database until cancel;
ORA-00279: 更改 10050832 (在 08/17/2017 21:36:13 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ARCHIVELOG\2017_08_17\O1_MF_1_718_%U_.ARC
ORA-00280: 更改 10050832 (用于线程 1) 在序列 #718 中
指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV661_.LOG
ORA-00283: recovery session canceled due to errors
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 60 change 23924938639111 time 08/17/2017 21:36:16
ORA-00334: archived log:
'D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ONLINELOG\O1_MF_1_DN9MV661_.LOG'
到这一步，只能通过屏蔽oracle 事务前滚，强制拉库恢复。
SQL> startup pfile='d:/app/pfile.txt' mount;
ORACLE 例程已经启动。
Total System Global Area 1.0301E+10 bytes
Fixed Size 3842760 bytes
Variable Size 1778388280 bytes
Database Buffers 8489271296 bytes
Redo Buffers 29708288 bytes
数据库装载完毕。
SQL> recover database until cancel;
ORA-00279: 更改 10050832 (在 08/17/2017 21:36:13 生成) 对于线程 1 是必需的
ORA-00289: 建议: D:\APP\ORACLE\FAST_RECOVERY_AREA\XIFENFEI\ARCHIVELOG\2017_08_17\O
1_MF_1_718_%U_.ARC
ORA-00280: 更改 10050832 (用于线程 1) 在序列 #718 中
指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: 'D:\APP\ORACLE\ORADATA\XIFENFEI\DATAFILE\O1_MF_SYSTEM_DN9M
N5OT_.DBF'
ORA-01112: 未启动介质恢复
SQL> alter database open resetlogs;
数据库已更改。
mount: wrong fs type, bad option, bad superblock恢复Oracle
有朋友找到我们,说对lv进行收缩操作之后,导致文件系统无法mount,提示超级块损坏.
尝试mount失败
[root@GZGSDB data]# mount /dev/vg_gzgsdb/lv_home /home
mount: wrong fs type, bad option, bad superblock on /dev/mapper/vg_gzgsdb-lv_home,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
系统日志报错
Aug 16 21:23:24 GZGSDB kernel: EXT4-fs (dm-5): ext4_check_descriptors: Block bitmap for group 1 not in group (block 0)! Aug 16 21:23:24 GZGSDB kernel: EXT4-fs (dm-5): group descriptors corrupted!
主要操作
这里可以看出来，明显操作错误，在没有收缩文件系统之前，直接收缩的lv.
这里可以看出来，在发生故障之后，又做了很多操作，包括fsck和testdisk等,最终无法恢复,请求我们协助处理。
找出来备份超级块
[root@GZGSDB dul]# dumpe2fs /dev/vg_gzgsdb/lv_home |grep superblock dumpe2fs 1.41.12 (17-May-2010) ext2fs_read_bb_inode: A block group is missing an inode table Primary superblock at 0, Group descriptors at 1-52 Backup superblock at 32768, Group descriptors at 32769-32820 Backup superblock at 98304, Group descriptors at 98305-98356 Backup superblock at 163840, Group descriptors at 163841-163892 Backup superblock at 229376, Group descriptors at 229377-229428 Backup superblock at 294912, Group descriptors at 294913-294964 Backup superblock at 819200, Group descriptors at 819201-819252 Backup superblock at 884736, Group descriptors at 884737-884788 Backup superblock at 1605632, Group descriptors at 1605633-1605684 Backup superblock at 2654208, Group descriptors at 2654209-2654260 Backup superblock at 4096000, Group descriptors at 4096001-4096052 Backup superblock at 7962624, Group descriptors at 7962625-7962676 Backup superblock at 11239424, Group descriptors at 11239425-11239476 Backup superblock at 20480000, Group descriptors at 20480001-20480052 Backup superblock at 23887872, Group descriptors at 23887873-23887924 Backup superblock at 71663616, Group descriptors at 71663617-71663668 Backup superblock at 78675968, Group descriptors at 78675969-78676020 Backup superblock at 102400000, Group descriptors at 102400001-102400052 Backup superblock at 214990848, Group descriptors at 214990849-214990900
使用fsck修复
[root@GZGSDB data]# fsck -y /dev/vg_gzgsdb/lv_home
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
fsck.ext4: Group descriptors look bad... trying backup blocks...
fsck.ext4: The ext2 superblock is corrupt when using the backup blocks
fsck.ext4: going back to original superblock
fsck.ext4: Device or resource busy while trying to open /dev/mapper/vg_gzgsdb-lv_home
Filesystem mounted or opened exclusively by another program?
--指定超级块恢复
[root@GZGSDB data]# fsck -y -b 102400000 /dev/vg_gzgsdb/lv_home
…………
Illegal block #0 (1296647770) in inode 354315. CLEARED.
Illegal block #1 (1398362886) in inode 354315. CLEARED.
Illegal block #3 (453538936) in inode 354315. CLEARED.
Illegal block #5 (808333361) in inode 354315. CLEARED.
Illegal block #6 (775434798) in inode 354315. CLEARED.
Illegal block #8 (1180306180) in inode 354315. CLEARED.
Illegal block #9 (1413893971) in inode 354315. CLEARED.
Illegal block #10 (1229347423) in inode 354315. CLEARED.
Illegal block #11 (1498613325) in inode 354315. CLEARED.
Illegal indirect block (1296389203) in inode 354315. CLEARED.
Inode 354315 is too big. Truncate? yes
Block #1074301965 (69632) causes directory to be too big. CLEARED.
Warning... fsck.ext4 for device /dev/mapper/vg_gzgsdb-lv_home exited with signal 11.
[root@GZGSDB data]# mount /dev/vg_gzgsdb/lv_home /home
mount: wrong fs type, bad option, bad superblock on /dev/mapper/vg_gzgsdb-lv_home,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
至此基本上可以判断,直接修复该文件系统,让其正常mount的概率很小.直接通过lun层面来恢复.
通过工具解析lun
比较明显,客户经过一系列操作现在的现象是lv 1.03T,文件系统只有821G,明显和客户给我反馈的操作不符(应该是lv 821G),证明客户做了大量操作,已经导致文件系统损坏
由于文件系统严重异常,工具获取到的文件都是没有名称,直接从inode里面读取的数据.获取到这些数据之后,然后结合oracle的特性,判断出来对应的文件号关系(这里有大量文件重复);另外有个别文件通过inode恢复丢失,通过底层碎片重组进行恢复出来文件,然后恢复出来其中数据(asm disk header 彻底损坏恢复).这个客户比较幸运,system文件在另外一个分区中,不然工作量会大很多.
再次提醒:对lv操作一定要谨慎,特别是lvreduce操作,另外出现发生误操作之后,应该第一时间保护现场,而不是百度着去乱操作,可能导致故障更加悲剧
Quick Reference to Patch Numbers for Database/GI PSU, SPU(CPU), Bundle Patches and Patchsets—201707
|
12.1.0.2 (12.1.0.2.0 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
11.2.0.4 (11.2.0.4.0 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
11.2.0.3 (11.2.0.3.0 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
11.2.0.2 (11.2.0.2.0 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
11.1.0.7 (11.1.0.7.0 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
A10.2.0.4 (10.2.0.4.0 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
B10.2.0.3 (10.2.0.3 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
10.2.0.2 (10.2.0.2 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
10.1.0.5 (10.1.0.5 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
10.1.0.4 (10.1.0.4 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
10.1.0.3 (10.1.0.3 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
9.2.0.8 (9.2.0.8 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
9.2.0.7 (9.2.0.7 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
9.2.0.6 (9.2.0.6 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
9.2.0.5 (ORACLE 9I DATABASE SERVER RELEASE 2 – PATCH SET 4 VERSION 9.2.0.5.0) |
|
|
9.2.0.4 (9.2.0.4 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
9.2.0.3 (9.2.0.3 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
9.2.0.2 (9.2.0.2 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
9.0.1.5 (9.0.1.5 PATCHSET) |
|
|
9.0.1.4 (9.0.1.4 PATCH SET FOR ORACLE DATABASE SERVER) |
|
|
9.0.1.3 (9.0.1.3. PATCH SET FOR ORACLE DATA SERVER) |
|
|
8.1.7.4 (8.1.7.4 PATCH SET FOR ORACLE DATA SERVER) |
|
|
8.1.7.3 (8.1.7.3 PATCH SET FOR ORACLE DATA SERVER) |
|
|
8.1.7.2 (8.1.7.2.1 PATCH SET FOR ORACLE DATA SERVER) |
|
12.2.0.1 |
|||
|
Description |
Database RU |
GI RU |
Windows Bundle Patch |
|
JUL2017 (12.2.0.1.170718) |
|||
|
12.1.0.2 |
||||
|
Description |
PSU |
GI PSU |
Proactive Bundle Patch |
Bundle Patch (Windows 32bit & 64bit) |
|
JUL2017 (12.1.0.2.170718) |
||||
|
APR2017 (12.1.0.2.170418) |
||||
|
JAN2017 (12.1.0.2.170117) |
||||
|
OCT2016 (12.1.0.2.161018) |
||||
|
JUL2016 (12.1.0.2.160719) |
||||
|
APR2016 (12.1.0.2.160419) |
||||
|
JAN2016 (12.1.0.2.160119) |
||||
|
OCT2015 |
21359755 (12.1.0.2.5) |
21523234 (12.1.0.2.5) |
21744410 (12.1.0.2.13) |
21821214 (12.1.0.2.10) |
|
JUL2015 |
20831110 (12.1.0.2.4) |
20996835 (12.1.0.2.4) |
21188742 (12.1.0.2.10) |
21126814 (12.1.0.2.7) |
|
APR2015 |
20299023 (12.1.0.2.3) |
20485724 (12.1.0.2.3) |
20698050 (12.1.0.2.7) |
20684004 (12.1.0.2.4) |
|
JAN2015 |
19769480 (12.1.0.2.2) |
19954978 (12.1.0.2.2) |
20141343 (12.1.0.2.4) |
19720843 (12.1.0.2.1) |
|
OCT2014 |
19303936 (12.1.0.2.1) |
19392646 (12.1.0.2.1) |
19404326 (12.1.0.2.1) |
N/A |
|
12.1.0.1 |
||||||
|
Description |
PSU |
GI PSU |
Bundle Patch |
|||
|
A |
B |
Windows 64 bit |
Windows 32 bit |
|||
|
JUL2016 (12.1.0.1.160719) |
23273935 | |||||
|
APR2016 (12.1.0.1.160419) |
||||||
|
JAN2016 (12.1.0.1.160119) |
||||||
|
OCT2015 |
21352619(12.1.0.1.9) |
21551666(12.1.0.1.9) |
21551685(12.1.0.1.9) |
21744907 (12.1.0.1.21) |
||
|
JUL2015 |
20831107(12.1.0.1.8) |
20996901(12.1.0.1.8) |
20996911(12.1.0.1.8) |
21076681 (12.1.0.1.20) |
||
|
APR2015 |
20299016(12.1.0.1.7) |
20485762(12.1.0.1.7) |
19971331(12.1.0.1.7) |
20558101 (12.1.0.1.18) |
||
|
JAN2015 |
19769486(12.1.0.1.6) |
19971324(12.1.0.1.6) |
19971331(12.1.0.1.6) |
20160748 (12.1.0.1.16) |
||
|
OCT2014 |
19121550(12.1.0.1.5) |
19392372(12.1.0.1.5) |
19392451(12.1.0.1.5) |
19542943 (12.1.0.1.14) |
||
|
JUL2014 |
18522516(12.1.0.1.4) |
18705901(12.1.0.1.4) |
18705972(12.1.0.1.4) |
19062327 (12.1.0.1.11) |
||
|
APR2014 |
18031528(12.1.0.1.3) |
18139660(12.1.0.1.3) |
18413105(12.1.0.1.3) |
18448604 (12.1.0.1.7) |
||
|
JAN2014 |
17552800(12.1.0.1.2) |
17735306 (12.1.0.1.2) |
17977915 (12.1.0.1.3) |
|||
|
OCT2013 |
17027533(12.1.0.1.1) |
17272829 (12.1.0.1.1) |
17363796(12.1.0.1.1) |
17363795(12.1.0.1.1) |
||
|
11.2.0.4 |
||||
|
Description |
PSU |
SPU(CPU) |
GI PSU |
Bundle Patch (Windows 32bit & 64bit) |
|
JUL2017 (11.2.0.4.170718) |
||||
|
APR2017 (11.2.0.4.170418) |
||||
|
JAN2017 |
N/A |
N/A |
N/A |
N/A |
|
OCT2016 (11.2.0.4.161018) |
||||
|
JUL2016 (11.2.0.4.160719) |
23530402 |
|||
|
APR2016 (11.2.0.4.160419) |
22839608 |
|||
|
JAN2016 (11.2.0.4.160119) |
22310544 |
|||
|
OCT2015 |
21352635 (11.2.0.4.8) |
21523375 (11.2.0.4.8) |
21821802 (11.2.0.4.20) |
|
|
JUL2015 |
20760982 (11.2.0.4.7) |
20996923 (11.2.0.4.7) |
21469106 (11.2.0.4.18) |
|
|
APR2015 |
20299013 (11.2.0.4.6) |
20485808 (11.2.0.4.6) |
20544696 (11.2.0.4.15) |
|
|
JAN2015 |
19769489 (11.2.0.4.5) |
19955028 (11.2.0.4.5) |
20127071 (11.2.0.4.12) |
|
|
OCT2014 |
19121551 (11.2.0.4.4) |
19380115 (11.2.0.4.4) |
19651773 (11.2.0.4.10) |
|
|
JUL2014 |
18522509 (11.2.0.4.3) |
18706472 (11.2.0.4.3) |
18842982 (11.2.0.4.7) |
|
|
APR2014 |
18031668 (11.2.0.4.2) |
18139609 (11.2.0.4.2) |
18296644 (11.2.0.4.4) |
|
|
JAN2014 |
17478514 (11.2.0.4.1) |
N/A |
17987366 (11.2.0.4.1) |
|
Assistant: Download Reference for Oracle Database/GI PSU, SPU(CPU), Bundle Patches, Patchsets and Base Releases (文档 ID 2118136.2)
ORA-12012: error on auto execute of job "SYS"."ORA$AT_OS_OPT_SY_383"
12.2.0.1版本数据库报类似ORA-12012: error on auto execute of job “SYS”.”ORA$AT_OS_OPT_SY_383″错误
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_j000_10213.trc: ORA-12012: error on auto execute of job "SYS"."ORA$AT_OS_OPT_SY_383" ORA-20001: Statistics Advisor: Invalid task name for the current user ORA-06512: at "SYS.DBMS_STATS", line 47207 ORA-06512: at "SYS.DBMS_STATS_ADVISOR", line 882 ORA-06512: at "SYS.DBMS_STATS_INTERNAL", line 20059 ORA-06512: at "SYS.DBMS_STATS_INTERNAL", line 22201 ORA-06512: at "SYS.DBMS_STATS", line 47197
trace文件信息
Trace file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_j000_10379.trc Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production Build label: RDBMS_12.2.0.1.0_LINUX.X64_170125 ORACLE_HOME: /u01/app/oracle/product/12.2.0/db_1 System name: Linux Node name: yoridb2 Release: 3.10.0-514.26.2.el7.x86_64 Version: #1 SMP Tue Jul 4 15:04:05 UTC 2017 Machine: x86_64 Instance name: xifenfei Redo thread mounted by this instance: 1 Oracle process number: 148 Unix process pid: 10379, image: oracle@yoridb2 (J000) *** 2017-08-04T14:41:26.486692+08:00 *** SESSION ID:(508.52539) 2017-08-04T14:41:26.486716+08:00 *** CLIENT ID:() 2017-08-04T14:41:26.486722+08:00 *** SERVICE NAME:(SYS$USERS) 2017-08-04T14:41:26.486727+08:00 *** MODULE NAME:(DBMS_SCHEDULER) 2017-08-04T14:41:26.486732+08:00 *** ACTION NAME:(ORA$AT_OS_OPT_SY_384) 2017-08-04T14:41:26.486738+08:00 *** CLIENT DRIVER:() 2017-08-04T14:41:26.486743+08:00 ORA-12012: error on auto execute of job "SYS"."ORA$AT_OS_OPT_SY_384" ORA-20001: Statistics Advisor: Invalid task name for the current user ORA-06512: at "SYS.DBMS_STATS", line 47207 ORA-06512: at "SYS.DBMS_STATS_ADVISOR", line 882 ORA-06512: at "SYS.DBMS_STATS_INTERNAL", line 20059 ORA-06512: at "SYS.DBMS_STATS_INTERNAL", line 22201 ORA-06512: at "SYS.DBMS_STATS", line 47197
错误比较明显，是在执行统计信息分析的时候报错了，报错原因是由于SCHEDULER的task name无效.
解决方法
[oracle@yoridb2 trace]$ ss
SQL*Plus: Release 12.2.0.1.0 Production on Fri Aug 4 23:40:33 2017
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
SQL> select name, ctime, how_created
2 from sys.wri$_adv_tasks
3 where owner_name = 'SYS'
and name in ('AUTO_STATS_ADVISOR_TASK','INDIVIDUAL_STATS_ADVISOR_TASK'); 4
no rows selected
SQL> EXEC dbms_stats.init_package();
PL/SQL procedure successfully completed.
SQL> select name, ctime, how_created
2 from sys.wri$_adv_tasks
3 where owner_name = 'SYS'
and name in ('AUTO_STATS_ADVISOR_TASK','INDIVIDUAL_STATS_ADVISOR_TASK'); 4
NAME
--------------------------------------------------------------------------------
CTIME HOW_CREATED
--------- ------------------------------
AUTO_STATS_ADVISOR_TASK
04-AUG-17 CMD
INDIVIDUAL_STATS_ADVISOR_TASK
04-AUG-17 CMD
Disable Transparent HugePages
从redhat 6开始引入了Transparent HugePages,但是oracle一直建议disable 它,而使用标准HugePages方式.对于6和7的禁用方式有一些区别.
linux 6
修改/etc/grub.conf之后重启系统生效
vi /etc/grub.conf
title Oracle Linux Server (2.6.32-300.25.1.el6uek.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-300.25.1.el6uek.x86_64 ro root=LABEL=/ transparent_hugepage=never
initrd /initramfs-2.6.32-300.25.1.el6uek.x86_64.img
linux 7
修改/etc/default/grub然后执行grub2-mkconfig并重启系统生效
[root@xifenfei u01]# vi /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="nomodeset vconsole.font=latarcyrheb-sun16 vconsole.keymap=us crashkernel=auto
biosdevname=0 transparent_hugepage=never"
GRUB_DISABLE_RECOVERY="true"
~
[root@xifenfei u01]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-514.26.2.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-514.26.2.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-123.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-123.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-cb0b6b4de89a4fe4acfc8774c2f01486
Found initrd image: /boot/initramfs-0-rescue-cb0b6b4de89a4fe4acfc8774c2f01486.img
done
临时禁用
该方法对于linux 6和7均有效,不用重启系统
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi
警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703
继续上篇的tab$被清空(ORA-600 16703故障解析—tab$表被清空),导致数据库启动异常的case
ORA-600 16703报错
数据库日志分析
数据库open成功同时报ORA-7445 kokeicbegin和ORA-600 kzrini:!uprofile错误
再次启动数据库直接报ORA-600 16703错误
这里有一个其他现象,就是数据库在open成功的同时(同一秒内),开始报异常.重启之后直接报
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
根据10046分析结果
===================== select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1 END OF STMT PARSE #140048443935120:c=0,e=390,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905161433 ===================== select blevel, leafcnt, distkey, lblkkey, dblkkey, clufac, nvl(degree,1), nvl(instances,1) from ind$ where bo# = :1 and obj# = :2 END OF STMT PARSE #140048443934176:c=1000,e=601,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162088 ===================== PARSING IN CURSOR #140048443933232 len=70 dep=1 uid=0 oct=3 lid=0 tim=1499185905162444 hv=3377894161 ad='84f13d70' sqlid='32d4jrb4pd4sj' select charsetid, charsetform from col$ where obj# = :1 and col# = :2 END OF STMT PARSE #140048443933232:c=0,e=294,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162443 ===================== PARSING IN CURSOR #140048443932288 len=52 dep=1 uid=0 oct=3 lid=0 tim=1499185905247020 hv=429618617 ad='84f0f2d0' sqlid='4krwuz0ctqxdt' select ctime, mtime, stime from obj$ where obj# = :1 END OF STMT PARSE #140048443932288:c=0,e=549,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905247019 BINDS #140048443932288: select blevel, leafcnt, distkey, lblkkey, dblkkey, clufac, nvl(degree,1), nvl(instances,1) from ind$ where bo# = :1 and obj# = :2 END OF STMT PARSE #140048443934176:c=1000,e=601,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162088 ===================== PARSING IN CURSOR #140048443933232 len=70 dep=1 uid=0 oct=3 lid=0 tim=1499185905162444 hv=3377894161 ad='84f13d70' sqlid='32d4jrb4pd4sj' select charsetid, charsetform from col$ where obj# = :1 and col# = :2 END OF STMT PARSE #140048443933232:c=0,e=294,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162443 ===================== PARSING IN CURSOR #140048443932288 len=52 dep=1 uid=0 oct=3 lid=0 tim=1499185905247020 hv=429618617 ad='84f0f2d0' sqlid='4krwuz0ctqxdt' select ctime, mtime, stime from obj$ where obj# = :1 END OF STMT PARSE #140048443932288:c=0,e=549,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905247019 BINDS #140048443932288: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=00 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=7f5f91b87bd0 bln=22 avl=02 flg=05 value=20 EXEC #140048443932288:c=2000,e=2686,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=1218588913,tim=1499185905249810 WAIT #140048443932288: nam='db file sequential read' ela= 6205 file#=1 block#=337 blocks=1 obj#=36 tim=1499185905256132 WAIT #140048443932288: nam='db file sequential read' ela= 3739 file#=1 block#=338 blocks=1 obj#=36 tim=1499185905266704 WAIT #140048443932288: nam='db file sequential read' ela= 4966 file#=1 block#=241 blocks=1 obj#=18 tim=1499185905271761 FETCH #140048443932288:c=0,e=21976,p=3,cr=3,cu=0,mis=0,r=1,dep=1,og=4,plh=1218588913,tim=1499185905271820 STAT #140048443932288 id=1 cnt=1 pid=0 pos=1 obj=18 op='TABLE ACCESS BY INDEX ROWID OBJ$ (cr=3 pr=3 pw=0 time=21993 us)' STAT #140048443932288 id=2 cnt=1 pid=1 pos=1 obj=36 op='INDEX RANGE SCAN I_OBJ1 (cr=2 pr=2 pw=0 time=16923 us)' CLOSE #140048443932288:c=0,e=63,dep=1,type=0,tim=1499185905271941 BINDS #140048443935120: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=7f5f91c07de8 bln=22 avl=02 flg=05 value=20 EXEC #140048443935120:c=1000,e=795,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=2970138452,tim=1499185905272802 WAIT #140048443935120: nam='db file sequential read' ela= 3197 file#=1 block#=169 blocks=1 obj#=3 tim=1499185905276069 WAIT #140048443935120: nam='db file sequential read' ela= 3459 file#=1 block#=170 blocks=1 obj#=3 tim=1499185905404084 WAIT #140048443935120: nam='db file sequential read' ela= 6358 file#=1 block#=145 blocks=1 obj#=4 tim=1499185905410548 FETCH #140048443935120:c=999,e=137805,p=3,cr=3,cu=0,mis=0,r=0,dep=1,og=4,plh=2970138452,tim=1499185905410635 STAT #140048443935120 id=1 cnt=0 pid=0 pos=1 obj=4 op='TABLE ACCESS CLUSTER TAB$ (cr=3 pr=3 pw=0 time=137810 us)' STAT #140048443935120 id=2 cnt=1 pid=1 pos=1 obj=3 op='INDEX UNIQUE SCAN I_OBJ# (cr=2 pr=2 pw=0 time=131330 us)' *** 2017-07-05 00:31:46.094 Incident 176395 created, dump file: /oracle/diag/rdbms/orcl/orcl2/incident/incdir_176395/orcl_ora_51261_i176395.trc ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
以及以往恢复经验和mos,基本上可以确定是由于tab$和obj$记录不匹配导致该问题.而且是obj#=20的记录在tab$和obj$中不匹配.
分析tab$和obj$记录
Data UnLoader: 11.2.0.1.5 - Internal Only - on Wed Jul 05 01:28:53 2017 with 64-bit io functions and the decompression option Copyright (c) 1994 2017 Bernard van Duijnen All rights reserved. Strictly Oracle Internal Use Only Found db_id = 1334610369 Found db_name = orcl DUL> unload table TAB$( OBJ# number, DATAOBJ# number, 2 TS# number, FILE# number, BLOCK# number, 3 BOBJ# number, TAB# number, COLS number, CLUCOLS number, 4 PCTFREE$ ignore, PCTUSED$ ignore, INITRANS ignore, MAXTRANS ignore, 5 FLAGS ignore, AUDIT$ ignore, ROWCNT ignore, BLKCNT ignore, 6 EMPCNT ignore, AVGSPC ignore, CHNCNT ignore, AVGRLN ignore, 7 AVGSPC_FLB ignore, FLBCNT ignore, 8 ANALYZETIME ignore, SAMPLESIZE ignore, 9 DEGREE ignore, INSTANCES ignore, 10 INTCOLS ignore, KERNELCOLS number, PROPERTY number) 11 cluster C_OBJ#(OBJ#) 12 storage ( tablespace 0 segobjno 2 tabno 1 file 1 block 144); . unloading table TAB$ 0 rows unloaded DUL> unload table OBJ$( OBJ# number, DATAOBJ# number, OWNER# number, 2 NAME clean varchar2(30), NAMESPACE ignore, SUBNAME clean varchar2(30), 3 TYPE# number, CTIME ignore, MTIME ignore, STIME ignore, 4 STATUS ignore, REMOTEOWNER ignore, LINKNAME ignore, 5 FLAGS ignore, OID$ hexraw) 6 storage ( tablespace 0 segobjno 18 file 1 block 240); . unloading table OBJ$ 89804 rows unloaded DUL>
这里可以明确表示tab$无记录,obj$有记录,从而启动的过程出现ORA-600 16703错误可以很好解释.由于数据库启动成功和报错几乎同时进行，以及上面看到的tab$记录不存在的情况，初步怀疑是有startup触发器清空tab$表记录
工具分析触发器表trigger$
这里果然看到一个after startup on database的触发器,名字为DBMS_SUPPORT_DBMONITOR,而它调用的是DBMS_SUPPORT_DBMONITORP存储.
工具分析pl/sql表source$
对wraped加密的程序进行解密
这里可以明确的看到DBMS_SUPPORT_DBMONITORP存储过程备份tab$表到orachk中然后delete tab$表,现在已经明确是由于DBMS_SUPPORT_DBMONITOR触发器在数据库重启之后开始执行调用DBMS_SUPPORT_DBMONITORP程序,如果判断数据库创建时间大于等于300天,便干掉tab$表,实现数据库破坏.
查找DBMS_SUPPORT_DBMONITOR等程序源头
分析相关程序创建时间,通过obj$表的ctime和name来判断
这里可以看出来DBMS_SUPPORT_DBMONITOR和DBMS_SUPPORT_DBMONITORP的创建时间基本上和数据库核心对象的创建时间相差无几,我们可以大概排除掉,是由于pl sql等工具连接数据库导致该发问题(类似:plsql dev引起的数据库被黑勒索比特币实现原理分析和解决方案),很可能是在dbca创建库的过程中就已经带有了DBMS_SUPPORT_DBMONITOR等程序,如果这样那很可能是由于数据库的安装介质被破坏导致该问题.
分析DBMS_SUPPORT_DBMONITOR来源
这里已经很清晰,由于prvtsupp.plb被人注入了恶意脚本从而使得数据库被创建了DBMS_SUPPORT_DBMONITOR的触发器和DBMS_SUPPORT_DBMONITORP的存储过程,从而实现了对数据库的而易破坏.
确定破坏文件prvtsupp.plb来源于介质
这里比较明显,文件就是来源database\stage\Components\oracle.rdbms.dbscripts\11.2.0.4.0\1\DataFiles\filegroup2.jar\rdbms\admin\prvtsupp.plb文件被修改导致
通过md5码对比,可以确定是有人对软件的安装介质进行了破坏,从而实现了恶意代码的注入,从而实现了数据库300天之后重启之后无法正常启动而是出现类似ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []的错误.
温馨提示
各位一定要从官方途径下载oracle安装介质,如果是从其他互联网途径下载一定要验证md5,确保文件没有被人恶意篡改,造成无可挽回的损坏.如果真的不幸遇到这类问题,请保护现场联系我们
Tel:17813235971 Q Q:107644445
E-Mail:dba@xifenfei.com
我们可以通过使用bbed对tab$表数据数据进行恢复实现数据库正常启动,实现数据0丢失,最大限度抢救您的数据和减少业务恢复时间
ORA-600 16703故障解析—tab$表被清空
最近连续遇到两次数据库启动报ORA-600 16703错误,而导致数据库无法正常启动的问题
ORA-600 16703报错
Completed: ALTER DATABASE RECOVER database alter database open Beginning crash recovery of 1 threads parallel recovery started with 32 processes Started redo scan Completed redo scan read 0 KB redo, 0 data blocks need recovery Started redo application at Thread 1: logseq 993752, block 2, scn 14872376881763 Recovery of Online Redo Log: Thread 1 Group 6 Seq 993752 Reading mem 0 Mem# 0: /oracle/oradata/xifenfei/redo06.log Completed redo application of 0.00MB Completed crash recovery at Thread 1: logseq 993752, block 3, scn 14872376901765 0 data blocks read, 0 data blocks written, 0 redo k-bytes read Tue Jul 04 23:13:29 2017 Thread 1 advanced to log sequence 993753 (thread open) Thread 1 opened at log sequence 993753 Current log# 7 seq# 993753 mem# 0: /oracle/oradata/xifenfei/redo07.log Successful open of redo thread 1 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Tue Jul 04 23:13:29 2017 SMON: enabling cache recovery Errors in file /oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_15886.trc (incident=163595): ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], [] Incident details in: /oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_163595/xifenfei_ora_15886_i163595.trc Tue Jul 04 23:13:30 2017 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_15886.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], [] Errors in file /oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_15886.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], [] Error 704 happened during db open, shutting down database USER (ospid: 15886): terminating the instance due to error 704 Instance terminated by USER, pid = 15886 ORA-1092 signalled during: alter database open... opiodr aborting process unknown ospid (15886) as a result of ORA-1092
这里报错比较明显ORA-600 16703,而且是在启动时bootstrap$中的对象出现该问题.
10046分析启动过程
===================== select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1 END OF STMT PARSE #140048443935120:c=0,e=390,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905161433 ===================== select blevel, leafcnt, distkey, lblkkey, dblkkey, clufac, nvl(degree,1), nvl(instances,1) from ind$ where bo# = :1 and obj# = :2 END OF STMT PARSE #140048443934176:c=1000,e=601,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162088 ===================== PARSING IN CURSOR #140048443933232 len=70 dep=1 uid=0 oct=3 lid=0 tim=1499185905162444 hv=3377894161 ad='84f13d70' sqlid='32d4jrb4pd4sj' select charsetid, charsetform from col$ where obj# = :1 and col# = :2 END OF STMT PARSE #140048443933232:c=0,e=294,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162443 ===================== PARSING IN CURSOR #140048443932288 len=52 dep=1 uid=0 oct=3 lid=0 tim=1499185905247020 hv=429618617 ad='84f0f2d0' sqlid='4krwuz0ctqxdt' select ctime, mtime, stime from obj$ where obj# = :1 END OF STMT PARSE #140048443932288:c=0,e=549,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905247019 BINDS #140048443932288: select blevel, leafcnt, distkey, lblkkey, dblkkey, clufac, nvl(degree,1), nvl(instances,1) from ind$ where bo# = :1 and obj# = :2 END OF STMT PARSE #140048443934176:c=1000,e=601,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162088 ===================== PARSING IN CURSOR #140048443933232 len=70 dep=1 uid=0 oct=3 lid=0 tim=1499185905162444 hv=3377894161 ad='84f13d70' sqlid='32d4jrb4pd4sj' select charsetid, charsetform from col$ where obj# = :1 and col# = :2 END OF STMT PARSE #140048443933232:c=0,e=294,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162443 ===================== PARSING IN CURSOR #140048443932288 len=52 dep=1 uid=0 oct=3 lid=0 tim=1499185905247020 hv=429618617 ad='84f0f2d0' sqlid='4krwuz0ctqxdt' select ctime, mtime, stime from obj$ where obj# = :1 END OF STMT PARSE #140048443932288:c=0,e=549,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905247019 BINDS #140048443932288: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=00 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=7f5f91b87bd0 bln=22 avl=02 flg=05 value=20 EXEC #140048443932288:c=2000,e=2686,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=1218588913,tim=1499185905249810 WAIT #140048443932288: nam='db file sequential read' ela= 6205 file#=1 block#=337 blocks=1 obj#=36 tim=1499185905256132 WAIT #140048443932288: nam='db file sequential read' ela= 3739 file#=1 block#=338 blocks=1 obj#=36 tim=1499185905266704 WAIT #140048443932288: nam='db file sequential read' ela= 4966 file#=1 block#=241 blocks=1 obj#=18 tim=1499185905271761 FETCH #140048443932288:c=0,e=21976,p=3,cr=3,cu=0,mis=0,r=1,dep=1,og=4,plh=1218588913,tim=1499185905271820 STAT #140048443932288 id=1 cnt=1 pid=0 pos=1 obj=18 op='TABLE ACCESS BY INDEX ROWID OBJ$ (cr=3 pr=3 pw=0 time=21993 us)' STAT #140048443932288 id=2 cnt=1 pid=1 pos=1 obj=36 op='INDEX RANGE SCAN I_OBJ1 (cr=2 pr=2 pw=0 time=16923 us)' CLOSE #140048443932288:c=0,e=63,dep=1,type=0,tim=1499185905271941 BINDS #140048443935120: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=7f5f91c07de8 bln=22 avl=02 flg=05 value=20 EXEC #140048443935120:c=1000,e=795,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=2970138452,tim=1499185905272802 WAIT #140048443935120: nam='db file sequential read' ela= 3197 file#=1 block#=169 blocks=1 obj#=3 tim=1499185905276069 WAIT #140048443935120: nam='db file sequential read' ela= 3459 file#=1 block#=170 blocks=1 obj#=3 tim=1499185905404084 WAIT #140048443935120: nam='db file sequential read' ela= 6358 file#=1 block#=145 blocks=1 obj#=4 tim=1499185905410548 FETCH #140048443935120:c=999,e=137805,p=3,cr=3,cu=0,mis=0,r=0,dep=1,og=4,plh=2970138452,tim=1499185905410635 STAT #140048443935120 id=1 cnt=0 pid=0 pos=1 obj=4 op='TABLE ACCESS CLUSTER TAB$ (cr=3 pr=3 pw=0 time=137810 us)' STAT #140048443935120 id=2 cnt=1 pid=1 pos=1 obj=3 op='INDEX UNIQUE SCAN I_OBJ# (cr=2 pr=2 pw=0 time=131330 us)' *** 2017-07-05 00:31:46.094 Incident 176395 created, dump file: /oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_176395/xifenfei_ora_51261_i176395.trc ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
报错信息明显,由于select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1无法正常执行.从而出现了ORA-600 16703的错误,更加直接一点的解释就是obj#=20的对象在tab$中找不到记录，从而出现此类报错.和官方解释ORA-600 TAB$和obj$不匹配一致.
分析system文件
通过dul等工具分析system文件发现tab$表记录为空
Data UnLoader: 11.2.0.1.5 - Internal Only - on Wed Jul 05 01:28:53 2017 with 64-bit io functions and the decompression option Copyright (c) 1994 2017 Bernard van Duijnen All rights reserved. Strictly Oracle Internal Use Only Found db_id = 1334610369 Found db_name = xifenfei DUL> unload table TAB$( OBJ# number, DATAOBJ# number, 2 TS# number, FILE# number, BLOCK# number, 3 BOBJ# number, TAB# number, COLS number, CLUCOLS number, 4 PCTFREE$ ignore, PCTUSED$ ignore, INITRANS ignore, MAXTRANS ignore, 5 FLAGS ignore, AUDIT$ ignore, ROWCNT ignore, BLKCNT ignore, 6 EMPCNT ignore, AVGSPC ignore, CHNCNT ignore, AVGRLN ignore, 7 AVGSPC_FLB ignore, FLBCNT ignore, 8 ANALYZETIME ignore, SAMPLESIZE ignore, 9 DEGREE ignore, INSTANCES ignore, 10 INTCOLS ignore, KERNELCOLS number, PROPERTY number) 11 cluster C_OBJ#(OBJ#) 12 storage ( tablespace 0 segobjno 2 tabno 1 file 1 block 144); . unloading table TAB$ 0 rows unloaded DUL> unload table OBJ$( OBJ# number, DATAOBJ# number, OWNER# number, 2 NAME clean varchar2(30), NAMESPACE ignore, SUBNAME clean varchar2(30), 3 TYPE# number, CTIME ignore, MTIME ignore, STIME ignore, 4 STATUS ignore, REMOTEOWNER ignore, LINKNAME ignore, 5 FLAGS ignore, OID$ hexraw) 6 storage ( tablespace 0 segobjno 18 file 1 block 240); . unloading table OBJ$ 89804 rows unloaded DUL>
发现在obj$中有创建表ORACHKBEC66CBE055000000000001(ORACHK+16进制24位)的一个表名字
该表用途通过分析数据库日志发现
create table ORACHKBEC66CBE055000000000001 tablespace system as select * from sys.tab$;
也就是说,这个orachk的表是用来备份tab$的,然后进一步发现有delete from tab$.至此基本上分析清楚,tab$表备份到ORACHK表中,然后delete tab$表数据.实现数据库破坏以及难以恢复的效果.有点类似plsql dev引起的数据库被黑勒索比特币实现原理分析和解决方案的破坏案例
我们可以通过使用bbed对tab$表数据数据进行恢复实现数据库正常启动,实现数据0丢失,最大限度抢救您的数据和减少业务恢复时间
Phone:17813235971 Q Q:107644445 E-Mail:dba@xifenfei.com
源头已经分析清楚,是有人篡改了oracle安装介质,导致数据库运行300天之后出现该问题,具体见:警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703
Oracle 12c active dataguard switchover
从12.1开始adg的切换发生了一些改变,直接使用alter database switchover to [target standby db_unique_name] verify; alter database switchover to [target standby db_unique_name]; 即可完成切换,以下是一次生产环境的具体操作步骤
主库操作
SQL> select database_role from v$database;
DATABASE_ROLE
----------------
PRIMARY
SQL> alter database switchover to xifenfei verify;
数据库已更改。
--alert日志
Sun Jun 25 09:07:08 2017
diag_adl:SWITCHOVER VERIFY: Send VERIFY request to switchover target xifenfei
diag_adl:SWITCHOVER VERIFY COMPLETE
SQL> alter database switchover to xifenfei;
数据库已更改。
--alert日志
Sun Jun 25 09:07:46 2017
diag_adl:Starting switchover [Process ID: 37024]
Sun Jun 25 09:07:46 2017
diag_adl:ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY [Process Id: 37024] (xifenfei)
diag_adl:Waiting for target standby to receive all redo
Sun Jun 25 09:07:46 2017
diag_adl:Waiting for all non-current ORLs to be archived...
Sun Jun 25 09:07:46 2017
diag_adl:All non-current ORLs have been archived.
Sun Jun 25 09:07:46 2017
diag_adl:Waiting for all FAL entries to be archived...
Sun Jun 25 09:07:46 2017
diag_adl:All FAL entries have been archived.
Sun Jun 25 09:07:46 2017
diag_adl:Waiting for dest_id 2 to become synchronized...
Sun Jun 25 09:07:47 2017
diag_adl:Active, synchronized Physical Standby switchover target has been identified
diag_adl:Preventing updates and queries at the Primary
diag_adl:Generating and shipping final logs to target standby
diag_adl:Switchover End-Of-Redo Log thread 1 sequence 96534 has been fixed
diag_adl:Switchover: Primary highest seen SCN set to 0x3.0x109d7502
diag_adl:ARCH: Noswitch archival of thread 1, sequence 96534
diag_adl:ARCH: End-Of-Redo Branch archival of thread 1 sequence 96534
diag_adl:ARCH: LGWR is scheduled to archive destination LOG_ARCHIVE_DEST_2 after log switch
diag_adl:ARCH: Standby redo logfile selected for thread 1 sequence 96534 for destination LOG_ARCHIVE_DEST_2
diag_adl:ARCH: Archiving is disabled due to current logfile archival
diag_adl:Primary will check for some target standby to have received all redo
diag_adl:Waiting for target standby to apply all redo
diag_adl:Backup controlfile written to trace file
/u01/app/oracle/diag/rdbms/xifenfeildg/xifenfei/trace/xifenfei_ora_37024.trc
diag_adl:Converting the primary database to a new standby database
diag_adl:Clearing standby activation ID 612004791 (0x247a73b7)
diag_adl:The primary database controlfile was created using the
diag_adl:'MAXLOGFILES 16' clause.
diag_adl:There is space for up to 11 standby redo logfiles
diag_adl:Use the following SQL commands on the standby database to create
diag_adl:standby redo logfiles that match the primary database:
diag_adl:ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 1073741824;
diag_adl:ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 1073741824;
diag_adl:ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 1073741824;
diag_adl:ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 1073741824;
diag_adl:ALTER DATABASE ADD STANDBY LOGFILE 'srl5.f' SIZE 1073741824;
diag_adl:ALTER DATABASE ADD STANDBY LOGFILE 'srl6.f' SIZE 1073741824;
diag_adl:Archivelog for thread 1 sequence 96534 required for standby recovery
diag_adl:Switchover: Primary controlfile converted to standby controlfile succesfully.
diag_adl:Switchover: Complete - Database shutdown required
diag_adl:Sending request(convert to primary database) to switchover target xifenfei
Sun Jun 25 09:07:58 2017
diag_adl:Switchover complete. Database shutdown required
diag_adl:USER (ospid: 37024): terminating the instance
Sun Jun 25 09:07:59 2017
diag_adl:Instance terminated by USER, pid = 37024
diag_adl:Shutting down instance (abort)
diag_adl:License high water mark = 527
Sun Jun 25 09:07:59 2017
Instance shutdown complete
备库alert日志
Sun Jun 25 09:05:54 2017
SWITCHOVER VERIFY BEGIN
SWITCHOVER VERIFY COMPLETE
Sun Jun 25 09:06:35 2017
RFS[107]: Assigned to RFS process (PID:7330)
RFS[107]: Selected log 12 for thread 1 sequence 96534 dbid 588725663 branch 916962073
Sat Jun 24 20:06:35 2017
Archived Log entry 100576 added for thread 1 sequence 96534 ID 0x247a73b7 dest 1:
Sat Jun 24 20:06:35 2017
Resetting standby activation ID 612004791 (0x247a73b7)
Sat Jun 24 20:06:35 2017
Media Recovery End-Of-Redo indicator encountered
Sat Jun 24 20:06:35 2017
Media Recovery Continuing
Media Recovery Waiting for thread 1 sequence 96535
Sun Jun 25 09:06:36 2017
SWITCHOVER: received request 'ALTER DTABASE COMMIT TO SWITCHOVER TO PRIMARY' from primary database.
Sun Jun 25 09:06:36 2017
ALTER DATABASE SWITCHOVER TO PRIMARY (xifenfei)
Maximum wait for role transition is 15 minutes.
Switchover: Media recovery is still active
Role Change: Canceling MRP - no more redo to apply
Sat Jun 24 20:06:36 2017
MRP0: Background Media Recovery cancelled with status 16037
Sat Jun 24 20:06:36 2017
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_pr00_4590.trc:
ORA-16037: user requested cancel of managed recovery operation
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Sat Jun 24 20:06:36 2017
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_pr00_4590.trc:
ORA-16037: user requested cancel of managed recovery operation
Sat Jun 24 20:06:37 2017
MRP0: Background Media Recovery process shutdown (xifenfei)
Sun Jun 25 09:06:38 2017
Role Change: Canceled MRP
Killing 2 processes(PIDS:7328,4704)(all RFS)
in order to disallow current and future RFS connections.Requested by OS process 7334
Stopping Emon pool
All dispatchers and shared servers shutdown
CLOSE: killing server sessions.
Active process 5428 user 'oracle' program 'oracle@kage7.hk0620.com (TNS V1-V3)'
Active process 5161 user 'oracle' program 'oracle@kage7.hk0620.com'
…………
Active process 5428 user 'oracle' program 'oracle@kage7.hk0620.com (TNS V1-V3)'
Active process 5161 user 'oracle' program 'oracle@kage7.hk0620.com'
Active process 5178 user 'oracle' program 'oracle@kage7.hk0620.com'
CLOSE: all sessions shutdown successfully.
Stopping Emon pool
Sat Jun 24 20:06:43 2017
SMON: disabling cache recovery
Sat Jun 24 20:06:44 2017
Buffer Cache Full DB Caching mode changing from FULL CACHING DISABLED to FULL CACHING ENABLED
Sun Jun 25 09:06:44 2017
Backup controlfile written to trace file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_rmi_7334.trc
SwitchOver after complete recovery through change 13163656450
Online logfile pre-clearing operation disabled by switchover
Online log /u01/app/oracle/oradata/xifenfei/redo01n.log: Thread 1 Group 1 was previously cleared
Online log /u01/app/oracle/fast_recovery_area/xifenfei/redo01n.log: Thread 1 Group 1 was previously cleared
Online log /u01/app/oracle/oradata/xifenfei/redo02n.log: Thread 1 Group 2 was previously cleared
Online log /u01/app/oracle/fast_recovery_area/xifenfei/redo02n.log: Thread 1 Group 2 was previously cleared
Online log /u01/app/oracle/oradata/xifenfei/redo03n.log: Thread 1 Group 3 was previously cleared
Online log /u01/app/oracle/fast_recovery_area/xifenfei/redo03n.log: Thread 1 Group 3 was previously cleared
Online log /u01/app/oracle/oradata/xifenfei/redo04n.log: Thread 1 Group 4 was previously cleared
Online log /u01/app/oracle/fast_recovery_area/xifenfei/redo04n.log: Thread 1 Group 4 was previously cleared
Online log /u01/app/oracle/oradata/xifenfei/redo05n.log: Thread 1 Group 5 was previously cleared
Online log /u01/app/oracle/fast_recovery_area/xifenfei/redo05n.log: Thread 1 Group 5 was previously cleared
Standby became primary SCN: 13163656448
Switchover: Complete - Database mounted as primary
SWITCHOVER: completed request from primary database.
Sat Jun 24 20:07:12 2017
ARC0: Becoming the 'no SRL' ARCH
原备库(现主库)操作
SQL> conn / as sysdba Connected. SQL> select database_role,open_mode from v$database; DATABASE_ROLE OPEN_MODE ---------------- -------------------- PRIMARY MOUNTED SQL> alter database open; Database altered.
原主库(现备库)操作
[oracle@localhost scripts]$ ss SQL*Plus: Release 12.1.0.2.0 Production on 星期日 6月 25 09:09:40 2017 Copyright (c) 1982, 2014, Oracle. All rights reserved. 已连接到空闲例程。 SQL> startup ORACLE 例程已经启动。 Total System Global Area 8.5899E+10 bytes Fixed Size 7654304 bytes Variable Size 1.2616E+10 bytes Database Buffers 7.3014E+10 bytes Redo Buffers 260780032 bytes 数据库装载完毕。 数据库已经打开。 SQL> alter database recover managed standby database disconnect; 数据库已更改。
oracle rac 12.2 执行root.sh报CLSRSC-400
在redhat 7.3版本中安装oracle rac 12.2的过程中,执行root.sh脚本的第14步的时候报如下错误,导致无法继续
CLSRSC-400: A system reboot is required to continue installing.
The command ‘/u01/app/grid/product/12.2.0/grid/perl/bin/perl -I/u01/app/grid/product/12.2.0/grid/perl/lib
-I/u01/app/grid/product/12.2.0/grid/crs/install /u01/app/grid/product/12.2.0/grid/crs/install/rootcrs.pl ‘ execution failed
os版本信息
[grid@xifenfei01 ~]$ more /etc/redhat-release Red Hat Enterprise Linux Server release 7.3 (Maipo) [grid@xifenfei01 ~]$ uname -a Linux xifenfei01 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
root.sh报错
[root@xifenfei01 ~]# /u01/app/grid/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/grid/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/grid/oraInventory to oinstall.
The execution of the script is complete.
[root@xifenfei01 ~]# /u01/app/grid/product/12.2.0/grid/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/grid/product/12.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/grid/product/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/grid_bash/crsdata/xifenfei01/crsconfig/rootcrs_xifenfei01_2017-06-11_09-52-55AM.log
2017/06/11 09:53:00 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2017/06/11 09:53:00 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/06/11 09:53:27 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/06/11 09:53:27 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2017/06/11 09:53:30 CLSRSC-363: User ignored prerequisites during installation
2017/06/11 09:53:30 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2017/06/11 09:53:31 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2017/06/11 09:53:32 CLSRSC-594: Executing installation step 5 of 19: 'SaveParamFile'.
2017/06/11 09:53:37 CLSRSC-594: Executing installation step 6 of 19: 'SetupOSD'.
2017/06/11 09:53:38 CLSRSC-594: Executing installation step 7 of 19: 'CheckCRSConfig'.
2017/06/11 09:53:38 CLSRSC-594: Executing installation step 8 of 19: 'SetupLocalGPNP'.
2017/06/11 09:53:51 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2017/06/11 09:53:56 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2017/06/11 09:53:56 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2017/06/11 09:54:00 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2017/06/11 09:54:15 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2017/06/11 09:54:44 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2017/06/11 09:54:48 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'xifenfei01'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'xifenfei01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/06/11 09:55:15 CLSRSC-400: A system reboot is required to continue installing.
The command '/u01/app/grid/product/12.2.0/grid/perl/bin/perl -I/u01/app/grid/product/12.2.0/grid/perl/lib
-I/u01/app/grid/product/12.2.0/grid/crs/install /u01/app/grid/product/12.2.0/grid/crs/install/rootcrs.pl'execution failed
主要报错信息：
2017/06/11 09:55:15 CLSRSC-400: A system reboot is required to continue installing.
The command ‘/u01/app/grid/product/12.2.0/grid/perl/bin/perl -I/u01/app/grid/product/12.2.0/grid/perl/lib -I/u01/app/grid/product/12.2.0/grid/crs/install /u01/app/grid/product/12.2.0/grid/crs/install/rootcrs.pl ‘ execution failed
查询mos发下:ACFS Drivers Install reports CLSRSC-400: A system reboot is required to continue installing (Doc ID 2025056.1),主要是由于12c gi开始，acfs默认是安装的，由于acfs在redhat 7.3中不支持导致上述的错误信息.
[grid@xifenfei01 ~]$ acfsdriverstate -orahome $ORACLE_HOME supported ACFS-9459: ADVM/ACFS is not supported on this OS version: '3.10.0-514.el7.x86_64' ACFS-9201: Not Supported
处理方法
停掉crs,kill 进程(如果有不能停掉的，通过kill处理),执行root.sh
[root@xifenfei01 ~]# /u01/app/grid/product/12.2.0/grid/bin/crsctl status res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 OFFLINE OFFLINE STABLE
ora.cluster_interconnect.haip
1 OFFLINE OFFLINE STABLE
ora.crf
1 OFFLINE OFFLINE STABLE
ora.crsd
1 OFFLINE OFFLINE STABLE
ora.cssd
1 OFFLINE OFFLINE STABLE
ora.cssdmonitor
1 OFFLINE OFFLINE STABLE
ora.ctssd
1 OFFLINE OFFLINE STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.drivers.acfs
1 OFFLINE OFFLINE STABLE
ora.evmd
1 OFFLINE OFFLINE STABLE
ora.gipcd
1 OFFLINE OFFLINE STABLE
ora.gpnpd
1 OFFLINE OFFLINE STABLE
ora.mdnsd
1 OFFLINE OFFLINE STABLE
ora.storage
1 OFFLINE OFFLINE STABLE
--------------------------------------------------------------------------------
[root@xifenfei01 ~]# /u01/app/grid/product/12.2.0/grid/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'xifenfei01'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'xifenfei02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@xifenfei02 ~]# ps -ef|grep d.bin
root 29155 11754 0 10:46 pts/0 00:00:00 grep --color=auto d.bin
[root@xifenfei01 ~]# /u01/app/grid/product/12.2.0/grid/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/grid/product/12.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/grid/product/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/grid_bash/crsdata/xifenfei01/crsconfig/rootcrs_xifenfei01_2017-06-11_10-33-57AM.log
2017/06/11 10:33:59 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2017/06/11 10:33:59 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/06/11 10:34:00 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/06/11 10:34:00 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2017/06/11 10:34:01 CLSRSC-363: User ignored prerequisites during installation
2017/06/11 10:34:01 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2017/06/11 10:34:02 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2017/06/11 10:34:02 CLSRSC-594: Executing installation step 5 of 19: 'SaveParamFile'.
2017/06/11 10:34:03 CLSRSC-594: Executing installation step 6 of 19: 'SetupOSD'.
2017/06/11 10:34:04 CLSRSC-594: Executing installation step 7 of 19: 'CheckCRSConfig'.
2017/06/11 10:34:04 CLSRSC-594: Executing installation step 8 of 19: 'SetupLocalGPNP'.
2017/06/11 10:34:06 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2017/06/11 10:34:06 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2017/06/11 10:34:53 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2017/06/11 10:34:54 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2017/06/11 10:35:09 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2017/06/11 10:35:31 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2017/06/11 10:35:33 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'xifenfei01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'xifenfei01'
CRS-2673: Attempting to stop 'ora.evmd' on 'xifenfei01'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'xifenfei01'
CRS-2677: Stop of 'ora.mdnsd' on 'xifenfei01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'xifenfei01' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'xifenfei01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'xifenfei01'
CRS-2677: Stop of 'ora.gipcd' on 'xifenfei01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'xifenfei01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/06/11 10:35:57 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2017/06/11 10:36:01 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'xifenfei01'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'xifenfei01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'xifenfei01'
CRS-2672: Attempting to start 'ora.mdnsd' on 'xifenfei01'
CRS-2676: Start of 'ora.mdnsd' on 'xifenfei01' succeeded
CRS-2676: Start of 'ora.evmd' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'xifenfei01'
CRS-2676: Start of 'ora.gpnpd' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'xifenfei01'
CRS-2672: Attempting to start 'ora.gipcd' on 'xifenfei01'
CRS-2676: Start of 'ora.cssdmonitor' on 'xifenfei01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'xifenfei01'
CRS-2672: Attempting to start 'ora.diskmon' on 'xifenfei01'
CRS-2676: Start of 'ora.diskmon' on 'xifenfei01' succeeded
CRS-2676: Start of 'ora.cssd' on 'xifenfei01' succeeded
Disk groups created successfully. Check /u01/app/grid/grid_bash/cfgtoollogs/asmca/asmca-170611AM103637.log for details.
2017/06/11 10:37:40 CLSRSC-482: Running command: '/u01/app/grid/product/12.2.0/grid/bin/ocrconfig -upgrade grid oinstall'
CRS-2672: Attempting to start 'ora.crf' on 'xifenfei01'
CRS-2672: Attempting to start 'ora.storage' on 'xifenfei01'
CRS-2676: Start of 'ora.storage' on 'xifenfei01' succeeded
CRS-2676: Start of 'ora.crf' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'xifenfei01'
CRS-2676: Start of 'ora.crsd' on 'xifenfei01' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 49af246c7d2e4f5dbf0d9ea09cc047d5.
Successfully replaced voting disk group with +DATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 49af246c7d2e4f5dbf0d9ea09cc047d5 (/dev/mapper/data1) [DATA]
Located 1 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'xifenfei01'
CRS-2673: Attempting to stop 'ora.crsd' on 'xifenfei01'
CRS-2677: Stop of 'ora.crsd' on 'xifenfei01' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'xifenfei01'
CRS-2673: Attempting to stop 'ora.crf' on 'xifenfei01'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'xifenfei01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'xifenfei01'
CRS-2677: Stop of 'ora.crf' on 'xifenfei01' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'xifenfei01' succeeded
CRS-2677: Stop of 'ora.storage' on 'xifenfei01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'xifenfei01'
CRS-2677: Stop of 'ora.mdnsd' on 'xifenfei01' succeeded
CRS-2677: Stop of 'ora.asm' on 'xifenfei01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'xifenfei01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'xifenfei01' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'xifenfei01'
CRS-2673: Attempting to stop 'ora.evmd' on 'xifenfei01'
CRS-2677: Stop of 'ora.ctssd' on 'xifenfei01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'xifenfei01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'xifenfei01'
CRS-2677: Stop of 'ora.cssd' on 'xifenfei01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'xifenfei01'
CRS-2677: Stop of 'ora.gipcd' on 'xifenfei01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'xifenfei01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2017/06/11 10:38:40 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'xifenfei01'
CRS-2672: Attempting to start 'ora.evmd' on 'xifenfei01'
CRS-2676: Start of 'ora.mdnsd' on 'xifenfei01' succeeded
CRS-2676: Start of 'ora.evmd' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'xifenfei01'
CRS-2676: Start of 'ora.gpnpd' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'xifenfei01'
CRS-2676: Start of 'ora.gipcd' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'xifenfei01'
CRS-2674: Start of 'ora.drivers.acfs' on 'xifenfei01' failed
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'xifenfei01'
CRS-2676: Start of 'ora.cssdmonitor' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'xifenfei01'
CRS-2672: Attempting to start 'ora.diskmon' on 'xifenfei01'
CRS-2676: Start of 'ora.diskmon' on 'xifenfei01' succeeded
CRS-2676: Start of 'ora.cssd' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'xifenfei01'
CRS-2672: Attempting to start 'ora.ctssd' on 'xifenfei01'
CRS-2676: Start of 'ora.ctssd' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'xifenfei01'
CRS-2674: Start of 'ora.drivers.acfs' on 'xifenfei01' failed
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'xifenfei01'
CRS-2676: Start of 'ora.asm' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'xifenfei01'
CRS-2676: Start of 'ora.storage' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'xifenfei01'
CRS-2676: Start of 'ora.crf' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'xifenfei01'
CRS-2676: Start of 'ora.crsd' on 'xifenfei01' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: xifenfei01
CRS-6016: Resource auto-start has completed for server xifenfei01
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2017/06/11 10:40:23 CLSRSC-343: Successfully started Oracle Clusterware stack
2017/06/11 10:40:23 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'xifenfei01'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'xifenfei01'
CRS-2676: Start of 'ora.asm' on 'xifenfei01' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'xifenfei01'
CRS-2676: Start of 'ora.DATA.dg' on 'xifenfei01' succeeded
2017/06/11 10:42:19 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2017/06/11 10:43:16 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
其他剩余节点也是类似处理,最终跳过acfs安装成功
[grid@xifenfei01 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE xifenfei01 STABLE
ONLINE ONLINE xifenfei02 STABLE
ora.DATA.dg
ONLINE ONLINE xifenfei01 STABLE
ONLINE ONLINE xifenfei02 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE xifenfei01 STABLE
ONLINE ONLINE xifenfei02 STABLE
ora.chad
ONLINE ONLINE xifenfei01 STABLE
ONLINE ONLINE xifenfei02 STABLE
ora.net1.network
ONLINE ONLINE xifenfei01 STABLE
ONLINE ONLINE xifenfei02 STABLE
ora.ons
ONLINE ONLINE xifenfei01 STABLE
ONLINE ONLINE xifenfei02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE xifenfei01 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE xifenfei01 169.254.20.214 192.1
68.1.20 192.168.2.20
,STABLE
ora.asm
1 ONLINE ONLINE xifenfei01 Started,STABLE
2 ONLINE ONLINE xifenfei02 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE xifenfei01 STABLE
ora.mgmtdb
1 ONLINE ONLINE xifenfei01 Open,STABLE
ora.qosmserver
1 ONLINE ONLINE xifenfei01 STABLE
ora.scan1.vip
1 ONLINE ONLINE xifenfei01 STABLE
ora.xifenfei01.vip
1 ONLINE ONLINE xifenfei01 STABLE
ora.xifenfei02.vip
1 ONLINE ONLINE xifenfei02 STABLE
--------------------------------------------------------------------------------
最新官方处理方案:CLSRSC-400: A system reboot is required to continue installing.