解决CON$ ORA-600 kdsgrp1错误

数据库报ORA 600 kdsgrp1错误
数据库报ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []错

Thread 1 advanced to log sequence 23861 (LGWR switch)
  Current log# 7 seq# 23861 mem# 0: /oradata/easdb/redo07.log
Tue Nov 15 10:00:42 2016
Errors in file /u01/oracle/diag/rdbms/easdb/easdb/trace/easdb_dw00_3165.trc  (incident=908262):
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle/diag/rdbms/easdb/easdb/incident/incdir_908262/easdb_dw00_3165_i908262.trc
Tue Nov 15 10:00:55 2016
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Tue Nov 15 10:00:56 2016
Errors in file /u01/oracle/diag/rdbms/easdb/easdb/trace/easdb_dw00_3165.trc  (incident=908263):
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPW$WORKER", line 1751
ORA-06512: at line 2
Incident details in: /u01/oracle/diag/rdbms/easdb/easdb/incident/incdir_908263/easdb_dw00_3165_i908263.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
DW00 terminating with fatal err=600, pid=40, wid=1, job SYSTEM.
Tue Nov 15 10:01:01 2016
Thread 1 advanced to log sequence 23862 (LGWR switch)
  Current log# 2 seq# 23862 mem# 0: /oradata/easdb/redo02.log
Tue Nov 15 10:01:23 2016
Errors in file /u01/oracle/diag/rdbms/easdb/easdb/trace/easdb_dm00_3163.trc  (incident=908254):
ORA-31671: Worker process DW00 had an unhandled exception.
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPW$WORKER", line 1751
ORA-06512: at line 2
Incident details in: /u01/oracle/diag/rdbms/easdb/easdb/incident/incdir_908254/easdb_dm00_3163_i908254.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Tue Nov 15 10:01:26 2016
Tue Nov 15 10:01:28 2016
Thread 1 advanced to log sequence 23863 (LGWR switch)
  Current log# 4 seq# 23863 mem# 0: /oradata/easdb/redo04.log

trace文件中信息

*** 2016-11-15 10:00:35.977
* kdsgrp1-1: *************************************************
            row 0x004459e6.26 continuation at
            0x004459e6.26 file# 1 block# 285158 slot 38 not found
KDSTABN_GET: 0 ..... ntab: 1
curSlot: 38 ..... nrows: 208
kdsgrp - dump CR block dba=0x004459e6
Block header dump:  0x004459e6
 Object id on Block? Y
 seg/obj: 0x1c  csc: 0x01.c712f743  itc: 3  flg: -  typ: 1 - DATA
     fsl: 0  fnx: 0x0 ver: 0x01
 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x000b.015.0036d715  0x00c01bba.0fbd.02  C---    0  scn 0x0001.c6b4cb1a
0x02   0x000c.004.00044d36  0x04c0dd93.3eec.33  C---    0  scn 0x0001.c6d2c65b
0x03   0x000d.008.00008eb9  0x04c0777a.10e3.02  --U-    2  fsc 0x0056.c7346f21

确定报错对象和确认异常

SQL> select object_name from dba_objects where object_id=28;
OBJECT_NAME
---------------------------------------------------------
CON$
SQL> ANALYZE TABLE sys.CON$ VALIDATE STRUCTURE CASCADE online;
ANALYZE TABLE sys.CON$ VALIDATE STRUCTURE CASCADE online
*
ERROR at line 1:
ORA-01499: table/index cross reference failure - see trace file
SQL> SET LINES 122
SQL> COL INDEX_OWNER FOR A20
SQL> COL INDEX_NAME FOR A30
SQL> COL TABLE_OWNER FOR A20
SQL> COL COLUMN_NAME FOR A25
SQL> SELECT TABLE_OWNER,INDEX_NAME,COLUMN_NAME,COLUMN_POSITION
2  FROM Dba_Ind_Columns
3  WHERE table_name = upper('&TABLE_NAME') order by TABLE_OWNER,INDEX_OWNER,INDEX_NAME,COLUMN_POSITION;
Enter value for table_name: CON$
old   3:  WHERE table_name = upper('&TABLE_NAME') order by TABLE_OWNER,INDEX_OWNER,INDEX_NAME,COLUMN_POSITION
new   3:  WHERE table_name = upper('CON$') order by TABLE_OWNER,INDEX_OWNER,INDEX_NAME,COLUMN_POSITION
TABLE_OWNER	     INDEX_NAME 		    COLUMN_NAME 	      COLUMN_POSITION
-------------------- ------------------------------ ------------------------- ---------------
SYS		     I_CON1			    OWNER#				    1
SYS		     I_CON1			    NAME				    2
SYS		     I_CON2			    CON#				    1
SQL> select owner#,name from con$
2    minus
3   select /*+ full(t) */owner#,name from con$ t;
no rows selected
SQL> select /*+ full(t) */owner#,name from con$ t
2    minus
3   select owner#,name from con$  ;
no rows selected
SQL> select /*+ full(t) */ con# from con$ t
2    minus
3   select con# from con$ ;
no rows selected
SQL> select con# from con$
2    minus
3   select /*+ full(t) */ con# from con$ t   ;
      CON#
----------
   1037224
   1037225
   1037386
   1037387
   1037388
   ……
   1037846
62 rows selected.

通过上述分析,可以确定是由于CON$和I_CON2数据不一致,而且是index的数据比表中多了62条.针对这样情况,考虑通过重建index来解决.

尝试rebuild index

SQL> alter index I_CON2 rebuild online;
alter index I_CON2 rebuild online
*
ERROR at line 1:
ORA-00701: object necessary for warmstarting database cannot be altered
SQL>
SQL>
SQL>
SQL>
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup upgrade
ORACLE instance started.
Total System Global Area 2421825536 bytes
Fixed Size                  2215744 bytes
Variable Size            1828716736 bytes
Database Buffers          570425344 bytes
Redo Buffers               20467712 bytes
Database mounted.
Database opened.
SQL> alter index I_CON2 rebuild;
alter index I_CON2 rebuild
*
ERROR at line 1:
ORA-00701: object necessary for warmstarting database cannot be altered

因为是数据库核心index,无法直接rebuild解决,只能通过bootstrap$核心index(I_OBJ1,I_USER1,I_FILE#_BLOCK#,I_IND1,I_TS#,I_CDEF1等)异常恢复—ORA-00701错误解决 方式解决

plsql dev引起的数据库被黑勒索比特币实现原理分析和解决方案

数据库启动alert报错

Mon Jul 10 19:51:24 2016
Errors in file e:\app\administrator\diag\rdbms\zhxh\zhxh\trace\zhxh_ora_3584.trc:
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-20313: 你的数据库已被SQL RUSH Team锁死  发送5个比特币到这个地址 166xk1FXMB2g8JxBVF5T4Aw1Z5JaZ6vrSE (大小写一致)  之后把你的Oracle SID邮寄地址 sqlrush@mail.com 我们将让你知道如何解锁你的数据库  Hi buddy, your database was hacked by SQL RUSH Team, send 5 bitcoin to address 166xk1FXMB2g8JxBVF5T4Aw1Z5JaZ6vrSE (case sensitive),  after that send your Oracle SID to mail address sqlrush@mail.com, we will let you know how to unlock your database.
ORA-06512: 在 "ZHXH.DBMS_SYSTEM_INTERNAL         ", line 15
ORA-06512: 在 line 2
Mon Jul 10 19:51:30 2016
OER 7451 in Load Indicator : Error Code = OSD-04500: 指定了非法选项
O/S-Error: (OS 1) 函数不正确。 !
Mon Jul 10 19:51:34 2017
Errors in file e:\app\administrator\diag\rdbms\zhxh\zhxh\trace\zhxh_ora_824.trc:
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-20313: 你的数据库已被SQL RUSH Team锁死  发送5个比特币到这个地址 166xk1FXMB2g8JxBVF5T4Aw1Z5JaZ6vrSE (大小写一致)  之后把你的Oracle SID邮寄地址 sqlrush@mail.com 我们将让你知道如何解锁你的数据库  Hi buddy, your database was hacked by SQL RUSH Team, send 5 bitcoin to address 166xk1FXMB2g8JxBVF5T4Aw1Z5JaZ6vrSE (case sensitive),  after that send your Oracle SID to mail address sqlrush@mail.com, we will let you know how to unlock your database.
ORA-06512: 在 "ZHXH.DBMS_SYSTEM_INTERNAL         ", line 15
ORA-06512: 在 line 2

afterconnect.sql
是plsql dev登录后自动执行脚本,非Oralce官方脚本
1


数据库启动后执行触发器DBMS_SUPPORT_INTERNAL
2
3


DBMS_SUPPORT_INTERNAL主要的意义是:
1. 当数据库创建时间大于1200天之后,开始备份tab$表
2. 删除tab$中除掉owner#为0和38的记录(sys,xdb)
4


3. 通过SYS.DBMS_BACKUP_RESTORE.RESETCFILESECTION清理掉备份信息(v$controlfile_record_section)
4. 然后通过DBMS_SYSTEM.KSDWRT在你的alert日志中写上2046次的提示信息
Hi buddy, your database was hacked by SQL RUSH Team, send 5 bitcoin to address 166xk1FXMB2g8JxBVF5T4Aw1Z5JaZ6vrSE (case sensitive), after that send your Oracle SID to mail address sqlrush@mail.com, we will let you know how to unlock your database.
你的数据库已被SQL RUSH Team锁死 发送5个比特币到这个地址 166xk1FXMB2g8JxBVF5T4Aw1Z5JaZ6vrSE (大小写一致) 之后把你的Oracle SID邮寄地址 sqlrush@mail.com 我们将让你知道如何解锁你的数据库
5. 再抛出一个前台的和4类似的警告信息
数据库登录触发器DBMS_SYSTEM_INTERNAL
6
7


当你的非SYSTEM,SYSAUX,EXAMPLE之外的所有表的最小统计信息时间大于1200天,而且非C89239.EXE程序,就会报出来” 你的数据库已被SQL RUSH Team锁死 发送5个比特币到这个地址 166xk1FXMB2g8JxBVF5T4Aw1Z5JaZ6vrSE (大小写一致) 之后把你的Oracle SID邮寄地址 sqlrush@mail.com 我们将让你知道如何解锁你的数据库 Hi buddy, your database was hacked by SQL RUSH Team, send 5 bitcoin to address 166xk1FXMB2g8JxBVF5T4Aw1Z5JaZ6vrSE
(case sensitive), after that send your Oracle SID to mail address sqlrush@mail.com, we will let you know how to unlock your database.”的信息
数据库登录触发器DBMS_CORE_INTERNAL
8
9
5


这里比较明显,把表名不含$,不含ORACHK,不是cluster的表放到一个游标里面,然后取非SYSTEM,SYSAUX,EXAMPLE之外的表空间的表的最小统计信息收集时间和当前时间比较如果大于1200天就执行truncate table操作,操作完成之后判断如果登录程序不为C89239.EXE,则报出来异常,” 你的数据库已被SQL RUSH Team锁死 发送5个比特币到这个地址 166xk1FXMB2g8JxBVF5T4Aw1Z5JaZ6vrSE (大小写一致) 之后把你的Oracle SID邮寄地址 sqlrush@mail.com 我们将让你知道如何解锁你的数据库 Hi buddy, your database was hacked by SQL RUSH Team, send 5 bitcoin to address 166xk1FXMB2g8JxBVF5T4Aw1Z5JaZ6vrSE
(case sensitive), after that send your Oracle SID to mail address sqlrush@mail.com, we will let you know how to unlock your database.”。
对于这次故障处理方法
1. 如果SELECT NVL(TO_CHAR(SYSDATE-MIN(LAST_ANALYZED)),0) FROM ALL_TABLES WHERE TABLESPACE_NAME NOT IN (‘SYSTEM’,’SYSAUX’,’EXAMPLE’); 小于1200,查询下列语句,然后删除掉(正常库查询为空)
2. 如果SYSDATE-MIN(LAST_ANALYZED)大于1200, SYSDATE-CREATED大于1200天未重启,或者SYSDATE-CREATED小于1200;就是tab$还未被清理,但是表被truncate,这样情况可以通过oracle原厂dul工具恢复
3. 如果SYSDATE-CREATED大于1200天,而且数据库重启过,但是SYSDATE-MIN(LAST_ANALYZED)小于1200天,那可以直接通过把ORACHK’||SUBSTR(SYS_GUID,10)中备份信息插入到$tab中
4. SYSDATE-CREATED大于1200天,而且数据库重启过,但是SYSDATE-MIN(LAST_ANALYZED)大于1200天,Oracle 原厂dul之类工具结合ORACHK’||SUBSTR(SYS_GUID,10)备份表中数据进行恢复
dul


预防策略
1)数据库里面查询下,如果有这些对象,及时给与清理(注意% ‘中间有空格)

select 'DROP TRIGGER '||owner||'."'||TRIGGER_NAME||'";' from dba_triggers
where TRIGGER_NAME like  'DBMS_%_INTERNAL% '
union all
select 'DROP PROCEDURE '||owner||'."'||a.object_name||'";' from dba_procedures a
where a.object_name like 'DBMS_%_INTERNAL% ';
--注意% '之间的空格

2)建议业务用户尽量限制dba 权限
3)检查相关登陆工具的自动运行脚本 清理掉有风险脚本
sqlplus中的glogin.sql/login.sql
toad中的toad.ini
plsql dev中的login.sql/afterconnect.sql
4)建议从官方下载工具,不要使用绿色版/破解版等
如果不幸数据库被感染此种勒索比特币事件,而且无法自行恢复的,可以联系我们给予技术支持
Phone:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com

ext3/ext4 超级块修复

创建ext4文件系统

[root@localhost ~]# mkfs.ext4 /dev/sdb1
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1310720 inodes, 5242624 blocks
262131 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2153775104
160 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
[root@localhost ~]# mkdir /sdb
[root@localhost ~]# mount /dev/sdb1 /sdb
[root@localhost ~]# df -h
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/ol-root   36G  4.0G   32G  12% /
devtmpfs             1.8G     0  1.8G   0% /dev
tmpfs                1.8G     0  1.8G   0% /dev/shm
tmpfs                1.8G  8.9M  1.8G   1% /run
tmpfs                1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/sda1            497M  195M  303M  40% /boot
tmpfs                369M     0  369M   0% /run/user/0
/dev/sdb1             20G   45M   19G   1% /sdb

准备测试数据

[root@localhost sdb]# cd /etc/sysctl.d/
[root@localhost sysctl.d]# ls
99-sysctl.conf
[root@localhost sysctl.d]# cp 99-sysctl.conf /sdb
[root@localhost sysctl.d]# more 99-sysctl.conf
# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

破坏ext4文件系统

[root@localhost ~]#  dd if=/dev/zero of=/dev/sdb1 bs=1024 count=5
5+0 records in
5+0 records out
5120 bytes (5.1 kB) copied, 0.00270838 s, 1.9 MB/s
[root@localhost ~]# mount /dev/sdb1 /sdb
mount: unknown filesystem type '(null)'

日志报错

[ 8868.362628] sd 32:0:1:0: [sdb] Cache data unavailable
[ 8868.362632] sd 32:0:1:0: [sdb] Assuming drive cache: write through
[ 8868.363714]  sdb: sdb1
[ 8868.390297] sd 32:0:1:0: [sdb] Cache data unavailable
[ 8868.390301] sd 32:0:1:0: [sdb] Assuming drive cache: write through
[ 8868.391462]  sdb: sdb1
[ 8900.130143] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)
[ 8900.130163] SELinux: initialized (dev sdb1, type ext4), uses xattr
[ 8902.803966] sdb1: WRITE SAME failed. Manually zeroing.

fsck修复

[root@localhost ~]# fsck -t ext4 /dev/sdb1
fsck from util-linux 2.23.2
e2fsck 1.42.9 (28-Dec-2013)
ext2fs_open2: Bad magic number in super-block
fsck.ext4: Superblock invalid, trying backup blocks...
/dev/sdb1 was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group #1 (31740, counted=31739).
Fix<y>? yes
Free blocks count wrong (5116302, counted=5116301).
Fix<y>? yes
Free inodes count wrong for group #0 (8181, counted=8180).
Fix<y>? yes
Free inodes count wrong (1310709, counted=1310708).
Fix<y>? yes
/dev/sdb1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdb1: 12/1310720 files (0.0% non-contiguous), 126323/5242624 blocks

测试修复结果

[root@localhost ~]#
[root@localhost ~]#
[root@localhost ~]# mount /dev/sdb1 /sdb
[root@localhost ~]# df -h
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/ol-root   36G  4.0G   32G  12% /
devtmpfs             1.8G     0  1.8G   0% /dev
tmpfs                1.8G     0  1.8G   0% /dev/shm
tmpfs                1.8G  8.9M  1.8G   1% /run
tmpfs                1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/sda1            497M  195M  303M  40% /boot
tmpfs                369M     0  369M   0% /run/user/0
/dev/sdb1             20G   45M   19G   1% /sdb
[root@localhost ~]# cd /sdb
[root@localhost sdb]# ls
99-sysctl.conf  lost+found
[root@localhost sdb]# more 99-sysctl.conf
# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

ext4文件系统修复

[root@localhost ~]# mkfs.ext3 /dev/sdb1
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1310720 inodes, 5242624 blocks
262131 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
160 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
[root@localhost ~]# mount /dev/sdb1 /sdb
[root@localhost ~]# df -h
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/ol-root   36G  4.0G   32G  12% /
devtmpfs             1.8G     0  1.8G   0% /dev
tmpfs                1.8G     0  1.8G   0% /dev/shm
tmpfs                1.8G  8.9M  1.8G   1% /run
tmpfs                1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/sda1            497M  195M  303M  40% /boot
tmpfs                369M     0  369M   0% /run/user/0
/dev/sdb1             20G   45M   19G   1% /sdb
[root@localhost ~]# dd if=/dev/zero of=/dev/sdb1 bs=1024 count=5
5+0 records in
5+0 records out
5120 bytes (5.1 kB) copied, 0.0138915 s, 369 kB/s
[root@localhost ~]# fsck -t ext3 /dev/sdb1
fsck from util-linux 2.23.2
e2fsck 1.42.9 (28-Dec-2013)
ext2fs_open2: Bad magic number in super-block
fsck.ext3: Superblock invalid, trying backup blocks...
/dev/sdb1 was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdb1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdb1: 11/1310720 files (0.0% non-contiguous), 126322/5242624 blocks
[root@localhost ~]# mount /dev/sdb1 /sdb
[root@localhost ~]# df -h
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/ol-root   36G  4.0G   32G  12% /
devtmpfs             1.8G     0  1.8G   0% /dev
tmpfs                1.8G     0  1.8G   0% /dev/shm
tmpfs                1.8G  8.9M  1.8G   1% /run
tmpfs                1.8G     0  1.8G   0% /sys/fs/cgroup
/dev/sda1            497M  195M  303M  40% /boot
tmpfs                369M     0  369M   0% /run/user/0
/dev/sdb1             20G   45M   19G   1% /sdb

fsck修复危险性较大,建议先备份对应的分区(dd命令备份分区)然后再处理,有导致分区数据全部或者部分丢失的风险,如果超级块彻底损坏无法恢复,请联系我们,提供专业ORACLE数据库恢复技术支持
Phone:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com

File #N is offline, but is part of an online tablespace

在看一个客户的数据库恢复日志的时候发现类似警告(File #N is offline, but is part of an online tablespace.),以前没有注意,这次通过试验来重现该部分内容
创建表空间

SQL> create tablespace readonly datafile '/home/oracle/.oradata/test/readonly01.dbf' size 128M;
Tablespace created.
SQL> alter tablespace  readonly add datafile '/home/oracle/.oradata/test/readonly02.dbf' size 128M;
Tablespace altered.

写入数据

SQL> create table t_readonly tablespace readonly as select * from dba_objects;
Table created.

read only 表空间

SQL> select count(*) from t_readonly;
  COUNT(*)
----------
     72226
SQL> alter system switch logfile;
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL>  alter tablespace readonly  read only;
Tablespace altered.
SQL> alter system switch logfile;
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL> /
System altered.

备份数据库

[oracle@localhost ~]$ rman target /
Recovery Manager: Release 11.2.0.1.0 - Production on Tue Nov 1 21:15:51 2016
Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.
connected to target database: TEST (DBID=2210907828)
RMAN> backup database format '/home/oracle/full_%U.rman';
Starting backup at 01-NOV-16
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=197 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=/home/oracle/.oradata/test/system01.dbf
input datafile file number=00002 name=/home/oracle/.oradata/test/sysaux01.dbf
input datafile file number=00005 name=/home/oracle/.oradata/test/test01.dbf
input datafile file number=00006 name=/home/oracle/.oradata/test/test02.dbf
input datafile file number=00007 name=/home/oracle/.oradata/test/readonly01.dbf
input datafile file number=00008 name=/home/oracle/.oradata/test/readonly02.dbf
input datafile file number=00003 name=/home/oracle/.oradata/test/undotbs01.dbf
input datafile file number=00004 name=/home/oracle/.oradata/test/users01.dbf
channel ORA_DISK_1: starting piece 1 at 01-NOV-16
channel ORA_DISK_1: finished piece 1 at 01-NOV-16
piece handle=/home/oracle/full_03rjrp0t_1_1.rman tag=TAG20161101T211613 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:25
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 01-NOV-16
channel ORA_DISK_1: finished piece 1 at 01-NOV-16
piece handle=/home/oracle/full_04rjrp1m_1_1.rman tag=TAG20161101T211613 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 01-NOV-16
RMAN> sql 'alter system archive log current';
backup  as compressed backupset archivelog  all format '/home/oracle/arch_%T_%U.rman'  delete input;
sql statement: alter system archive log current
RMAN>
Starting backup at 01-NOV-16
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting compressed archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=494 RECID=1 STAMP=926802386
input archived log thread=1 sequence=495 RECID=2 STAMP=926802386
input archived log thread=1 sequence=496 RECID=3 STAMP=926802389
input archived log thread=1 sequence=497 RECID=4 STAMP=926802693
input archived log thread=1 sequence=498 RECID=5 STAMP=926802693
input archived log thread=1 sequence=499 RECID=6 STAMP=926802696
input archived log thread=1 sequence=500 RECID=7 STAMP=926802787
input archived log thread=1 sequence=501 RECID=8 STAMP=926802789
input archived log thread=1 sequence=502 RECID=9 STAMP=926802792
input archived log thread=1 sequence=503 RECID=10 STAMP=926802793
input archived log thread=1 sequence=504 RECID=11 STAMP=926802812
input archived log thread=1 sequence=505 RECID=12 STAMP=926802813
input archived log thread=1 sequence=506 RECID=13 STAMP=926802816
input archived log thread=1 sequence=507 RECID=14 STAMP=926803076
input archived log thread=1 sequence=508 RECID=15 STAMP=926803077
channel ORA_DISK_1: starting piece 1 at 01-NOV-16
channel ORA_DISK_1: finished piece 1 at 01-NOV-16
piece handle=/home/oracle/arch_20161101_05rjrp45_1_1.rman tag=TAG20161101T211757 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
channel ORA_DISK_1: deleting archived log(s)
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_494_d1k4tkot_.arc RECID=1 STAMP=926802386
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_495_d1k4tln7_.arc RECID=2 STAMP=926802386
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_496_d1k4tot5_.arc RECID=3 STAMP=926802389
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_497_d1k544w3_.arc RECID=4 STAMP=926802693
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_498_d1k545wc_.arc RECID=5 STAMP=926802693
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_499_d1k548bm_.arc RECID=6 STAMP=926802696
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_500_d1k5734v_.arc RECID=7 STAMP=926802787
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_501_d1k5752s_.arc RECID=8 STAMP=926802789
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_502_d1k578c2_.arc RECID=9 STAMP=926802792
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_503_d1k579hy_.arc RECID=10 STAMP=926802793
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_504_d1k57w6s_.arc RECID=11 STAMP=926802812
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_505_d1k57xj1_.arc RECID=12 STAMP=926802813
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_506_d1k580hj_.arc RECID=13 STAMP=926802816
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_507_d1k5j4q0_.arc RECID=14 STAMP=926803076
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_508_d1k5j4yq_.arc RECID=15 STAMP=926803077
Finished backup at 01-NOV-16
RMAN> backup  format '/home/oracle/ctl_%T_%U.rman' current controlfile;
Starting backup at 01-NOV-16
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 01-NOV-16
channel ORA_DISK_1: finished piece 1 at 01-NOV-16
piece handle=/home/oracle/ctl_20161101_06rjrp75_1_1.rman tag=TAG20161101T211933 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 01-NOV-16

清理环境还原数据库

RMAN> shutdown immediate;
database closed
database dismounted
Oracle instance shut down
RMAN> exit
Recovery Manager complete.
[oracle@localhost .oradata]$ mv test  test_20161101
[oracle@localhost .oradata]$ mkdir test

还原数据库

[oracle@localhost ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Tue Nov 1 21:21:09 2016
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup nomount;
ORACLE instance started.
Total System Global Area 2421825536 bytes
Fixed Size                  2215744 bytes
Variable Size            1795162304 bytes
Database Buffers          603979776 bytes
Redo Buffers               20467712 bytes
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@localhost ~]$ rman target /
Recovery Manager: Release 11.2.0.1.0 - Production on Tue Nov 1 21:21:22 2016
Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.
connected to target database: TEST (not mounted)
RMAN> restore controlfile from '/home/oracle/ctl_20161101_06rjrp75_1_1.rman';
Starting restore at 01-NOV-16
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=63 device type=DISK
channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
output file name=/home/oracle/.oradata/test/control01.ctl
Finished restore at 01-NOV-16
RMAN> alter database mount;
database mounted
released channel: ORA_DISK_1
RMAN> restore database;
Starting restore at 01-NOV-16
Starting implicit crosscheck backup at 01-NOV-16
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=63 device type=DISK
Crosschecked 3 objects
Finished implicit crosscheck backup at 01-NOV-16
Starting implicit crosscheck copy at 01-NOV-16
using channel ORA_DISK_1
Finished implicit crosscheck copy at 01-NOV-16
searching for all files in the recovery area
cataloging files...
no files cataloged
using channel ORA_DISK_1
datafile 5 not processed because file is offline
datafile 6 not processed because file is offline
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /home/oracle/.oradata/test/system01.dbf
channel ORA_DISK_1: restoring datafile 00002 to /home/oracle/.oradata/test/sysaux01.dbf
channel ORA_DISK_1: restoring datafile 00003 to /home/oracle/.oradata/test/undotbs01.dbf
channel ORA_DISK_1: restoring datafile 00004 to /home/oracle/.oradata/test/users01.dbf
channel ORA_DISK_1: restoring datafile 00007 to /home/oracle/.oradata/test/readonly01.dbf
channel ORA_DISK_1: restoring datafile 00008 to /home/oracle/.oradata/test/readonly02.dbf
channel ORA_DISK_1: reading from backup piece /home/oracle/full_03rjrp0t_1_1.rman
channel ORA_DISK_1: piece handle=/home/oracle/full_03rjrp0t_1_1.rman tag=TAG20161101T211613
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:15
Finished restore at 01-NOV-16
RMAN> exit
Recovery Manager complete.

检查数据库文件
read_only1


通过Oracle Database Recovery Check很明显发现,这里看到文件状态是read only的.
recover database

[oracle@localhost ~]$ rman target /
Recovery Manager: Release 11.2.0.1.0 - Production on Tue Nov 1 21:28:14 2016
Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.
connected to target database: TEST (DBID=2210907828, not open)
RMAN> recover database;
Starting recover at 01-NOV-16
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=63 device type=DISK
datafile 7 not processed because file is read-only    <<<<=====注意
datafile 8 not processed because file is read-only    <<<<=====注意
starting media recovery
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=507
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=508
channel ORA_DISK_1: reading from backup piece /home/oracle/arch_20161101_05rjrp45_1_1.rman
channel ORA_DISK_1: piece handle=/home/oracle/arch_20161101_05rjrp45_1_1.rman tag=TAG20161101T211757
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_507_d1k63pj8_.arc thread=1 sequence=507
channel default: deleting archived log(s)
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_507_d1k63pj8_.arc RECID=16 STAMP=926803702
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_508_d1k63pww_.arc thread=1 sequence=508
channel default: deleting archived log(s)
archived log file name=/opt/oracle/flash_recovery_area/TEST/archivelog/2016_11_01/o1_mf_1_508_d1k63pww_.arc RECID=17 STAMP=926803702
unable to find archived log
archived log thread=1 sequence=509
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 11/01/2016 21:28:24
RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 509 and starting SCN of 8933048
RMAN> exit

resetlogs 打开数据库

SQL> set numw 16
SQL> SELECT status,
checkpoint_change#,
  2    3  checkpoint_time,last_change#,
count(*) ROW_NUM
FROM v$datafile
GROUP BY status, checkpoint_change#, checkpoint_time,last_change#
ORDER BY status, checkpoint_change#, checkpoint_time;  4    5    6    7
STATUS  CHECKPOINT_CHANGE# CHECKPOIN     LAST_CHANGE#          ROW_NUM
------- ------------------ --------- ---------------- ----------------
ONLINE             8932792 01-NOV-16                                 2
ONLINE             8933048 01-NOV-16                                 3
SYSTEM             8933048 01-NOV-16                                 1
SQL> SELECT status,
  2  checkpoint_change#,
  3  checkpoint_time,FUZZY,
  4  count(*) ROW_NUM
  5  FROM v$datafile_header
GROUP BY status, checkpoint_change#, checkpoint_time,fuzzy
ORDER BY status, checkpoint_change#, checkpoint_time;
  6    7
STATUS  CHECKPOINT_CHANGE# CHECKPOIN FUZ          ROW_NUM
------- ------------------ --------- --- ----------------
ONLINE             8932792 01-NOV-16 NO                 2
ONLINE             8933048 01-NOV-16 NO                 4
SQL> alter database open resetlogs;
Database altered.

alert日志信息

Tue Nov 01 21:29:56 2016
alter database open resetlogs
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_ora_14479.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '/home/oracle/.oradata/test/redo01.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_ora_14479.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '/home/oracle/.oradata/test/redo01.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_ora_14479.trc:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/home/oracle/.oradata/test/redo02.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_ora_14479.trc:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/home/oracle/.oradata/test/redo02.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_ora_14479.trc:
ORA-00313: open failed for members of log group 3 of thread 1
ORA-00312: online log 3 thread 1: '/home/oracle/.oradata/test/redo03.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_ora_14479.trc:
ORA-00313: open failed for members of log group 3 of thread 1
ORA-00312: online log 3 thread 1: '/home/oracle/.oradata/test/redo03.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
RESETLOGS after incomplete recovery UNTIL CHANGE 8933048
Resetting resetlogs activation ID 2210869684 (0x83c731b4)
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_ora_14479.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '/home/oracle/.oradata/test/redo01.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_ora_14479.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '/home/oracle/.oradata/test/redo01.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_ora_14479.trc:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/home/oracle/.oradata/test/redo02.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_ora_14479.trc:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/home/oracle/.oradata/test/redo02.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_ora_14479.trc:
ORA-00313: open failed for members of log group 3 of thread 1
ORA-00312: online log 3 thread 1: '/home/oracle/.oradata/test/redo03.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_ora_14479.trc:
ORA-00313: open failed for members of log group 3 of thread 1
ORA-00312: online log 3 thread 1: '/home/oracle/.oradata/test/redo03.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Tue Nov 01 21:29:58 2016
Setting recovery target incarnation to 3
Tue Nov 01 21:29:58 2016
Assigning activation ID 2224900353 (0x849d4901)
LGWR: STARTING ARCH PROCESSES
Tue Nov 01 21:29:58 2016
ARC0 started with pid=20, OS id=14486
ARC0: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Tue Nov 01 21:29:59 2016
ARC1 started with pid=21, OS id=14488
Tue Nov 01 21:29:59 2016
ARC2 started with pid=22, OS id=14490
ARC1: Archival started
Tue Nov 01 21:29:59 2016
ARC3 started with pid=23, OS id=14492
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /home/oracle/.oradata/test/redo01.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Nov 01 21:29:59 2016
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Dictionary check beginning
File #7 is offline, but is part of an online tablespace.
data file 7: '/home/oracle/.oradata/test/readonly01.dbf'
Successfuly brought file #7 online.
File #8 is offline, but is part of an online tablespace.
data file 8: '/home/oracle/.oradata/test/readonly02.dbf'
Successfuly brought file #8 online.
Tue Nov 01 21:29:59 2016
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_dbw0_14226.trc:
ORA-01157: cannot identify/lock data file 201 - see DBWR trace file
ORA-01110: data file 201: '/home/oracle/.oradata/test/temp01.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_dbw0_14226.trc:
ORA-01186: file 201 failed verification tests
ORA-01157: cannot identify/lock data file 201 - see DBWR trace file
ORA-01110: data file 201: '/home/oracle/.oradata/test/temp01.dbf'
File 201 not verified due to error ORA-01157
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Re-creating tempfile /home/oracle/.oradata/test/temp01.dbf
Database Characterset is AL32UTF8
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Tue Nov 01 21:30:00 2016
QMNC started with pid=24, OS id=14494
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Completed: alter database open resetlogs

这里看到了我们期待的警告
File #7 is offline, but is part of an online tablespace.
data file 7: ‘/home/oracle/.oradata/test/readonly01.dbf’
Successfuly brought file #7 online.
File #8 is offline, but is part of an online tablespace.
data file 8: ‘/home/oracle/.oradata/test/readonly02.dbf’
Successfuly brought file #8 online.
结论:如果数据库的表空间之前是read only,然后resetlogs操作就会有类似提示(File #N is offline, but is part of an online tablespace).这样的整个恢复过程不影响read only表空间中的数据

ORA-27300 ORA-27301 ORA-27302

有朋友库中alert日志报ORA-27300 ORA-27301 ORA-27302错误,让帮忙分析原因
ORA-27300 ORA-27301 ORA-27302

Fri Oct 28 22:15:33 2016
Process startup failed, error stack:
Errors in file d:\app\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_psp0_4856.trc:
ORA-27300: OS system dependent operation:CreateThread failed with status: 8
ORA-27301: OS failure message: 存储空间不足,无法处理此命令。
ORA-27302: failure occurred at: ssthrddcr
Fri Oct 28 22:15:34 2016
Process J001 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file d:\app\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_cjq0_5664.trc:
Fri Oct 28 22:16:35 2016
Process startup failed, error stack:
Errors in file d:\app\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_psp0_4856.trc:
ORA-27300: OS system dependent operation:CreateThread failed with status: 8
ORA-27301: OS failure message: 存储空间不足,无法处理此命令。
ORA-27302: failure occurred at: ssthrddcr
Fri Oct 28 22:16:36 2016
Process J001 died, see its trace file
kkjcre1p: unable to spawn jobq slave process
Errors in file d:\app\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_cjq0_5664.trc:
Fri Oct 28 22:17:36 2016
Process startup failed, error stack:
Errors in file d:\app\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_psp0_4856.trc:
ORA-27300: OS system dependent operation:CreateThread failed with status: 8
ORA-27301: OS failure message: 存储空间不足,无法处理此命令。
ORA-27302: failure occurred at: ssthrddcr

系统版本win 2008 32位
os


系统状态
os2


数据库版本
version


数据库内存配置
db-memory


新会话无法连接

28-10月-2016 22:46:59 * (CONNECT_DATA=(SID=xifenfei)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=Administrator))) * (ADDRESS=(PROTOCOL=tcp)(HOST=172.16.91.5)(PORT=54383)) * establish * xifenfei * 12518
TNS-12518: TNS: 监听程序无法分发客户机连接
 TNS-12560: TNS: 协议适配器错误
28-10月-2016 22:46:59 * (CONNECT_DATA=(SID=xifenfei)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=Administrator))) * (ADDRESS=(PROTOCOL=tcp)(HOST=172.16.91.5)(PORT=54384)) * establish * xifenfei * 12518
TNS-12518: TNS: 监听程序无法分发客户机连接
 TNS-12560: TNS: 协议适配器错误
28-10月-2016 22:46:59 * (CONNECT_DATA=(SID=xifenfei)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=Administrator))) * (ADDRESS=(PROTOCOL=tcp)(HOST=172.16.91.5)(PORT=54385)) * establish * xifenfei * 12518
TNS-12518: TNS: 监听程序无法分发客户机连接
 TNS-12560: TNS: 协议适配器错误
28-10月-2016 22:46:59 * (CONNECT_DATA=(SID=xifenfei)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=Administrator))) * (ADDRESS=(PROTOCOL=tcp)(HOST=172.16.91.5)(PORT=54386)) * establish * xifenfei * 12518
TNS-12518: TNS: 监听程序无法分发客户机连接
 TNS-12560: TNS: 协议适配器错误
28-10月-2016 22:46:59 * (CONNECT_DATA=(SID=xifenfei)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=Administrator))) * (ADDRESS=(PROTOCOL=tcp)(HOST=172.16.91.5)(PORT=54388)) * establish * xifenfei * 12518
TNS-12518: TNS: 监听程序无法分发客户机连接
 TNS-12560: TNS: 协议适配器错误
28-10月-2016 22:46:59 * (CONNECT_DATA=(SID=xifenfei)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=Administrator))) * (ADDRESS=(PROTOCOL=tcp)(HOST=172.16.91.5)(PORT=54389)) * establish * xifenfei * 12518
TNS-12518: TNS: 监听程序无法分发客户机连接
 TNS-12560: TNS: 协议适配器错误
28-10月-2016 22:46:59 * (CONNECT_DATA=(SID=xifenfei)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=Administrator))) * (ADDRESS=(PROTOCOL=tcp)(HOST=172.16.91.5)(PORT=54390)) * establish * xifenfei * 12518
TNS-12518: TNS: 监听程序无法分发客户机连接
 TNS-12560: TNS: 协议适配器错误
28-10月-2016 22:47:00 * (CONNECT_DATA=(SID=xifenfei)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=Administrator))) * (ADDRESS=(PROTOCOL=tcp)(HOST=172.16.91.5)(PORT=54391)) * establish * xifenfei * 12518
TNS-12518: TNS: 监听程序无法分发客户机连接
 TNS-12560: TNS: 协议适配器错误

20161028224020


故障处理

HKEY_LOCAL_MACHINE SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters
Value Name: IRPStackSize
Data Type: DWORD
The default value of the IRPStackSize parameter is 15. The range is from 11 (0xb hexadecimal) through 50 (0x32 hexadecimal).

register


参考:Ora-27300: OS System Dependent Operation:Createthread Failed With Status: 8 (Doc ID 1310453.1)

ORA-600 kcrfr_update_nab_2 故障恢复

由于控制器掉线导致数据库启动报ORA-600 kcrfr_update_nab_2错误,导致无法正常open
数据库版本信息

ORACLE V10.2.0.4.0 - 64bit Production vsnsta=0
vsnsql=14 vsnxtr=3
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
Windows Server 2003 Version V5.2 Service Pack 2
CPU                 : 12 - type 8664, 2 Physical Cores
Process Affinity    : 0x0000000000000000
Memory (Avail/Total): Ph:22579M/32754M, Ph+PgF:24594M/33845M

ORA-600 kcrfr_update_nab_2报错

Mon Oct 24 17:42:57 2016
Database mounted in Exclusive Mode
Completed: ALTER DATABASE   MOUNT
Mon Oct 24 17:42:58 2016
ALTER DATABASE OPEN
Mon Oct 24 17:43:14 2016
Beginning crash recovery of 1 threads
 parallel recovery started with 11 processes
Mon Oct 24 17:43:14 2016
Started redo scan
Mon Oct 24 17:43:16 2016
Errors in file d:\oracle\product\10.2.0\admin\spcsjkdb\udump\spcsjkdb_ora_10108.trc:
ORA-00600: internal error code, arguments: [kcrfr_update_nab_2], [0x7FFC22A2150], [2], [], [], [], [], []
Mon Oct 24 17:43:18 2016
Aborting crash recovery due to error 600
Mon Oct 24 17:43:18 2016
Errors in file d:\oracle\product\10.2.0\admin\spcsjkdb\udump\spcsjkdb_ora_10108.trc:
ORA-00600: internal error code, arguments: [kcrfr_update_nab_2], [0x7FFC22A2150], [2], [], [], [], [], []
ORA-600 signalled during: ALTER DATABASE OPEN...

trace文件信息

*** 2016-10-24 17:43:14.515
*** ACTION NAME:() 2016-10-24 17:43:14.515
*** MODULE NAME:(sqlplus.exe) 2016-10-24 17:43:14.515
*** SERVICE NAME:() 2016-10-24 17:43:14.515
*** SESSION ID:(356.3) 2016-10-24 17:43:14.515
Successfully allocated 11 recovery slaves
Using 101 overflow buffers per recovery slave
Thread 1 checkpoint: logseq 33251, block 2, scn 14624215134369
  cache-low rba: logseq 33251, block 2463324
    on-disk rba: logseq 33251, block 2803965, scn 14624216078841
  start recovery at logseq 33251, block 2463324, scn 0
*** 2016-10-24 17:43:16.406
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [kcrfr_update_nab_2], [0x7FFC22A2150], [2], [], [], [], [], []
Current SQL statement for this session:
ALTER DATABASE OPEN
----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp+663           CALL???  ksedst+55            003C878B8 000000000 012B863E8
                                                   000000000
ksfdmp+19            CALL???  ksedmp+663           000000003 015572A70 007222698
                                                   003CACC80
kgerinv+158          CALL???  ksfdmp+19            015572430 000000000 0FFFFFFFF
                                                   000000000
kgeasnmierr+62       CALL???  kgerinv+158          000000000 000000000 000000000
                                                   004FD788F
kcrfr_update_nab+18  CALL???  kgeasnmierr+62       00BDA1170 000000000 000000000
6                                                  000000002
kcrfr_read+1078      CALL???  kcrfr_update_nab+18  007222698 00001E650 015572430
                              6                    0072229B8
kcrfrgv+8134         CALL???  kcrfr_read+1078      000000000 0051525D7 000000000
                                                   0051525D7
kcratr1+488          CALL???  kcrfrgv+8134         007222698 000000000 000000000
                                                   000000000
kcratr+412           CALL???  kcratr1+488          012B891C8 012B890A4 00727FFB8
                                                   00BEA7FF0
kctrec+1910          CALL???  kcratr+412           012B891C8 012B91E18 000000000
                                                   012B91E48
kcvcrv+3585          CALL???  kctrec+1910          012B92C58 000000000 00726DF00
                                                   00726BDB0
kcfopd+1007          CALL???  kcvcrv+3585          012B93350 000000000 000000000
                                                   000000000
adbdrv+55820         CALL???  kcfopd+1007          000000000 000000000 000000000
                                                   000000000
opiexe+13897         CALL???  adbdrv+55820         000000023 000000003 000000102
                                                   000000000
opiosq0+3558         CALL???  opiexe+13897         000000004 000000000 012B9B238
                                                   4155474E414C5F45
kpooprx+339          CALL???  opiosq0+3558         000000003 00000000E 012B9B3C8
                                                   0000000A4
kpoal8+894           CALL???  kpooprx+339          015587550 000000018 0041AE700
                                                   000000001
opiodr+1136          CALL???  kpoal8+894           00000005E 000000017 012B9E868
                                                   0072F5100
ttcpip+5146          CALL???  opiodr+1136          00000005E 000000017 012B9E868
                                                   2D8C00000000
opitsk+1818          CALL???  ttcpip+5146          015587550 000000000 000000000
                                                   000000000
opiino+1129          CALL???  opitsk+1818          00000001E 000000000 000000000
                                                   000000000
opiodr+1136          CALL???  opiino+1129          00000003C 000000004 012B9FB20
                                                   000000000
opidrv+815           CALL???  opiodr+1136          00000003C 000000004 012B9FB20
                                                   000000000
sou2o+52             CALL???  opidrv+815           00000003C 000000004 012B9FB20
                                                   7FF7FC48580
opimai_real+131      CALL???  sou2o+52             000000000 012B9FC40
                                                   7FFFFF7F258 077EF4D1C
opimai+96            CALL???  opimai_real+131      7FF7FC48580 7FFFFF7E000
                                                   0001F0003 000000000
OracleThreadStart+6  CALL???  opimai+96            012B9FEF0 01289FF3C 012B9FCC0
40                                                 7FF7FC48580
0000000077D6B6DA     CALL???  OracleThreadStart+6  01289FF3C 000000000 000000000
                              40                   012B9FFA8

官方描述
The assert ORA-600: [kcrfr_update_nab_2] is a direct result of a lost write in the current on line log that we are attempting to resolve.So, this confirms the theory that this is a OS/hardware lost write issue not an internal oracle bug. In fact the assert ORA-600: [kcrfr_update_nab_2] is how we detect a lost log write.
Bug 5692594
Hdr: 5692594 10.2.0.1 RDBMS 10.2.0.1 RECOVERY PRODID-5 PORTID-226 ORA-600
Abstract: AFTER DATABASE CRASHED DOESN’T OPEN ORA-600 [KCRFR_UPDATE_NAB_2]
Status: 95,Closed, Vendor OS Problem
Bug 6655116
Hdr: 6655116 10.2.0.3 RDBMS 10.2.0.3 RECOVERY PRODID-5 PORTID-23
Abstract: INSTANCES CRASH WITH ORA-600 [KCRFR_UPDATE_NAB_2] AFTER DISK FAILURE
根据官方的描述,结合故障情况,基本上可以确定是由于硬件异常导致Oracle写丢失,从而除非oracle相关bug导致数据库无法正常启动

ORA-600 [kcrfr_update_nab_2] [a] [b]
VERSIONS:
versions 10.2 to 11.1
DESCRIPTION:
Failure of upgrade of recovery node (RN) enqueue to SSX mode
ARGUMENTS:
Arg [a] State Object for redo nab enqueue for resilvering
Arg [b] Redo nab enqueue mode
FUNCTIONALITY:
Kernel Cache Redo File Read
IMPACT:
INSTANCE FAILURE

处理方法
1.如果有备份,利用备份进行不完全恢复,跳过最后异常的redo,数据库resetlogs打开
2.如果没有备份,尝试使用历史的控制文件进行不完全恢复,或者直接跳过数据库一致性打开库.
3.互联网有人解决删除redo第二组成员数据库open成功(http://blog.itpub.net/16976507/viewspace-1266952/)
redo


存储精简卷导致asm磁盘组异常

有朋友在一个存储空间给asm使用,发生空间不足,然后使用另外一个存储中的lun给asm的data磁盘组增加asm disk,运行了大概1天之后,asm磁盘组直接dismount,数据库crash.然后就无法正常mount.包括这个存储上的几个其他磁盘组也无法正常mount.
数据库异常日志

Sun Oct 23 08:43:59 2016
SUCCESS: diskgroup DATA was dismounted
SUCCESS: diskgroup DATA was dismounted
Sun Oct 23 08:44:00 2016
Errors in file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lmon_79128.trc:
ORA-00202: control file: '+DATA/orcl/controlfile/current.278.892363163'
ORA-15078: ASM diskgroup was forcibly dismounted
Sun Oct 23 08:44:00 2016
Errors in file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lgwr_79174.trc:
ORA-00345: redo log write error block 15924 count 2
ORA-00312: online log 2 thread 1: '+DATA/orcl/onlinelog/group_2.274.892363167'
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lgwr_79174.trc:
ORA-00202: control file: '+DATA/orcl/controlfile/current.278.892363163'
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lgwr_79174.trc:
ORA-00204: error in reading (block 1, # blocks 1) of control file
ORA-00202: control file: '+DATA/orcl/controlfile/current.278.892363163'
ORA-15078: ASM diskgroup was forcibly dismounted
Sun Oct 23 08:44:00 2016
LGWR (ospid: 79174): terminating the instance due to error 204
Sun Oct 23 08:44:00 2016
opiodr aborting process unknown ospid (79742) as a result of ORA-1092
Sun Oct 23 08:44:01 2016
ORA-1092 : opitsk aborting process
Sun Oct 23 08:44:01 2016
ORA-1092 : opitsk aborting process
System state dump requested by (instance=1, osid=79174 (LGWR)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_diag_79118.trc
Instance terminated by LGWR, pid = 79174

很明显,数据库异常是由于asm diskgroup dismount,因此分析asm 日志

asm 日志

Sun Oct 23 07:00:31 2016
Time drift detected. Please check VKTM trace file for more details.
Sun Oct 23 08:43:55 2016
Errors in file /oracle/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_8755.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
WARNING: Write Failed. group:1 disk:2 AU:1222738 offset:0 size:1048576
ERROR: failed to copy file +DATA.524, extent 15030
ERROR: ORA-15080 thrown in ARB0 for group number 1
Errors in file /oracle/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_8755.trc:
ORA-15080: synchronous I/O operation to a disk failed
Sun Oct 23 08:43:55 2016
NOTE: stopping process ARB0
NOTE: rebalance interrupted for group 1/0xec689cdd (DATA)
NOTE: ASM did background COD recovery for group 1/0xec689cdd (DATA)
NOTE: starting rebalance of group 1/0xec689cdd (DATA) at power 1
Starting background process ARB0
Sun Oct 23 08:43:56 2016
ARB0 started with pid=24, OS id=103554
NOTE: assigning ARB0 to group 1/0xec689cdd (DATA) with 1 parallel I/O
Errors in file /oracle/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_103554.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1048576
WARNING: Write Failed. group:1 disk:2 AU:1222738 offset:0 size:1048576
ERROR: failed to copy file +DATA.256, extent 6570
ERROR: ORA-15080 thrown in ARB0 for group number 1
Errors in file /oracle/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_103554.trc:
ORA-15080: synchronous I/O operation to a disk failed
NOTE: stopping process ARB0
Sun Oct 23 08:43:58 2016
Errors in file /oracle/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dbw0_8521.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 4096
WARNING: Write Failed. group:1 disk:3 AU:6789 offset:24576 size:4096
NOTE: cache initiating offline of disk 3 group DATA
NOTE: process _dbw0_+asm1 (8521) initiating offline of disk 3.3915934787 (DATA_0003) with mask 0x7e in group 1
Sun Oct 23 08:43:58 2016
WARNING: Disk 3 (DATA_0003) in group 1 mode 0x7f is now being offlined
WARNING: Disk 3 (DATA_0003) in group 1 in mode 0x7f is now being taken offline on ASM inst 1
NOTE: initiating PST update: grp = 1, dsk = 3/0xe9686c43, mask = 0x6a, op = clear
GMON updating disk modes for group 1 at 14 for pid 14, osid 8521
ERROR: Disk 3 cannot be offlined, since diskgroup has external redundancy.
ERROR: too many offline disks in PST (grp 1)
Sun Oct 23 08:43:58 2016
NOTE: cache dismounting (not clean) group 1/0xEC689CDD (DATA)
NOTE: messaging CKPT to quiesce pins Unix process pid: 103577, image: oracle@node1 (B000)
WARNING: Disk 3 (DATA_0003) in group 1 mode 0x7f offline is being aborted
WARNING: Offline of disk 3 (DATA_0003) in group 1 and mode 0x7f failed on ASM inst 1
NOTE: halting all I/Os to diskgroup 1 (DATA)
Sun Oct 23 08:43:59 2016
NOTE: LGWR doing non-clean dismount of group 1 (DATA)
NOTE: LGWR sync ABA=160.10145 last written ABA 160.10145

错误信息很明显,由于Write Failed导致asm diskgroup dismount.

系统日志

Oct 23 08:43:55 node1 kernel: sd 6:0:12:1: [sdd]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 23 08:43:55 node1 kernel: sd 6:0:12:1: [sdd]  Sense Key : Data Protect [current]
Oct 23 08:43:55 node1 kernel: sd 6:0:12:1: [sdd]  Add. Sense: Space allocation failed write protect
Oct 23 08:43:55 node1 kernel: sd 6:0:12:1: [sdd] CDB: Write(16): 8a 00 00 00 00 02 e7 18 37 f9 00 00 00 07 00 00
Oct 23 08:43:55 node1 kernel: end_request: critical space allocation error, dev sdd, sector 12467058681
Oct 23 08:43:55 node1 kernel: end_request: critical space allocation error, dev dm-3, sector 12467058681
Oct 23 08:43:55 node1 kernel: sd 8:0:6:1: [sdh]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 23 08:43:55 node1 kernel: sd 8:0:6:1: [sdh]  Sense Key : Data Protect [current]
Oct 23 08:43:55 node1 kernel: sd 8:0:6:1: [sdh]  Add. Sense: Space allocation failed write protect
Oct 23 08:43:55 node1 kernel: sd 8:0:6:1: [sdh] CDB: Write(16): 8a 00 00 00 00 02 e7 18
Oct 23 08:43:55 node1 kernel: sd 6:0:4:1: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 23 08:43:55 node1 kernel: sd 6:0:4:1: [sdb]  Sense Key : Data Protect [current]
Oct 23 08:43:55 node1 kernel: sd 6:0:4:1: [sdb]   33Add. Sense: Space allocation failed write protect
Oct 23 08:43:55 node1 kernel: sd 6:0:4:1: [sdb] CDB: Write(16): 8a 00 00 00 00 02 e7 18 30 00 00 00 03 f9 00 00
Oct 23 08:43:55 node1 kernel: end_request: critical space allocation error, dev sdb, sector 12467056640
Oct 23 08:43:55 node1 kernel: f9 00 00 04 00
Oct 23 08:43:55 node1 kernel: end_request: critical space allocation error, dev dm-3, sector 12467056640
Oct 23 08:43:55 node1 kernel: 00 00
Oct 23 08:43:55 node1 kernel: end_request: critical space allocation error, dev sdh, sector 12467057657
Oct 23 08:43:55 node1 kernel: end_request: critical space allocation error, dev dm-3, sector 12467057657
Oct 23 08:43:57 node1 kernel: sd 8:0:6:1: [sdh]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 23 08:43:57 node1 kernel: sd 8:0:6:1: [sdh]  Sense Key : Data Protect [current]
Oct 23 08:43:57 node1 kernel: sd 8:0:6:1: [sdh]  Add. Sense: Space allocation failed write protect
Oct 23 08:43:57 node1 kernel: sd 8:0:6:1: [sdh] CDB: Write(16): 8a 00 00 00 00 02 e7 18 37 f9 00 00 00 07 00 00
Oct 23 08:43:57 node1 kernel: end_request: critical space allocation error, dev sdh, sector 12467058681
Oct 23 08:43:57 node1 kernel: end_request: critical space allocation error, dev dm-3, sector 12467058681
Oct 23 08:43:57 node1 kernel: sd 8:0:12:1: [sdj]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 23 08:43:57 node1 kernel: sd 8:0:12:1: [sdj]  Sense Key : Data Protect [current]
Oct 23 08:43:57 node1 kernel: sd 8:0:12:1: [sdj]  Add. Sense: Space allocation failed write protect
Oct 23 08:43:57 node1 kernel: sd 8:0:12:1: [sdj] CDB: Write(16): 8a 00 00 00 00 02 e7 18 30 00 00 00 03 f9 00 00
Oct 23 08:43:57 node1 kernel: end_request: critical space allocation error, dev sdj, sector 12467056640
Oct 23 08:43:57 node1 kernel: end_request: critical space allocation error, dev dm-3, sector 12467056640
Oct 23 08:43:57 node1 kernel: sd 6:0:4:1: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 23 08:43:57 node1 kernel: sd 6:0:4:1: [sdb]  Sense Key : Data Protect [current]
Oct 23 08:43:57 node1 kernel: sd 6:0:4:1: [sdb]  Add. Sense: Space allocation failed write protect
Oct 23 08:43:57 node1 kernel: sd 6:0:4:1: [sdb] CDB: Write(16): 8a 00 00 00 00 02 e7 18 33 f9 00 00 04 00 00 00
Oct 23 08:43:58 node1 kernel: sd 6:0:4:1: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 23 08:43:58 node1 kernel: sd 6:0:4:1: [sdb]  Sense Key : Data Protect [current]
Oct 23 08:43:58 node1 kernel: sd 6:0:4:1: [sdb]  Add. Sense: Space allocation failed write protect
Oct 23 08:43:58 node1 kernel: sd 6:0:4:1: [sdb] CDB: Write(16): 8a 00 00 00 00 03 3b 7e 78 30 00 00 00 08 00 00
Oct 23 10:50:59 node1 init: oracle-ohasd main process (6150) killed by TERM signal

错误信息为:critical space allocation error,严重空间分配错误.也就是linux在分配空间之时发生错误.在换而言之,由于分配空间错误导致asm 磁盘组dismount.

查看多路径信息

[root@node1 ~]# multipath -ll
36000d31003190c000000000000000003 dm-3 COMPELNT,Compellent Vol
size=80T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 6:0:9:1  sdd 8:48  active ready running
  `- 8:0:9:1  sdi 8:128 active ready running
delldisk2 (36000d310031908000000000000000003) dm-4 COMPELNT,Compellent Vol
size=8.0T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 6:0:12:1 sde 8:64  active ready running
  |- 8:0:6:1  sdh 8:112 active ready running
  |- 6:0:4:1  sdb 8:16  active ready running
  `- 8:0:12:1 sdj 8:144 active ready running
delldisk1 (36000d31003190a000000000000000007) dm-2 COMPELNT,Compellent Vol
size=12T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 6:0:1:1  sda 8:0   active ready running
  |- 8:0:2:1  sdf 8:80  active ready running
  |- 6:0:7:1  sdc 8:32  active ready running
  `- 8:0:3:1  sdg 8:96  active ready running

很明显报错的都是同一个lun(delldisk2),也就是存储空间使用完的存储.也就是说,由于delldisk2存储的空间使用尽了导致系统出现分配空间错误,从而导致asm 写失败,进而导致数据库异常.这种问题的本质其实就是存储给系统分配了8T,但是实际存储可以使用的空间不足8T,而os按照8T来使用从而出现该问题.专业名字叫做”存储精简卷”.因此各位在存储配置之时需要注意该问题.因为这种情况的出现一般只是写io异常,读依旧正常,因此不会丢失数据.

ORA-20011 KUP-11024错误分析

数据库alert日志出现ORA-20011 KUP-11024等错误

Thu Sep 22 18:00:31 2016
DBMS_STATS: GATHER_STATS_JOB encountered errors.  Check the trace file.
Errors in file /u1/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_j002_2686.trc:
ORA-20011: Approximate NDV failed: ORA-29913: error in executing ODCIEXTTABLEOPEN callout
KUP-11024: This external table can only be accessed from within a Data Pump job.

从报错的信息看应该是数据库收集统计信息报错(GATHER_STATS_JOB),但是报错原因是由于访问外部表导致,而该外部表很可能和data pump有关系.

查看trace日志

[oracle@xifenfei]$ more /u1/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_j002_2686.trc
Trace file /u1/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_j002_2686.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /u1/oracle/pruduct/11.2.0.3
System name:    Linux
Node name:      xifenfei
Release:        2.6.32-220.el6.x86_64
Version:        #1 SMP Wed Nov 9 08:03:13 EST 2011
Machine:        x86_64
Instance name: xifenfei
Redo thread mounted by this instance: 1
Oracle process number: 356
Unix process pid: 2686, image: oracle@xifenfei (J002)
*** 2016-09-22 18:00:31.939
*** SESSION ID:(835.16363) 2016-09-22 18:00:31.939
*** CLIENT ID:() 2016-09-22 18:00:31.939
*** SERVICE NAME:(SYS$USERS) 2016-09-22 18:00:31.939
*** MODULE NAME:(DBMS_SCHEDULER) 2016-09-22 18:00:31.939
*** ACTION NAME:(ORA$AT_OS_OPT_SY_10669) 2016-09-22 18:00:31.939
ORA-20011: Approximate NDV failed: ORA-29913: error in executing ODCIEXTTABLEOPEN callout
KUP-11024: This external table can only be accessed from within a Data Pump job.
*** 2016-09-22 18:00:31.939
DBMS_STATS: GATHER_STATS_JOB: GATHER_TABLE_STATS('"DWDBA"','"ET$012D00070001"','""', ...)
DBMS_STATS: ORA-20011: Approximate NDV failed: ORA-29913: error in executing ODCIEXTTABLEOPEN callout
KUP-11024: This external table can only be accessed from within a Data Pump job.
*** 2016-09-22 18:00:31.960
DBMS_STATS: GATHER_STATS_JOB: GATHER_TABLE_STATS('"DWDBA"','"ET$01D10D4F0001"','""', ...)
DBMS_STATS: ORA-20011: Approximate NDV failed: ORA-29913: error in executing ODCIEXTTABLEOPEN callout
KUP-11024: This external table can only be accessed from within a Data Pump job.

通过trace文件,我们已经可以明确是由于数据库对DWDBA.ET$012D00070001和DWDBA.ET$01D10D4F0001这两个表收集统计信息时候报的上述alert日志中看到的错误.

查询数据库记录

SYS@xifenfei>select OWNER,OBJECT_NAME,OBJECT_TYPE, status,
  2  to_char(CREATED,'dd-mon-yyyy hh24:mi:ss') created
  3  ,to_char(LAST_DDL_TIME , 'dd-mon-yyyy hh24:mi:ss') last_ddl_time
  4  from dba_objects
  5  where object_name like 'ET$%'
  6  /
OWNER     OBJECT_NAME      OBJECT_TYPE  STATUS  CREATED               LAST_DDL_TIME
--------- ---------------- ------------ ------- ------------------------- ----------------
DWDBA     ET$012D00070001  TABLE        VALID   10-mar-2016 16:32:25  10-mar-2016 16:32:25
DWDBA     ET$01D10D4F0001  TABLE        VALID   10-mar-2016 17:29:29  10-mar-2016 17:29:29
SYS@xifenfei> select owner, TABLE_NAME, DEFAULT_DIRECTORY_NAME, ACCESS_TYPE
  2  from dba_external_tables
  3  order by 1,2
  4  /
OWNER       TABLE_NAME                     DEFAULT_DIRECTORY_NAME         ACCESS_
----------- ------------------------------ ------------------------------ -------
DWDBA       ET$012D00070001                EXP_FILE_DIR                   CLOB
DWDBA       ET$01D10D4F0001                EXP_FILE_DIR                   CLOB

到这一步,我们已经完全清楚,ET$012D00070001和ET$01D10D4F0001是两个外部表,由于他们的存在使得收集统计信息异常。

分析ET$012D00070001表

SYS@xifenfei>desc DWDBA.ET$012D00070001
 Name                                                  Null?    Type
 ----------------------------------------------------- -------- ------------------------------------
 STORE_NO                                                       NUMBER(3)
 ITEM_NO                                                        NUMBER(6)
 WORK_DATE                                                      DATE
 DIVISION_NO                                                    NUMBER(2)
 SECTION_NO                                                     NUMBER(3)
 SUP_NO                                                         NUMBER(6)
 GRP_NO                                                         NUMBER(3)
 SUBGRP_NO                                                      NUMBER(3)
 USR                                                            VARCHAR2(30)
 TYPE                                                           NUMBER(1)
 ACTIVE_SELL_PRICE                                              NUMBER(8,2)
 SELL_PRICE                                                     NUMBER(8,2)
 SELL_VAT                                                       NUMBER(1)
 BUY_PRICE                                                      NUMBER(10,4)
 BUY_VAT                                                        NUMBER(1)
 PROMOTION_NO                                                   NUMBER(10)
 PROM_CLASS                                                     VARCHAR2(1)
 PROM_LEVEL                                                     NUMBER(1)
 STOCK                                                          NUMBER(10,3)
 STOCK_ADJ                                                      NUMBER(10,3)
 RECPT                                                          NUMBER(10,3)
 SALES                                                          NUMBER(10,3)
 STOCK_ADJ_AMNT                                                 NUMBER(10,2)
 RECPT_AMNT                                                     NUMBER(10,2)
 SALES_AMNT                                                     NUMBER(10,2)
 PROF_AMNT                                                      NUMBER(10,2)
 COST_CHANGE                                                    NUMBER(10,2)
 DISC                                                           NUMBER(10,3)
 RTN_QTY                                                        NUMBER(9,3)
 DISC_AMNT                                                      NUMBER(10,2)
 RTN_AMNT                                                       NUMBER(10,2)
 LOSS_AMNT                                                      NUMBER(10,2)
 CREATED_DATE                                                   DATE
 COST                                                           NUMBER(10,4)
 NBR_PK                                                         NUMBER(5)
 NBR_VISIT                                                      NUMBER(5)
 NBR_PK_LINE                                                    NUMBER(5)
 N_N_PROF_AMNT                                                  NUMBER(9,2)
 CON_FORE                                                       NUMBER(10,2)
 CON_FORE_OTH                                                   NUMBER(10,2)
 SALES_B                                                        NUMBER(10,3)
 SALES_AMNT_B                                                   NUMBER(10,2)
SYS@xifenfei>select count(*) from DWDBA.ET$012D00070001;
select count(*) from DWDBA.ET$012D00070001
*
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
KUP-11024: This external table can only be accessed from within a Data Pump job.

通过对ET$012D00070001表查询重新了alert日志一样的错误,可以明确定位问题就是由于该外部表异常导致.通过查询mos,确定Bug 10327346 DBMS_WORKLOAD_CAPTURE does not drop external tables (causing ORA-20011 from DBMS_STATS)可能导致DBMS_WORKLOAD_CAPTURE无法正常清理掉Data pump的外部表导致出现Datapump出现孤立的外部表对象,从而出现该问题.解决方案就是直接drop 相关外部表.也就是这里的(ET$012D00070001和ET$01D10D4F0001)

Quick Reference to Patch Numbers for Database PSU, SPU(CPU), Bundle Patches and Patchsets—201610

Patchsets

 l12.1.0.2 (12.1.0.2.0 PATCH SET FOR ORACLE DATABASE SERVER)

 21419221

 11.2.0.4 (11.2.0.4.0 PATCH SET FOR ORACLE DATABASE SERVER)

 13390677

 11.2.0.3 (11.2.0.3.0 PATCH SET FOR ORACLE DATABASE SERVER)

 10404530

 11.2.0.2 (11.2.0.2.0 PATCH SET FOR ORACLE DATABASE SERVER)

 10098816

 11.1.0.7 (11.1.0.7.0 PATCH SET FOR ORACLE DATABASE SERVER)

 6890831

 10.2.0.5 (10.2.0.5 PATCH SET FOR ORACLE DATABASE SERVER)

 8202632

 d10.2.0.4 (10.2.0.4.0 PATCH SET FOR ORACLE DATABASE SERVER)

 6810189

 e10.2.0.3 (10.2.0.3 PATCH SET FOR ORACLE DATABASE SERVER)

 5337014

 10.2.0.2 (10.2.0.2 PATCH SET FOR ORACLE DATABASE SERVER)

 4547817

 10.1.0.5 (10.1.0.5 PATCH SET FOR ORACLE DATABASE SERVER)

 4505133

 10.1.0.4 (10.1.0.4 PATCH SET FOR ORACLE DATABASE SERVER)

 4163362

 10.1.0.3 (10.1.0.3 PATCH SET FOR ORACLE DATABASE SERVER)

 3761843

 9.2.0.8 (9.2.0.8 PATCH SET FOR ORACLE DATABASE SERVER)

 4547809

 9.2.0.7 (9.2.0.7 PATCH SET FOR ORACLE DATABASE SERVER)

 4163445

 9.2.0.6 (9.2.0.6 PATCH SET FOR ORACLE DATABASE SERVER)

 3948480

 9.2.0.5 (ORACLE 9I DATABASE SERVER RELEASE 2 – PATCH SET 4 VERSION 9.2.0.5.0)

 3501955

 9.2.0.4 (9.2.0.4 PATCH SET FOR ORACLE DATABASE SERVER) 

 3095277

 9.2.0.3 (9.2.0.3 PATCH SET FOR ORACLE DATABASE SERVER)

 2761332

 9.2.0.2 (9.2.0.2 PATCH SET FOR ORACLE DATABASE SERVER)

 2632931

 9.0.1.5 (9.0.1.5 PATCHSET)

 3301544

 9.0.1.4 (9.0.1.4 PATCH SET FOR ORACLE DATABASE SERVER)

 2517300

 9.0.1.3 (9.0.1.3. PATCH SET FOR ORACLE DATA SERVER)

 2271678

 8.1.7.4 (8.1.7.4 PATCH SET FOR ORACLE DATA SERVER)

 2376472

 8.1.7.3 (8.1.7.3 PATCH SET FOR ORACLE DATA SERVER)

 2189751

 8.1.7.2 (8.1.7.2.1 PATCH SET FOR ORACLE DATA SERVER)

 1909158

PSU, SPU(CPU), Bundle Patches

12.1.0.2

 Description

 PSU

   GI PSU

Proactive Bundle Patch

 Bundle Patch(Windows 32bit & 64bit)

 OCT2016

 24006101 (12.1.0.2.161018)

 24412235 (12.1.0.2.161018)

  24448103 (12.1.0.2.161018)

 24591642(12.1.0.2.161018)

 JUL2016

 23054246 (12.1.0.2.160719)

 23273629 (12.1.0.2.160719)

 23273686 (12.1.0.2.160719)

 23530387(12.1.0.2.160719)

 APR2016

 22291127 (12.1.0.2.160419)

 22646084 (12.1.0.2.160419)

 22899531

 22809813(12.1.0.2.160419)

 JAN2016

 21948354 (12.1.0.2.160119)

 22191349 (12.1.0.2.160119)

 22243551

 22310559(12.1.0.2.160119)

 OCT2015

 21359755 (12.1.0.2.5)

 21523234 (12.1.0.2.5)

 21744410 (12.1.0.2.13)

 21821214(12.1.0.2.10)

 JUL2015

 20831110 (12.1.0.2.4)

 20996835 (12.1.0.2.4)

 21188742 (12.1.0.2.10)

 21126814(12.1.0.2.7)

 APR2015

 20299023 (12.1.0.2.3)

 20485724 (12.1.0.2.3)

 20698050 (12.1.0.2.7)

 20684004(12.1.0.2.4)

 JAN2015

 19769480 (12.1.0.2.2)

 19954978 (12.1.0.2.2)

 20141343 (12.1.0.2.4)

 19720843(12.1.0.2.1)

 OCT2014

 19303936 (12.1.0.2.1)

 19392646 (12.1.0.2.1)

 19404326 (12.1.0.2.1)

 N/A

 

12.1.0.1

 Description

 PSU

 GI PSU

  Bundle Patch (Windows64bit)

 Bundle Patch (Windows32bit)

 JUL2016

 23054354 (12.1.0.1.160719)

 i23273935 / k23273958 (12.1.0.1.160719)

23530410 (12.1.0.1.160719)

 APR2016

 22291141 (12.1.0.1.160419)

 i22654153 / k22654166(12.1.0.1.160419)

22617408 (12.1.0.1.160419)

 JAN2016

 21951844 (12.1.0.1.160119)

 j22191492 / k22191511(12.1.0.1.160119)

22494866 (12.1.0.2.160119)

 OCT2015

 21352619 (12.1.0.1.9)

 j21551666 / k21551685 (12.1.0.1.9)

21744907 (12.1.0.1.21)

 JUL2015

 20831107 (12.1.0.1.8)

  j20996901 / k20996911(12.1.0.1.8)

21076681  (12.1.0.1.20)

 APR2015

 20299016 (12.1.0.1.7)

  j20485762 / k19971331(12.1.0.1.7)

20558101 (12.1.0.1.18)

 JAN2015

 19769486 (12.1.0.1.6)

 j19971324 / k19971331 (12.1.0.1.6)

20160748 (12.1.0.1.16)

 OCT2014

 19121550 (12.1.0.1.5)

 j19392372 / k19392451 (12.1.0.1.5)

19542943 (12.1.0.1.14)

 JUL2014

 18522516 (12.1.0.1.4)

 j18705901 / k18705972 (12.1.0.1.4)

19062327 (12.1.0.1.11)

 APR2014

 18031528 (12.1.0.1.3)

 j18139660 / k18413105  (12.1.0.1.3)

18448604 (12.1.0.1.7)

 JAN2014

 17552800 (12.1.0.1.2)

 17735306 (12.1.0.1.2)

17977915 (12.1.0.1.3)

 OCT2013

 17027533 (12.1.0.1.1)

 17272829 (12.1.0.1.1)

 17363796 (12.1.0.1.1)

 17363795 (12.1.0.1.1)

11.2.0.4

 Description

 PSU

 SPU(CPU)

 GI PSU

 Bundle Patch (Windows 32bit & 64bit)

 OCT2016

 24006111 (11.2.0.4.161018)

 24433711 (11.2.0.4.161018)

 24436338 (11.2.0.4.161018)

 24591646 (11.2.0.4.161018)

 JUL2016

 23054359 (11.2.0.4.160719)

 23177648 (11.2.0.4.160719)

 23274134 (11.2.0.4.160719)

 23530402 (11.2.0.4.160719)

 APR2016

 22502456 (11.2.0.4.160419)

 22502493 (11.2.0.4.160419)

 22646198 (11.2.0.4.160419)

 22839608 (11.2.0.4.160419)

 JAN2016

 21948347 (11.2.0.4.160119)

 21972320 (11.2.0.4.160119)

 22191577 (11.2.0.4.160119)

 22310544 (11.2.0.4.160119)

 OCT2015

 21352635 (11.2.0.4.8)

 21352646

 21523375 (11.2.0.4.8)

 21821802 (11.2.0.4.20)

 JUL2015

 20760982 (11.2.0.4.7)

 20803583

 20996923 (11.2.0.4.7)

 21469106 (11.2.0.4.18)

 APR2015

 20299013 (11.2.0.4.6)

 20299015

 20485808 (11.2.0.4.6)

 20544696 (11.2.0.4.15)

 JAN2015

 19769489 (11.2.0.4.5)

 19854503

 19955028 (11.2.0.4.5)

 20127071 (11.2.0.4.12)

 OCT2014

 19121551 (11.2.0.4.4)

 19271443

 19380115 (11.2.0.4.4)

 19651773 (11.2.0.4.10)

 JUL2014

 18522509 (11.2.0.4.3)

 18681862

 18706472 (11.2.0.4.3)

 18842982 (11.2.0.4.7)

 APR2014

 18031668 (11.2.0.4.2)

 18139690

 18139609 (11.2.0.4.2)

 18296644 (11.2.0.4.4)

 JAN2014

 17478514 (11.2.0.4.1)

 17551709

 N/A

 17987366 (11.2.0.4.1)

11.2.0.3

 Description

 PSU

 SPU(CPU)

 GI PSU

 Bundle Patch (Windows64bit)

 Bundle Patch(Windows32bit)

 aJUL2015

 20760997 (11.2.0.3.15)

 20803576

 20996944 (11.2.0.3.15)

 21104036

 21104035

 APR2015

 20299017 (11.2.0.3.14)

 20299010

 20485830 (11.2.0.3.14)

 20420395

 20420394

 JAN2015

 19769496 (11.2.0.3.13)

 19854461

 19971343 (11.2.0.3.13)

 20233168

 20233167

 OCT2014

 19121548 (11.2.0.3.12)

 19271438

 19440385 (11.2.0.3.12)

 19618575

 19618574

 JUL2014

 18522512 (11.2.0.3.11)

 18681866

 18706488 (11.2.0.3.11)

 18940194

 18940193

 APR2014

 18031683 (11.2.0.3.10)

 18139695

 18139678 (11.2.0.3.10)

 18372244

 18372243

 JAN2014

 17540582 (11.2.0.3.9)

 17478415

 17735354 (11.2.0.3.9)

 18075406

 17906981

 OCT2013

 16902043 (11.2.0.3.8)

 17082364

 17272731 (11.2.0.3.8)

 17363850

 17363844

 JUL2013

 16619892 (11.2.0.3.7)

 16742095

 16742216 (11.2.0.3.7)

 16803775

 16803774

 APR2013

 16056266 (11.2.0.3.6)

 16294378

 16083653 (11.2.0.3.6)

 16345834

 16345833

 JAN2013

 14727310 (11.2.0.3.5)

 14841409

 14727347 (11.2.0.3.5)

 16042648

 16042647

 OCT2012

 14275605 (11.2.0.3.4)

 14390252

 14275572 (11.2.0.3.4)

 14613223

 14613222

 JUL2012

 13923374 (11.2.0.3.3)

 14038787

 13919095 (11.2.0.3.3)

 14223718

 14223717

 APR2012

 13696216 (11.2.0.3.2)

 13632717

 13696251 (11.2.0.3.2)

 13885389

 13885388

 JAN2012

 13343438 (11.2.0.3.1)

 13466801

 13348650 (11.2.0.3.1)

 13413168

 13413167

11.2.0.2

 Description

 PSU

  SPU(CPU)

 GI PSU

 Bundle Patch (Windows64bit)

 Bundle Patch(Windows32bit)

 aOCT2013

 17082367 (11.2.0.2.12)

 17082375

 17272753 (11.2.0.2.12)

 17363838

 17363837

 JUL2013

 16619893 (11.2.0.2.11)

 16742100

 16742320 (11.2.0.2.11)

 16345852

 16345851

 APR2013

 16056267 (11.2.0.2.10)

 16294412

 16166868 (11.2.0.2.10)

 16345846

 16345845

 JAN2013

 14727315 (11.2.0.2.9)

 14841437

 14841385 (11.2.0.2.9)

 16100399

 16100398

 OCT2012

 14275621 (11.2.0.2.8)

 14390377

 14390437 (11.2.0.2.8)

 14672268

 14672267

 JUL2012

 13923804 (11.2.0.2.7)

 14038791

 14192201 (11.2.0.2.7)

 14134043

 14134042

 APR2012

 13696224 (11.2.0.2.6)

 13632725

 13696242 (11.2.0.2.6)

 13697074

 13697073

 JAN2012

 13343424 (11.2.0.2.5)

 13343244

 13653086 (11.2.0.2.5)

 13413155

 13413154

 OCT2011

 12827726 (11.2.0.2.4)

 12828071

 12827731 (11.2.0.2.4)

 13038788

 13038787

 JUL2011

 12419331 (11.2.0.2.3)

 12419321

 12419353 (11.2.0.2.3)

 12714463

 12714462

 APR2011

 11724916 (11.2.0.2.2)

 11724984

 12311357 (11.2.0.2.2)

 11896292

 11896290

 JAN2011

 10248523 (11.2.0.2.1)

 N/A

 N/A

 10432053

 10432052

11.2.0.1

 Description

 PSU

 CPU

 Bundle Patch (Windows64bit)

 Bundle Patch (Windows32bit)

 aJUL2011

 12419378 (11.2.0.1.6)

 12419278

 12429529

 12429528

 APR2011

 11724930 (11.2.0.1.5)

 11724991

 11731176

 11883240

 JAN2011

 10248516 (11.2.0.1.4)

 10249532

 10432045

 10432044

 OCT2010

 9952216 (11.2.0.1.3)

 9952260

 10100101

 10100100

 JUL2010

 9654983 (11.2.0.1.2)

 9655013

 9736865

 9736864

 APR2010

 9352237 (11.2.0.1.1)

 9369797

 N/A

 N/A

11.1.0.7

 Description

 PSU

 SPU(CPU)

 Bundle Patch (Windows64bit)

 Bundle Patch (Windows32bit)

JUL2015

 20761024 (11.1.0.7.24)

 20803573

 21104030

 21104029

APR2015

 20299012 (11.1.0.7.23)

 20299020

 20420391

 20420390

JAN2015

 19769499 (11.1.0.7.22)

 19854433

 20126915

 20126914

OCT2014

 19152553 (11.1.0.7.21)

 19274522

 19609034

 19609032

JUL2014

 18522513 (11.1.0.7.20)

 18681875

 18944208

 18944207

APR2014

 18031726 (11.1.0.7.19)

 18139703

 18372258

 18372257

JAN2014

 17465583 (11.1.0.7.18)

 17551415

 17906936

 17906935

OCT2013

 17082366 (11.1.0.7.17)

 17082374

 17363760

 17363759

JUL2013

 16619896 (11.1.0.7.16)

 16742110

 16803788

 16803787

APR2013

 16056268 (11.1.0.7.15)

 16308394

 16345862

 16345861

JAN2013

 14739378 (11.1.0.7.14)

 14841452

 15848067

 15848066

OCT2012

 14275623 (11.1.0.7.13)

 14390384

 14672313

 14672312

 JUL2012

 13923474 (11.1.0.7.12)

 14038803

 14109868

 14109867

 APR2012

 13621679 (11.1.0.7.11)

 13632731

 13715810

 13715809

 JAN2012

 13343461 (11.1.0.7.10)

 13343453

 13460956

 13460955

 OCT2011

 12827740 (11.1.0.7.9)

 12828097

 12914916

 12914915

 JUL2011

 12419384 (11.1.0.7.8)

 12419265

 12695278

 12695277

 APR2011

 11724936 (11.1.0.7.7)

 11724999

 11741170

 11741169

 JAN2011

 10248531 (11.1.0.7.6)

 10249534

 10350788

 10350787

 OCT2010

 9952228  (11.1.0.7.5)

 9952269

 9773825

 9773817

 JUL2010

 9654987 (11.1.0.7.4)

 9655014

 9869912

 9869911

 APR2010

 9352179 (11.1.0.7.3)

 9369783

 9392335

 9392331

 JAN2010

 9209238 (11.1.0.7.2)

 9114072

 9166861

 9166858

 OCT2009

 8833297 (11.1.0.7.1)

 8836375

 8928977

 8928976

 JUL2009

 N/A

 8534338

 8553515

 8553512

 APR2009

 N/A

 8290478

 8343070

 8343061

11.1.0.6

 Description

 CPU

 Bundle Patch (Windows64bit)

 Bundle Patch (Windows32bit)

 aJUL2009

 8534378

 8563155

 8563154

 APR2009

 8290402

 8333657

 8333655

 JAN2009

 7592335

 7631981

 7631980

 OCT2008

 7375639

 7378393

 7378392

 JUL2008

 7150417

 7210197

 7210195

 APR2008

 6864063

 6867180

 6867178

10.2.0.5

 Description

 PSU

 SPU(CPU)

 Bundle Patch (Windows64bit)

 Bundle Patch (Windows32bit)

 Bundle Patch(WindowsItanium)

 bJUL2015

 20299014 (10.2.0.5.19)

 20299021

 20420387

 20420386

 N/A

 APR2015

 N/A

 N/A

 N/A

 N/A

 N/A

 JAN2015

 19769505 (10.2.0.5.18)

 19854436

 20126868

 20126867

 N/A

 OCT2014

 19274523 (10.2.0.5.17)

 19274521

 19618565

 19618563

 N/A

 JUL2014

 18522511 (10.2.0.5.16)

 18681879

 18940198

 18940196

 N/A

 APR2014

 18031728 (10.2.0.5.15)

 18139709

 18372261

 18372259

 N/A

 JAN2014

 17465584 (10.2.0.5.14)

 17551414

 17906974

 17906972

 N/A

 OCT2014

 17082365 (10.2.0.5.13)

 17082371

 N/A

 17363822

 N/A

 JUL2013

 16619894 (10.2.0.5.12)

 16742123

 16803782

 16803780

 16803781

 APR2013

 16056270 (10.2.0.5.11)

 16270946

 16345857

 16345855

 16345856

 JAN2013

 14727319 (10.2.0.5.10)

 14841459

 15848062

 15848060

 15848061

 OCT2012

 14275629 (10.2.0.5.9)

 14390396

 14553358

 14553356

 14553357

 JUL2012

 13923855 (10.2.0.5.8)

 14038805

 14134053

 14134051

 14134052

 APR2012

 13632743 (10.2.0.5.7)

 13632738

 13654815

 13654814

 13870404

 JAN2012

 13343471 (10.2.0.5.6)

 13343467

 13460968

13460967

 N/A

 OCT2011

 12827745 (10.2.0.5.5)

 12828105

 c12914913

 12914911

 N/A

 JUL2011

 12419392 (10.2.0.5.4)

 12419258

 12429524

 12429523

 N/A

 APR2011

 11724962 (10.2.0.5.3)

 11725006

 12328269

 12328268

 N/A

 JAN2011

 10248542 (10.2.0.5.2)

 10249537

 10352673

 10352672

 N/A

 OCT2010

 9952230 (10.2.0.5.1)

 9952270

 10099855

 10058290

 N/A

10.2.0.4

 Description

 PSU

 SPU(CPU)

 Bundle Patch (Windows32bit)

 Bundle Patch (Windows64bit)

 Bundle Patch(WindowsItanium)

 gJUL2013

 16619897 (10.2.0.4.17)

 16742253

 N/A

 N/A

 N/A

 gAPR2013

 16056269 (10.2.0.4.16)

 16270931

 N/A

 N/A

 N/A

 gJAN2013

 14736542 (10.2.0.4.15)

 14841471

 N/A

 N/A

 N/A

gOCT2012

 14275630 (10.2.0.4.14)

 14390410

 N/A

 N/A

 N/A

gJUL2012

 13923851 (10.2.0.4.13)

 14038814

 N/A

 N/A

 N/A

 aAPR2012

 12879933 (10.2.0.4.12)

 12879926

 13928775

 13928776

 N/A

 JAN2012

 12879929 (10.2.0.4.11)

 12879912

 b13654060

 N/A

 N/A

 OCT2011

 12827778 (10.2.0.4.10)

 12828112

 12914908

 12914910

 12914909

 JUL2011

 12419397 (10.2.0.4.9)

 12419249

 12429519

 12429521

 12429520

 APR2011

 11724977 (10.2.0.4.8)

 11725015

 12328501

 12328503

 12328502

 JAN2011

 10248636 (10.2.0.4.7)

 10249540

 10349197

 10349200

 10349198

 OCT2010

 9952234 (10.2.0.4.6)

 9952272

 10084980

 10084982

 10084981

 JUL2010

 9654991 (10.2.0.4.5)

 9655017

 9777076

 9777078

 9777077

 APR2010

 9352164 (10.2.0.4.4)

 9352191

 9393548

 9393550

 9393549

 JAN2010

 9119284 (10.2.0.4.3)

 9119226

 9169457

 9169460

 9169458

 OCT2009

 8833280 (10.2.0.4.2)

 8836308

 8880857

 8880861

 8880858

 JUL2009

 8576156 (10.2.0.4.1)

 8534387

 8559466

 8559467

 8541782

 APR2009

 N/A

 8290506

 8307237

 8307238

 8333678

 JAN2009

 N/A

 7592346

 7584866

 7584867

 N/A

 OCT2008

 N/A

 7375644

 7386320

 7386321

 N/A

 JUL2008

 N/A

 7150470

 7218676

 7218677

 N/A

10.2.0.3

 Description

 CPU (Unix/Linux)

 Bundle Patch (Windows32bit)

 Bundle Patch (WindowsItanium)

 Bundle Patch (Windows64bit)

 aJAN2009

 7592354

 7631956

 7631958

 7631957

 OCT2008

 7369190

 7353782

 7353784

 7353785

 JUL2008

 7150622

 7252496

 7252497

 7252498

 APR2008

 6864068

 6867054

 6867055

 6867056

 JAN2008

 6646853

 6637237

 6637238

 6637239

 OCT2007

 6394981

 6430171

 6430173

 6430174

 JUL2007

 6079591

 6116131

 6038242

 6116139

 APR2007

 5901891

 5948242

 5916262

 5948243

 JAN2007

 5881721

 5846376

 5846377

 5846378

10.2.0.2

 Description

 CPU (Unix/Linux)

 Bundle Patch (Windows32bit)

  Bundle Patch (Windows64bit)

 Bundle Patch (WindowsItanium)

 iJAN2009

 7592355

 N/A

 N/A

 N/A

 hOCT2008

 7375660

 N/A

 N/A

 N/A

 hJUL2008

 7154083

 N/A

 N/A

 N/A

 hAPR2008

 6864071

 N/A

 N/A

 N/A

 aJAN2008

 6646850

 N/A

 N/A

 N/A

 fOCT2007

 6394997

 6397028

 6397030

 6397029

 JUL2007

 6079588

 6013105

 6013121

 6013118

 APR2007

 5901881

 5912173

 5912179

 5912176

 JAN2007

 5689957

 5716143

 5699839

 5699824

 OCT2006

 5490848

 5502226

 5500921

 5500894

 JUL2006

 5225799

 5251025

 5251028

 5251026

 APR2006

 5079037

 5140461

 5140567

 5140508

10.2.0.1

 Description

 CPU (Unix/Linux)

 Bundle Patch (Windows32bit)

 Bundle Patch (Windows64bit)

 Bundle Patch (WindowsItanium)

 APR2007

 5901880

 N/A

 N/A

 N/A

 JAN2007

 5689937

 5695784

 5695786

 5695785

 OCT2006

 5490846

 5500927

 5500954

 5500951

 JUL2006

 5225798

 5239698

 5239701

 5239699

 APR2006

 5049080

 5059238

 5059261

 5059251

 JAN2006

 4751931

 4751539

 4770480

 4751549

10.1.0.5

 Description

 CPU (Unix/Linux)

 Bundle Patch (Windows32bit)

  Bundle Patch (WindowsItanium)

 JAN2012

 13343482

 13413002

 13413003

 OCT2011

 12828135

 12914905

 12914906

 JUL2011

 12419228

 12429517

 12429518

 APR2011

 11725035

 11731119

 11731120

 JAN2011

 N/A

 N/A

 N/A

 OCT2010

 9952279

 10089559

 10089560

 JUL2010

 9655023

 9683651

 9683652

 APR2010

 9352208

 9390288

 9390289

 JAN2010

 9119261

 9187104

 9187105

 OCT2009

 8836540

 8785211

 8785212

 JUL2009

 8534394

 8656224

 8656226

 APR2009

 8290534

 8300356

 8300360

 JAN2009

 7592360

 7486619

 7586049

 OCT2008

 7375686

 7367493

 7367494

 JUL2008

 7154097

 7047034

 7047037

 APR2008

 6864078

 6867107

 6867108

 JAN2008

 6647005

 6637274

 6637275

 OCT2007

 6395024

 6408393

 6408394

 JUL2007

 6079585

 6115804

 6115818

 APR2007

 5901877

 5907304

 5907305

 JAN2007

 5689908

 5716295

 5634747

 OCT2006

 5490845

 5500883

 5500885

 JUL2006

 5225797

 5251148

 5251140

 APR2006

 5049074

 5057606

 5057609

 JAN2006

 4751932

 4882231

 4882236

10.1.0.4

 Description

 CPU (Unix/Linux)

 Bundle Patch (Windows32bit)

 Bundle Patch (WindowsItanium)

 APR2007

 5901876

 5909871

 5909879

 JAN2007

 5689894

 5695771

 5695772

 OCT2006

 5490844

 5500878

 5500880

 JUL2006

 5225796

 5239736

 5239737

 APR2006

 5049067

 5059200

 5059227

 JAN2006

 4751928

 4751259

 4745040

 OCT2005

 4567866

 4579182

 4579188

 JUL2005

 4392423

 4440706

 4404600

 APR2005

 4210374

 4287619

 4287611

10.1.0.3

 Description

 CPU (Unix/Linux)

 Bundle Patch (Windows32bit)

 Bundle Patch (WindowsItanium)

 JAN2007

 5923277

 N/A

 N/A

 OCT2006

 5566825

 N/A

 N/A

 JUL2006

 5435164

 N/A

 N/A

 APR2006

 5158022

 N/A

 N/A

 JAN2006

 4751926

 4741077

 4741084

 OCT2005

 4567863

 4567518

 4567523

 JUL2005

 4392409

 4389012

 4389014

 APR2005

 4193286

 4269715

 4158888

 JAN2005

 4003062

 4074232

 3990812

10.1.0.2

 Description

 CPU (Unix/Linux)

 Bundle Patch (Windows32bit)

 Bundle Patch (WindowsItanium)

 APR2005

 4193293

 4181849

 4213305

 JUL2005

 4400766

 4388944

 4388948

 JAN2005

 4003051

 4104364

 4083038

9.2.0.8

 Description

 CPU (Unix/Linux)

 Bundle Patch (Windows32bit)

 Bundle Patch (WindowsItanium)

 JUL2010

 9655027

 9683644

 9683645

 APR2010

 9352224

 9390286

 N/A

 JAN2010

 9119275

 9187106

 N/A

 OCT2009

 8836758

 8785185

 8785186

 JUL2009

 8534403

 8427417

 8427418

 APR2009

 8290549

 8300340

 8300346

 JAN2009

 7592365

 7703210

 7703212

 OCT2008

 7375695

 7394394

 7394402

 JUL2008

 7154111

 7047026

 7047029

 APR2008

 6864082

 6867138

 6867139

 JAN2008

 6646842

 6637265

 6637266

 OCT2007

 6395038

 6417013

 6417014

 JUL2007

 6079582

 6130293

 6130295

 APR2007

 5901875

 5916268

 5916275

 JAN2007

 N/A

 N/A

 N/A

 OCT2006

 5490859

 5652380

 5639519

9.2.0.7

 Description

 CPU (Unix/Linux)

 Bundle Patch (Windows32bit)

 Bundle Patch (WindowsItanium)

 JUL2007

 6079579

 6146759

 6146748

 APR2007

 5901872

 5907274

 5907275

 JAN2007

 5689875

 5654905

 5654909

 OCT2006

 5490841

 5500873

 5500874

 JUL2006

 5225794

 5250980

 5250981

 APR2006

 5049060

 5064365

 5064364

 JAN2006

 4751923

 4751528

 4741074

 OCT2005

 4567854

 4579590

 4579599

 JUL2005

 4547566

 N/A

 N/A

9.2.0.6

 Description

 CPU (Unix/Linux)

 Bundle Patch (Windows32bit)

 Bundle Patch (WindowsItanium)

 OCT2006

 5490840

 5500865

 5500871

 JUL2006

 5225793

 5239794

 5239793

 APR2006

 5049051

 5059614

 5059615

 JAN2006

 4751921

 4751261

 4751262

 OCT2005

 4567846

 4579093

 4579097

 JUL2005

 4392392

 4445852

 4401917

 APR2005

 4193295

 4269928

 4213298

9.2.0.5

 Description

 CPU (Unix/Linux)

 Bundle Patch (Windows32bit)

 Bundle Patch (WindowsItanium)

 OCT2006

 5689708

 N/A

 N/A

 JUL2006

 5435138

 N/A

 N/A

 APR2006

 5219762

 N/A

 N/A

 OCT2005

 4560421

 N/A

 N/A

 JUL2005

 4392256

 4387563

 4391819

 APR2005

 4193299

 4195791

 4214192

 JAN2005

 4003006

 4104374

 3990809

9.2.0.4

 Description

 CPU (Unix/Linux)

 Bundle Patch (Windows32bit)

 Bundle Patch (WindowsItanium)

 JAN2005

 4002994

 4104369

 4083202

8.1.7.4

 Description

 CPU (Unix/Linux)

 Bundle Patch (Windows32bit)

 JAN2007

 5689799

 5686514

 OCT2006

 5490835

 5496067

 JUL2006

 5225788

 5236412

 APR2006

 5045247

 5057601

 JAN2006

 4751906

 4751570

 OCT2005

 4560405

 4554818

 JUL2005

 4392446

 4437058

 APR2005

 4193312

 4180163

 JAN2005

 4002909

 3921893

OJVM PSU Patches

12.1.0.2

 Description

 OJVM PSU (Linux/Unix)

 OJVM BP (Windows)

 Combo OJVM + DB PSU

 Combo OJVM + GI PSU

 Generic JDBC

 OCT2016

 24315824 (12.1.0.2.161018)

 24591630 (12.1.0.2.161018)

 24433133 (12.1.0.2.161018)

 24433148 (12.1.0.2.161018)

 Included in OJVM PSU

 JUL2016

 23177536 (12.1.0.2.160719)

 23515290 (12.1.0.2.160719)

 23615289 (12.1.0.2.160719)

 23615308 (12.1.0.2.160719)

 23727148 (Included in OJVM PSU)

 APR2016

 22674709 (12.1.0.2.160419)

 22839633 (12.1.0.2.160419)

 22738582 (12.1.0.2.160419)

 22738641 (12.1.0.2.160419)

 JAN2016

 22139226 (12.1.0.2.160119)

 22311086 (12.1.0.2.160119)

 22191659 (12.1.0.2.160119)

 22191676 (12.1.0.2.160119)

 OCT2015

 21555660 (12.1.0.2.5)

 21788394 (12.1.0.2.4)

 21520444

 21523260

 JUL2015

 21068507 (12.1.0.2.4)

 21153530 (12.1.0.2.3)

 21150768

 21150782

 APR2015

 20415564 (12.1.0.2.3)

 20391199 (12.1.0.2.2)

 20834354

 20834538

 JAN2015

 19877336 (12.1.0.2.2)

 20225938 (12.1.0.2.1)

 20132434

 20132450

 OCT2014 (12.1.0.2.1)

 19282028

 19791366

 19791375

12.1.0.1

 Description

 OJVM PSU (Linux/Unix)

 OJVM BP (Windows)

 Combo OJVM + DB PSU

 Combo OJVM + GI PSU

 Generic JDBC

 JUL2016 (12.1.0.1.160719)

 23177541

 23515285

 23615355

 23615368

 23727043 (Included in OJVM PSU)

 APR2016 (12.1.0.1.160419)

 22674703

 22839627

 22738678

 22738715

 Included in OJVM PSU

 JAN2016 (12.1.0.1.160119)

 22139235

 22311072

 22191711

 22191721

 OCT2015 (12.1.0.1.5)

 21555669

 21788365

 21744318

 21744328

 JUL2015 (12.1.0.1.4)

 21068523

 21153513

 21150806

 21150817

 APR2015 (12.1.0.1.3)

 20406245

 20225909

 20834568

 20834579

 JAN2015 (12.1.0.1.2)

 19877342

 20225916

 20132482

 20132489

 OCT2014 (12.1.0.1.1)

 19282024

 19801531

 19791363

 19791360

 19852357

11.2.0.4

 Description

 OJVM PSU (Linux/Unix)

 OJVM BP (Windows)

  Combo OJVM + DB PSU

 Combo OJVM + DB SPU

 Combo OJVM + GI PSU

 Generic JDBC

 OCT2016 (11.2.0.4.161018)

 24315821

 24591637

 24436313

 24433791

 24436346

 Included in OJVM PSU

 JUL2016 (11.2.0.4.160719)

 23177551

 23515277

 23615392

 23615381

 23615403

 23727132 (Included in OJVM PSU)

 APR2016 (11.2.0.4.160419)

 22674697

 22839614

 22738777

 22738732

 22738793

Included in OJVM PSU

 JAN2016 (11.2.0.4.160119)

 22139245

 22311053

 22378146

 22378121

 22378167

 OCT2015 (11.2.0.4.5)

 21555791

 21788344

 21744343

 21744335

 21744348

 JUL2015 (11.2.0.4.4)

 21068539

 21153498

 21150851

 21150829

 21150864

 APR2015 (11.2.0.4.3)

 20406239

 20225988

 20834611

 20834597

 20834621

 JAN2015 (11.2.0.4.2)

 19877440

 20225982

 20132580

 20132517

 20132615

 OCT2014 (11.2.0.4.1)

 19282021

19799291

 19791364

 19791358

 19791420

 19852360

11.2.0.3

 Description

 OJVM PSU (Linux/Unix)

 OJVM BP (Windows)

Combo OJVM + DB PSU

 Combo OJVM + DB SPU

Combo OJVM + GI PSU

 Generic JDBC

 JUL2015 (11.2.0.3.4)

21068553

 21153470

21150891

21150885

 21150904

 Included in OJVM PSU

 APR2015 (11.2.0.3.3)

20406220

 20391185

20834670

20834653

 20834686

 JAN2015 (11.2.0.3.2)

19877443

 20227195

20132646

20132635

 20132651

 OCT2014 (11.2.0.3.1)

19282015

 19806120

19791427

19791426

19791428

19852361

11.1.0.7

Description

OJVM PSU (Linux/Unix)

 OJVM BP (Windows)

Combo OJVM + DB PSU

Combo OJVM + DB SPU

 Combo OJVM + GI PSU

Generic JDBC

 JUL2015 (11.1.0.7.4)

 21068565

 21153423

 21150939

 21150929

  N/A

  Included in OJVM PSU

 APR2015 (11.1.0.7.3)

 20406213

 20391156

 20834724

 20834712

  N/A

 JAN2015 (11.1.0.7.2)

 19877446

 20227146

 20132677

 20132669

  N/A

 OCT2014 (11.1.0.7.1)

 19282002

 19806118

 19791436

 19791434

  N/A

 19852363

参考:Quick Reference to Patch Numbers for Database PSU, SPU(CPU), Bundle Patches and Patchsets (文档 ID 1454618.1)

alter database create datafile 导致数据文件丢失恢复

alter database create datafile导致原始数据文件丢失
有客户一个小系统找我们恢复,通过Oracle Database Recovery Check 检测之后我们红框部分发现一奇怪现象
ctl
create-datafile-ctl
create-datafile


1.文件头fuzzy为NO,不符合数据库异常crash常识,也和其他文件该状态不匹配
2.文件的创建时间,scn均和checkpoint时间,scn一致(也就是说该文件是创建之后就checkpoint,然后就没有其他操作)
3.文件开始应用的归档为5,110和其他数据文件要求的3115相差甚远
结合这些情况,怀疑该文件被重新创建,查找alert日志果如发现如下信息

两个文件通过create datafile创建之后,然后offline操作.通过alert日志核查file 6和8的创建时间和seq信息

Fri Jan 16 15:03:36 2015
Thread 1 advanced to log sequence 5 (LGWR switch)
  Current log# 2 seq# 5 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\MYCHOICE\REDO02.LOG
Fri Jan 16 15:13:19 2015
CREATE BIGFILE TABLESPACE "FBAUDIT"
DATAFILE 'E:\ZDSoft\ZDFood\databak\FBAUDIT' SIZE 10M
AUTOEXTEND ON NEXT 10M
MAXSIZE UNLIMITED LOGGING EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO
Completed: CREATE BIGFILE TABLESPACE "FBAUDIT"
DATAFILE 'E:\ZDSoft\ZDFood\databak\FBAUDIT' SIZE 10M
AUTOEXTEND ON NEXT 10M
MAXSIZE UNLIMITED LOGGING EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO
Sat Feb 07 15:03:46 2015
Thread 1 advanced to log sequence 110 (LGWR switch)
  Current log# 2 seq# 110 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\MYCHOICE\REDO02.LOG
Sat Feb 07 15:20:41 2015
CREATE BIGFILE TABLESPACE "CARD"
DATAFILE 'E:\ZDSoft\ZDCARD\databak\CARD1' SIZE 10M
AUTOEXTEND ON NEXT 10M
MAXSIZE UNLIMITED LOGGING EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO
Completed: CREATE BIGFILE TABLESPACE "CARD"
DATAFILE 'E:\ZDSoft\ZDCARD\databak\CARD1' SIZE 10M
AUTOEXTEND ON NEXT 10M
MAXSIZE UNLIMITED LOGGING EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO

通过结合alert日志判断,我们可以确定,当前我们Oracle Database Recovery Check检查出来的情况,是由于执行了create datafile命令导致故障前的文件丢失,创建了一个新的数据文件,而由于该库为非归档模式,导致该文件数据无法恢复(备注:不光是非归档模式不行,就算是归档模式,也需要从文件创建到现在的所有归档才行).在大部分生产系统,我相信不可能有这么的归档,因为在执行alter database create datafile命令之时一定要慎重,评估确定是否丢失归档,否则可能导致不可理的损坏).
客户意识到了悲剧的发生,但是希望我们帮忙恢复一张核心数据,用户的余额信息.

对于alter database create datafile丢失文件恢复
通过工具扫描原始文件相关的记录(由于写入大量数据,无法完整恢复,只能通过工具扫描,恢复部分数据)[asm disk header 彻底损坏恢复]
scan-datafile


因为原库虽然丢失了这两个文件,但是已经open成功,通过相关的data obj结合这个里面扫描到的文件,抽取出来需要的对象的block,然后对block里面的数据进行读取恢复出来相关数据.在这里我们还有一个难点就是由于这两个文件都是bigfile,给恢复过程增加了难度
recover-data


至此我们已经实现了对于alter database create datafile导致文件丢失的核心数据的恢复.尽可能的减小的客户的损坏.这种恢复是取决运气,数据在磁盘上的block没有被覆盖.如果覆盖了基本无望.
如果需要数据库恢复,请联系我们(ORACLE数据库恢复技术支持),将为您提供专业数据库技术支持:
Phone:17813235971    Q Q:107644445    E-Mail:dba@xifenfei.com
再次提醒
1.在数据库出现故障之时,尽可能保护现场,做操作之前要之后后果别百度了就不分青红皂白的直接操作,导致不可逆的破坏,数据可能永久性丢失[Oracle异常恢复前备份保护现场建议—FileSystem环境|Oracle异常恢复前备份保护现场建议—ASM环境
2.使用alter database create datafile命令之前需要慎重,评估是否所有的归档都存在