使用nas存储存放控制文件,数据文件的数据库服务器,因为突然断电后,数据库系统无法正常启动,报ORA-27086: skgfglk: unable to lock file – already in use错误,通过分析是因为netapp的nfs锁导致该故障.本文为同事的处理过程记录
数据库启动报错

[oraprod@erpdb dbs]$ sqlplus "/ as sysdba"
SQL*Plus: Release 9.2.0.6.0 - Production on Thu May 29 22:03:00 2014
Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area  581506668 bytes
Fixed Size                   452204 bytes
Variable Size             402653184 bytes
Database Buffers          167772160 bytes
Redo Buffers               10629120 bytes
ORA-00205: error in identifying controlfile, check alert log for more info

alert日志

Thu May 29 23:37:56 2014
ORA-00202: controlfile: '/erpdata/PROD/proddata/cntrl01.dbf'
ORA-27086: skgfglk: unable to lock file - already in use
Linux Error: 11: Resource temporarily unavailable
Additional information: 8
Thu May 29 23:37:56 2014
ORA-205 signalled during: ALTER DATABASE   MOUNT...

查看控制文件

[oraprod@erpdb ~]$ ls -ltr /erpdata/PROD/proddata/cntrl01.dbf
-rw-r-----  1 oraprod dba 16293888 May 30  2014 /erpdata/PROD/proddata/cntrl01.dbf
[oraprod@erpdb ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              97G  4.3G   87G   5% /
/dev/sda1             190M   12M  169M   7% /boot
none                  8.0G     0  8.0G   0% /dev/shm
/dev/sdb1             577G  169G  379G  31% /erpdata
/dev/sdb2             241G   16G  213G   7% /oracle
/dev/sda3             9.7G   55M  9.1G   1% /tmp
192.168.1.13:/vol/vol1/backup
                      1.4T  199G  1.3T  14% /erpdbbak
192.168.1.11:/vol/vol1/erpdb
                      150G   84G   67G  56% /erpdata/PROD

这里可以看出来,控制文件是存在的,但是控制文件是存放在nfs中,重试umount/mount /erpdata/PROD文件系统,问题依旧.检查nfs系统所在存储(netapp存储)锁情况

nas设备锁情况

netapp2*> priv set advanced
netapp2*> lock status -h
======== NLM host erpdb.xifenfei.com
10688 0x01b70b28:0x00b83a2c 0:0 1 GRANTED (0x000000005fc63a58)
10688 0x01b70b28:0x00b83a2b 0:0 1 GRANTED (0x000000005fc63398)
10686 0x01b70b28:0x0135d0b3 0:0 1 GRANTED (0x0000000065f73a58)
10686 0x01b70b28:0x0135d0b2 0:0 1 GRANTED (0x00000000161a3b78)
10686 0x01b70b28:0x0135d0b1 0:0 1 GRANTED (0x00000001286246f8)
…………
10686 0x01b70b28:0x01770860 0:0 1 GRANTED (0x000000007e8fb938)
10690 0x01b70b28:0x007adc2a 0:0 1 GRANTED (0x0000000133e7e818)
10690 0x01b70b28:0x007adc29 0:0 1 GRANTED (0x00000000161a3c98)
10690 0x01b70b28:0x007adc28 0:0 1 GRANTED (0x000000017a2606f8)

使用相关命令解锁

storage*> sm_mon -l erpdb.xifenfei.com
storage*> lock status -h

启动数据库恢复正常

查询mos文档,发现相关文章主要有:
NFS Locking Problems Encountered During Database Startup (Doc ID 236794.1)
ORA-1157 ORA-1110 ORA-27086 Starting up Database (Doc ID 145194.1)