联系：手机/微信(+86 17813235971) QQ(107644445)
标题：ORA-600 kcffo_online_pdb_check: fno_system 和 ORA-600 kcvfdb_pdb_set_clean_scn: cleanckpt错误
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
在做18c模拟故障测试中,经过自己一系列折腾,主要遭遇了ORA-600 kcffo_online_pdb_check: fno_system 和 ORA-600 kcvfdb_pdb_set_clean_scn: cleanckpt错误,这些都是pdb特有的,主要是由于一些bug引起,在非pdb环境中不太可能遇到.其实这也就是说明由于pdb机制的引入,使得后续的数据库异常恢复中会更加复杂.
18c数据库open ORA-00603 ORA-01092 ORA-00600报错
[oracle@ora11g tmp]$ ss
SQL*Plus: Release 18.0.0.0.0 - Production on Sat Apr 20 21:18:09 2019
Version 18.3.0.0.0
Copyright (c) 1982, 2018, Oracle. All rights reserved.
Connected to:
Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.3.0.0.0
SQL> select * from v$version;
BANNER
--------------------------------------------------------------------------------
BANNER_FULL
--------------------------------------------------------------------------------
BANNER_LEGACY
--------------------------------------------------------------------------------
CON_ID
----------
Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.3.0.0.0
Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
0
BANNER
--------------------------------------------------------------------------------
BANNER_FULL
--------------------------------------------------------------------------------
BANNER_LEGACY
--------------------------------------------------------------------------------
CON_ID
----------
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [], [], [], [], [], [], [], [], [],
[], [], []
Process ID: 55775
Session ID: 135 Serial number: 49652
alert日志信息
Database Characterset is AL32UTF8 2019-04-20T18:20:54.256841+08:00 No Resource Manager plan active 2019-04-20T18:20:56.751241+08:00 replication_dependency_tracking turned off (no async multimaster replication found) 2019-04-20T18:20:57.862516+08:00 Starting background process AQPC 2019-04-20T18:20:58.341991+08:00 AQPC started with pid=45, OS id=55830 2019-04-20T18:21:01.476252+08:00 PDB$SEED(2):Autotune of undo retention is turned on. 2019-04-20T18:21:01.738732+08:00 Pdb PDB$SEED hit error 1157 during open read only (2) and will be closed. 2019-04-20T18:21:01.755310+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/trace/orcl18c_ora_55775.trc: ORA-01157: cannot identify/lock data file 5 - see DBWR trace file ORA-01110: data file 5: '/u01/app/oracle/oradata/ORCL18C/pdbseed/system01.dbf' PDB$SEED(2):JIT: pid 55775 requesting stop PDB$SEED(2):Buffer Cache flush deferred for PDB 2 Could not open PDB$SEED error=1157 2019-04-20T18:21:01.887601+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/trace/orcl18c_ora_55775.trc: ORA-01157: cannot identify/lock data file 5 - see DBWR trace file ORA-01110: data file 5: '/u01/app/oracle/oradata/ORCL18C/pdbseed/system01.dbf' 2019-04-20T18:21:03.385503+08:00 PDB1(3):Autotune of undo retention is turned on. Errors in file /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/trace/orcl18c_p000_55808.trc (incident=66865) (PDBNAME=CDB$ROOT): ORA-00600: internal error code, arguments: [kcffo_online_pdb_check: fno_system], [3], [], [], [], [], [], [], [], [], [], [] 2019-04-20T18:21:03.682428+08:00 PDB2(4):Autotune of undo retention is turned on. Incident details in: /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/incident/incdir_66865/orcl18c_p000_55808_i66865.trc 2019-04-20T18:21:12.863880+08:00 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. 2019-04-20T18:21:12.879921+08:00 Pdb PDB1 hit error 600 during open read write (5) and will be closed. 2019-04-20T18:21:12.880506+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/trace/orcl18c_p000_55808.trc: ORA-00600: internal error code, arguments: [kcffo_online_pdb_check: fno_system], [3], [], [], [], [], [], [], [], [], [], [] PDB1(3):JIT: pid 55808 requesting stop 2019-04-20T18:21:12.915407+08:00 Dumping diagnostic data in directory=[cdmp_20190420182112], requested by (instance=1, osid=55808 (P000)), summary=[incident=66865]. 2019-04-20T18:21:12.989890+08:00 PDB1(3):Buffer Cache flush deferred for PDB 3 2019-04-20T18:21:13.004575+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/trace/orcl18c_p001_55810.trc (incident=66873) (PDBNAME=CDB$ROOT): ORA-00600: internal error code, arguments: [kcffo_online_pdb_check: fno_system], [4], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/incident/incdir_66873/orcl18c_p001_55810_i66873.trc 2019-04-20T18:21:17.218642+08:00 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. 2019-04-20T18:21:17.222057+08:00 Pdb PDB2 hit error 600 during open read write (5) and will be closed. 2019-04-20T18:21:17.222236+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/trace/orcl18c_p001_55810.trc: ORA-00600: internal error code, arguments: [kcffo_online_pdb_check: fno_system], [4], [], [], [], [], [], [], [], [], [], [] 2019-04-20T18:21:17.260941+08:00 Dumping diagnostic data in directory=[cdmp_20190420182117], requested by (instance=1, osid=55810 (P001)), summary=[incident=66873]. 2019-04-20T18:21:17.262023+08:00 PDB2(4):JIT: pid 55810 requesting stop PDB2(4):Buffer Cache flush deferred for PDB 4 2019-04-20T18:21:17.352939+08:00 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. 2019-04-20T18:21:17.483695+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/trace/orcl18c_ora_55775.trc (incident=66857) (PDBNAME=CDB$ROOT): ORA-00600: internal error code, arguments: [kcffo_online_pdb_check: fno_system], [3], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/incident/incdir_66857/orcl18c_ora_55775_i66857.trc 2019-04-20T18:21:22.612339+08:00 ***************************************************************** An internal routine has requested a dump of selected redo. This usually happens following a specific internal error, when analysis of the redo logs will help Oracle Support with the diagnosis. It is recommended that you retain all the redo logs generated (by all the instances) during the past 12 hours, in case additional redo dumps are required to help with the diagnosis. ***************************************************************** 2019-04-20T18:21:26.062635+08:00 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/trace/orcl18c_ora_55775.trc (incident=66858) (PDBNAME=CDB$ROOT): ORA-00600: internal error code, arguments: [], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/incident/incdir_66858/orcl18c_ora_55775_i66858.trc 2019-04-20T18:21:26.506644+08:00 Dumping diagnostic data in directory=[cdmp_20190420182126], requested by (instance=1, osid=55775), summary=[incident=66857]. 2019-04-20T18:21:30.119381+08:00 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. 2019-04-20T18:21:30.119505+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/trace/orcl18c_ora_55775.trc: ORA-00600: internal error code, arguments: [], [], [], [], [], [], [], [], [], [], [], [] 2019-04-20T18:21:30.119629+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/trace/orcl18c_ora_55775.trc: ORA-00600: internal error code, arguments: [], [], [], [], [], [], [], [], [], [], [], [] 2019-04-20T18:21:30.119719+08:00 Error 600 happened during db open, shutting down database Errors in file /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/trace/orcl18c_ora_55775.trc (incident=66859) (PDBNAME=CDB$ROOT): ORA-00603: ORACLE server session terminated by fatal error ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00600: internal error code, arguments: [], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/incident/incdir_66859/orcl18c_ora_55775_i66859.trc 2019-04-20T18:21:30.346811+08:00 Dumping diagnostic data in directory=[cdmp_20190420182130], requested by (instance=1, osid=55775), summary=[incident=66858]. 2019-04-20T18:21:34.551418+08:00 opiodr aborting process unknown ospid (55775) as a result of ORA-603 2019-04-20T18:21:34.720885+08:00 ORA-603 : opitsk aborting process License high water mark = 4 2019-04-20T18:21:34.754766+08:00 USER (ospid: 55775): terminating the instance due to ORA error 600 2019-04-20T18:21:35.839992+08:00 Instance terminated by USER, pid = 55775
alert日志提示文件不存在，实际上文件是存在的
[root@ora11g ~]# [root@ora11g ~]# ls -l /u01/app/oracle/oradata/ORCL18C/pdbseed/system01.dbf -rw-r-----. 1 oracle oinstall 283123712 4月 13 23:59 /u01/app/oracle/oradata/ORCL18C/pdbseed/system01.dbf
主要错误是ORA-600 kcffo_online_pdb_check: fno_system 查询mos发现主要是由于数据库bug导致,查询mos发现不少bug
![KCFFO_ONLINE_PDB_CHECK FNO_SYSTEM]](http://www.xifenfei.com/wp-content/uploads/2019/04/KCFFO_ONLINE_PDB_CHECK-FNO_SYSTEM.jpg)
官方解释,主要是由于kcffo_online_pdb函数执行异常导致
Function kcffo_online_pdb_check Check if it ok to online the files in a pluggable database. An error is signalled if it is not ok. This routine will grab a file enqueue for the relevant files and save it in the NULL-terminated array fenqsp. The caller must release these after kcffo_online_pdb() or if an error occurs.
通过人工修改文件状态,绕过该错误，cdb数据库open成功，但是pdb依旧无法正常open
SQL> alter database open; Database altered. SQL> alter session set container=PDB1; Session altered. SQL> alter database open; alter database open * ERROR at line 1: ORA-00600: internal error code, arguments: [kcvfdb_pdb_set_clean_scn: cleanckpt], [3], [1739494], [38655308813], [2], [], [], [], [], [], [], []
alert日志
PDB1(3):alter database open PDB1(3):Autotune of undo retention is turned on. Errors in file /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/trace/orcl18c_ora_56178.trc (incident=69185) (PDBNAME=CDB$ROOT): ORA-00600: internal error code, arguments: [kcvfdb_pdb_set_clean_scn: cleanckpt], [3], [1739494], [38655308813], [2], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/incident/incdir_69185/orcl18c_ora_56178_i69185.trc 2019-04-20T18:31:41.761479+08:00 ***************************************************************** An internal routine has requested a dump of selected redo. This usually happens following a specific internal error, when analysis of the redo logs will help Oracle Support with the diagnosis. It is recommended that you retain all the redo logs generated (by all the instances) during the past 12 hours, in case additional redo dumps are required to help with the diagnosis. ***************************************************************** 2019-04-20T18:31:42.097465+08:00 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Pdb PDB1 hit error 600 during open read write (1) and will be closed. 2019-04-20T18:31:42.097847+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl18c/orcl18c/trace/orcl18c_ora_56178.trc: ORA-00600: internal error code, arguments: [kcvfdb_pdb_set_clean_scn: cleanckpt], [3], [1739494], [38655308813], [2], [], [], [], [], [], [], [] PDB1(3):JIT: pid 56178 requesting stop 2019-04-20T18:31:42.098808+08:00 Dumping diagnostic data in directory=[cdmp_20190420183142], requested by (instance=1, osid=56178), summary=[incident=69185]. 2019-04-20T18:31:42.138818+08:00 PDB1(3):Buffer Cache flush deferred for PDB 3 PDB1(3):ORA-600 signalled during: alter database open...
主要错误是ORA-600 kcvfdb_pdb_set_clean_scn: cleanckpt，通过查询mos，依旧发现mos上有的主要可能的bug
通过人工修改数据文件的checkpoint scn解决该问题，pdb open成功
