2023年11月 – Database SOS

DBV-00107: Unknown header format 故障处理

标题：DBV-00107: Unknown header format 故障处理

客户linux平台被勒索病毒加密,其中有oracle数据库.客户联系黑客进行解密【勒索解密oracle失败】,但是数据库无法正常启动,dbv检查数据库文件报错

[oracle@hisdb ~]$ dbv file=system01.dbf

DBVERIFY: Release 11.2.0.1.0 - Production on 星期一 11月 27 21:49:17 2023

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.


DBV-00107: 未知标头格式 (31) (287942924)

对应的英文为:DBV-00107: Unknown header format (31) (287942924)，检查数据文件信息发现提示为 FILE NOT FOUND,使用脚本为:Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)检测结果

通过分区确认是文件头损坏

修复正确的文件头

再次dbv检查数据文件

[oracle@hisdb ~]$ dbv file=system01.dbf

DBVERIFY: Release 11.2.0.1.0 - Production on 星期一 11月 27 22:05:41 2023

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = /u01/app/oracle/oradata/system01.dbf
页 12800 标记为损坏
Corrupt block relative dba: 0x00403200 (file 1, block 12800)
Bad header found during dbv:
Data in bad block:
 type: 88 format: 1 rdba: 0x33877808
 last change scn: 0x257a.7b3a44e3 seq: 0xe8 flg: 0xe6
 spare1: 0x4e spare2: 0x73 spare3: 0x0
 consistency value in tail: 0x65251001
 check value in block header: 0xc3b4
 computed block checksum: 0x4ca7



DBVERIFY - 验证完成

检查的页总数: 13440
处理的页总数 (数据): 3297
失败的页总数 (数据): 0
处理的页总数 (索引): 2097
失败的页总数 (索引): 0
处理的页总数 (其他): 1441
处理的总页数 (段)  : 1
失败的总页数 (段)  : 0
空的页总数: 6604
标记为损坏的总页数: 1
流入的页总数: 0
加密的总页数        : 0
最高块 SCN            : 1667927064 (12.1667927064)

修复其他文件头,并dbv检查,发现均在12800位置损坏.尝试recover database恢复数据库,报ORA-00742 ORA-00312之类错误.【由于redo损坏,报出来数据文件更多坏块】

Sat Nov 25 17:03:39 2023
ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 40 slaves
Sat Nov 25 17:03:40 2023
Recovery of Online Redo Log: Thread 1 Group 7 Seq 27220 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/redo07.log
Sat Nov 25 17:03:41 2023
Hex dump of (file 3, block 7) in trace file /u01/app/oracle/diag/rdbms/his/his/trace/his_pr0l_52669.trc
Corrupt block relative dba: 0x00c00007 (file 3, block 7)
Bad header found during media recovery
Data in bad block:
 type: 124 format: 7 rdba: 0x1698b845
 last change scn: 0x4fa1.3eaa638f seq: 0x6 flg: 0x24
 spare1: 0x26 spare2: 0x42 spare3: 0x0
 consistency value in tail: 0xa39e1e01
 check value in block header: 0x2ca4
 computed block checksum: 0x3b25
Reading datafile '/u01/app/oracle/oradata/undotbs01.dbf' for corruption at rdba: 0x00c00007 (file 3, block 7)
Reread (file 3, block 7) found same corrupt data (no logical check)
Sat Nov 25 17:03:41 2023
Hex dump of (file 46, block 3) in trace file /u01/app/oracle/diag/rdbms/his/his/trace/his_pr0w_52691.trc
Corrupt block relative dba: 0x0b800003 (file 46, block 3)
Bad header found during media recovery
Data in bad block:
 type: 7 format: 7 rdba: 0x77922022
 last change scn: 0xdff3.c40df5b6 seq: 0x6f flg: 0xe5
 spare1: 0xcd spare2: 0x6d spare3: 0x83d7
 consistency value in tail: 0x63c63d2c
 check value in block header: 0xf662
 computed block checksum: 0xec49
Data in bad block:
 type: 135 format: 4 rdba: 0x45ad2864
 last change scn: 0x9d7e.34949c73 seq: 0x32 flg: 0x3e
 spare1: 0x89 spare2: 0x0 spare3: 0x9f9f
 consistency value in tail: 0xa5807800
 check value in block header: 0xb2c9
 computed block checksum: 0x3aea
Reread (file 5, block 11259) found same corrupt data (no logical check)
 type: 214 format: 1 rdba: 0x0228dbe9
Bad header found during media recovery
 last change scn: 0xed57.ca4f7559 seq: 0x9b flg: 0x4a
Data in bad block:
 spare1: 0x97 spare2: 0x77 spare3: 0x2bab
 type: 33 format: 6 rdba: 0x018d584a
 consistency value in tail: 0x359f90d6
 last change scn: 0xaeb8.2fa361eb seq: 0x60 flg: 0x92
 check value in block header: 0x6b26
 spare1: 0xea spare2: 0xe spare3: 0xb405 block checksum disabled
Reread (file 3, block 4) found same corrupt data (no logical check)
Corrupt block relative dba: 0x0b800e61 (file 46, block 3681)
Bad header found during media recovery
Data in bad block:
 type: 131 format: 6 rdba: 0xc7edd0fc
 last change scn: 0xd319.d0e54941 seq: 0x6f flg: 0x6d
 spare1: 0xe7 spare2: 0x82 spare3: 0x439f
 consistency value in tail: 0x18dc47b6
 check value in block header: 0xe9c8
 computed block checksum: 0x204d
Reread (file 46, block 3681) found same corrupt data (no logical check)
Hex dump of (file 1, block 2017) in trace file /u01/app/oracle/diag/rdbms/his/his/trace/his_pr10_52699.trc
Corrupt block relative dba: 0x004007e1 (file 1, block 2017)
Bad header found during media recovery
Data in bad block:
 type: 159 format: 2 rdba: 0x52c5b2b0
 last change scn: 0x2ed8.e0bc5af9 seq: 0x62 flg: 0xe9
 spare1: 0x81 spare2: 0x1e spare3: 0xda98
 consistency value in tail: 0xc5753dd3
 check value in block header: 0x2bba
 block checksum disabled
Reading datafile '/u01/app/oracle/oradata/system01.dbf' for corruption at rdba: 0x004007e1 (file 1, block 2017)
Reread (file 1, block 2017) found same corrupt data (no logical check)
Media Recovery failed with error 742
Errors in file /u01/app/oracle/diag/rdbms/his/his/trace/his_pr00_52622.trc:
ORA-00283: recovery session canceled due to errors
ORA-00742: Log read detects lost write in thread %d sequence %d block %d
ORA-00312: online log 7 thread 1: '/u01/app/oracle/oradata/redo07.log'
ORA-10877 signalled during: ALTER DATABASE RECOVER  database  ...

尝试强制打开数据库报ORA-600 krsi_al_hdr_update.15,参考:Oracle断电故障处理中有类似报错

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [krsi_al_hdr_update.15],
[4294967295], [], [], [], [], [], [], [], [], [], []

由于redo问题无法resetlogs成功,解决异常redo,再次尝试open库,由于undo坏块无法open成功,报ORA-01092 ORA-01578等错误

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-01578: ORACLE data block corrupted (file # 3, block # 1848)
ORA-01110: data file 3: '/u01/app/oracle/oradata/undotbs01.dbf'
Process ID: 55655
Session ID: 2623 Serial number: 5

解决undo异常,数据库open成功.导出客户需要数据,完成此次恢复工作

AZRECOVERDATA勒索恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：A____Z____RECOVER____DATA勒索恢复

有客户MySQL数据库被黑,业务库中表被删除,并创建A____Z____RECOVER____DATA库,里面有一张readme表,内容为:

mysql> select * from readme \G;
*************************** 1. row ***************************
zh_content: 请尽快与我们取得联系，否则我们将会公布你的数据库信息在网络中，联系邮件：datacenterback@keemail.me
en_content: 请尽快与我们取得联系，否则我们将会公布你的数据库信息在网络中，联系邮件：datacenterback@keemail.me
     email: datacenterback@keemail.me
1 row in set (0.00 sec)

对于这种情况,本质就是mysql drop 库或者drop表级别的恢复,通过反删除软件恢复,可惜恢复效果很差（底层发生了大量的覆盖）

对于这种情况,只能采用底层block级别恢复,通过底层扫描分析

并解析扫描结果恢复需要数据

对于类似这种A____Z____RECOVER____DATA勒索恢复,建议先对系统进行镜像或者快照,然后按照先os层面恢复,在block级别恢复的方法处理,如果无法自行解决,可以联系我们进行技术支持,最大限度抢救和数据,减少损失
电话/微信:17813235971 Q Q:107644445 E-Mail:dba@xifenfei.com
另外建议加强系统和mysql安全加固,数据库尽量不要暴露在公网上

ORA-1200/ORA-1207数据库恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-1200/ORA-1207数据库恢复

由于系统性能问题或者底层io问题,数据库alert日志报一下控制文件损坏错误然后crash掉

Mon Nov 13 08:06:44 2023
Thread 1 advanced to log sequence 12100 (LGWR switch)
  Current log# 1 seq# 12100 mem# 0: /u01/oracle/oradata/xifenfei/redo01.log
Mon Nov 13 09:23:59 2023
********************* ATTENTION: ******************** 
 The controlfile header block returned by the OS
 has a sequence number that is too old. 
 The controlfile might be corrupted.
 PLEASE DO NOT ATTEMPT TO START UP THE INSTANCE 
 without following the steps below.
 RE-STARTING THE INSTANCE CAN CAUSE SERIOUS DAMAGE 
 TO THE DATABASE, if the controlfile is truly corrupted.
 In order to re-start the instance safely, 
 please do the following:
 (1) Save all copies of the controlfile for later 
     analysis and contact your OS vendor and Oracle support.
 (2) Mount the instance and issue: 
     ALTER DATABASE BACKUP CONTROLFILE TO TRACE;
 (3) Unmount the instance. 
 (4) Use the script in the trace file to
     RE-CREATE THE CONTROLFILE and open the database. 
*****************************************************
USER (ospid: 17064): terminating the instance
Mon Nov 13 09:24:00 2023
System state dump requested by (instance=1, osid=17064), summary=[abnormal instance termination].

重启数据库报ORA-01122 ORA-01110 ORA-01207错误

Mon Nov 13 10:11:21 2023
ALTER DATABASE OPEN
Errors in file /u01/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_25824.trc:
ORA-01122: database file 1 failed verification check
ORA-01110: data file 1: '/u01/oracle/oradata/xifenfei/system01.dbf'
ORA-01207: file is more recent than control file - old control file
ORA-1122 signalled during: ALTER DATABASE OPEN...

处理好上述错误之后遭遇ORA-01122 ORA-01110 ORA-01200,类似文章:
bbed处理ORA-01200故障
 ORA-01122 ORA-01200故障处理

Mon Nov 13 10:51:48 2023
alter database open
Read of datafile '/u01/oracle/oradata/xifenfei/sysaux01.dbf' (fno 2) header failed with ORA-01200
Rereading datafile 2 header failed with ORA-01200
Errors in file /u01/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_24148.trc:
ORA-01122: database file 2 failed verification check
ORA-01110: data file 2: '/u01/oracle/oradata/xifenfei/sysaux01.dbf'
ORA-01200: actual file size of 2860800 is smaller than correct size of 2867200 blocks
ORA-1122 signalled during: alter database open...

解决上述错误之后,尝试open库报ORA-00314 ORA-00312之类错误

Mon Nov 13 18:00:43 2023
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 61894 KB redo, 589 data blocks need recovery
Started redo application at
 Thread 1: logseq 12100, block 112760
Recovery of Online Redo Log: Thread 1 Group 1 Seq 12100 Reading mem 0
  Mem# 0: /u01/oracle/oradata/xifenfei/redo01.log
Completed redo application of 1.20MB
Mon Nov 13 18:00:44 2023
Hex dump of (file 2, block 39078) in trace file /u01/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_p011_27469.trc
Reading datafile '/u01/oracle/oradata/xifenfei/sysaux01.dbf' for corruption at rdba: 0x008098a6 (file 2, block 39078)
Reread (file 2, block 39078) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 39078 OF FILE 2
Mon Nov 13 18:00:44 2023
Exception [type: SIGSEGV, Address not mapped to object][ADDR:0xC][PC:0x95FB838, kdxlin()+4946][flags: 0x0, count: 1]
Mon Nov 13 18:00:44 2023
Exception [type: SIGSEGV, Address not mapped to object][ADDR:0xC][PC:0x95FB4DE, kdxlin()+4088][flags: 0x0, count: 1]
Errors in file /u01/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_p011_27469.trc:
ORA-00314: log 2 of thread 1, expected sequence# 12093 doesn't match 12085
ORA-00312: online log 2 thread 1: '/u01/oracle/oradata/xifenfei/redo02.log'
Errors in file /u01/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_p011_27469.trc:
ORA-00314: log 3 of thread 1, expected sequence# 12096 doesn't match 12080
ORA-00312: online log 3 thread 1: '/u01/oracle/oradata/xifenfei/redo03.log'
ORA-00314: log 2 of thread 1, expected sequence# 12093 doesn't match 12085
ORA-00312: online log 2 thread 1: '/u01/oracle/oradata/xifenfei/redo02.log'

后面继续处理遇到类似这样错误

ALTER DATABASE RECOVER    CANCEL  
Errors in file /u01/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_pr00_31110.trc:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/u01/oracle/oradata/xifenfei/system01.dbf'
ORA-1547 signalled during: ALTER DATABASE RECOVER    CANCEL  ...
ALTER DATABASE RECOVER CANCEL 
ORA-1112 signalled during: ALTER DATABASE RECOVER CANCEL ...
Mon Nov 13 19:06:28 2023
alter database open resetlogs
ORA-1194 signalled during: alter database open resetlogs...

最后确认其他数据文件均可recover 成功,只有file 2 无法正常recover

SQL> recover datafile 1;
Media recovery complete.
SQL> recover datafile 2;
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [3020], [2], [950840], [9339448],
[], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 2, block# 950840, file
offset is 3494313984 bytes)
ORA-10564: tablespace SYSAUX
ORA-01110: data file 2: '/u01/oracle/oradata/xifenfei/sysaux01.dbf'
ORA-10560: block type 'FIRST LEVEL BITMAP BLOCK'
SQL> recover datafile 3;
Media recovery complete.
SQL> recover datafile 4;
Media recovery complete.
SQL> recover datafile 5;
Media recovery complete.
SQL> recover datafile 6;
Media recovery complete.
SQL> recover datafile 7;
Media recovery complete.

SQL> recover  datafile 2 allow 1 corruption;
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [3020], [2], [2410240], [10798848],
[], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 2, block# 2410240, file
offset is 2564816896 bytes)
ORA-10564: tablespace SYSAUX
ORA-01110: data file 2: '/u01/oracle/oradata/xifenfei/sysaux01.dbf'
ORA-10560: block type 'FIRST LEVEL BITMAP BLOCK'

通过bbed修改文件头,直接open数据库成功,并协助客户顺利导出数据
参考类似文章:
使用bbed修复损坏datafile header
使用bbed让rac中的sysaux数据文件online

存储断电导致ORA-00325故障恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：存储断电导致ORA-00325故障恢复

由于存储断电,数据库启动最初报错

Reconfiguration complete
 parallel recovery started with 32 processes
Started redo scan
Completed redo scan
 read 22887 KB redo, 2025 data blocks need recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl11/trace/orcl11_ora_25449.trc  (incident=25522):
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [203596], [1537], [3046], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl11/incident/incdir_25522/orcl11_ora_25449_i25522.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Abort recovery for domain 0
Aborting crash recovery due to error 600
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl11/trace/orcl11_ora_25449.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [203596], [1537], [3046], []
Abort recovery for domain 0
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl11/trace/orcl11_ora_25449.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [203596], [1537], [3046], []
ORA-600 signalled during: ALTER DATABASE OPEN...

尝试重建ctl之后,恢复数据库报错ORA-00325错误

[oracle@xifenfei ~]$ oerr ora 325
00325, 00000, "archived log for thread %s, wrong thread # %s in header"
// *Cause:  The archived log is corrupted or for another thread. Can not
//          use the log for applying redo.
// *Action: Find correct archived log.

归档日志文件的头部中的线程编号和实际使用的线程编号不匹配的时候报出的错误。该错误的常见情形是由于线程 ID (thread ID) 没有被正确分配或更新造成的，如从主控制档（control file）恢复用户表空间后更改线程 ID 但系统没有知道，导致系统在访问归档日志文件时，由于归档日志文件的头部中的线程编号与实际使用的线程编号不匹配，而出现该错误。

尝试强制拉库,报ORA-600 2662错误

尝试修改scn继续启动库报ORA-600 kclchkblk_4

继续调整scn,数据库open成功.后续就是各种收尾工作(比如undo回滚段,tempfile,各种ORA-600错误等)

重建ctl遭遇ORA-01163 ORA-01110和ORA-600 kccccfl_1错误

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：重建ctl遭遇ORA-01163 ORA-01110和ORA-600 kccccfl_1错误

有客户联系我,数据库强制关闭之后无法正常启动,通过他们一系列尝试,数据库报open 报ORA-01555错误

通过分析确认是由于客户屏蔽一致性强制拉库失败导致

Wed Nov 01 19:49:03 2023
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 1199697777
Archived Log entry 68106 added for thread 1 sequence 3829 ID 0x761a769a dest 1:
Archived Log entry 68107 added for thread 1 sequence 3830 ID 0x761a769a dest 1:
Archived Log entry 68108 added for thread 1 sequence 3828 ID 0x761a769a dest 1:
Clearing online redo logfile 1 /u01/app/oracle/oradata/ORACLE/redo01.log
Clearing online log 1 of thread 1 sequence number 3829
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 /u01/app/oracle/oradata/ORACLE/redo02.log
Clearing online log 2 of thread 1 sequence number 3830
Clearing online redo logfile 2 complete
Clearing online redo logfile 3 /u01/app/oracle/oradata/ORACLE/redo03.log
Clearing online log 3 of thread 1 sequence number 3828
Clearing online redo logfile 3 complete
Resetting resetlogs activation ID 1981445786 (0x761a769a)
Online log /u01/app/oracle/oradata/ORACLE/redo01.log: Thread 1 Group 1 was previously cleared
Online log /u01/app/oracle/oradata/ORACLE/redo02.log: Thread 1 Group 2 was previously cleared
Online log /u01/app/oracle/oradata/ORACLE/redo03.log: Thread 1 Group 3 was previously cleared
Wed Nov 01 19:49:09 2023
Setting recovery target incarnation to 3
Wed Nov 01 19:49:09 2023
Assigning activation ID 2024817472 (0x78b04340)
LGWR: STARTING ARCH PROCESSES
Wed Nov 01 19:49:09 2023
ARC0 started with pid=21, OS id=2078 
ARC0: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Wed Nov 01 19:49:10 2023
ARC1 started with pid=22, OS id=2080 
Thread 1 advanced to log sequence 2 (thread open)
Wed Nov 01 19:49:10 2023
ARC2 started with pid=23, OS id=2082 
Wed Nov 01 19:49:10 2023
ARC3 started with pid=24, OS id=2084 
ARC1: Archival started
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
Thread 1 opened at log sequence 2
  Current log# 2 seq# 2 mem# 0: /u01/app/oracle/oradata/ORACLE/redo02.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Nov 01 19:49:10 2023
SMON: enabling cache recovery
ARC2: LGWR is scheduled to archive destination LOG_ARCHIVE_DEST_2 after log switch
Wed Nov 01 19:49:10 2023
NSA2 started with pid=25, OS id=2086 
ORA-01555 caused by SQL statement below (SQL ID: 4krwuz0ctqxdt, SCN: 0x0000.4781ef78):
select ctime, mtime, stime from obj$ where obj# = :1
Errors in file /u01/app/oracle/diag/rdbms/oracle_pd/ORACLE/trace/ORACLE_ora_2076.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 11 with name "_SYSSMU11_467363169$" too small
Errors in file /u01/app/oracle/diag/rdbms/oracle_pd/ORACLE/trace/ORACLE_ora_2076.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 11 with name "_SYSSMU11_467363169$" too small
Error 704 happened during db open, shutting down database
USER (ospid: 2076): terminating the instance due to error 704
Instance terminated by USER, pid = 2076
ORA-1092 signalled during: alter database open resetlogs...
opiodr aborting process unknown ospid (2076) as a result of ORA-1092

这种错误比较常见恢复简单,以前有过类似的文章:
在数据库open过程中常遇到ORA-01555汇总
 数据库open过程遭遇ORA-1555对应sql语句补充
 重建control遗漏数据文件,reseltogs报ORA-1555错误处理
 Oracle Recovery Tools恢复—ORA-00704 ORA-01555故障
这个错误恢复本身不难,只是在数据库的恢复过程中需要重建文件,在这次的重建文件中由于ctl语句错误【语句是通过sql查询拼接而成】,导致重建文件报了两个错误,供参考:
ORA-01163 ORA-01110错误

SQL> CREATE CONTROLFILE REUSE DATABASE "ORACLE" NORESETLOGS  NOARCHIVELOG
        MAXLOGFILES 50
       MAXLOGMEMBERS 5
       MAXDATAFILES 1000
       MAXINSTANCES 8
       MAXLOGHISTORY 2920
LOGFILE
       group 1   '/u01/app/oracle/oradata/ORACLE/redo01.log' size 200M,
  2         group 3   '/u01/app/oracle/oradata/ORACLE/redo03.log' size 200M,
       group 2   '/u01/app/oracle/oradata/ORACLE/redo02.log' size 200M,
DATAFILE
        '/u01/app/oracle/oradata/ORACLE/system01.dbf',
        …………
        '/u01/app/oracle/oradata/ORACLE/xifenfei06.dbf',
CHARACTER SET  ZHS16GBK ;
 23   24   25   26   27  CREATE CONTROLFILE REUSE DATABASE "ORACLE" NORESETLOGS	NOARCHIVELOG
*
ERROR at line 1:
ORA-01503: CREATE CONTROLFILE failed
ORA-01163: SIZE clause indicates 12800 (blocks), but should match header
1564672
ORA-01110: data file 15: '/u01/app/oracle/oradata/ORACLE/xifenfei06.dbf'

出现该错误是由于我写路径的时候在最后一个数据文件路径后面多了一个【,】导致

ORA-600 kccccfl_1错误

SQL> CREATE CONTROLFILE REUSE DATABASE "ORACLE" NORESETLOGS  NOARCHIVELOG
        MAXLOGFILES 50
       MAXLOGMEMBERS 5
       MAXDATAFILES 1000
       MAXINSTANCES 8
       MAXLOGHISTORY 2920
LOGFILE
       group 1   '/u01/app/oracle/oradata/ORACLE/redo01.log'  size 200M,
       group 3   '/u01/app/oracle/oradata/ORACLE/redo03.log' size 200M,
       group 2   '/u01/app/oracle/oradata/ORACLE/redo02.log' size 200M,
DATAFILE
        '/u01/app/oracle/oradata/ORACLE/system01.dbf',
        ………… 
        '/u01/app/oracle/oradata/ORACLE/xifenfei06.dbf'
CHARACTER SET  ZHS16GBK ;
 23   24   25   26   27  CREATE CONTROLFILE REUSE DATABASE "ORACLE" NORESETLOGS	NOARCHIVELOG
*
ERROR at line 1:
ORA-01503: CREATE CONTROLFILE failed
ORA-00600: internal error code, arguments: [kccccfl_1], [], [], [], [], [], [],
[], [], [], [], []

出现该错误是由于在最后redo文件路径后面多了【,】导致

RFS[22349]: Database mount ID mismatch

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：RFS[22349]: Database mount ID mismatch

今天检查数据库发现一套主库上面报大量RFS[22349]: Database mount ID mismatch之类的错误

[oracle@hisdb2 trace]$ tail -f alert_hisdb2.log 
LNS: Standby redo logfile selected for thread 2 sequence 133319 for destination LOG_ARCHIVE_DEST_2
Mon Nov 06 20:56:09 2023
RFS[27080]: Assigned to RFS process 15518
RFS[27080]: Database mount ID mismatch [0x7afb8c50:0x70b2cf7b] (2063305808:1890766715)
Mon Nov 06 20:57:09 2023
RFS[27081]: Assigned to RFS process 16299
RFS[27081]: Database mount ID mismatch [0x7afb8c50:0x70b2cf7b] (2063305808:1890766715)
Mon Nov 06 20:57:09 2023
RFS[27082]: Assigned to RFS process 16319
RFS[27082]: Database mount ID mismatch [0x7afb8c50:0x70b2cf7b] (2063305808:1890766715)

这种错误表示rfs进程在传输的时候检测到database mount id不匹配(也就是认为这个库不是主库该传输或者接受日志的),出现类似这样的错误,大概率是由于这个dg成员(主库和备库)中应该有不匹配的库,回想这个库近期的操作,突然想到对这个主库在虚拟化平台的备库做了一次克隆,然后吧克隆库激活作为测试库的操作.基于此种情况,先判断该库是否和上次克隆的库之间创建有会话连接

--主库
[oracle@hisdb2 trace]$ netstat -natp|grep 192.168.106.63
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 192.168.115.12:52785        192.168.106.63:1521         ESTABLISHED -                   
tcp        0      0 192.168.115.14:1521         192.168.106.63:21598        TIME_WAIT   - 

--克隆库
[oracle@HIS_DG ~]$ netstat -atpn|grep 192.168.115.12
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 ::ffff:192.168.106.63:1521  ::ffff:192.168.115.12:52785 ESTABLISHED 6126/oraclehisdb

两个库之间确实存在会话,检查克隆库相关的dg配置

[oracle@HIS_DG ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Mon Nov 6 21:01:05 2023

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> show parameter fal;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
fal_client                           string      HISDBDG
fal_server                           string      HISDB
SQL> select open_mode,database_role from v$database;

OPEN_MODE            DATABASE_ROLE
-------------------- ----------------
READ WRITE           PRIMARY

SQL> show parameter archive;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
archive_lag_target                   integer     1800
log_archive_config                   string
log_archive_dest                     string
log_archive_dest_1                   string      LOCATION=USE_DB_RECOVERY_FILE_
                                                 DEST valid_for=(all_logfiles,a
                                                 ll_roles) db_unique_name=hisdb
                                                 dg
log_archive_dest_10                  string
log_archive_dest_11                  string
log_archive_dest_12                  string
log_archive_dest_13                  string

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_14                  string
log_archive_dest_15                  string
log_archive_dest_16                  string
log_archive_dest_17                  string
log_archive_dest_18                  string
log_archive_dest_19                  string
log_archive_dest_2                   string      service=hisdb lgwr async valid
                                                 _for=(online_logfiles,primary_
                                                 role) db_unique_name=hisdb

该库的dg配置没有清除,当该库变为主库时,log_archive_dest_2刚好指向主库,从而使得主库上出现了类似RFS[22349]: Database mount ID mismatch的错误.处理方法是清除掉克隆库上面dg备库相关配置

SQL> alter system set log_archive_dest_2='';

System altered.

SQL> alter system set fal_server='';

System altered.

主库日志中未再出现类似错误

Mon Nov 06 20:48:08 2023
RFS[22349]: Assigned to RFS process 7383
RFS[22349]: Database mount ID mismatch [0x7afb8c50:0x70b2cf7b] (2063305808:1890766715)
[oracle@hisdb1 trace]$ tail -f alert_hisdb1.log 
RFS[22352]: Database mount ID mismatch [0x7afb8c50:0x70b2cf7b] (2063305808:1890766715)
Mon Nov 06 20:58:09 2023
RFS[22353]: Assigned to RFS process 14958
RFS[22353]: Database mount ID mismatch [0x7afb8c50:0x70b2cf7b] (2063305808:1890766715)
Mon Nov 06 21:01:09 2023
RFS[22354]: Assigned to RFS process 18580
RFS[22354]: Database mount ID mismatch [0x7afb8c50:0x70b2cf7b] (2063305808:1890766715)
Mon Nov 06 21:01:09 2023
RFS[22355]: Assigned to RFS process 18598
RFS[22355]: Database mount ID mismatch [0x7afb8c50:0x70b2cf7b] (2063305808:1890766715)
Mon Nov 06 21:03:28 2023
Thread 1 advanced to log sequence 129769 (LGWR switch)
  Current log# 9 seq# 129769 mem# 0: +DATA/hisdb/onlinelog/group_9.276.976991877
  Current log# 9 seq# 129769 mem# 1: +FRA/hisdb/onlinelog/group_9.2334.976991877
Mon Nov 06 21:03:28 2023
Archived Log entry 524082 added for thread 1 sequence 129768 ID 0x70864b41 dest 1:
Mon Nov 06 21:03:28 2023
LNS: Standby redo logfile selected for thread 1 sequence 129769 for destination LOG_ARCHIVE_DEST_2

事后在MOS上有一篇文档供参考:Database mount ID mismatch ORA-16009: invalid redo transport destination (Doc ID 1450132.1)

触发器找出密码错误应用

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：触发器找出密码错误应用

经常会遇到修改oracle业务用户密码之后,由于部分应用密码没有被正确修改(忘记修改,或者修改错误了),导致数据库被hang或者用户被锁的情况,对于这样的情况,通过一个logon触发器可以快速找到

create or replace trigger logon_denied_to_alert
after servererror on database
declare
 message   varchar2(120);
 IP        varchar2(15);
 v_os_user varchar2(80);
 v_module  varchar2(50);
 v_action  varchar2(50);
 v_pid     varchar2(10);
 v_sid     number;
 v_username  varchar2(50);
 v_suser      varchar2(50);
begin
 IF (ora_is_servererror(1017)) THEN
   if sys_context('userenv', 'network_protocol') = 'tcp' then
     IP := sys_context('userenv', 'ip_address');
   else
     select distinct sid into v_sid from sys.v_$mystat;
     SELECT p.SPID
       into v_pid
       FROM V$PROCESS p, V$SESSION v
      WHERE p.ADDR = v.PADDR
        AND v.sid = v_sid;
   end if;
   v_os_user := sys_context('userenv', 'os_user');
   v_username := sys_context('userenv', 'CURRENT_USER');
   v_suser := SYS_CONTEXT('USERENV','SESSION_USER');
   dbms_application_info.READ_MODULE(v_module, v_action);
   message := to_char(sysdate, 'Dy Mon dd HH24:MI:SS YYYY') ||
              ' logon denied from '||v_username||' '||v_suser||' '|| nvl(IP, v_pid) || ' ' || v_os_user ||
              ' with ' || v_module || ' ' || v_action;
   sys.dbms_system.ksdwrt(2, message);
 end if;
end;
/

测试

SQL> create user test identified by oracle;

User created.

SQL> grant dba to test;

Grant succeeded.

直接本地登录

SQL> conn test/test
ERROR:
ORA-01017: invalid username/password; logon denied


Warning: You are no longer connected to ORACLE.

---alert日志记录
Wed Nov 01 23:15:04 2023 logon denied from SYS  12886 oracle with sqlplus@iZbp1hx0enix3hix1kvyrxZ (TNS V1-V3)

通过tns登录

SQL> conn  test/oracl1@172.27.54.81:1522/orcl
ERROR:
ORA-01017: invalid username/password; logon denied


Warning: You are no longer connected to ORACLE.

---alert日志报错
Wed Nov 01 23:31:51 2023 logon denied from SYS  172.27.54.81 oracle with sqlplus@iZbp1hx0enix3hix1kvyrxZ (TNS V1-V3)

月度归档：11月 2023