Oracle – 第8页 – Database SOS

.ncov加密oracle数据库恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：.ncov加密oracle数据库恢复

有朋友oracle数据库文件被加密,后缀名为:.id-09C1B27D.[3441546223@qq.com].ncov,

通过分析,运气不错,这个病毒只是加密了少量的oracle block,通过底层分析,该数据库可以open成功

通过一系列处理,数据库open,数据使用expdp导出

对于该类型加密,我们可以对sql server、mysql、oracle恢复出来绝大多数数据,通过不向黑客交赎金的方式,实现绝绝大部分业务数据恢复.

再次提醒plsql dev勒索事件

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：再次提醒plsql dev勒索事件

在2016年发现plsql dev被注入恶意脚本,导致数据库被破坏的事件,我当时写过相关分析blog(plsql dev引起的数据库被黑勒索比特币实现原理分析和解决方案),最近又接到两个客户类似故障请求,分享这次处理经历,提醒大家注意
aler日志报错

Mon Oct 21 16:13:06 2019
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_68593.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-20315: 你的数据库已被SQL RUSH Team锁死  发送5个比特币到这个地址 166xk1FXMB2g8JxBVF5T4Aw1Z5JaZ6vrSE (大小写一致)
  之后把你的Oracle SID邮寄地址 sqlrush@mail.com 我们将让你知道如何解锁你的数据库
Hi buddy, your database was hacked by SQL RUSH Team, send 5 bitcoin to address
166xk1FXMB2g8JxBVF5T4Aw1Z5JaZ6vrSE (case sensitive),  after that send your Oracle SID to mail
address sqlrush@mail.com, we will let you know how to unlock your database.
ORA-06512: at "XIFENFEI.DBMS_CORE_INTERNAL         ", line 25
ORA-06512: at line 2

证明该库在2019年10月份就已经被注入恶意脚本,只是由于该库无专业维护,没有定期检查,不然该问题再就被发现.直到前几天有数据丢失影响业务运行,才开始找原因,发现该问题.

查询相关恶意脚本创建时间

SQL> select owner||'.'||object_name,to_char(created,'yyyy-mm-dd hh24:mi:ss')
 2  from dba_objects where object_name like 'DBMS_%_INTERNAL% ';
OWNER||'.'||OBJECT_NAME
-----------------------------------------------
TO_CHAR(CREATED,'YY
-------------------
XFF.DBMS_SUPPORT_INTERNAL
2019-11-19 11:29:38
XFF.DBMS_SUPPORT_INTERNAL
2019-11-19 11:29:38
XFF.DBMS_SYSTEM_INTERNAL
2019-11-19 11:29:39
XFF.DBMS_SYSTEM_INTERNAL
2019-11-19 11:29:39
XFF.DBMS_CORE_INTERNAL
2019-11-19 11:29:39
XFF.DBMS_CORE_INTERNAL
2019-11-19 11:29:39
XIFENFEI.DBMS_SUPPORT_INTERNAL
2019-10-21 15:49:06
XIFENFEI.DBMS_SUPPORT_INTERNAL
2019-10-21 15:49:06
XIFENFEI.DBMS_SYSTEM_INTERNAL
2019-10-21 15:49:06
XIFENFEI.DBMS_SYSTEM_INTERNAL
2019-10-21 15:49:06
XIFENFEI.DBMS_CORE_INTERNAL
2019-10-21 15:49:06
XIFENFEI.DBMS_CORE_INTERNAL
2019-10-21 15:49:06
12 rows selected.

证明在xifenfei和xff用户下面均已经被注入了恶意脚本(也可以通过alert日志分析出来类似结论)

生产truncate表job

SQL> select count(*) from dba_jobs;
  COUNT(*)
----------
  50283485
SQL> select count(job) from dba_jobs
where what like '%DBMS_STANDARD_FUN9%';  2
COUNT(JOB)
----------
  50283483

证明该库本身自带job只有2个,剩余全部是恶意脚本生成的job.

处理思路
1.保护现场:停掉监听,kill所有业务会话
2.job处理:禁止数据库启动任何job,kill已经启动job,清除异常job

begin
for i in (select job from dba_jobs
where what like '%DBMS_STANDARD_FUN9%' )
loop
dbms_ijob.remove(i.job);
commit;
end loop;
end;
/

3. 清理恶意脚本

select 'DROP TRIGGER '||owner||'."'||TRIGGER_NAME||'";'
from dba_triggers where TRIGGER_NAME like 'DBMS_%_INTERNAL% '
union all
select 'DROP PROCEDURE '||owner||'."'||a.object_name||'";'
from dba_procedures a where a.object_name like 'DBMS_%_INTERNAL% ';

4.分析异常表:通过查询相关视图和业务数据,分析哪些表异常,需要对其进行数据恢复(一定要确认恢复数据ok之后才能够导入,不然可能导致原环境破坏,彻底无法恢复)
再次提醒检查plsql dev工具的afterconnect.sql脚本,请从正规途径下载oracle工具和软件(警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703)如果不幸数据库被感染此种勒索比特币事件,而且无法自行恢复的,可以联系我们给予技术支持
Phone:17813235971 Q Q:107644445 E-Mail:dba@xifenfei.com

oracle to sql 透明网络配置

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：oracle to sql 透明网络配置

一直比较抵触oracle 透明网关,最近有朋友说他们客户坚持要使用这个(不想修改程序),无奈之下进行了配置.
透明网关需求
Oracle透明网关链接sqlserver

Oracle：
OS：Red Hat Enterprise Linux Server release 6.8
DB：Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit
Ip：192.168.222.11

Sqlserver
OS：windows server 2003 x86
DB：sqlserver2005
IP：192.168.222.1
实例名称：MSSQLSERVER
指向的DB：xifenfei

安装 Gateway软件
使用oracle用户安装，并且使用空目录(不建议使用oracle_home目录)
配置oracle_base和oracle_home
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/product/11.2.0/wangguan

配置Gateway数据库连接信息

[oracle@ora11g admin]$ pwd
/u01/app/oracle/product/11.2.0/wangguan/dg4msql/admin
[oracle@ora11g admin]$  cat initdg4msql.ora
# This is a customized agent init file that contains the HS parameters
# that are needed for the Database Gateway for Microsoft SQL Server
#
# HS init parameters
#
HS_FDS_CONNECT_INFO=[192.168.222.1]:1433//xifenfei
# alternate connect format is hostname/serverinstance/databasename
HS_FDS_TRACE_LEVEL=OFF
HS_FDS_RECOVERY_ACCOUNT=RECOVER
HS_FDS_RECOVERY_PWD=RECOVER

配置Gateway监听配置

[oracle@ora11g admin]$ pwd
/u01/app/oracle/product/11.2.0/wangguan/network/admin
[oracle@ora11g admin]$ cat listener.ora
# Generated by Oracle configuration tools.
LISTENERSQL =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = ora11g)(PORT = 1522))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1522))
    )
  )
SID_LIST_LISTENERSQL =
  (SID_LIST =
    (SID_DESC=
      (SID_NAME = dg4msql)
      (ORACLE_HOME = /u01/app/oracle/product/11.2.0/wangguan)
      (PROGRAM = dg4msql)
    )
  )
ADR_BASE_LISTENERSQL = /u01/app/oracle

配置Gateway tns信息
在oracle数据库的oracle_home中配置,如果有设置tns_admin,在该目录下处理

[oracle@ora11g admin]$ pwd
/u01/app/oracle/product/11.2.0/db_1/network/admin
[oracle@ora11g admin]$ cat tnsnames.ora
# Generated by Oracle configuration tools.
tomssql =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.222.11)(PORT = 1522))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = dg4msql)
    )
     (HS = OK)
  )
[oracle@ora11g admin]$ tnsping tomssql
TNS Ping Utility for Linux: Version 11.2.0.4.0 - Production on 09-JAN-2020 09:10:06
Copyright (c) 1997, 2013, Oracle.  All rights reserved.
Used parameter files:
/u01/app/oracle/product/11.2.0/db_1/network/admin/sqlnet.ora
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.222.11)(PORT = 1522))
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dg4msql)) (HS = OK))
OK (20 msec)

创建dblink to sql

SQL> create public database link to_sql2012 connect to sa identified by "sa" using 'tomssql';
Database link created.

验证Gateway
sql server中查询

oracle 通过gateway查询

至此oracle to sql server Gateway 配置成功.如果需要配置多个$ORACLE_HOME/dg4msql/admin/init网关SID.ora文件并新增多个静态监听和tns指向网关SID即可

ORACLE中比较常见的一些错误的解释

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORACLE中比较常见的一些错误的解释

ORA-1578	ORA-1578 The data block indicated was corrupt. This was a physical corruption, also called a media corruption. The cause is unknown but is most likely external to the database. If ORA-26040 is also signaled, the corruption is due to NOLOGGING or UNRECOVERABLE operations.
ORA-1410	This error is raised when an operation refers to a ROWID in a table for which there is no such row. The reference to a ROWID may be implicit from a WHERE CURRENT OF clause or directly from a WHERE ROWID=… clause. ORA-1410 indicates the ROWID is for a BLOCK that is not part of this table.
ORA-8103	The object has been deleted by another user since the operation began; example: another session truncated or dropped the segment while the SQL statement was still active. If the error is reproducible, following may be the reasons: a.) The header block has an invalid block type. b.) The data_object_id (seg/obj) stored in the block is different than the data_object_id stored in the segment header. See dba_objects.data_object_id and compare it to the decimal value stored in the block (field seg/obj).
ORA-8102	An ORA-08102 indicates that there is a mismatch between the key(s) stored in the index and the values stored in the table. What typically happens is the index is built and at some future time, some type of corruption occurs, either in the table or index, to cause the mismatch.
ORA-1498	Generally this is a result of an ANALYZE … VALIDATE … command. This error generally manifests itself when there is inconsistency in the data/Index block. Some of the block check errors that may be found:- a.) Row locked by a non-existent transaction b.) The amount of space used is not equal to block size c.) Transaction header lock count mismatch. While support are processing the tracefile it may be worth the re-running the ANALYZE after restarting the database to help show if the corruption is consistent or if it ‘moves’. Send the tracefile to support for analysis. If the ANALYZE was against an index you should check the whole object. Eg: Find the tablename and execute: ANALYZE TABLE xxx VALIDATE STRUCTURE CASCADE;
ORA-1499	An error occurred when validating an index or a table using the ANALYZE command. One or more entries does not point to the appropriate cross-reference.
ORA-752	Media recovery detected a lost write of a data block. A data block write to storage was lost during normal database operation on the primary database.
ORA-26040	Trying to access data in block that was loaded without redo generation using the NOLOGGING/UNRECOVERABLE option. This Error raises always together with ORA-1578
ORA-600 [12700]	Oracle is trying to access a row using its ROWID, which has been obtained from an index. A mismatch was found between the index rowid and the data block it is pointing to. The rowid points to a non-existent row in the data block. The corruption can be in data and/or index blocks. ORA-600 [12700] can also be reported due to a consistent read (CR) problem.
ORA-600 [3020]	This is called a ‘STUCK RECOVERY’. There is an inconsistency between the information stored in the redo and the information stored in a database block being recovered.This error indicates a lost write or a lost change in the database
ORA-600 [4194]	A mismatch has been detected between Redo records and rollback (Undo) records. Oracle is validating the Undo record number relating to the change being applied against the maximum undo record number recorded in the undo block. This error is reported when the validation fails.
ORA-600 [4193]	A mismatch has been detected between Redo records and Rollback (Undo) records. Oracle is validating the Undo block sequence number in the undo block against the Redo block sequence number relating to the change being applied. This error is reported when this validation fails.
ORA-600 [4137]	While backing out an undo record (i.e. at the time of rollback) Oracle found a transaction id mismatch indicating either a corruption in the rollback segment or corruption in an object which the rollback segment is trying to apply undo records on. This would indicate a corrupted rollback segment.
ORA-600 [6101]	Not enough free space was found when inserting a row into an index leaf block during the application of undo.
ORA-600 [2103]	Oracle is attempting to read or update a generic entry in the control file. If the entry number is invalid, ORA-600 [2130] is logged.
ORA-600 [4512]	Oracle is checking the status of transaction locks within a block. If the lock number is greater than the number of lock entries, ORA-600 [4512] is reported followed by a stack trace, process state and block dump. This error possibly indicates a block corruption.
ORA-600 [2662]	A data block SCN is ahead of the current SCN. The ORA-600 [2662] occurs when an SCN is compared to the dependent SCN identified by the process that is normally close to the database scn. If the SCN is less than the dependent SCN then ORA-600 [2662] is signaled.
ORA-600 [4097]	Oracle is accessing a rollback segment header to review if a transaction has been committed. However, the xid given is in the future of the transaction table. This could be due to a rollback segment corruption issue.
ORA-600 [4000]	It means that Oracle has tried to find an undo segment number in the data dictionary and this undo segment number was not found.
ORA-600 [6006]	Oracle is undoing an index leaf key operation. If the key is not found, ORA-00600 [6006] is logged. ORA-600[6006] is usually caused by a media corruption problem related to either a lost write to disk or a corruption on disk.
ORA-600 [4552]	This assertion is raised because Oracle is trying to unlock the rows in a block, but receive an incorrect block type. The second argument is the block type received.
ORA-600[6856]	Oracle is checking that the row slot that is about to be freed is not already on the free list. This internal error is raised when this check fails.
ORA-600[13011]	During a delete operation Oracle is deleting from a view via an instead-of trigger or an Index organized table and have exceeded a 5000 pass count
ORA-600[13013]	During the execution of an UPDATE statement, after several attempts (Arg [a] passcount) Oracle is unable to get a stable set of rows that conform to the WHERE clause.
ORA-600[13030]
ORA-600[25012]	Oracle is trying to generate the absolute file number given a tablespace number and relative file number and cannot find a matching file number or the file number is zero.
ORA-600[25026]	Looking up/checking a tablespace invalid tablespace ID and/or rdba found
ORA-600[25027]	Invalid tsn and/or rfn found
ORA-600 [kcbz_check_objd_typ_3]	An object block buffer in memory is checked and is found to have the wrong object id. This is most likely due to corruption.
ORA-600[kddummy_blkchk] ORA-600[kdblkcheckerror]	ORA-600 [kddummy_blkchk] is for 10g and ORA-600[kdblkcheckerror] for 11g onward. This error reports a Logical Block Corruption
ORA-600[ktadrprc-1]	Orphan segment or invalid rdba in Index,Table,Partition etc. Example: An entry in sys.ind$ does not exist in sys.seg$Note 136697.1 : “hcheck.sql” Script to Check for Known Problems in Oracle8i, Oracle9i, Oracle10g, Oracle 11g and Oracle 12c
ORA-600[ktsircinfo_num1]	This exception occurs when there are problems obtaining the row cache information correctly from sys.seg$. In most cases there is no information in sys.seg$.Note 136697.1 : “hcheck.sql” Script to Check for Known Problems in Oracle8i, Oracle9i, Oracle10g, Oracle 11g and Oracle 12c
ORA-600[qertbfetchbyrowid]	This error might be that a row was not found in an Index. Perform the check in section “Identify TABLE / INDEX Mismatch” in:Note 836658.1 : Identify the Corruption Extension for Block Corruption, Table/Index Inconsistency, Data Dictionary and Lost Writes
ORA-600[ktbdchk1-bad dscn]	This exception is raised when Oracle is performing a sanity check on the dependent SCN and fail. The dependent scn is greater than the current scn.

因为篇章问题，无法详细描述,如果需要深入了解的，可以单独进行沟通

WARNING: too many parse errors

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：WARNING: too many parse errors

从12.2开始,如果sql解析失败超过一定次数,会在alert日志中记录类似警告信息,便于对其进行排查

2020-01-05T20:30:50.559569+08:00
ARC0 (PID:29542): Archived Log entry 25044 added for T-1.S-12532 ID 0x6564f341 LAD:1
2020-01-05T20:30:50.626457+08:00
TT02 (PID:29552): SRL selected for T-1.S-12533 for LAD:2
2020-01-05T20:40:34.688814+08:00
WARNING: too many parse errors, count=1551097 SQL hash=0xec3987d3
PARSE ERROR: ospid=2240, error=923 for statement:
2020-01-05T20:40:34.688981+08:00
SELECT 1
Additional information: hd=0xef18d268 phd=0xee42c9c8 flg=0x28 cisid=106 sid=106 ciuid=106 uid=106 sqlid=4bwp0wrq3m1ym
...Current username=XIFENFEI
...Application: JDBC Thin Client Action:
2020-01-05T21:00:50.528469+08:00
Thread 1 advanced to log sequence 12534 (LGWR switch)
  Current log# 4 seq# 12534 mem# 0: /u01/app/oracle/oradata/XFF/redo04.log
  Current log# 4 seq# 12534 mem# 1: /u01/app/oracle/fast_recovery_area/XFF/onlinelog/redo04.log
2020-01-05T21:00:50.572028+08:00
ARC1 (PID:29546): Archived Log entry 25046 added for T-1.S-12533 ID 0x6564f341 LAD:1
2020-01-05T21:00:50.636035+08:00
TT02 (PID:29552): SRL selected for T-1.S-12534 for LAD:2
2020-01-05T21:05:32.671478+08:00
WARNING: too many parse errors, count=1551197 SQL hash=0xec3987d3
PARSE ERROR: ospid=4172, error=923 for statement:
2020-01-05T21:05:32.671641+08:00
SELECT 1
Additional information: hd=0xef18d268 phd=0xee42c9c8 flg=0x28 cisid=106 sid=106 ciuid=106 uid=106 sqlid=4bwp0wrq3m1ym
...Current username=XIFENFEI
...Application: JDBC Thin Client Action:

通过查询mos,确认相关记录是通过_kks_parse_error_warning参数来控制,默认每100次解析失败写入alert日志一次

[oracle@kage28 trace]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Sun Jan 5 21:23:21 2020
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
SQL> col name for a52
SQL> col value for a24
SQL> col description for a50
set linesize 150
select a.ksppinm name,b.ksppstvl value,a.ksppdesc description
  from x$ksppi a,x$ksppcv b
 where a.inst_id = USERENV ('Instance')
   and b.inst_id = USERENV ('Instance')
   and a.indx = b.indx
   and upper(a.ksppinm) LIKE upper('%&param%')
SQL> SQL>   2    3    4    5    6    7  order by name
  8  /
Enter value for param: _kks_parse_error_warning
old   6:    and upper(a.ksppinm) LIKE upper('%&param%')
new   6:    and upper(a.ksppinm) LIKE upper('%_kks_parse_error_warning%')
NAME                                                 VALUE                    DESCRIPTION
---------------------------------------------------- ------------------------ ----------------------
_kks_parse_error_warning                             100                      Parse error warning

由于某种原因,解析错误sql,短期内无法进行修改,又不想让其在alert中提示,可以通过设置该值为0,实现让其不记录在alert日志中

alter system set "_kks_parse_error_warning"=0;

在12.2之前版本,解析错误默认不记录到alert日志中,但是也可以通过event 10035实现类似功能,具体参见:failed parse elapsed time过大分析案例

dbca删除库和rm删库恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：dbca删除库和rm删库恢复

由于误操作删除库的事情偶有发生,遇到过dbca误删除数据库,遇到过rm删除数据文件的故障.近期有过两次类似的恢复请求,一次是win平台dbca本该删除测试库,结果选择错误把生产库给删除,发生误操作之后,没有及时对磁盘进行卸载,而是有一些写操作,当事人找了各种反删除软件进行恢复,结果只是成功了恢复了部分文件,核心文件全部丢失.另外一次是linux平台误操作,rm -rf 删除了整个oracle目录,而且数据文件也全部在里面,所幸没有任何写操作.

对于dbca客户已经使用反删除软件进行恢复,但是效果不行,我们直接从底层扫描,恢复出来需要数据

有个别文件少量block被覆盖,而且客户只要其中部分核心表数据,因此直接使用dul恢复出客户需要数据,完美完成恢复

对于rm掉所有数据文件的客户,通过extundelete进行反删除恢复(参考:extundelete恢复Linux被删除文件),由于句柄丢失,导致有几个数据文件恢复失败.通过linux平台底层恢复,实现需要数据文件完全恢复,数据库正常open,实现完美恢复

对于这类删库的操作,一定要保护好现场,对数据文件存在分区不要有二次写操作,尽可能的减少覆盖.理论上数据文件在磁盘上,都可以恢复出来.如果有遇到数据库文件误删除,或者文件系统损坏的恢复需求,无法自行解决,可以联系我们从底层进行恢复:Phone:17813235971 Q Q:107644445 E-Mail:dba@xifenfei.com
类似恢复案例:
又一例asm格式化文件系统恢复
 文件系统损坏导致数据文件异常恢复
 Oracle 数据文件大小为0kb或者文件丢失恢复
 oracle asm disk格式化恢复—格式化为ntfs文件系统
 oracle asm disk格式化恢复—格式化为ext4文件系统
 ORA-15042: ASM disk “N” is missing from group number “M” 故障恢复

文件系统加密解密之后数据库异常处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：文件系统加密解密之后数据库异常处理

有朋友数据库被文件系统勒索病毒加密,付费进行解密之后,发现所有的文本文件解密成功,但是Oracle数据库无法正常open,通过分析,虽然所有的文件名都已经恢复正常

但是文件内容没有被正常恢复成功

对于这次的情况,我们通过底层分析,发现是由于解密bug,导致文件没有被解密完成,通过对文件进行二次修复,实现数据库直接open,并导出数据

ORA-600 kkdlcob-objn-exist 40

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 kkdlcob-objn-exist 40

有客户晚上对数据库进行正常的重启维护,结果去遭遇数据库无法正常启动故障,报ORA-600 kkdlcob-objn-exist 40错误.

alert日志信息

Sat Jul 06 12:05:50 2019
alter database open
Sat Jul 06 12:05:51 2019
LGWR: STARTING ARCH PROCESSES
Sat Jul 06 12:05:51 2019
ARC0 started with pid=19, OS id=7056
ARC0: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Thread 1 opened at log sequence 12
  Current log# 3 seq# 12 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sat Jul 06 12:05:52 2019
SMON: enabling cache recovery
Sat Jul 06 12:05:52 2019
ARC1 started with pid=21, OS id=7948
Sat Jul 06 12:05:52 2019
ARC2 started with pid=22, OS id=11552
Sat Jul 06 12:05:52 2019
ARC3 started with pid=23, OS id=5504
ARC1: Archival started
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_3092.trc  (incident=20562):
ORA-00600: internal error code, arguments: [kkdlcob-objn-exists], [40], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_20562\orcl_ora_3092_i20562.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_3092.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kkdlcob-objn-exists], [40], [], [], [], [], [], [], [], [], [], []
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_3092.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kkdlcob-objn-exists], [40], [], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 3092): terminating the instance due to error 704
Instance terminated by USER, pid = 3092
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (3092) as a result of ORA-1092
Sat Jul 06 12:05:56 2019
ORA-1092 : opitsk aborting process

ORA-600 kkdlcob-objn-exists官方解释，一般是由于ddl操作时dataobj#不合适引起,最大的可能是_NEXT_OBJECT值不对

对数据库启动过程进行跟踪

CREATE UNIQUE INDEX I_OBJ5 ON OBJ$(SPARE3,NAME,NAMESPACE,TYPE#,OWNER#,REMOTEOWNER,LINKNAME,SUBNAME,OBJ#)
PCTFREE 10 INITRANS 2 MAXTRANS 255 STORAGE (  INITIAL 64K NEXT 1024K MINEXTENTS 1
MAXEXTENTS 2147483645 PCTINCREASE 0 OBJNO 40 EXTENTS (FILE 1 BLOCK 368))
END OF STMT
PARSE #299964504:c=0,e=368,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=440321312,tim=8823997737
ORA-00600: internal error code, arguments: [kkdlcob-objn-exists], [40], [], [], [], [], [], [], [], [], [], []
EXEC #299964504:c=875000,e=1213391,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=440321312,tim=8825211183
ERROR #299964504:err=600 tim=8825211203
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kkdlcob-objn-exists], [40], [], [], [], [], [], [], [], [], [], []
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kkdlcob-objn-exists], [40], [], [], [], [], [], [], [], [], [], []

发现是I_OBJ5无法正常创建,导致数据库无法正常启动,由于这个是obj$对象的一个index(而且是11.2之后版本才引进),不是数据库启动必须的对象,通过bbed修改相关基表,让数据库启动不在跳过该index,数据库启动成功.因为进行了非常规的方法恢复,建议客户导出数据,导入到新库,数据库恢复工作完成.

ORA-00600 kcratr_scan_rc

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-00600 kcratr_scan_rc

11.2.0.4数据库启动报ORA-600 kcratr_scan_rc错误

ORA-600 kcratr_scan_rc错误相关alert日志

Thu May 09 01:56:01 2019
alter database   open
Beginning crash recovery of 1 threads
 parallel recovery started with 23 processes
Started redo scan
Errors in file /home/u01/diag/rdbms/orcl/orcl/trace/orcl_ora_9975.trc  (incident=171):
ORA-00600: internal error code, arguments: [kcratr_scan_rc], [4], [1], [39821], [190063], [], [], [], [], [], [], []
Incident details in: /home/u01/diag/rdbms/orcl/orcl/incident/incdir_171/orcl_ora_9975_i171.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Aborting crash recovery due to error 600
Errors in file /home/u01/diag/rdbms/orcl/orcl/trace/orcl_ora_9975.trc:
ORA-00600: internal error code, arguments: [kcratr_scan_rc], [4], [1], [39821], [190063], [], [], [], [], [], [], []
Errors in file /home/u01/diag/rdbms/orcl/orcl/trace/orcl_ora_9975.trc:
ORA-00600: internal error code, arguments: [kcratr_scan_rc], [4], [1], [39821], [190063], [], [], [], [], [], [], []
ORA-600 signalled during: alter database   open...

从这里看是由于数据库在做实例恢复的时候无法正确的应用日志导致.通过屏蔽数据库前滚恢复,强制open库

SQL> startup mount pfile='/tmp/pfile'
ORACLE instance started.
Total System Global Area 6998261760 bytes
Fixed Size                  2266624 bytes
Variable Size            2684357120 bytes
Database Buffers         4294967296 bytes
Redo Buffers               16670720 bytes
Database mounted.
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 91 with name
"_SYSSMU91_1360910548$" too small
Process ID: 15929
Session ID: 5272 Serial number: 3

数据库报出来比较熟悉的ORA-01092 ORA-00704 ORA-00604 ORA-01555错误,分析trace文件

Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu May 09 02:10:27 2019
SMON: enabling cache recovery
ORA-01555 caused by SQL statement below (SQL ID: bqbdby3c400p7, SCN: 0x0000.3e785fc7):
select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1
Errors in file /home/u01/diag/rdbms/orcl/orcl/trace/orcl_ora_15929.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 91 with name "_SYSSMU91_1360910548$" too small
Errors in file /home/u01/diag/rdbms/orcl/orcl/trace/orcl_ora_15929.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 91 with name "_SYSSMU91_1360910548$" too small
Error 704 happened during db open, shutting down database
USER (ospid: 15929): terminating the instance due to error 704
Instance terminated by USER, pid = 15929
ORA-1092 signalled during: alter database open resetlogs...
opiodr aborting process unknown ospid (15929) as a result of ORA-1092
Thu May 09 02:10:28 2019
ORA-1092 : opitsk aborting process

这次的是比较少见的ORA-1555的错误语句select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1,通过bbed进行处理之后

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 17776
Session ID: 5272 Serial number: 3

数据库出现ORA-03113错误,通过分析alert日志

ORA-00600: internal error code, arguments: [4198], [], [], [], [], [], [], [], [], [], [], []
Errors in file /home/u01/diag/rdbms/orcl/orcl/trace/orcl_smon_17739.trc  (incident=128132):
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], []
Incident details in: /home/u01/diag/rdbms/orcl/orcl/incident/incdir_128132/orcl_smon_17739_i128132.trc
Errors in file /home/u01/diag/rdbms/orcl/orcl/trace/orcl_mmon_17743.trc  (incident=128151):
ORA-00600: internal error code, arguments: [4412], [0x1E6BC4DF8], [0x000000000], [1], [6283], [], [], [], [], [], [], []
Incident details in: /home/u01/diag/rdbms/orcl/orcl/incident/incdir_128151/orcl_mmon_17743_i128151.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
SMON: Parallel transaction recovery slave got internal error
SMON: Downgrading transaction recovery to serial
Errors in file /home/u01/diag/rdbms/orcl/orcl/trace/orcl_smon_17739.trc  (incident=128133):
ORA-00600: internal error code, arguments: [4137], [10.4.1100583], [0], [0], [], [], [], [], [], [], [], []
Incident details in: /home/u01/diag/rdbms/orcl/orcl/incident/incdir_128133/orcl_smon_17739_i128133.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /home/u01/diag/rdbms/orcl/orcl/trace/orcl_mmon_17743.trc  (incident=128152):
ORA-00600: internal error code, arguments: [4406], [0x1E6BC4DF8], [0x000000000], [2], [6289], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4412], [0x1E6BC4DF8], [0x000000000], [1], [6283], [], [], [], [], [], [], []
Incident details in: /home/u01/diag/rdbms/orcl/orcl/incident/incdir_128152/orcl_mmon_17743_i128152.trc
ORACLE Instance orcl (pid = 15) - Error 600 encountered while recovering transaction (10, 4).
Errors in file /home/u01/diag/rdbms/orcl/orcl/trace/orcl_smon_17739.trc  (incident=128134):
ORA-00600: internal error code, arguments: [4137], [98.33.13158], [0], [0], [], [], [], [], [], [], [], []
Incident details in: /home/u01/diag/rdbms/orcl/orcl/incident/incdir_128134/orcl_smon_17739_i128134.trc
Starting background process SMCO
Thu May 09 02:16:25 2019
SMCO started with pid=21, OS id=18119
Errors in file /home/u01/diag/rdbms/orcl/orcl/trace/orcl_smon_17739.trc  (incident=128135):
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /home/u01/diag/rdbms/orcl/orcl/incident/incdir_128135/orcl_smon_17739_i128135.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Non-fatal internal error happenned while SMON was doing non-existent object cleanup.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
Thu May 09 02:16:27 2019
Thu May 09 02:16:32 2019
Errors in file /home/u01/diag/rdbms/orcl/orcl/trace/orcl_ora_18168.trc  (incident=128620):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-28000: the account is locked
Incident details in: /home/u01/diag/rdbms/orcl/orcl/incident/incdir_128620/orcl_ora_18168_i128620.trc
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
ORA-28000: the account is locked
Block recovery from logseq 3, block 239 to scn 2147483855
Recovery of Online Redo Log: Thread 1 Group 3 Seq 3 Reading mem 0
  Mem# 0: /home/u01/oradata/orcl/redo03.log
Block recovery stopped at EOT rba 3.241.16
Block recovery completed at rba 3.241.16, scn 0.2147483854
Block recovery from logseq 3, block 239 to scn 2147483853
Recovery of Online Redo Log: Thread 1 Group 3 Seq 3 Reading mem 0
  Mem# 0: /home/u01/oradata/orcl/redo03.log
Block recovery completed at rba 3.241.16, scn 0.2147483854
Thu May 09 02:16:33 2019
Errors in file /home/u01/diag/rdbms/orcl/orcl/trace/orcl_ora_18170.trc  (incident=128621):
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
ORA-28000: the account is locked
Incident details in: /home/u01/diag/rdbms/orcl/orcl/incident/incdir_128621/orcl_ora_18170_i128621.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Thu May 09 02:16:34 2019
PMON (ospid: 17710): terminating the instance due to error 474
Instance terminated by PMON, pid = 17710

大量的ORA-600 4198,ORA-600 6006,ORA-600 4412,ORA-600 4137,ORA-600 4406,ORA-600 kdsgrp1等错误,根据以往经验,主要是undo异常导致,通过相关处理之后,数据库open正常,数据逻辑导出.

分类目录归档：Oracle