Database SOS – 第16页 – Professional Database Recovery Support@+8617813235971

.mdf.locked加密sql server完美恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：.mdf.locked加密sql server完美恢复

有可能用友ERP软件的sql server 数据库所在机器被勒索病毒加密,扩展名为.locked和昨天恢复的基本类似(.locked加密勒索数据库级别恢复),通过分析确认sql server被这种病毒加密,也可以完美恢复

通过恢复之后数据库正常挂载成功

测试应用一切正常

对于类似这种被加密的勒索的数据文件,我们可以实现比较好的恢复效果,如果此类的数据库(oracle,mysql,sql server)等被加密,需要专业恢复技术支持，请联系我们:
电话/微信:17813235971 Q Q:107644445 E-Mail:dba@xifenfei.com
系统安全防护措施建议：
1.多台机器，不要使用相同的账号和口令
2.登录口令要有足够的长度和复杂性，并定期更换登录口令
3.重要资料的共享文件夹应设置访问权限控制，并进行定期备份
4.定期检测系统和软件中的安全漏洞，及时打上补丁。
5.定期到服务器检查是否存在异常。
6.安装安全防护软件，并确保其正常运行。
7.从正规渠道下载安装软件。
8.对不熟悉的软件，如果已经被杀毒软件拦截查杀，不要添加信任继续运行。
9.保存良好的备份习惯，尽量做到每日备份，异地备份。

.locked加密勒索数据库级别恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：.locked加密勒索数据库级别恢复

有客户数据库被加密成.locked结尾的扩展名,数据库无法正常使用

对应的READ_ME1.html文件中信息类似:
send 0.1btc to my address:bc1ql8an5slxutu3yjyu9rvhsfcpv29tsfhv3j9lr4. contact email:service@hellowinter.online,if you can’t contact my email, please contact some data recovery company(suggest taobao.com), may they can contact to me .your id: ATNtkQxKZ8uaTNt3etgDXVdN2MjvPC8XoWOpoc7l25Dyg5ePEA6AFtj0sE8rYUX+Jm+1ajXASCAREJO8GD+53eqiUX26/s2aTX4uTkaAY90xTg6UIQ1SqhPpylXRsY759aUnOPeHWuv8fwj03g7vN68ctgjiRgHz3h1gNoOEDzT0GLGbr9+X6G2oCx23kNyWj3m2IXPJtyY8XqRa0UI1zNyjwyNoE6qNu0ZufInCnHKVrAp95ngibtUSmc6z+vYl8ROQlsPqi9UfyO0P3Wk2k9pLxLKsej7cBl0qtYEfRTou1v0/Ca4y57o1Djpaieklk2Ne03gYGztrGtS77S/zVzrg5B3x64qVuYNzQH9+LtmPE+6RqJY06CSmYr5f3GN1kEBuhsnGlNWjVJjjWimEMJXYMqqAY8fwb5DYxC/XCYg6jbg85R4CTOeE9DfC6yoPpsG1yXpDilBIFhFiTjWj9t94g0TBHi11dNBz1Fr+0x6TpmgfA9d0L/yjZSOTr7hQ9CeMn2eWjkYZ2jIoSV7XlHyDCbPRPlCTOnOX3ZP9zgQ48tXgJWbkds8SATOzjJk3+ttl+JNN7jScNvs7MZX0oVSyXVDrAdCHpftY63riuDuyE7rVBjtkOPr5njKSpA6K9EBD+FnCwQDzja7z78kOKXA/ddFj79mznzcnbODE8qnxitzNdv9SuQkJU+qVoH+fSndDGkDDxNliZJypHdwKM3ooO2q7fayq1Iw0dEN3FHbqgY/m3drUqge3UV07VrV9ht+tUzozUPoVBZbKSgZuYK2FKlXhlxhV/PXlskNtx+YFi1T4yzqjcT0a8HE5huQQzehDPeKQTRRpU+u9aq4TAxotjcTS1sfe/qgRKYyq+qLHg9rfFCcr5NjJPDSpWLTlhB+hUTK/iZqjYBaEccJjyvi0i+/5bDqmGmqkFeyWToyqhwlk4hN1IGyOHJY7Jf6a1S6CogVb34ArTOPFk2gKD7o4vXbERUmhw5xiP9c6Zwir2ENBdDaCl5p9oh44jpeoJqptZR4Uu1k8eqQNtUuXqlU9xvjaxt+dgXMQaul65nmgxgMmLUHjSSXZhGNzyeVGkz2KrP97ZLqYCq29iYoUt9WTLAmFtj0F+l4uYTj58+7xHM1/oXIMEOt3RmFv183I/yDesJaoV11cpbQUp1KJZvRvbws6CcajMnaH3zHJ7FHg74=
通过自研的oracle勒索加密恢复工具快速恢复文件

实现数据库直接open成功,实现数据0丢失

ORA-16038 ORA-00354故障处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-16038 ORA-00354故障处理

遇到一个案例,数据库open报ORA-16038,ORA-00354等错误

查询该redo状态(使用Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)脚本收集),确认为inactive

由于inactive 状态的redo损坏,无法被arch进程归档导致数据库无法正常open,尝试强制clear联机日志

由于25号文件属于offline状态,导致联机日志无法正常被clear,报ORA-00393 ORA-00312等错误.通过试验重现该问题.

SQL> alter database datafile 5 offline;

Database altered.

--使用一些技巧让数据库无法归档

SQL> select group#,status,archived from v$log;

    GROUP# STATUS           ARC
---------- ---------------- ---
         1 INACTIVE         NO
         2 ACTIVE           NO
         3 CURRENT          NO

SQL> shutdown abort;
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area  626327552 bytes
Fixed Size                  2255832 bytes
Variable Size             234882088 bytes
Database Buffers          385875968 bytes
Redo Buffers                3313664 bytes
Database mounted.
SQL> alter database clear unarchived logfile group 1;
alter database clear unarchived logfile group 1
*
ERROR at line 1:
ORA-00393: log 1 of thread 1 is needed for recovery of offline datafiles
ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log'
ORA-01110: data file 5: '/u01/app/oracle/oradata/orcl/t_xifenfei01.dbf'

SQL> !oerr ora 393
00393, 00000, "log %s of thread %s is needed for recovery of offline datafiles"
// *Cause:  Log cannot be cleared because the redo in it is needed to recover
//          offline datafiles. It has not been archived so there is no
//          other copy available. If the log is cleared the tablespaces
//          containing the files will have to be dropped.
// *Action: Archive the log then repeat the clear command. If archiving is not
//          possible, and dropping the tablespaces is acceptible, then add the
//          clause UNRECOVERABLE DATAFILE at the end of the clear command.


SQL>  alter database clear unarchived logfile group 1 unrecoverable datafile;

Database altered.

SQL> select group#,status,archived from v$log;

    GROUP# STATUS           ARC
---------- ---------------- ---
         1 UNUSED           YES
         3 CURRENT          NO
         2 ACTIVE           NO

客户的问题也是通过unrecoverable datafile 方式强制clear联机日志成功,数据库open成功

unknown variable ‘default-character-set=utf8′ 处理

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：unknown variable ‘default-character-set=utf8′ 处理

分享一例由于mysql参数配置不当,当时mysql无法正常启动的case

[root@XIFENFEI ~]# service mysql start
Redirecting to /bin/systemctl start mysql.service
Job for mysqld.service failed because the control process exited with error code. 
See "systemctl status mysqld.service" and "journalctl -xe" for details.
[root@XIFENFEI ~]# systemctl status mysqld.service
 mysqld.service - LSB: start and stop MySQL
   Loaded: loaded (/etc/rc.d/init.d/mysqld; bad; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2023-05-22 16:09:57 CST; 13s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 101951 ExecStop=/etc/rc.d/init.d/mysqld stop (code=exited, status=0/SUCCESS)
  Process: 102545 ExecStart=/etc/rc.d/init.d/mysqld start (code=exited, status=1/FAILURE)

May 22 16:09:53 RHAA08-0901 systemd[1]: Starting LSB: start and stop MySQL...
May 22 16:09:57 RHAA08-0901 mysqld[102545]: Starting MySQL.... ERROR! 
            The server quit without updating PID file (/www/se....pid).
May 22 16:09:57 RHAA08-0901 systemd[1]: mysqld.service: control process exited, code=exited status=1
May 22 16:09:57 RHAA08-0901 systemd[1]: Failed to start LSB: start and stop MySQL.
May 22 16:09:57 RHAA08-0901 systemd[1]: Unit mysqld.service entered failed state.
May 22 16:09:57 RHAA08-0901 systemd[1]: mysqld.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

mysql日志

2023-05-22T06:59:57.646266Z 0 [Warning] option 'max_allowed_packet': unsigned value 107374182400 adjusted to 1073741824
2023-05-22T06:59:57.646480Z 0 [Note] --secure-file-priv is set to NULL. 
          Operations related to importing and exporting data are disabled
2023-05-22T06:59:57.646548Z 0 [Note] /www/server/mysql/bin/mysqld (mysqld 5.7.41-log) starting as process 79451 ...
2023-05-22T06:59:57.726167Z 0 [Note] InnoDB: PUNCH HOLE support available
2023-05-22T06:59:57.726257Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2023-05-22T06:59:57.726269Z 0 [Note] InnoDB: Uses event mutexes
2023-05-22T06:59:57.726286Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2023-05-22T06:59:57.726296Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
2023-05-22T06:59:57.726306Z 0 [Note] InnoDB: Using Linux native AIO
2023-05-22T06:59:57.726975Z 0 [Note] InnoDB: Number of pools: 1
2023-05-22T06:59:57.727161Z 0 [Note] InnoDB: Using CPU crc32 instructions
2023-05-22T06:59:57.733999Z 0 [Note] InnoDB: Initializing buffer pool, total size = 4G, instances = 8, chunk size = 128M
2023-05-22T06:59:58.051369Z 0 [Note] InnoDB: Completed initialization of buffer pool
2023-05-22T06:59:58.087559Z 0 [Note] InnoDB: If the mysqld execution user is authorized,
 page cleaner thread priority can be changed. See the man page of setpriority().
2023-05-22T06:59:58.099727Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2023-05-22T06:59:58.202034Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-05-22T06:59:58.202150Z 0 [Note] InnoDB: Setting file '/www/server/data/ibtmp1' size to 12 MB. 
Physically writing the file full; Please wait ...
2023-05-22T06:59:58.242233Z 0 [Note] InnoDB: File '/www/server/data/ibtmp1' size is now 12 MB.
2023-05-22T06:59:58.244128Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2023-05-22T06:59:58.244163Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2023-05-22T06:59:58.244910Z 0 [Note] InnoDB: Waiting for purge to start
2023-05-22T06:59:58.295153Z 0 [Note] InnoDB: 5.7.41 started; log sequence number 1184000067
2023-05-22T06:59:58.295727Z 0 [Note] InnoDB: Loading buffer pool(s) from /www/server/data/ib_buffer_pool
2023-05-22T06:59:58.296135Z 0 [Note] Plugin 'FEDERATED' is disabled.
2023-05-22T06:59:58.296288Z 0 [Note] InnoDB: Buffer pool(s) load completed at 230522 14:59:58
2023-05-22T06:59:58.299095Z 0 [ERROR] unknown variable 'default-character-set=utf8'
2023-05-22T06:59:58.299131Z 0 [ERROR] Aborting

2023-05-22T06:59:58.299179Z 0 [Note] Binlog end
2023-05-22T06:59:58.299530Z 0 [Note] Shutting down plugin 'ngram'
2023-05-22T06:59:58.299560Z 0 [Note] Shutting down plugin 'partition'
2023-05-22T06:59:58.299572Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2023-05-22T06:59:58.299588Z 0 [Note] Shutting down plugin 'ARCHIVE'
2023-05-22T06:59:58.299599Z 0 [Note] Shutting down plugin 'MyISAM'
2023-05-22T06:59:58.299626Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2023-05-22T06:59:58.299641Z 0 [Note] Shutting down plugin 'MEMORY'
2023-05-22T06:59:58.299657Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2023-05-22T06:59:58.299797Z 0 [Note] Shutting down plugin 'CSV'
2023-05-22T06:59:58.299811Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2023-05-22T06:59:58.299822Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2023-05-22T06:59:58.299832Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2023-05-22T06:59:58.299843Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2023-05-22T06:59:58.299853Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2023-05-22T06:59:58.299863Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2023-05-22T06:59:58.299873Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2023-05-22T06:59:58.299883Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2023-05-22T06:59:58.299893Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2023-05-22T06:59:58.299904Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2023-05-22T06:59:58.299914Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2023-05-22T06:59:58.299924Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2023-05-22T06:59:58.299934Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2023-05-22T06:59:58.299944Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2023-05-22T06:59:58.299954Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2023-05-22T06:59:58.299964Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2023-05-22T06:59:58.299974Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
2023-05-22T06:59:58.299984Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2023-05-22T06:59:58.299995Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2023-05-22T06:59:58.300005Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2023-05-22T06:59:58.300015Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2023-05-22T06:59:58.300025Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2023-05-22T06:59:58.300035Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2023-05-22T06:59:58.300045Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2023-05-22T06:59:58.300055Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2023-05-22T06:59:58.300066Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2023-05-22T06:59:58.300075Z 0 [Note] Shutting down plugin 'INNODB_CMP'
2023-05-22T06:59:58.300085Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2023-05-22T06:59:58.300095Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2023-05-22T06:59:58.300105Z 0 [Note] Shutting down plugin 'INNODB_TRX'
2023-05-22T06:59:58.300116Z 0 [Note] Shutting down plugin 'InnoDB'
2023-05-22T06:59:58.300271Z 0 [Note] InnoDB: FTS optimize thread exiting.
2023-05-22T06:59:58.300437Z 0 [Note] InnoDB: Starting shutdown...
2023-05-22T06:59:58.400764Z 0 [Note] InnoDB: Dumping buffer pool(s) to /www/server/data/ib_buffer_pool
2023-05-22T06:59:58.401153Z 0 [Note] InnoDB: Buffer pool(s) dump completed at 230522 14:59:58
2023-05-22T07:00:00.340582Z 0 [Note] InnoDB: Shutdown completed; log sequence number 1184000086
2023-05-22T07:00:00.343128Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2023-05-22T07:00:00.343157Z 0 [Note] Shutting down plugin 'sha256_password'
2023-05-22T07:00:00.343165Z 0 [Note] Shutting down plugin 'mysql_native_password'
2023-05-22T07:00:00.343347Z 0 [Note] Shutting down plugin 'binlog'
2023-05-22T07:00:00.344547Z 0 [Note] /www/server/mysql/bin/mysqld: Shutdown complete

提示比较明显由于default-character-set=utf8参数设置不当当时,检查my.cnf配置，发现
在mysqld中配置了default-character-set=utf8,该参数正确配置为:character-set-server = utf8,设置正确参数值之后,mysql启动正常

[root@XIFENFEI data]# service mysql start
Redirecting to /bin/systemctl start mysql.service
[root@XIFENFEI data]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.41-log Source distribution

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| 365xxxy             |
| xxxxhu         |
| xxxiuye              |
| mysql              |
| performance_schema |
| xxxxchegn         |
| sxxxxp               |
| sys                |
| xxxxc               |
+--------------------+
10 rows in set (0.00 sec)

ORA-600 16703故障,客户找人恢复数据库,数据库被进一步恶意破坏—ORA-00704 ORA-00922

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-600 16703故障,客户找人恢复数据库,数据库被进一步恶意破坏—ORA-00704 ORA-00922

有朋友找到我,数据库报ORA-600 16703错误,这个本来是一个比较常见的破坏故障(警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703数据库启动报错如下:

修复tab$启动库报ORA-00704 ORA-00922错误

SQL> alter database Open;
alter database Open
*
第 1 行出现错误:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00922: missing or invalid option
进程 ID: 1340
会话 ID: 191 序列号: 3

ORA-00704 ORA-00922是比较少见的错误,第一感觉bootstrap$损坏了,对数据库启动过程进行跟踪

PARSING IN CURSOR #11700472 len=600 dep=1 uid=0 oct=1 lid=0 tim=338738406773 hv=4034608779 
ad='7ffdef83f360' sqlid='asgjp8bs7qgnb'
CREATE TABLE UNDO$("US#" 
END OF STMT
PARSE #11700472:c=0,e=361,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=338738406773
EXEC #11700472:c=0,e=73,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=0,tim=338738406917
CLOSE #11700472:c=0,e=3,dep=1,type=0,tim=338738406997
=====================
PARSE ERROR #635423520:len=841 dep=1 uid=0 oct=1 lid=0 tim=338738407066 err=922
CREATE TABLE TS$<"TS#" NU ...
ORA-00704: 引导程序进程失败
ORA-00922: 选项缺失或无效
ORA-00704: 引导程序进程失败
ORA-00922: 选项缺失或无效

*** 2023-05-17 19:27:51.813
USER (ospid: 1340): terminating the instance due to error 704

*** 2023-05-17 19:27:54.050
EXEC #11710688:c=0,e=2481834,p=16,cr=62,cu=0,mis=0,r=0,dep=0,og=1,plh=0,tim=338740646732
ERROR #11710688:err=1092 tim=338740646777

进一步分析bootstrap$表记录

通过上述分析,可以确认原库的CREATE TABLE TS$(“TS#”被人修改为CREATE TABLE TS$<“TS#”,通过观察客户机器以及和客户确认,客户找的技术人员上传了bbed工具,并进行了一些操作.基于上述信息,大概率是被人通过bbed工具把TS$(修改为了TS$<,从而使得数据库修复tab$之后也无法正常启动.

gpk-update-icon进程占用CPU资源100％

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：gpk-update-icon进程占用CPU资源100％

在一个客户的机器上发现gpk-update-icon进程长期占用cpu 100%

gpk-update-icon进程在GUI模式下会自动通知rpm软件包更新，是由gnome-packagekit的bug造成的。
gpk-update-icon使用递归主循环，递归循环从dbus回调调用。因此，它处于调度操作的中间，并且在操作完成之前dbus无法进一步调度。
临时解决方法

killall gpk-update-icon

掉掉相关进程之后临时恢复正常,如果要防止后续再发生该问题,可以把系统启动到非图形化界面

[root@HIS_DG ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.8 (Santiago)
[root@HIS_DG ~]# runlevel
N 5
[root@HIS_DG ~]# who -r
         run-level 5  2021-11-13 12:04
[root@HIS_DG ~]# init 3
[root@HIS_DG ~]# who -r
         run-level 3  2023-05-17 14:12                   last=5
[root@HIS_DG ~]# runlevel
5 3

或者卸载相关包

#yum remove gnome-packagekit 或 rpm -e gpk-update-icon

难见的oracle 9i恢复—2023年

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：难见的oracle 9i恢复—2023年

时过境迁,以前恢复大量oracle 8/9版本的库,现在一套oracle 9i的库都比较稀奇了.今天恢复客户一套9.2.0.6的aix环境rac库,通过分析确认主要问题:
1. 重建控制文件，resetlogs库遗漏数据文件

2. 数据库启动主要报错ORA-600 2663和ORA-600 kclchkblk_4

Tue Nov  8 09:10:05 2022
Successfully onlined Undo Tablespace 1.
Dictionary check beginning
Tablespace 'TEMP' #2 found in data dictionary,
but not in the controlfile. Adding to controlfile.
File #84 found in data dictionary but not in controlfile.
Creating OFFLINE file 'MISSING00084' in the controlfile.
This file can no longer be recovered so it must be dropped.
Dictionary check complete
Tue Nov  8 09:10:05 2022
SMON: enabling tx recovery
Tue Nov  8 09:10:05 2022
Database Characterset is ZHS16GBK
Tue Nov  8 09:10:05 2022
Errors in file /u01/prod/proddb/9.2.0/admin/udump/prod1_ora_536662.trc:
ORA-00600: internal error code, arguments: [2663], [3301], [2638369768], [3301], [2640322622], [], [], []
Tue Nov  8 09:10:06 2022
Errors in file /u01/prod/proddb/9.2.0/admin/bdump/prod1_smon_647352.trc:
ORA-00600: internal error code, arguments: [kclchkblk_4], [3301], [18446744072061740072],[3301],[18446744072052954088]
Tue Nov  8 09:10:06 2022
Errors in file /u01/prod/proddb/9.2.0/admin/udump/prod1_ora_536662.trc:
ORA-00600: internal error code, arguments: [2663], [3301], [2638369768], [3301], [2640322622], [], [], []
Error 600 happened during db open, shutting down database
USER: terminating instance due to error 600
Instance terminated by USER, pid = 536662
ORA-1092 signalled during: alter database open...

根据客户文件名称的规则,推算出来84号文件实际的文件名(因为使用的是lv[aix的hacmp管理的lv的裸设备方式]),通过dbv确认文件无坏块

DBVERIFY: Release 9.2.0.6.0 - Production on Sat May 13 16:44:09 2023

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

DBVERIFY - Verification starting : FILE = /dev/ra_txn_ind12.dbf


DBVERIFY - Verification complete

Total Pages Examined         : 256000
Total Pages Processed (Data) : 0
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 299
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 13
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 255688
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Highest block SCN            : 11177081099136 (2602.1576194944)

bbed验证文件该文件是否是84号文件

$ bbed blocksize=8192 filename='/dev/ra_txn_ind12.dbf'   
Password: 

BBED: Release 2.0.0.0.0 - Limited Production on Mon May 15 09:45:44 2023

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

************* !!! For Oracle Internal Use only !!! ***************

BBED> map
 File: /dev/ra_txn_ind12.dbf (0)
 Block: 1                                     Dba:0x00000000
------------------------------------------------------------
 Data File Header

 struct kcvfh, 608 bytes                    @0       

 ub4 tailchk                                @8188    


BBED> p kcvfh
struct kcvfh, 608 bytes                     @0       
   struct kcvfhbfh, 20 bytes                @0       
      ub1 type_kcbh                         @0        0x0b
      ub1 frmt_kcbh                         @1        0x02
      ub1 spare1_kcbh                       @2        0x00
      ub1 spare2_kcbh                       @3        0x00
      ub4 rdba_kcbh                         @4        0x15000001
      ub4 bas_kcbh                          @8        0x00000000
      ub2 wrp_kcbh                          @12       0x0000
      ub1 seq_kcbh                          @14       0x01
      ub1 flg_kcbh                          @15       0x04 (KCBHFCKV)
      ub2 chkval_kcbh                       @16       0x1b4a
      ub2 spare3_kcbh                       @18       0x0000
   struct kcvfhhdr, 76 bytes                @20      
      ub4 kccfhswv                          @20       0x09200000
      ub4 kccfhcvn                          @24       0x08000000
      ub4 kccfhdbi                          @28       0x05d15ccf
      ……
      ub4 kccfhcsq                          @40       0x00525a20
      ub4 kccfhfsz                          @44       0x0003e800
      s_blkz kccfhbsz                       @48       0x00
      ub2 kccfhfno                          @52       0x0054
      ub2 kccfhtyp                          @54       0x0003
   ……
   ub4 kcvfhrfn                             @528      0x00000054  ---确认是84号文件
  ……

通过bbed修改文件相关信息,然后尝试rename文件,但是recover datafile 84报错

Mon May 15 09:49:44 2023
alter database rename file '/u01/prod/proddb/9.2.0/dbs/MISSING00084' to '/dev/ra_txn_ind12.dbf'
Mon May 15 09:49:44 2023
Completed: alter database rename file '/u01/prod/proddb/9.2.0
Mon May 15 09:51:15 2023
ALTER DATABASE RECOVER  datafile 84  
Media Recovery Start
Mon May 15 09:51:15 2023
Errors in file /u01/prod/proddb/9.2.0/admin/udump/prod1_ora_467190.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []

通过处理之后,数据库recover 正常,但是open报ORA-600 4193错误

Mon May 15 09:57:53 2023
ALTER DATABASE RECOVER  DATABASE  
Media Recovery Start
Mon May 15 09:57:53 2023
Recovery of Online Redo Log: Thread 1 Group 1 Seq 4 Reading mem 0
  Mem# 0 errs 0: /dev/rlog01a.dbf
  Mem# 1 errs 0: /dev/rlog01b.dbf
Media Recovery Complete
Completed: ALTER DATABASE RECOVER  DATABASE  
Mon May 15 09:59:24 2023
alter database open
Mon May 15 09:59:24 2023
Beginning crash recovery of 1 threads
Mon May 15 09:59:24 2023
Started redo scan
Mon May 15 09:59:24 2023
Completed redo scan
 75 redo blocks read, 0 data blocks need recovery
Mon May 15 09:59:24 2023
Started recovery at
 Thread 1: logseq 4, block 2, scn 3301.2638369687
Mon May 15 09:59:24 2023
Recovery of Online Redo Log: Thread 1 Group 1 Seq 4 Reading mem 0
  Mem# 0 errs 0: /dev/rlog01a.dbf
  Mem# 1 errs 0: /dev/rlog01b.dbf
Mon May 15 09:59:24 2023
Completed redo application
Mon May 15 09:59:24 2023
Ended recovery at
 Thread 1: logseq 4, block 77, scn 3301.2638389765
 0 data blocks read, 0 data blocks written, 75 redo blocks read
Crash recovery completed successfully
Mon May 15 09:59:25 2023
Thread 1 advanced to log sequence 5
Thread 1 opened at log sequence 5
  Current log# 2 seq# 5 mem# 0: /dev/rlog02a.dbf
  Current log# 2 seq# 5 mem# 1: /dev/rlog02b.dbf
Successful open of redo thread 1
Mon May 15 09:59:25 2023
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon May 15 09:59:25 2023
SMON: enabling cache recovery
Mon May 15 09:59:25 2023
ARC0: Media recovery disabled
Mon May 15 09:59:25 2023
Successfully onlined Undo Tablespace 1.
Dictionary check beginning
Tablespace 'TEMP' #2 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Mon May 15 09:59:25 2023
SMON: enabling tx recovery
Mon May 15 09:59:25 2023
Database Characterset is ZHS16GBK
Mon May 15 09:59:25 2023
Errors in file /u01/prod/proddb/9.2.0/admin/bdump/prod1_smon_413872.trc:
ORA-00600: internal error code, arguments: [4193], [781], [6399], [], [], [], [], []
Mon May 15 09:59:25 2023
Errors in file /u01/prod/proddb/9.2.0/admin/udump/prod1_ora_844004.trc:
ORA-00600: internal error code, arguments: [4193], [56042], [1895], [], [], [], [], []
Mon May 15 09:59:26 2023
Doing block recovery for fno: 12 blk: 153
Mon May 15 09:59:26 2023
Doing block recovery for fno: 12 blk: 2893
Mon May 15 09:59:26 2023
Recovery of Online Redo Log: Thread 1 Group 2 Seq 5 Reading mem 0
  Mem# 0 errs 0: /dev/rlog02a.dbf
Mon May 15 09:59:26 2023
Recovery of Online Redo Log: Thread 1 Group 2 Seq 5 Reading mem 0
Mon May 15 09:59:26 2023
  Mem# 1 errs 0: /dev/rlog02b.dbf
Mon May 15 09:59:26 2023
  Mem# 0 errs 0: /dev/rlog02a.dbf
  Mem# 1 errs 0: /dev/rlog02b.dbf
Doing block recovery for fno: 12 blk: 3009
Mon May 15 09:59:26 2023
Recovery of Online Redo Log: Thread 1 Group 2 Seq 5 Reading mem 0
  Mem# 0 errs 0: /dev/rlog02a.dbf
  Mem# 1 errs 0: /dev/rlog02b.dbf
Mon May 15 09:59:26 2023
Doing block recovery for fno: 12 blk: 89
Mon May 15 09:59:26 2023
Recovery of Online Redo Log: Thread 1 Group 2 Seq 5 Reading mem 0
  Mem# 0 errs 0: /dev/rlog02a.dbf
  Mem# 1 errs 0: /dev/rlog02b.dbf
Mon May 15 09:59:26 2023
Errors in file /u01/prod/proddb/9.2.0/admin/udump/prod1_ora_844004.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4193], [56042], [1895], [], [], [], [], []
Error 607 happened during db open, shutting down database
USER: terminating instance due to error 607
Instance terminated by USER, pid = 844004
ORA-1092 signalled during: alter database open...

绕过该错误之后,数据库启动报ORA-600 2662错误

$ sqlplus "/ as sysdba"

SQL*Plus: Release 9.2.0.6.0 - Production on Mon May 15 10:04:44 2023

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

Connected to an idle instance.

SQL> startup mount pfile='/tmp/pfile'
ORACLE instance started.

Total System Global Area 1102023336 bytes
Fixed Size                   744104 bytes
Variable Size             922746880 bytes
Database Buffers          167772160 bytes
Redo Buffers               10760192 bytes
Database mounted.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel

Mon May 15 10:05:03 2023
SMON: enabling cache recovery
Mon May 15 10:05:03 2023
ARC0: Media recovery disabled
Mon May 15 10:05:03 2023
SMON: enabling tx recovery
Mon May 15 10:05:03 2023
Database Characterset is ZHS16GBK
Mon May 15 10:05:03 2023
Errors in file /u01/prod/proddb/9.2.0/admin/bdump/prod1_smon_413880.trc:
ORA-00600: internal error code, arguments: [2662], [3301], [2638409995], [3301], [2644132966], [4195678]
Mon May 15 10:05:04 2023
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
Mon May 15 10:05:04 2023
Errors in file /u01/prod/proddb/9.2.0/admin/bdump/prod1_smon_413880.trc:
ORA-00600: internal error code, arguments: [2662], [3301], [2638409998], [3301], [2644132966], [4195678]
Mon May 15 10:05:04 2023
Errors in file /u01/prod/proddb/9.2.0/admin/bdump/prod1_smon_413880.trc:
ORA-00600: internal error code, arguments: [2662], [3301], [2638409998], [3301], [2644132966], [4195678]
SMON: terminating instance due to error 600
Instance terminated by SMON, pid = 413880

解决该错误之后,数据库open正常

$ sqlplus "/ as sysdba"

SQL*Plus: Release 9.2.0.6.0 - Production on Mon May 15 10:10:30 2023

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

Connected to an idle instance.

SQL> startup mount pfile='/tmp/pfile'
ORACLE instance started.

Total System Global Area 1102023336 bytes
Fixed Size                   744104 bytes
Variable Size             922746880 bytes
Database Buffers          167772160 bytes
Redo Buffers               10760192 bytes
Database mounted.
SQL> alter database open;

Database altered.

逻辑方式导出数据,本次恢复任务基本完成.
以前有过的类似恢复案例(类似较多选择典型几个):
ORA-600 2663
ORA-600 2663 故障恢复
 ORA-600 2662
ora-600 2662和ora-600 kclchkblk_4恢复
 redo异常 ORA-600 kclchkblk_4 故障恢复
 ORA-600 4193 错误说明和解决
 ORA-00600 [2662]和ORA-00600 [4194]恢复

udev_start导致vip漂移(参见情况:rac在线加盘操作引起)

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：udev_start导致vip漂移(参见情况:rac在线加盘操作引起)

客户对asm进行扩容,执行udev_start命令之后,所有的vip全部漂移,业务全部中断

优先恢复业务,把所有vip漂移回来

[grid@rac3 ~]$  srvctl relocate vip -i rac1 -n rac1 -f -v
VIP was relocated successfully.
[grid@rac3 ~]$  srvctl relocate vip -i rac2 -n rac2 -f -v
VIP was relocated successfully.
[grid@rac3 ~]$  srvctl relocate vip -i rac3 -n rac3 -f -v
VIP was relocated successfully.
[grid@rac3 ~]$  srvctl relocate vip -i rac4 -n rac4 -f -v
VIP was relocated successfully.

vip恢复正常，业务也恢复正常

出现该问题的原因是由于udev_start命令引起网卡瞬间中断,从而使得vip发生漂移

查看ifcfg配置文件

引起该问题的原因是udev对网卡进行了操作,从而引起该问题,处理建议在对应的ifcfg文件中加上 HOTPLUG=”no” （pulbic,private和其他需要关注的网络）
参考:Network interface going down when dynamically adding disks to storage using udev in RHEL 6 (Doc ID 1569028.1)

又一例ORA-600 kcbzpbuf_1恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：又一例ORA-600 kcbzpbuf_1恢复

数据库突然报ORA-600 kdddgb1和ORA-600 kcl_snd_cur_2错误,并且导致实例crash

Tue May 09 22:29:40 2023
Errors in file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_338012.trc  (incident=962050):
ORA-00600: internal error code, arguments: [kdddgb1], [0], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_962050/orcl1_ora_338012_i962050.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Tue May 09 22:29:43 2023
Hex dump of (file 75, block 1154926) in trace file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lms3_217928.trc
Corrupt block relative dba: 0x12d19f6e (file 75, block 1154926)
Bad header found during preparing block for transfer
Data in bad block:
 type: 0 format: 2 rdba: 0x1affe051
 last change scn: 0x0009.a2266e65 seq: 0x2 flg: 0x10
 spare1: 0x83 spare2: 0x36 spare3: 0x3700
 consistency value in tail: 0x6e650002
 check value in block header: 0x0
 block checksum disabled
Errors in file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lms3_217928.trc  (incident=960186):
ORA-00600: internal error code, arguments: [kcl_snd_cur_2], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_960186/orcl1_lms3_217928_i960186.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Tue May 09 22:29:43 2023
Sweep [inc][962050]: completed
Sweep [inc][960186]: completed
Sweep [inc2][962050]: completed
Errors in file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lms3_217928.trc:
ORA-00600: internal error code, arguments: [kcl_snd_cur_2], [], [], [], [], [], [], [], [], [], [], []
LMS3 (ospid: 217928): terminating the instance due to error 484
System state dump requested by (instance=1, osid=217928 (LMS3)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_diag_217897_20230509222949.trc
Tue May 09 22:29:52 2023
ORA-1092 : opitsk aborting process
Tue May 09 22:29:53 2023
ORA-1092 : opitsk aborting process
Tue May 09 22:29:54 2023
Instance terminated by LMS3, pid = 217928

另外一个正在运行的实例做instance recovery,然后节点报ORA-600 kcbzpbuf_1,节点也crash,再次启动一直该错误无法正常启动.

Wed May 10 08:17:07 2023
Hex dump of (file 75, block 1154926) in trace file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbw9_134621.trc
Corrupt block relative dba: 0x12d19f6e (file 75, block 1154926)
Bad header found during preparing block for write
Data in bad block:
 type: 0 format: 2 rdba: 0x1affe051
 last change scn: 0x0009.a2266e65 seq: 0x2 flg: 0x34
 spare1: 0x83 spare2: 0x36 spare3: 0x3700
 consistency value in tail: 0x6e650002
 check value in block header: 0xf894
 computed block checksum: 0x0
Errors in file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbw9_134621.trc  (incident=2240402):
ORA-00600: internal error code, arguments: [kcbzpbuf_1], [4], [1], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_2240402/orcl1_dbw9_134621_i2240402.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbw9_134621.trc:
ORA-00600: internal error code, arguments: [kcbzpbuf_1], [4], [1], [], [], [], [], [], [], [], [], []
DBW9 (ospid: 134621): terminating the instance due to error 471
Wed May 10 08:17:08 2023
System state dump requested by (instance=1, osid=134621 (DBW9)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_diag_134555_20230510081708.trc
Instance terminated by DBW9, pid = 134621

尝试直接recover datafile 75失败,报ORA-03113

SQL> recover datafile 75;
ORA-03113: end-of-file on communication channel
Process ID: 281304
Session ID: 14161 Serial number: 1503

dbv检查file 75,发现15个block逻辑坏块

[oracle@oradb21 ~]$ dbv userid=xxx/xxx file=+datadg/orcl/datafile/xifenfei01.377.1130539753

DBVERIFY: Release 11.2.0.4.0 - Production on Wed May 10 08:29:44 2023

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = +datadg/orcl/datafile/xifenfei01.377.1130539753
Block Checking: DBA = 314866909, Block Type = KTB-managed data block
data header at 0x7f852b573064
kdbchk: row locked by non-existent transaction
        table=0   slot=13
        lockid=101   ktbbhitc=2
Page 294109 failed with check code 6101
Block Checking: DBA = 314866928, Block Type = KTB-managed data block
data header at 0x7f852b599064
kdbchk: row locked by non-existent transaction
        table=0   slot=18
        lockid=101   ktbbhitc=2
Page 294128 failed with check code 6101
Block Checking: DBA = 315415269, Block Type = KTB-managed data block
data header at 0x7f852b583064
kdbchk: the amount of space used is not equal to block size
        used=7470 fsc=0 avsp=625 dtl=8088
Page 842469 failed with check code 6110
Block Checking: DBA = 315415302, Block Type = KTB-managed data block
data header at 0x7f852b3c3064
kdbchk: row locked by non-existent transaction
        table=0   slot=13
        lockid=101   ktbbhitc=2
Page 842502 failed with check code 6101
Block Checking: DBA = 315415350, Block Type = KTB-managed data block
data header at 0x7f852b423064
kdbchk: row locked by non-existent transaction
        table=0   slot=14
        lockid=101   ktbbhitc=2
Page 842550 failed with check code 6101
Block Checking: DBA = 315415351, Block Type = KTB-managed data block
data header at 0x7f852b425064
kdbchk: row locked by non-existent transaction
        table=0   slot=10
        lockid=101   ktbbhitc=2
Page 842551 failed with check code 6101
Block Checking: DBA = 315415397, Block Type = KTB-managed data block
data header at 0x7f852b481064
kdbchk: row locked by non-existent transaction
        table=0   slot=14
        lockid=101   ktbbhitc=2
Page 842597 failed with check code 6101
Block Checking: DBA = 315415414, Block Type = KTB-managed data block
data header at 0x7f852b4a3064
kdbchk: row locked by non-existent transaction
        table=0   slot=14
        lockid=101   ktbbhitc=2
Page 842614 failed with check code 6101
Block Checking: DBA = 315665300, Block Type = KTB-managed data block
data header at 0x7f852b2dd0ac
kdbchk: the amount of space used is not equal to block size
        used=7191 fsc=0 avsp=832 dtl=8016
Page 1092500 failed with check code 6110
Block Checking: DBA = 315665302, Block Type = KTB-managed data block
data header at 0x7f852b2e10ac
kdbchk: row locked by non-existent transaction
        table=0   slot=14
        lockid=101   ktbbhitc=5
Page 1092502 failed with check code 6101
Block Checking: DBA = 315665316, Block Type = KTB-managed data block
data header at 0x7f852b2fd0ac
kdbchk: the amount of space used is not equal to block size
        used=7140 fsc=0 avsp=883 dtl=8016
Page 1092516 failed with check code 6110
Block Checking: DBA = 315665491, Block Type = KTB-managed data block
data header at 0x7f852f4170c4
kdbchk: row locked by non-existent transaction
        table=0   slot=3
        lockid=101   ktbbhitc=6
Page 1092691 failed with check code 6101
Block Checking: DBA = 315727518, Block Type = KTB-managed data block
data header at 0x7f852b4f50c4
kdbchk: row locked by non-existent transaction
        table=0   slot=8
        lockid=101   ktbbhitc=6
Page 1154718 failed with check code 6101
Block Checking: DBA = 315727614, Block Type = KTB-managed data block
data header at 0x7f852b5b50ac
kdbchk: row locked by non-existent transaction
        table=0   slot=15
        lockid=101   ktbbhitc=5
Page 1154814 failed with check code 6101
Block Checking: DBA = 315727646, Block Type = KTB-managed data block
data header at 0x7f852b3f30ac
kdbchk: row locked by non-existent transaction
        table=0   slot=3
        lockid=101   ktbbhitc=5
Page 1154846 failed with check code 6101


DBVERIFY - Verification complete

Total Pages Examined         : 1835008
Total Pages Processed (Data) : 250749
Total Pages Failing   (Data) : 15
Total Pages Processed (Index): 74532
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 1244181
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 265546
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 2720428335 (9.2720428335)

通过对坏块一些处理,数据库open成功,以前有过类似恢复ORA-600 kcbzpbuf_1故障恢复

SQL> alter database open;

Database altered.

alert日志报事务异常

ORACLE Instance orcl1 (pid = 34) - Error 1578 encountered while recovering transaction (697, 6) on object 170692.
Errors in file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_smon_301450.trc:
ORA-01578: ORACLE data block corrupted (file # 75, block # 1154926)
ORA-01110: data file 75: '+DATADG/orcl/datafile/xifenfei01.377.1130539753'
Archived Log entry 9299 added for thread 1 sequence 4781 ID 0x5f4a1865 dest 1:
Wed May 10 08:24:03 2023
NOTE: dependency between database orcl and diskgroup resource ora.ARCHDG.dg is established
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Wed May 10 08:24:04 2023
Starting background process EMNC
Wed May 10 08:24:04 2023
EMNC started with pid=49, OS id=305303 
Archived Log entry 9300 added for thread 2 sequence 4530 ID 0x5f4a1865 dest 1:
ARC2: Archiving disabled thread 2 sequence 4531
Archived Log entry 9301 added for thread 2 sequence 4531 ID 0x5f4a1865 dest 1:
Wed May 10 08:24:13 2023
Errors in file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_p000_305307.trc  (incident=2560578):
ORA-01578: ORACLE data block corrupted (file # 75, block # 1154926)
ORA-01110: data file 75: '+DATADG/orcl/datafile/xifenfei01.377.1130539753'
Incident details in: /oracle/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_2560578/orcl1_p000_305307_i2560578.trc
Errors in file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_p000_305307.trc  (incident=2560579):
ORA-01578: ORACLE data block corrupted (file # , block # )
Incident details in: /oracle/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_2560579/orcl1_p000_305307_i2560579.trc
Wed May 10 08:24:15 2023
Errors in file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_smon_301450.trc  (incident=2560427):
ORA-01578: ORACLE data block corrupted (file # 75, block # 1154926)
ORA-01110: data file 75: '+DATADG/orcl/datafile/xifenfei01.377.1130539753'
Incident details in: /oracle/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_2560427/orcl1_smon_301450_i2560427.trc
Errors in file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_smon_301450.trc  (incident=2560432):
ORA-01578: ORACLE data block corrupted (file # 75, block # 1154926)
ORA-01110: data file 75: '+DATADG/orcl/datafile/xifenfei01.377.1130539753'
ORACLE Instance orcl1 (pid = 34) - Error 1578 encountered while recovering transaction (717, 20) on object 170692.
Errors in file /oracle/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_smon_301450.trc:
ORA-01578: ORACLE data block corrupted (file # 75, block # 1154926)
ORA-01110: data file 75: '+DATADG/orcl/datafile/xifenfei01.377.1130539753'

处理异常事务,并且定位异常对象表

SQL> select owner,object_name,object_type from dba_objects where object_id=170692;

OWNER
--------------------------------------------------------------------------------
OBJECT_NAME
--------------------------------------------------------------------------------
OBJECT_TYPE
---------------------------------------------------------
XFF
T_XIFENFEI
TABLE

rman检测逻辑坏块所属对象也是这个表（15个坏块均为该表）,对该表数据进行重建抛弃损坏数据,完成本次恢复

ORA-01172 ORA-01151 故障恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-01172 ORA-01151 故障恢复

节点2报Error: Controlfile sequence number in file header is different from the one in memory,导致实例异常

Tue May 09 23:03:24 2023
Thread 2 cannot allocate new log, sequence 16728
Checkpoint not complete
  Current log# 3 seq# 16727 mem# 0: +DATA/xff/onlinelog/group_3.265.941900045
  Current log# 3 seq# 16727 mem# 1: +FRA/xff/onlinelog/group_3.259.941900045
Thread 2 advanced to log sequence 16728 (LGWR switch)
  Current log# 4 seq# 16728 mem# 0: +DATA/xff/onlinelog/group_4.266.941900045
  Current log# 4 seq# 16728 mem# 1: +FRA/xff/onlinelog/group_4.260.941900045
Tue May 09 23:03:31 2023
LNS: Standby redo logfile selected for thread 2 sequence 16728 for destination LOG_ARCHIVE_DEST_2
Tue May 09 23:03:32 2023
Archived Log entry 431615 added for thread 2 sequence 16727 ID 0x5ffc99b5 dest 1:
Tue May 09 23:05:30 2023
Error: Controlfile sequence number in file header is different from the one in memory
       Please check that the correct mount options are used if controlfile is located on NFS
USER (ospid: 30162): terminating the instance
Tue May 09 23:05:30 2023
System state dump requested by (instance=2, osid=30162), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_diag_6650.trc
Instance terminated by USER, pid = 30162

在节点1 进行实例重组之后,节点1 实例异常

Tue May 09 23:04:54 2023
Thread 1 cannot allocate new log, sequence 2060
Checkpoint not complete
  Current log# 1 seq# 2059 mem# 0: +DATA/xff/onlinelog/group_1.261.941899887
  Current log# 1 seq# 2059 mem# 1: +FRA/xff/onlinelog/group_1.257.941899887
Thread 1 advanced to log sequence 2060 (LGWR switch)
  Current log# 2 seq# 2060 mem# 0: +DATA/xff/onlinelog/group_2.262.941899889
  Current log# 2 seq# 2060 mem# 1: +FRA/xff/onlinelog/group_2.258.941899889
Tue May 09 23:04:58 2023
********************* ATTENTION: ******************** 
 The controlfile header block returned by the OS
 has a sequence number that is too old. 
 The controlfile might be corrupted.
 PLEASE DO NOT ATTEMPT TO START UP THE INSTANCE 
 without following the steps below.
 RE-STARTING THE INSTANCE CAN CAUSE SERIOUS DAMAGE 
 TO THE DATABASE, if the controlfile is truly corrupted.
 In order to re-start the instance safely, 
 please do the following:
 (1) Save all copies of the controlfile for later 
     analysis and contact your OS vendor and Oracle support.
 (2) Mount the instance and issue: 
     ALTER DATABASE BACKUP CONTROLFILE TO TRACE;
 (3) Unmount the instance. 
 (4) Use the script in the trace file to
     RE-CREATE THE CONTROLFILE and open the database. 
*****************************************************
Tue May 09 23:05:31 2023
Reconfiguration started (old inc 20, new inc 22)
List of instances:
 1 (myinst: 1) 
 Global Resource Directory frozen
 * dead instance detected - domain 0 invalid = TRUE 
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Tue May 09 23:05:31 2023
 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Tue May 09 23:05:31 2023
 LMS 0: 3 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info 
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Post SMON to start 1st pass IR
Tue May 09 23:05:32 2023
Instance recovery: looking for dead threads
 Submitted all GCS remote-cache requests
 Post SMON to start 1st pass IR
 Fix write in gcs resources
Reconfiguration complete
Tue May 09 23:06:00 2023
ARC1 (ospid: 26512): terminating the instance
Tue May 09 23:06:00 2023
System state dump requested by (instance=1, osid=26512 (ARC1)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_diag_26311.trc
Tue May 09 23:06:01 2023
ORA-1092 : opitsk aborting process
Instance terminated by ARC1, pid = 26512

实例重启报错

Recovery of Online Redo Log: Thread 1 Group 1 Seq 2059 Reading mem 0
  Mem# 0: +DATA/dbm/onlinelog/group_1.261.941899887
  Mem# 1: +FRA/dbm/onlinelog/group_1.257.941899887
Recovery of Online Redo Log: Thread 2 Group 3 Seq 16727 Reading mem 0
  Mem# 0: +DATA/dbm/onlinelog/group_3.265.941900045
  Mem# 1: +FRA/dbm/onlinelog/group_3.259.941900045
Recovery of Online Redo Log: Thread 2 Group 4 Seq 16728 Reading mem 0
  Mem# 0: +DATA/dbm/onlinelog/group_4.266.941900045
  Mem# 1: +FRA/dbm/onlinelog/group_4.260.941900045
Hex dump of (file 1, block 102777) in trace file /u01/app/oracle/diag/rdbms/dbm/dbm2/trace/dbm2_ora_30749.trc
Reading datafile '+DATA/dbm/datafile/system.256.941899799' for corruption at rdba: 0x00419179 (file 1, block 102777)
Reread (file 1, block 102777) found different corrupt data (logically corrupt)
Hex dump of (file 1, block 102777) in trace file /u01/app/oracle/diag/rdbms/dbm/dbm2/trace/dbm2_ora_30749.trc
RECOVERY OF THREAD 2 STUCK AT BLOCK 102777 OF FILE 1
Abort recovery for domain 0
Aborting crash recovery due to error 1172
Errors in file /u01/app/oracle/diag/rdbms/dbm/dbm2/trace/dbm2_ora_30749.trc:
ORA-01172: recovery of thread 2 stuck at block 102777 of file 1
ORA-01151: use media recovery to recover block, restore backup if needed
Abort recovery for domain 0
Errors in file /u01/app/oracle/diag/rdbms/dbm/dbm2/trace/dbm2_ora_30749.trc:
ORA-01172: recovery of thread 2 stuck at block 102777 of file 1
ORA-01151: use media recovery to recover block, restore backup if needed
ORA-1172 signalled during: ALTER DATABASE OPEN /* db agent *//* {0:890:17} */...

人工recover操作失败报ORA-600 3020错误

SQL> recover datafile 1;
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [3020], [1], [102777], [4297081],[], []
ORA-10567: Redo is inconsistent with data block (file# 1, block# 102777, file
offset is 841949184 bytes)
ORA-10564: tablespace SYSTEM
ORA-01110: data file 1: '+DATA/dbm/datafile/system.256.941899799'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 469884

---alert日志
Tue May 09 23:28:44 2023
ALTER DATABASE RECOVER  datafile 1  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 2 Group 3 Seq 16727 Reading mem 0
  Mem# 0: +DATA/xff/onlinelog/group_3.265.941900045
  Mem# 1: +FRA/xff/onlinelog/group_3.259.941900045
ORA-279 signalled during: ALTER DATABASE RECOVER  datafile 1  ...
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log +FRA/xff/archivelog/2023_05_09/thread_1_seq_2055.20899.1136415701
ORA-279 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log +FRA/xff/archivelog/2023_05_09/thread_1_seq_2056.20837.1136415753
ORA-279 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log +FRA/xff/archivelog/2023_05_09/thread_1_seq_2057.20911.1136415803
ORA-279 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log +FRA/xff/archivelog/2023_05_09/thread_1_seq_2058.21898.1136415853
Recovery of Online Redo Log: Thread 2 Group 4 Seq 16728 Reading mem 0
  Mem# 0: +DATA/xff/onlinelog/group_4.266.941900045
  Mem# 1: +FRA/xff/onlinelog/group_4.260.941900045
Recovery of Online Redo Log: Thread 1 Group 1 Seq 2059 Reading mem 0
  Mem# 0: +DATA/xff/onlinelog/group_1.261.941899887
  Mem# 1: +FRA/xff/onlinelog/group_1.257.941899887
Hex dump of (file 1, block 102777) in trace file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_16246.trc
Reading datafile '+DATA/xff/datafile/system.256.941899799' for corruption at rdba: 0x00419179 (file 1, block 102777)
Reread (file 1, block 102777) found different corrupt data (logically corrupt)
Hex dump of (file 1, block 102777) in trace file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_16246.trc
Tue May 09 23:28:59 2023
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_16246.trc  (incident=6868615):
ORA-00600: internal error code, arguments: [3020], [1], [102777], [4297081], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 1, block# 102777, file offset is 841949184 bytes)
ORA-10564: tablespace SYSTEM
ORA-01110: data file 1: '+DATA/xff/datafile/system.256.941899799'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 469884
Incident details in: /u01/app/oracle/diag/rdbms/xff/xff1/incident/incdir_6868615/xff1_ora_16246_i6868615.trc
Tue May 09 23:29:00 2023
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Media Recovery failed with error 600
ORA-283 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER CANCEL 
ORA-1112 signalled during: ALTER DATABASE RECOVER CANCEL ...

根据上述报错信息可以确认报错的是一个index,而且非系统核心对象,可以通过allow 1 corruption方式进行恢复,并且open库成功

SQL> recover  datafile 1 allow 1 corruption;
Media recovery complete.
SQL> alter database open;

Database altered.

SQL> select owner,object_name,object_type from dba_objects where object_id=469884;

OWNER
--------------------------------------------------------------------------------
OBJECT_NAME
--------------------------------------------------------------------------------
OBJECT_TYPE
---------------------------------------------------------
SYSTEM
PK_XFF_SERVERS
INDEX

SQL> alter index system.PK_XFF_SERVERS rebuild online;

Index altered.

数据库完美恢复,数据0丢失,业务可以直接正常使用