ORA-600 krse_arc_complete.4

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-600 krse_arc_complete.4

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

11.2.0.4版本数据库对于虚拟化环境中的备库进行克隆,然后尝试failover方式激活备库,结果遭遇ORA-600 krse_arc_complete.4错误

SQL> ALTER DATABASE ACTIVATE PHYSICAL STANDBY DATABASE;
ALTER DATABASE ACTIVATE PHYSICAL STANDBY DATABASE
*
ERROR at line 1:
ORA-01196: file 1 is inconsistent due to a failed media recovery session
ORA-01110: data file 1: '/u01/app/oracle/oradata/hisdb/system.256.975233377'

alert日志显示

Wed Oct 23 21:45:46 2024
ALTER DATABASE ACTIVATE PHYSICAL STANDBY DATABASE
ALTER DATABASE ACTIVATE [PHYSICAL] STANDBY DATABASE (hisdb)
Begin: Standby Redo Logfile archival
End: Standby Redo Logfile archival
Wed Oct 23 21:45:46 2024
ARC0: Detected ARCH process failure
ARC0: STARTING ARCH PROCESSES
Wed Oct 23 21:45:46 2024
ARC3 started with pid=22, OS id=28848 
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Beginning Standby Crash Recovery.
Serial Media Recovery started
Managed Standby Recovery starting Real Time Apply
Errors in file /u01/app/oracle/diag/rdbms/hisdbdg/hisdb/trace/hisdb_arc0_28695.trc  (incident=528155):
ORA-00600: internal error code, arguments: [krse_arc_complete.4], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/hisdbdg/hisdb/incident/incdir_528155/hisdb_arc0_28695_i528155.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Master archival failure: 600
Wed Oct 23 21:45:47 2024
Deleted Oracle managed file /fra/fast_recovery_area/HISDBDG/archivelog/2024_10_23/o1_mf_2_158418_mkkzjbqt_.arc
Warning: Datafile 1 (/u01/app/oracle/oradata/hisdb/system.256.975233377) is infinitely media recovery fuzzy
Standby database will not open with this datafile online!
Standby Crash Recovery aborted due to error 10554.
Errors in file /u01/app/oracle/diag/rdbms/hisdbdg/hisdb/trace/hisdb_ora_28842.trc:
ORA-10554: Media recovery failed to bring datafile 1 to a consistent point
ORA-01110: data file 1: '/u01/app/oracle/oradata/hisdb/system.256.975233377'
Completed Standby Crash Recovery.
Wed Oct 23 21:45:47 2024
Errors in file /u01/app/oracle/diag/rdbms/hisdbdg/hisdb/trace/hisdb_arc2_28840.trc  (incident=528172):
ORA-00600: internal error code, arguments: [krse_arc_complete.4], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/hisdbdg/hisdb/incident/incdir_528172/hisdb_arc2_28840_i528172.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Master archival failure: 600
ORA-1196 signalled during: ALTER DATABASE ACTIVATE PHYSICAL STANDBY DATABASE...
ARC3: Detected ARCH process failure

查询mos 发现是尝试应用standby log,并且arc进程尝试对其进行归档,发现归档失败从而报该错误,可以尝试对standby log进行clear,然后再激活备库

SQL> select group#,status from v$standby_log;

          GROUP# STATUS
---------------- ----------
              11 UNASSIGNED
              12 UNASSIGNED
              13 UNASSIGNED
              14 ACTIVE
              15 UNASSIGNED
              16 UNASSIGNED
              17 UNASSIGNED
              18 UNASSIGNED

8 rows selected.

SQL> alter database clear unarchived logfile group 14;

Database altered.

SQL> ALTER DATABASE ACTIVATE PHYSICAL STANDBY DATABASE;

Database altered.

Wed Oct 23 21:55:12 2024
ALTER DATABASE ACTIVATE PHYSICAL STANDBY DATABASE
ALTER DATABASE ACTIVATE [PHYSICAL] STANDBY DATABASE (hisdb)
Begin: Standby Redo Logfile archival
End: Standby Redo Logfile archival
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
Standby terminal recovery start SCN: 35738445353
RESETLOGS after incomplete recovery UNTIL CHANGE 35738178180
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
Resetting resetlogs activation ID 1887849281 (0x70864b41)
Online log /u01/app/oracle/oradata/hisdb/group_5.272.976991793: Thread 1 Group 5 was previously cleared
Online log /u01/app/oracle/oradata/hisdb/group_5.2338.976991793: Thread 1 Group 5 was previously cleared
Online log /u01/app/oracle/oradata/hisdb/group_6.273.976991805: Thread 1 Group 6 was previously cleared
Online log /u01/app/oracle/oradata/hisdb/group_6.2339.976991805: Thread 1 Group 6 was previously cleared
Online log /u01/app/oracle/oradata/hisdb/group_7.274.976991825: Thread 2 Group 7 was previously cleared
Online log /u01/app/oracle/oradata/hisdb/group_7.2336.976991825: Thread 2 Group 7 was previously cleared
Online log /u01/app/oracle/oradata/hisdb/group_8.275.976991863: Thread 2 Group 8 was previously cleared
Online log /u01/app/oracle/oradata/hisdb/group_8.2337.976991863: Thread 2 Group 8 was previously cleared
Online log /u01/app/oracle/oradata/hisdb/group_9.276.976991877: Thread 1 Group 9 was previously cleared
Online log /u01/app/oracle/oradata/hisdb/group_9.2334.976991877: Thread 1 Group 9 was previously cleared
Online log /u01/app/oracle/oradata/hisdb/group_10.277.976991891: Thread 2 Group 10 was previously cleared
Online log /u01/app/oracle/oradata/hisdb/group_10.2333.976991893: Thread 2 Group 10 was previously cleared
Standby became primary SCN: 35738445352
Wed Oct 23 21:55:21 2024
Setting recovery target incarnation to 3
ACTIVATE STANDBY: Complete - Database mounted as primary
Completed: ALTER DATABASE ACTIVATE PHYSICAL STANDBY DATABASE

参考:Activate Standby Database failed with ORA-00600: [krse_arc_complete.4] (Doc ID 2409336.1)

Oracle 19c 202410补丁(RUs+OJVM)

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:Oracle 19c 202410补丁(RUs+OJVM)

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

19.0.0.0
 Description  Database Update  GI Update  Windows Bundle Patch
 OCT2024 (19.25.0.0.0) 36912597  36916690  36878821
 JUL2024 (19.24.0.0.0) 36582781  36582629  36521936
 APR2024 (19.23.0.0.0) 36233263  36233126  36219938
 JAN2024 (19.22.0.0.0) 35943157  35940989  35962832
 OCT2023 (19.21.0.0.0) 35643107  35642822  35681552
 JUL2023 (19.20.0.0.0) 35320081  35319490  35348034
 APR2023 (19.19.0.0.0) 35042068  35037840  35046439
 JAN2023 (19.18.0.0.0) 34765931  34762026  34750795
 Oct2022 (19.17.0.0.0) 34419443  34416665  34468114
 JUL2022 (19.16.0.0.0) 34133642  34130714  34110685
 APR2022 (19.15.0.0.0) 33806152  33803476  33829175
 JAN2022 (19.14.0.0.0) 33515361  33509923  33575656
 OCT2021(19.13.0.0.0) 33192793  33182768  33155330
 JUL2021 (19.12.0.0.0) 32904851  32895426  32832237
 APR2021 (19.11.0.0.0) 32545013  32545008  32409154
 JAN2021 (19.10.0.0.0) 32218454  32226239  32062765
 OCT2020 (19.9.0.0.0) 31771877  31750108  31719903
 JUL2020  (19.8.0.0.0) 31281355  31305339  31247621
 APR2020 (19.7.0.0.0) 30869156  30899722  30901317
 JAN2020 (19.6.0.0.0) 30557433  30501910  30445947
 OCT2019 (19.5.0.0.0) 30125133  30116789  30151705
 JUL2019 (19.4.0.0.0) 29834717  29708769   NA
 APR2019 (19.3.0.0.0) 29517242  29517302   NA
19.0.0.0
 Description  OJVM Update  OJVM + DB Update  OJVM + GI Update
 OCT2024 (19.25.0.0.241015)  36878697  36866623  36866740
 JUL2024 (19.24.0.0.240716)  36414915  36522340  36522439
 APR2024 (19.23.0.0.240416)  36199232  36209492  36209493
 JAN2024 (19.22.0.0.240116)  35926646  36031426  36031453
 OCT2023 (19.21.0.0.231017)  35648110  35742413  35742441
 JUL2023 (19.20.0.0.230718)  35354406  35370174  35370167
 APR2023 (19.19.0.0.230418)  35050341  35058163  35058172
 JAN2023 (19.18.0.0.230117)  34786990  34773489  34773504
 OCT2022 (19.17.0.0.221018)  34411846  34449114  34449117
 JUL2022 (19.16.0.0.220719)  34086870  34160831  34160854
 APR2022 (19.15.0.0.220419)  33808367  33859194  33859214
 JAN2022 (19.14.0.0.220118)  33561310  33567270  33567274
 OCT2021 (19.13.0.0.211019)  33192694  33248420  33248471
 JUL2021 (19.12.0.0.210720)  32876380  32900021  32900083
 APR2021 (19.11.0.0.210420)  32399816  32578972  32578973
 JAN2021 (19.10.0.0.210119)  32067171  32126828  32126842
 OCT2020 (19.9.0.0.201020)  31668882  31720396  31720429
 JUL2020 (19.8.0.0.200714)  31219897  31326362  31326369
 APR2020 (19.7.0.0.200414)  30805684  30783543  30783556
 JAN2020 (19.6.0.0.200114)  30484981  30463595  30463609
 OCT2019 (19.5.0.0.191015)  30128191  30133124  30133178
 JUL2019 (19.4.0.0.190716)  29774421  29699079  29699097
 APR2019 (19.3.0.0.190416)  29548437  29621253  29621299

参考:Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), Bundle Patches, Patchsets and Base Releases (Doc ID 2118136.2)

ntfs MFT损坏(ntfs文件系统故障)导致oracle异常恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ntfs MFT损坏(ntfs文件系统故障)导致oracle异常恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户虚拟化环境,由于断电,启动数据库报ORA-01157错误,通过操作系统层面查看,发现文件是存在的,但是dbv检测报不可访问
ora-01157


感觉是文件系统损坏了,尝试把该文件拷贝到其他磁盘
221509

查看操作系统事件,确认是ntfs文件系统的MFT损坏
mft

基于这种情况,通过文件系统恢复工具进行恢复该文件尝试,提示恢复文件大小和实际元数据中记录大小不一致
214712

通过对比实际恢复大小和文件本身大小,发现7811899392-7791460352,几乎等于20M大小(也就是说恢复出来的数据文件少了20M),通过分析数据库alert日志,确认该系统在前端时间刚好扩展了20M(增加数据文件之时指定了每次扩展20m)

2023-08-11T11:29:21.397236+08:00
ALTER TABLESPACE "HSHIS" ADD DATAFILE
'D:\APP\ADMINISTRATOR\ORADATA\HIS\HSHIS01.DBF' SIZE 10M AUTOEXTEND ON NEXT 20M MAXSIZE 8001M
Completed: ALTER TABLESPACE "HSHIS" ADD DATAFILE
'D:\APP\ADMINISTRATOR\ORADATA\HIS\HSHIS01.DBF' SIZE 10M AUTOEXTEND ON NEXT 20M MAXSIZE 8001M

2024-10-09T00:18:31.058537+08:00
Resize operation completed for file# 66, old size 7608320K, new size 7628800K

通过对该文件底层block分析,确认最终丢失block就是最后20M(直接的数据文件的block的rdba均正确),对于这种故障,通过填补数据文件尾部,欺骗数据库完成该文件的恢复(最后20M中如果写入了业务数据,可能会丢失),做好该文件修复工作之后,尝试打开数据库,结果很不乐观,redo也损坏
recover-error


屏蔽一致性,强制打开库成功

2024-10-18T04:24:43.911107+08:00
ALTER DATABASE RECOVER    CANCEL  
2024-10-18T04:24:47.098637+08:00
Errors in file E:\TRACE\diag\rdbms\his\his\trace\his_pr00_2608.trc:
ORA-01547: 警告: RECOVER 成功但 OPEN RESETLOGS 将出现如下错误
ORA-01194: 文件 1 需要更多的恢复来保持一致性
ORA-01110: 数据文件 1: 'E:\ORADATA\SYSTEM01.DBF'
2024-10-18T04:24:47.114278+08:00
ORA-1547 signalled during: ALTER DATABASE RECOVER    CANCEL  ...
ALTER DATABASE RECOVER CANCEL 
ORA-1112 signalled during: ALTER DATABASE RECOVER CANCEL ...
2024-10-18T04:25:03.989398+08:00
alter database open resetlogs
2024-10-18T04:25:05.598781+08:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 2666786639 time 
Resetting resetlogs activation ID 3659241623 (0xda1b9897)
2024-10-18T04:25:12.380089+08:00
Setting recovery target incarnation to 3
2024-10-18T04:25:15.052071+08:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
Endian type of dictionary set to little
2024-10-18T04:25:15.458286+08:00
Assigning activation ID 3703362676 (0xdcbcd474)
2024-10-18T04:25:15.505102+08:00
TT00 (PID:4092): Gap Manager starting
2024-10-18T04:25:15.551992+08:00
Redo log for group 1, sequence 1 is not located on DAX storage
2024-10-18T04:25:17.833250+08:00
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: E:\ORADATA\REDO01.LOG
Successful open of redo thread 1
2024-10-18T04:25:17.848888+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2024-10-18T04:25:22.052035+08:00
Undo initialization recovery: err:0 start: 24275578 end: 24276578 diff: 1000 ms (1.0 seconds)
Undo initialization online undo segments: err:0 start: 24276578 end: 24276593 diff: 15 ms (0.0 seconds)
Undo initialization finished serial:0 start:24275578 end:24276640 diff:1062 ms (1.1 seconds)
Dictionary check beginning
Dictionary check complete
Verifying minimum file header compatibility for tablespace encryption..
Verifying file header compatibility for tablespace encryption completed for pdb 0
2024-10-18T04:25:23.114610+08:00
Database Characterset is AL32UTF8
No Resource Manager plan active
2024-10-18T04:25:29.036475+08:00
replication_dependency_tracking turned off (no async multimaster replication found)
2024-10-18T04:25:32.833386+08:00
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Starting background process AQPC
2024-10-18T04:25:33.145881+08:00
AQPC started with pid=37, OS id=5560 
2024-10-18T04:25:35.677167+08:00
Starting background process CJQ0
2024-10-18T04:25:35.708430+08:00
CJQ0 started with pid=39, OS id=2728 
2024-10-18T04:25:36.724036+08:00
Completed: alter database open resetlogs

然后导出数据到新库,其中遇到了file# 66号文件最后丢失的20M引起的数据无法正常导出的问题处理(丢弃损坏部分数据,把剩余好的表中数据恢复到新库中)

.mkp扩展名oracle数据文件加密恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:.mkp扩展名oracle数据文件加密恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

又有客户的oracle数据库文件被勒索,扩展名被加密为:.[tsai.shen@mailfence.com].mkp
[tsai.shen@mailfence.com].mkp


对应的+README-WARNING+.txt文件内容类似:

::: Greetings :::

Little FAQ:

.1. 
Q: Whats Happen?
A: Your files have been encrypted. The file structure was not damaged, we did everything possible so that this could not happen.

.2. 
Q: How to recover files?
A: If you wish to decrypt your files you will need to pay us.

.3. 
Q: What about guarantees?
A: Its just a business. We absolutely do not care about you and your deals, except getting benefits. If we do not do our work and liabilities - nobody will cooperate with us. Its not in our interests.
To check the ability of returning files, you can send to us any 2 files with SIMPLE extensions(jpg,xls,doc, etc... not databases!) and low sizes(max 1 mb), we will decrypt them and send back to you. That is our guarantee.

.4.
Q: How to contact with you?
A: You can write us to our mailboxes: tsai.shen@mailfence.com or nicetomeetyou@onionmail.org

.5.
Q: How will the decryption process proceed after payment?
A: After payment we will send to you our scanner-decoder program and detailed instructions for use. With this program you will be able to decrypt all your encrypted files.

.6.
Q: If I don抰 want to pay bad people like you?
A: If you will not cooperate with our service - for us, its does not matter. But you will lose your time and data, cause only we have the private key. In practice - time is much more valuable than money.



:::BEWARE:::
DON'T try to change encrypted files by yourself! 
If you will try to use any third party software for restoring your data or antivirus solutions - please make a backup for all encrypted files!
Any changes in encrypted files may entail damage of the private key and, as result, the loss all data.

通过分析确认该文件就是头部32个block被破坏,因为oracle数据库中前面32个block是不含业务数据(主要是文件头和位图信息),通过自研的Oracle数据文件加密勒索恢复工具对其进行修复
mkp-recover-tools


重建ctl之后,open数据库查询用户创建时间证明库是直接open成功的
dba_users

根据客户需求截图业务数据
112

类似勒索病毒预防建议:
1. 教育和培训:提高用户的网络安全意识非常重要。通过定期的网络安全培训和教育,向用户传达有关勒索病毒及其传播方式的知识,让他们能够警惕潜在的威胁,并学会如何正确应对可疑的电子邮件、链接和附件。
2. 更新和维护:及时更新操作系统、应用程序和安全软件,以修补已知的漏洞,并确保系统能够及时获取最新的安全补丁。此外,定期进行系统维护和检查,确保系统的安全配置和设置。
3. 备份数据:定期备份重要的数据和文件,并将备份存储在安全的离线或云存储中。确保备份是完整的、可靠的,并且能够及时恢复,以便在发生勒索病毒感染或其他数据丢失事件时能够快速恢复数据。
4. 网络安全工具:使用可信赖的网络安全工具,包括防病毒软件、防火墙、入侵检测系统等,以提高系统的安全性和防护能力。定期对系统进行全面的安全扫描和检测,及时发现并清除潜在的威胁。
5. 访问控制:实施严格的访问控制措施,限制用户对系统和文件的访问权限,避免使用管理员权限进行日常操作,以减少恶意软件感染的风险。此外,定期审查和更新访问控制策略,确保系统安全性得到有效维护。
6. 应急响应计划:制定和实施应急响应计划,明确团队成员的责任和任务,建立应对勒索病毒和其他安全事件的应急响应流程,以最大程度地减少损失并快速恢复业务正常运营。

如果此类的数据库(oracle,mysql,sql server)等被加密,需要专业恢复技术支持,请联系我们:
电话/微信:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com

清空redo,导致ORA-27048: skgfifi: file header information is invalid

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:清空redo,导致ORA-27048: skgfifi: file header information is invalid

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户由于空间不足,使用> redo命令清空了oracle的redo文件
redo


数据库挂掉之后,启动报错

Fri Oct 04 10:32:57 2024
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 31 processes
Started redo scan
Errors in file /home/oracle/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_24876.trc:
ORA-00313: open failed for members of log group 3 of thread 1
ORA-00312: online log 3 thread 1: '/u01/app/oracle/oradata/xifenfei/redo03.log'
ORA-27048: skgfifi: file header information is invalid
Additional information: 13
Aborting crash recovery due to error 313
Errors in file /home/oracle/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_24876.trc:
ORA-00313: open failed for members of log group 3 of thread 1
ORA-00312: online log 3 thread 1: '/u01/app/oracle/oradata/xifenfei/redo03.log'
ORA-27048: skgfifi: file header information is invalid
Additional information: 13
Errors in file /home/oracle/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_24876.trc:
ORA-00313: open failed for members of log group 3 of thread 1
ORA-00312: online log 3 thread 1: '/u01/app/oracle/oradata/xifenfei/redo03.log'
ORA-27048: skgfifi: file header information is invalid
Additional information: 13
ORA-313 signalled during: alter database open...
Fri Oct 04 10:32:58 2024
Errors in file /home/oracle/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_m000_29646.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/xifenfei/redo01.log'
ORA-27047: unable to read the header block of file
Linux-x86_64 Error: 25: Inappropriate ioctl for device
Additional information: 1
Errors in file /home/oracle/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_m000_29646.trc:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/u01/app/oracle/oradata/xifenfei/redo02.log'
ORA-27047: unable to read the header block of file
Linux-x86_64 Error: 25: Inappropriate ioctl for device
Additional information: 1
Errors in file /home/oracle/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_m000_29646.trc:
ORA-00313: open failed for members of log group 3 of thread 1
ORA-00312: online log 3 thread 1: '/u01/app/oracle/oradata/xifenfei/redo03.log'
ORA-27048: skgfifi: file header information is invalid
Additional information: 11
Checker run found 6 new persistent data failures
Fri Oct 04 10:47:32 2024
db_recovery_file_dest_size of 4182 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.

这种情况下,所有redo全部被清空(包含current,active的redo),只能强制拉库,运气不错,拉库成功.

Sun Oct 06 10:09:01 2024
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 25668466513
Resetting resetlogs activation ID 4222555315 (0xfbaf14b3)
Sun Oct 06 10:09:10 2024
Setting recovery target incarnation to 3
Sun Oct 06 10:09:10 2024
Assigning activation ID 79943739 (0x4c3d83b)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /u01/app/oracle/oradata/xifenfei/redo01.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sun Oct 06 10:09:11 2024
SMON: enabling cache recovery
Undo initialization finished serial:0 start:70198684 end:70198794 diff:110 (1 seconds)
Dictionary check beginning
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is AL32UTF8
No Resource Manager plan active
Sun Oct 06 10:09:12 2024
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Sun Oct 06 10:09:13 2024
QMNC started with pid=23, OS id=4328 
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Sun Oct 06 10:09:16 2024
db_recovery_file_dest_size of 4182 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Sun Oct 06 10:09:16 2024
Starting background process CJQ0
Sun Oct 06 10:09:16 2024
CJQ0 started with pid=25, OS id=4413 
Completed: alter database open resetlogs

A_H_README_TO_RECOVER勒索恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:A_H_README_TO_RECOVER勒索恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户mysql数据库被黑(业务数据库被删除),创建了一个A_H_README_TO_RECOVER库

[root@www.xifenfei.com ~]# mysql -uroot -pxxxxx
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4539028
Server version: 5.6.50-log Source distribution

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+-----------------------+
| Database              |
+-----------------------+
| information_schema    |
| A_H_README_TO_RECOVER |
| mysql                 |
| performance_schema    |
+-----------------------+
8 rows in set (0.00 sec)

mysql> use A_H_README_TO_RECOVER;
Database changed
mysql> show tables;
+---------------------------------+
| Tables_in_A_H_README_TO_RECOVER |
+---------------------------------+
| README                          |
+---------------------------------+
1 row in set (0.00 sec)

mysql> desc README;
+------------+----------+------+-----+---------+-------+
| Field      | Type     | Null | Key | Default | Extra |
+------------+----------+------+-----+---------+-------+
| zh_content | longtext | YES  |     | NULL    |       |
| en_content | longtext | YES  |     | NULL    |       |
| email      | longtext | YES  |     | NULL    |       |
+------------+----------+------+-----+---------+-------+
3 rows in set (0.00 sec)

mysql>  select *from README ;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------+-------------------------+
| zh_content                                                                                                                                                                                                                                | en_content                                        | email                   |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------+-------------------------+
| 请与我们联系进行数据恢复,或者你对我们的项目感兴趣,也可以与我们取得联系。未与我们联系的,数据和组织信息将会公布在国内各大平台中。联系邮件:honey_xiaowu@keemail.me                                                                       | honey_xiaowu@keemail.me or honey_xiaowu@proton.me | honey_xiaowu@keemail.me |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------+-------------------------+
1 row in set (0.00 sec)

mysql> exit
Bye

对于这类情况类似以前处理的case:
read_me_recover_tn勒索恢复
RECOVER_YOUR_DATA勒索恢复
A____Z____RECOVER____DATA勒索恢复
处理方法一般也就是先考虑os层面恢复,如果os层面无法恢复,就从block层面进行恢复,这个客户通过最终分析,恢复出来客户需要的表数据
101244
对于类似这种A_H_README_TO_RECOVER勒索恢复,建议先对系统进行镜像或者快照,然后按照先os层面恢复,在block级别恢复的方法处理,如果无法自行解决,可以联系我们进行技术支持,最大限度抢救和数据,减少损失
电话/微信:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com
另外建议加强系统和mysql安全加固,数据库尽量不要暴露在公网上

通过alert日志分析客户自行对一个数据库恢复的来龙去脉和点评

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:通过alert日志分析客户自行对一个数据库恢复的来龙去脉和点评

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

12.1.0.2数据库由于异常断电,导致无法正常启动,通过alert日志对客户的整个操作过程进行分析(不含我的操作部分)
12.1.0.2


通过alert日志分析最初故障原因是由于控制文件有坏块导致

Tue Sep 24 11:49:48 2024
alter database open
Tue Sep 24 11:49:48 2024
Ping without log force is disabled
.
Tue Sep 24 11:49:48 2024
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_4715.trc:
ORA-01113: file 10 needs media recovery
ORA-01110: data file 10: '/u01/app/oracle/oradata/xifenfei.dbf'
ORA-1113 signalled during: alter database open...
alter database recover datafile '/u01/app/oracle/oradata/xifenfei.dbf'

offline 无法正常recover的数据文件

Tue Sep 24 13:13:30 2024
Media Recovery Complete (orcl)
Completed: ALTER DATABASE RECOVER  datafile 15  
ALTER DATABASE DATAFILE '/u01/app/oracle/oradata/xifenfei.dbf' END BACKUP
ORA-1235 signalled during: ALTER DATABASE DATAFILE '/u01/app/oracle/oradata/xifenfei.dbf' END BACKUP...
ALTER DATABASE DATAFILE '/u01/app/oracle/oradata/xifenfei.dbf' offline
Completed: ALTER DATABASE DATAFILE '/u01/app/oracle/oradata/xifenfei.dbf' offline
Tue Sep 24 13:25:16 2024
 ALTER DATABASE DATAFILE '/u01/app/oracle/oradata/xff.dbf' offline
Completed:  ALTER DATABASE DATAFILE '/u01/app/oracle/oradata/xff.dbf' offline

然后尝试打开数据库,遭遇ORA-600 4193错误,没有open成功

Tue Sep 24 13:27:06 2024
Media Recovery Complete (orcl)
Completed: ALTER DATABASE RECOVER  datafile 13   
alter database open
Tue Sep 24 13:27:16 2024
Ping without log force is disabled
.
Tue Sep 24 13:27:16 2024
Beginning crash recovery of 1 threads
 parallel recovery started with 7 processes
Tue Sep 24 13:27:16 2024
Started redo scan
Tue Sep 24 13:27:16 2024
Completed redo scan
 read 67 KB redo, 0 data blocks need recovery
Tue Sep 24 13:27:16 2024
Started redo application at
 Thread 1: logseq 7422, block 2, scn 119284797
Tue Sep 24 13:27:16 2024
Recovery of Online Redo Log: Thread 1 Group 3 Seq 7422 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/orcl/redo03.log
Tue Sep 24 13:27:16 2024
Completed redo application of 0.00MB
Tue Sep 24 13:27:16 2024
Completed crash recovery at
 Thread 1: logseq 7422, block 136, scn 119284798
 0 data blocks read, 0 data blocks written, 67 redo k-bytes read
Initializing SCN for created control file
Database SCN compatibility initialized to 3
Starting background process TMON
Tue Sep 24 13:27:16 2024
TMON started with pid=32, OS id=10617 
Tue Sep 24 13:27:16 2024
Thread 1 advanced to log sequence 7423 (thread open)
Thread 1 opened at log sequence 7423
  Current log# 1 seq# 7423 mem# 0: /u01/app/oracle/oradata/orcl/redo01.log
Successful open of redo thread 1
Tue Sep 24 13:27:16 2024
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Sep 24 13:27:16 2024
SMON: enabling cache recovery
Tue Sep 24 13:27:20 2024
[10553] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:6974064 end:6975474 diff:1410 ms (1.4 seconds)
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
File #10 is offline, but is part of an online tablespace.
data file 10: '/u01/app/oracle/oradata/tbs_data.dbf'
File #14 is offline, but is part of an online tablespace.
data file 14: '/u01/app/oracle/oradata/corsmf03.dbf'
Dictionary check complete
Verifying minimum file header compatibility (11g) for tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
Tue Sep 24 13:27:21 2024
SMON: enabling tx recovery
Tue Sep 24 13:27:21 2024
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Updating character set in controlfile to AL32UTF8
Starting background process SMCO
Tue Sep 24 13:27:21 2024
SMCO started with pid=34, OS id=10632 
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_10523.trc  (incident=108129):
ORA-00600: internal error code, arguments: [4193], [21368], [21372], [], [], [], [], [], [], [], [], []
Incident details in:/u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_108129/orcl_smon_10523_i108129.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
…………
Tue Sep 24 13:27:24 2024
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_10553.trc:
ORA-00600: internal error code, arguments: [4193], [21652], [21539], [], []
Tue Sep 24 13:27:24 2024
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_10553.trc:
ORA-00600: internal error code, arguments: [4193], [21652], [21539], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 10553): terminating the instance due to error 600
Tue Sep 24 13:27:25 2024
Instance terminated by USER, pid = 10553
ORA-1092 signalled during: alter database open...

重建了ctl,加入_allow_resetlogs_corruption隐含参数,尝试使用resetlogs方式打开数据库,报ORA-600 2662错误

Tue Sep 24 14:30:22 2024
alter database open RESETLOGS
Tue Sep 24 14:32:09 2024
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 119237645 time 
Online log /u01/app/oracle/oradata/orcl/redo01.log: Thread 1 Group 1 was previously cleared
Online log /u01/app/oracle/oradata/orcl/redo02.log: Thread 1 Group 2 was previously cleared
Online log /u01/app/oracle/oradata/orcl/redo03.log: Thread 1 Group 3 was previously cleared
Tue Sep 24 14:32:09 2024
Setting recovery target incarnation to 2
Tue Sep 24 14:32:09 2024
Ping without log force is disabled
.
Initializing SCN for created control file
Database SCN compatibility initialized to 3
Tue Sep 24 14:32:09 2024
Warning - High Database SCN: Current SCN value is 119237648, threshold SCN value is 0
If you have not previously reported this warning on this database, 
please notify Oracle Support so that additional diagnosis can be performed.
Starting background process TMON
Tue Sep 24 14:32:09 2024
TMON started with pid=25, OS id=15032 
Tue Sep 24 14:32:09 2024
Assigning activation ID 1708301307 (0x65d29bfb)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /u01/app/oracle/oradata/orcl/redo01.log
Successful open of redo thread 1
Tue Sep 24 14:32:09 2024
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Sep 24 14:32:09 2024
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_14937.trc  (incident=122458):
ORA-00600: internal error code, arguments: [2662], [0], [119484861], [0], [119484868], [16777344]……
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_122458/orcl_ora_14937_i122458.trc
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_14937.trc  (incident=122459):
………………
Tue Sep 24 14:32:16 2024
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_124802/orcl_ora_14937_i124802.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [0], [119484866], [0], [119484868], [16777344]……
ORA-00600: internal error code, arguments: [2662], [0], [119484865], [0], [119484868], [16777344]……
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [0], [119484861], [0], [119484868], [16777344]……

客户的自行恢复到此为止,没有成功,这里客户的恢复没有犯原则性错误(破坏文件的resetlogs 信息),同时也没有解决两个ORA-600错误
1. 在offline部分文件的情况下,打开数据库(没有使用resetlogs,避免了进一步破坏offline文件的resetlogs 信息),但是数据库报ORA-600 4193错误没有打开库成功
2. 后面强制拉库之前重建了ctl文件,避免了offline数据文件在resetlogs之后导致文件头resetlogs 信息和其他文件不一致的可能(因为重建ctl,offline的文件自动onlinne)
3. 最初offline数据文件,启动库报ORA-600 4193故障没有解决,这个故障一般是undo异常导致,这个故障大概率在后面强制拉库open过程中还可能遇到
4. 强制拉库过程中遭遇ORA-600 2662问题,需要修改scn,如果这个问题不解决,数据库无法open成功

ORA-12514: TNS: 监听进程不能解析在连接描述符中给出的SERVICE_NAME

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-12514: TNS: 监听进程不能解析在连接描述符中给出的SERVICE_NAME

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

一个10g的库应用访问报ORA-12514: TNS: 监听进程不能解析在连接描述符中给出的SERVICE_NAME
ORA-12514


通过分析alert日志,确认是数据库启动报ORA-600 4194错误

Mon Sep 23 16:12:42 2024
SMON: enabling cache recovery
Mon Sep 23 16:12:43 2024
Successfully onlined Undo Tablespace 1.
Mon Sep 23 16:12:43 2024
SMON: enabling tx recovery
Mon Sep 23 16:12:43 2024
Database Characterset is ZHS16GBK
Mon Sep 23 16:12:43 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\udump\xifenfei_ora_7832.trc:
ORA-00600: 内部错误代码, 参数: [4194], [66], [50], [], [], [], [], []

DEBUG: Replaying xcb 0xae312888, pmd 0x9058f4d4 for failed op 8
Doing block recovery for file 2 block 5547
No block recovery was needed
Mon Sep 23 16:13:31 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\udump\xifenfei_ora_7832.trc:
ORA-00600: 内部错误代码, 参数: [4194], [66], [50], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [4194], [66], [50], [], [], [], [], []

Mon Sep 23 16:13:32 2024
DEBUG: Replaying xcb 0xae312888, pmd 0x9058f4d4 for failed op 8
Mon Sep 23 16:13:32 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\udump\xifenfei_ora_7832.trc:
ORA-00600: 内部错误代码, 参数: [4194], [66], [50], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [4194], [66], [50], [], [], [], [], []

Doing block recovery for file 2 block 5547
No block recovery was needed
Mon Sep 23 16:13:33 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\udump\xifenfei_ora_7832.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4194], [66], [50], [], [], [], [], []
ORA-00600: internal error code, arguments: [4194], [66], [50], [], [], [], [], []

Mon Sep 23 16:14:18 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_smon_5880.trc:
ORA-00600: internal error code, arguments: [4194], [66], [50], [], [], [], [], []

Mon Sep 23 16:14:19 2024
DEBUG: Replaying xcb 0xae312888, pmd 0x9058f4d4 for failed op 8
Mon Sep 23 16:14:19 2024
Non-fatal internal error happenned while SMON was doing shrinking of rollback segments.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
Mon Sep 23 16:14:19 2024
Doing block recovery for file 2 block 5547
No block recovery was needed
Mon Sep 23 16:15:06 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_pmon_6952.trc:
ORA-00600: internal error code, arguments: [4194], [66], [50], [], [], [], [], []

Mon Sep 23 16:15:06 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_pmon_6952.trc:
ORA-00600: internal error code, arguments: [4194], [66], [50], [], [], [], [], []

Mon Sep 23 16:15:06 2024
PMON: terminating instance due to error 472
Mon Sep 23 16:15:07 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_psp0_2104.trc:
ORA-00472: PMON  process terminated with error

Mon Sep 23 16:15:07 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_lgwr_3200.trc:
ORA-00472: PMON  process terminated with error

Mon Sep 23 16:15:07 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_dbw1_448.trc:
ORA-00472: PMON  process terminated with error

Mon Sep 23 16:15:07 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_dbw0_7436.trc:
ORA-00472: PMON  process terminated with error

Mon Sep 23 16:15:07 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_mman_1704.trc:
ORA-00472: PMON  process terminated with error

Mon Sep 23 16:15:07 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_dbw2_5072.trc:
ORA-00472: PMON  process terminated with error

Mon Sep 23 16:15:07 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_ckpt_6628.trc:
ORA-00472: PMON  process terminated with error

Mon Sep 23 16:15:07 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_reco_7924.trc:
ORA-00472: PMON  process terminated with error

Mon Sep 23 16:15:07 2024
Errors in file d:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_smon_5880.trc:
ORA-00472: PMON  process terminated with error

Instance terminated by PMON, pid = 6952

这个比较简单一般就是undo异常,对undo设置为人工管理,然后重建undo完成本次恢复任务

ORA-01092 ORA-00604 ORA-01558故障处理

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-01092 ORA-00604 ORA-01558故障处理

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

客户一个11.2.0.1的库,在重启之前报ORA-00604和ORA-01558: out of transaction ID’s in rollback segment SYSTEM错误

Sat Mar 16 05:51:44 2024
Errors in file /opt/oracle/app/diag/rdbms/xff/xff/trace/xff_smon_47709.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTEM
Sat Mar 16 05:51:55 2024
Errors in file /opt/oracle/app/diag/rdbms/xff/xff/trace/xff_smon_47709.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTEM

数据库关闭之后无法open,报ORA-01092 ORA-00604 ORA-01558错误

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTEM
Process ID: 44308
Session ID: 769 Serial number: 3

对应的alert日志信息

Fri Aug 30 20:05:40 2024
alter database open
Beginning crash recovery of 1 threads
Fri Aug 30 20:06:08 2024
 parallel recovery started with 32 processes
Started redo scan
Completed redo scan
 read 0 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 7583120, block 130906, scn 832944617073
Recovery of Online Redo Log: Thread 1 Group 6 Seq 7583120 Reading mem 0
  Mem# 0: /OracleData/db_mc/xff/redo06.log
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 7583120, block 130906, scn 832944637074
 0 data blocks read, 0 data blocks written, 0 redo k-bytes read
Fri Aug 30 20:06:09 2024
Thread 1 advanced to log sequence 7583121 (thread open)
Thread 1 opened at log sequence 7583121
  Current log# 4 seq# 7583121 mem# 0: /OracleData/db_mc/xff/redo04.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Aug 30 20:06:09 2024
SMON: enabling cache recovery
Errors in file /opt/oracle/app/diag/rdbms/xff/xff/trace/xff_ora_6066.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTEM
Errors in file /opt/oracle/app/diag/rdbms/xff/xff/trace/xff_ora_6066.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTEM
Error 604 happened during db open, shutting down database
USER (ospid: 6066): terminating the instance due to error 604
Instance terminated by USER, pid = 6066
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (6066) as a result of ORA-1092

通过对异常block进行dump

  index  state cflags  wrap#    uel         scn            dba            parent-xid    nub     stmt_num
  ------------------------------------------------------------------------------------------------
   0x00    9    0x00  0xfffffffe  0x0059  0x00c1.ee6ab58f  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x01    9    0x00  0xfffffffe  0x0008  0x00c1.ee6ab5a7  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x02    9    0x00  0xfffffffe  0x005b  0x00c1.ee6ab59f  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x03    9    0x00  0xfffffffe  0x0007  0x00c1.ee6ab5b5  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x04    9    0x00  0xfffffffe  0x005f  0x00c1.ee6ab593  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x05    9    0x00  0xfffffffe  0x000a  0x00c1.ee6ab5ad  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x06    9    0x00  0xfffffffe  0x0019  0x00c1.ee6ab5dd  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x07    9    0x00  0xfffffffe  0x0013  0x00c1.ee6ab5b7  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x08    9    0x00  0xfffffffe  0x0035  0x00c1.ee6ab5a9  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x09    9    0x00  0xfffffffe  0x0002  0x00c1.ee6ab59d  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x0a    9    0x00  0xfffffffe  0x003c  0x00c1.ee6ab5af  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x0b    9    0x00  0xfffffffe  0x0011  0x00c1.ee6ab5d7  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x0c    9    0x00  0xfffffffe  0x0026  0x00c1.ee6ab605  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x0d    9    0x00  0xfffffffe  0x000f  0x00c1.ee6ab5c5  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x0e    9    0x00  0xfffffffe  0x0014  0x00c1.ee6ab5d3  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x0f    9    0x00  0xfffffffe  0x0012  0x00c1.ee6ab5c7  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x10    9    0x00  0xfffffffe  0x0029  0x00c1.ee6ab5c1  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x11    9    0x00  0xfffffffe  0x0018  0x00c1.ee6ab5d9  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x12    9    0x00  0xfffffffe  0x001e  0x00c1.ee6ab5c9  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x13    9    0x00  0xfffffffe  0x001b  0x00c1.ee6ab5b9  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x14    9    0x00  0xfffffffe  0x000b  0x00c1.ee6ab5d5  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x15    9    0x00  0xfffffffe  0x0003  0x00c1.ee6ab5b3  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x16    9    0x00  0xfffffffe  0x0023  0x00c1.ee6ab5f1  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x17    9    0x00  0xfffffffe  0x0021  0x00c1.ee6ab5bd  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x18    9    0x00  0xfffffffe  0x0006  0x00c1.ee6ab5db  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x19    9    0x00  0xfffffffe  0x0049  0x00c1.ee6ab5df  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x1a    9    0x00  0xfffffffe  0x0043  0x00c1.ee6ab617  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x1b    9    0x00  0xfffffffe  0x0017  0x00c1.ee6ab5bb  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x1c    9    0x00  0xfffffffe  0x002e  0x00c1.ee6ab5e9  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x1d    9    0x00  0xfffffffe  0x001c  0x00c1.ee6ab5e7  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x1e    9    0x00  0xfffffffe  0x002a  0x00c1.ee6ab5cb  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x1f    9    0x00  0xffffff28  0x003e  0x00c1.ee6ab62d  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x20    9    0x00  0xfffffffe  0x0022  0x00c1.ee6ab5cf  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x21    9    0x00  0xfffffffe  0x0010  0x00c1.ee6ab5bf  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x22    9    0x00  0xfffffffe  0x000e  0x00c1.ee6ab5d1  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x23    9    0x00  0xfffffffe  0x0025  0x00c1.ee6ab5f3  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x24    9    0x00  0xfffffffe  0x0032  0x00c1.ee6ab5f7  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x25    9    0x00  0xfffffffd  0x0024  0x00c1.ee6ab5f5  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x26    9    0x00  0xfffffffe  0x003d  0x00c1.ee6ab607  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x27    9    0x00  0xfffffffe  0x0041  0x00c1.ee6ab63b  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x28    9    0x00  0xfffffffe  0x003f  0x00c1.ee6ab613  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x29    9    0x00  0xfffffffb  0x000d  0x00c1.ee6ab5c3  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x2a    9    0x00  0xfffffffe  0x0020  0x00c1.ee6ab5cd  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x2b    9    0x00  0xfffffffe  0x0037  0x00c1.ee6ab60d  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x2c    9    0x00  0xfffffffe  0x002d  0x00c1.ee6ab5ed  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x2d    9    0x00  0xfffffffe  0x0016  0x00c1.ee6ab5ef  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x2e    9    0x00  0xfffffffe  0x002c  0x00c1.ee6ab5eb  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x2f    9    0x00  0xfffffffe  0x0034  0x00c1.ee6ab5ff  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x30    9    0x00  0xfffffffe  0x001f  0x00c1.ee6ab62b  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x31    9    0x00  0xfffffffe  0x000c  0x00c1.ee6ab603  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x32    9    0x00  0xfffffffe  0x003b  0x00c1.ee6ab5f9  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x33    9    0x00  0xfffffffa  0x0038  0x00c1.ee6ab61b  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x34    9    0x00  0xfffffffe  0x0031  0x00c1.ee6ab601  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x35    9    0x00  0xfffffffd  0x0005  0x00c1.ee6ab5ab  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x36    9    0x00  0xfffffffe  0x0028  0x00c1.ee6ab611  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x37    9    0x00  0xfffffffe  0x0036  0x00c1.ee6ab60f  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x38    9    0x00  0xfffffffe  0x0048  0x00c1.ee6ab61d  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x39    9    0x00  0xfffffffe  0x0052  0x00c1.ee6ab64d  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x3a    9    0x00  0xfffffffe  0x002f  0x00c1.ee6ab5fd  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x3b    9    0x00  0xfffffffe  0x003a  0x00c1.ee6ab5fb  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x3c    9    0x00  0xfffffffd  0x0015  0x00c1.ee6ab5b1  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x3d    9    0x00  0xfffffffe  0x0061  0x00c1.ee6ab609  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x3e    9    0x00  0xfffffffe  0x004d  0x00c1.ee6ab62f  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x3f    9    0x00  0xfffffffe  0x001a  0x00c1.ee6ab615  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x40    9    0x00  0xfffffffd  0x001d  0x00c1.ee6ab5e5  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x41    9    0x00  0xfffffffe  0x0058  0x00c1.ee6ab63d  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x42    9    0x00  0xfffffffe  0x0027  0x00c1.ee6ab639  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x43    9    0x00  0xfffffffe  0x0033  0x00c1.ee6ab619  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x44    9    0x00  0xfffffffe  0x0050  0x00c1.ee6ab635  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x45    9    0x00  0xfffffffe  0x0044  0x00c1.ee6ab633  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x46    9    0x00  0xfffffffe  0x0057  0x00c1.ee6ab625  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x47    9    0x00  0xfffffffe  0x0030  0x00c1.ee6ab629  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x48    9    0x00  0xfffffffe  0x004c  0x00c1.ee6ab61f  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x49    9    0x00  0xfffffffd  0x004e  0x00c1.ee6ab5e1  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x4a    9    0x00  0xfffffffe  0x0054  0x00c1.ee6ab641  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x4b    9    0x00  0xfffffffe  0x0046  0x00c1.ee6ab623  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x4c    9    0x00  0xfffffffe  0x004b  0x00c1.ee6ab621  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x4d    9    0x00  0xfffffffe  0x0045  0x00c1.ee6ab631  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x4e    9    0x00  0xfffffffd  0x0040  0x00c1.ee6ab5e3  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x4f    9    0x00  0xfffffffd  0x005a  0x00c1.ee6ab599  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x50    9    0x00  0xfffffffe  0x0042  0x00c1.ee6ab637  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x51    9    0x00  0xfffffffe  0x0053  0x00c1.ee6ab645  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x52    9    0x00  0xfffffffe  0x0060  0x00c1.ee6ab64f  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x53    9    0x00  0xfffffffe  0x0056  0x00c1.ee6ab647  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x54    9    0x00  0xfffffffe  0x0051  0x00c1.ee6ab643  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x55    9    0x00  0xfffffffe  0x0039  0x00c1.ee6ab64b  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x56    9    0x00  0xfffffffe  0x0055  0x00c1.ee6ab649  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x57    9    0x00  0xfffffffe  0x0047  0x00c1.ee6ab627  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x58    9    0x00  0xfffffffe  0x004a  0x00c1.ee6ab63f  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x59    9    0x00  0xfffffffd  0x0004  0x00c1.ee6ab591  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x5a    9    0x00  0xfffffffd  0x0009  0x00c1.ee6ab59b  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x5b    9    0x00  0xfffffffd  0x005e  0x00c1.ee6ab5a1  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x5c    9    0x00  0xfffffffd  0x0001  0x00c1.ee6ab5a5  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x5d    9    0x00  0xfffffffd  0x004f  0x00c1.ee6ab597  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x5e    9    0x00  0xfffffffd  0x005c  0x00c1.ee6ab5a3  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x5f    9    0x00  0xfffffffd  0x005d  0x00c1.ee6ab595  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x60    9    0x00  0xfffffffe  0xffff  0x00c1.ee6ab651  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x61    9    0x00  0xfffffdb4  0x002b  0x00c1.ee6ab60b  0x03000438  0x0000.000.00000000  0x00000001   0x00000000

确实是Wrap#达到了极限值,通过bbed也可以进一步确认

BBED> p ktuxe[1]
struct ktuxe[1], 40 bytes                   @4292
   ub4 ktuxexid                             @4292     0xfffffffe
   ub4 ktuxebrb                             @4296     0x03000436
   struct ktuxescn, 8 bytes                 @4300
      ub4 kscnbas                           @4300     0xee6ab5a7
      ub2 kscnwrp                           @4304     0x00c1
   sb4 ktuxesta                             @4308     524297 (KTU_SCO, KTU_DEAD)
   ub1 ktuxecfl                             @4309     0x00
   sb2 ktuxeuel                             @4310     8

BBED> p ktuxe[2]
struct ktuxe[2], 40 bytes                   @4332
   ub4 ktuxexid                             @4332     0xfffffffe
   ub4 ktuxebrb                             @4336     0x03000436
   struct ktuxescn, 8 bytes                 @4340
      ub4 kscnbas                           @4340     0xee6ab59f
      ub2 kscnwrp                           @4344     0x00c1
   sb4 ktuxesta                             @4348     5963785 (KTU_SCO, KTU_DEAD)
   ub1 ktuxecfl                             @4349     0x00
   sb2 ktuxeuel                             @4350     91

处理方法,通过bbed对异常的block进行编辑,修改Wrap#的值,重新dumpblock进行确认

  index  state cflags  wrap#    uel         scn            dba            parent-xid    nub     stmt_num
  ------------------------------------------------------------------------------------------------
   0x00    9    0x00  0x0015  0x0003  0x0000.11e1a313  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x01    9    0x00  0x0015  0x0007  0x0000.11e1a317  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x02    9    0x00  0x0015  0x000c  0x0000.11e32087  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x03    9    0x00  0x0015  0x005a  0x0000.11e1a314  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x04    9    0x00  0x0015  0x0060  0x0000.11e3206e  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x05    9    0x00  0x0015  0x005d  0x0000.11e1a311  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x06    9    0x00  0x0015  0x0015  0x0000.11e3208b  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x07    9    0x00  0x0015  0x0004  0x0000.11e1a318  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x08    9    0x00  0x0015  0x0014  0x0000.11e32084  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x09    9    0x00  0x0015  0x0001  0x0000.11e1a316  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x0a    9    0x00  0x0015  0x0013  0x0000.11e3207d  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x0b    9    0x00  0x0015  0x0011  0x0000.11e3207f  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x0c    9    0x00  0x0015  0x0059  0x0000.11e32089  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x0d    9    0x00  0x0015  0x000a  0x0000.11e3207c  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x0e    9    0x00  0x0015  0x0017  0x0000.11e32082  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x0f    9    0x00  0x0015  0x002a  0x0000.120964dc  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x10    9    0x00  0x0015  0x001a  0x0000.11e3208d  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x11    9    0x00  0x0015  0x005f  0x0000.11e32080  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x12    9    0x00  0x0015  0x0028  0x0000.120964d9  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x13    9    0x00  0x0015  0x000b  0x0000.11e3207e  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x14    9    0x00  0x0015  0x0018  0x0000.11e32085  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x15    9    0x00  0x0015  0x0010  0x0000.11e3208c  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x16    9    0x00  0x0015  0x0032  0x0000.120964e2  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x17    9    0x00  0x0015  0x0008  0x0000.11e32083  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x18    9    0x00  0x0015  0x0002  0x0000.11e32086  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x19    9    0x00  0x0015  0x0020  0x0000.120964d3  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x1a    9    0x00  0x0015  0x0025  0x0000.120964c5  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x1b    9    0x00  0x0015  0x0021  0x0000.120964d6  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x1c    9    0x00  0x0015  0x001b  0x0000.120964d5  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x1d    9    0x00  0x0015  0x0019  0x0000.120964d2  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x1e    9    0x00  0x0015  0x0030  0x0000.1210ab57  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x1f    9    0x00  0x0015  0x000f  0x0000.120964db  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x20    9    0x00  0x0015  0x001c  0x0000.120964d4  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x21    9    0x00  0x0015  0x0012  0x0000.120964d8  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x22    9    0x00  0x0015  0x0029  0x0000.120964de  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x23    9    0x00  0x0015  0x0031  0x0000.120964e4  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x24    9    0x00  0x0015  0x001d  0x0000.120964d1  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x25    9    0x00  0x0014  0x0024  0x0000.120964cf  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x26    9    0x00  0x0015  0x003b  0x0000.1210ab5b  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x27    9    0x00  0x0014  0x0058  0x0000.0ec9fb55  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x28    9    0x00  0x0015  0x001f  0x0000.120964da  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x29    9    0x00  0x0015  0x002b  0x0000.120964df  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x2a    9    0x00  0x0015  0x0022  0x0000.120964dd  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x2b    9    0x00  0x0015  0x002d  0x0000.120964e0  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x2c    9    0x00  0x0015  0x003d  0x0000.1210ab60  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x2d    9    0x00  0x0015  0x0016  0x0000.120964e1  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x2e    9    0x00  0x0014  0x003c  0x0000.0ec9fb51  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x2f    9    0x00  0x0015  0x0045  0x0000.1210ab65  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x30    9    0x00  0x0015  0x0034  0x0000.1210ab58  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x31    9    0x00  0x0015  0x0036  0x0000.1210ab4b  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x32    9    0x00  0x0015  0x0023  0x0000.120964e3  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x33    9    0x00  0x0015  0x0041  0x0000.1210ab67  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x34    9    0x00  0x0015  0x003a  0x0000.1210ab59  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x35    9    0x00  0x0015  0x002c  0x0000.1210ab5e  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x36    9    0x00  0x0015  0x001e  0x0000.1210ab55  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x37    9    0x00  0x0014  0x0050  0x0000.0ec9fb53  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x38    9    0x00  0x0015  0x003f  0x0000.1210ab62  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x39    9    0x00  0x0015  0x0035  0x0000.1210ab5d  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x3a    9    0x00  0x0015  0x0026  0x0000.1210ab5a  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x3b    9    0x00  0x0015  0x0039  0x0000.1210ab5c  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x3c    9    0x00  0x0014  0x0037  0x0000.0ec9fb52  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x3d    9    0x00  0x0015  0x0038  0x0000.1210ab61  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x3e    9    0x00  0x0014  0x0040  0x0000.0ec9fb49  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x3f    9    0x00  0x0015  0x0044  0x0000.1210ab63  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x40    9    0x00  0x0014  0x004b  0x0000.0ec9fb4a  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x41    9    0x00  0x0015  0x0043  0x0000.1210ab68  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x42    9    0x00  0x0015  0xffff  0x0000.1210ab6b  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x43    9    0x00  0x0015  0x0042  0x0000.1210ab69  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x44    9    0x00  0x0015  0x002f  0x0000.1210ab64  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x45    9    0x00  0x0015  0x0033  0x0000.1210ab66  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x46    9    0x00  0x0014  0x004e  0x0000.0ec9fb4e  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x47    9    0x00  0x0014  0x004f  0x0000.0ec9fb4c  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x48    9    0x00  0x0014  0x0054  0x0000.11e1a306  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x49    9    0x00  0x0014  0x0055  0x0000.0ec9fd5b  0x00400223  0x0000.000.00000000  0x00000002   0x00000000
   0x4a    9    0x00  0x0014  0x003e  0x0000.0ec9fb48  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x4b    9    0x00  0x0014  0x0047  0x0000.0ec9fb4b  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x4c    9    0x00  0x0014  0x000d  0x0000.11e3207b  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x4d    9    0x00  0x0014  0x0061  0x0000.11e1a30f  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x4e    9    0x00  0x0014  0x002e  0x0000.0ec9fb4f  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x4f    9    0x00  0x0014  0x0046  0x0000.0ec9fb4d  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x50    9    0x00  0x0014  0x0027  0x0000.0ec9fb54  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x51    9    0x00  0x0014  0x0053  0x0000.11e1a30a  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x52    9    0x00  0x0014  0x005b  0x0000.11e1a30c  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x53    9    0x00  0x0014  0x0052  0x0000.11e1a30b  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x54    9    0x00  0x0014  0x0057  0x0000.11e1a307  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x55    9    0x00  0x0014  0x0048  0x0000.11e1a304  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x56    9    0x00  0x0014  0x0051  0x0000.11e1a309  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x57    9    0x00  0x0014  0x0056  0x0000.11e1a308  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x58    9    0x00  0x0014  0x0049  0x0000.0ec9fb56  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x59    9    0x00  0x0014  0x0006  0x0000.11e3208a  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x5a    9    0x00  0x0014  0x0009  0x0000.11e1a315  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x5b    9    0x00  0x0014  0x005e  0x0000.11e1a30d  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x5c    9    0x00  0x0014  0x004c  0x0000.11e3207a  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x5d    9    0x00  0x0014  0x0000  0x0000.11e1a312  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x5e    9    0x00  0x0014  0x004d  0x0000.11e1a30e  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x5f    9    0x00  0x0014  0x000e  0x0000.11e32081  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x60    9    0x00  0x0014  0x005c  0x0000.11e32078  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x61    9    0x00  0x0014  0x0005  0x0000.11e1a310  0x00400223  0x0000.000.00000000  0x00000001   0x00000000

直接启动数据库成功

[oracle@www.xifenfei.com ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Sat Sep 21 13:45:23 2024

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area 8551575552 bytes
Fixed Size		    2215064 bytes
Variable Size		 2483028840 bytes
Database Buffers	 6039797760 bytes
Redo Buffers		   26533888 bytes
Database mounted.
SQL> recover database;
Media recovery complete.
SQL> alter database open;       

Database altered.

由于客户遇到故障之后,第一时间保护了现场,没有进行二次破坏,使用bbed进行修改block,实现完美恢复.
以前有过一次在11.2.0.4的环境中恢复的同样案例:ORA-01558: out of transaction ID’s in rollback segment SYSTEM
通过查询mos,确认类似Bug 19700135 – ORA-600 [4187] when the undo segment wrap# is close to the max value of 0xffffffff (Doc ID 19700135.8)
bug-19700135

Description
ORA-600 [4187] can occur for undo segments where wrap# is close to the max value of 0xffffffff (KSQNMAXVAL).
This normally affects databases with high transaction rate that have existed for a relatively long time.
 
To identify undo segments causing the above error and others that may potentially cause it 
in the future, run the next query:
 
 select b.segment_name, b.tablespace_name 
         ,a.ktuxeusn "Undo Segment Number"
         ,a.ktuxeslt "Slot"
         ,a.ktuxesqn "Wrap#"
   from  x$ktuxe a, dba_rollback_segs b
   where a.ktuxesqn > -429496730 and a.ktuxesqn < 0
       and a.ktuxeusn = b.segment_id;
 
Then drop the undo segments or the undo tablespace from the output above.
 
With this fix in place an error ORA-1558 is eventually produced for the affected undo segment
which still requires dropping the undo segment:
  ORA-1558 "out of transaction ID's in rollback segment %s"
   Cause: All the available transaction id's have been used
   Action: Shutdown the instance and restart using other rollback segment(s),
                then drop the rollback segment that has no more transaction id's.
 
Workaround
 Once the ORA-600 [4187] is produced, drop the affected undo segment or undo tablespace.

由于本次故障是在system的rollback回滚段上,无法删除该回滚段.所以处理起来比较棘手.

ORA-65088: database open should be retried

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-65088: database open should be retried

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

在12.2以及后续的cdb版本中,如果重建ctl并且resetlogs库,很可能会遇到ORA-65088: database open should be retried错误

SQL> startup nomount force pfile='/<path>/<filename>.ora';
ORACLE instance started.

Total System Global Area 1593835520 bytes
Fixed Size                  8793256 bytes
Variable Size             402654040 bytes
Database Buffers         1174405120 bytes
Redo Buffers                7983104 bytes
SQL> !vi ctl.sql

SQL> @ctl.sql

Control file created.

SQL> select count(*) ,fhsta from x$kcvfh group by fhsta;

  COUNT(*)      FHSTA
---------- ----------
        11      32768
         4      40960

SQL> select count(*) ,FHSCN from x$kcvfh group by FHSCN;

  COUNT(*) FHSCN
---------- --------------------
         3 1820866
         4 2281969
         4 2281978
         4 2281982

SQL> select file#,error from v$datafile_header where length(error)>=1;

no rows selected

SQL> select count(*) ,fhrba_seq from x$kcvfh group by fhrba_seq;

  COUNT(*)  FHRBA_SEQ
---------- ----------
         3         20
        12         32

SQL> recover database using backup controlfile until cancel;
ORA-00279: change 2281978 generated at 09/19/2018 00:52:00 needed for thread 1
ORA-00289: suggestion : /<path>/1_32_981800889.dbf
ORA-00280: change 2281978 for thread 1 is in sequence #32


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/<path>/1_32_981800889.dbf
ORA-00279: change 2282008 generated at 09/19/2018 00:52:13 needed for thread 1
ORA-00289: suggestion : /<path>/1_33_981800889.dbf
ORA-00280: change 2282008 for thread 1 is in sequence #33
ORA-00278: log file '/<path>/1_32_981800889.dbf' no longer needed for this recovery


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.    << Expected message "Media recovery complete." !!
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MOUNTED
         3 _###_UNKNOWN_PDB_#_3           MOUNTED
         4 _###_UNKNOWN_PDB_#_4           MOUNTED
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-65088: database open should be retried
Process ID: 32688
Session ID: 10 Serial number: 38416

alert日志类似错误

 Dictionary check beginning
 Pluggable Database <pdb_name_1> (#3) found in data dictionary,
 but not in the control file. Adding it to control file.
 Pluggable Database <pdb_name_2> (#4) found in data dictionary,
 but not in the control file. Adding it to control file.
 Tablespace '<tablespace_name>' #3 found in data dictionary,
 but not in the controlfile. Adding to controlfile.
 --
 File 8 not verified due to error ORA-01122
 File 9 not verified due to error ORA-01122
 File 11 not verified due to error ORA-01122
 File 16 not verified due to error ORA-01122
 File 17 not verified due to error ORA-01122
 File 18 not verified due to error ORA-01122
 File 19 not verified due to error ORA-01122
 File 20 not verified due to error ORA-01122
  --
 ORA-65088: database open should be retried
 2018-09-19T01:00:54.083814+05:30
 Errors in file /<path>/trace/<oracle_sid>_ora_12412.trc:
 ORA-65088: database open should be retried
 Error 65088 happened during db open, shutting down database
 Errors in file /<path>/trace/<oracle_sid>_ora_12412.trc  (incident=12289) (PDBNAME=CDB$ROOT):
 ORA-00603: ORACLE server session terminated by fatal error
 ORA-01092: ORACLE instance terminated. Disconnection forced
 ORA-65088: database open should be retried

出现这类故障的原因是由于:
we see that the created controlfile is not aware of PDB and open resetlogs process trying to add information in newly created file . Hence, recovery process ,in newly created controlfile didn’t applied the archives to datafiles part of PDB which says later it will ask for recovery once controlfile is aware of PDB files During the resetlogs process, its pushing the required information to controlfile and shutting the database with suggestion to re-try opening the DB.

$ sqlplus "/as sysdba"

SQL*Plus: Release 12.2.0.1.0 Production on Wed Sep 19 01:34:01 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount force pfile='/<path>/<filename>.ora';
ORACLE instance started.

Total System Global Area 1593835520 bytes
Fixed Size                  8793256 bytes
Variable Size             402654040 bytes
Database Buffers         1174405120 bytes
Redo Buffers                7983104 bytes
SQL> alter database mount;

Database altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MOUNTED
         3 PDB1                           MOUNTED
         4 APDB                           MOUNTED
        
SQL> select count(*) ,FHSCN from x$kcvfh group by FHSCN;

  COUNT(*) FHSCN
---------- --------------------
         3 1820866
         4 2281969
         4 2281982
         4 2282012        
        
//* Here , we see controlfile is aware of PDB



$ sqlplus "/as sysdba"

SQL*Plus: Release 12.2.0.1.0 Production on Wed Sep 19 01:02:13 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL>  startup nomount force pfile='/<path>/<filename>.ora';
ORACLE instance started.

Total System Global Area 1593835520 bytes
Fixed Size                  8793256 bytes
Variable Size             402654040 bytes
Database Buffers         1174405120 bytes
Redo Buffers                7983104 bytes
SQL> alter database mount;

Database altered.

SQL> recover database;
ORA-00279: change 2281969 generated at 09/19/2018 00:51:35 needed for thread 1
ORA-00289: suggestion : /<path>/1_32_981800889.dbf
ORA-00280: change 2281969 for thread 1 is in sequence #32


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/<path>/1_32_981800889.dbf
Log applied.
Media recovery complete.
SQL> alter database open;

Database altered.

SQL>

官方的进一步解释:
We clearly see that the recovery steps applies the same archivelog file twice. When a controlfile is recreated, the recovery initiated will apply archivelog files to only the CDB datafiles, not to the PDB. Once the database open returns the ORA-65088 error, the next database re-start will apply the archivelog files to the PDB for the sake of database consistency.This should explain why Oracle is looking to apply the same archivelog sequence a second time. The following bugs report similar issues. They have both been closed as ‘not a bug’ as this is expected behavior:
BUG 24951417 – ERROR OPENING DATABASE WITH RESETLOGS AFTER CREATE CONTROLFILE
BUG 25172530 – MULTITENANT RESTORE FAILED WITH ORA-65088: DATABASE OPEN
参考:ORA-65088 while opening DB with resetlogs for multi-tenant DB in 12.2 (Doc ID 2449591.1)