Database SOS – Professional Database Recovery Support@+8617813235971

最近有客户在虚拟化平台运行oracle,由于机房掉电,导致oracle数据库无法正常启动,通过第三方恢复,oracle被强制拉起,但是无法进行ddl操作,比如创建表报ORA-08102: 未找到索引关键字, 对象号 39, 文件 1, 块 122448 (2) 错误

通过obj#确认具体对象为i_obj#(也就是obj$对象上的一个index)

由于这类对象属于数据库的底层核心对象,无法直接rebulid他们,根据以往经验,可以通过bbed对其进行修复,或者参考类似文章进行重建:
分享I_OBJ4 ORA-8102故障恢复案例
使用bbed 修复I_OBJ4 index 报ORA-8102错误
通过bbed修改obj$中dataobj$重现I_OBJ4索引报ORA-08102错误
bootstrap$核心index(I_OBJ1,I_USER1,I_FILE#_BLOCK#,I_IND1,I_TS#,I_CDEF1等)异常恢复—ORA-00701错误解决
这个问题解决之后,该客户还有另外一个问题需要解决（不然数据库运行一段时间之后就会crash）

Wed Dec 18 09:13:03 2024
Errors in file E:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_smon_2536.trc  (incident=1105672):
ORA-00600: 内部错误代码, 参数: [13013], [5001], [268], [8459081], [1], [8459081], [3], [], [], [], [], []
Incident details in: E:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_1105672\orcl_smon_2536_i1105672.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Non-fatal internal error happenned while SMON was doing logging scn->time mapping.
SMON encountered 1 out of maximum 100 non-fatal internal errors.

这个问题本质就是SMON_SCN_TIME表的异常导致(一般13013是由于表和index不一致导致),对于这类问题处理，参考:
关于SMON_SCN_TIME若干问题说明
处理完上述两个明显故障之后,然后使用expdp不落地方式把客户数据迁移到新库,完成本次恢复任务

ORA-00227: corrupt block detected in control file

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：ORA-00227: corrupt block detected in control file

由于服务器断电，导致oracle数据库无法正常启动，recover报ORA-00353 ORA-00312错

SQL> recover datafile 1;
ORA-00283: recovery session canceled due to errors
ORA-00368: checksum error in redo log block
ORA-00353: log corruption near block 8 change 237896529 time 12/03/2024 22:03:32
ORA-00312: online log 1 thread 1: 'E:\ORADATA\xifenfei\REDO01.LOG'

报错提示比较明显,是由于oracle恢复需要的redo损坏,导致无法进行正常恢复,这种情况下,只能尝试强制打开库

SQL> recover database until cancel;
ORA-00279: change 237857808 generated at 12/03/2024 07:06:31 needed for thread 1
ORA-00289: suggestion : C:\ORACLE\PRODUCT\10.2.0.3\DB_1\RDBMS\ARC20892_0929553713.001
ORA-00280: change 237857808 for thread 1 is in sequence #20892


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: 'E:\ORADATA\xifenfei\SYSTEM01.DBF'


ORA-01112: media recovery not started


SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced

对于这种ORA-01092错误,需要查看alert日志确认具体报错原因

Tue Dec 17 22:03:59 2024
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 237857808
Resetting resetlogs activation ID 2015910641 (0x78285af1)
Tue Dec 17 22:04:00 2024
Setting recovery target incarnation to 2
Tue Dec 17 22:04:00 2024
Advancing SCN to 16106127360 according to _minimum_giga_scn
Tue Dec 17 22:04:00 2024
Assigning activation ID 2274407914 (0x8790b5ea)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: E:\ORADATA\xifenfei\REDO01.LOG
Successful open of redo thread 1
Tue Dec 17 22:04:01 2024
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Dec 17 22:04:01 2024
SMON: enabling cache recovery
Tue Dec 17 22:04:01 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\udump\xifenfei_ora_816.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option

Tue Dec 17 22:04:01 2024
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Tue Dec 17 22:04:01 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_pmon_2472.trc:
ORA-00704: bootstrap process failure

Tue Dec 17 22:04:01 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_reco_2576.trc:
ORA-00704: bootstrap process failure

Tue Dec 17 22:04:01 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_smon_2584.trc:
ORA-00704: bootstrap process failure

Tue Dec 17 22:04:02 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_ckpt_2216.trc:
ORA-00704: bootstrap process failure

Tue Dec 17 22:04:02 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_lgwr_1556.trc:
ORA-00704: bootstrap process failure

Tue Dec 17 22:04:02 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_dbw0_1528.trc:
ORA-00704: bootstrap process failure

Tue Dec 17 22:04:02 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_psp0_2896.trc:
ORA-00704: bootstrap process failure

Tue Dec 17 22:04:02 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_mman_904.trc:
ORA-00704: bootstrap process failure

Tue Dec 17 22:04:02 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_dbw1_1732.trc:
ORA-00704: bootstrap process failure

Instance terminated by USER, pid = 816
ORA-1092 signalled during: alter database open resetlogs...

由于对oracle粗心对于oracle版本判断失误,导致打开数据库失败,使用正确版本打开数据库发现ctl有报错,导致打开依旧失败(这种错误一般比较少见，大部分ctl异常都是在oracle mount状态无法成功)

Tue Dec 17 22:10:48 2024
Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0
  Mem# 0: E:\ORADATA\xifenfei\REDO01.LOG
Tue Dec 17 22:10:48 2024
Completed redo application
Tue Dec 17 22:10:48 2024
Completed crash recovery at
 Thread 1: logseq 1, block 3, scn 16106147363
 0 data blocks read, 0 data blocks written, 1 redo blocks read
Tue Dec 17 22:10:48 2024
Read from controlfile member 'E:\ORADATA\xifenfei\CONTROL01.CTL' has found a corrupted block (blk# 432, seq# 132194)
Hex dump of (file 0, block 432) in trace file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_lgwr_2728.trc
Corrupt block relative dba: 0x000001b0 (file 0, block 432)
Fractured block found during control file block read
Data in bad block:
 type: 0 format: 0 rdba: 0x00000000
 last change scn: 0x0000.00000000 seq: 0x0 flg: 0x00
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x03e71501
 check value in block header: 0x0
 block checksum disabled
Hex dump of (file 0, block 432) in trace file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_lgwr_2728.trc
Corrupt block relative dba: 0x000001b0 (file 0, block 432)
Fractured block found during control file block read
Data in bad block:
 type: 0 format: 0 rdba: 0x00000000
 last change scn: 0x0000.00000000 seq: 0x0 flg: 0x00
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x03e71501
 check value in block header: 0x0
 block checksum disabled
Tue Dec 17 22:10:48 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_lgwr_2728.trc:
ORA-00202: control file: 'E:\ORADATA\xifenfei\CONTROL01.CTL'

Tue Dec 17 22:10:48 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_lgwr_2728.trc:
ORA-00227: corrupt block detected in control file: (block 432, # blocks 1)
ORA-00202: control file: 'E:\ORADATA\xifenfei\CONTROL01.CTL'

LGWR: terminating instance due to error 227
Tue Dec 17 22:10:48 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_pmon_2056.trc:
ORA-00227: corrupt block detected in control file: (block , # blocks )

Tue Dec 17 22:10:49 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_dbw0_1636.trc:
ORA-00227: corrupt block detected in control file: (block , # blocks )

Tue Dec 17 22:10:49 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_reco_2468.trc:
ORA-00227: corrupt block detected in control file: (block , # blocks )

Tue Dec 17 22:10:49 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_smon_2996.trc:
ORA-00227: corrupt block detected in control file: (block , # blocks )

Tue Dec 17 22:10:49 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_ckpt_2292.trc:
ORA-00227: corrupt block detected in control file: (block , # blocks )

Tue Dec 17 22:10:49 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_dbw1_508.trc:
ORA-00227: corrupt block detected in control file: (block , # blocks )

Tue Dec 17 22:10:49 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_mman_1728.trc:
ORA-00227: corrupt block detected in control file: (block , # blocks )

Tue Dec 17 22:10:49 2024
Errors in file c:\oracle\product\10.2.0\admin\xifenfei\bdump\xifenfei_psp0_1724.trc:
ORA-00227: corrupt block detected in control file: (block , # blocks )

Instance terminated by LGWR, pid = 2728

重建ctl,打开数据库成功,导出数据,完成本次恢复任务

手工删除19c rac

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：手工删除19c rac

在某些时候,需要删除掉手工删除19c RAC,重新安装,以下是比较简便的操作(root用户操作)

source /home/grid/.bash_profile
crsctl stop crs
systemctl disable oracle-ohasd
systemctl stop oracle-ohasd
systemctl disable oracle-tfa.service
systemctl stop oracle-tfa

rm -rf /etc/oracle
rm -rf /etc/ora*
rm -rf /u01
rm -rf /tmp/CVU*
rm -rf /tmp/.oracle
rm -rf /var/tmp/.oracle
rm -f /etc/init.d/init.ohasd
rm -f /etc/systemd/system/oracle-ohasd.service
rm -f /etc/systemd/system/oracle-tfa.service

dd if=/dev/zero of=/dev/asm_ocr1 bs=1024 count=1
dd if=/dev/zero of=/dev/asm_ocr2 bs=1024 count=1
dd if=/dev/zero of=/dev/asm_ocr3 bs=1024 count=1

解决oracle数据文件路径有回车故障

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：解决oracle数据文件路径有回车故障

最近遇到一个硬件恢复朋友的请求,oracle数据库文件恢复出来了,但是在linux上面启动的时候,有两个文件无法检测到,dbv检测正常.

通过分析是由于文件无法找到原因导致

进一步检查发现原库这两个文件结尾带有回车,但是恢复出来的文件不带回车

对于这个故障,我在测试环境进行了重现并且给予解决
1. 创建带回车键数据文件

SQL> create tablespace xifenfei datafile '/u01/app/oracle/oradata/xifenfei/xff01.dbf
  2  ' size 128m;

Tablespace created.

SQL> alter tablespace xifenfei add datafile '/u01/app/oracle/oradata/xifenfei/xff02.dbf' size 128M;

Tablespace altered.

SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/xifenfei/system01.dbf
/u01/app/oracle/oradata/xifenfei/sysaux01.dbf
/u01/app/oracle/oradata/xifenfei/undotbs01.dbf
/u01/app/oracle/oradata/xifenfei/users01.dbf
/u01/app/oracle/oradata/xifenfei/xff01.dbf
/u01/app/oracle/oradata/xifenfei/xff02.dbf

6 rows selected.

2.操作系统层面查看文件（在我的ssh工具中,可以看到带回车键文件和不带回车文件不一样,使用的是crt工具,其他工具是否显示不确定）

[oracle@xifenfei ~]$ cd /u01/app/oracle/oradata/xifenfei/
[oracle@xifenfei xifenfei]$ ls -l xff*
-rw-r----- 1 oracle oinstall 134225920 Dec 14 08:05 xff01.dbf?
-rw-r----- 1 oracle oinstall 134225920 Dec 14 08:05 xff02.dbf

3. 操作系统层面重命名数据文件

[oracle@xifenfei xifenfei]$ mv xff01.dbf* xff01.dbf
[oracle@xifenfei xifenfei]$ ls -l xff*
-rw-r----- 1 oracle oinstall 134225920 Dec 14 08:05 xff01.dbf
-rw-r----- 1 oracle oinstall 134225920 Dec 14 08:05 xff02.dbf

3. 数据库层面重启看文件情况,发现文件不能被正常发现(当然不能,文件被os层面mv了)

SQL> shutdown abort;
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area  551165952 bytes
Fixed Size                  2255112 bytes
Variable Size             369100536 bytes
Database Buffers          171966464 bytes
Redo Buffers                7843840 bytes
Database mounted.
SQL> select file#, CHECKPOINT_CHANGE# from v$datafile_header;

     FILE# CHECKPOINT_CHANGE#
---------- ------------------
         1          306775013
         2          306775013
         3          306775013
         4          306775013
         5                  0
         6          306779423

6 rows selected.

RMAN> report schema;

Report of database schema for database with db_unique_name XIFENFEI

List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    770      SYSTEM               ***     /u01/app/oracle/oradata/xifenfei/system01.dbf
2    1950     SYSAUX               ***     /u01/app/oracle/oradata/xifenfei/sysaux01.dbf
3    70       UNDOTBS1             ***     /u01/app/oracle/oradata/xifenfei/undotbs01.dbf
4    12       USERS                ***     /u01/app/oracle/oradata/xifenfei/users01.dbf
5    0        XIFENFEI             ***     /u01/app/oracle/oradata/xifenfei/xff01.dbf

6    128      XIFENFEI             ***     /u01/app/oracle/oradata/xifenfei/xff02.dbf

4. 解决控制文件和数据文件实际名称不一致问题

RMAN> catalog datafilecopy '/u01/app/oracle/oradata/xifenfei/xff01.dbf';

using target database control file instead of recovery catalog
cataloged datafile copy
datafile copy file name=/u01/app/oracle/oradata/xifenfei/xff01.dbf RECID=1 STAMP=1187684217

RMAN> switch datafile 5 to copy;

datafile 5 switched to datafile copy "/u01/app/oracle/oradata/xifenfei/xff01.dbf"

RMAN> report schema;

Report of database schema for database with db_unique_name XIFENFEI

List of Permanent Datafiles
===========================
File Size(MB) Tablespace           RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1    770      SYSTEM               ***     /u01/app/oracle/oradata/xifenfei/system01.dbf
2    1950     SYSAUX               ***     /u01/app/oracle/oradata/xifenfei/sysaux01.dbf
3    70       UNDOTBS1             ***     /u01/app/oracle/oradata/xifenfei/undotbs01.dbf
4    12       USERS                ***     /u01/app/oracle/oradata/xifenfei/users01.dbf
5    128      XIFENFEI             ***     /u01/app/oracle/oradata/xifenfei/xff01.dbf
6    128      XIFENFEI             ***     /u01/app/oracle/oradata/xifenfei/xff02.dbf

List of Temporary Files
=======================
File Size(MB) Tablespace           Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1    123      TEMP                 32767       /u01/app/oracle/oradata/xifenfei/temp01.dbf


RMAN> alter database open;

database opened

.wstop扩展名勒索数据库恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：.wstop扩展名勒索数据库恢复

操作系统文件被加密成.[[gmtaP2R5]].[[dataserver@airmail.cc]].wstop扩展名,类似

运行的oracle数据库文件,从名称上看没有被加上明显的后缀名

通过winhex打开文件分析,虽然文件名称没有改变,但是文件依旧被破坏

通过专业工具检测具体破坏情况,每个文件破坏三段,破坏24个block左右

因为损坏block较少,这种情况,可以通过我开发的Oracle数据文件勒索加密工具进行处理,然后open数据库

类似勒索病毒预防建议：
1. 教育和培训：提高用户的网络安全意识非常重要。通过定期的网络安全培训和教育，向用户传达有关勒索病毒及其传播方式的知识，让他们能够警惕潜在的威胁，并学会如何正确应对可疑的电子邮件、链接和附件。
2. 更新和维护：及时更新操作系统、应用程序和安全软件，以修补已知的漏洞，并确保系统能够及时获取最新的安全补丁。此外，定期进行系统维护和检查，确保系统的安全配置和设置。
3. 备份数据：定期备份重要的数据和文件，并将备份存储在安全的离线或云存储中。确保备份是完整的、可靠的，并且能够及时恢复，以便在发生勒索病毒感染或其他数据丢失事件时能够快速恢复数据。
4. 网络安全工具：使用可信赖的网络安全工具，包括防病毒软件、防火墙、入侵检测系统等，以提高系统的安全性和防护能力。定期对系统进行全面的安全扫描和检测，及时发现并清除潜在的威胁。
5. 访问控制：实施严格的访问控制措施，限制用户对系统和文件的访问权限，避免使用管理员权限进行日常操作，以减少恶意软件感染的风险。此外，定期审查和更新访问控制策略，确保系统安全性得到有效维护。
6. 应急响应计划：制定和实施应急响应计划，明确团队成员的责任和任务，建立应对勒索病毒和其他安全事件的应急响应流程，以最大程度地减少损失并快速恢复业务正常运营。

如果此类的数据库(oracle,mysql,sql server)等被加密,需要专业恢复技术支持，请联系我们:
电话/微信:17813235971 Q Q:107644445 E-Mail:dba@xifenfei.com

Oracle Recovery Tools工具一键解决ORA-00376 ORA-01110故障（文件offline）

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle Recovery Tools工具一键解决ORA-00376 ORA-01110故障（文件offline）

客户在win上面迁移数据文件,由于原库非归档,结果导致有两个文件scn不一致,无法打开库,结果他们选择offline文件,然后打开数据库

Wed Dec 04 14:06:04 2024
alter database open
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_6056.trc:
ORA-01113: 文件 10 需要介质恢复
ORA-01110: 数据文件 10: 'C:\PROGRAM FILES\ORACLE\XFF1.DBF'
ORA-1113 signalled during: alter database open...
Wed Dec 04 14:08:18 2024
alter database datafile 'c:\program files\oracle\XFF1.dbf' offline drop
Completed: alter database datafile 'c:\program files\oracle\XFF1.dbf' offline drop
Wed Dec 04 14:08:31 2024
alter database open
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_6056.trc:
ORA-01113: 文件 26 需要介质恢复
ORA-01110: 数据文件 26: 'C:\PROGRAM FILES\ORACLE\XFF2.DBF'
ORA-1113 signalled during: alter database open...
Wed Dec 04 14:08:31 2024
Checker run found 1 new persistent data failures
Wed Dec 04 14:08:51 2024
alter database datafile 'c:\program files\oracle\XFF2.dbf' offline drop
Completed: alter database datafile 'c:\program files\oracle\XFF2.dbf' offline drop
alter database open
Wed Dec 04 14:08:57 2024
Thread 1 opened at log sequence 136210
  Current log# 1 seq# 136210 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Dec 04 14:08:57 2024
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is AL32UTF8
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Wed Dec 04 14:08:59 2024
QMNC started with pid=20, OS id=4264 
Completed: alter database open

后面自行尝试recover 数据文件没有成功

Wed Dec 04 14:42:50 2024
ALTER DATABASE RECOVER  datafile 26  
Media Recovery Start
Serial Media Recovery started
ORA-279 signalled during: ALTER DATABASE RECOVER  datafile 26  ...
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log D:\APP\ADMINISTRATOR\FLASH_RECOVERY_AREA\ORCL\ARCHIVELOG\2024_12_04\O1_MF_1_135983_%U_.ARC
Errors with log D:\APP\ADMINISTRATOR\FLASH_RECOVERY_AREA\ORCL\ARCHIVELOG\2024_12_04\O1_MF_1_135983_%U_.ARC
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER CANCEL 
Media Recovery Canceled
Completed: ALTER DATABASE RECOVER CANCEL

由于这两个文件处于offline状态导致客户很多操作报ORA-00376 ORA-01110之类错

ORA-00376: file 10 cannot be read at this time
ORA-01110: data file 10: 'C:\PROGRAM FILES\ORACLE\XFF1.DBF'

对于这类故障使用Oracle Recovery Tools工具,一键恢复

然后直接recover 数据文件成功

对于这类缺少归档数据文件offline的故障Oracle Recovery Tools可以快速傻瓜式恢复
软件下载:OraRecovery下载
使用说明:使用说明

OGG-02771 Input trail file format RELEASE 19.1 is different from previous trail file form at RELEASE 11.2.

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：OGG-02771 Input trail file format RELEASE 19.1 is different from previous trail file form at RELEASE 11.2.

源端数据库从11.2.0.4升级到19c版本(目标端本身就是19.1版本ogg),对应的ogg也从11.2升级到了19.1版本,ogg的trail文件传输到目标端之后,replicat进程直接ABENDED

GGSCI (xifenfei) 3> info replicat HISCA01,detail

REPLICAT   HISCA01   Last Started 2024-12-06 17:18   Status ABENDED
Checkpoint Lag       00:00:00 (updated 13:35:38 ago)
Log Read Checkpoint  File /data/ogg/dirdat/his/re000148
                     2024-12-06 01:12:04.078756  RBA 51446

查看view report查看报错详细

***********************************************************************
**                     Run Time Messages                             **
***********************************************************************


2024-12-06 17:50:55  INFO    OGG-02243  Opened trail file /data/ogg/dirdat/his/re000148 at 2024-12-06 17:50:55.559447.

2024-12-06 17:50:55  INFO    OGG-02232  Switching to next trail file /data/ogg/dirdat/his/re000000149 
     at 2024-12-06 17:50:55.559447 due to EOF. with current RBA 51,446.

Source Context :
  SourceModule            : [er.replicat.processloop]
  SourceID                : [er/replicat/processloop.cpp]
  SourceMethod            : [processReplicatLoop]
  SourceLine              : [1111]
  ThreadBacktrace         : [12] elements
                          : [/data/ogg/libgglog.so(CMessageContext::AddThreadContext())]
                          : [/data/ogg/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...))]
                          : [/data/ogg/libgglog.so(_MSG_Int32_String(CSourceContext*, int, int, char const*, CMessageFactory::MessageDisposition))]
                          : [/data/ogg/replicat()]
                          : [/data/ogg/replicat(ggs::er::ReplicatContext::run())]
                          : [/data/ogg/replicat()]
                          : [/data/ogg/replicat(ggs::gglib::MultiThreading::MainThread::ExecMain())]
                          : [/data/ogg/replicat(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*))]
                          : [/data/ogg/replicat(ggs::gglib::MultiThreading::MainThread::Run(int, char**))]
                          : [/data/ogg/replicat(main)]
                          : [/lib64/libc.so.6(__libc_start_main)]
                          : [/data/ogg/replicat()]

2024-12-06 17:50:55  ERROR   OGG-02171  Error reading LCR from data source. Status 524, data source type TrailDataSource.

Source Context :
  SourceModule            : [er.replicat.ReplicatContext]
  SourceID                : [er/replicat/ReplicatContext.cpp]
  SourceMethod            : [onTrailFormatChange]
  SourceLine              : [564]
  ThreadBacktrace         : [17] elements
                          : [/data/ogg/libgglog.so(CMessageContext::AddThreadContext())]
                          : [/data/ogg/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...))]
                          : [/data/ogg/libgglog.so(_MSG_String_String_String(CSourceContext*, int, char const*, char const*,
                             char const*, CMessageFactory::MessageDisposition))]
                          : [/data/ogg/replicat(ggs::er::ReplicatContext::onTrailFormatChange(char const*, unsigned short, unsigned short) const)]
                          : [/data/ogg/replicat(ggs::gglib::ggtrail::TrailDataSource::updateTrailCompat(ggs::gglib::ggtrail::TrailFile const&))]
                          : [/data/ogg/replicat(ggs::er::ReplicatTrailDataSource::updateTrailCompat(ggs::gglib::ggtrail::TrailFile const&))]
                          : [/data/ogg/replicat(ggs::gglib::ggtrail::TrailDataSource::
                             readNextTrailRecord(ggs::gglib::gglcr::CommonLCR**, long*, int&, int&, bool, bool))]
                          : [/data/ogg/replicat(ggs::er::ReplicatTrailDataSource::readLCR(ggs::gglib::gglcr::CommonLCR**, long&, bool&))]
                          : [/data/ogg/replicat(ggs::er::ReplicatContext::processReplicatLoop())]
                          : [/data/ogg/replicat(ggs::er::ReplicatContext::run())]
                          : [/data/ogg/replicat()]
                          : [/data/ogg/replicat(ggs::gglib::MultiThreading::MainThread::ExecMain())]
                          : [/data/ogg/replicat(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*))]
                          : [/data/ogg/replicat(ggs::gglib::MultiThreading::MainThread::Run(int, char**))]
                          : [/data/ogg/replicat(main)]
                          : [/lib64/libc.so.6(__libc_start_main)]
                          : [/data/ogg/replicat()]

2024-12-06 17:50:55  ERROR   OGG-02771  Input trail file /data/ogg/dirdat/his/re000000149 format RELEASE 19.1 
                                        is different from previous trail file form at RELEASE 11.2.

trail文件情况

[oracle@xifenfei his]$ ls -ltr
total 2167648
-rw-r----- 1 oracle oinstall 157604039 Nov 14 11:44 re000144
-rw-r----- 1 oracle oinstall 499999979 Nov 21 16:48 re000145
-rw-r----- 1 oracle oinstall 499999866 Dec  2 10:06 re000146
-rw-r----- 1 oracle oinstall 266123675 Dec  6 03:36 re000147
-rw-r----- 1 oracle oinstall     51446 Dec  6 04:15 re000148
-rw-r----- 1 oracle oinstall      1211 Dec  6 04:15 re000000149
-rw-r----- 1 oracle oinstall  43711175 Dec  6 17:50 re000000150

大概的意思就是解析完成了148文件,但是在解析149文件时发现trail的版本从11.2变成了19.1,从而导致进程abend.
解决这个问题,需要人工重新指定解析149文件即可

GGSCI (xifenfei) 5>  Alter replicat HISCA01 EXTSEQNO 149, EXTRBA 0

2024-12-06 17:53:01  INFO    OGG-06594  Replicat HISCA01 has been altered. 
Even the start up position might be updated, duplicate suppression remains active in next startup.
To override duplicate suppression, start HISCA01 with NOFILTERDUPTRANSACTIONS option.

REPLICAT altered.


GGSCI (xifenfei) 6> start HISCA01

Sending START request to MANAGER ...
REPLICAT HISCA01 starting

GGSCI (xifenfei) 8> stats HISCA01

Sending STATS request to REPLICAT HISCA01 ...

Start of Statistics at 2024-12-06 17:53:20.

Replicating from U_XFF_A.T_XFF to U_XFF_B.T_XFF:

*** Total statistics since 2024-12-06 17:53:12 ***
        Total inserts                                    431.00
        Total updates                                      0.00
        Total deletes                                    307.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total operations                                 738.00

*** Daily statistics since 2024-12-06 17:53:12 ***
        Total inserts                                    431.00
        Total updates                                      0.00
        Total deletes                                    307.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total operations                                 738.00

*** Hourly statistics since 2024-12-06 17:53:12 ***
        Total inserts                                    431.00
        Total updates                                      0.00
        Total deletes                                    307.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total operations                                 738.00

*** Latest statistics since 2024-12-06 17:53:12 ***
        Total inserts                                    431.00
        Total updates                                      0.00
        Total deletes                                    307.00
        Total upserts                                      0.00
        Total discards                                     0.00
        Total operations                                 738.00

End of Statistics.

OGG-02246 Source redo compatibility level 19.0.0 requires trail FORMAT 12.2 or higher

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：OGG-02246 Source redo compatibility level 19.0.0 requires trail FORMAT 12.2 or higher

在一些情况下,我们会遇到某些原因,在源端和目标端部署不同版本的ogg,如果是目标端版本高于源端版本,一般没有问题,但是如果源端版本较高,需要考虑在抽取和传输进程中加上类似这样的配置

EXTTRAIL <trail file>, FORMAT RELEASE 11.2
 
RMTTRAIL <trail file>, FORMAT RELEASE 11.2

需要注意ogg 19版本设置FORMAT RELEASE 参数最低只能12.2,否则就会出现类似报错,导致进程无法启动

2024-12-05 15:06:59  INFO    OGG-02089  Source redo compatibility version is: 19.0.0.

2024-12-05 15:06:59  INFO    OGG-00546  Default thread stack size: 10485760.

Source Context :
  SourceModule     : [er.redo.ora]
  SourceID         : [er/redo/oracle/redoora.c]
  SourceMethod     : [validateOutTrailFileCompatibility]
  SourceLine       : [6931]
  ThreadBacktrace  : [15] elements
                   : [/ogg/libgglog.so(CMessageContext::AddThreadContext())]
                   : [/ogg/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...))]
                   : [/ogg/libgglog.so(_MSG_String_String(CSourceContext*, int, char const*, char const*, CMessageFactory::MessageDisposition))]
                   : [/ogg/extract()]
                   : [/ogg/extract(RedoAPI::createInstance(ggs::gglib::ggdatasource::DataSource*, ggs::gglib::ggapp::ReplicationContext*))]
                   : [/ogg/extract(ggs::er::OraTranLogDataSource::setup())]
                   : [/ogg/extract(ggs::gglib::ggapp::ReplicationContext::establishStartPoints(char, ggs::gglib::ggdatasource::DataSourceParams c
onst&))]
                   : [/ogg/extract(ggs::gglib::ggapp::ReplicationContext::initializeDataSources(ggs::gglib::ggdatasource::DataSourceParams&))]
                   : [/ogg/extract()]
                   : [/ogg/extract(ggs::gglib::MultiThreading::MainThread::ExecMain())]
                   : [/ogg/extract(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*))]
                   : [/ogg/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**))]
                   : [/ogg/extract(main)]
                   : [/lib64/libc.so.6(__libc_start_main)]
                   : [/ogg/extract()]

2024-12-05 15:06:59  ERROR   OGG-02246  Source redo compatibility level 19.0.0 requires trail FORMAT 12.2 or higher.

2024-12-05 15:06:59  ERROR   OGG-01668  PROCESS ABENDING.

出现这个问题主要是从oracle 12.2开始引入了BigSCN机制,基于目前客户比较常见的数据库版本,最可能出现这类问题的是:源端19c版本数据库,目标端是11.2版本数据库.对于这类同步需求,可以在目标端也部署19版本ogg for 11.2数据库版本来解决

GoldenGate 19安装和打patch

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：GoldenGate 19安装和打patch

1. 下载V983658-01.zip,并上传到/tmp中
2. 安装19.1.0.0.4版本ogg,需要修改静默文件/tmp/fbo_ggs_Linux_x64_shiphome/Disk1/response/oggcore.rsp中修改这两个参数

INSTALL_OPTION=ORA19c
SOFTWARE_LOCATION=/tmp/ogg

3. 静默安装ogg

[oracle@ora19c:/home/oracle]$ /tmp/fbo_ggs_Linux_x64_shiphome/Disk1/runInstaller -silent \
-responseFile /tmp/fbo_ggs_Linux_x64_shiphome/Disk1/response/oggcore.rsp
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB. Actual 6106 MB Passed
Checking swap space: must be greater than 150 MB. Actual 1257 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2024-12-02_10-57-00PM. Please wait ...
[oracle@ora19c:/home/oracle]$ You can find the log of this install session at:
/data/app/oraInventory/logs/installActions2024-12-02_10-57-00PM.log
Successfully Setup Software.
The installation of Oracle GoldenGate Core was successful.
Please check '/data/app/oraInventory/logs/silentInstall2024-12-02_10-57-00PM.log' for more details.

[oracle@ora19c:/home/oracle]$ cd /tmp/ogg
[oracle@ora19c:/tmp/ogg]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29
Operating system character set identified as UTF-8.

GGSCI (ora19c) 1> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER STOPPED

GGSCI (ora19c) 2> create subdirs

Creating subdirectories under current directory /tmp/ogg

Parameter file /tmp/ogg/dirprm: created.
Report file /tmp/ogg/dirrpt: created.
Checkpoint file /tmp/ogg/dirchk: created.
Process status files /tmp/ogg/dirpcs: created.
SQL script files /tmp/ogg/dirsql: created.
Database definitions files /tmp/ogg/dirdef: created.
Extract data files /tmp/ogg/dirdat: created.
Temporary files /tmp/ogg/dirtmp: created.
Credential store files /tmp/ogg/dircrd: created.
Masterkey wallet files /tmp/ogg/dirwlt: created.
Dump files /tmp/ogg/dirdmp: created.

GGSCI (ora19c) 3> exit

4. 下载patch p37236684_1925000OGGRU_Linux-x86-64.zip 和opatch p6880880_190000_Linux-x86-64.zip（12.2.0.1.44版本）并解压

[oracle@ora19c:/tmp]$ cd /tmp/
[oracle@ora19c:/tmp]$ unzip p37236684_1925000OGGRU_Linux-x86-64.zip 
[oracle@ora19c:/tmp]$ cd /tmp/ogg
[oracle@ora19c:/tmp/ogg]$ mv OPatch/ OPatch_bak
[oracle@ora19c:/tmp/ogg]$ unzip /tmp/p6880880_190000_Linux-x86-64.zip 
[oracle@ora19c:/tmp/ogg]$ /tmp/ogg/OPatch/opatch version
OPatch Version: 12.2.0.1.44

OPatch succeeded.

5. 对19.1.0.0.4版本ogg 打上最新patch(37236684)

[oracle@ora19c:/tmp/ogg]$ export ORACLE_HOME=/tmp/ogg
[oracle@ora19c:/tmp/ogg]$ OPatch/opatch apply /tmp/37236684/
Oracle Interim Patch Installer version 12.2.0.1.44
Copyright (c) 2024, Oracle Corporation.  All rights reserved.


Oracle Home       : /tmp/ogg
Central Inventory : /data/app/oraInventory
   from           : /tmp/ogg/oraInst.loc
OPatch version    : 12.2.0.1.44
OUI version       : 12.2.0.4.0
Log file location : /tmp/ogg/cfgtoollogs/opatch/opatch2024-12-02_23-04-26PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   37236684  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/tmp/ogg')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '37236684' to OH '/tmp/ogg'

Patching component oracle.oggcore.ora19c, 19.1.0.0.0...
Patch 37236684 successfully applied.
Log file location: /tmp/ogg/cfgtoollogs/opatch/opatch2024-12-02_23-04-26PM_1.log

OPatch succeeded.
[oracle@ora19c:/tmp/ogg]$ OPatch/opatch lspatches
37236684;

OPatch succeeded.
[oracle@ora19c:/tmp/ogg]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 19.25.0.0.241105 OGGCORE_19.25.0.0.0OGGRU_PLATFORMS_241118.0932_FBO
Linux, x64, 64bit (optimized), Oracle 19c  on Nov 18 2024 13:19:46
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2024, Oracle and/or its affiliates. All rights reserved.



GGSCI (ora19c) 1>

dd破坏asm磁盘头恢复

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：dd破坏asm磁盘头恢复

有朋友对asm disk的磁盘头dd了2048byte的数据

通过分析,gi软件版本,确认是11.2.0.4

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options.
ORACLE_HOME = /u01/app/11.2.0/grid
System name:	Linux
Node name:	rac1
Release:	4.1.12-37.4.1.el6uek.x86_64
Version:	#2 SMP Tue May 17 07:23:38 PDT 2016
Machine:	x86_64

从10.2.0.5之后版本,在第二个au的倒数第二个block上面,有asm disk header备份（每个block大小为4k）,分析au大小(通过分析正常的asm disk快速找到au 大小【使用dd备份的正常的磁盘头查看】)

H:\TEMP\tmp\asmbak>kfed read sdcp.dd |grep ausize
kfdhdb.ausize:                 16777216 ; 0x0bc: 0x01000000

找到被破坏的asm disk的备份磁盘头信息

H:\TEMP\tmp\asmbak>kfed read sdc.dd blkn=4094 aun=1 aus=16777216|more
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                    4094 ; 0x004: blk=4094
kfbh.block.obj:              2147483648 ; 0x008: disk=0
kfbh.check:                   229348702 ; 0x00c: 0x0dab955e
kfbh.fcn.base:                 11727032 ; 0x010: 0x00b2f0b8
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:         ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]:            0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]:            0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]:            0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000
kfdhdb.compat:                186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum:                        0 ; 0x024: 0x0000
kfdhdb.grptyp:                        1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname:               DATA_0000 ; 0x028: length=9
kfdhdb.grpname:                    DATA ; 0x048: length=4
kfdhdb.fgname:                DATA_0000 ; 0x068: length=9
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.crestmp.hi:             33123276 ; 0x0a8: HOUR=0xc DAYS=0x1e MNTH=0xa YEAR=0x7e5
kfdhdb.crestmp.lo:           2259134464 ; 0x0ac: USEC=0x0 MSEC=0x1ea SECS=0x2a MINS=0x21
kfdhdb.mntstmp.hi:             33162836 ; 0x0b0: HOUR=0x14 DAYS=0x12 MNTH=0x1 YEAR=0x7e8
kfdhdb.mntstmp.lo:           3600987136 ; 0x0b4: USEC=0x0 MSEC=0xad SECS=0x2a MINS=0x35
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                 16777216 ; 0x0bc: 0x01000000
kfdhdb.mfact:                    454272 ; 0x0c0: 0x0006ee80
kfdhdb.dsksize:                   65536 ; 0x0c4: 0x00010000
kfdhdb.pmcnt:                         2 ; 0x0c8: 0x00000002
kfdhdb.fstlocn:                       1 ; 0x0cc: 0x00000001
kfdhdb.altlocn:                       2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn:                      0 ; 0x0d4: 0x00000000
kfdhdb.redomirrors[0]:                0 ; 0x0d8: 0x0000
kfdhdb.redomirrors[1]:                0 ; 0x0da: 0x0000
kfdhdb.redomirrors[2]:                0 ; 0x0dc: 0x0000
…………

确认被损坏的磁盘只有磁盘头信息损坏(即确认第二个block是否是好的)

H:\TEMP\tmp\asmbak>kfed read sdc.dd blkn=0
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
0065D8400 00000000 00000000 00000000 00000000  [................]
  Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]


H:\TEMP\tmp\asmbak>kfed read sdc.dd blkn=1|more
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            2 ; 0x002: KFBTYP_FREESPC
kfbh.datfmt:                          2 ; 0x003: 0x02
kfbh.block.blk:                       1 ; 0x004: blk=1
kfbh.block.obj:              2147483648 ; 0x008: disk=0
kfbh.check:                  2781697777 ; 0x00c: 0xa5cd56f1
kfbh.fcn.base:                 39359331 ; 0x010: 0x02589363
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdfsb.aunum:                         0 ; 0x000: 0x00000000
kfdfsb.max:                        1014 ; 0x004: 0x03f6
kfdfsb.cnt:                         147 ; 0x006: 0x0093
kfdfsb.bound:                         0 ; 0x008: 0x0000
kfdfsb.flag:                          1 ; 0x00a: B=1
kfdfsb.ub1spare:                      0 ; 0x00b: 0x00
kfdfsb.spare[0]:                      0 ; 0x00c: 0x00000000
kfdfsb.spare[1]:                      0 ; 0x010: 0x00000000
kfdfsb.spare[2]:                      0 ; 0x014: 0x00000000
kfdfse[0].fse:                        0 ; 0x018: FREE=0x0 FRAG=0x0
…………

基于上述分析,直接使用备份的asm disk header信息进行merge或者repair修复之后,asm 磁盘头状态恢复正常

这个客户运气比较好,库非常大,只是破坏了2k的数据,如果超过4k可能就是比较麻烦的事故了,再次提醒对asm磁盘的dd操作一定要小心谨慎.如果不慎破坏asm磁盘过多,参考以前类似文档:
asm磁盘dd破坏恢复

断电引起的ORA-08102: 未找到索引关键字, 对象号 39故障处理