最初是由于数据库sysaux文件无法正常恢复,重建ctl抛弃sysaux文件,然后打开数据库,但是无法expdp导出数据
Export: Release 12.2.0.1.0 - Production on Wed Jun 24 17:18:04 2026
Copyright (c) 1982, 2017, Oracle and/or its affiliates. All rights reserved.
Connected to: Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
ORA-31626: job does not exist
ORA-31637: cannot create job SYS_EXPORT_SCHEMA_02 for user SYS
ORA-06512: at "SYS.KUPV$FT", line 1140
ORA-06512: at "SYS.KUPV$FT", line 1741
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 95
ORA-06512: at "SYS.KUPV$FT_INT", line 823
ORA-39080: failed to create queues "KUPC$C_1_20260624171804" and "" for Data Pump job
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 95
ORA-06512: at "SYS.KUPC$QUE_INT", line 1541
ORA-00376: file 3 cannot be read at this time
ORA-06512: at "SYS.DBMS_AQADM", line 742
ORA-06512: at "SYS.DBMS_AQADM_SYS", line 8060
ORA-01110: data file 3: '/u01/app/oracle/product/12.2.0.1/dbhome_2/dbs/MISSING00003'
ORA-06512: at "SYS.DBMS_AQADM_SYSCALLS", line 912
ORA-06512: at "SYS.DBMS_AQADM_SYS", line 8036
ORA-06512: at "SYS.DBMS_AQADM", line 737
ORA-06512: at "SYS.KUPC$QUE_INT", line 1461
ORA-06512: at line 1
ORA-06512: at "SYS.KUPC$QUEUE_INT", line 158
ORA-06512: at "SYS.KUPV$FT_INT", line 758
ORA-06512: at "SYS.KUPV$FT", line 1645
ORA-06512: at "SYS.KUPV$FT", line 1101
然后通过各方人员一顿操作猛如虎,导致数据库启动报ORA-600 4137和ORA-600 4193错误,数据库无法open成功
2026-06-24T18:38:50.158906+08:00
alter database open
2026-06-24T18:38:50.182720+08:00
Ping without log force is disabled:
instance mounted in exclusive mode.
2026-06-24T18:38:50.219449+08:00
…………
2026-06-24T18:38:50.514016+08:00
ARC3: Archival started
ARCH: STARTING ARCH PROCESSES COMPLETE
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_48840.trc (incident=304968):
ORA-00600: internal error code, arguments: [4137], [0.77.1546], [0], [0], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_304968/orcl1_ora_48840_i304968.trc
Use ADRCI or Support Workbench to package the incident.
ORACLE Instance orcl1 (pid = 53) - Error 600 encountered while recovering transaction (0, 77).
2026-06-24T18:38:51.313973+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_48840.trc:
ORA-00600: internal error code, arguments: [4137], [0.77.1546], [0], [0], [], [], [], [], [], [], [], []
2026-06-24T18:38:51.649361+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_48840.trc (incident=304969):
ORA-00600: internal error code, arguments: [4193], [1112], [1122], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_304969/orcl1_ora_48840_i304969.trc
2026-06-24T18:38:53.412782+08:00
opiodr aborting process unknown ospid (48840) as a result of ORA-603
需要open故障库,并且正常导出数据,需要处理两个问题
1. 被抛弃的sysaux文件需要正常online起来，不然expdp无法正常导出用户或者全库数据
2. 需要解决open过程的ORA-600 4137/ORA-600 4193错误
对于sysaux文件进行检查，由于重建ctl没有包含异常的sysaux文件，因此无法直接从库中查询到当前各种文件头相关情况,通过obet直接解析文件头获取相关信息(Oracle数据块编辑工具( Oracle Block Editor Tool)-obet)
对于这种情况，可以使用obet的修改文件头checkpoint scn和resetlogs scn功能进行快速修复
OBET> set file 2
filename set to: /u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_sysaux_go991cmw_.dbf (file#2)
OBET> backup
Created backup directory: backup_blk
Successfully backed up block 1 from current file to /tmp/backup_blk/o1_mf_sysaux_go991cmw_.dbf_1.20260624191357
OBET> copy chkscn file 1 to file 2
Error: Edit mode not enabled. Use 'set mode edit' first.
OBET> set mode edit
mode set to: edit
OBET> copy chkscn file 1 to file 2
Confirm Modify chkscn:
Source: file#1 (/u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_system_go990lcg_.dbf)
Target: file#2 (/u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_sysaux_go991cmw_.dbf)
Proceed? (Y/YES to confirm): yes
Successfully copied checkpoint SCN information from file#1 to file#2.
OBET> copy resetlogscn file 1 to file 2
Confirm Modify resetlogscn:
Source: file#1 (/u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_system_go990lcg_.dbf)
Target: file#2 (/u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_sysaux_go991cmw_.dbf)
Proceed? (Y/YES to confirm): yes
Successfully copied resetlog SCN information from file#1 to file#2.
OBET> sum
Check value for File /u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_sysaux_go991cmw_.dbf, Block 1:
current = 0xF21B, required = 0x6651
OBET> sum apply
Confirm applying checksum:
File: /u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_sysaux_go991cmw_.dbf
Block: 1
Offset in block: 16 (file offset: 0x00002010)
Original value: 0xF21B
New value: 0x6651
Confirm? (Y/YES to proceed): y
Verification successful: Stored checksum matches calculated value (0x6651).
Checksum applied successfully.
OBET> tailchk
Check tailchk for File /u02/app/oracle/oradata/PYJHYSYS/PYJHYSYS/datafile/o1_mf_sysaux_go991cmw_.dbf, Block 1:
current = 0x010B0000, required = 0x010B0000
OBET>
然后重建ctl，包含该sysaux，尝试打开数据库,报ORA-600 4193错误
SYS@ORCL> alter database open ;
alter database open
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4193], [1112], [1122], [], [], [], [], [], [], [], [], []
Process ID: 93550
Session ID: 1123 Serial number: 55884
进一步跟踪启动过程，确认报错在update undo$上
PARSING IN CURSOR
#140446136869016
len=160 dep=1 uid=0 oct=6 lid=0 tim=3161302405543 hv=1292341136 ad='9bbd4828' sqlid='8vyjutx6hg3wh'
update /*+ rule */ undo$ set name=:2,file#=:3,block#=:4,status$=:5,user#=:6,undosqn=:7,
xactsqn=:8,scnbas=:9,scnwrp=:10,inst#=:11,ts#=:12,spare1=:13 where us#=:1
END OF STMT
PARSE
#140446136869016
:c=11966,e=11918,p=18,cr=94,cu=0,mis=1,r=0,dep=1,og=3,plh=0,tim=3161302405542
BINDS
#140446136869016
:
Bind
#0
oacdty=01 mxl=32(21) mxlc=00 mal=00 scl=00 pre=00
oacflg=18 fl2=0001 frm=01 csi=852 siz=32 off=0
kxsbbbfp=9bbdac32 bln=32 avl=21 flg=09
value="_SYSSMU12_3861134380$"
Bind
#1
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=7fbc23eda370 bln=24 avl=02 flg=05
value=5
Bind
#2
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=7fbc23eda340 bln=24 avl=03 flg=05
value=144
Bind
#3
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=7fbc23eda310 bln=24 avl=02 flg=05
value=5
Bind
#4
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=7fbc23eda2e0 bln=24 avl=02 flg=05
value=1
Bind
#5
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=7fbc23eda2b0 bln=24 avl=04 flg=05
value=46221
Bind
#6
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=7fbc23eda280 bln=24 avl=05 flg=05
value=30810931
Bind
#7
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=7fbc23eda250 bln=24 avl=06 flg=05
value=3399756014
Bind
#8
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=7fbc23eda220 bln=24 avl=03 flg=05
value=2429
Bind
#9
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=7fbc23eda1f0 bln=24 avl=02 flg=05
value=2
Bind
#10
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=7fbc23eda1c0 bln=24 avl=02 flg=05
value=4
Bind
#11
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=7fbc23eda190 bln=24 avl=02 flg=05
value=2
Bind
#12
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=7fbc23eda3a0 bln=22 avl=02 flg=05
value=12
WAIT
#140446136869016
: nam='db file sequential read' ela= 16 file#=1 block#=547 blocks=1 obj#=0 tim=3161302406306
2026-06-24T19:59:40.979075+08:00
ORA-00600: internal error code, arguments: [4193], [1112], [1122], [], [], [], [], [], [], [], [], []
alert日志中还有ORA-600 4137等错误
ORACLE Instance orcl1 (pid = 53) - Error 600 encountered while recovering transaction (0, 77).
2026-06-24T19:59:40.387459+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_83245.trc:
ORA-00600: internal error code, arguments: [4137], [0.77.1546], [0], [0], [], [], [], [], [], [], [], []
通过这个报错,可以确认是由于0号回滚段,也就是rollback中事务异常，获取相关的trace
[TOC00003]
----- Beginning of Customized Incident Dump(s) -----
XID passed in = xid: 0x0000.04d.0000060a
XID from Undo block = xid: 0x0000.060.00000600
Dump of buffer cache at level 7 for pdb=0 tsn=0 rdba=4194432
BH (0x3ddfd26b8) file#: 1 rdba: 0x00400080 (1/128) class: 15 ba: 0x3ddb80000
set: 166 pool: 3 bsz: 8192 bsi: 0 sflg: 2 pwc: 0,0
dbwrid: 3 obj: -1 objn: 0 tsn: [0/0] afn: 1 hint: f
hash: [0x3ddece808,0xc6bdc2d8] lru: [0xbc1db108,0xbc1db108]
ckptq: [NULL] fileq: [NULL]
objq: [0xa2267bc0,0xa2267bc0] objaq: [0xa2267bb0,0xa2267bb0]
st: XCURRENT md: NULL fpin: 'ktuwh72: ktugus:ktuswr1' fscn: 0x980cfff669f tch: 1
flags:
LRBA: [0x0.0.0] LSCN: [0x0] HSCN: [0x0] HSUB: [65535]
Printing buffer operation history (latest change first):
cnt: 10
01. sid:00 L353:gcur:set:MEXCL 02. sid:00 L145:zib:mk:EXCL
03. sid:00 L212:zib:bic:FSQ 04. sid:00 L122:zgb:set:st
05. sid:00 L830:olq1:clr:WRT+CKT 06. sid:00 L951:zgb:lnk:objq
07. sid:00 L372:zgb:set:MEXCL 08. sid:00 L123:zgb:no:FEN
09. sid:00 L083:zgb:ent:fn 10. sid:01 L203:w_ini_dc:bic:FVB
buffer tsn: 0 rdba: 0x00400080 (1/128)
scn: 0x980cffc5958 seq: 0x01 flg: 0x04 tail: 0x59580e01
frmt: 0x02 chkval: 0x2688 type: 0x0e=KTU UNDO HEADER W/UNLIMITED EXTENTS
基于这样的情况,可以判断通过清理undo$中的相关记录,让其重新分配新的回滚块
Block Header:
block type=0x0e (KTU UNDO HEADER W/UNLIMITED EXTENTS)
block format=0xa2 (oracle 10+)
block rdba=0x00400080 (file#=1, block#=128)
scn=0x0980.cff7c56d, seq=1, tail=0xc56d0e01
block checksum value=0x2683=9859, flag=4
Extent Control Header
-------------------------------------------------------------
Extent Header:: extents: 10 blocks: 79
last map: 0x00000000
#maps
: 0 offset: 4128
Highwater:: 0x00400225 (rfile#=1,block#=549)
ext#: 6 blk#: 5 ext size:8
#blocks
in seg. hdr's freelists: 0
#blocks
below: 0
mapblk: 0x00000000 offset: 6
Map Header:: next: 0x00000000
#extents
: 10 obj#: 0 flag: 0x40000000
Extent Control Header
-------------------------------------------------------------
0x00400081 length: 7
0x004206a8 length: 8
0x004206b0 length: 8
0x00400088 length: 8
0x00400210 length: 8
0x00400218 length: 8
0x00400220 length: 8
0x00400228 length: 8
0x004206a0 length: 8
0x00400230 length: 8
TRN CTL:: seq: 0x0462 chd: 0x005e ctl: 0x000d inc: 0x00000000 nbf: 0x0000
mgc: 0x8002 xts: 0x0068 flg: 0x0001 opt: 2147483646(0x7ffffffe)
uba: 0x00000225.0462.1d scn: 0x0980.cf1f2121
Version: 0x01
FREE BLOCK POOL::
uba: 0x00000000.0462.1c ext: 0x6 spc: 0x11a2
uba: 0x00000000.0462.26 ext: 0x6 spc: 0xc86
uba: 0x00000000.0462.03 ext: 0x6 spc: 0x1e5c
uba: 0x00000000.0460.03 ext: 0x4 spc: 0x1e5c
uba: 0x00000000.043c.21 ext: 0x8 spc: 0xd2c
然后数据库打开成功,使用expdp完美导出数据，完成本次恢复任务
