<< Back to previous view

[QA-63] ORA-600 [3020] on the standby after adding a datafile on primary Created: 16/Feb/18  Updated: 16/Feb/18

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: 11.2.0.4
Operating System: Linux
Host Name: .
Database Name: .

 Description   
Problem:

The customer has added a datafile on the primary database. After the datafile was created on the standby database, ORA-600 [3020] was encountered while applying an archivelog to this datafile on standby.

ORA-600 [3020]:

This is called a 'STUCK RECOVERY'.

  There is an inconsistency between the information stored in the redo 
  and the information stored in a database block being recovered.

Ref: Doc ID 30866.1



 Comments   
Comment by ubTools Support [ 16/Feb/18 10:36 AM ]
PROBLEM OCCURRENCE:

Adding datafile on the Primary:

Tue Feb 13 07:44:52 2018
ALTER TABLESPACE MENKUL2018_DATA
  ADD DATAFILE '/orassd/orcl/datafile/menkul2018_data02.dbf'
  SIZE 5G
  AUTOEXTEND ON
  NEXT 100M
  MAXSIZE UNLIMITED
Completed: ALTER TABLESPACE MENKUL2018_DATA
  ADD DATAFILE '/orassd/orcl/datafile/menkul2018_data02.dbf'
  SIZE 5G
  AUTOEXTEND ON
  NEXT 100M
  MAXSIZE UNLIMITED
Tue Feb 13 07:45:38 2018

Applying Archivelogs on the Standby:

Tue Feb 13 07:46:55 2018
ALTER DATABASE RECOVER AUTOMATIC STANDBY DATABASE UNTIL CHANGE 69358633799
Media Recovery Start
 started logmerger process
Tue Feb 13 07:46:55 2018
Managed Standby Recovery not using Real Time Apply
Parallel Media Recovery started with 24 slaves
Media Recovery Log /u01/ORCL/archive/52b86b9a_1_91417_922239972.arc
Tue Feb 13 07:47:07 2018
Successfully added datafile 232 to media recovery
Datafile #232: '/u01/oracle/app/oradata/ORCL/datafile/ORCL_STBY/datafile/o1_mf_menkul20_f84vg0lt_.dbf'
Incomplete Recovery applied until change 69358633799 time 02/13/2018 07:45:38
Tue Feb 13 07:47:08 2018
Media Recovery Complete (ORCL)
Completed: ALTER DATABASE RECOVER AUTOMATIC STANDBY DATABASE UNTIL CHANGE 69358633799

.....

Tue Feb 13 08:26:55 2018
ALTER DATABASE RECOVER AUTOMATIC STANDBY DATABASE UNTIL CHANGE 69358697423
Media Recovery Start
 started logmerger process
Tue Feb 13 08:26:55 2018
Managed Standby Recovery not using Real Time Apply
Parallel Media Recovery started with 24 slaves
Media Recovery Log /u01/ORCL/archive/52b86b9a_1_91425_922239972.arc
Tue Feb 13 08:26:57 2018
Errors in file /u01/oracle/app/diag/rdbms/orcl_stby/ORCL/trace/ORCL_pr0i_21945.trc  (incident=131897):
ORA-00600: internal error code, arguments: [3020], [232], [3], [973078531], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 232, block# 3, file offset is 24576 bytes)
ORA-10564: tablespace MENKUL2018_DATA
ORA-01110: data file 232: '/u01/oracle/app/oradata/ORCL/datafile/ORCL_STBY/datafile/o1_mf_menkul20_f84vg0lt_.dbf'
ORA-10560: block type '0'
Incident details in: /u01/oracle/app/diag/rdbms/orcl_stby/ORCL/incident/incdir_131897/ORCL_pr0i_21945_i131897.trc
Tue Feb 13 08:26:59 2018
Comment by ubTools Support [ 16/Feb/18 11:06 AM ]
ANALYSIS of the RESULT:
Ref: ORCL_pr0i_21945.trc

Data:

REDO Dump:

KCOX_FUTURE: CHANGE IN FUTURE OF BLOCK

*** 2018-02-13 08:26:56.826
RECOVERY STUCK AT BLOCK 3 OF FILE 232
Redo record scn: 0x0010.2619b07d
CHANGE #1 TYP:0 CLS:12 AFN:232 DBA:0x3a000003 OBJ:4294967295
 SCN:0x0010.2618c0fb SEQ:2 OP:22.5 ENC:0 RBL:0

Buffer read during recovery:
.....

The stuck recovery happened at file#232 block#3 with KCOX_FUTURE: CHANGE IN FUTURE OF BLOCK information.

Block Dump:

buffer tsn: 141 rdba: 0x3a000003 (232/3)
scn: 0x0000.00000000 seq: 0x01 flg: 0x05 tail: 0x00000001
frmt: 0x02 chkval: 0x9d03 type: 0x00=unknown
on-disk scn: 0x0.0

SCN is 0, type is unknown. flg is 0x05:

Where flg: 0x05 contains flag 0x1 (unused,unformatted block).

Ref: Oracle Doc ID 17896895.8

Comment:

The change vector was expecting SCN:0x0010.2618c0fb on the block. But, the SCN on the block was 0x0000.00000000.

Oracle was trying to apply archivelog to an unformatted block. REDO in archivelog is beyond block in datafile. This inconsistency causes stuck recovery.

Comment by ubTools Support [ 16/Feb/18 12:12 PM ]
ANALYSIS of the ROOT CAUSE:

Data:

The datafile has been created at sequence#91417 and the problem happened at sequence#91425 at file#232 block#3.

REDO Dump Commands:

SQL> ALTER SYSTEM dump  logfile '/u01/ORCL/archive/52b86b9a_1_91417_922239972.arc' dba min 232 3 dba max 232 3;

System altered.

SQL> ALTER SYSTEM dump  logfile '/u01/ORCL/archive/52b86b9a_1_91418_922239972.arc' dba min 232 3 dba max 232 3;

System altered.

SQL> ALTER SYSTEM dump  logfile '/u01/ORCL/archive/52b86b9a_1_91419_922239972.arc' dba min 232 3 dba max 232 3;

System altered.

SQL> ALTER SYSTEM dump  logfile '/u01/ORCL/archive/52b86b9a_1_91420_922239972.arc' dba min 232 3 dba max 232 3;

System altered.

SQL> ALTER SYSTEM dump  logfile '/u01/ORCL/archive/52b86b9a_1_91421_922239972.arc' dba min 232 3 dba max 232 3;

System altered.

SQL> ALTER SYSTEM dump  logfile '/u01/ORCL/archive/52b86b9a_1_91422_922239972.arc' dba min 232 3 dba max 232 3;

System altered.

SQL>  ALTER SYSTEM dump  logfile '/u01/ORCL/archive/52b86b9a_1_91423_922239972.arc' dba min 232 3 dba max 232 3;

System altered.

SQL> ALTER SYSTEM dump  logfile '/u01/ORCL/archive/52b86b9a_1_91424_922239972.arc' dba min 232 3 dba max 232 3;

System altered.

SQL>  ALTER SYSTEM dump  logfile '/u01/ORCL/archive/52b86b9a_1_91425_922239972.arc' dba min 232 3 dba max 232 3;

System altered.

SQL> 

REDO Dumps:

DUMP OF REDO FROM FILE '/u01/ORCL/archive/52b86b9a_1_91417_922239972.arc'
.....
REDO RECORD - Thread:1 RBA: 0x016519.00000765.01e8 LEN: 0x0058 VLD: 0x01
SCN: 0x0010.2618c0fb SUBSCN:  1 02/13/2018 07:44:59
(LWN RBA: 0x016519.00000763.0010 LEN: 0004 NST: 0002 SCN: 0x0010.2618c0f9)
CHANGE #1 TYP:1 CLS:12 AFN:232 DBA:0x3a000003 OBJ:4294967295 SCN:0x0010.2618c0fb SEQ:1 OP:22.4 ENC:0 RBL:0
ktfbbfo - File BitMap Block Format:
BitMap Control:
RelFno: 232, BeginBlock: 128, Flag: 0, First: 0, Free: 63488

REDO RECORD - Thread:1 RBA: 0x016519.00000763.0010 LEN: 0x0244 VLD: 0x05
SCN: 0x0010.2618c0fb SUBSCN:  1 02/13/2018 07:44:59
CHANGE #1 TYP:0 CLS:69 AFN:3 DBA:0x00c018c0 OBJ:4294967295 SCN:0x0010.2618c0ee SEQ:1 OP:5.4 ENC:0 RBL:0
ktucm redo: slt: 0x0007 sqn: 0x000060e7 srt: 0 sta: 9 flg: 0x2 ktucf redo: uba: 0x3000319f.07d5.06 ext: 2 spc: 7334 fbi: 0
CHANGE #2 MEDIA RECOVERY MARKER SCN:0x0000.00000000 SEQ:0 OP:17.30 ENC:0
Add datafiles to tablespace #141
file #232  relative file #232. '/orassd/orcl/datafile/menkul2018_data02.dbf'
flags(reuse): 0x0
Checkpointed at scn:  0x0010.2618c0f0 02/13/2018 07:44:56
.....
DUMP OF REDO FROM FILE '/u01/ORCL/archive/52b86b9a_1_91425_922239972.arc'
.....
REDO RECORD - Thread:1 RBA: 0x016521.0001a022.0034 LEN: 0x0040 VLD: 0x01
SCN: 0x0010.2619b07d SUBSCN: 14 02/13/2018 08:23:51
(LWN RBA: 0x016521.00019dca.0010 LEN: 1012 NST: 0002 SCN: 0x0010.2619b069)
CHANGE #1 TYP:0 CLS:12 AFN:232 DBA:0x3a000003 OBJ:4294967295 SCN:0x0010.2618c0fb SEQ:2 OP:22.5 ENC:0 RBL:0
ktfbbredo - File BitMap Block Redo:
Use Bits:

Comment:

The datafile has been created at sequence#91417 by REDO OP code 17.30, which means:

the OP:17.30 redo which adds the <file#> datafile

Ref: Oracle Doc ID 27229389.8

There are some other OP codes 22.4 and 5.4 before adding the datafile.

Change Vector for OP Code 22:4:

It tries to change absolute file#232 (AFN:232 DBA:0x3a000003). This is the problem that Oracle tries to apply a change vector to a file which was not created yet.

Change Vector for OP Code 5:4:

It tries to change absolute file#3 (AFN:3 DBA:0x00c018c0). This is a different file. So, it's out of the scope.

Comment by ubTools Support [ 16/Feb/18 12:26 PM ]
SOLUTION:

Problem:

Oracle tries to apply archivelog to a file which was not created on standby yet.

Fix:

This is Oracle bug 27229389.

Workaround:

Copy datafile from primary to standby that doesn't require corrupted archivelogs.





[QA-41] Startup database fails with ORA-600 [4000], ORA-600 [4137]. Created: 16/Jun/08  Updated: 17/Jun/08

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: Oracle 8.1.7.3.0
Operating System: HP-UX

 Description   
After an hardware problem, database crashed.

Since an ARCHIVELOG is missed, and restoring the previous backup is not acceptable, the customer wanted to open database in inconsistent state.



 Comments   
Comment by ubTools Support [ 16/Jun/08 11:38 PM ]
Steps to open the database:
  • setting _ALLOW_RESETLOGS_CORRUPTION=TRUE in init<SID>.ora.
  • startup mount;
  • recover database until cancel;
    <--cancel
  • alter database open resetlogs;

But, it failed with the following error:

ORA-00600: internal error code, arguments: [4000], [9], [], [], [], [], [], []

Oracle Note:47456.1:

DESCRIPTION:

  This has the potential to be a very serious error.

  It means that Oracle has tried to find an undo segment number in the 
  dictionary cache and failed.

ARGUMENTS:
  Arg [a] Undo segment number

FUNCTIONALITY:      
  KERNEL TRANSACTION UNDO

IMPACT:             
  INSTANCE FAILURE - Instance will not restart
  STATEMENT FAILURE
Comment by ubTools Support [ 17/Jun/08 12:13 AM ]
An exerpt from the trace file:
ORA-00600: internal error code, arguments: [4000], [9], [], [], [], [], [], []
Current SQL statement for this session:
select ctime, mtime, stime from obj$ where obj# = :1
...
Block header dump:  0x0080003e
 Object id on Block? Y
 seg/obj: 0x12  csc: 0x570.b8368d16  itc: 1  flg: -  typ: 1 - DATA
     fsl: 0  fnx: 0x0 ver: 0x01
...
 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   xid:  0x0009.019.000dc23f    uba: 0x58c13ddb.0523.46  --U-    1  fsc 0x0000.b8368d17

Looks like a problem regarding obj$ and its undo...If undo requirement is bypassed, there will be no requirement for undo. In order to do that, bumping SNC further needed.

csc shows the the SCN of last block cleanout. We guessed it may be used a target bumping SCN as below:

  • 0x570.b8368d16 => 0x570b8368d16 => Decimal: 5981685058838 => divide by 1024/1024/1024 = 5571

Bump SCN as below and restart:

  • Setting _MINIMUM_GIGA_SCN = 5571 in init<SID>.ora
  • startup mount;
  • recover database until cancel;
    <--cancel
  • alter database open resetlogs;

ORA-600 [4000] disappeared. But now, the following error appeared:

ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []

Oracle Note:47456.1:

DESCRIPTION:        

  While backing out an undo record (i.e. at the time of rollback) we found a
  transaction id mis-match indicating either a corruption in the rollback 
  segment or corruption in an object which the rollback segment is trying to
  apply undo records on.

  This would indicate a corrupted rollback segment. 

FUNCTIONALITY:      
 Kernel Transaction Undo Recovery
 
IMPACT:             
  POSSIBLE PHYSICAL CORRUPTION in Rollback segments
Comment by ubTools Support [ 17/Jun/08 12:21 AM ]
Restart the database:
  • Setting _CORRUPTED_ROLLBACK_SEGMENTS in init<SID>.ora
  • startup mount;
  • recover database until cancel;
    <--cancel
  • alter database open resetlogs;

The database is opened.

Since it's opened in inconsistent state, a full export and then import into a new database is required to get rid of the inconsistency in Oracle dictionary. But, the customer data will not be consistent after the import. It should be reviewed by the customer.

Comment by ubTools Support [ 17/Jun/08 09:28 AM ]
The database was opened inconsistently. It'll be recreated with full export/import.




[QA-38] DBMS_XMLPARSER.FREEPARSER doesn't release UGA memory. Created: 06/Jun/08  Updated: 06/Jun/08

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production
Operating System: Solaris

 Description   
DBMS_XMLPARSER.FREEPARSER doesn't release UGA memory.

Session memory statistics before operation:

SQL> select name,value from v$sesstat a, v$statname b
  2  where a.statistic#=b.statistic#
  3  and b.name like '%memory%'
  4  and sid = 58
  5  order by value desc;

NAME                                                                  VALUE
---------------------------------------------------------------- ----------
session pga memory                                                   424336
session pga memory max                                               424336
session uga memory                                                   209872
session uga memory max                                               209872
sorts (memory)                                                           16
workarea memory allocated                                                14

6 rows selected.

Operation:

...
dbms_xmlparser.parseclob (v_parser, data_for_table);
...
dbms_xmlparser.freeParser(v_parser);
...

Session memory statistics after operation:

SQL> select name,value from v$sesstat a, v$statname b
  2  where a.statistic#=b.statistic#
  3  and b.name like '%memory%'
  4  and sid = 58
  5  order by value desc;

NAME                                                                  VALUE
---------------------------------------------------------------- ----------
session pga memory                                                 52396928
session pga memory max                                             52396928
session uga memory                                                 51816784
session uga memory max                                             51816784
sorts (memory)                                                           19
workarea memory allocated                                                14

6 rows selected.

An excerpt from HEAPDUMP LEVEL 4 (UGA) dump:

...
EXTENT 788 addr=ffffffff7ce90080
  Chunk ffffffff7ce90090 sz=      392    free      "               "
  Chunk ffffffff7ce90218 sz=      184    freeable  "kgiobdtb       "
  Chunk ffffffff7ce902d0 sz=     1112    recreate  "koh-kghu sessi "  latch=0
     ds ffffffff7ce9db50 sz=     1112 ct=        1
  Chunk ffffffff7ce90728 sz=     2136    freeable  "PLS non-lib hp "  ds=ffffffff7cf6abd8
  Chunk ffffffff7ce90f80 sz=     4288    freeable  "qmxdpls_subhea "  ds=ffffffff7ce96b78
  Chunk ffffffff7ce92040 sz=     4288    freeable  "qmxdpls_subhea "  ds=ffffffff7ce96b78
  Chunk ffffffff7ce93100 sz=     4288    freeable  "qmxdpls_subhea "  ds=ffffffff7ce96b78
  Chunk ffffffff7ce941c0 sz=     4288    freeable  "qmxdpls_subhea "  ds=ffffffff7ce96b78
  Chunk ffffffff7ce95280 sz=     4328    freeable  "qmxdpls_subhea "  ds=ffffffff7ce96b78
  Chunk ffffffff7ce96368 sz=       48    freeable  "allocator state"
  Chunk ffffffff7ce96398 sz=       72    freeable  "persistant defi"
  Chunk ffffffff7ce963e0 sz=       48    freeable  "kgbt           "
  Chunk ffffffff7ce96410 sz=       48    freeable  "frame segment  "
  Chunk ffffffff7ce96440 sz=       64    freeable  "qmxdpls_init_ug"
  Chunk ffffffff7ce96480 sz=       48    freeable  "frame segment  "
  Chunk ffffffff7ce964b0 sz=       72    freeable  "frame segment  "
  Chunk ffffffff7ce964f8 sz=       72    freeable  "kxsxsi: frame  "
  Chunk ffffffff7ce96540 sz=     1568    recreate  "qmxdpls_subhea "  latch=0
     ds ffffffff7ce96b78 sz= 50681480 ct=    11820
        ffffffff779d6940 sz=     4288
        ffffffff779d7a00 sz=     4288
        ffffffff779d8ac0 sz=     4288
        ffffffff779d9b80 sz=     4288
        ffffffff779dac40 sz=     4288
        ffffffff779dbd00 sz=     4288
        ffffffff779dcdc0 sz=     4288
        ffffffff779dde80 sz=     4288
        ffffffff779def40 sz=     4288
        ffffffff779c04c0 sz=     4288
        ffffffff779c1580 sz=     4288
        ffffffff779c2640 sz=     4288
        ffffffff779c3700 sz=     4288
        ffffffff779c47c0 sz=     4288
        ffffffff779c5880 sz=     4288
        ffffffff779c6940 sz=     4288
        ffffffff779c7a00 sz=     4288
...
       ffffffff7ce93100 sz=     4288
        ffffffff7ce941c0 sz=     4288
        ffffffff7ce95280 sz=     4328
  Chunk ffffffff7ce96b60 sz=      160    freeable  "qmxdpls_heapptr"
  Chunk ffffffff7ce96c00 sz=      232    freeable  "lob ctl struct "
  Chunk ffffffff7ce96ce8 sz=       80    freeable  "frame          "
  Chunk ffffffff7ce96d38 sz=       40    freeable  "private oac inf"
  Chunk ffffffff7ce96d60 sz=      128    freeable  "bnrdef and uac "
  Chunk ffffffff7ce96de0 sz=      600    recreate  "bind var heap  "  latch=0
     ds ffffffff7ce971f0 sz=      600 ct=        1
  Chunk ffffffff7ce97038 sz=      928    freeable  "kgiob          "
  Chunk ffffffff7ce973d8 sz=     4160    freeable  "koh-kghu sessi "  ds=ffffffff7cf65710
  Chunk ffffffff7ce98418 sz=     8192    freeable  "kdit           "
  Chunk ffffffff7ce9a418 sz=       40    free      "               "
  Chunk ffffffff7ce9a440 sz=     8192    freeable  "kdit           "
  Chunk ffffffff7ce9c440 sz=       48    freeable  "ktatt          "
  Chunk ffffffff7ce9c470 sz=       48    freeable  "kdit           "
  Chunk ffffffff7ce9c4a0 sz=       80    freeable  "kgicu          "
  Chunk ffffffff7ce9c4f0 sz=     5672    free      "               "
  Chunk ffffffff7ce9db18 sz=     2520    freeable  "koh-kghu sessio"
  Chunk ffffffff7ce9e4f0 sz=       48    freeable  "frame segment  "
  Chunk ffffffff7ce9e520 sz=       40    freeable  "frame segment  "
  Chunk ffffffff7ce9e548 sz=       72    freeable  "kxsxsi: frame  "
  Chunk ffffffff7ce9e590 sz=     2464    perm      "perm           "  alo=432
  Chunk ffffffff7ce9ef30 sz=       48    freeable  "allocator state"
  Chunk ffffffff7ce9ef60 sz=       80    freeable  "frame          "
  Chunk ffffffff7ce9efb0 sz=      128    freeable  "bnrdef and uac "
  Chunk ffffffff7ce9f030 sz=      600    recreate  "bind var heap  "  latch=0
     ds ffffffff7ce9f440 sz=      600 ct=        1
  Chunk ffffffff7ce9f288 sz=      928    freeable  "kgiob          "
  Chunk ffffffff7ce9f628 sz=     2520    freeable  "koh-kghu sessio"
EXTENT 789 addr=ffffffff7ce30080
  Chunk ffffffff7ce30090 sz=     2016    perm      "perm           "  alo=2016
 ...
Total heap size    = 51790440
FREE LISTS:
 Bucket 0 size=56
...
 Bucket 16 size=524312
 Bucket 17 size=2097176
Total free space   =   870336
UNPINNED RECREATABLE CHUNKS (lru first):
PERMANENT CHUNKS:
  Chunk ffffffff7ce9e590 sz=     2464    perm      "perm           "  alo=432
  Chunk ffffffff7ce30090 sz=     2016    perm      "perm           "  alo=2016
  Chunk ffffffff7cf70090 sz=      288    perm      "perm           "  alo=288
  Chunk ffffffff7cf600a8 sz=    20320    perm      "perm           "  alo=20320
Permanent space    =    25088
******************************************************

DBMS_SESSION.FREE_UNUSED_USER_MEMORY did not help.



 Comments   
Comment by ubTools Support [ 06/Jun/08 01:35 PM ]
The UGA of PGA had been filled with a big chunk which has recreatable "qmxdpls_subhea". This chunk is 50681480 byte. (See QA-8 for the simple definitions of HEAPDUMP).

Oracle Note:3518909.8:

Calling Dbms_xmlparser.freeParser / dbms_xmldom.freeDocument in the procudure do not 
appear to free the memory.

The leaked memory shows in heapdumps as "qmxdpls_subheap"

Although the mentioned bug fixed in Oracle 9.2.0.6; the customer encounters the same problem in Oracle 9.2.0.8.

Since the next usage of DBMS_XMLPARSER.PARSECLOB after a previous DBMS_XMLPARSER.FREEPARSER within the same session, the UGA did not grow. This is acceptable by the customer.





[QA-35] ORA-00600 [kturrur11], [65535], [0]: Instance crashed. Created: 08/Nov/07  Updated: 08/Nov/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
Operating System: IBM-AIX

 Description   
The instance crashes with the following error code:
ORA-00600: internal error code, arguments: [kturrur11], [65535], [0], [], [], [], [], []

Stack trace:

----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
ksedst+001c          bl       ksedst1              FFFFFFFFFFFA3B0 ? 000000000 ?
ksedmp+0290          bl       ksedst               1047C9C10 ?
ksfdmp+0018          bl       03F53584             
kgerinv+00dc         bl       _ptrgl               
kgeasnmierr+0040     bl       kgerinv              0FFFFFFFF ? 0000000BA ?
                                                   FFFFFFFFFFFAD18 ?
                                                   FFFFFFFFFFFAD20 ?
                                                   FFFFFFFFFFFAC08 ?
kturgmbu+02b8        bl       kgeasnmierr          000085188 ? 000000000 ?
                                                   000550021 ? 200000002 ?
                                                   000000000 ? 00000FFFF ?
                                                   000000000 ? 000000000 ?
kturrur+01c8         bl       kturgmbu             1001AEFA8 ? 70000020F745AC0 ?
                                                   0000F4240 ? 104BF4B00 ?
                                                   000085188 ? FFFFFFFFFFFA950 ?
                                                   70000020A3BC630 ? 1102B0D58 ?
ktundo+016c          bl       kturrur              1102B0D58 ? 000000000 ?
                                                   100000000 ? FFFFFFFFFFFA9D0 ?
                                                   FFFFFF00000003 ? 15453015F ?
                                                   000000000 ? 000000000 ?
ktubko+0794          bl       ktundo               1FFFFBC80 ?
                                                   5B16B1B30F745928 ?
                                                   1001DD940 ? 70000020A3C8EA0 ?
                                                   000000528 ? FFFFFFFFFFFBE08 ?
                                                   400000000 ? FFFFFFFFFFFBF08 ?
kturrt+15fc          bl       ktubko               70000020A3C52F0 ? 600000000 ?
                                                   000000000 ? 0E8E65D60 ?
                                                   3B008CEC89 ? 55002100000000 ?
kturec+0dcc          bl       kturrt               FFFFFFFFFFFC528 ?
                                                   21000000000000 ? 1FFFFC5E0 ?
                                                   000000000 ? 0000009A0 ?
                                                   40A288838 ? 110021A88 ?
kturax+0300          bl       kturec               5522880400 ? 000000000 ?
                                                   19E370001 ? 000000001 ?
                                                   FFFFFFFFFFFFCAC0 ?
                                                   11FFFFFEFF ? 400000000 ?
ktprbeg+02b0         bl       kturax               10FDB1B2B0 ? 004AD8530 ?
ktmmon+0ebc          bl       ktprbeg              080000000 ?
ktmSmonMain+0030     bl       ktmmon               000000000 ?
ksbrdp+03e0          bl       _ptrgl               
opirip+03fc          bl       01FC66A0             
opidrv+0448          bl       opirip               1103BD070 ? 4103BE990 ?
                                                   FFFFFFFFFFFF860 ?
sou2o+0090           bl       opidrv               32023373FC ? 400000020 ?
                                                   FFFFFFFFFFFF860 ?
opimai_real+0150     bl       01FC0DF4             
main+0098            bl       opimai_real          000000000 ? 000000000 ?
__start+0090         bl       main                 000000000 ? 000000000 ?


 Comments   
Comment by ubTools Support [ 08/Nov/07 08:40 AM ]
From Oracle Note:4940513.8:
Bug 4940513  OERI[kturrur11] can occur with multi block undo
 This note gives a brief overview of bug 4940513.
Affects:

    Product (Component)	Oracle Server (Rdbms)
    Range of versions believed to be affected	Versions < 11
    Versions confirmed as being affected	

        * 9.2.0.6
        * 9.2.0.7
        * 10.1.0.4
        * 10.1.0.5
        * 10.2.0.2 

    Platforms affected	Generic (all / most platforms affected)

Fixed:

    This issue is fixed in	

        * 9.2.0.8 (Server Patch Set)
        * 10.2.0.3 (Server Patch Set)
        * 11g (Future version) 

Symptoms:
	
Related To:

    * Internal Error May Occur (ORA-600)
    * ORA-600 [kturrur11] 

	

    * (None Specified) 

Description

    In rare situations the server could raise ORA-600 [kturrur11][65535][0]

    Workaround:
      Avoid the multi block undo code path by making sure that the 
      block size in the undo tablespace is large enough to accomodate the 
      largest column that is changed by any SQL statement. If the block size 
      of the data tablespaces is larger than the block size of the undo 
      tablespace, increase the blocksize of the undo tablespace to that of 
      the data tablespace.

Comment by ubTools Support [ 08/Nov/07 10:32 AM ]
Workaround:

Drop segments which need recovery.

Finding the UNDO segment from ALERT LOG:

Errors in file /product/10g/admin/DTWP/bdump/dtwp1_smon_7024648.trc:
ORA-00600: internal error code, arguments: [kturrur11], [65535], [0], [], [], [], [], []
replication_dependency_tracking turned off (no async multimaster replication found)
Sat Nov  3 13:19:38 2007
ORACLE Instance DTWP1 (pid = 15) - Error 600 encountered while recovering transaction (85, 33).
Sat Nov  3 13:19:38 2007
Errors in file /product/10g/admin/DTWP/bdump/dtwp1_smon_7024648.trc:
ORA-00600: internal error code, arguments: [kturrur11], [65535], [0], [], [], [], [], []

SMON is trying to rollback a transaction in (UNDOSEGMENT#85, UNDOSLOT#33).

Identifiying UNDO segment:

select segment_name,owner,tablespace_name from dba_rollback_segs
where segment_id=85;  

SEGMENT_NAME                   OWNER  TABLESPACE_NAME
------------------------------ ------ ------------------------------
_SYSSMU85$                     PUBLIC UNDOTBS1

Undo block in the SMON trace:

********************************************************************************
UNDO BLK:  
xid: 0x0055.021.00085188  seq: 0xffff cnt: 0x1   irb: 0x1   icl: 0x0   flg: 0x0000
 
 Rec Offset      Rec Offset      Rec Offset      Rec Offset      Rec Offset
---------------------------------------------------------------------------
0x01 0x0018     
 
*-----------------------------
* Rec #0x1  slt: 0x21  objn: 125213(0x0001e91d)  objd: 564547  tblspc: 10(0x0000000a)
*       Layer:   5 (Transaction Undo)   opc: 1   rci 0x00   
Undo type:  Multi-block undo Mid-piece   Last buffer split:  Yes 
Temp Object:  No 
Tablespace Undo:  No 
rdba: 0x5b16b1af
*-----------------------------

Transaction ID: xid: 0x0055.021.00085188

Hexadecimal 55 = Decimal 85
Hexadecimal 21 = Decimal 33

That means this UNDO block is the block which SMON is reading to rollback a segment.

irb points to last UNDO RECORD in UNDO block. rci points to previous UNDO RECORD. if rci=0, it's the first UNDO RECORD. Recovery operation starts from irb and chain is followed by rci until rci is zero.

In this case, the UNDO block includes just one UNDO RECORD. This UNDO RECORD inludes UNDO DATA for object#125213.

Object needs recovery:

select owner,object_name,object_type from dba_objects where object_id=125213

Owner : OWBRUN
Object_name : sm_post_ind2
Object_type  INDEX

Index dropped. But problem did not disappear. Then, it's decided to drop this UNDO segment after identifiying all objects in.

Reading Transaction Table in the UNDO header:

ALTER SYSTEM DUMP UNDO HEADER '_SYSSMU85$';

...
********************************************************************************
Undo Segment:  _SYSSMU85$ (85)
********************************************************************************
...
TRN TBL::
 
  index  state cflags  wrap#    uel         scn            dba            parent-xid    nub     stmt_num    cmt
  ------------------------------------------------------------------------------------------------
   0x00    9    0x00  0x85435  0xffff  0x0847.4024f907  0x00000000  0x0000.000.00000000  0x00000000
   0x00000000  1194066440
   0x01    9    0x00  0x84b3c  0x0004  0x0847.4024f89b  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x02    9    0x00  0x85237  0x0006  0x0847.4024f895  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x03    9    0x00  0x85406  0x0011  0x0847.4024f877  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x04    9    0x00  0x851d9  0x000a  0x0847.4024f89e  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x05    9    0x00  0x85234  0x002f  0x0847.4024f881  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x06    9    0x00  0x8543f  0x002b  0x0847.4024f897  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x07    9    0x00  0x850ce  0x002e  0x0847.4024f8ac  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x08    9    0x00  0x853f3  0x001a  0x0847.4024f88a  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x09    9    0x00  0x85188  0x001f  0x0847.4024f87d  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x0a    9    0x00  0x84f75  0x0014  0x0847.4024f8a0  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x0b    9    0x00  0x832f2  0x0007  0x0847.4024f8aa  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x0c    9    0x00  0x85313  0x001e  0x0847.4024f8d2  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x0d    9    0x00  0x85320  0x000c  0x0847.4024f8d0  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x0e    9    0x00  0x849fb  0x0012  0x0847.4024f890  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x0f    9    0x00  0x8530c  0x000e  0x0847.4024f88e  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x10    9    0x00  0x84ac9  0x0015  0x0847.4024f8de  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x11    9    0x00  0x854f4  0x0009  0x0847.4024f87a  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x12    9    0x00  0x84ce9  0x0002  0x0847.4024f892  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x13    9    0x00  0x85220  0x001b  0x0847.4024f8c4  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x14    9    0x00  0x85119  0x001d  0x0847.4024f8a2  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x15    9    0x00  0x8540c  0x0025  0x0847.4024f8e0  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x16    9    0x00  0x85177  0x0017  0x0847.4024f8ea  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x17    9    0x00  0x84f02  0x002a  0x0847.4024f8ec  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x18    9    0x00  0x84e2d  0x0027  0x0847.4024f8f7  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x19    9    0x00  0x8537a  0x0020  0x0847.4024f8a6  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x1a    9    0x00  0x8530b  0x000f  0x0847.4024f88c  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x1b    9    0x00  0x841bc  0x0029  0x0847.4024f8c6  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x1c    9    0x00  0x852a9  0x002d  0x0847.4024f8fb  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x1d    9    0x00  0x84d24  0x0019  0x0847.4024f8a4  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x1e    9    0x00  0x85419  0x0010  0x0847.4024f8d3  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x1f    9    0x00  0x84ea2  0x0005  0x0847.4024f87f  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x20    9    0x00  0x853a5  0x000b  0x0847.4024f8a8  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x21   10    0x10  0x85188  0x0919  0x0847.4024f8fd  0x5b16b1b3  0x0000.000.00000000  0x00000002   0x00000000  0
   0x22    9    0x00  0x85279  0x002c  0x0847.4024f886  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x23    9    0x00  0x847b0  0x0028  0x0847.4024f8bb  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x24    9    0x00  0x851cf  0x0023  0x0847.4024f8b9  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x25    9    0x00  0x84a9c  0x0016  0x0847.4024f8e1  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x26   10    0x90  0x7539b  0x0003  0x0847.3fcdcb21  0x4f81d3d2  0x0000.000.00000000  0x0000dbd9   0x00000000  0
   0x27    9    0x00  0x850ac  0x001c  0x0847.4024f8f9  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x28    9    0x00  0x8531b  0x0013  0x0847.4024f8bc  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x29    9    0x00  0x854f0  0x000d  0x0847.4024f8c7  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x2a    9    0x00  0x85301  0x0018  0x0847.4024f8ed  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x2b    9    0x00  0x83c38  0x0001  0x0847.4024f899  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x2c    9    0x00  0x85051  0x0008  0x0847.4024f888  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x2d    9    0x00  0x84a3c  0x0000  0x0847.4024f904  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x2e    9    0x00  0x84f35  0x0024  0x0847.4024f8ae  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x2f    9    0x00  0x85100  0x0022  0x0847.4024f884  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440

State#10 means active transaction. dba points to starting UNDO block address.

There are 2 active transactions. The one of them points to the slot of 0x21, which is the same as seen in the SMON trace that causes this ORA-600 [kturrur11] error. The other active transaction is available in the slot of 0x26, which has a dba of 0x4f81d3d2.

The object in the slot of 0x21 had been found above; but the object in slot of 0x26 is not known yet.

Object needs recovery:

Hexadecimal 4f81d3d2 = Decimal 1333908434

select DBMS_UTILITY.DATA_BLOCK_ADDRESS_FILE(1333908434) from x$dual;
318

select DBMS_UTILITY.DATA_BLOCK_ADDRESS_BLOCK(1333908434) from x$dual;
119762

alter system dump datafile 318 block 119762;

...
*** SESSION ID:(489.37) 2007-11-03 16:23:56.878
Start dump data blocks tsn: 1 file#: 318 minblk 119762 maxblk 119762
buffer tsn: 1 rdba: 0x4f81d3d2 (318/119762)
...
UNDO BLK:  
xid: 0x0055.026.0007539b  seq: 0xf72d cnt: 0x5b  irb: 0x1   icl: 0x0   flg: 0x0000
 
 Rec Offset      Rec Offset      Rec Offset      Rec Offset      Rec Offset
---------------------------------------------------------------------------
0x01 0x1f8c     0x02 0x1f30     0x03 0x1ed4     0x04 0x1e78     0x05 0x1e1c     
0x06 0x1dc0     0x07 0x1d64     0x08 0x1d08     0x09 0x1cac     0x0a 0x1c50     
0x0b 0x1bf4     0x0c 0x1b9c     0x0d 0x1b48     0x0e 0x1af0     0x0f 0x1a98     
0x10 0x1a40     0x11 0x19e8     0x12 0x1990     0x13 0x193c     0x14 0x18e4     
0x15 0x1890     0x16 0x1838     0x17 0x17e0     0x18 0x178c     0x19 0x1738     
0x1a 0x16e0     0x1b 0x1688     0x1c 0x1634     0x1d 0x15dc     0x1e 0x1584     
0x1f 0x152c     0x20 0x14d4     0x21 0x147c     0x22 0x1424     0x23 0x13cc     
0x24 0x1374     0x25 0x131c     0x26 0x12c4     0x27 0x126c     0x28 0x1214     
0x29 0x11bc     0x2a 0x1168     0x2b 0x1110     0x2c 0x10b8     0x2d 0x1064     
0x2e 0x1010     0x2f 0x0fbc     0x30 0x0f64     0x31 0x0f0c     0x32 0x0eb8     
0x33 0x0e60     0x34 0x0e0c     0x35 0x0db8     0x36 0x0d64     0x37 0x0d0c     
0x38 0x0cb4     0x39 0x0c60     0x3a 0x0c08     0x3b 0x0bb0     0x3c 0x0b58     
0x3d 0x0b04     0x3e 0x0aac     0x3f 0x0a58     0x40 0x0a04     0x41 0x09ac     
0x42 0x0954     0x43 0x08fc     0x44 0x08a4     0x45 0x0850     0x46 0x07f8     
0x47 0x07a0     0x48 0x074c     0x49 0x06f4     0x4a 0x06a0     0x4b 0x0648     
0x4c 0x05f0     0x4d 0x0598     0x4e 0x0540     0x4f 0x04e8     0x50 0x0490     
0x51 0x0438     0x52 0x03e0     0x53 0x0388     0x54 0x0334     0x55 0x02dc     
0x56 0x0284     0x57 0x022c     0x58 0x01d4     0x59 0x0180     0x5a 0x012c     
0x5b 0x00d4     
 
*-----------------------------

irb points to the UNDO RECORD of 0x1.

*-----------------------------
* Rec #0x1  slt: 0x26  objn: 125212(0x0001e91c)  objd: 564548  tblspc: 10(0x0000000a)
*       Layer:  10 (Index)   opc: 22   rci 0x00   
Undo type:  Regular undo    User Undo Applied  Last buffer split:  No 
Temp Object:  No 
Tablespace Undo:  No 
rdba: 0x4f81d3d1
*-----------------------------
...

rci of UNDO RECORD of 0x1 is 0x00. That means this is the first and last UNDO RECORD.

Object ID in this UNDO RECORD is 125212.

SQL> select owner,object_name,object_type from dba_objects where object_id in (125213,125212);

OWNER
------------------------------
OBJECT_NAME
--------------------------------------------------------------------------------
OBJECT_TYPE
-------------------
OWBRUN
SM_POST_IND1
INDEX

It's another lucky object that its type is INDEX. This index dropped. Now, after being sure that there is no new active transactions in this UNDO segment, the followings were done:

  • Shutdown the database
  • Set the following parameter to PFILE/SPFILE:

_smu_debug_mode=4
_offline_rollback_segments=(_SYSSMU85$)

  • Startup the database
  • drop rollback segment "_SYSSMU85$";

After UNDO segment is successfuly dropped, the INTERNAL parameters above should be removed. But, in our case, while dropping UNDO segment, although the current internal error (ORA-600) [kturrur11]) disappeared; the another internal error (ORA-600 [kddummy_blkchk]) was encountered. It's created as another issue as QA-34.

Since all objects needing recovery in the UNDO segment were dropped, there is no need to re-create the database after using _offline_rollback_segments parameter.





[QA-34] ORA-00600 [kddummy_blkchk] while dropping UNDO segment. Created: 08/Nov/07  Updated: 08/Nov/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Blocker
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
Operating System: IBM-AIX

 Description   
While dropping an offlined UNDO segment (by _offline_rollback_segments), the following error appeared:
SQL> drop rollback segment "_SYSSMU85$";
>
> drop rollback segment "_SYSSMU85$"
> *
> ERROR at line 1:
> ORA-00607: Internal error occurred while making a change to a data block
> ORA-00600: internal error code, arguments: [kddummy_blkchk], [2], [846985],
> [38508], [], [], [], []

Then, the instance crashed. After re-starting the instance, it crashed again.

Stack trace:

ORA-00600: internal error code, arguments: [kddummy_blkchk], [2], [846985], [38508], [], [], [], []
----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
ksedst+001c          bl       ksedst1              FFFFFFFFFFF9D10 ? 000000000 ?
ksedmp+0290          bl       ksedst               1047C9C10 ?
ksfdmp+0018          bl       03F53584             
kgerinv+00dc         bl       _ptrgl               
kseinpre+0040        bl       kgerinv              110040AA0 ? 000000000 ?
                                                   1048470A0 ? 07FFFFFFF ?
                                                   000000000 ?
ksesin+0048          bl       kseinpre             1048470A0 ? 07FFFFFFF ?
                                                   000000000 ?
kco_blkchk+0778      bl       ksesin               10484752C ? 300000003 ?
                                                   000000000 ? 000000002 ?
                                                   000000000 ? 0000CEC89 ?
                                                   000000000 ? 00000966C ?
kcoapl+0d24          bl       kco_blkchk           FFF00FFFFFFA310 ?
                                                   284422800B4E4358 ?
                                                   102FD4FDC ? 7000001F5151F50 ?
                                                   000000080 ?
kcbapl+0178          bl       kcoapl               FFFFFFFFFFFC218 ?
                                                   7000001E815A000 ? 100000001 ?
                                                   7FFFFFFF000000F7 ?
                                                   200000000000 ? 20BD260C8 ?
                                                   000000000 ?
kcrfw_redo_gen+2964  bl       kcbapl               000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ?
kcbchg1_main+25e0    bl       kcrfw_redo_gen       102DA5AC7 ?
                                                   2D30491F0A2889F8 ?
                                                   FFFFFFFFFFFAB20 ?
                                                   700000010008000 ? 1000024A4 ?
                                                   000000001 ? 400000000000001 ?
                                                   000000000 ?
kcbchg1+038c         bl       kcbchg1_main         000000000 ? 0000001F4 ?
                                                   000000000 ? 110366678 ?
                                                   0000023A0 ? 70000020B29AFD8 ?
ktbchgro+0380        bl       kcbchg1              00A288AD0 ? 30A2889FA ?
                                                   FFFFFFFFFFFB620 ?
                                                   FFFFFFFFFFFB658 ? 000000000 ?
                                                   000000000 ?
ktfbapp+0044         bl       ktbchgro             000000000 ? 300000003 ?
                                                   FFFFFFFFFFFCB48 ?
                                                   FFFFFFFFFFFC218 ?
                                                   FFFFFFFFFFFBFD8 ?
                                                   FFFFFFFFFFFC0B0 ?
                                                   FFFFFFFFFFFC4D8 ?
                                                   FFFFFFFFFFFC5B0 ?
kteopgen+00ec        bl       ktfbapp              000000000 ? FFFFFFFFFFFC218 ?
                                                   044244040 ? 0FFFFFFFF ?
                                                   FFFFFFFFFFFBFD8 ?
kteopdelete+1468     bl       kteopgen             FFFFFFFFFFFCB48 ? 000000000 ?
                                                   FFFFFFFFFFFBFD8 ?
                                                   FFFFFFFFFFFC140 ?
                                                   FFFFFFFFFFFC218 ?
                                                   FFFFFFFFFFFC0B0 ? 000000000 ?
                                                   1101EBDCC ?
ktsxfastdele+0118    bl       kteopdelete          700000209E9B238 ? 100000001 ?
                                                   100B5A770 ? 000000000 ?
                                                   FFFFFFFFFFFC270 ? 000000000 ?
                                                   000000000 ?
kteopshrink+0308     bl       01FC21A0             
ktssdrbm_segment+0a  bl       kteopshrink          100000001 ? FFFFFFFFFFFCAD8 ?
f8                                                 000000001 ? 000000001 ?
                                                   0000001A0 ? 700000200C016A0 ?
                                                   000000000 ?
ktssdro_segment+06c  bl       ktssdrbm_segment     FFFFFFFFFFFD498 ?
8                                                  FFFFFFFFFFFD560 ? 100008043 ?
                                                   1FFFFFFFF ?
ktssdt_segs+0350     bl       ktssdro_segment      70000020A3C52F0 ? 600007530 ?
                                                   0001DCE78 ?
ktmmon+1048          bl       ktssdt_segs          000000000 ?
                                                   7FFFFFFF7FFFFFFF ?
                                                   7FFFFFFF7FFFFFFF ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ?
                                                   7FFFFFFC7FFFFFFC ?
                                                   0472CD5BE ?
ktmSmonMain+0030     bl       ktmmon               000000000 ?
ksbrdp+03e0          bl       _ptrgl               
opirip+03fc          bl       01FC66A0             
opidrv+0448          bl       opirip               1103BD070 ? 4103BE990 ?
                                                   FFFFFFFFFFFF860 ?
sou2o+0090           bl       opidrv               32023373FC ? 400000020 ?
                                                   FFFFFFFFFFFF860 ?
opimai_real+0150     bl       01FC0DF4             
main+0098            bl       opimai_real          000000000 ? 000000000 ?
__start+0090         bl       main                 000000000 ? 000000000 ?

UNDO segment status:

SQL> select status$ from undo$ where us#=85;

   STATUS$
----------
         1

UNDO$ structure from $ORACLE_HOME/rdbms/admin/sql.bsq:

create table undo$                                     /* undo segment table */
( us#           number not null,                      /* undo segment number */
  name          varchar2("M_IDEN") not null,    /* name of this undo segment */
  user#         number not null,      /* owner: 0 = SYS(PRIVATE), 1 = PUBLIC */
  file#         number not null,               /* segment header file number */
  block#        number not null,              /* segment header block number */
  scnbas        number,           /* highest commit time in rollback segment */
  scnwrp        number,              /* scnbas - scn base, scnwrp - scn wrap */
  xactsqn       number,               /* highest transaction sequence number */
  undosqn       number,                /* highest undo block sequence number */
  inst#         number,    /* parallel server instance that owns the segment */
  status$       number not null,              /* segment status (see KTS.H): */
  /* 1 = INVALID, 2 = AVAILABLE, 3 = IN USE, 4 = OFFLINE, 5 = NEED RECOVERY,
   * 6 = PARTLY AVAILABLE (contains in-doubt txs)
   */
  ts#           number,                                 /* tablespace number */
  ugrp#         number,                      /* The undo group it belongs to */
  keep          number,
  optimal       number,
  flags         number,
  spare1        number,
  spare2        number,
  spare3        number,
  spare4        varchar2(1000),
  spare5        varchar2(1000),
  spare6        date
)

status$=1 means INVALID or DOES NOT EXIST. That means the UNDO segment doesn't exist.



 Comments   
Comment by ubTools Support [ 08/Nov/07 08:03 AM ]
Since the UNDO segment doesn't exist, the most probably its type is converted to TEMP. After setting the following event in the SPFILE/PFILE, the problem disappeared.
event="10061 trace name context forever, level 10" 

This event disables SMON from cleaning temp segment.

Comment by ubTools Support [ 08/Nov/07 12:39 PM ]
The current UNDO TABLESPACE was dropped, and a new one has been created. Then, Event 10061 has been removed.




[QA-29] ORA-600 [2845] while selecting, Invalid ROWID. Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: 7.3.4.5.0
Operating System: TRU64

 Description   
ORA-600 [2845] while selecting, Invalid ROWID.

 Comments   
Comment by ubTools Support [ 15/Jul/07 06:14 PM ]

Error code:

ORA-00600: internal error code, arguments: [2845], [0], [30], [0], [], [], [], []

Error code definition:

Oracle is reading a range of blocks from a database file.
If the starting block number or file number is 0, or the file number is  greater than can be accommodated in
the SGA (DB_FILES), error ORA-600 [2845] is raised.

Ref: Metalink Note: 31057.1 ORA-600 [2845] "Read of bad DBA Requested"

Cursor dump:

********************   Cursor Dump   ************************
Current cursor: 30, pgadep: 0
  pgactx: 14a415b8 ctxcbk: 0 ctxqbc: 14a41990 qbcrws: 14a40ef0
Cursor Dump:
----------------------------------------
...
----------------------------------------
Cursor 30 (1400ea8e0): CURFETCH  curiob: 14019be40
curflg: 46 curpar: 0 curusr: c curses 8a2f38
cursor name: SELECT "NOTE" FROM "MEDIX"."PAT_SES" WHERE "ROWID"=:1
child pin: 5485b30, child lock: 54f2960, parent lock: 54c5f68
xscflg: 80110676, parent handle: 14a46070
 nxt: 3.0x00000018  nxt: 2.0x000007d8  nxt: 1.0x000004e0
Cursor frame allocation dump:
frm: -------- Comment --------  Size  Seg Off
bind 0: dty=1 mxl=32(18) mal=00 scl=00 pre=00 oacflg=01
  bfp=140192748 bln=18 avl=18 flg=05
  value="00000000.0000.0000"
----------------------------------------
...

Problem explanation:

As seen in the cursor dump above, the current cursor number is 30. The cursor#30 has a bind variable using a ROWID. But, the value of this bind variable is "00000000.0000.0000". In Oracle7, this ROWID points to block#0, slot#0, file#0. This is wrong.

Recommendation:

  • Check application against any possible wrong ROWID usage.
  • Call Oracle Support.




[QA-28] ORA-00600 [729]: UGA memory leak. Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: 8.1.7.3.0
Operating System: Windows
Operating System Version: 2000

 Description   
ORA-00600 [729]: UGA memory leak.

 Comments   
Comment by ubTools Support [ 15/Jul/07 06:04 PM ]

Error code:

ORA-00600: internal error code, arguments: [729], [480], [space leak], [], [], [], [], []

Error code definition:

A space leak has been detected in the User Global Area (UGA). There is no data corruption as a result of this error. It is an internal memory housekeeping problem. Second argument is the number of bytes leaked.

UGA Heap dump:

******** ERROR: UGA memory leak detected 480 ********
******************************************************
HEAP DUMP heap name="session heap"  desc=0x222bd6f4
extent sz=0x108c alt=32767 het=32767 rec=0 flg=3 opc=3
parent=212550 owner=ad83d50 nex=0 xsz=0x108c
EXTENT 0
 Chunk 2330b100 sz=     3844    free      "               "
EXTENT 1
 Chunk 232f5174 sz=      516    free      "               "
EXTENT 2
 Chunk 236f0050 sz=     4176    free      "               "
EXTENT 3
 Chunk 236d0050 sz=     1228    free      "               "
EXTENT 4
 Chunk 236f18e4 sz=     1280    free      "               "
EXTENT 5
 Chunk 23307098 sz=     4228    free      "               "
EXTENT 6
 Chunk 2330a27c sz=     3696    free      "               "
EXTENT 7
 Chunk 23308130 sz=     1008    free      "               "
 Chunk 23308520 sz=      480    freeable  "define var info"
 Chunk 23308700 sz=     2740    free      "               "
EXTENT 8
 Chunk 23306214 sz=     2832    perm      "perm           "  alo=2832
 Chunk 23306d24 sz=      864    free      "               "
EXTENT 9
 Chunk 233091c8 sz=     4228    free      "               "
EXTENT 10
 Chunk 232f405c sz=      612    free      "               "
EXTENT 11
...

Problem explanation:

As seen in the UGA heap dump, there is a freeable chunk of define var info memory type. This chunk looks leaked.

Workaround:

There is no data corruption in this error, and can be safely ignore for small memory leaks by adding the following event to init.ora:

  • event = "10262 trace name context forever, level 500"

Then, restart your database. This event disables space leaks less than 500 bytes.

You can see the details at Metalink Note:31056.1 ORA-600 [729] "UGA Space Leak"

Bug:

Bug:2177050: ORA-600 [729] after application of the 8.1.7.3 patchset. The resulting trace file will include a memory dump which shows unfreed memory chunks with the tags "define var info" and/or "oactoid info".
Ref: Metalink Note:31056.1 ORA-600 [729] "UGA Space Leak"





[QA-27] ORA-00600 [kcbgcur_1] by PQ operation. Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: 8.1.6.1.0
Operating System: Linux
Operating System Version: 2.2.14-5.0

 Description   
ORA-00600 [kcbgcur_1] by PQ operation.

 Comments   
Comment by ubTools Support [ 15/Jul/07 05:59 PM ]

Error code:

ORA-00600: internal error code, arguments: [kcbgcur_1], [], [], [], [], [], [], []

Oracle kernel function from which the problem is raised:

kcbgcur().

This function is a function of Oracle Cache Layer.

Undo block dump:

UNDO BLK:  
xid: 0x0005.05e.000000c4  seq: 0x8c  cnt: 0x31  irb: 0x19  icl: 0x0   flg: 0x0000

Rec Offset      Rec Offset      Rec Offset      Rec Offset      Rec Offset
---------------------------------------------------------------------------
0x01 0x1f38     0x02 0x1e88     0x03 0x1de4     0x04 0x1d3c     0x05 0x1c94    
0x06 0x1bf4     0x07 0x1b54     0x08 0x1ac4     0x09 0x1a20     0x0a 0x1978    
0x0b 0x18d4     0x0c 0x1820     0x0d 0x1784     0x0e 0x16e0     0x0f 0x1638    
0x10 0x1598     0x11 0x14e8     0x12 0x1448     0x13 0x13a4     0x14 0x1308    
0x15 0x126c     0x16 0x11d0     0x17 0x112c     0x18 0x1084     0x19 0x0fe0    
0x1a 0x0f0c     0x1b 0x0e60     0x1c 0x0db8     0x1d 0x0d28     0x1e 0x0c90    
0x1f 0x0bf0     0x20 0x0b28     0x21 0x0a88     0x22 0x09ec     0x23 0x0950    
0x24 0x08ac     0x25 0x0814     0x26 0x077c     0x27 0x06e4     0x28 0x0650    
0x29 0x05b4     0x2a 0x0524     0x2b 0x0480     0x2c 0x03f4     0x2d 0x035c    
0x2e 0x02c0     0x2f 0x0230     0x30 0x01a0     0x31 0x0108
...
*-----------------------------
* Rec #0x19  slt: 0x5e  objn: 0(0x00000000)  objd: 0  tblspc: 0(0x00000000)
*       Layer:  11 (Row)   opc: 1   rci 0x18  
Undo type:  Regular undo   Last buffer split:  No
Temp Object:  No
Tablespace Undo:  No
rdba: 0x00000000
*-----------------------------
KDO undo record:
KTB Redo
op: 0x02  ver: 0x01  
op: C  uba: 0x00c0083d.008c.18
KDO Op code: IRP  xtype: XA  bdba: 0x0040760a  hdba: 0x004075d9
itli: 1  ispac: 0  maxfr: 4863
tabn: 0 slot: 130(0x82) size/delt: 56
fb: --H-FL-- lb: 0x0 cc: 4
null: ----
col  0: [ 3]  37 34 34
col  1: [20]  45 6c 65 63 74 72 6f 6e 69 63 20 73 74 72 75 63 74 75 72 65
col  2: [ 0]
col  3: [ 0]
*-----------------------------
...

Problem explanation:

irb points the first undo record in undo block to begin rollback. So, the record 0x19 is your first undo record. The object number of the block, and the object number of the block undo applied to are 0. I think this may be your problem. Oracle may not be able to know the real object number during this rollback.

Bug:

It looks like:

  • Bug:984947 A PARALLEL QUERY SLAVE GOT ORA-600[KCBGCUR_1]




[QA-26] ORA-00600 [12700] by SNP process. Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: 8.0.5.2.1
Operating System: Windows

 Description   
ORA-00600 [12700] by SNP process.

 Comments   
Comment by ubTools Support [ 15/Jul/07 05:54 PM ]

Error code:

ORA-00600: internal error code, arguments: [12700], [62], [4202128], [133], [], [], [], []

Current SQL statement for this session:

SELECT source from source$ WHERE obj# =:1 ORDER BY line

Oracle kernel function from which the problem is raised:

rtbhiopn().

Error code definition:

Oracle is trying to access a row using its ROWID, which has been obtained from an index.

A mismatch was found between the index rowid and the data block it is   pointing to.
The rowid points to a non-existent row in the data block.
The corruption can be in data and/or index blocks.

ORA-600 [12700] can also be reported due to a consistent read (CR) problem.

The information dumped to the trace file varies greatly between releases:

- in Oracle 7.3.x it is ORA-600 [12700][a1][a2] , where
 Arg [a1] dba (Data Block Address)
 Arg [a2] slot number (number of the row in the block pointed by the dba)

- in Oracle 8.x and 9.x, it is ORA-600 [12700][a1][a2][a3] , where
 Arg [a1] dataobj# from sys.obj$
 Arg [a2] relative dba of the data block
 Arg [a3] slot number of the row in the data block  

Details: Metalink Note:28229.1 ORA-600 [12700] "Index entry Points to Missing ROWID"

Error code interpretation:

Argument             Dec              Hex
----------    ----------    ----------
        [62]              62             0x3E
[4202128]      4202128       0x401E90
      [133]             133             0x85

This problem is related to the slot#133 of the rdba#4202128 of the object#62.

Index block dump:

Block header dump: rdba: 0x00401ede
Object id on Block? Y
seg/obj: 0x63  csc: 0x00.2fbe43  itc: 2  flg: -  typ: 2 - INDEX
    fsl: 0  fnx: 0x0 ver: 0x01

Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x0012.01a.00000130  0x008027fb.000e.02  C---    0  scn 0x0000.0000c71c
0x02   0x0002.013.00000768  0x00807cf2.4f11.08  --U-  217  fsc 0x0000.002fbe45

Leaf block dump
===============
header address 74698844=0x473d05c
kdxcolev 0
kdxcolok 0
kdxcoopc 0x80: opcode=0: iot flags=--- is converted=Y
kdxconco 2
kdxcosdc 1
kdxconro 217
kdxcofbo 470=0x1d6
kdxcofeo 471=0x1d7
kdxcoavs 1
kdxlespl 0
kdxlende 0
kdxlenxt 4202207=0x401edf
kdxleprv 4203488=0x4023e0
kdxledsz 6
kdxlecol 0
kdxlebksz 3940
row#0[471] flag: ----, lock: 2, data:(6):  00 40 1e 93 00 5b
col 0; len 3; (3):  c2 27 11
col 1; len 3; (3):  c2 02 62
...
row#213[3876] flag: ----, lock: 2, data:(6):  00 40 1e 90 00 85
col 0; len 3; (3):  c2 27 11
col 1; len 3; (3):  c2 05 0b
row#214[3892] flag: ----, lock: 2, data:(6):  00 40 1e 90 00 86
col 0; len 3; (3):  c2 27 11
col 1; len 3; (3):  c2 05 0c
...

Data block dump:

Block header dump: rdba: 0x00401e90
Object id on Block? Y
seg/obj: 0x3e  csc: 0x00.2fbe43  itc: 1  flg: -  typ: 1 - DATA
    fsl: 0  fnx: 0x0 ver: 0x01

Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x0002.013.00000768  0x00807cf2.4f11.0b  --U-  139  fsc 0x009b.002fbe45

data_block_dump
===============
tsiz: 0xfb8
hsiz: 0x128
pbl: 0x04b43044
bdba: 0x00401e90
flag=---------
ntab=1
nrow=139
frre=-1
fsbo=0x128
fseo=0x2d4
avsp=0x111
tosp=0x23a
0xe:pti[0] nrow=139 offs=0
0x12:pri[0] offs=0xfb6
0x14:pri[1] offs=0xfb4
0x16:pri[2] offs=0xfb2
.
0x11a:pri[132] offs=0x382
0x11c:pri[133] sfll=0
0x11e:pri[134] sfll=0
0x120:pri[135] sfll=0
0x122:pri[136] sfll=0
0x124:pri[137] sfll=0
0x126:pri[138] sfll=0
block_row_dump:
tab 0, row 0, @0xfb6
tl: 2 fb: --HDFL-- lb: 0x1
tab 0, row 1, @0xfb4
tl: 2 fb: --HDFL-- lb: 0x1
.
tab 0, row 132, @0x382
tl: 42 fb: --H-FL-- lb: 0x1 cc: 3
col  0: [ 3]  c2 27 11
col  1: [ 3]  c2 05 0a
col  2: [30]
09 09 09 09 09 09 6c 5f 6e 65 78 74 48 6f 6c 64 44 65 73 69 72 65 4e 75 6d
62 65 72 2c 0a
end_of_block_dump

Problem explanation:

As seen in the index block dump, kdxledsz is 6. That means this index is a unique B*Tree index which uses restricted ROWID format in 6 bytes. The first 4 bytes are used for rdba, and the last 2 bytes are used for slot#.

This internal error code had returned 0x401E90 for the rdba, and 0x85 for the slot#. The restricted ROWID in the index dump has to be the combination of them. So, it's 0x00401E900085. This restricted ROWID is available in the index dump.

The pri[] field shows slot# of rows in data block. In this error, the returned slot# is 133. But, as seen in the data block dump, there is no row allocated for this slot. The max slot# in the block dump is 132.

Although there is a value in the index block, there is no matching row in the data block. The data block looks corrupted.

Workaround:

The most probably the object#62 is source$. Restore SYSTEM tablespace from the backup, and recover it.





[QA-25] ORA-00600 [kkslgop1] in SELECT when CURSOR_SHARING IS NOT EXACT. Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: 8.1.7.2.0
Operating System: IBM-AIX

 Description   
ORA-00600 [kkslgop1] in SELECT when CURSOR_SHARING IS NOT EXACT.

 Comments   
Comment by ubTools Support [ 15/Jul/07 05:42 PM ]

Error code:

ORA-00600: internal error code, arguments: [kkslgop1], [], [], [], [], [], [], []

Current SQL statement for this session:

SELECT COMP_TIME FROM CSTMAPSTATUS  WHERE CSTID = :"SYS_B_0" AND  SLOTNO = :"SYS_B_1"

Oracle kernel function from which the problem is raised:

kkslgop().

This is a function of Oracle Compilation Layer.

Process state:

PROCESS STATE
-------------
...
   ----------------------------------------
   SO: 404c6264, type: 3, owner: 403cde98, pt: 0, flag: INIT/-/-/0x00
   (session) trans: 40e83928, creator: 403cde98, flag: (8000041) USR/- BSY/-/-/-/-/-
             DID: 0001-0014-00000002, short-term DID: 0000-0000-00000000
             txn branch: 40f8201c
             oct: 3, prv: 0, user: 24/APPMGR
   O/S info: user: Administrator, term: CIMMB, ospid: 219:228, machine: PDP1_MES_DOM\CIMMB
             program: TIME_GAP.exe
   last wait for 'SQL*Net message from dblink' blocking sess=0x0 seq=60687 wait_time=-1
               driver id=54435000, #bytes=1, =0
     ----------------------------------------
...

Problem explanation:

As you see in your SQL statement, your bind variables are system generated bind variables. In other words, cursor sharing is enabled in your database.

Also, as seen in the process state, your last wait event is SQL*Net message from dblink. That means a dblink operation had been done before.

Workaround:

Use cursor_sharing=exact

Bug:

  • Bug:2169897 ORA-600 ARGUMENTS: [KKSLGOP1] VIA SELECT ACROSS DB_LINK
  • Bug:2159152 CURSOR_SHARING=FORCE MAY NOT SHARE STATEMENTS USING VIEWS IN 8172/8173
    Back to top




[QA-24] ORA-07445 [000000010112A75C] during import. Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: 8.1.7.2.0
Operating System: Solaris
Operating System Version: 5.8

 Description   

Error code:

ORA-07445: exception encountered: core dump [000000010112A75C] [SIGSEGV] [Address not mapped to object] [260] [] []

Current SQL statement for this session:

CREATE PROCEDURE TableParse_Proc wrapped
0
abcd
abcd
abcd
abcd
abcd
..

Oracle kernel function from which the problem is raised:

parfs4_freelist_sort()

Process state:

PROCESS STATE
-------------
...
   ----------------------------------------
   SO: 399071ce0, type: 3, owner: 39905dc18, pt: 0, flag: INIT/-/-/0x00
   (session) trans: 399e4f3b8, creator: 39905dc18, flag: (10000041) USR/- BSY/-/-/-/-/-
             DID: 0001-0008-00000002, short-term DID: 0000-0000-00000000
             txn branch: 0
             oct: 24, prv: 0, user: 360/WINRLS
   O/S info: user: mtrxdev, term: pts/2, ospid: 8915, machine: vhdcap5g
             program: imp@vhdcap5g (TNS V1-V3)
   last wait for 'SQL*Net more data from client' blocking sess=0x0 seq=45627 wait_time=-2
               driver id=62657100, #bytes=2882, =0
     ----------------------------------------

Problem explanation:

As seen above, this problem was encountered in import while creating a wrapped package.

Bug:

There are several bugs about this problem with additional ORA-4030 error. The base bug is below:

  • Bug:2278310 IMPORT OF WRAPPED PL/SQL PROCEDURE FAILS WITH ORA-04030





[QA-23] ORA-00600 [15851] while creating unique index. Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: 7.3.4.0.0
Operating System: Windows
Operating System Version: 4.0

 Description   
ORA-00600 [15851] while creating unique index.

 Comments   
Comment by ubTools Support [ 15/Jul/07 05:31 PM ]

Error code:

ORA-00600: internal error code, arguments: [15851], [8], [8], [1], [2], [], [], []

Oracle kernel function from which the problem is raised:

srsqb1nx().

Problem explanation:

Most probably, this is a sort problem while creating index.

Bug:

Metalink Note:1032586.6 ORA-600 [15851]





[QA-22] ORA-00600 [13004] while creating index. Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: 7.3.4.0.0
Operating System: Windows
Operating System Version: 4.0

 Description   
ORA-00600 [13004] while creating index.

 Comments   
Comment by ubTools Support [ 15/Jul/07 05:27 PM ]

Error code:

ORA-00600: internal error code, arguments: [13004], [], [], [], [], [], [], []

Oracle kernel function from which the problem is raised:

kkrirop().

This is a function of Oracle Compilation Layer.

Bug:

Bug:994802 CREATE INDEX RESULTS IN ORA-600 [13004]





[QA-21] ORA-07445 [11]: SMON crashed. Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: 8.1.7.1.0
Operating System: HP-UX
Operating System Version: B.11.00

 Description   
ORA-07445 [11]: SMON crashed.

 Comments   
Comment by ubTools Support [ 15/Jul/07 05:23 PM ]

Error code:

 
ORA-07445: exception encountered: core dump [11] [3221212616] [240] [0] [] []

Oracle kernel function from which the problem is raised:

 
kdb4_dup_keys().

This is a function of Oracle Data Layer.

Cursor dump:

 
********************   Cursor Dump   ************************
Current cursor: 1, pgadep: 1
  pgactx: c00000014e736d90 ctxcbk: c00000014e776720 ctxqbc: 0 ctxrws: c00000014e7253c8
Cursor Dump:
----------------------------------------
Cursor 1 (80000001000befe8): CURBOUND  curiob: 80000001000c1358
curflg: 5 curpar: 0 curusr: 0 curses c00000012c18a070
cursor name: delete from uet$ where ts#=:1 and segfile#=:2 and segblock#=:3 and ext#=:4
child pin: c00000013511e670, child lock: c00000013511d630, parent lock: c00000013511d6a0
xscflg: 20100466, parent handle: c00000014e748b20, xscfl2: 5100400
 nxt: 3.0x00000560  nxt: 2.0x000005e0  nxt: 1.0x000005e0
Cursor frame allocation dump:
frm: -------- Comment --------  Size  Seg Off
bind 0: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=08 oacfl2=1 size=24 offset=0
  bfp=80000001000d2470 bln=22 avl=02 flg=05
  value=1
bind 1: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=08 oacfl2=1 size=24 offset=0
  bfp=80000001000d2440 bln=24 avl=02 flg=05
  value=2
bind 2: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=08 oacfl2=1 size=24 offset=0
  bfp=80000001000d2410 bln=24 avl=02 flg=05
  value=2
bind 3: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=08 oacfl2=1 size=24 offset=0
  bfp=80000001000d23e0 bln=24 avl=02 flg=05
  value=59
End of cursor dump

Recomendation:

Check if sys.uet$ is corrupted.

Bug:

Bug:2106455 SMON CRASHES WITH ORA-07445 IN KDB4_DUP_KEYS





[QA-20] ORA-00600 [723]: Memory leak in LGWR. Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: 8.1.6.0.0
Operating System: HP-UX
Operating System Version: B.11.00

 Description   
ORA-00600 [723]: Memory leak in LGWR.

 Comments   
Comment by ubTools Support [ 15/Jul/07 05:17 PM ]

Error code:

 
ORA-00600: internal error code, arguments: [723], [5200], [5200], [memory leak], [], [], [], []

Oracle kernel function from which the problem is raised:

 
ksmdpg()

Deallocate variable PGA. Just free top PGA heap, the callback will free. the extents to the OSD.
Ref: Bug:1283286

Process state:

 
PROCESS STATE
-------------
Process global information:
    process: 0, call: 0, xact: 0, curses: 0, usrses: 0
No process is allocated.
END OF PROCESS STATE

PGA Heap dump:

 

******** ERROR: PGA memory leak detected 5200 > 3616 ********
******************************************************
HEAP DUMP heap name="pga heap"  desc=0x40003190
extent sz=0x2148 alt=40 het=32767 rec=0 flg=3 opc=3
parent=0 owner=0 nex=0 xsz=0x2148
EXTENT 0
 Chunk 400de8e0 sz=     4432    free      "               "
 Chunk 400dfa30 sz=      256    freeable  "LGWR PIC bds ar"
 Chunk 400dfb30 sz=      896    freeable  "LGWR PIC ins ar"
 Chunk 400dfeb0 sz=      896    freeable  "LGWR PIC ins ar"
 Chunk 400e0230 sz=      568    free      "               "
 Chunk 400e0468 sz=      896    freeable  "LGWR PIC ins ar"
 Chunk 400e07e8 sz=      568    free      "               "
EXTENT 1
.

Problem explanation:

As seen above and included in your trace, the memory class of some chunks are "LGWR PIC ins ar" and similar. If you notice that sum of them is 5200 bytes, and they are freeable chunks. These chunks are leaked.

Also, there is no allocated process for LGWR. The most probably, you are closing the database.

Workaround:

There is no data corruption in this error, and can be safely ignore for small memory leaks by adding the following event to init.ora:

  • event = "10262 trace name context forever, level 6000"

Then, restart your database. This event disables space leaks less than 6000 bytes.

You can see the details at Metalink Note:39308.1 ORA-600 [723] "PGA memory leak"

Bug:

Bug:1125724 ORA-600[723] DURING SHUTDOWN





[QA-19] ORA-00600 [2845] in UPDATE. WRONG ROWID VALUE. Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: 7.3.2.3.0
Operating System: HP-UX
Operating System Version: B.10.20

 Description   

Error code:

 
ORA-00600: internal error code, arguments: [2845], [0], [50], [39314], [], [], [], []

Current SQL statement for this session:

 
update pers_auth_str_tbl  set asgn_str=:b1 where rowid=:b2

Oracle kernel function from which the problem is raised:

 
kcfrbd()

This funtion is a funtion of Oracle's Cache Layer.

Values of bind variables:

 
:b1 = 0
:b2 = "9992"

Data types of bind variables:

 
:b1 : Number
:b2 : Varchar2

Problem explanation:

As you see, your ROWID value in :b2 is "9992". This is incorrect. ROWID format in Oracle7 is 'BBBBBBBB.SSSS.FFFF' (Block.Slot.File).

Bug:

Check Bug#632396. This bug says:

  • The correct behaviour is to return an "Invalid Rowid" message.

Recomendation:

Use proper datatype in the bind variable.






[QA-18] ORA-00600 [6033] in SELECT. Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: 8.1.7.0.0
Operating System: Solaris
Operating System Version: 5.8

 Description   

Error code:

 
ORA-00600: internal error code, arguments: [6033], [], [], [], [], [], [], []

Current SQL statement for this session:

 
SELECT *   FROM CSD_BOUNCE_CONTENT_BODY  WHERE BOUNCE_CONTENT_ID = :b1

Oracle kernel function from which the problem is raised:

 
kdifxs()

This is a function of Oracle Data Layer and responsible for fetching a row in an index scan.

Leaf block dump:

 
Leaf block dump
===============
header address 2567438500=0x990800a4
kdxcolev 0
kdxcolok 0
kdxcoopc 0xa0: opcode=0: iot flags=-C- is converted=Y
kdxconco 2
kdxcosdc 0
kdxconro 663
kdxcofbo 1410=0x582
kdxcofeo 4421=0x1145
kdxcoavs 8687
kdxlespl 0
kdxlende 0
kdxlenxt 0=0x0
kdxleprv 62923132=0x3c0217c
kdxledsz 0
kdxlebksz 16152
kdxlepnro 11
kdxlepnco 1
...

This is an index object# 0x1ec41. As seen above, kdxcoopc is 0xa0. That means, this index is a key compressed V8 B*Tree index. Also, kdxledsz is 0. In other words, this index is a non-unique index.

Recommendations:

Check your table by the following statement against any possible corruption:

  • SQL > analyze table CSD_BOUNCE_CONTENT_BODY validate structure cascade;

If no corruption is detected, please see the following bugs:

  • ORA-600 [6033] DURING WORK FLOW ORDER IMPORT PROCESS
  • ORA-600 [711], [1], [0X2EDFE84] [KDIFXS - PREFIX CONTEXT] WITH COMPRESSED INDEX





[QA-11] How to see the tasks of Oracle background processes ? Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: 10.1.0.3.0
Operating System: Generic

 Description   
How to see the tasks of Oracle background processes ?

 Comments   
Comment by ubTools Support [ 15/Jul/07 01:29 PM ]

Answer:

Use the following query:

select substr(DEST,1,10) DEST, DESCRIPTION from x$messages order by DEST;

 
DEST DESCRIPTION

---------- ----------------------------------------------------------------

* Monitor Cleanup

* KSB action for X-instance calls

* generic shutdown background

* Scumnt mount lock

* database close in progress

* Poll system events broadcast channel

* svr actn for shrd grp reg/dereg

ARB* ASM to slave BG msg

ARC* Archiver wakeup

ARCH Archiver Message

ARCH Archiver shutdown


DEST DESCRIPTION

---------- ----------------------------------------------------------------

CJQ* Shutdown Job Queue Process

CJQ* Job Queue Interupt

CJQ* Job Queue Interupt

CJQ* Job Queue Interupt

CJQ* Job Queue Timout

CJQ0 Check for async messages from other instances

CJQ0 Coordinator send broadcast timeout

CKPT create/scrub cmon foregrounds

CKPT perform RM action in CKPT

CKPT identify control file

CKPT close control file


DEST DESCRIPTION

---------- ----------------------------------------------------------------

CKPT release (XR,4,0) enqueue

CKPT CKPT stat update timeout action

CKPT CKPT reuse call completion action

CKPT CKPT reuse range call continuation

CKPT CKPT reuse call continuation

CKPT refresh control file

CKPT check for parameters from other instances

CKPT start background

CKPT CPU dynamic reconfiguration

CKPT check for quiesce messages

CKPT unquiesce the instance during database close


DEST DESCRIPTION

---------- ----------------------------------------------------------------

CKPT unsubscribe to quiesce channel

CKPT subscribe to quiesce channel

CKPT Get Proxy Lock

CKPT Db Checkpt Compl check

CKPT Db Checkpt Request check

CKPT update recovery-based i/o statistics

CKPT Compile Environment Monitor

CKPT SQL Memory Management Calculation

CKPT free PX memory chunks in background

CKPT KKX: drop ncomp dll action

CKPT Flashback barrier


DEST DESCRIPTION

---------- ----------------------------------------------------------------

CKPT hold alert level

CKPT recovery area alert action

CKPT start change tracking in ckpt

CKPT get (XR,4,0) enqueue

CKPT sense a heartbeat

CKPT set heartbeat sensing

CKPT emulate i/o errors on a disk

CKPT timeout

CKPT Run self test on group

CKPT asynchronously dismount disk group

CKPT dismount disk group


DEST DESCRIPTION

---------- ----------------------------------------------------------------

CKPT query disk group status

CKPT check disk status

CKPT update disk status

CKPT update disk group status

CKPT kfc CKPT dismount disk group

CKPT kfc CKPT mount disk group

CTWR change tracking message

CTWR change tracking timeout action

DBW* hardware clock went backwards

DBW* DBWR write buffers

DBW* get/release open thread enqueue


DEST DESCRIPTION

---------- ----------------------------------------------------------------

DBW* mount/dismount all db files

DBW0 SGA memory tuning parameter update - DBW0

DBW0 Db mount lock

DBW0 kfcb Poke DBW0

DBW0 kfc mount disk group

DBW0 kfc dismount disk group

DBW0 kfc invalidate file extent

DBW0 Reserve lock name space lock

DBW0 Release lock name space lock

DBW0 complete Release space call

DBW0 verify/invalidate all db files


DEST DESCRIPTION

---------- ----------------------------------------------------------------

DBW0 recovery db file verification

DBW0 identify db file

DBW0 close and unlock db file

DBW0 lock db file

DBW0 offline db file

DBW0 Db File check

DBW0 Message to flush IMU txns

DBW0 Db Instance Lock Mgmt

DIAG write trace records out

DIAG Clusterwise dump request

DIAG poradebug commands


DEST DESCRIPTION

---------- ----------------------------------------------------------------

DIAG write trace records out

DIAG write trace records out

DIAG write trace records out

DMON DMON Wakeup

DMON DMON shutdown

DMON DMON Verify Standby shutdown for PM violation

DMON Standby site request resync

DMON Metadata file available

DMON DMON rcv NS status

DMON DMON Receive Message

DMON DMON Disable DRC


DEST DESCRIPTION

---------- ----------------------------------------------------------------

DMON DMON Interrupt Routine

INSV INSV Wakeup

INSV NetSlave Shutdown Message

INSV INSV Receive Message

LCK0 ksim LCK0 functions

LCK0 ksim reg/dereg instance group

LCK0 ksim query instance group

LCK0 ksim polling interrupt action

LCK0 KSXR remote instance died

LCK0 KSXR finialize

LCK0 kxfp signal recv function


DEST DESCRIPTION

---------- ----------------------------------------------------------------

LCK0 get and hold global enqueue

LCK0 perform a user instance lock operation

LCK0 SMON purge object number cache

LCK0 KQLM interrupt action

LCK0 KQLM invalidation instance lock operation

LCK0 KQLM pin instance lock operation

LCK0 KQR timeout action

LCK0 KQR get instance lock

LCK0 sequence bckgrnd instance lock

LCK0 release TS enq for sort segment

LCK0 kea signal recv function


DEST DESCRIPTION

---------- ----------------------------------------------------------------

LCK0 get TS enq for sort segment

LCK0 release quiesce enqueue

LCK0 get quiesce enqueue

LCK0 KCL lock affinity timeout action

LCK0 Check SCN adjust

LCK0 Cross-instance broadcast message

LCK0 ksim get value

LGWR LGWR failure

LGWR kfr ACD relocation

LGWR kfr Incr Ckpt

LGWR kfr Poke LGWR


DEST DESCRIPTION

---------- ----------------------------------------------------------------

LGWR kfr Dismount disk group

LGWR kfr mount disk group

LGWR LGWR to Start DMON

LGWR free KTU instance lock

LGWR convert KTU instance lock

LGWR get KTU instance lock

LGWR dml_locks = 0 global enforcement

LGWR Open/close/mount/dismount thread

LGWR Redo writer generate offline immed marker

LGWR Redo writer log switch operations

LGWR LGWR re-eval standby locks


DEST DESCRIPTION

---------- ----------------------------------------------------------------

LGWR Redo writer interrupt action

LGWR Redo writer IO's

LMD* Flush side-channel msgs LMD

LNS* Network Server wakeup

LNS* Network Server forced

LNS* Network Server shutdown

LNS* Network Server reinit

MMAN lock memory at startup

MMAN Memory Management

MMAN Handle sga_target resize

MMAN Reset advisory pool when advisory turned ON


DEST DESCRIPTION

---------- ----------------------------------------------------------------

MMAN Complete deferred initialization of components

MMAN lock memory timeout action

MMNL tune undo retention

MMNL MMNL Periodic MQL Selector

MMNL ASH Sampler (KEWA)

MMNL MMON SWRF Raw Metrics Capture

MMON reload failed KSPD callbacks

MMON SGA memory tuning

MMON background recovery area alert action

MMON Flashback Marker

MMON tablespace alert monitor


DEST DESCRIPTION

---------- ----------------------------------------------------------------

MMON UNDO MMON ACTION

MMON MMON Local action Listener

MMON MMON Remote action Listener

MMON Advisor delete expired tasks

MMON ASH Emergency Flusher (KEWA)

MMON MMON SWRF Auto DBFUS Task

MMON MMON SWRF Auto Purge Task

MMON MMON SWRF Auto Flush Task

MMON alert message purge

MMON alert message cleanup

MMON Check for sync messages from other instances


DEST DESCRIPTION

---------- ----------------------------------------------------------------

MMON ADDM (KEH)

MMON threshold reconciliation

MMON metrics monitoring

MMON shutdown MMON

MMON run-once action driver

MMON MMON testing slave

MMON MMON testing action

MMON MMON Completion Callback Dispatcher

MMON Job Autostart action force

MMON Coordinator autostart timeout

MMON Check for autostart messages from other instances


DEST DESCRIPTION

---------- ----------------------------------------------------------------

MMON Compute cache stats in background

MMON undo usage

MMON recovery area alert action

MMON SGA memory tuning parameter update

MMON reconfiguration MMON action

NSV* NetSlave Wakeup Message

NSV* NetSlave Receive Message

NSV* NetSlave Metadata Resync

NSV* NetSlave Health Check Message

NSV* NetSlave Shutdown Message

NSV* NetSlave request Primary to resync


DEST DESCRIPTION

---------- ----------------------------------------------------------------

NSV* NetSlave Check DRC version

QMNC Shutdown Q Monitor Coord

RBAL ASM to master BG msg

RBAL BG load lib msg

RBAL|SMON OSM to BG mesg

RECO distributed recovery wakeup

RECO distributed recovery shutdown

RSM* RSM Wakeup

RSM* RSM Receive Message

RSM* RSM Receive Message Response

RVWR Open/close flashback thread


DEST DESCRIPTION

---------- ----------------------------------------------------------------

RVWR RVWR IO's

SMON kfcl instance recovery

TEST Reliable Test Dummy Call


212 rows selected.


SQL>




[QA-8] Heapdump Interpretation Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: ???
Operating System: Generic

 Description   
I have a process which is taking up way more memory than I'd expected. The process runs a PL/SQL that does some nested loop joins on a PL/SQL table.

The background process is using > 200Mb of private memory and this number goes up if we tweak the WHERE clause in the join to return more data.

I did a heapdump of the process and the trace file looks like this (lots of stuff trimmed):

 
...
EXTENT 437
Chunk 925dfe4 sz= 1836 perm "perm " alo=1836
Chunk 925e710 sz= 1156 recreate "session heap " latch=0
ds 92693fc sz= 30315156 ct= 440
b7aa56c sz= 3980
92f30a0 sz= 1072
afb6e34 sz= 16472
afb2dcc sz= 16472
afaed64 sz= 16472
...

I presume that "session heap" is the UGA for this process'
session. Basically it goes on like this for several pages with sz anywhere between 16k and 1Mb. How can I interpret this? I presume the memory is to do with cursor information. This is a sort but the sort area size is only 10Mb and cannot account for all the private memory in use.

I'm just trying to decide if this is a reasonable amount of memory to be using (i.e. explain what it is using it for) and just put up with it, or if something has gone wrong. I'm on 8.1.5 on Linux 2.2 (I know, I know...)

Thanks for any insight!



 Comments   
Comment by ubTools Support [ 15/Jul/07 01:03 PM ]

Answer:

A heap consists of memory areas named extent. Each extent consists of memory areas named chunks.

Interpretation:

 
EXTENT 437
  Chunk 925dfe4 sz=     1836    perm      "perm           "
alo=1836
  Chunk  925e710 sz=     1156    recreate  "session heap   "
latch=0

EXTENT 437        ---> extent number
925dfe4           ----> chunk address
sz=               -----> size of chunk
perm              ------> permanent memory class
"perm           " ------> chunk comment

Memory classes can be the followings:

  • Recreatable (can be removed and then recreated when requested. i.e: shared SQL statements)
  • Free (free, no object in it)
  • Freeable(used in session/call duration)
  • Permanent(for permament objects)

Each chunk in same extent is contiguous. For your case, the first chunk address(0x925dfe4) + its size(1836) = the second chunk address (0x925e710)

For your problem:

Shared memory segments such as SGA are included in process address space. So, You may be encoutering this problem. Search metalink for pmap command.





[QA-7] _TRACE_FILES_PUBLIC parameter Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: ???
Operating System: Generic

 Description   

Parameter:

 
Name.................: _TRACE_FILES_PUBLIC
Values...............: TRUE/FALSE
Default value........: FALSE
Initial Release......: ?
Scope................: Instance

Explanation:

Trace files are not created with read permission by default for non-dba groups. Here is a sample on Linux:

 
$ ls -ltr
total 4
-rw-r-----    1 oracle   oinstall     2146 Jan  6 11:37 linkplus_ora_18653.trc

With _TRACE_FILES_PUBLIC=TRUE, other groups can read trace files.

 
$ ls -ltr
total 8
-rw-r--r--    1 oracle   oinstall     2742 Jan  6 12:00 linkplus_ora_18759.trc

Warning:

Setting this parameter to TRUE should be done for trusted users since trace files may include security data in BIND variables.






[QA-6] _OPTIM_PEEK_USER_BINDS parameter Created: 15/Jul/07  Updated: 16/Sep/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: 9.0.1
Operating System: Generic

 Description   

Parameter:

Name.................: _OPTIM_PEEK_USER_BINDS
Values...............: TRUE/FALSE
Default value........: TRUE
Initial Release......: 9.0.1
Scope................: Instance/Session

Explanation:

Until Oracle 9.0.1, values of bind variables are known in the PARSE phase. Since it's not known, it's not possible to generate execution plans according to bind values.

With 9i and onwards, Oracle peeks the values of bind variables in the FIRST PARSE phase and generates execution plans according to the values in this first PARSE. If subsequent bind values are skewed, then execution plans may not be optimal for the subsequent binds.






Generated at Sat Oct 23 20:40:39 UTC 2021 using JIRA Standard Edition, Version: 3.12.3-#302.