<< Back to previous view

[QA-35] ORA-00600 [kturrur11], [65535], [0]: Instance crashed. Created: 08/Nov/07  Updated: 08/Nov/07

Status: Closed
Project: Questions & Answers
Fix Version/s: None

Type: Oracle - Internals Priority: Major
Reporter: ubTools Support Assignee: ubTools Support
Resolution: Answered Votes: 0

Product Version: Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
Operating System: IBM-AIX

 Description   
The instance crashes with the following error code: 
ORA-00600: internal error code, arguments: [kturrur11], [65535], [0], [], [], [], [], []

Stack trace:

----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
ksedst+001c          bl       ksedst1              FFFFFFFFFFFA3B0 ? 000000000 ?
ksedmp+0290          bl       ksedst               1047C9C10 ?
ksfdmp+0018          bl       03F53584             
kgerinv+00dc         bl       _ptrgl               
kgeasnmierr+0040     bl       kgerinv              0FFFFFFFF ? 0000000BA ?
                                                   FFFFFFFFFFFAD18 ?
                                                   FFFFFFFFFFFAD20 ?
                                                   FFFFFFFFFFFAC08 ?
kturgmbu+02b8        bl       kgeasnmierr          000085188 ? 000000000 ?
                                                   000550021 ? 200000002 ?
                                                   000000000 ? 00000FFFF ?
                                                   000000000 ? 000000000 ?
kturrur+01c8         bl       kturgmbu             1001AEFA8 ? 70000020F745AC0 ?
                                                   0000F4240 ? 104BF4B00 ?
                                                   000085188 ? FFFFFFFFFFFA950 ?
                                                   70000020A3BC630 ? 1102B0D58 ?
ktundo+016c          bl       kturrur              1102B0D58 ? 000000000 ?
                                                   100000000 ? FFFFFFFFFFFA9D0 ?
                                                   FFFFFF00000003 ? 15453015F ?
                                                   000000000 ? 000000000 ?
ktubko+0794          bl       ktundo               1FFFFBC80 ?
                                                   5B16B1B30F745928 ?
                                                   1001DD940 ? 70000020A3C8EA0 ?
                                                   000000528 ? FFFFFFFFFFFBE08 ?
                                                   400000000 ? FFFFFFFFFFFBF08 ?
kturrt+15fc          bl       ktubko               70000020A3C52F0 ? 600000000 ?
                                                   000000000 ? 0E8E65D60 ?
                                                   3B008CEC89 ? 55002100000000 ?
kturec+0dcc          bl       kturrt               FFFFFFFFFFFC528 ?
                                                   21000000000000 ? 1FFFFC5E0 ?
                                                   000000000 ? 0000009A0 ?
                                                   40A288838 ? 110021A88 ?
kturax+0300          bl       kturec               5522880400 ? 000000000 ?
                                                   19E370001 ? 000000001 ?
                                                   FFFFFFFFFFFFCAC0 ?
                                                   11FFFFFEFF ? 400000000 ?
ktprbeg+02b0         bl       kturax               10FDB1B2B0 ? 004AD8530 ?
ktmmon+0ebc          bl       ktprbeg              080000000 ?
ktmSmonMain+0030     bl       ktmmon               000000000 ?
ksbrdp+03e0          bl       _ptrgl               
opirip+03fc          bl       01FC66A0             
opidrv+0448          bl       opirip               1103BD070 ? 4103BE990 ?
                                                   FFFFFFFFFFFF860 ?
sou2o+0090           bl       opidrv               32023373FC ? 400000020 ?
                                                   FFFFFFFFFFFF860 ?
opimai_real+0150     bl       01FC0DF4             
main+0098            bl       opimai_real          000000000 ? 000000000 ?
__start+0090         bl       main                 000000000 ? 000000000 ?


 Comments   
Comment by ubTools Support [ 08/Nov/07 08:40 AM ]
From Oracle Note:4940513.8: 
Bug 4940513  OERI[kturrur11] can occur with multi block undo
 This note gives a brief overview of bug 4940513.
Affects:

    Product (Component)	Oracle Server (Rdbms)
    Range of versions believed to be affected	Versions < 11
    Versions confirmed as being affected	

        * 9.2.0.6
        * 9.2.0.7
        * 10.1.0.4
        * 10.1.0.5
        * 10.2.0.2 

    Platforms affected	Generic (all / most platforms affected)

Fixed:

    This issue is fixed in	

        * 9.2.0.8 (Server Patch Set)
        * 10.2.0.3 (Server Patch Set)
        * 11g (Future version) 

Symptoms:
	
Related To:

    * Internal Error May Occur (ORA-600)
    * ORA-600 [kturrur11] 

	

    * (None Specified) 

Description

    In rare situations the server could raise ORA-600 [kturrur11][65535][0]

    Workaround:
      Avoid the multi block undo code path by making sure that the 
      block size in the undo tablespace is large enough to accomodate the 
      largest column that is changed by any SQL statement. If the block size 
      of the data tablespaces is larger than the block size of the undo 
      tablespace, increase the blocksize of the undo tablespace to that of 
      the data tablespace.
Comment by ubTools Support [ 08/Nov/07 10:32 AM ]
Workaround:

Drop segments which need recovery.

Finding the UNDO segment from ALERT LOG:

Errors in file /product/10g/admin/DTWP/bdump/dtwp1_smon_7024648.trc:
ORA-00600: internal error code, arguments: [kturrur11], [65535], [0], [], [], [], [], []
replication_dependency_tracking turned off (no async multimaster replication found)
Sat Nov  3 13:19:38 2007
ORACLE Instance DTWP1 (pid = 15) - Error 600 encountered while recovering transaction (85, 33).
Sat Nov  3 13:19:38 2007
Errors in file /product/10g/admin/DTWP/bdump/dtwp1_smon_7024648.trc:
ORA-00600: internal error code, arguments: [kturrur11], [65535], [0], [], [], [], [], []

SMON is trying to rollback a transaction in (UNDOSEGMENT#85, UNDOSLOT#33).

Identifiying UNDO segment:

select segment_name,owner,tablespace_name from dba_rollback_segs
where segment_id=85;  

SEGMENT_NAME                   OWNER  TABLESPACE_NAME
------------------------------ ------ ------------------------------
_SYSSMU85$                     PUBLIC UNDOTBS1

Undo block in the SMON trace:

********************************************************************************
UNDO BLK:  
xid: 0x0055.021.00085188  seq: 0xffff cnt: 0x1   irb: 0x1   icl: 0x0   flg: 0x0000
 
 Rec Offset      Rec Offset      Rec Offset      Rec Offset      Rec Offset
---------------------------------------------------------------------------
0x01 0x0018     
 
*-----------------------------
* Rec #0x1  slt: 0x21  objn: 125213(0x0001e91d)  objd: 564547  tblspc: 10(0x0000000a)
*       Layer:   5 (Transaction Undo)   opc: 1   rci 0x00   
Undo type:  Multi-block undo Mid-piece   Last buffer split:  Yes 
Temp Object:  No 
Tablespace Undo:  No 
rdba: 0x5b16b1af
*-----------------------------

Transaction ID: xid: 0x0055.021.00085188

Hexadecimal 55 = Decimal 85
Hexadecimal 21 = Decimal 33

That means this UNDO block is the block which SMON is reading to rollback a segment.

irb points to last UNDO RECORD in UNDO block. rci points to previous UNDO RECORD. if rci=0, it's the first UNDO RECORD. Recovery operation starts from irb and chain is followed by rci until rci is zero.

In this case, the UNDO block includes just one UNDO RECORD. This UNDO RECORD inludes UNDO DATA for object#125213.

Object needs recovery:

select owner,object_name,object_type from dba_objects where object_id=125213

Owner : OWBRUN
Object_name : sm_post_ind2
Object_type  INDEX

Index dropped. But problem did not disappear. Then, it's decided to drop this UNDO segment after identifiying all objects in.

Reading Transaction Table in the UNDO header:

ALTER SYSTEM DUMP UNDO HEADER '_SYSSMU85$';

...
********************************************************************************
Undo Segment:  _SYSSMU85$ (85)
********************************************************************************
...
TRN TBL::
 
  index  state cflags  wrap#    uel         scn            dba            parent-xid    nub     stmt_num    cmt
  ------------------------------------------------------------------------------------------------
   0x00    9    0x00  0x85435  0xffff  0x0847.4024f907  0x00000000  0x0000.000.00000000  0x00000000
   0x00000000  1194066440
   0x01    9    0x00  0x84b3c  0x0004  0x0847.4024f89b  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x02    9    0x00  0x85237  0x0006  0x0847.4024f895  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x03    9    0x00  0x85406  0x0011  0x0847.4024f877  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x04    9    0x00  0x851d9  0x000a  0x0847.4024f89e  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x05    9    0x00  0x85234  0x002f  0x0847.4024f881  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x06    9    0x00  0x8543f  0x002b  0x0847.4024f897  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x07    9    0x00  0x850ce  0x002e  0x0847.4024f8ac  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x08    9    0x00  0x853f3  0x001a  0x0847.4024f88a  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x09    9    0x00  0x85188  0x001f  0x0847.4024f87d  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x0a    9    0x00  0x84f75  0x0014  0x0847.4024f8a0  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x0b    9    0x00  0x832f2  0x0007  0x0847.4024f8aa  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x0c    9    0x00  0x85313  0x001e  0x0847.4024f8d2  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x0d    9    0x00  0x85320  0x000c  0x0847.4024f8d0  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x0e    9    0x00  0x849fb  0x0012  0x0847.4024f890  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x0f    9    0x00  0x8530c  0x000e  0x0847.4024f88e  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x10    9    0x00  0x84ac9  0x0015  0x0847.4024f8de  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x11    9    0x00  0x854f4  0x0009  0x0847.4024f87a  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x12    9    0x00  0x84ce9  0x0002  0x0847.4024f892  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x13    9    0x00  0x85220  0x001b  0x0847.4024f8c4  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x14    9    0x00  0x85119  0x001d  0x0847.4024f8a2  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x15    9    0x00  0x8540c  0x0025  0x0847.4024f8e0  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x16    9    0x00  0x85177  0x0017  0x0847.4024f8ea  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x17    9    0x00  0x84f02  0x002a  0x0847.4024f8ec  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x18    9    0x00  0x84e2d  0x0027  0x0847.4024f8f7  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x19    9    0x00  0x8537a  0x0020  0x0847.4024f8a6  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x1a    9    0x00  0x8530b  0x000f  0x0847.4024f88c  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x1b    9    0x00  0x841bc  0x0029  0x0847.4024f8c6  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x1c    9    0x00  0x852a9  0x002d  0x0847.4024f8fb  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x1d    9    0x00  0x84d24  0x0019  0x0847.4024f8a4  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x1e    9    0x00  0x85419  0x0010  0x0847.4024f8d3  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x1f    9    0x00  0x84ea2  0x0005  0x0847.4024f87f  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x20    9    0x00  0x853a5  0x000b  0x0847.4024f8a8  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x21   10    0x10  0x85188  0x0919  0x0847.4024f8fd  0x5b16b1b3  0x0000.000.00000000  0x00000002   0x00000000  0
   0x22    9    0x00  0x85279  0x002c  0x0847.4024f886  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x23    9    0x00  0x847b0  0x0028  0x0847.4024f8bb  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x24    9    0x00  0x851cf  0x0023  0x0847.4024f8b9  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x25    9    0x00  0x84a9c  0x0016  0x0847.4024f8e1  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x26   10    0x90  0x7539b  0x0003  0x0847.3fcdcb21  0x4f81d3d2  0x0000.000.00000000  0x0000dbd9   0x00000000  0
   0x27    9    0x00  0x850ac  0x001c  0x0847.4024f8f9  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x28    9    0x00  0x8531b  0x0013  0x0847.4024f8bc  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x29    9    0x00  0x854f0  0x000d  0x0847.4024f8c7  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x2a    9    0x00  0x85301  0x0018  0x0847.4024f8ed  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x2b    9    0x00  0x83c38  0x0001  0x0847.4024f899  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x2c    9    0x00  0x85051  0x0008  0x0847.4024f888  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x2d    9    0x00  0x84a3c  0x0000  0x0847.4024f904  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x2e    9    0x00  0x84f35  0x0024  0x0847.4024f8ae  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440
   0x2f    9    0x00  0x85100  0x0022  0x0847.4024f884  0x00000000  0x0000.000.00000000  0x00000000   0x00000000
  1194066440

State#10 means active transaction. dba points to starting UNDO block address.

There are 2 active transactions. The one of them points to the slot of 0x21, which is the same as seen in the SMON trace that causes this ORA-600 [kturrur11] error. The other active transaction is available in the slot of 0x26, which has a dba of 0x4f81d3d2.

The object in the slot of 0x21 had been found above; but the object in slot of 0x26 is not known yet.

Object needs recovery:

Hexadecimal 4f81d3d2 = Decimal 1333908434

select DBMS_UTILITY.DATA_BLOCK_ADDRESS_FILE(1333908434) from x$dual;
318

select DBMS_UTILITY.DATA_BLOCK_ADDRESS_BLOCK(1333908434) from x$dual;
119762

alter system dump datafile 318 block 119762;

...
*** SESSION ID:(489.37) 2007-11-03 16:23:56.878
Start dump data blocks tsn: 1 file#: 318 minblk 119762 maxblk 119762
buffer tsn: 1 rdba: 0x4f81d3d2 (318/119762)
...
UNDO BLK:  
xid: 0x0055.026.0007539b  seq: 0xf72d cnt: 0x5b  irb: 0x1   icl: 0x0   flg: 0x0000
 
 Rec Offset      Rec Offset      Rec Offset      Rec Offset      Rec Offset
---------------------------------------------------------------------------
0x01 0x1f8c     0x02 0x1f30     0x03 0x1ed4     0x04 0x1e78     0x05 0x1e1c     
0x06 0x1dc0     0x07 0x1d64     0x08 0x1d08     0x09 0x1cac     0x0a 0x1c50     
0x0b 0x1bf4     0x0c 0x1b9c     0x0d 0x1b48     0x0e 0x1af0     0x0f 0x1a98     
0x10 0x1a40     0x11 0x19e8     0x12 0x1990     0x13 0x193c     0x14 0x18e4     
0x15 0x1890     0x16 0x1838     0x17 0x17e0     0x18 0x178c     0x19 0x1738     
0x1a 0x16e0     0x1b 0x1688     0x1c 0x1634     0x1d 0x15dc     0x1e 0x1584     
0x1f 0x152c     0x20 0x14d4     0x21 0x147c     0x22 0x1424     0x23 0x13cc     
0x24 0x1374     0x25 0x131c     0x26 0x12c4     0x27 0x126c     0x28 0x1214     
0x29 0x11bc     0x2a 0x1168     0x2b 0x1110     0x2c 0x10b8     0x2d 0x1064     
0x2e 0x1010     0x2f 0x0fbc     0x30 0x0f64     0x31 0x0f0c     0x32 0x0eb8     
0x33 0x0e60     0x34 0x0e0c     0x35 0x0db8     0x36 0x0d64     0x37 0x0d0c     
0x38 0x0cb4     0x39 0x0c60     0x3a 0x0c08     0x3b 0x0bb0     0x3c 0x0b58     
0x3d 0x0b04     0x3e 0x0aac     0x3f 0x0a58     0x40 0x0a04     0x41 0x09ac     
0x42 0x0954     0x43 0x08fc     0x44 0x08a4     0x45 0x0850     0x46 0x07f8     
0x47 0x07a0     0x48 0x074c     0x49 0x06f4     0x4a 0x06a0     0x4b 0x0648     
0x4c 0x05f0     0x4d 0x0598     0x4e 0x0540     0x4f 0x04e8     0x50 0x0490     
0x51 0x0438     0x52 0x03e0     0x53 0x0388     0x54 0x0334     0x55 0x02dc     
0x56 0x0284     0x57 0x022c     0x58 0x01d4     0x59 0x0180     0x5a 0x012c     
0x5b 0x00d4     
 
*-----------------------------

irb points to the UNDO RECORD of 0x1.

*-----------------------------
* Rec #0x1  slt: 0x26  objn: 125212(0x0001e91c)  objd: 564548  tblspc: 10(0x0000000a)
*       Layer:  10 (Index)   opc: 22   rci 0x00   
Undo type:  Regular undo    User Undo Applied  Last buffer split:  No 
Temp Object:  No 
Tablespace Undo:  No 
rdba: 0x4f81d3d1
*-----------------------------
...

rci of UNDO RECORD of 0x1 is 0x00. That means this is the first and last UNDO RECORD.

Object ID in this UNDO RECORD is 125212.

SQL> select owner,object_name,object_type from dba_objects where object_id in (125213,125212);

OWNER
------------------------------
OBJECT_NAME
--------------------------------------------------------------------------------
OBJECT_TYPE
-------------------
OWBRUN
SM_POST_IND1
INDEX

It's another lucky object that its type is INDEX. This index dropped. Now, after being sure that there is no new active transactions in this UNDO segment, the followings were done:

  • Shutdown the database
  • Set the following parameter to PFILE/SPFILE:

_smu_debug_mode=4
_offline_rollback_segments=(_SYSSMU85$)

  • Startup the database
  • drop rollback segment "_SYSSMU85$";

After UNDO segment is successfuly dropped, the INTERNAL parameters above should be removed. But, in our case, while dropping UNDO segment, although the current internal error (ORA-600) [kturrur11]) disappeared; the another internal error (ORA-600 [kddummy_blkchk]) was encountered. It's created as another issue as QA-34.

Since all objects needing recovery in the UNDO segment were dropped, there is no need to re-create the database after using _offline_rollback_segments parameter.

Generated at Thu Oct 16 08:36:09 UTC 2025 using JIRA Standard Edition, Version: 3.12.3-#302.