ubTools Support http://jira.ubtools.com/jira/secure/IssueNavigator.jspa?reset=true&type=8&pid=10042&sorter/field=issuekey&sorter/order=DESC An XML representation of a search request en-us RE: [QA-63] ORA-600 [3020] on the standby after adding a datafile on primary http://jira.ubtools.com/jira/browse/QA-63?focusedCommentId=33027#action_33027 Fri, 16 Feb 2018 12:46:45 +0000 ubTools Support <b>SOLUTION:</b> <p><b>Problem:</b></p> <p>Oracle tries to apply archivelog to a file which was not created on standby yet.</p> <p><b>Fix:</b></p> <p>This is Oracle bug 27229389.</p> <p><b>Workaround:</b></p> <p>Copy datafile from primary to standby that doesn't require corrupted archivelogs.</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-63">QA-63</a>)</td> </tr> <tr> <td>Edited by:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a></td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-63?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=33027#action_33027 RE: [QA-63] ORA-600 [3020] on the standby after adding a datafile on primary http://jira.ubtools.com/jira/browse/QA-63?focusedCommentId=33026#action_33026 Fri, 16 Feb 2018 12:40:44 +0000 ubTools Support <b>ANALYSIS of the ROOT CAUSE:</b> <p><b>Data:</b></p> <p> The datafile has been created at sequence#91417 and the problem happened at sequence#91425 at file#232 block#3.</p> <p><ins>REDO Dump Commands:</ins></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SQL&gt; ALTER SYSTEM dump logfile '/u01/ORCL/archive/52b86b9a_1_91417_922239972.arc' dba min 232 3 dba max 232 3; System altered. SQL&gt; ALTER SYSTEM dump logfile '/u01/ORCL/archive/52b86b9a_1_91418_922239972.arc' dba min 232 3 dba max 232 3; System altered. SQL&gt; ALTER SYSTEM dump logfile '/u01/ORCL/archive/52b86b9a_1_91419_922239972.arc' dba min 232 3 dba max 232 3; System altered. SQL&gt; ALTER SYSTEM dump logfile '/u01/ORCL/archive/52b86b9a_1_91420_922239972.arc' dba min 232 3 dba max 232 3; System altered. SQL&gt; ALTER SYSTEM dump logfile '/u01/ORCL/archive/52b86b9a_1_91421_922239972.arc' dba min 232 3 dba max 232 3; System altered. SQL&gt; ALTER SYSTEM dump logfile '/u01/ORCL/archive/52b86b9a_1_91422_922239972.arc' dba min 232 3 dba max 232 3; System altered. SQL&gt; ALTER SYSTEM dump logfile '/u01/ORCL/archive/52b86b9a_1_91423_922239972.arc' dba min 232 3 dba max 232 3; System altered. SQL&gt; ALTER SYSTEM dump logfile '/u01/ORCL/archive/52b86b9a_1_91424_922239972.arc' dba min 232 3 dba max 232 3; System altered. SQL&gt; ALTER SYSTEM dump logfile '/u01/ORCL/archive/52b86b9a_1_91425_922239972.arc' dba min 232 3 dba max 232 3; System altered. SQL&gt; </pre> </div></div> <p><ins>REDO Dumps:</ins></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>DUMP OF REDO FROM FILE '/u01/ORCL/archive/52b86b9a_1_91417_922239972.arc' ..... REDO RECORD - Thread:1 RBA: 0x016519.00000765.01e8 LEN: 0x0058 VLD: 0x01 SCN: 0x0010.2618c0fb SUBSCN: 1 02/13/2018 07:44:59 (LWN RBA: 0x016519.00000763.0010 LEN: 0004 NST: 0002 SCN: 0x0010.2618c0f9) CHANGE #1 TYP:1 CLS:12 AFN:232 DBA:0x3a000003 OBJ:4294967295 SCN:0x0010.2618c0fb SEQ:1 OP:22.4 ENC:0 RBL:0 ktfbbfo - File BitMap Block Format: BitMap Control: RelFno: 232, BeginBlock: 128, Flag: 0, First: 0, Free: 63488 REDO RECORD - Thread:1 RBA: 0x016519.00000763.0010 LEN: 0x0244 VLD: 0x05 SCN: 0x0010.2618c0fb SUBSCN: 1 02/13/2018 07:44:59 CHANGE #1 TYP:0 CLS:69 AFN:3 DBA:0x00c018c0 OBJ:4294967295 SCN:0x0010.2618c0ee SEQ:1 OP:5.4 ENC:0 RBL:0 ktucm redo: slt: 0x0007 sqn: 0x000060e7 srt: 0 sta: 9 flg: 0x2 ktucf redo: uba: 0x3000319f.07d5.06 ext: 2 spc: 7334 fbi: 0 CHANGE #2 MEDIA RECOVERY MARKER SCN:0x0000.00000000 SEQ:0 OP:17.30 ENC:0 Add datafiles to tablespace #141 file #232 relative file #232. '/orassd/orcl/datafile/menkul2018_data02.dbf' flags(reuse): 0x0 Checkpointed at scn: 0x0010.2618c0f0 02/13/2018 07:44:56 ..... DUMP OF REDO FROM FILE '/u01/ORCL/archive/52b86b9a_1_91425_922239972.arc' ..... REDO RECORD - Thread:1 RBA: 0x016521.0001a022.0034 LEN: 0x0040 VLD: 0x01 SCN: 0x0010.2619b07d SUBSCN: 14 02/13/2018 08:23:51 (LWN RBA: 0x016521.00019dca.0010 LEN: 1012 NST: 0002 SCN: 0x0010.2619b069) CHANGE #1 TYP:0 CLS:12 AFN:232 DBA:0x3a000003 OBJ:4294967295 SCN:0x0010.2618c0fb SEQ:2 OP:22.5 ENC:0 RBL:0 ktfbbredo - File BitMap Block Redo: Use Bits: </pre> </div></div> <p><b>Comment:</b></p> <p>The datafile has been created at sequence#91417 by REDO OP code 17.30, which means:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>the OP:17.30 redo which adds the &lt;file#&gt; datafile </pre> </div></div> <p><em>Ref: Oracle Doc ID 27229389.8</em></p> <p>There are some other OP codes 22.4 and 5.4 before adding the datafile.</p> <p><ins>Change Vector for OP Code 22:4:</ins></p> <p>It tries to change absolute file#232 <em>(AFN:232 DBA:0x3a000003)</em>. This is the problem that Oracle tries to apply a change vector to a file which was not created yet.</p> <p><ins>Change Vector for OP Code 5:4:</ins></p> <p>It tries to change absolute file#3 <em>(AFN:3 DBA:0x00c018c0)</em>. This is a different file. So, it's out of the scope.</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-63">QA-63</a>)</td> </tr> <tr> <td>Edited by:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a></td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-63?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=33026#action_33026 RE: [QA-63] ORA-600 [3020] on the standby after adding a datafile on primary http://jira.ubtools.com/jira/browse/QA-63?focusedCommentId=33025#action_33025 Fri, 16 Feb 2018 12:38:42 +0000 ubTools Support <b>ANALYSIS of the RESULT:</b><br/> <em>Ref: ORCL_pr0i_21945.trc</em> <p><b>Data:</b></p> <p><ins>REDO Dump:</ins></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>KCOX_FUTURE: CHANGE IN FUTURE OF BLOCK *** 2018-02-13 08:26:56.826 RECOVERY STUCK AT BLOCK 3 OF FILE 232 Redo record scn: 0x0010.2619b07d CHANGE #1 TYP:0 CLS:12 AFN:232 DBA:0x3a000003 OBJ:4294967295 SCN:0x0010.2618c0fb SEQ:2 OP:22.5 ENC:0 RBL:0 Buffer read during recovery: ..... </pre> </div></div> <p>The stuck recovery happened at file#232 block#3 with KCOX_FUTURE: CHANGE IN FUTURE OF BLOCK information.</p> <p><ins>Block Dump</ins>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>buffer tsn: 141 rdba: 0x3a000003 (232/3) scn: 0x0000.00000000 seq: 0x01 flg: 0x05 tail: 0x00000001 frmt: 0x02 chkval: 0x9d03 type: 0x00=unknown on-disk scn: 0x0.0 </pre> </div></div> <p>SCN is 0, type is unknown. flg is 0x05:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Where flg: 0x05 contains flag 0x1 (unused,unformatted block). </pre> </div></div> <p><em>Ref: Oracle Doc ID 17896895.8</em></p> <p><b>Comment:</b></p> <p>The change vector was expecting SCN:0x0010.2618c0fb on the block. But, the SCN on the block was 0x0000.00000000.</p> <p>Oracle was trying to apply archivelog to an unformatted block. REDO in archivelog is beyond block in datafile. This inconsistency causes stuck recovery.</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-63">QA-63</a>)</td> </tr> <tr> <td>Edited by:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a></td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-63?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=33025#action_33025 RE: [QA-63] ORA-600 [3020] on the standby after adding a datafile on primary http://jira.ubtools.com/jira/browse/QA-63?focusedCommentId=33024#action_33024 Fri, 16 Feb 2018 12:31:58 +0000 ubTools Support <b>PROBLEM OCCURRENCE:</b> <p><b>Adding datafile on the Primary:</b></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Tue Feb 13 07:44:52 2018 ALTER TABLESPACE MENKUL2018_DATA ADD DATAFILE '/orassd/orcl/datafile/menkul2018_data02.dbf' SIZE 5G AUTOEXTEND ON NEXT 100M MAXSIZE UNLIMITED Completed: ALTER TABLESPACE MENKUL2018_DATA ADD DATAFILE '/orassd/orcl/datafile/menkul2018_data02.dbf' SIZE 5G AUTOEXTEND ON NEXT 100M MAXSIZE UNLIMITED Tue Feb 13 07:45:38 2018 </pre> </div></div> <p><b>Applying Archivelogs on the Standby:</b></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Tue Feb 13 07:46:55 2018 ALTER DATABASE RECOVER AUTOMATIC STANDBY DATABASE UNTIL CHANGE 69358633799 Media Recovery Start started logmerger process Tue Feb 13 07:46:55 2018 Managed Standby Recovery not using Real Time Apply Parallel Media Recovery started with 24 slaves Media Recovery Log /u01/ORCL/archive/52b86b9a_1_91417_922239972.arc Tue Feb 13 07:47:07 2018 Successfully added datafile 232 to media recovery Datafile #232: '/u01/oracle/app/oradata/ORCL/datafile/ORCL_STBY/datafile/o1_mf_menkul20_f84vg0lt_.dbf' Incomplete Recovery applied until change 69358633799 time 02/13/2018 07:45:38 Tue Feb 13 07:47:08 2018 Media Recovery Complete (ORCL) Completed: ALTER DATABASE RECOVER AUTOMATIC STANDBY DATABASE UNTIL CHANGE 69358633799 ..... Tue Feb 13 08:26:55 2018 ALTER DATABASE RECOVER AUTOMATIC STANDBY DATABASE UNTIL CHANGE 69358697423 Media Recovery Start started logmerger process Tue Feb 13 08:26:55 2018 Managed Standby Recovery not using Real Time Apply Parallel Media Recovery started with 24 slaves Media Recovery Log /u01/ORCL/archive/52b86b9a_1_91425_922239972.arc Tue Feb 13 08:26:57 2018 Errors in file /u01/oracle/app/diag/rdbms/orcl_stby/ORCL/trace/ORCL_pr0i_21945.trc (incident=131897): ORA-00600: internal error code, arguments: [3020], [232], [3], [973078531], [], [], [], [], [], [], [], [] ORA-10567: Redo is inconsistent with data block (file# 232, block# 3, file offset is 24576 bytes) ORA-10564: tablespace MENKUL2018_DATA ORA-01110: data file 232: '/u01/oracle/app/oradata/ORCL/datafile/ORCL_STBY/datafile/o1_mf_menkul20_f84vg0lt_.dbf' ORA-10560: block type '0' Incident details in: /u01/oracle/app/diag/rdbms/orcl_stby/ORCL/incident/incdir_131897/ORCL_pr0i_21945_i131897.trc Tue Feb 13 08:26:59 2018 </pre> </div></div> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-63">QA-63</a>)</td> </tr> <tr> <td>Edited by:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a></td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-63?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=33024#action_33024 RE: [QA-41] Startup database fails with ORA-600 [4000], ORA-600 [4137]. http://jira.ubtools.com/jira/browse/QA-41?focusedCommentId=10645#action_10645 Tue, 17 Jun 2008 09:28:35 +0000 ubTools Support The database was opened inconsistently. It'll be recreated with full export/import. <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-41">QA-41</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10645#action_10645 RE: [QA-41] Startup database fails with ORA-600 [4000], ORA-600 [4137]. http://jira.ubtools.com/jira/browse/QA-41?focusedCommentId=10644#action_10644 Tue, 17 Jun 2008 09:17:27 +0000 ubTools Support Restart the database: <ul class="alternate" type="square"> <li>Setting _CORRUPTED_ROLLBACK_SEGMENTS in init&lt;SID&gt;.ora</li> <li>startup mount;</li> <li>recover database until cancel;<br/> &lt;--cancel</li> <li>alter database open resetlogs;</li> </ul> <p>The database is opened.</p> <p>Since it's opened in inconsistent state, a full export and then import into a new database is required to get rid of the inconsistency in Oracle dictionary. But, the customer data will not be consistent after the import. It should be reviewed by the customer.</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-41">QA-41</a>)</td> </tr> <tr> <td>Edited by:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a></td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10644#action_10644 RE: [QA-41] Startup database fails with ORA-600 [4000], ORA-600 [4137]. http://jira.ubtools.com/jira/browse/QA-41?focusedCommentId=10643#action_10643 Tue, 17 Jun 2008 00:13:00 +0000 ubTools Support An exerpt from the trace file: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ORA-00600: internal error code, arguments: [4000], [9], [], [], [], [], [], [] Current SQL statement for this session: select ctime, mtime, stime from obj$ where obj# = :1 ... Block header dump: 0x0080003e Object id on Block? Y seg/obj: 0x12 csc: 0x570.b8368d16 itc: 1 flg: - typ: 1 - DATA fsl: 0 fnx: 0x0 ver: 0x01 ... Itl Xid Uba Flag Lck Scn/Fsc 0x01 xid: 0x0009.019.000dc23f uba: 0x58c13ddb.0523.46 --U- 1 fsc 0x0000.b8368d17 </pre> </div></div> <p>Looks like a problem regarding obj$ and its undo...If undo requirement is bypassed, there will be no requirement for undo. In order to do that, bumping SNC further needed.</p> <p><em>csc</em> shows the the SCN of last block cleanout. We <em>guessed</em> it may be used a target bumping SCN as below:</p> <ul class="alternate" type="square"> <li>0x570.b8368d16 =&gt; 0x570b8368d16 =&gt; Decimal: 5981685058838 =&gt; divide by 1024/1024/1024 = 5571</li> </ul> <p>Bump SCN as below and restart:</p> <ul class="alternate" type="square"> <li>Setting _<em>MINIMUM_GIGA_SCN</em> = 5571 in init&lt;SID&gt;.ora</li> <li>startup mount;</li> <li>recover database until cancel;<br/> &lt;--cancel</li> <li>alter database open resetlogs;</li> </ul> <p>ORA-600 <span class="error">&#91;4000&#93;</span> disappeared. But now, the following error appeared:</p> <blockquote><p>ORA-00600: internal error code, arguments: <span class="error">&#91;4137&#93;</span>, [], [], [], [], [], [], []</p></blockquote> <p>Oracle Note:47456.1: </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>DESCRIPTION: While backing out an undo record (i.e. at the time of rollback) we found a transaction id mis-match indicating either a corruption in the rollback segment or corruption in an object which the rollback segment is trying to apply undo records on. This would indicate a corrupted rollback segment. FUNCTIONALITY: Kernel Transaction Undo Recovery IMPACT: POSSIBLE PHYSICAL CORRUPTION in Rollback segments </pre> </div></div> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-41">QA-41</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10643#action_10643 RE: [QA-41] Startup database fails with ORA-600 [4000], ORA-600 [4137]. http://jira.ubtools.com/jira/browse/QA-41?focusedCommentId=10642#action_10642 Mon, 16 Jun 2008 23:38:47 +0000 ubTools Support Steps to open the database: <ul class="alternate" type="square"> <li>setting _<em>ALLOW_RESETLOGS_CORRUPTION=TRUE</em> in init&lt;SID&gt;.ora.</li> <li>startup mount;</li> <li>recover database until cancel;<br/> &lt;--cancel</li> <li>alter database open resetlogs;</li> </ul> <p>But, it failed with the following error:</p> <blockquote><p>ORA-00600: internal error code, arguments: <span class="error">&#91;4000&#93;</span>, <span class="error">&#91;9&#93;</span>, [], [], [], [], [], []</p></blockquote> <p>Oracle Note:47456.1:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>DESCRIPTION: This has the potential to be a very serious error. It means that Oracle has tried to find an undo segment number in the dictionary cache and failed. ARGUMENTS: Arg [a] Undo segment number FUNCTIONALITY: KERNEL TRANSACTION UNDO IMPACT: INSTANCE FAILURE - Instance will not restart STATEMENT FAILURE </pre> </div></div> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-41">QA-41</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10642#action_10642 RE: [QA-38] DBMS_XMLPARSER.FREEPARSER doesn't release UGA memory. http://jira.ubtools.com/jira/browse/QA-38?focusedCommentId=10627#action_10627 Fri, 6 Jun 2008 13:35:03 +0000 ubTools Support The UGA of PGA had been filled with a big chunk which has recreatable "qmxdpls_subhea". This chunk is 50681480 byte. (See <a href="http://jira.ubtools.com/jira/browse/QA-8" title="Heapdump Interpretation"><del>QA-8</del></a> for the simple definitions of HEAPDUMP). <p>Oracle Note:3518909.8:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Calling Dbms_xmlparser.freeParser / dbms_xmldom.freeDocument in the procudure do not appear to free the memory. The leaked memory shows in heapdumps as "qmxdpls_subheap" </pre> </div></div> <p>Although the mentioned bug fixed in Oracle 9.2.0.6; the customer encounters the same problem in Oracle 9.2.0.8.</p> <p>Since the next usage of DBMS_XMLPARSER.PARSECLOB after a previous DBMS_XMLPARSER.FREEPARSER within the same session, the UGA did not grow. This is acceptable by the customer.</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-38">QA-38</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-38?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10627#action_10627 RE: [QA-34] ORA-00600 [kddummy_blkchk] while dropping UNDO segment. http://jira.ubtools.com/jira/browse/QA-34?focusedCommentId=10215#action_10215 Thu, 8 Nov 2007 12:39:08 +0000 ubTools Support The current UNDO TABLESPACE was dropped, and a new one has been created. Then, Event 10061 has been removed. <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-34">QA-34</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10215#action_10215 RE: [QA-35] ORA-00600 [kturrur11], [65535], [0]: Instance crashed. http://jira.ubtools.com/jira/browse/QA-35?focusedCommentId=10214#action_10214 Thu, 8 Nov 2007 11:57:22 +0000 ubTools Support <b>Workaround:</b> <p>Drop segments which need recovery.</p> <p><b>Finding the UNDO segment from ALERT LOG:</b></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Errors in file /product/10g/admin/DTWP/bdump/dtwp1_smon_7024648.trc: ORA-00600: internal error code, arguments: [kturrur11], [65535], [0], [], [], [], [], [] replication_dependency_tracking turned off (no async multimaster replication found) Sat Nov 3 13:19:38 2007 ORACLE Instance DTWP1 (pid = 15) - Error 600 encountered while recovering transaction (85, 33). Sat Nov 3 13:19:38 2007 Errors in file /product/10g/admin/DTWP/bdump/dtwp1_smon_7024648.trc: ORA-00600: internal error code, arguments: [kturrur11], [65535], [0], [], [], [], [], [] </pre> </div></div> <p>SMON is trying to rollback a transaction in (UNDOSEGMENT#85, UNDOSLOT#33).</p> <p><b>Identifiying UNDO segment:</b></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>select segment_name,owner,tablespace_name from dba_rollback_segs where segment_id=85; SEGMENT_NAME OWNER TABLESPACE_NAME ------------------------------ ------ ------------------------------ _SYSSMU85$ PUBLIC UNDOTBS1 </pre> </div></div> <p><b>Undo block in the SMON trace:</b></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>******************************************************************************** UNDO BLK: xid: 0x0055.021.00085188 seq: 0xffff cnt: 0x1 irb: 0x1 icl: 0x0 flg: 0x0000 Rec Offset Rec Offset Rec Offset Rec Offset Rec Offset --------------------------------------------------------------------------- 0x01 0x0018 *----------------------------- * Rec #0x1 slt: 0x21 objn: 125213(0x0001e91d) objd: 564547 tblspc: 10(0x0000000a) * Layer: 5 (Transaction Undo) opc: 1 rci 0x00 Undo type: Multi-block undo Mid-piece Last buffer split: Yes Temp Object: No Tablespace Undo: No rdba: 0x5b16b1af *----------------------------- </pre> </div></div> <p>Transaction ID: xid: 0x0055.021.00085188</p> <p>Hexadecimal 55 = Decimal 85<br/> Hexadecimal 21 = Decimal 33</p> <p>That means this UNDO block is the block which SMON is reading to rollback a segment.</p> <p><b>irb</b> points to last UNDO RECORD in UNDO block. <b>rci</b> points to previous UNDO RECORD. if rci=0, it's the first UNDO RECORD. Recovery operation starts from irb and chain is followed by rci until rci is zero.</p> <p>In this case, the UNDO block includes just one UNDO RECORD. This UNDO RECORD inludes UNDO DATA for object#125213.</p> <p><b>Object needs recovery:</b></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>select owner,object_name,object_type from dba_objects where object_id=125213 Owner : OWBRUN Object_name : sm_post_ind2 Object_type INDEX </pre> </div></div> <p>Index dropped. But problem did not disappear. Then, it's decided to drop this UNDO segment after identifiying all objects in.</p> <p><b>Reading Transaction Table in the UNDO header:</b></p> <p>ALTER SYSTEM DUMP UNDO HEADER '_SYSSMU85$';</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>... ******************************************************************************** Undo Segment: _SYSSMU85$ (85) ******************************************************************************** ... TRN TBL:: index state cflags wrap# uel scn dba parent-xid nub stmt_num cmt ------------------------------------------------------------------------------------------------ 0x00 9 0x00 0x85435 0xffff 0x0847.4024f907 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x01 9 0x00 0x84b3c 0x0004 0x0847.4024f89b 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x02 9 0x00 0x85237 0x0006 0x0847.4024f895 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x03 9 0x00 0x85406 0x0011 0x0847.4024f877 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x04 9 0x00 0x851d9 0x000a 0x0847.4024f89e 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x05 9 0x00 0x85234 0x002f 0x0847.4024f881 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x06 9 0x00 0x8543f 0x002b 0x0847.4024f897 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x07 9 0x00 0x850ce 0x002e 0x0847.4024f8ac 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x08 9 0x00 0x853f3 0x001a 0x0847.4024f88a 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x09 9 0x00 0x85188 0x001f 0x0847.4024f87d 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x0a 9 0x00 0x84f75 0x0014 0x0847.4024f8a0 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x0b 9 0x00 0x832f2 0x0007 0x0847.4024f8aa 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x0c 9 0x00 0x85313 0x001e 0x0847.4024f8d2 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x0d 9 0x00 0x85320 0x000c 0x0847.4024f8d0 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x0e 9 0x00 0x849fb 0x0012 0x0847.4024f890 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x0f 9 0x00 0x8530c 0x000e 0x0847.4024f88e 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x10 9 0x00 0x84ac9 0x0015 0x0847.4024f8de 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x11 9 0x00 0x854f4 0x0009 0x0847.4024f87a 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x12 9 0x00 0x84ce9 0x0002 0x0847.4024f892 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x13 9 0x00 0x85220 0x001b 0x0847.4024f8c4 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x14 9 0x00 0x85119 0x001d 0x0847.4024f8a2 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x15 9 0x00 0x8540c 0x0025 0x0847.4024f8e0 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x16 9 0x00 0x85177 0x0017 0x0847.4024f8ea 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x17 9 0x00 0x84f02 0x002a 0x0847.4024f8ec 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x18 9 0x00 0x84e2d 0x0027 0x0847.4024f8f7 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x19 9 0x00 0x8537a 0x0020 0x0847.4024f8a6 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x1a 9 0x00 0x8530b 0x000f 0x0847.4024f88c 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x1b 9 0x00 0x841bc 0x0029 0x0847.4024f8c6 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x1c 9 0x00 0x852a9 0x002d 0x0847.4024f8fb 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x1d 9 0x00 0x84d24 0x0019 0x0847.4024f8a4 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x1e 9 0x00 0x85419 0x0010 0x0847.4024f8d3 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x1f 9 0x00 0x84ea2 0x0005 0x0847.4024f87f 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x20 9 0x00 0x853a5 0x000b 0x0847.4024f8a8 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x21 10 0x10 0x85188 0x0919 0x0847.4024f8fd 0x5b16b1b3 0x0000.000.00000000 0x00000002 0x00000000 0 0x22 9 0x00 0x85279 0x002c 0x0847.4024f886 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x23 9 0x00 0x847b0 0x0028 0x0847.4024f8bb 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x24 9 0x00 0x851cf 0x0023 0x0847.4024f8b9 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x25 9 0x00 0x84a9c 0x0016 0x0847.4024f8e1 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x26 10 0x90 0x7539b 0x0003 0x0847.3fcdcb21 0x4f81d3d2 0x0000.000.00000000 0x0000dbd9 0x00000000 0 0x27 9 0x00 0x850ac 0x001c 0x0847.4024f8f9 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x28 9 0x00 0x8531b 0x0013 0x0847.4024f8bc 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x29 9 0x00 0x854f0 0x000d 0x0847.4024f8c7 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x2a 9 0x00 0x85301 0x0018 0x0847.4024f8ed 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x2b 9 0x00 0x83c38 0x0001 0x0847.4024f899 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x2c 9 0x00 0x85051 0x0008 0x0847.4024f888 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x2d 9 0x00 0x84a3c 0x0000 0x0847.4024f904 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x2e 9 0x00 0x84f35 0x0024 0x0847.4024f8ae 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 0x2f 9 0x00 0x85100 0x0022 0x0847.4024f884 0x00000000 0x0000.000.00000000 0x00000000 0x00000000 1194066440 </pre> </div></div> <p><b>State#10</b> means active transaction. <b>dba</b> points to starting UNDO block address.</p> <p>There are 2 active transactions. The one of them points to the slot of 0x21, which is the same as seen in the SMON trace that causes this ORA-600 <span class="error">&#91;kturrur11&#93;</span> error. The other active transaction is available in the slot of 0x26, which has a dba of 0x4f81d3d2.</p> <p>The object in the slot of 0x21 had been found above; but the object in slot of 0x26 is not known yet.</p> <p><b>Object needs recovery:</b></p> <blockquote> <p>Hexadecimal 4f81d3d2 = Decimal 1333908434</p> <p>select DBMS_UTILITY.DATA_BLOCK_ADDRESS_FILE(1333908434) from x$dual;<br/> 318</p> <p>select DBMS_UTILITY.DATA_BLOCK_ADDRESS_BLOCK(1333908434) from x$dual;<br/> 119762</p> <p>alter system dump datafile 318 block 119762;</p></blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>... *** SESSION ID:(489.37) 2007-11-03 16:23:56.878 Start dump data blocks tsn: 1 file#: 318 minblk 119762 maxblk 119762 buffer tsn: 1 rdba: 0x4f81d3d2 (318/119762) ... UNDO BLK: xid: 0x0055.026.0007539b seq: 0xf72d cnt: 0x5b irb: 0x1 icl: 0x0 flg: 0x0000 Rec Offset Rec Offset Rec Offset Rec Offset Rec Offset --------------------------------------------------------------------------- 0x01 0x1f8c 0x02 0x1f30 0x03 0x1ed4 0x04 0x1e78 0x05 0x1e1c 0x06 0x1dc0 0x07 0x1d64 0x08 0x1d08 0x09 0x1cac 0x0a 0x1c50 0x0b 0x1bf4 0x0c 0x1b9c 0x0d 0x1b48 0x0e 0x1af0 0x0f 0x1a98 0x10 0x1a40 0x11 0x19e8 0x12 0x1990 0x13 0x193c 0x14 0x18e4 0x15 0x1890 0x16 0x1838 0x17 0x17e0 0x18 0x178c 0x19 0x1738 0x1a 0x16e0 0x1b 0x1688 0x1c 0x1634 0x1d 0x15dc 0x1e 0x1584 0x1f 0x152c 0x20 0x14d4 0x21 0x147c 0x22 0x1424 0x23 0x13cc 0x24 0x1374 0x25 0x131c 0x26 0x12c4 0x27 0x126c 0x28 0x1214 0x29 0x11bc 0x2a 0x1168 0x2b 0x1110 0x2c 0x10b8 0x2d 0x1064 0x2e 0x1010 0x2f 0x0fbc 0x30 0x0f64 0x31 0x0f0c 0x32 0x0eb8 0x33 0x0e60 0x34 0x0e0c 0x35 0x0db8 0x36 0x0d64 0x37 0x0d0c 0x38 0x0cb4 0x39 0x0c60 0x3a 0x0c08 0x3b 0x0bb0 0x3c 0x0b58 0x3d 0x0b04 0x3e 0x0aac 0x3f 0x0a58 0x40 0x0a04 0x41 0x09ac 0x42 0x0954 0x43 0x08fc 0x44 0x08a4 0x45 0x0850 0x46 0x07f8 0x47 0x07a0 0x48 0x074c 0x49 0x06f4 0x4a 0x06a0 0x4b 0x0648 0x4c 0x05f0 0x4d 0x0598 0x4e 0x0540 0x4f 0x04e8 0x50 0x0490 0x51 0x0438 0x52 0x03e0 0x53 0x0388 0x54 0x0334 0x55 0x02dc 0x56 0x0284 0x57 0x022c 0x58 0x01d4 0x59 0x0180 0x5a 0x012c 0x5b 0x00d4 *----------------------------- </pre> </div></div> <p>irb points to the UNDO RECORD of 0x1.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>*----------------------------- * Rec #0x1 slt: 0x26 objn: 125212(0x0001e91c) objd: 564548 tblspc: 10(0x0000000a) * Layer: 10 (Index) opc: 22 rci 0x00 Undo type: Regular undo User Undo Applied Last buffer split: No Temp Object: No Tablespace Undo: No rdba: 0x4f81d3d1 *----------------------------- ... </pre> </div></div> <p>rci of UNDO RECORD of 0x1 is 0x00. That means this is the first and last UNDO RECORD.</p> <p>Object ID in this UNDO RECORD is 125212.</p> <p>SQL&gt; select owner,object_name,object_type from dba_objects where object_id in (125213,125212);</p> <p>OWNER<br/> ------------------------------<br/> OBJECT_NAME<br/> -------------------------------------------------------------------------------- <br/> OBJECT_TYPE<br/> -------------------<br/> OWBRUN<br/> SM_POST_IND1<br/> INDEX</p> <p>It's another lucky object that its type is INDEX. This index dropped. Now, after being sure that there is no new active transactions in this UNDO segment, the followings were done:</p> <ul class="alternate" type="square"> <li>Shutdown the database</li> <li>Set the following parameter to PFILE/SPFILE:</li> </ul> <p> _smu_debug_mode=4<br/> _offline_rollback_segments=(_SYSSMU85$)</p> <ul class="alternate" type="square"> <li>Startup the database</li> <li>drop rollback segment "_SYSSMU85$";</li> </ul> <p>After UNDO segment is successfuly dropped, the INTERNAL parameters above should be removed. But, in our case, while dropping UNDO segment, although the current internal error (ORA-600) <span class="error">&#91;kturrur11&#93;</span>) disappeared; the another internal error (ORA-600 <span class="error">&#91;kddummy_blkchk&#93;</span>) was encountered. It's created as another issue as <a href="http://jira.ubtools.com/jira/browse/QA-34" title="ORA-00600 [kddummy_blkchk] while dropping UNDO segment."><del>QA-34</del></a>.</p> <p>Since all objects needing recovery in the UNDO segment were dropped, there is no need to re-create the database after using _<em>offline_rollback_segments</em> parameter.</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-35">QA-35</a>)</td> </tr> <tr> <td>Edited by:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a></td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10214#action_10214 RE: [QA-35] ORA-00600 [kturrur11], [65535], [0]: Instance crashed. http://jira.ubtools.com/jira/browse/QA-35?focusedCommentId=10213#action_10213 Thu, 8 Nov 2007 11:17:03 +0000 ubTools Support <b>From Oracle Note:4940513.8:</b> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Bug 4940513 OERI[kturrur11] can occur with multi block undo This note gives a brief overview of bug 4940513. Affects: Product (Component) Oracle Server (Rdbms) Range of versions believed to be affected Versions &lt; 11 Versions confirmed as being affected * 9.2.0.6 * 9.2.0.7 * 10.1.0.4 * 10.1.0.5 * 10.2.0.2 Platforms affected Generic (all / most platforms affected) Fixed: This issue is fixed in * 9.2.0.8 (Server Patch Set) * 10.2.0.3 (Server Patch Set) * 11g (Future version) Symptoms: Related To: * Internal Error May Occur (ORA-600) * ORA-600 [kturrur11] * (None Specified) Description In rare situations the server could raise ORA-600 [kturrur11][65535][0] Workaround: Avoid the multi block undo code path by making sure that the block size in the undo tablespace is large enough to accomodate the largest column that is changed by any SQL statement. If the block size of the data tablespaces is larger than the block size of the undo tablespace, increase the blocksize of the undo tablespace to that of the data tablespace. </pre> </div></div> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-35">QA-35</a>)</td> </tr> <tr> <td>Edited by:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a></td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-35?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10213#action_10213 RE: [QA-34] ORA-00600 [kddummy_blkchk] while dropping UNDO segment. http://jira.ubtools.com/jira/browse/QA-34?focusedCommentId=10212#action_10212 Thu, 8 Nov 2007 08:03:56 +0000 ubTools Support Since the UNDO segment doesn't exist, the most <b>probably</b> its type is converted to TEMP. After setting the following event in the SPFILE/PFILE, the problem disappeared. <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>event="10061 trace name context forever, level 10" </pre> </div></div> <p>This event disables SMON from cleaning temp segment.</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-34">QA-34</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10212#action_10212 RE: [QA-29] ORA-600 [2845] while selecting, Invalid ROWID. http://jira.ubtools.com/jira/browse/QA-29?focusedCommentId=10087#action_10087 Sun, 15 Jul 2007 18:14:14 +0000 ubTools Support <h4><a name="Errorcode%3A"></a>Error code:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ORA-00600: internal error code, arguments: [2845], [0], [30], [0], [], [], [], [] </pre> </div></div> <h4><a name="Errorcodedefinition%3A"></a>Error code definition:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oracle is reading a range of blocks from a database file. If the starting block number or file number is 0, or the file number is greater than can be accommodated in the SGA (DB_FILES), error ORA-600 [2845] is raised. Ref: Metalink Note: 31057.1 ORA-600 [2845] "Read of bad DBA Requested" </pre> </div></div> <h4><a name="Cursordump%3A"></a>Cursor dump:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>******************** Cursor Dump ************************ Current cursor: 30, pgadep: 0 pgactx: 14a415b8 ctxcbk: 0 ctxqbc: 14a41990 qbcrws: 14a40ef0 Cursor Dump: ---------------------------------------- ... ---------------------------------------- Cursor 30 (1400ea8e0): CURFETCH curiob: 14019be40 curflg: 46 curpar: 0 curusr: c curses 8a2f38 cursor name: SELECT "NOTE" FROM "MEDIX"."PAT_SES" WHERE "ROWID"=:1 child pin: 5485b30, child lock: 54f2960, parent lock: 54c5f68 xscflg: 80110676, parent handle: 14a46070 nxt: 3.0x00000018 nxt: 2.0x000007d8 nxt: 1.0x000004e0 Cursor frame allocation dump: frm: -------- Comment -------- Size Seg Off bind 0: dty=1 mxl=32(18) mal=00 scl=00 pre=00 oacflg=01 bfp=140192748 bln=18 avl=18 flg=05 value="00000000.0000.0000" ---------------------------------------- ... </pre> </div></div> <h4><a name="Problemexplanation%3A"></a>Problem explanation:</h4> <p>As seen in the cursor dump above, the current cursor number is 30. The cursor#30 has a bind variable using a ROWID. But, the value of this bind variable is "00000000.0000.0000". In Oracle7, this ROWID points to block#0, slot#0, file#0. This is wrong.</p> <h4><a name="Recommendation%3A"></a>Recommendation:</h4> <ul class="alternate" type="square"> <li>Check application against any possible wrong ROWID usage.</li> <li>Call Oracle Support.</li> </ul> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-29">QA-29</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-29?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10087#action_10087 RE: [QA-28] ORA-00600 [729]: UGA memory leak. http://jira.ubtools.com/jira/browse/QA-28?focusedCommentId=10086#action_10086 Sun, 15 Jul 2007 18:04:37 +0000 ubTools Support <h4><a name="Errorcode%3A"></a>Error code:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ORA-00600: internal error code, arguments: [729], [480], [space leak], [], [], [], [], [] </pre> </div></div> <h4><a name="Errorcodedefinition%3A"></a>Error code definition:</h4> <p>A space leak has been detected in the User Global Area (UGA). There is no data corruption as a result of this error. It is an internal memory housekeeping problem. Second argument is the number of bytes leaked.</p> <h4><a name="UGAHeapdump%3A"></a>UGA Heap dump:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>******** ERROR: UGA memory leak detected 480 ******** ****************************************************** HEAP DUMP heap name="session heap" desc=0x222bd6f4 extent sz=0x108c alt=32767 het=32767 rec=0 flg=3 opc=3 parent=212550 owner=ad83d50 nex=0 xsz=0x108c EXTENT 0 Chunk 2330b100 sz= 3844 free " " EXTENT 1 Chunk 232f5174 sz= 516 free " " EXTENT 2 Chunk 236f0050 sz= 4176 free " " EXTENT 3 Chunk 236d0050 sz= 1228 free " " EXTENT 4 Chunk 236f18e4 sz= 1280 free " " EXTENT 5 Chunk 23307098 sz= 4228 free " " EXTENT 6 Chunk 2330a27c sz= 3696 free " " EXTENT 7 Chunk 23308130 sz= 1008 free " " Chunk 23308520 sz= 480 freeable "define var info" Chunk 23308700 sz= 2740 free " " EXTENT 8 Chunk 23306214 sz= 2832 perm "perm " alo=2832 Chunk 23306d24 sz= 864 free " " EXTENT 9 Chunk 233091c8 sz= 4228 free " " EXTENT 10 Chunk 232f405c sz= 612 free " " EXTENT 11 ... </pre> </div></div> <h4><a name="Problemexplanation%3A"></a>Problem explanation:</h4> <p>As seen in the UGA heap dump, there is a freeable chunk of <em>define var info</em> memory type. This chunk looks leaked.</p> <h4><a name="Workaround%3A"></a>Workaround:</h4> <p>There is no data corruption in this error, and can be safely ignore for small memory leaks by adding the following event to init.ora:</p> <ul class="alternate" type="square"> <li>event = "10262 trace name context forever, level 500"</li> </ul> <p>Then, restart your database. This event disables space leaks less than 500 bytes.</p> <p>You can see the details at Metalink Note:31056.1 ORA-600 <span class="error">&#91;729&#93;</span> "UGA Space Leak"</p> <h4><a name="Bug%3A"></a>Bug:</h4> <p>Bug:2177050: ORA-600 <span class="error">&#91;729&#93;</span> after application of the 8.1.7.3 patchset. The resulting trace file will include a memory dump which shows unfreed memory chunks with the tags "define var info" and/or "oactoid info".<br/> Ref: Metalink Note:31056.1 ORA-600 <span class="error">&#91;729&#93;</span> "UGA Space Leak"</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-28">QA-28</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10086#action_10086 RE: [QA-27] ORA-00600 [kcbgcur_1] by PQ operation. http://jira.ubtools.com/jira/browse/QA-27?focusedCommentId=10085#action_10085 Sun, 15 Jul 2007 17:59:03 +0000 ubTools Support <h4><a name="Errorcode%3A"></a>Error code:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ORA-00600: internal error code, arguments: [kcbgcur_1], [], [], [], [], [], [], [] </pre> </div></div> <h4><a name="Oraclekernelfunctionfromwhichtheproblemisraised%3A"></a>Oracle kernel function from which the problem is raised:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>kcbgcur(). This function is a function of Oracle Cache Layer. </pre> </div></div> <h4><a name="Undoblockdump%3A"></a>Undo block dump:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>UNDO BLK: xid: 0x0005.05e.000000c4 seq: 0x8c cnt: 0x31 irb: 0x19 icl: 0x0 flg: 0x0000 Rec Offset Rec Offset Rec Offset Rec Offset Rec Offset --------------------------------------------------------------------------- 0x01 0x1f38 0x02 0x1e88 0x03 0x1de4 0x04 0x1d3c 0x05 0x1c94 0x06 0x1bf4 0x07 0x1b54 0x08 0x1ac4 0x09 0x1a20 0x0a 0x1978 0x0b 0x18d4 0x0c 0x1820 0x0d 0x1784 0x0e 0x16e0 0x0f 0x1638 0x10 0x1598 0x11 0x14e8 0x12 0x1448 0x13 0x13a4 0x14 0x1308 0x15 0x126c 0x16 0x11d0 0x17 0x112c 0x18 0x1084 0x19 0x0fe0 0x1a 0x0f0c 0x1b 0x0e60 0x1c 0x0db8 0x1d 0x0d28 0x1e 0x0c90 0x1f 0x0bf0 0x20 0x0b28 0x21 0x0a88 0x22 0x09ec 0x23 0x0950 0x24 0x08ac 0x25 0x0814 0x26 0x077c 0x27 0x06e4 0x28 0x0650 0x29 0x05b4 0x2a 0x0524 0x2b 0x0480 0x2c 0x03f4 0x2d 0x035c 0x2e 0x02c0 0x2f 0x0230 0x30 0x01a0 0x31 0x0108 ... *----------------------------- * Rec #0x19 slt: 0x5e objn: 0(0x00000000) objd: 0 tblspc: 0(0x00000000) * Layer: 11 (Row) opc: 1 rci 0x18 Undo type: Regular undo Last buffer split: No Temp Object: No Tablespace Undo: No rdba: 0x00000000 *----------------------------- KDO undo record: KTB Redo op: 0x02 ver: 0x01 op: C uba: 0x00c0083d.008c.18 KDO Op code: IRP xtype: XA bdba: 0x0040760a hdba: 0x004075d9 itli: 1 ispac: 0 maxfr: 4863 tabn: 0 slot: 130(0x82) size/delt: 56 fb: --H-FL-- lb: 0x0 cc: 4 null: ---- col 0: [ 3] 37 34 34 col 1: [20] 45 6c 65 63 74 72 6f 6e 69 63 20 73 74 72 75 63 74 75 72 65 col 2: [ 0] col 3: [ 0] *----------------------------- ... </pre> </div></div> <h4><a name="Problemexplanation%3A"></a>Problem explanation:</h4> <p>irb points the first undo record in undo block to begin rollback. So, the record 0x19 is your first undo record. The object number of the block, and the object number of the block undo applied to are 0. I think this may be your problem. Oracle may not be able to know the real object number during this rollback.</p> <h4><a name="Bug%3A"></a>Bug:</h4> <p>It looks like:</p> <ul class="alternate" type="square"> <li>Bug:984947 A PARALLEL QUERY SLAVE GOT ORA-600<span class="error">&#91;KCBGCUR_1&#93;</span></li> </ul> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-27">QA-27</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10085#action_10085 RE: [QA-26] ORA-00600 [12700] by SNP process. http://jira.ubtools.com/jira/browse/QA-26?focusedCommentId=10084#action_10084 Sun, 15 Jul 2007 17:54:22 +0000 ubTools Support <h4><a name="Errorcode%3A"></a>Error code:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ORA-00600: internal error code, arguments: [12700], [62], [4202128], [133], [], [], [], [] </pre> </div></div> <h4><a name="CurrentSQLstatementforthissession%3A"></a>Current SQL statement for this session:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SELECT source from source$ WHERE obj# =:1 ORDER BY line </pre> </div></div> <h4><a name="Oraclekernelfunctionfromwhichtheproblemisraised%3A"></a>Oracle kernel function from which the problem is raised:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rtbhiopn(). </pre> </div></div> <h4><a name="Errorcodedefinition%3A"></a>Error code definition:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oracle is trying to access a row using its ROWID, which has been obtained from an index. A mismatch was found between the index rowid and the data block it is pointing to. The rowid points to a non-existent row in the data block. The corruption can be in data and/or index blocks. ORA-600 [12700] can also be reported due to a consistent read (CR) problem. The information dumped to the trace file varies greatly between releases: - in Oracle 7.3.x it is ORA-600 [12700][a1][a2] , where Arg [a1] dba (Data Block Address) Arg [a2] slot number (number of the row in the block pointed by the dba) - in Oracle 8.x and 9.x, it is ORA-600 [12700][a1][a2][a3] , where Arg [a1] dataobj# from sys.obj$ Arg [a2] relative dba of the data block Arg [a3] slot number of the row in the data block Details: Metalink Note:28229.1 ORA-600 [12700] "Index entry Points to Missing ROWID" </pre> </div></div> <h4><a name="Errorcodeinterpretation%3A"></a>Error code interpretation:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Argument Dec Hex ---------- ---------- ---------- [62] 62 0x3E [4202128] 4202128 0x401E90 [133] 133 0x85 This problem is related to the slot#133 of the rdba#4202128 of the object#62. </pre> </div></div> <h4><a name="Indexblockdump%3A"></a>Index block dump:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Block header dump: rdba: 0x00401ede Object id on Block? Y seg/obj: 0x63 csc: 0x00.2fbe43 itc: 2 flg: - typ: 2 - INDEX fsl: 0 fnx: 0x0 ver: 0x01 Itl Xid Uba Flag Lck Scn/Fsc 0x01 0x0012.01a.00000130 0x008027fb.000e.02 C--- 0 scn 0x0000.0000c71c 0x02 0x0002.013.00000768 0x00807cf2.4f11.08 --U- 217 fsc 0x0000.002fbe45 Leaf block dump =============== header address 74698844=0x473d05c kdxcolev 0 kdxcolok 0 kdxcoopc 0x80: opcode=0: iot flags=--- is converted=Y kdxconco 2 kdxcosdc 1 kdxconro 217 kdxcofbo 470=0x1d6 kdxcofeo 471=0x1d7 kdxcoavs 1 kdxlespl 0 kdxlende 0 kdxlenxt 4202207=0x401edf kdxleprv 4203488=0x4023e0 kdxledsz 6 kdxlecol 0 kdxlebksz 3940 row#0[471] flag: ----, lock: 2, data:(6): 00 40 1e 93 00 5b col 0; len 3; (3): c2 27 11 col 1; len 3; (3): c2 02 62 ... row#213[3876] flag: ----, lock: 2, data:(6): 00 40 1e 90 00 85 col 0; len 3; (3): c2 27 11 col 1; len 3; (3): c2 05 0b row#214[3892] flag: ----, lock: 2, data:(6): 00 40 1e 90 00 86 col 0; len 3; (3): c2 27 11 col 1; len 3; (3): c2 05 0c ... </pre> </div></div> <h4><a name="Datablockdump%3A"></a>Data block dump:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Block header dump: rdba: 0x00401e90 Object id on Block? Y seg/obj: 0x3e csc: 0x00.2fbe43 itc: 1 flg: - typ: 1 - DATA fsl: 0 fnx: 0x0 ver: 0x01 Itl Xid Uba Flag Lck Scn/Fsc 0x01 0x0002.013.00000768 0x00807cf2.4f11.0b --U- 139 fsc 0x009b.002fbe45 data_block_dump =============== tsiz: 0xfb8 hsiz: 0x128 pbl: 0x04b43044 bdba: 0x00401e90 flag=--------- ntab=1 nrow=139 frre=-1 fsbo=0x128 fseo=0x2d4 avsp=0x111 tosp=0x23a 0xe:pti[0] nrow=139 offs=0 0x12:pri[0] offs=0xfb6 0x14:pri[1] offs=0xfb4 0x16:pri[2] offs=0xfb2 . 0x11a:pri[132] offs=0x382 0x11c:pri[133] sfll=0 0x11e:pri[134] sfll=0 0x120:pri[135] sfll=0 0x122:pri[136] sfll=0 0x124:pri[137] sfll=0 0x126:pri[138] sfll=0 block_row_dump: tab 0, row 0, @0xfb6 tl: 2 fb: --HDFL-- lb: 0x1 tab 0, row 1, @0xfb4 tl: 2 fb: --HDFL-- lb: 0x1 . tab 0, row 132, @0x382 tl: 42 fb: --H-FL-- lb: 0x1 cc: 3 col 0: [ 3] c2 27 11 col 1: [ 3] c2 05 0a col 2: [30] 09 09 09 09 09 09 6c 5f 6e 65 78 74 48 6f 6c 64 44 65 73 69 72 65 4e 75 6d 62 65 72 2c 0a end_of_block_dump </pre> </div></div> <h4><a name="Problemexplanation%3A"></a>Problem explanation:</h4> <p>As seen in the index block dump, kdxledsz is 6. That means this index is a unique B*Tree index which uses restricted ROWID format in 6 bytes. The first 4 bytes are used for rdba, and the last 2 bytes are used for slot#.</p> <p>This internal error code had returned 0x401E90 for the rdba, and 0x85 for the slot#. The restricted ROWID in the index dump has to be the combination of them. So, it's 0x00401E900085. This restricted ROWID is available in the index dump.</p> <p>The pri[] field shows slot# of rows in data block. In this error, the returned slot# is 133. But, as seen in the data block dump, there is no row allocated for this slot. The max slot# in the block dump is 132.</p> <p>Although there is a value in the index block, there is no matching row in the data block. The data block looks corrupted.</p> <h4><a name="Workaround%3A"></a>Workaround:</h4> <p>The most probably the object#62 is source$. Restore SYSTEM tablespace from the backup, and recover it.</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-26">QA-26</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-26?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10084#action_10084 RE: [QA-25] ORA-00600 [kkslgop1] in SELECT when CURSOR_SHARING IS NOT EXACT. http://jira.ubtools.com/jira/browse/QA-25?focusedCommentId=10083#action_10083 Sun, 15 Jul 2007 17:42:45 +0000 ubTools Support <h4><a name="Errorcode%3A"></a>Error code:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ORA-00600: internal error code, arguments: [kkslgop1], [], [], [], [], [], [], [] </pre> </div></div> <h4><a name="CurrentSQLstatementforthissession%3A"></a>Current SQL statement for this session:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SELECT COMP_TIME FROM CSTMAPSTATUS WHERE CSTID = :"SYS_B_0" AND SLOTNO = :"SYS_B_1" </pre> </div></div> <h4><a name="Oraclekernelfunctionfromwhichtheproblemisraised%3A"></a>Oracle kernel function from which the problem is raised:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>kkslgop(). This is a function of Oracle Compilation Layer. </pre> </div></div> <h4><a name="Processstate%3A"></a>Process state:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PROCESS STATE ------------- ... ---------------------------------------- SO: 404c6264, type: 3, owner: 403cde98, pt: 0, flag: INIT/-/-/0x00 (session) trans: 40e83928, creator: 403cde98, flag: (8000041) USR/- BSY/-/-/-/-/- DID: 0001-0014-00000002, short-term DID: 0000-0000-00000000 txn branch: 40f8201c oct: 3, prv: 0, user: 24/APPMGR O/S info: user: Administrator, term: CIMMB, ospid: 219:228, machine: PDP1_MES_DOM\CIMMB program: TIME_GAP.exe last wait for 'SQL*Net message from dblink' blocking sess=0x0 seq=60687 wait_time=-1 driver id=54435000, #bytes=1, =0 ---------------------------------------- ... </pre> </div></div> <h4><a name="Problemexplanation%3A"></a>Problem explanation:</h4> <p>As you see in your SQL statement, your bind variables are system generated bind variables. In other words, cursor sharing is enabled in your database.</p> <p>Also, as seen in the process state, your last wait event is <em>SQL*Net message from dblink</em>. That means a dblink operation had been done before.</p> <h4><a name="Workaround%3A"></a>Workaround:</h4> <p>Use cursor_sharing=exact</p> <h4><a name="Bug%3A"></a>Bug:</h4> <ul class="alternate" type="square"> <li>Bug:2169897 ORA-600 ARGUMENTS: <span class="error">&#91;KKSLGOP1&#93;</span> VIA SELECT ACROSS DB_LINK</li> </ul> <ul class="alternate" type="square"> <li>Bug:2159152 CURSOR_SHARING=FORCE MAY NOT SHARE STATEMENTS USING VIEWS IN 8172/8173<br/> Back to top </li> </ul> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-25">QA-25</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10083#action_10083 RE: [QA-23] ORA-00600 [15851] while creating unique index. http://jira.ubtools.com/jira/browse/QA-23?focusedCommentId=10082#action_10082 Sun, 15 Jul 2007 17:31:15 +0000 ubTools Support <h4><a name="Errorcode%3A"></a>Error code:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ORA-00600: internal error code, arguments: [15851], [8], [8], [1], [2], [], [], [] </pre> </div></div> <h4><a name="Oraclekernelfunctionfromwhichtheproblemisraised%3A"></a>Oracle kernel function from which the problem is raised:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>srsqb1nx(). </pre> </div></div> <h4><a name="Problemexplanation%3A"></a>Problem explanation:</h4> <p>Most probably, this is a sort problem while creating index.</p> <h4><a name="Bug%3A"></a>Bug:</h4> <p>Metalink Note:1032586.6 ORA-600 <span class="error">&#91;15851&#93;</span></p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-23">QA-23</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-23?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10082#action_10082 RE: [QA-22] ORA-00600 [13004] while creating index. http://jira.ubtools.com/jira/browse/QA-22?focusedCommentId=10081#action_10081 Sun, 15 Jul 2007 17:27:43 +0000 ubTools Support <h4><a name="Errorcode%3A"></a>Error code:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ORA-00600: internal error code, arguments: [13004], [], [], [], [], [], [], [] </pre> </div></div> <h4><a name="Oraclekernelfunctionfromwhichtheproblemisraised%3A"></a>Oracle kernel function from which the problem is raised:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>kkrirop(). This is a function of Oracle Compilation Layer. </pre> </div></div> <h4><a name="Bug%3A"></a>Bug:</h4> <p>Bug:994802 CREATE INDEX RESULTS IN ORA-600 <span class="error">&#91;13004&#93;</span></p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-22">QA-22</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10081#action_10081 RE: [QA-21] ORA-07445 [11]: SMON crashed. http://jira.ubtools.com/jira/browse/QA-21?focusedCommentId=10080#action_10080 Sun, 15 Jul 2007 17:23:28 +0000 ubTools Support <h4><a name="Errorcode%3A"></a>Error code:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> ORA-07445: exception encountered: core dump [11] [3221212616] [240] [0] [] [] </pre> </div></div> <h4><a name="Oraclekernelfunctionfromwhichtheproblemisraised%3A"></a>Oracle kernel function from which the problem is raised:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> kdb4_dup_keys(). This is a function of Oracle Data Layer. </pre> </div></div> <h4><a name="Cursordump%3A"></a>Cursor dump:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> ******************** Cursor Dump ************************ Current cursor: 1, pgadep: 1 pgactx: c00000014e736d90 ctxcbk: c00000014e776720 ctxqbc: 0 ctxrws: c00000014e7253c8 Cursor Dump: ---------------------------------------- Cursor 1 (80000001000befe8): CURBOUND curiob: 80000001000c1358 curflg: 5 curpar: 0 curusr: 0 curses c00000012c18a070 cursor name: delete from uet$ where ts#=:1 and segfile#=:2 and segblock#=:3 and ext#=:4 child pin: c00000013511e670, child lock: c00000013511d630, parent lock: c00000013511d6a0 xscflg: 20100466, parent handle: c00000014e748b20, xscfl2: 5100400 nxt: 3.0x00000560 nxt: 2.0x000005e0 nxt: 1.0x000005e0 Cursor frame allocation dump: frm: -------- Comment -------- Size Seg Off bind 0: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=08 oacfl2=1 size=24 offset=0 bfp=80000001000d2470 bln=22 avl=02 flg=05 value=1 bind 1: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=08 oacfl2=1 size=24 offset=0 bfp=80000001000d2440 bln=24 avl=02 flg=05 value=2 bind 2: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=08 oacfl2=1 size=24 offset=0 bfp=80000001000d2410 bln=24 avl=02 flg=05 value=2 bind 3: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=08 oacfl2=1 size=24 offset=0 bfp=80000001000d23e0 bln=24 avl=02 flg=05 value=59 End of cursor dump </pre> </div></div> <h4><a name="Recomendation%3A"></a>Recomendation:</h4> <p>Check if sys.uet$ is corrupted.</p> <h4><a name="Bug%3A"></a>Bug:</h4> <p>Bug:2106455 SMON CRASHES WITH ORA-07445 IN KDB4_DUP_KEYS</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-21">QA-21</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-21?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10080#action_10080 RE: [QA-20] ORA-00600 [723]: Memory leak in LGWR. http://jira.ubtools.com/jira/browse/QA-20?focusedCommentId=10079#action_10079 Sun, 15 Jul 2007 17:17:28 +0000 ubTools Support <h4><a name="Errorcode%3A"></a>Error code:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> ORA-00600: internal error code, arguments: [723], [5200], [5200], [memory leak], [], [], [], [] </pre> </div></div> <h4><a name="Oraclekernelfunctionfromwhichtheproblemisraised%3A"></a>Oracle kernel function from which the problem is raised:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> ksmdpg() Deallocate variable PGA. Just free top PGA heap, the callback will free. the extents to the OSD. Ref: Bug:1283286 </pre> </div></div> <h4><a name="Processstate%3A"></a>Process state:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> PROCESS STATE ------------- Process global information: process: 0, call: 0, xact: 0, curses: 0, usrses: 0 No process is allocated. END OF PROCESS STATE </pre> </div></div> <h4><a name="PGAHeapdump%3A"></a>PGA Heap dump:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> ******** ERROR: PGA memory leak detected 5200 &gt; 3616 ******** ****************************************************** HEAP DUMP heap name="pga heap" desc=0x40003190 extent sz=0x2148 alt=40 het=32767 rec=0 flg=3 opc=3 parent=0 owner=0 nex=0 xsz=0x2148 EXTENT 0 Chunk 400de8e0 sz= 4432 free " " Chunk 400dfa30 sz= 256 freeable "LGWR PIC bds ar" Chunk 400dfb30 sz= 896 freeable "LGWR PIC ins ar" Chunk 400dfeb0 sz= 896 freeable "LGWR PIC ins ar" Chunk 400e0230 sz= 568 free " " Chunk 400e0468 sz= 896 freeable "LGWR PIC ins ar" Chunk 400e07e8 sz= 568 free " " EXTENT 1 . </pre> </div></div> <h4><a name="Problemexplanation%3A"></a>Problem explanation:</h4> <p>As seen above and included in your trace, the memory class of some chunks are "LGWR PIC ins ar" and similar. If you notice that sum of them is 5200 bytes, and they are freeable chunks. These chunks are leaked.</p> <p>Also, there is no allocated process for LGWR. The most probably, you are closing the database.</p> <h4><a name="Workaround%3A"></a>Workaround:</h4> <p>There is no data corruption in this error, and can be safely ignore for small memory leaks by adding the following event to init.ora:</p> <ul class="alternate" type="square"> <li>event = "10262 trace name context forever, level 6000"</li> </ul> <p>Then, restart your database. This event disables space leaks less than 6000 bytes.</p> <p>You can see the details at Metalink Note:39308.1 ORA-600 <span class="error">&#91;723&#93;</span> "PGA memory leak"</p> <h4><a name="Bug%3A"></a>Bug:</h4> <p>Bug:1125724 ORA-600<span class="error">&#91;723&#93;</span> DURING SHUTDOWN</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-20">QA-20</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-20?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10079#action_10079 RE: [QA-11] How to see the tasks of Oracle background processes ? http://jira.ubtools.com/jira/browse/QA-11?focusedCommentId=10073#action_10073 Sun, 15 Jul 2007 13:29:28 +0000 ubTools Support <h4><a name="Answer%3A"></a>Answer:</h4> <p>Use the following query:</p> <p>select substr(DEST,1,10) DEST, DESCRIPTION from x$messages order by DEST;</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> DEST DESCRIPTION ---------- ---------------------------------------------------------------- * Monitor Cleanup * KSB action for X-instance calls * generic shutdown background * Scumnt mount lock * database close in progress * Poll system events broadcast channel * svr actn for shrd grp reg/dereg ARB* ASM to slave BG msg ARC* Archiver wakeup ARCH Archiver Message ARCH Archiver shutdown DEST DESCRIPTION ---------- ---------------------------------------------------------------- CJQ* Shutdown Job Queue Process CJQ* Job Queue Interupt CJQ* Job Queue Interupt CJQ* Job Queue Interupt CJQ* Job Queue Timout CJQ0 Check for async messages from other instances CJQ0 Coordinator send broadcast timeout CKPT create/scrub cmon foregrounds CKPT perform RM action in CKPT CKPT identify control file CKPT close control file DEST DESCRIPTION ---------- ---------------------------------------------------------------- CKPT release (XR,4,0) enqueue CKPT CKPT stat update timeout action CKPT CKPT reuse call completion action CKPT CKPT reuse range call continuation CKPT CKPT reuse call continuation CKPT refresh control file CKPT check for parameters from other instances CKPT start background CKPT CPU dynamic reconfiguration CKPT check for quiesce messages CKPT unquiesce the instance during database close DEST DESCRIPTION ---------- ---------------------------------------------------------------- CKPT unsubscribe to quiesce channel CKPT subscribe to quiesce channel CKPT Get Proxy Lock CKPT Db Checkpt Compl check CKPT Db Checkpt Request check CKPT update recovery-based i/o statistics CKPT Compile Environment Monitor CKPT SQL Memory Management Calculation CKPT free PX memory chunks in background CKPT KKX: drop ncomp dll action CKPT Flashback barrier DEST DESCRIPTION ---------- ---------------------------------------------------------------- CKPT hold alert level CKPT recovery area alert action CKPT start change tracking in ckpt CKPT get (XR,4,0) enqueue CKPT sense a heartbeat CKPT set heartbeat sensing CKPT emulate i/o errors on a disk CKPT timeout CKPT Run self test on group CKPT asynchronously dismount disk group CKPT dismount disk group DEST DESCRIPTION ---------- ---------------------------------------------------------------- CKPT query disk group status CKPT check disk status CKPT update disk status CKPT update disk group status CKPT kfc CKPT dismount disk group CKPT kfc CKPT mount disk group CTWR change tracking message CTWR change tracking timeout action DBW* hardware clock went backwards DBW* DBWR write buffers DBW* get/release open thread enqueue DEST DESCRIPTION ---------- ---------------------------------------------------------------- DBW* mount/dismount all db files DBW0 SGA memory tuning parameter update - DBW0 DBW0 Db mount lock DBW0 kfcb Poke DBW0 DBW0 kfc mount disk group DBW0 kfc dismount disk group DBW0 kfc invalidate file extent DBW0 Reserve lock name space lock DBW0 Release lock name space lock DBW0 complete Release space call DBW0 verify/invalidate all db files DEST DESCRIPTION ---------- ---------------------------------------------------------------- DBW0 recovery db file verification DBW0 identify db file DBW0 close and unlock db file DBW0 lock db file DBW0 offline db file DBW0 Db File check DBW0 Message to flush IMU txns DBW0 Db Instance Lock Mgmt DIAG write trace records out DIAG Clusterwise dump request DIAG poradebug commands DEST DESCRIPTION ---------- ---------------------------------------------------------------- DIAG write trace records out DIAG write trace records out DIAG write trace records out DMON DMON Wakeup DMON DMON shutdown DMON DMON Verify Standby shutdown for PM violation DMON Standby site request resync DMON Metadata file available DMON DMON rcv NS status DMON DMON Receive Message DMON DMON Disable DRC DEST DESCRIPTION ---------- ---------------------------------------------------------------- DMON DMON Interrupt Routine INSV INSV Wakeup INSV NetSlave Shutdown Message INSV INSV Receive Message LCK0 ksim LCK0 functions LCK0 ksim reg/dereg instance group LCK0 ksim query instance group LCK0 ksim polling interrupt action LCK0 KSXR remote instance died LCK0 KSXR finialize LCK0 kxfp signal recv function DEST DESCRIPTION ---------- ---------------------------------------------------------------- LCK0 get and hold global enqueue LCK0 perform a user instance lock operation LCK0 SMON purge object number cache LCK0 KQLM interrupt action LCK0 KQLM invalidation instance lock operation LCK0 KQLM pin instance lock operation LCK0 KQR timeout action LCK0 KQR get instance lock LCK0 sequence bckgrnd instance lock LCK0 release TS enq for sort segment LCK0 kea signal recv function DEST DESCRIPTION ---------- ---------------------------------------------------------------- LCK0 get TS enq for sort segment LCK0 release quiesce enqueue LCK0 get quiesce enqueue LCK0 KCL lock affinity timeout action LCK0 Check SCN adjust LCK0 Cross-instance broadcast message LCK0 ksim get value LGWR LGWR failure LGWR kfr ACD relocation LGWR kfr Incr Ckpt LGWR kfr Poke LGWR DEST DESCRIPTION ---------- ---------------------------------------------------------------- LGWR kfr Dismount disk group LGWR kfr mount disk group LGWR LGWR to Start DMON LGWR free KTU instance lock LGWR convert KTU instance lock LGWR get KTU instance lock LGWR dml_locks = 0 global enforcement LGWR Open/close/mount/dismount thread LGWR Redo writer generate offline immed marker LGWR Redo writer log switch operations LGWR LGWR re-eval standby locks DEST DESCRIPTION ---------- ---------------------------------------------------------------- LGWR Redo writer interrupt action LGWR Redo writer IO's LMD* Flush side-channel msgs LMD LNS* Network Server wakeup LNS* Network Server forced LNS* Network Server shutdown LNS* Network Server reinit MMAN lock memory at startup MMAN Memory Management MMAN Handle sga_target resize MMAN Reset advisory pool when advisory turned ON DEST DESCRIPTION ---------- ---------------------------------------------------------------- MMAN Complete deferred initialization of components MMAN lock memory timeout action MMNL tune undo retention MMNL MMNL Periodic MQL Selector MMNL ASH Sampler (KEWA) MMNL MMON SWRF Raw Metrics Capture MMON reload failed KSPD callbacks MMON SGA memory tuning MMON background recovery area alert action MMON Flashback Marker MMON tablespace alert monitor DEST DESCRIPTION ---------- ---------------------------------------------------------------- MMON UNDO MMON ACTION MMON MMON Local action Listener MMON MMON Remote action Listener MMON Advisor delete expired tasks MMON ASH Emergency Flusher (KEWA) MMON MMON SWRF Auto DBFUS Task MMON MMON SWRF Auto Purge Task MMON MMON SWRF Auto Flush Task MMON alert message purge MMON alert message cleanup MMON Check for sync messages from other instances DEST DESCRIPTION ---------- ---------------------------------------------------------------- MMON ADDM (KEH) MMON threshold reconciliation MMON metrics monitoring MMON shutdown MMON MMON run-once action driver MMON MMON testing slave MMON MMON testing action MMON MMON Completion Callback Dispatcher MMON Job Autostart action force MMON Coordinator autostart timeout MMON Check for autostart messages from other instances DEST DESCRIPTION ---------- ---------------------------------------------------------------- MMON Compute cache stats in background MMON undo usage MMON recovery area alert action MMON SGA memory tuning parameter update MMON reconfiguration MMON action NSV* NetSlave Wakeup Message NSV* NetSlave Receive Message NSV* NetSlave Metadata Resync NSV* NetSlave Health Check Message NSV* NetSlave Shutdown Message NSV* NetSlave request Primary to resync DEST DESCRIPTION ---------- ---------------------------------------------------------------- NSV* NetSlave Check DRC version QMNC Shutdown Q Monitor Coord RBAL ASM to master BG msg RBAL BG load lib msg RBAL|SMON OSM to BG mesg RECO distributed recovery wakeup RECO distributed recovery shutdown RSM* RSM Wakeup RSM* RSM Receive Message RSM* RSM Receive Message Response RVWR Open/close flashback thread DEST DESCRIPTION ---------- ---------------------------------------------------------------- RVWR RVWR IO's SMON kfcl instance recovery TEST Reliable Test Dummy Call 212 rows selected. SQL&gt; </pre> </div></div> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-11">QA-11</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10073#action_10073 RE: [QA-8] Heapdump Interpretation http://jira.ubtools.com/jira/browse/QA-8?focusedCommentId=10071#action_10071 Sun, 15 Jul 2007 13:03:34 +0000 ubTools Support <h4><a name="Answer%3A"></a>Answer:</h4> <p>A heap consists of memory areas named extent. Each extent consists of memory areas named chunks.</p> <h4><a name="Interpretation%3A"></a>Interpretation:</h4> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> EXTENT 437 Chunk 925dfe4 sz= 1836 perm "perm " alo=1836 Chunk 925e710 sz= 1156 recreate "session heap " latch=0 EXTENT 437 ---&gt; extent number 925dfe4 ----&gt; chunk address sz= -----&gt; size of chunk perm ------&gt; permanent memory class "perm " ------&gt; chunk comment </pre> </div></div> <p>Memory classes can be the followings:</p> <ul class="alternate" type="square"> <li>Recreatable (can be removed and then recreated when requested. i.e: shared SQL statements)</li> <li>Free (free, no object in it)</li> <li>Freeable(used in session/call duration)</li> <li>Permanent(for permament objects)</li> </ul> <p>Each chunk in same extent is contiguous. For your case, the first chunk address(0x925dfe4) + its size(1836) = the second chunk address (0x925e710)</p> <h4><a name="Foryourproblem%3A"></a>For your problem:</h4> <p>Shared memory segments such as SGA are included in process address space. So, You may be encoutering this problem. Search metalink for pmap command.</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-8">QA-8</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=10071#action_10071