ubTools Support http://jira.ubtools.com/jira/secure/IssueNavigator.jspa?reset=true&pid=10042&resolution=-1&priority=3&sorter/field=issuekey&sorter/order=DESC An XML representation of a search request en-us [QA-56] "cursor: mutex X" http://jira.ubtools.com/jira/browse/QA-56 <style type="text/css"> * { border: 0; padding: 0; } .tableBorder, .grid { background-color: #fff; width: 100%; border-collapse: collapse; } h3.formtitle { } .tableBorder td, .grid td { vertical-align: top; padding: 2px; border: 1px solid #cccccc; border-collapse: collapse; } .noPadding { padding: 0px !important; } h3 .subText { font-size: 60%; font-weight: normal; } .tabLabel { font-weight: bold; border-top: 1px solid #cccccc; border-right: 1px solid #cccccc; border-left: 1px solid #cccccc; padding: 2px; border-collapse: collapse; display: inline; } td.blank { padding: 0; margin: 0; } .blank td { border: none; } #descriptionArea { margin: 0px; padding: 2px; border: 1px solid #cccccc; } hr.fullcontent { height: 15px; padding: 10px 0px; background: #fff url('http://jira.ubtools.com/jira/images/icons/hr.gif') no-repeat scroll center; } body { margin: 0px; font-size: 12px; font-family: Arial, Sans-Serif, sans-serif; color:black; } </style> <table class="tableBorder" cellpadding="0" cellspacing="0" border="0" width="100%"> <tr> <td bgcolor="#f0f0f0" width="100%" colspan="2" valign="top"> <h3 class="formtitle"> [QA-56]&nbsp;<a href="http://jira.ubtools.com/jira/browse/QA-56">&quot;cursor: mutex X&quot;</a> <span class="subText"> Created: 27/Jul/13 &nbsp;Updated: 28/Jul/13 </span> </h3> </td> </tr> <tr> <td width="20%"><b>Status:</b></td> <td width="80%">Open</td> </tr> <tr> <td width="20%"><b>Project:</b></td> <td width="80%"><a href="http://jira.ubtools.com/jira/secure/BrowseProject.jspa?id=10042">Questions &amp; Answers</a></td> </tr> <tr> <td><b>Fix Version/s:</b></td> <td> None </td> </tr> </table> <br /> <table class="grid" cellpadding="0" cellspacing="0" border="0" width="100%"> <tr> <td bgcolor="#f0f0f0" valign="top" width="20%"> <b>Type:</b> </td> <td bgcolor="#ffffff" valign="top" width="30%" > Oracle - Database Tuning </td> <td bgcolor="#f0f0f0"> <b>Priority:</b> </td> <td bgcolor="#ffffff" valign="top" nowrap> Major </td> </tr> <tr> <td bgcolor="#f0f0f0" valign="top" width="20%"> <b>Reporter:</b> </td> <td bgcolor="#ffffff" valign="top" width="30%" > <a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> </td> <td bgcolor="#f0f0f0" width="20%"> <b>Assignee:</b> </td> <td bgcolor="#ffffff" valign="top" nowrap width="30%" > <a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> </td> </tr> <tr> <td bgcolor="#f0f0f0" width="20%"> <b>Resolution:</b> </td> <td bgcolor="#ffffff" valign="top" width="30%" nowrap> Unresolved </td> <td bgcolor="#f0f0f0" width="20%"> <b>Votes:</b> </td> <td bgcolor="#ffffff" valign="top" width="30%" nowrap> 0 </td> </tr> </table> <br /> <table class="grid" cellpadding="0" cellspacing="0" border="0" width="100%"> <tr> <td bgcolor="#f0f0f0" width="20%" valign="top"><b>Product Version:</b></td> <td bgcolor="#ffffff" width="80%"> 11.2.0.3.7 </td> </tr> <tr> <td bgcolor="#f0f0f0" width="20%" valign="top"><b>Operating System:</b></td> <td bgcolor="#ffffff" width="80%"> Solaris </td> </tr> <tr> <td bgcolor="#f0f0f0" width="20%" valign="top"><b>Host Name:</b></td> <td bgcolor="#ffffff" width="80%"> . </td> </tr> <tr> <td bgcolor="#f0f0f0" width="20%" valign="top"><b>Database Name:</b></td> <td bgcolor="#ffffff" width="80%"> . </td> </tr> </table> <br/> <table cellpadding="2" cellspacing="0" border="0" width="100%" align="center"> <tr> <td bgcolor="#bbbbbb" width="1%" nowrap align="center"> &nbsp;<font color="#ffffff"><b>Description</b></font>&nbsp; </td> <td>&nbsp;</td> </tr> </table> <table cellpadding="0" cellspacing="0" border="0" width="100%"> <tr> <td id="descriptionArea"> The customer has upgraded from Oracle 10.2.0.5 to Oracle 11.2.0.3.7. They encountered <em>library cache lock</em> problem. <p>Some excerpt from the AWR:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Elapsed: 15.01 (mins) DB Time: 6,548.72 (mins) ..... Top 5 Timed Foreground Events Event Waits Time(s) Avg wait (ms) % DB time Wait Class library cache lock 425 364,186 856909 92.69 Concurrency enq: TX - row lock contention 288 11,500 39930 2.93 Application TCP Socket (KGAS) 136,317 9,552 70 2.43 Network DB CPU 4,167 1.06 db file sequential read 742,541 2,176 3 0.55 User I/O </pre> </div></div> <br/> </td> </tr> </table> <br/> <table cellpadding="2" cellspacing="0" border="0" width="100%" align="center"> <tr> <td bgcolor="#bbbbbb" width="1%" nowrap align="center"> &nbsp;<font color="#ffffff"><b>Comments</b></font>&nbsp; </td> <td>&nbsp;</td> </tr> </table> <table cellpadding="0" cellspacing="0" border="0" width="100%" class="grid" style="margin: 0;"> <tr id="comment-header-22743"><td bgcolor="#f0f0f0"> Comment by <a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> <font size="-2"> [ <font color="#336699">27/Jul/13 10:42 AM</font> ] </font> </td></tr> <tr id="comment-body-22743"><td bgcolor="#ffffff"> Some excerpt from the HANGANALYZE trace: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Chain 1: ------------------------------------------------------------------------------- Oracle session identified by: { instance: 1 (opusdata.opusdata) os id: 25465 process id: 256, oracle@&lt;hostname&gt; session id: 39 session serial #: 61345 } is waiting for 'library cache lock' with wait info: { p1: 'handle address'=0x22038dff68 p2: 'lock address'=0x21ef361978 p3: '100*mode+namespace'=0x520002 time in wait: 5 min 33 sec timeout after: never wait id: 5128 blocking: 0 sessions wait history: * time between current wait and wait #1: 0.001507 sec 1. event: 'SQL*Net message from client' time waited: 0.003166 sec wait id: 5127 p1: 'driver id'=0x54435000 p2: '#bytes'=0x1 * time between wait #1 and #2: 0.000002 sec 2. event: 'SQL*Net message to client' time waited: 0.000002 sec wait id: 5126 p1: 'driver id'=0x54435000 p2: '#bytes'=0x1 * time between wait #2 and #3: 0.000011 sec 3. event: 'SQL*Net message from client' time waited: 0.003906 sec wait id: 5125 p1: 'driver id'=0x54435000 p2: '#bytes'=0x1 } and is blocked by =&gt; Oracle session identified by: { instance: 1 (opusdata.opusdata) os id: 22379 process id: 500, oracle@&lt;hostname&gt; session id: 8008 session serial #: 62080 } which is waiting for 'cursor: mutex X' with wait info: { p1: 'idn'=0xd4d88873 p2: 'value'=0x1 p3: 'where'=0x400000000 time in wait: 0.000000 sec heur. time in wait: 1 min 3 sec timeout after: never wait id: 7092018 blocking: 426 sessions wait history: * time between current wait and wait #1: 0.000007 sec 1. event: 'cursor: mutex X' time waited: 0.000002 sec wait id: 7092017 p1: 'idn'=0xd4d88873 p2: 'value'=0x1 p3: 'where'=0x400000000 * time between wait #1 and #2: 0.000009 sec 2. event: 'cursor: mutex X' time waited: 0.000002 sec wait id: 7092016 p1: 'idn'=0xd4d88873 p2: 'value'=0x1 p3: 'where'=0x400000000 * time between wait #2 and #3: 0.000007 sec 3. event: 'cursor: mutex X' time waited: 0.000003 sec wait id: 7092015 p1: 'idn'=0xd4d88873 p2: 'value'=0x1 p3: 'where'=0x400000000 } Chain 1 Signature: 'cursor: mutex X'&lt;='library cache lock' Chain 1 Signature Hash: 0xfbcb6c60 ..... process id: 3136, oracle@&lt;hostname&gt; ..... is waiting for 'library cache lock' with wait info: ..... and is blocked by 'instance: 1, os id: 22379, session id: 8008', </pre> </div></div> <p>There are many other sessions waiting on <em>library cache lock</em> in the trace. And, they are blocked by SID#8008 which is waiting on <em>cursor: mutex X</em>. SID#8008 blocks 426 sessions.</p> </td></tr> <tr id="comment-header-22744"><td bgcolor="#f0f0f0"> Comment by <a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> <font size="-2"> [ <font color="#336699">27/Jul/13 01:03 PM</font> ] </font> </td></tr> <tr id="comment-body-22744"><td bgcolor="#ffffff"> Unfortunately, the blocker session SID#8008 exited. But, a new blocker SID#7579 appeared. Some excerpt from its ERRORSTACK LEVEL 3 trace: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>KGX Atomic Operation Log 3c6bb9388 Mutex 22c1df2c18(7579, 0) idn d4d88873 oper EXCL Cursor Parent uid 7579 efd 15 whr 1 slp 0 oper=OPERATION_DEFAULT pt1=0 pt2=0 pt3=0 pt4=0 u41=0 stt=0 KGX Atomic Operation Log 3c6bb93d8 Mutex 22c1df2d30(0, 1) idn d4d88873 oper GET_EXCL hash table uid 7579 efd 15 whr 4 slp 36782 oper=OPERATION_DEFAULT pt1=0 pt2=0 pt3=0 pt4=0 u41=0 stt=0 </pre> </div></div> <p>Mutex IDN 0xd4d88873 is hold in EXCL mode at mutex address 0x22c1df2c18. The holder SID is 7579. The mutex type is <em>Cursor Parent</em><br/> The same mutex IDN is requested in EXCL(oper GET_EXCL) mode at mutex address 0x22c1df2d30 and waited. The mutex type is <em>hash table</em><br/> See Oracle note <em>Understanding and Reading Systemstates (Doc ID 423153.1)</em> for interpreting.</p> <p>Waiting session holds the same mutex IDN in EXCL mode but in the different mutex address and the different mutex type. No other holder encountered in the SYSTEMSTATE trace.</p> </td></tr> <tr id="comment-header-22750"><td bgcolor="#f0f0f0"> Comment by <a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> <font size="-2"> [ <font color="#336699">28/Jul/13 01:26 PM</font> ] </font> </td></tr> <tr id="comment-body-22750"><td bgcolor="#ffffff"> Some excerpt from AWR: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Foreground Wait Events ..... Event Waits %Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn % DB time ..... cursor: mutex X 8,066,619 0 835 0 138.17 0.21 ..... Mutex Sleep Summary ordered by number of sleeps desc Mutex Type Location Sleeps Wait Time (ms) hash table kkshhcdel [KKSHBKLOC4] 8,061,510 0 </pre> </div></div> <p>The mutex sleep location is <em>kkshhcdel <span class="error">&#91;KKSHBKLOC4&#93;</span></em>. Nothing found about it in Metalink.</p> </td></tr> <tr id="comment-header-22759"><td bgcolor="#f0f0f0"> Comment by <a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> <font size="-2"> [ <font color="#336699">28/Jul/13 05:33 PM</font> ] </font> </td></tr> <tr id="comment-body-22759"><td bgcolor="#ffffff"> <b>Workaround:</b> <p>Change the SQL text.</p> </td></tr> <tr id="comment-header-22760"><td bgcolor="#f0f0f0"> Comment by <a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> <font size="-2"> [ <font color="#336699">28/Jul/13 05:36 PM</font> ] </font> </td></tr> <tr id="comment-body-22760"><td bgcolor="#ffffff"> The customer will open an SR to Oracle Support. I'll update this issue later with SR result. </td></tr> </table> Sun, 28 Jul 2013 17:36:08 +0000 http://jira.ubtools.com/jira/browse/QA-56 ubTools Support