ubTools Support http://jira.ubtools.com/jira/secure/IssueNavigator.jspa?reset=true&pid=10042&resolution=-1&assigneeSelect=specificuser&assignee=support&sorter/field=priority&sorter/order=DESC An XML representation of a search request en-us RE: [QA-56] "cursor: mutex X" http://jira.ubtools.com/jira/browse/QA-56?focusedCommentId=22760#action_22760 Sun, 28 Jul 2013 17:36:08 +0000 ubTools Support The customer will open an SR to Oracle Support. I'll update this issue later with SR result. <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-56">QA-56</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=22760#action_22760 RE: [QA-56] "cursor: mutex X" http://jira.ubtools.com/jira/browse/QA-56?focusedCommentId=22759#action_22759 Sun, 28 Jul 2013 17:33:59 +0000 ubTools Support <b>Workaround:</b> <p>Change the SQL text.</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-56">QA-56</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=22759#action_22759 RE: [QA-56] "cursor: mutex X" http://jira.ubtools.com/jira/browse/QA-56?focusedCommentId=22744#action_22744 Sun, 28 Jul 2013 17:25:58 +0000 ubTools Support Unfortunately, the blocker session SID#8008 exited. But, a new blocker SID#7579 appeared. Some excerpt from its ERRORSTACK LEVEL 3 trace: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>KGX Atomic Operation Log 3c6bb9388 Mutex 22c1df2c18(7579, 0) idn d4d88873 oper EXCL Cursor Parent uid 7579 efd 15 whr 1 slp 0 oper=OPERATION_DEFAULT pt1=0 pt2=0 pt3=0 pt4=0 u41=0 stt=0 KGX Atomic Operation Log 3c6bb93d8 Mutex 22c1df2d30(0, 1) idn d4d88873 oper GET_EXCL hash table uid 7579 efd 15 whr 4 slp 36782 oper=OPERATION_DEFAULT pt1=0 pt2=0 pt3=0 pt4=0 u41=0 stt=0 </pre> </div></div> <p>Mutex IDN 0xd4d88873 is hold in EXCL mode at mutex address 0x22c1df2c18. The holder SID is 7579. The mutex type is <em>Cursor Parent</em><br/> The same mutex IDN is requested in EXCL(oper GET_EXCL) mode at mutex address 0x22c1df2d30 and waited. The mutex type is <em>hash table</em><br/> See Oracle note <em>Understanding and Reading Systemstates (Doc ID 423153.1)</em> for interpreting.</p> <p>Waiting session holds the same mutex IDN in EXCL mode but in the different mutex address and the different mutex type. No other holder encountered in the SYSTEMSTATE trace.</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-56">QA-56</a>)</td> </tr> <tr> <td>Edited by:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a></td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=22744#action_22744 RE: [QA-56] "cursor: mutex X" http://jira.ubtools.com/jira/browse/QA-56?focusedCommentId=22750#action_22750 Sun, 28 Jul 2013 13:26:35 +0000 ubTools Support Some excerpt from AWR: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Foreground Wait Events ..... Event Waits %Time -outs Total Wait Time (s) Avg wait (ms) Waits /txn % DB time ..... cursor: mutex X 8,066,619 0 835 0 138.17 0.21 ..... Mutex Sleep Summary ordered by number of sleeps desc Mutex Type Location Sleeps Wait Time (ms) hash table kkshhcdel [KKSHBKLOC4] 8,061,510 0 </pre> </div></div> <p>The mutex sleep location is <em>kkshhcdel <span class="error">&#91;KKSHBKLOC4&#93;</span></em>. Nothing found about it in Metalink.</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-56">QA-56</a>)</td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=22750#action_22750 RE: [QA-56] "cursor: mutex X" http://jira.ubtools.com/jira/browse/QA-56?focusedCommentId=22743#action_22743 Sat, 27 Jul 2013 13:04:54 +0000 ubTools Support Some excerpt from the HANGANALYZE trace: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Chain 1: ------------------------------------------------------------------------------- Oracle session identified by: { instance: 1 (opusdata.opusdata) os id: 25465 process id: 256, oracle@&lt;hostname&gt; session id: 39 session serial #: 61345 } is waiting for 'library cache lock' with wait info: { p1: 'handle address'=0x22038dff68 p2: 'lock address'=0x21ef361978 p3: '100*mode+namespace'=0x520002 time in wait: 5 min 33 sec timeout after: never wait id: 5128 blocking: 0 sessions wait history: * time between current wait and wait #1: 0.001507 sec 1. event: 'SQL*Net message from client' time waited: 0.003166 sec wait id: 5127 p1: 'driver id'=0x54435000 p2: '#bytes'=0x1 * time between wait #1 and #2: 0.000002 sec 2. event: 'SQL*Net message to client' time waited: 0.000002 sec wait id: 5126 p1: 'driver id'=0x54435000 p2: '#bytes'=0x1 * time between wait #2 and #3: 0.000011 sec 3. event: 'SQL*Net message from client' time waited: 0.003906 sec wait id: 5125 p1: 'driver id'=0x54435000 p2: '#bytes'=0x1 } and is blocked by =&gt; Oracle session identified by: { instance: 1 (opusdata.opusdata) os id: 22379 process id: 500, oracle@&lt;hostname&gt; session id: 8008 session serial #: 62080 } which is waiting for 'cursor: mutex X' with wait info: { p1: 'idn'=0xd4d88873 p2: 'value'=0x1 p3: 'where'=0x400000000 time in wait: 0.000000 sec heur. time in wait: 1 min 3 sec timeout after: never wait id: 7092018 blocking: 426 sessions wait history: * time between current wait and wait #1: 0.000007 sec 1. event: 'cursor: mutex X' time waited: 0.000002 sec wait id: 7092017 p1: 'idn'=0xd4d88873 p2: 'value'=0x1 p3: 'where'=0x400000000 * time between wait #1 and #2: 0.000009 sec 2. event: 'cursor: mutex X' time waited: 0.000002 sec wait id: 7092016 p1: 'idn'=0xd4d88873 p2: 'value'=0x1 p3: 'where'=0x400000000 * time between wait #2 and #3: 0.000007 sec 3. event: 'cursor: mutex X' time waited: 0.000003 sec wait id: 7092015 p1: 'idn'=0xd4d88873 p2: 'value'=0x1 p3: 'where'=0x400000000 } Chain 1 Signature: 'cursor: mutex X'&lt;='library cache lock' Chain 1 Signature Hash: 0xfbcb6c60 ..... process id: 3136, oracle@&lt;hostname&gt; ..... is waiting for 'library cache lock' with wait info: ..... and is blocked by 'instance: 1, os id: 22379, session id: 8008', </pre> </div></div> <p>There are many other sessions waiting on <em>library cache lock</em> in the trace. And, they are blocked by SID#8008 which is waiting on <em>cursor: mutex X</em>. SID#8008 blocks 426 sessions.</p> <br/> <br/> <table> <tr> <td>Author:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a> (<a href="http://jira.ubtools.com/jira/browse/QA-56">QA-56</a>)</td> </tr> <tr> <td>Edited by:</td> <td><a href="http://jira.ubtools.com/jira/secure/ViewProfile.jspa?name=support">ubTools Support</a></td> </tr> </table> http://jira.ubtools.com/jira/browse/QA-56?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=22743#action_22743