Tuesday 10 May 2016

XAER_RMERR : A resource manager error has occured in the transaction branch oracle.jdbc.xa.OracleXAException

========================================================================
Applies To:
WLS - 10.3.6.0.11
OSB - 11.1.1.7.4
========================================================================
Problem:
We are not able to dequeue messages from AQ becuase of XAER_RMERR : A resource manager error has occured in the transaction branch oracle.jdbc.xa.OracleXAException.
If the server is restarted, it will work fine for some time. Then the exception will occur again and service stops working.

Error:
Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_RMERR start() failed on resource 'testDS': XAER_RMERR : A resource manager error has occured in the transaction branch  oracle.jdbc.xa.OracleXAException

Error obtaining database connection 'testDS': Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_RMERR start() failed on resource 'testDS': XAER_RMERR : A resource manager error has occured in the transaction branch
weblogic.transaction.internal.ResourceAccessException: Transaction has timed out when making request to XAResource 'testDS'.
at weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResourceDescriptor.java:697)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1249)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1183)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:286)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:561)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:488)
at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1706)
at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1610)

========================================================================
Cause:
WLS is not cleaning pending XA transactions.

========================================================================
Resolution:
We need to prepare the database for XA.Please execute below steps on DB.
connect to database as SYS user
@xaview.sql  ,the xaview.sql script resides in the $ORACLE_HOME/rdbms/admin directory
grant all on dba_2pc_pending to public;
grant all on dba_pending_transactions to public;
grant all on dba_2pc_neighbors to public;
grant select on pending_trans$ to public;
grant select on v$xatrans$ to public;

If the above steps are not performed on the database server, normal XA database queries and updates may work fine. However, when the Weblogic Server Transaction Manager performs recovery on a re-boot after a crash, recover for the Oracle resource will fail with XAER_RMERR. Crash recovery is a standard operation for an XA resource.

Also check for inactive sessions and CPU on DB side. If there are any inactive sessions clear them.
Rule for XA timeout is Global JTA Timeout (at WebLogic server level) < XA Transaction Timeout (at Data Source level) < DISTRIBUTED_LOCK_TIMEOUT (parameter in the Database)