java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread (lifecycle.c:1097)

========================================================================
Applies To:
Linux Version:  OEL-6.5
WLS - 10.3.6
OSB - 11.1.1.7
========================================================================
Problem:
OSB becomes unavailable/unresponsive under heavy load.

Error:
java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread (lifecycle.c:1097).

Attempting to allocate 8096M bytes

There is insufficient native memory for the Java
Runtime Environment to continue.

Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit

Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Disable compressed references (-XXcompressedRefs=false)


========================================================================
Cause:
/etc/security/limits.conf file nproc value is overwritten by /etc/security/limits.d/90-nproc.conf and setting max user processes 1024

========================================================================
Resolution:
We need to change nproc value in /etc/security/limits.d/90-nproc.conf

Default Values:
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     1024
root       soft    nproc     unlimited

New Values:
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     unlimited
root       soft    nproc     unlimited

Weblogic admin fails to start because of javax.naming.ServiceUnavailableException [Root exception is java.rmi.NoSuchObjectException

========================================================================
Applies To:
WLS - 10.3.6.0.11
OSB - 11.1.1.7.4
========================================================================
Problem:
We are unable to start weblogic admin server becuase of javax.naming.ServiceUnavailableException [Root exception is java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.]

Error:
####<May 30, 2016 6:34:56 AM CDT> <Info> <messaging.interception> <purushottam> <poc_osb_admin> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1464608096553> <BEA-400000> <Initializing message interception service>
####<May 30, 2016 6:34:56 AM CDT> <Error> <Configuration Management> <purushottam> <poc_osb_admin> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1464608096834> <BEA-150000> <An error occurred while establishing a connection back to the adminstration server t3://192.168.184.130:6609 during startup. Since bootstrap succeeded check that t3://192.168.184.130:6609 uniquely identifies the administration server.
javax.naming.ServiceUnavailableException [Root exception is java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.]
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:70)
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:32)
        at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:792)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:431)
        at weblogic.jndi.Environment.getInitialReference(Environment.java:245)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:174)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:147)
        at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:88)
        at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:174)
        at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:154)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.
        at weblogic.rmi.internal.OIDManager.getServerReference(OIDManager.java:251)
        at weblogic.rmi.extensions.server.ServerHelper.getLocalInitialReference(ServerHelper.java:476)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:131)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:124)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:427)
        at weblogic.jndi.Environment.getInitialReference(Environment.java:245)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:174)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:160)
        at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:89)
        at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:175)
        at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:155)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
>
####<May 30, 2016 6:34:56 AM CDT> <Critical> <WebLogicServer> <purushottam> <poc_osb_admin> <Main Thread> <<WLS Kernel>> <> <> <1464608096842> <BEA-000362> <Server failed. Reason:

There are 1 nested errors:

javax.naming.ServiceUnavailableException [Root exception is java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.]
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:70)
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:32)
        at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:792)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:431)
        at weblogic.jndi.Environment.getInitialReference(Environment.java:245)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:174)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:147)
        at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:88)
        at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:174)
        at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:154)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.
        at weblogic.rmi.internal.OIDManager.getServerReference(OIDManager.java:251)
        at weblogic.rmi.extensions.server.ServerHelper.getLocalInitialReference(ServerHelper.java:476)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:131)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:124)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:427)
        at weblogic.jndi.Environment.getInitialReference(Environment.java:245)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:174)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:160)
        at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:89)
        at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:175)
        at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:155)
        ... 3 more

>
####<May 30, 2016 6:34:56 AM CDT> <Notice> <WebLogicServer> <purushottam> <poc_osb_admin> <Main Thread> <<WLS Kernel>> <> <> <1464608096863> <BEA-000365> <Server state changed to FAILED>
####<May 30, 2016 6:34:56 AM CDT> <Error> <WebLogicServer> <purushottam> <poc_osb_admin> <Main Thread> <<WLS Kernel>> <> <> <1464608096863> <BEA-000383> <A critical service failed. The server will shut itself down>
####<May 30, 2016 6:34:56 AM CDT> <Notice> <WebLogicServer> <purushottam> <poc_osb_admin> <Main Thread> <<WLS Kernel>> <> <> <1464608096867> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

========================================================================
Cause:
Weblogic admin server listener address was empty,
<listen-address></listen-address>

========================================================================
Resolution:
1. Go to $DOMAIN_HOME/config/config.xml file
2. Edit config.xml and set valid listen-address for admin server.
3. Restart the server.

Weblogic Admin server fails to start Reason:java.security.AccessControlException: access denied (oracle.security.jps.JpsPermission idstore.config)

========================================================================
Applies To:
WLS - 10.3.6.0.11
OSB - 11.1.1.7.4
========================================================================
Problem:
We are unable to start weblogic admin server becuase of weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception.

Error:
####<May 30, 2016 6:25:36 AM CDT> <Critical> <WebLogicServer> <purushottam> <poc_osb_dit_admin> <Main Thread> <<WLS Kernel>> <> <> <1464607536237> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-03027: PDPService service instance failed during initialization due to lack of grant. If you are seeing this error after a new installation, make sure you have updated weblogic.policy as installation step suggested
weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-03027: PDPService service instance failed during initialization due to lack of grant. If you are seeing this error after a new installation, make sure you have updated weblogic.policy as installation step suggested
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1402)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)
        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: oracle.security.jps.JpsRuntimeException: JPS-03027: PDPService service instance failed during initialization due to lack of grant. If you are seeing this error after a new installation, make sure you have updated weblogic.policy as installation step suggested
        at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPDPService(PolicyUtil.java:2978)
        at oracle.security.jps.internal.policystore.PolicyUtil.getPDPService(PolicyUtil.java:3226)
        at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:167)
        at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:383)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at java.lang.Class.newInstance0(Class.java:357)
        at java.lang.Class.newInstance(Class.java:310)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1343)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)
        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.security.AccessControlException: access denied (oracle.security.jps.JpsPermission idstore.config)
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:374)
        at java.security.AccessController.checkPermission(AccessController.java:549)
        at oracle.security.jps.util.JpsAuth$AuthorizationMechanism$3.checkPermission(JpsAuth.java:463)
        at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:523)
        at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:549)
        at oracle.security.jps.internal.api.identitystore.IdentityStoreConfigurationUtil.checkPermission(IdentityStoreConfigurationUtil.java:548)
        at oracle.security.jps.internal.api.identitystore.IdentityStoreConfigurationUtil.getIgfLdapSpecifiedProperties(IdentityStoreConfigurationUtil.java:378)
        at oracle.security.jps.internal.api.identitystore.IdentityStoreConfigurationUtil.getLibOvdLdapPushData(IdentityStoreConfigurationUtil.java:526)
        at oracle.security.jps.internal.igf.ovd.OvdIGFServiceProvider$1.run(OvdIGFServiceProvider.java:232)
        at oracle.security.jps.internal.igf.ovd.OvdIGFServiceProvider$1.run(OvdIGFServiceProvider.java:229)
        at oracle.security.jps.internal.igf.ovd.OvdIGFServiceProvider.getConfigData(OvdIGFServiceProvider.java:228)
        at oracle.security.jps.internal.igf.ovd.OvdIGFServiceProvider.prepareDataPush2libOvd(OvdIGFServiceProvider.java:365)
        at oracle.security.jps.internal.idstore.ldap.LdapIdentityStoreProvider$NoLibOvd.getInstance(LdapIdentityStoreProvider.java:339)
        at oracle.security.jps.internal.idstore.ldap.LdapIdentityStoreProvider.getInstance(LdapIdentityStoreProvider.java:118)
        at oracle.security.jps.internal.idstore.ldap.LdapIdentityStoreProvider.getInstance(LdapIdentityStoreProvider.java:74)
        at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.findServiceInstance(ContextFactoryImpl.java:139)
        at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.getContext(ContextFactoryImpl.java:170)
        at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.getContext(ContextFactoryImpl.java:191)
        at oracle.security.jps.internal.core.runtime.JpsContextFactoryImpl.getContext(JpsContextFactoryImpl.java:132)
        at oracle.security.jps.internal.core.runtime.JpsContextFactoryImpl.getContext(JpsContextFactoryImpl.java:127)
        at oracle.security.jps.internal.policystore.PolicyUtil$3.run(PolicyUtil.java:2957)
        at oracle.security.jps.internal.policystore.PolicyUtil$3.run(PolicyUtil.java:2950)
        at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPDPService(PolicyUtil.java:2950)
        at oracle.security.jps.internal.policystore.PolicyUtil.getPDPService(PolicyUtil.java:3228)
        at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:167)
        at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:383)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at java.lang.Class.newInstance0(Class.java:357)
        at java.lang.Class.newInstance(Class.java:310)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1343)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1024)
        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:894)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
>
####<May 30, 2016 6:25:36 AM CDT> <Notice> <WebLogicServer> <purushottam> <poc_osb_dit_admin> <Main Thread> <<WLS Kernel>> <> <> <1464607536482> <BEA-000365> <Server state changed to FAILED>
####<May 30, 2016 6:25:36 AM CDT> <Error> <WebLogicServer> <purushottam> <poc_osb_dit_admin> <Main Thread> <<WLS Kernel>> <> <> <1464607536482> <BEA-000383> <A critical service failed. The server will shut itself down>
####<May 30, 2016 6:25:36 AM CDT> <Notice> <WebLogicServer> <purushottam> <poc_osb_dit_admin> <Main Thread> <<WLS Kernel>> <> <> <1464607536498> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

========================================================================
Cause:
Weblogic Admin server file $DOMAIN_HOME/config/fmwconfig/system-jazn-data.xml wasn't having execute permissions.
-rw-rw-r--  1 oracle oracle        132K May 30  2016 system-jazn-data.xml

========================================================================
Resolution:
Kindly provide 775 permissions to $DOMAIN_HOME/config/fmwconfig/system-jazn-data.xml and restart the server.

XAER_RMERR : A resource manager error has occured in the transaction branch oracle.jdbc.xa.OracleXAException

========================================================================
Applies To:
WLS - 10.3.6.0.11
OSB - 11.1.1.7.4
========================================================================
Problem:
We are not able to dequeue messages from AQ becuase of XAER_RMERR : A resource manager error has occured in the transaction branch oracle.jdbc.xa.OracleXAException.
If the server is restarted, it will work fine for some time. Then the exception will occur again and service stops working.

Error:
Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_RMERR start() failed on resource 'testDS': XAER_RMERR : A resource manager error has occured in the transaction branch  oracle.jdbc.xa.OracleXAException

Error obtaining database connection 'testDS': Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_RMERR start() failed on resource 'testDS': XAER_RMERR : A resource manager error has occured in the transaction branch
weblogic.transaction.internal.ResourceAccessException: Transaction has timed out when making request to XAResource 'testDS'.
at weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResourceDescriptor.java:697)
at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1249)
at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1183)
at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:286)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:561)
at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:488)
at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1706)
at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1610)

========================================================================
Cause:
WLS is not cleaning pending XA transactions.

========================================================================
Resolution:
We need to prepare the database for XA.Please execute below steps on DB.
connect to database as SYS user
@xaview.sql  ,the xaview.sql script resides in the $ORACLE_HOME/rdbms/admin directory
grant all on dba_2pc_pending to public;
grant all on dba_pending_transactions to public;
grant all on dba_2pc_neighbors to public;
grant select on pending_trans$ to public;
grant select on v$xatrans$ to public;

If the above steps are not performed on the database server, normal XA database queries and updates may work fine. However, when the Weblogic Server Transaction Manager performs recovery on a re-boot after a crash, recover for the Oracle resource will fail with XAER_RMERR. Crash recovery is a standard operation for an XA resource.

Also check for inactive sessions and CPU on DB side. If there are any inactive sessions clear them.
Rule for XA timeout is Global JTA Timeout (at WebLogic server level) < XA Transaction Timeout (at Data Source level) < DISTRIBUTED_LOCK_TIMEOUT (parameter in the Database)

Managed Server failed to start with Reason: [Management:141266]Parsing Failure in config.xml

========================================================================
Applies To:
WLS - 10.3.6.0.11
OSB - 11.1.1.7.4
========================================================================
Problem:
Unable to start managed server where as admin server came up fine with no issues.

Error:
<Apr 19, 2016 3:08:48 AM CDT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: [Management:141266]Parsing Failure in config.xml: failed to find method MethodName{methodName='setCacheInAppDirectory', paramTypes=[boolean]} on class weblogic.management.configuration.AppDeploymentMBeanImpl>
<Apr 19, 2016 3:08:48 AM CDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Apr 19, 2016 3:08:48 AM CDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<Apr 19, 2016 3:08:48 AM CDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
<Apr 19, 2016 3:08:58 AM> <INFO> <poc_osb_dev> <poc_osb_dev_ms01> <Server failed during startup so will not be restarted>
Apr 19, 2016 3:08:58 AM weblogic.nodemanager.server.AbstractServerManager log
INFO: Server failed during startup so will not be restarted

========================================================================
Cause:
WLS patches applied to admin host and managed server host were different.

Admin host:
WebLogic Server Temporary Patch for BUG20448343 Fri Feb 06 12:39:35 IST 2015 WebLogic Server 10.3.6.0.11 PSU Patch for BUG20181997 TUE NOV 12 15:54:42 IST 2015 WebLogic Server 10.3.6.0 Tue Nov 15 08:52:36 PST 2011 1441050

Managed server host:
WebLogic Server Temporary Patch for BUG14747231 Wed Dec 19 00:41:27 MST 2012 WebLogic Server Temporary Patch for BUG14030073 Tue Jul 31 11:44:25 IST 2012 WebLogic Server Temporary Patch for BUG13961904 Sat Oct 13 07:42:54 CDT 2012  WebLogic Server Temporary Patch for 13573621 Wed Jan 18 18:53:18 IST 2012  WebLogic Server Temporary Patch for BUG14322089 Wed Jul 25 16:00:29 IST 2012  WebLogic Server 10.3.6.0  Tue Nov 15 08:52:36 PST 2011 1441050
========================================================================
Resolution:
We need to apply same WLS patches to managed server host as like admin server and restart the server.
This should resolve your issue.

BAD_CERTIFICATE alert was received (java.lang.RuntimeException: Cannot convert identity certificate)

========================================================================
Applies To:
WLS - 10.3.6
OSB - 11.1.1.7
========================================================================
Problem:
Services which are having HTTPS/SSL endpoints aren't working properly.

Error:
<<Mar 29, 2016 3:48:10 PM CDT> <Warning> <Security> <puru-host> <poc_osb_admin> <ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <ecba9489e228dd77:-11ecd046:1535560a37c:-7ffd-000000000000fe95> <1459284490478> <BEA-090482> <BAD_CERTIFICATE alert was received from XX.XX.XXX.XXX - XX.XX.XXX.XXX. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.>

Reason:
java.lang.RuntimeException: Cannot convert identity certificate
========================================================================
Cause:
Recently target system renewed their certificates and new certificates installed on downstream side which has SHA256RSA signature algorithm and old certificates were having SHA1RSA which was working fine.
Weblogic is having default SSL implementation(CERTICOM)
========================================================================
Resolution:
We need to enforce weblogic server to use JSSE if any cipher suite is unsupported by default SSL implementation(CERTICOM).
To change SSL implementation from CERTICOM to JSSE and resolve issue kindly follow below steps,
1. Navigate Environment -> servers -> [server name] -> Configuration -> SSL -> Advanced tab -> Enable Use JSSE SSL
2. Also we need to import new root and intermediate certificates from renewed cert if they aren't existed in CACERT
Login to each domain host and perform below,
cd $JAVA_HOME/jre/bin
To list CACERTS, use below command,
keytool -list -v -keystore $JAVA_HOME/jre/lib/security/cacerts
To import cert, kindly use,
./keytool -import -alias alias_name -keystore $JAVA_HOME/jre/lib/security/cacerts -trustcacerts -file $Cert_location/cert.pem
3. Restart servers for changes to take effect and test the service.

Weblogic Managed server fails to start due to java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

========================================================================
Applies To:
WLS - 10.3.6
========================================================================
Problem:
Weblogic Managed server fails to start with an exception java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

Error:
<Mar 15, 2016 8:24:18 AM CDT> <Notice> <Cluster> <BEA-000142> <Trying to download cluster JNDI tree from server managed_server1.>
<Mar 15, 2016 8:24:18 AM CDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
        at java.util.ArrayList.RangeCheck(ArrayList.java:547)
        at java.util.ArrayList.get(ArrayList.java:322)
        at weblogic.rmi.cluster.BasicReplicaList.getPrimary(BasicReplicaList.java:90)
        at weblogic.rmi.cluster.ClusterableRemoteRef.getPrimaryRef(ClusterableRemoteRef.java:155)
        at weblogic.rmi.cluster.ClusterableRemoteRef.add(ClusterableRemoteRef.java:474)
        Truncated. see log file for complete stacktrace
>
<Mar 15, 2016 8:24:18 AM CDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Mar 15, 2016 8:24:18 AM CDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<Mar 15, 2016 8:24:18 AM CDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
========================================================================
Cause:
As per logs downloading of cluster JNDI tree failed from managed_server1.
There can be possibility of network or communication issue to servers in cluster.
========================================================================
Resolution:
Restart of managed_server1 can fix the issue. If it doesn't help then go for complete domain restart.

Unable to start Admin server becuase of java.lang.IllegalArgumentException: 0 is not a legal value for UnicastListenPort

========================================================================
Applies To:
BPEL/OSB - 11g PS6 (11.1.1.7)
WLS - 10.3.6
Oracle DB - 11.2.0.4
========================================================================
Problem:
Unable to start admin server using NM because of java.lang.IllegalArgumentException: 0 is not a legal value for UnicastListenPort.  It does not fall in the range [1, 65535]
Port configured for coherence/WKA setting are in range [1,65535] . Coherences settings are configured as per oracle WKA settings document and they are fine.

Recently domain was extended.

Error:
<Mar 15, 2016 8:24:16 AM CDT> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Oracle JRockit(R) Version R28.2.2-7-148152-1.6.0_29-20111221-2104-linux-x86_64 from Oracle Corporation>
<Mar 15, 2016 8:24:16 AM CDT> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.6.0.3 PSU Patch for BUG14736139 Fri Nov 23 10:16:54 IST 2012
WebLogic Server 10.3.6.0  Tue Nov 15 08:52:36 PST 2011 1441050 >
<Mar 15, 2016 8:24:18 AM CDT> <Error> <J2EE> <BEA-160197> <Unable to load descriptor java.net.URLClassLoader@1490a66f/null of module null. The error is weblogic.descriptor.DescriptorException: Unmarshaller failed
        at weblogic.descriptor.internal.MarshallerFactory$1.createDescriptor(MarshallerFactory.java:161)
        at weblogic.descriptor.BasicDescriptorManager.createDescriptor(BasicDescriptorManager.java:323)
        at weblogic.application.descriptor.AbstractDescriptorLoader2.getDescriptorBeanFromReader(AbstractDescriptorLoader2.java:788)
        at weblogic.application.descriptor.AbstractDescriptorLoader2.createDescriptorBean(AbstractDescriptorLoader2.java:409)
        at weblogic.application.descriptor.AbstractDescriptorLoader2.loadDescriptorBeanWithoutPlan(AbstractDescriptorLoader2.java:759)
        at weblogic.application.descriptor.AbstractDescriptorLoader2.loadDescriptorBean(AbstractDescriptorLoader2.java:768)
        at weblogic.cacheprovider.coherence.CoherenceClusterDescriptorHelper.getCoherenceBean(CoherenceClusterDescriptorHelper.java:64)
        at weblogic.cacheprovider.coherence.CoherenceClusterDescriptorHelper.createCoherenceDescriptor(CoherenceClusterDescriptorHelper.java:32)
        at weblogic.management.mbeans.custom.CoherenceClusterSystemResource.loadDescriptor(CoherenceClusterSystemResource.java:202)
        at weblogic.management.mbeans.custom.ConfigurationExtension.getExtensionRoot(ConfigurationExtension.java:178)
        at weblogic.management.mbeans.custom.CoherenceClusterSystemResource.getCoherenceClusterResource(CoherenceClusterSystemResource.java:70)
        at weblogic.management.mbeans.custom.CoherenceClusterSystemResource._postCreate(CoherenceClusterSystemResource.java:174)
        at weblogic.management.configuration.CoherenceClusterSystemResourceMBeanImpl._postCreate(CoherenceClusterSystemResourceMBeanImpl.java:448)
        at weblogic.descriptor.internal.AbstractDescriptorBean._postCreate(AbstractDescriptorBean.java:671)
        at weblogic.management.configuration.DomainMBeanImpl.setCoherenceClusterSystemResources(DomainMBeanImpl.java:14964)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.bea.staxb.runtime.internal.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:48)
        at com.bea.staxb.runtime.internal.RuntimeBindingType$BeanRuntimeProperty.setValue(RuntimeBindingType.java:539)
        at com.bea.staxb.runtime.internal.AttributeRuntimeBindingType$QNameRuntimeProperty.fillCollection(AttributeRuntimeBindingType.java:382)
        at com.bea.staxb.runtime.internal.MultiIntermediary.getFinalValue(MultiIntermediary.java:52)
        at com.bea.staxb.runtime.internal.AttributeRuntimeBindingType.getFinalObjectFromIntermediary(AttributeRuntimeBindingType.java:140)
        at com.bea.staxb.runtime.internal.UnmarshalResult.unmarshalBindingType(UnmarshalResult.java:200)
        at com.bea.staxb.runtime.internal.UnmarshalResult.unmarshalDocument(UnmarshalResult.java:169)
        at com.bea.staxb.runtime.internal.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:65)
        at weblogic.descriptor.internal.MarshallerFactory$1.createDescriptor(MarshallerFactory.java:100)
        at weblogic.descriptor.BasicDescriptorManager.createDescriptor(BasicDescriptorManager.java:257)
        at weblogic.management.provider.internal.DescriptorManagerHelper.loadDescriptor(DescriptorManagerHelper.java:73)
        at weblogic.management.provider.internal.RuntimeAccessImpl$IOHelperImpl.parseXML(RuntimeAccessImpl.java:690)
        at weblogic.management.provider.internal.RuntimeAccessImpl.parseNewStyleConfig(RuntimeAccessImpl.java:270)
        at weblogic.management.provider.internal.RuntimeAccessImpl.<init>(RuntimeAccessImpl.java:115)
        at weblogic.management.provider.internal.RuntimeAccessService.start(RuntimeAccessService.java:42)
        at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)
        at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)
        at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:882)
        at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:572)
        at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
        at weblogic.Server.main(Server.java:71)
Caused by: com.bea.xml.XmlException: java.lang.IllegalArgumentException: 0 is not a legal value for UnicastListenPort.  It does not fall in the range [1, 65535]
        at com.bea.staxb.runtime.internal.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:54)
        at com.bea.staxb.runtime.internal.RuntimeBindingType$BeanRuntimeProperty.setValue(RuntimeBindingType.java:539)
        at com.bea.staxb.runtime.internal.ByNameRuntimeBindingType$ElementQNameProperty.fill(ByNameRuntimeBindingType.java:351)
        at com.bea.staxb.runtime.internal.LiteralUnmarshalResult.extractAndFillElementProp(LiteralUnmarshalResult.java:159)
        at com.bea.staxb.runtime.internal.ByNameUnmarshaller.deserializeContents(ByNameUnmarshaller.java:51)
        at com.bea.staxb.runtime.internal.AttributeUnmarshaller.unmarshalIntoIntermediary(AttributeUnmarshaller.java:47)
        at com.bea.staxb.runtime.internal.LiteralUnmarshalResult.unmarshalElementProperty(LiteralUnmarshalResult.java:184)
        at com.bea.staxb.runtime.internal.LiteralUnmarshalResult.extractAndFillElementProp(LiteralUnmarshalResult.java:156)
        at com.bea.staxb.runtime.internal.ByNameUnmarshaller.deserializeContents(ByNameUnmarshaller.java:51)
        at com.bea.staxb.runtime.internal.AttributeUnmarshaller.unmarshalIntoIntermediary(AttributeUnmarshaller.java:47)
        at com.bea.staxb.runtime.internal.UnmarshalResult.unmarshalBindingType(UnmarshalResult.java:199)
        at com.bea.staxb.runtime.internal.UnmarshalResult.unmarshalDocument(UnmarshalResult.java:169)
        at com.bea.staxb.runtime.internal.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:65)
        at weblogic.descriptor.internal.MarshallerFactory$1.createDescriptor(MarshallerFactory.java:150)
        ... 39 more
Caused by: java.lang.IllegalArgumentException: 0 is not a legal value for UnicastListenPort.  It does not fall in the range [1, 65535]
        at weblogic.descriptor.beangen.LegalChecks.checkInRange(LegalChecks.java:80)
        at weblogic.coherence.descriptor.wl.CoherenceClusterParamsBeanImpl.setUnicastListenPort(CoherenceClusterParamsBeanImpl.java:137)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.bea.staxb.runtime.internal.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:48)
        at com.bea.staxb.runtime.internal.RuntimeBindingType$BeanRuntimeProperty.setValue(RuntimeBindingType.java:539)
        at com.bea.staxb.runtime.internal.ByNameRuntimeBindingType$ElementQNameProperty.fill(ByNameRuntimeBindingType.java:351)
        at com.bea.staxb.runtime.internal.LiteralUnmarshalResult.extractAndFillElementProp(LiteralUnmarshalResult.java:159)
        at com.bea.staxb.runtime.internal.ByNameUnmarshaller.deserializeContents(ByNameUnmarshaller.java:51)
        at com.bea.staxb.runtime.internal.AttributeUnmarshaller.unmarshalIntoIntermediary(AttributeUnmarshaller.java:48)
        at com.bea.staxb.runtime.internal.LiteralUnmarshalResult.unmarshalElementProperty(LiteralUnmarshalResult.java:184)
        at com.bea.staxb.runtime.internal.LiteralUnmarshalResult.extractAndFillElementProp(LiteralUnmarshalResult.java:156)
        at com.bea.staxb.runtime.internal.ByNameUnmarshaller.deserializeContents(ByNameUnmarshaller.java:51)
        at com.bea.staxb.runtime.internal.AttributeUnmarshaller.unmarshalIntoIntermediary(AttributeUnmarshaller.java:48)
        ... 43 more
.>
<Mar 15, 2016 8:24:18 AM CDT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: [Management:141266]Parsing Failure in config.xml: weblogic.management.ManagementRuntimeException: weblogic.application.ModuleException: Could not create the Coherence Cluster descriptor>
<Mar 15, 2016 8:24:18 AM CDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Mar 15, 2016 8:24:18 AM CDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<Mar 15, 2016 8:24:18 AM CDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
========================================================================
Cause:
While extending domain config.xml got corrupted. It was wrongly copied from 12c(12.1.3) environment which was on same host.
========================================================================
Resolution:
1. Replace domain config from backup/manually correct it.
2. Restart the server.

Unable to start admin server because of OPSS failed with ORA-12514, TNS:listener does not currently know of service requested in connect descriptor

========================================================================
Applies To:
BPEL/OSB - 12c (12.1.3)
WLS - 12.1.3
Oracle DB - 12.1.0.2.0
========================================================================
Problem:
Unable to start Admin server because of weblogic.security.SecurityInitializationException: The loading of an OPSS java security policy provider failed due to an exception.

SOA DB is changed and it's working fine at the moment. If we test directly with SQLDEV/DBPing utility we are able to connect to DB. We have changed URL in all $Domain_home/config/jdbc/*.xml
When we are trying to start Admin server still it's referring to old DB instance and failing with ORA-12514, TNS:listener does not currently know of service requested in connect descriptor

Error:
NM Log:
<Mar 6, 2016 2:06:38 AM IST> <WARNING> <Server start command for WebLogic server 'osb_poc_admin' failed due to: [Server failed to start up but Node Manager was not aware of the re
ason]. Please check Node Manager log and/or server 'osb_poc_admin' log for detailed information.>

Server Log:
####<Mar 6, 2016 5:23:47 AM IST> <Error> <Security> <purushottam> <osb_poc_admin> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'>
<<WLS Kernel>> <> <> <1457263427955> <BEA-090892> <The loading of an OPSS java security policy provider failed due to an exception. See the exception stack trace or the server log file for
the root cause. If there is no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: null>
####<Mar 6, 2016 5:23:47 AM IST> <Critical> <WebLogicServer> <purushottam> <osb_poc_admin> <main> <<WLS Kernel>> <> <> <1457263427970> <BEA-000386> <Server subsyst
em failed. Reason: A MultiException has 12 exceptions.  They are:
1. weblogic.security.SecurityInitializationException: The loading of an OPSS java security policy provider failed due to an exception. See the exception stack trace or the server log file f
or the root cause. If there is no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: null
.
.
.
Caused By: oracle.security.jps.JpsException: oracle.security.jps.service.policystore.PolicyStoreException: javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Pers
istence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLRecoverableException: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor

Error Code: 12514
        at oracle.security.jps.internal.config.OpssCommonStartup.start(OpssCommonStartup.java:211)
        at oracle.security.jps.wls.JpsWlsStartup.start(JpsWlsStartup.java:80)
        at oracle.security.jps.JpsStartup.start(JpsStartup.java:186)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
.
.
.
Caused By: oracle.net.ns.NetException: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor

        at oracle.net.ns.NSProtocolStream.negotiateConnection(NSProtocolStream.java:288)
        at oracle.net.ns.NSProtocol.connect(NSProtocol.java:275)
        at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1606)
========================================================================
Cause:
$Domain_home/config/fmwconfig/jps-config.xml file was still referring to old DB URL
========================================================================
Resolution:
In SOA 12c(12.1.3), we need to change DB URL in 2 places
1.$Domain_home/config/jdbc/*.xml
2.$Domain_home/config/fmwconfig/jps-config.xml

$Domain_home/config/fmwconfig/jps-config.xml file was still referring to old DB URL. Change this to new DB URL

Restart the server.

java.lang.OutOfMemoryError: getNewTla

========================================================================
Applies To:
BPEL/OSB - PS6 (11.1.1.7.0)
WLS - 10.3.6
Oracle DB - Enterprise Edition - Version 11.2.0.3
Java - R28.2.7
========================================================================
Problem:
Initially server is responding slow and after sometine it's getting crashed. Exception in logs says java.lang.OutOfMemoryError: getNewTla

Error:
####<Feb 14, 2016 5:10:09 PM UTC> <Error> <EJB> <purushottam_vm> <MS02> <[ACTIVE] ExecuteThread: '47' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1455469809711> <BEA-010080> <An error occurred while attempting to process a message inside a message-driven bean: java.lang.OutOfMemoryError
Exception is : java.lang.OutOfMemoryError
>
####<Feb 14, 2016 5:10:09 PM UTC> <Error> <EJB> <purushottam_vm> <MS02> <[ACTIVE] ExecuteThread: '48' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1455469809919> <BEA-010080> <An error occurred while attempting to process a message inside a message-driven bean: java.lang.OutOfMemoryError: getNewTla
Exception is : java.lang.OutOfMemoryError: getNewTla
        at weblogic.transaction.internal.XidImpl.create(XidImpl.java:291)
        at weblogic.transaction.internal.TransactionManagerImpl.getNewXID(TransactionManagerImpl.java:1912)
        at weblogic.transaction.internal.TransactionManagerImpl.internalBegin(TransactionManagerImpl.java:254)
        at weblogic.transaction.internal.ServerTransactionManagerImpl.internalBegin(ServerTransactionManagerImpl.java:360)
        at weblogic.transaction.internal.ServerTransactionManagerImpl.begin(ServerTransactionManagerImpl.java:335)
        at weblogic.ejb.container.internal.NewJMSMessagePoller.processOneMessage(NewJMSMessagePoller.java:250)
        at weblogic.ejb.container.internal.NewJMSMessagePoller.run(NewJMSMessagePoller.java:120)
        at weblogic.ejb.container.internal.NewJMSMessagePoller.timerExpired(NewJMSMessagePoller.java:235)
        at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
========================================================================
Cause: 
Thread Local Area (TLA) size for object allocation is not sufficient.
========================================================================
Resolution:
We need to fine tune thread-local areas (TLA) for object allocation in JRockit as per application requirement. Sometimes default settings might not work expected.
-XXtlaSize option sets the thread-local area size,
Usage:
-XXtlaSize:min=size,preferred=size,wasteLimit=size

Default TLA settings were,
min=2k
preferred=16k
wasteLimit=1k

Below settings resolved OOM issue in my local environment and have seen performance improvement as well.
We can configure it as startup arguments,
-XXtlaSize:min=8k,preferred=512k,wasteLimit=8k

Restart the server for changes to take effect.

ADMIN server failed to start because of Internal Exception: java.sql.SQLException: ORA-28001: the password has expired for SOA Schemas

========================================================================

Applies To:

BPEL/OSB - 12.1.3

WLS - 12.1.3

Oracle DB - Enterprise Edition - Version 11.2.0.3

========================================================================

Problem:

ADMIN server failed to start because of Internal Exception: java.sql.SQLException: ORA-28001: the password has expired for SOA Schemas

Error:

Caused By: oracle.security.jps.JpsException: oracle.security.jps.service.policystore.PolicyStoreException: javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException

Internal Exception: java.sql.SQLException: ORA-28001: the password has expired

Error Code: 28001

        at oracle.security.jps.internal.config.OpssCommonStartup.start(OpssCommonStartup.java:211)

        at oracle.security.jps.wls.JpsWlsStartup.start(JpsWlsStartup.java:80)

        at oracle.security.jps.JpsStartup.start(JpsStartup.java:186)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.startJPS(CommonSecurityServiceManagerDelegateImpl.java:1499)

        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1422)

        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.preInitialize(CommonSecurityServiceManagerDelegateImpl.java:1090)

        at weblogic.security.service.SecurityServiceManager.preInitialize(SecurityServiceManager.java:925)

        at weblogic.security.PreSecurityService.start(PreSecurityService.java:139)

        at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:78)

        at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

========================================================================

Cause: 

Passwords used by SOA (with it’s different users) to access the Database have expired. This is the designed behaviour.

java.sql.SQLException: ORA-28001: the password has expired

========================================================================

Resolution:

1.Log in to SQLPLUS with system as sysdba

2.Execute the following sql command to see all the schemas that are used in your repository and account status,

SET linesize 120

SET pagesize 1000 

col owner FORMAT a20 

col comp_name FORMAT a40 

col version FORMAT a12

SELECT username, profile, account_status, expiry_date, default_tablespace from DBA_USERS where username IN (SELECT owner FROM schema_version_registry);

You should change the passwords of all schemas whose password is expired.

3.Change the SOA schema db password using below command,

ALTER USER user_name identified BY new_password;

Would like to suggest to set new password same as old so that we need not to do DS password update on console.

4.Set password expiry to unlimited using below command,

ALTER PROFILE default LIMIT PASSWORD_LIFE_TIME UNLIMITED;

It will set password expiry to default for all users under the default profile. In case you want to apply this just for some specific users, you can create a dedicated security profile and associate users with this profile,

CREATE PROFILE <profile_name_here> LIMIT PASSWORD_LIFE_TIME UNLIMITED;

ALTER USER <user_name_here> PROFILE <profile_name_here>;

5.Check the changes by logging to DB using SOA schema/password.

6.Log in to weblogic console and go to "Domain Structure/Services/Data Source" to change password in Connection Pool tab for required schema. You can ignore this step if you have set new password same as old.

7.Restart server.