Weblogic Managed server fails to start due to java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

========================================================================
Applies To:
WLS - 10.3.6
========================================================================
Problem:
Weblogic Managed server fails to start with an exception java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

Error:
<Mar 15, 2016 8:24:18 AM CDT> <Notice> <Cluster> <BEA-000142> <Trying to download cluster JNDI tree from server managed_server1.>
<Mar 15, 2016 8:24:18 AM CDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
        at java.util.ArrayList.RangeCheck(ArrayList.java:547)
        at java.util.ArrayList.get(ArrayList.java:322)
        at weblogic.rmi.cluster.BasicReplicaList.getPrimary(BasicReplicaList.java:90)
        at weblogic.rmi.cluster.ClusterableRemoteRef.getPrimaryRef(ClusterableRemoteRef.java:155)
        at weblogic.rmi.cluster.ClusterableRemoteRef.add(ClusterableRemoteRef.java:474)
        Truncated. see log file for complete stacktrace
>
<Mar 15, 2016 8:24:18 AM CDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Mar 15, 2016 8:24:18 AM CDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<Mar 15, 2016 8:24:18 AM CDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
========================================================================
Cause:
As per logs downloading of cluster JNDI tree failed from managed_server1.
There can be possibility of network or communication issue to servers in cluster.
========================================================================
Resolution:
Restart of managed_server1 can fix the issue. If it doesn't help then go for complete domain restart.

Unable to start Admin server becuase of java.lang.IllegalArgumentException: 0 is not a legal value for UnicastListenPort

========================================================================
Applies To:
BPEL/OSB - 11g PS6 (11.1.1.7)
WLS - 10.3.6
Oracle DB - 11.2.0.4
========================================================================
Problem:
Unable to start admin server using NM because of java.lang.IllegalArgumentException: 0 is not a legal value for UnicastListenPort.  It does not fall in the range [1, 65535]
Port configured for coherence/WKA setting are in range [1,65535] . Coherences settings are configured as per oracle WKA settings document and they are fine.

Recently domain was extended.

Error:
<Mar 15, 2016 8:24:16 AM CDT> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Oracle JRockit(R) Version R28.2.2-7-148152-1.6.0_29-20111221-2104-linux-x86_64 from Oracle Corporation>
<Mar 15, 2016 8:24:16 AM CDT> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.6.0.3 PSU Patch for BUG14736139 Fri Nov 23 10:16:54 IST 2012
WebLogic Server 10.3.6.0  Tue Nov 15 08:52:36 PST 2011 1441050 >
<Mar 15, 2016 8:24:18 AM CDT> <Error> <J2EE> <BEA-160197> <Unable to load descriptor java.net.URLClassLoader@1490a66f/null of module null. The error is weblogic.descriptor.DescriptorException: Unmarshaller failed
        at weblogic.descriptor.internal.MarshallerFactory$1.createDescriptor(MarshallerFactory.java:161)
        at weblogic.descriptor.BasicDescriptorManager.createDescriptor(BasicDescriptorManager.java:323)
        at weblogic.application.descriptor.AbstractDescriptorLoader2.getDescriptorBeanFromReader(AbstractDescriptorLoader2.java:788)
        at weblogic.application.descriptor.AbstractDescriptorLoader2.createDescriptorBean(AbstractDescriptorLoader2.java:409)
        at weblogic.application.descriptor.AbstractDescriptorLoader2.loadDescriptorBeanWithoutPlan(AbstractDescriptorLoader2.java:759)
        at weblogic.application.descriptor.AbstractDescriptorLoader2.loadDescriptorBean(AbstractDescriptorLoader2.java:768)
        at weblogic.cacheprovider.coherence.CoherenceClusterDescriptorHelper.getCoherenceBean(CoherenceClusterDescriptorHelper.java:64)
        at weblogic.cacheprovider.coherence.CoherenceClusterDescriptorHelper.createCoherenceDescriptor(CoherenceClusterDescriptorHelper.java:32)
        at weblogic.management.mbeans.custom.CoherenceClusterSystemResource.loadDescriptor(CoherenceClusterSystemResource.java:202)
        at weblogic.management.mbeans.custom.ConfigurationExtension.getExtensionRoot(ConfigurationExtension.java:178)
        at weblogic.management.mbeans.custom.CoherenceClusterSystemResource.getCoherenceClusterResource(CoherenceClusterSystemResource.java:70)
        at weblogic.management.mbeans.custom.CoherenceClusterSystemResource._postCreate(CoherenceClusterSystemResource.java:174)
        at weblogic.management.configuration.CoherenceClusterSystemResourceMBeanImpl._postCreate(CoherenceClusterSystemResourceMBeanImpl.java:448)
        at weblogic.descriptor.internal.AbstractDescriptorBean._postCreate(AbstractDescriptorBean.java:671)
        at weblogic.management.configuration.DomainMBeanImpl.setCoherenceClusterSystemResources(DomainMBeanImpl.java:14964)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.bea.staxb.runtime.internal.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:48)
        at com.bea.staxb.runtime.internal.RuntimeBindingType$BeanRuntimeProperty.setValue(RuntimeBindingType.java:539)
        at com.bea.staxb.runtime.internal.AttributeRuntimeBindingType$QNameRuntimeProperty.fillCollection(AttributeRuntimeBindingType.java:382)
        at com.bea.staxb.runtime.internal.MultiIntermediary.getFinalValue(MultiIntermediary.java:52)
        at com.bea.staxb.runtime.internal.AttributeRuntimeBindingType.getFinalObjectFromIntermediary(AttributeRuntimeBindingType.java:140)
        at com.bea.staxb.runtime.internal.UnmarshalResult.unmarshalBindingType(UnmarshalResult.java:200)
        at com.bea.staxb.runtime.internal.UnmarshalResult.unmarshalDocument(UnmarshalResult.java:169)
        at com.bea.staxb.runtime.internal.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:65)
        at weblogic.descriptor.internal.MarshallerFactory$1.createDescriptor(MarshallerFactory.java:100)
        at weblogic.descriptor.BasicDescriptorManager.createDescriptor(BasicDescriptorManager.java:257)
        at weblogic.management.provider.internal.DescriptorManagerHelper.loadDescriptor(DescriptorManagerHelper.java:73)
        at weblogic.management.provider.internal.RuntimeAccessImpl$IOHelperImpl.parseXML(RuntimeAccessImpl.java:690)
        at weblogic.management.provider.internal.RuntimeAccessImpl.parseNewStyleConfig(RuntimeAccessImpl.java:270)
        at weblogic.management.provider.internal.RuntimeAccessImpl.<init>(RuntimeAccessImpl.java:115)
        at weblogic.management.provider.internal.RuntimeAccessService.start(RuntimeAccessService.java:42)
        at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)
        at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)
        at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:882)
        at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:572)
        at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
        at weblogic.Server.main(Server.java:71)
Caused by: com.bea.xml.XmlException: java.lang.IllegalArgumentException: 0 is not a legal value for UnicastListenPort.  It does not fall in the range [1, 65535]
        at com.bea.staxb.runtime.internal.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:54)
        at com.bea.staxb.runtime.internal.RuntimeBindingType$BeanRuntimeProperty.setValue(RuntimeBindingType.java:539)
        at com.bea.staxb.runtime.internal.ByNameRuntimeBindingType$ElementQNameProperty.fill(ByNameRuntimeBindingType.java:351)
        at com.bea.staxb.runtime.internal.LiteralUnmarshalResult.extractAndFillElementProp(LiteralUnmarshalResult.java:159)
        at com.bea.staxb.runtime.internal.ByNameUnmarshaller.deserializeContents(ByNameUnmarshaller.java:51)
        at com.bea.staxb.runtime.internal.AttributeUnmarshaller.unmarshalIntoIntermediary(AttributeUnmarshaller.java:47)
        at com.bea.staxb.runtime.internal.LiteralUnmarshalResult.unmarshalElementProperty(LiteralUnmarshalResult.java:184)
        at com.bea.staxb.runtime.internal.LiteralUnmarshalResult.extractAndFillElementProp(LiteralUnmarshalResult.java:156)
        at com.bea.staxb.runtime.internal.ByNameUnmarshaller.deserializeContents(ByNameUnmarshaller.java:51)
        at com.bea.staxb.runtime.internal.AttributeUnmarshaller.unmarshalIntoIntermediary(AttributeUnmarshaller.java:47)
        at com.bea.staxb.runtime.internal.UnmarshalResult.unmarshalBindingType(UnmarshalResult.java:199)
        at com.bea.staxb.runtime.internal.UnmarshalResult.unmarshalDocument(UnmarshalResult.java:169)
        at com.bea.staxb.runtime.internal.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:65)
        at weblogic.descriptor.internal.MarshallerFactory$1.createDescriptor(MarshallerFactory.java:150)
        ... 39 more
Caused by: java.lang.IllegalArgumentException: 0 is not a legal value for UnicastListenPort.  It does not fall in the range [1, 65535]
        at weblogic.descriptor.beangen.LegalChecks.checkInRange(LegalChecks.java:80)
        at weblogic.coherence.descriptor.wl.CoherenceClusterParamsBeanImpl.setUnicastListenPort(CoherenceClusterParamsBeanImpl.java:137)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.bea.staxb.runtime.internal.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:48)
        at com.bea.staxb.runtime.internal.RuntimeBindingType$BeanRuntimeProperty.setValue(RuntimeBindingType.java:539)
        at com.bea.staxb.runtime.internal.ByNameRuntimeBindingType$ElementQNameProperty.fill(ByNameRuntimeBindingType.java:351)
        at com.bea.staxb.runtime.internal.LiteralUnmarshalResult.extractAndFillElementProp(LiteralUnmarshalResult.java:159)
        at com.bea.staxb.runtime.internal.ByNameUnmarshaller.deserializeContents(ByNameUnmarshaller.java:51)
        at com.bea.staxb.runtime.internal.AttributeUnmarshaller.unmarshalIntoIntermediary(AttributeUnmarshaller.java:48)
        at com.bea.staxb.runtime.internal.LiteralUnmarshalResult.unmarshalElementProperty(LiteralUnmarshalResult.java:184)
        at com.bea.staxb.runtime.internal.LiteralUnmarshalResult.extractAndFillElementProp(LiteralUnmarshalResult.java:156)
        at com.bea.staxb.runtime.internal.ByNameUnmarshaller.deserializeContents(ByNameUnmarshaller.java:51)
        at com.bea.staxb.runtime.internal.AttributeUnmarshaller.unmarshalIntoIntermediary(AttributeUnmarshaller.java:48)
        ... 43 more
.>
<Mar 15, 2016 8:24:18 AM CDT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: [Management:141266]Parsing Failure in config.xml: weblogic.management.ManagementRuntimeException: weblogic.application.ModuleException: Could not create the Coherence Cluster descriptor>
<Mar 15, 2016 8:24:18 AM CDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Mar 15, 2016 8:24:18 AM CDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<Mar 15, 2016 8:24:18 AM CDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
========================================================================
Cause:
While extending domain config.xml got corrupted. It was wrongly copied from 12c(12.1.3) environment which was on same host.
========================================================================
Resolution:
1. Replace domain config from backup/manually correct it.
2. Restart the server.

Unable to start admin server because of OPSS failed with ORA-12514, TNS:listener does not currently know of service requested in connect descriptor

========================================================================
Applies To:
BPEL/OSB - 12c (12.1.3)
WLS - 12.1.3
Oracle DB - 12.1.0.2.0
========================================================================
Problem:
Unable to start Admin server because of weblogic.security.SecurityInitializationException: The loading of an OPSS java security policy provider failed due to an exception.

SOA DB is changed and it's working fine at the moment. If we test directly with SQLDEV/DBPing utility we are able to connect to DB. We have changed URL in all $Domain_home/config/jdbc/*.xml
When we are trying to start Admin server still it's referring to old DB instance and failing with ORA-12514, TNS:listener does not currently know of service requested in connect descriptor

Error:
NM Log:
<Mar 6, 2016 2:06:38 AM IST> <WARNING> <Server start command for WebLogic server 'osb_poc_admin' failed due to: [Server failed to start up but Node Manager was not aware of the re
ason]. Please check Node Manager log and/or server 'osb_poc_admin' log for detailed information.>

Server Log:
####<Mar 6, 2016 5:23:47 AM IST> <Error> <Security> <purushottam> <osb_poc_admin> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'>
<<WLS Kernel>> <> <> <1457263427955> <BEA-090892> <The loading of an OPSS java security policy provider failed due to an exception. See the exception stack trace or the server log file for
the root cause. If there is no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: null>
####<Mar 6, 2016 5:23:47 AM IST> <Critical> <WebLogicServer> <purushottam> <osb_poc_admin> <main> <<WLS Kernel>> <> <> <1457263427970> <BEA-000386> <Server subsyst
em failed. Reason: A MultiException has 12 exceptions.  They are:
1. weblogic.security.SecurityInitializationException: The loading of an OPSS java security policy provider failed due to an exception. See the exception stack trace or the server log file f
or the root cause. If there is no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: null
.
.
.
Caused By: oracle.security.jps.JpsException: oracle.security.jps.service.policystore.PolicyStoreException: javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Pers
istence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLRecoverableException: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor

Error Code: 12514
        at oracle.security.jps.internal.config.OpssCommonStartup.start(OpssCommonStartup.java:211)
        at oracle.security.jps.wls.JpsWlsStartup.start(JpsWlsStartup.java:80)
        at oracle.security.jps.JpsStartup.start(JpsStartup.java:186)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
.
.
.
Caused By: oracle.net.ns.NetException: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor

        at oracle.net.ns.NSProtocolStream.negotiateConnection(NSProtocolStream.java:288)
        at oracle.net.ns.NSProtocol.connect(NSProtocol.java:275)
        at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1606)
========================================================================
Cause:
$Domain_home/config/fmwconfig/jps-config.xml file was still referring to old DB URL
========================================================================
Resolution:
In SOA 12c(12.1.3), we need to change DB URL in 2 places
1.$Domain_home/config/jdbc/*.xml
2.$Domain_home/config/fmwconfig/jps-config.xml

$Domain_home/config/fmwconfig/jps-config.xml file was still referring to old DB URL. Change this to new DB URL

Restart the server.

java.lang.OutOfMemoryError: getNewTla

========================================================================
Applies To:
BPEL/OSB - PS6 (11.1.1.7.0)
WLS - 10.3.6
Oracle DB - Enterprise Edition - Version 11.2.0.3
Java - R28.2.7
========================================================================
Problem:
Initially server is responding slow and after sometine it's getting crashed. Exception in logs says java.lang.OutOfMemoryError: getNewTla

Error:
####<Feb 14, 2016 5:10:09 PM UTC> <Error> <EJB> <purushottam_vm> <MS02> <[ACTIVE] ExecuteThread: '47' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1455469809711> <BEA-010080> <An error occurred while attempting to process a message inside a message-driven bean: java.lang.OutOfMemoryError
Exception is : java.lang.OutOfMemoryError
>
####<Feb 14, 2016 5:10:09 PM UTC> <Error> <EJB> <purushottam_vm> <MS02> <[ACTIVE] ExecuteThread: '48' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1455469809919> <BEA-010080> <An error occurred while attempting to process a message inside a message-driven bean: java.lang.OutOfMemoryError: getNewTla
Exception is : java.lang.OutOfMemoryError: getNewTla
        at weblogic.transaction.internal.XidImpl.create(XidImpl.java:291)
        at weblogic.transaction.internal.TransactionManagerImpl.getNewXID(TransactionManagerImpl.java:1912)
        at weblogic.transaction.internal.TransactionManagerImpl.internalBegin(TransactionManagerImpl.java:254)
        at weblogic.transaction.internal.ServerTransactionManagerImpl.internalBegin(ServerTransactionManagerImpl.java:360)
        at weblogic.transaction.internal.ServerTransactionManagerImpl.begin(ServerTransactionManagerImpl.java:335)
        at weblogic.ejb.container.internal.NewJMSMessagePoller.processOneMessage(NewJMSMessagePoller.java:250)
        at weblogic.ejb.container.internal.NewJMSMessagePoller.run(NewJMSMessagePoller.java:120)
        at weblogic.ejb.container.internal.NewJMSMessagePoller.timerExpired(NewJMSMessagePoller.java:235)
        at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
========================================================================
Cause: 
Thread Local Area (TLA) size for object allocation is not sufficient.
========================================================================
Resolution:
We need to fine tune thread-local areas (TLA) for object allocation in JRockit as per application requirement. Sometimes default settings might not work expected.
-XXtlaSize option sets the thread-local area size,
Usage:
-XXtlaSize:min=size,preferred=size,wasteLimit=size

Default TLA settings were,
min=2k
preferred=16k
wasteLimit=1k

Below settings resolved OOM issue in my local environment and have seen performance improvement as well.
We can configure it as startup arguments,
-XXtlaSize:min=8k,preferred=512k,wasteLimit=8k

Restart the server for changes to take effect.