Unable to start node manager becuase of java.net.BindException: Address already in use

========================================================================
Applies to:
Weblogic Server: 10.3.6 and later
========================================================================
Problem Summary:
Trying to start node manager but couldn't able to because of java.net.BindException: Address already in use

Error:
<Jan 8, 2017 10:43:12 PM CST> <SEVERE> <Fatal error in NodeManager server>
java.io.IOException: The server is unable to create a server socket for listening. The address puruhost might be incorrect or another process is using port 25000: java.net.BindException: Address already in use
        at weblogic.nodemanager.server.NMServer.start(NMServer.java:351)
        at weblogic.nodemanager.server.NMServer.main(NMServer.java:529)
        at weblogic.NodeManager.main(NodeManager.java:31)
Caused by: java.net.BindException: Address already in use
        at java.net.PlainSocketImpl.socketBind(Native Method)
        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
        at java.net.ServerSocket.bind(ServerSocket.java:375)
        at java.net.ServerSocket.<init>(ServerSocket.java:237)
        at javax.net.ssl.SSLServerSocket.<init>(SSLServerSocket.java:181)
        at sun.security.ssl.SSLServerSocketImpl.<init>(SSLServerSocketImpl.java:140)
        at sun.security.ssl.SSLServerSocketFactoryImpl.createServerSocket(SSLServerSocketFactoryImpl.java:95)
        at weblogic.nodemanager.server.SSLListener.privilegedInit(SSLListener.java:134)
        at weblogic.nodemanager.server.SSLListener.access$000(SSLListener.java:33)
        at weblogic.nodemanager.server.SSLListener$1.run(SSLListener.java:97)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.nodemanager.server.SSLListener.init(SSLListener.java:94)
        at weblogic.nodemanager.server.NMServer.start(NMServer.java:349)
        ... 2 more

========================================================================
Root Cause:
Port number used by node manager process is being already used by some other process.

========================================================================
How to verify:
1. Login to host
2. Verify process which is using node manager port using
 /usr/sbin/lsof -i:port_num
 netstat -an | grep port_num
 e.g. /usr/sbin/lsof -i:25000
netstat -an | grep 25000

========================================================================
Resolution:
1. Once you find a process which is using Node manager port and can be killed, kill it using,
 kill -9 PID
2. Restart node manager using $DOMAIN_HOME/bin/startNodeManager.sh

Not able to connect to node manager because of Connection refused: connect

========================================================================
Applies to:
Weblogic Server: 10.3.6 and later
========================================================================
Problem Summary:
We are trying to connect to node manager to perform restart using nmConnect but couldn't able to.

Error:
wls:/offline> nmConnect('weblogic','weblogic123','localhost','25000','weblogic_poc_dit')
Connecting to Node Manager ...
Traceback (innermost last):
  File "<console>", line 1, in ?
  File "<iostream>", line 123, in nmConnect
  File "<iostream>", line 648, in raiseWLSTException
WLSTException: Error occured while performing nmConnect : Cannot connect to Node
 Manager. : Connection refused: connect. Could not connect to NodeManager. Check
 that it is running at localhost:25000.
Use dumpStack() to view the full stacktrace
wls:/offline>
========================================================================
Root Cause:
Node manager isn't running at localhost:25000 becuase of which connection is refused.
========================================================================
How to verify:
1. Login to localhost
2. Use /usr/sbin/lsof -i:25000 or ps -ef | grep weblogic.NodeManager command to verify if node manager is up and running.
========================================================================
Resolution:
1. Go to $DOMAIN_HOME/bin and start node manager using startNodeManager.sh
<Feb 11, 2017 12:29:20 PM CST> <INFO> <Secure socket listener started on port 25000, host localhost/192.168.184.130>
2. Once it is started listening, try to connect to node manager using nmConnect
wls:/offline> nmConnect('weblogic','weblogic123','localhost','25000','weblogic_poc_dit')
Connecting to Node Manager ...
Successfully Connected to Node Manager.
wls:/nm/weblogic_poc_dit>

Couldn’t able to start Node Manager because of invalid java directory.

========================================================================

Applies to:

Weblogic Server: 10.3.6 and later

========================================================================

Problem Summary:

Couldn’t able to start Node Manager because of missing java directory.

Error:

C:\weblogic1036\Middleware\Oracle_Home\user_projects\domains\base_domain\bin>startNodeManager.cmd

NODEMGR_HOME is already set to C:\WEBLOG~1\MIDDLE~1\ORACLE~1\USER_P~1\domains\BASE_D~1\NODEMA~1

The JDK wasn't found in directory C:\oracle1221\Java\jdk1.8.0_66.

Please edit this script so that the JAVA_HOME

variable points to the location of your JDK.

C:\weblogic1036\Middleware\Oracle_Home\user_projects\domains\base_domain\bin>

========================================================================

Root Cause:

JAVA_HOME variable was set to invalid location C:\oracle1221\Java\jdk1.8.0_66 which is invalid location.

========================================================================

How to verify:

Use echo command to verify it.

C:\WINDOWS\system32>echo %JAVA_HOME%

C:\oracle1221\Java\jdk1.8.0_66

========================================================================

Resolution:

1. Go to Control Panel => System and Security => System => Advance System Settings => Environment Variables

2. Locate JAVA_HOME variable and change it’s value form invalid to valid location.

In our case from C:\oracle1221\Java\jdk1.8.0_66 to C:\oracle12211\Java\jdk1.8.0_66

3. Verify JAVA_HOME

C:\WINDOWS\system32>echo %JAVA_HOME%

C:\oracle12211\Java\jdk1.8.0_66

4. Start Node Manager,

Domain name mappings:

base_domain -> C:\weblogic1036\Middleware\Oracle_Home\user_projects\domains\base_domain

<Feb 11, 2017 1:16:36 PM IST> <INFO> <WebLogic Server 12.1.3.0.0  Wed May 21 18:53:34 PDT 2014 1604337 >

<Feb 11, 2017 1:16:36 PM IST> <INFO> <Secure socket listener started on port 5556, host localhost/127.0.0.1>

java.io.FileNotFoundException: Domain directory '/u01/oracle11g/mw1036/user_projects/domains/poc_domain' invalid (domain salt file not found)

========================================================================
Applies to:
Weblogic Server: 10.3.6 and later
========================================================================
Problem Summary:
Case 1: When we are trying to start managed server through the node manager facing java.io.FileNotFoundException: Domain directory '/u01/oracle11g/mw1036/user_projects/domains/poc_domain' invalid (domain salt file not found) error.

WebLogic server is complaining about a domain that is not correctly registered and enrolled for node Manager.

<Mar 2, 2016 3:16:43 AM CST> <WARNING> <There was a problem initializing the domain 'poc_domain' at '/u01/oracle11g/mw1036/user_projects/domains/poc_domain'. Please make sure that this domainName: 'poc_domain' is registered and is fully enrolled for this NodeManager at: '/u01/oracle11g/mw1036/user_projects/domains/poc_domain'.>

Case 2: Node Manager Status is Inactive on weblogic admin console,
Problem description:weblogic.nodemanager.NMException
Change: Node manager config was copied from box A to B of same domain, changed ListenAddress, Port and started nodemanager.

Error:
<Mar 2, 2016 3:16:43 AM CST> <WARNING> <I/O error while reading domain directory>
java.io.FileNotFoundException: Domain directory '/u01/oracle11g/mw1036/user_projects/domains/poc_domain' invalid (domain salt file not found)
        at weblogic.nodemanager.server.DomainManager.initialize(DomainManager.java:95)
        at weblogic.nodemanager.server.DomainManager.<init>(DomainManager.java:60)
        at weblogic.nodemanager.server.NMServer.findOrCreateDomainManager(NMServer.java:332)
        at weblogic.nodemanager.server.NMServer.getDomainManager(NMServer.java:316)
        at weblogic.nodemanager.server.Handler.handleDomain(Handler.java:277)
        at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:131)
        at weblogic.nodemanager.server.Handler.run(Handler.java:81)
        at java.lang.Thread.run(Thread.java:724)

========================================================================
Root Cause:
Case 1: Domain root directory(nodemanager.properties) was pointing to wrong directory.
Node manager was referring to wrong domain directory and looking for a SerializedSystemIni.dat file to decrypt the credentials required to start the server causing the exception.

Case 2: Domain isn't registered with node manager.
========================================================================
Resolution:
Case1:
1. Go to your nodemanager home,
$WL_HOME/common/nodemanager/
2. Edit nodemanager.domains file and make sure it has valid domain root directory.
3. Save your changes.
4. Restart Node Manager
5. Restart Server using Node manager.

Case2:
1. Enroll your nodemanager using NMEnroll command,
nmEnroll('$DOMAIN_HOME','$WL_HOME/common/nodemanager/')
2. Restart your node manager.
3. Check NM status on console.

java.lang.Exception: Failed to load url; https://host:port/tls12Tester/DIT?wsdl, 0 in SoapUI

========================================================================
Applies to:
SoapUI: 5.2.1
========================================================================
Problem Summary:
We aren't able to load a WSDL in SOAPUI which has TLS1.2 as minimum protocol version.

SOAPUI Error:
Error loading wsdl
There was something wrong with the wsdl you are trying to import
Error loading [https://host:port/tls12Tester/DIT?wsdl]: java.lang.Exception: Failed to load url; https://host:port/tls12Tester/DIT?wsdl, 0

2017-02-08 10:07:22,947 ERROR [errorlog] com.eviware.soapui.impl.support.definition.support.InvalidDefinitionException
com.eviware.soapui.impl.support.definition.support.InvalidDefinitionException
at com.eviware.soapui.impl.wsdl.support.wsdl.WsdlLoader.makeInvalidDefinitionException(WsdlLoader.java:119)
at com.eviware.soapui.impl.wsdl.support.wsdl.WsdlLoader.loadXmlObject(WsdlLoader.java:112)
at com.eviware.soapui.impl.wsdl.support.xsd.SchemaUtils.getDefinitionParts(SchemaUtils.java:488)
at com.eviware.soapui.impl.wsdl.support.xsd.SchemaUtils.getDefinitionParts(SchemaUtils.java:477)
at com.eviware.soapui.impl.support.definition.support.AbstractDefinitionCache.update(AbstractDefinitionCache.java:94)
at com.eviware.soapui.impl.support.definition.support.AbstractDefinitionContext$Loader.construct(AbstractDefinitionContext.java:209)
at com.eviware.soapui.support.swing.SwingWorkerDelegator.construct(SwingWorkerDelegator.java:46)
at com.eviware.soapui.support.swing.SwingWorker$2.run(SwingWorker.java:131)
at java.lang.Thread.run(Unknown Source)

========================================================================
Root Cause:
Minimum security protocol version for webservice is set to TLS1.2
SSL.minimumProtocolVersion=TLSv1.2
========================================================================
How to verify:
 1. Load wsdl in a browser to see if it's loading fine or not.
 2. use openssl to verify protocol,
openssl s_client -connect host:port -tls1_2
Result: Secure Renegotiation IS supported
SSL-Session:
Protocol  : TLSv1.2
========================================================================
Resolution:
Edit SoapUI-5.2.1.vmoptions file located at C:\SmartBear\SoapUI-5.2.1\bin\SoapUI-5.2.1.vmoptions

Add the following parameter, save and exit.
-Dsoapui.https.protocols=TLSv1.2

Kindly note it will enable only TLSv1.2 protocol

Close and reopen SoapUI for changes to take effect.

Retry to load wsdl in SoapUI. This time it should work fine.

Exception in HttpOutboundMessageContext.RetrieveHttpResponseWork.run: java.io.IOException: SocketMuxer detected socket closure

========================================================================
Applies To:
Linux Version:  OEL-6.5
WLS - 10.3.6
OSB - 11.1.1.7 and later
========================================================================
Problem Summary:
While calling buiness service which has 3DNS endpoint uri of a different domain which is hosted on the same box giving java.io.IOException: SocketMuxer detected socket closure while waiting for a response.

Error:
[WliSbTransports:381304]Exception in HttpOutboundMessageContext.RetrieveHttpResponseWork.run: java.io.IOException: SocketMuxer detected socket closure while waiting for a response
java.io.IOException: SocketMuxer detected socket closure while waiting for a response
 at weblogic.net.http.SocketClosedNotification.&lt;clinit>(SocketClosedNotification.java:10)
 at weblogic.net.http.AsyncResponseHandler$MuxableSocketHTTPAsyncResponse.handleError(AsyncResponseHandler.java:396)
 at weblogic.net.http.AsyncResponseHandler$MuxableSocketHTTPAsyncResponse.hasException(AsyncResponseHandler.java:504)
 at weblogic.socket.SocketMuxer.deliverExceptionAndCleanup(SocketMuxer.java:827)
 at weblogic.socket.SocketMuxer.deliverHasException(SocketMuxer.java:767)
 at weblogic.socket.EPollSocketMuxer.dataReceived(EPollSocketMuxer.java:222)
 at weblogic.socket.EPollSocketMuxer.processSockets(EPollSocketMuxer.java:177)
 at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
 at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
 at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
 at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

========================================================================
Root Cause:
Service is failing when using 3DNS where as working fine if we use individual managedServerHost:port url becuase of which suspect goes to 3DNS settings.
During investion found that 3DNS doesn't have SNAT settings enabled for members becuase of which local call which is travelling through network ( from domain1 to domain2 on same host) is failing.

========================================================================
How to verify:
1. Login to MS host and use wget with 3DNS url. You will recevie below error,
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.

2. Use OSB test console to test BS/PS

3. If we use SOAPUI to test service with 3DNS url should work fine.
========================================================================
Resolution:
Permanent Solution:
Configure SNAT for 3DNS members (LTM setting).
Work around:
Change business service endpoint URL from load balancer URL to individual server managedServerHost:port URL.

java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread (lifecycle.c:1097)

========================================================================
Applies To:
Linux Version:  OEL-6.5
WLS - 10.3.6
OSB - 11.1.1.7
========================================================================
Problem:
OSB becomes unavailable/unresponsive under heavy load.

Error:
java.lang.OutOfMemoryError: Resource temporarily unavailable in tsStartJavaThread (lifecycle.c:1097).

Attempting to allocate 8096M bytes

There is insufficient native memory for the Java
Runtime Environment to continue.

Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit

Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Disable compressed references (-XXcompressedRefs=false)


========================================================================
Cause:
/etc/security/limits.conf file nproc value is overwritten by /etc/security/limits.d/90-nproc.conf and setting max user processes 1024

========================================================================
Resolution:
We need to change nproc value in /etc/security/limits.d/90-nproc.conf

Default Values:
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     1024
root       soft    nproc     unlimited

New Values:
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     unlimited
root       soft    nproc     unlimited