SOA-Infra is not coming up beause of exception com.tangosol.net.RequestTimeoutException

SOA infra isn't coming up and below error message has been shown in the server logs.

Error:
Caused By: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
  MemberSet=MasterMemberSet(
    ThisMember=null
    OldestMember=null
    ActualMemberSet=MemberSet(Size=0
      )
    MemberId|ServiceVersion|ServiceJoined|MemberState
    RecycleMillis=1200000
    RecycleSet=MemberSet(Size=0
      )
    )
)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
        at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:58)

Root Cause:
Tangosole Configuration wasn't correct. Localhost was missing from WKA settings

Solution:
Corrected WKA/Tangosole settings by providing localhost and restarted server

Oracle Fusion Middleware Node manager

Node Manager is a WebLogic Server utility that enables you to start, shut down, and restart Administration Server and Managed Server instances from a remote location

A Node Manager process is not associated with a specific WebLogic domain but with a machine. Although Node Manager is optional but it's recommended by oracle.

WebLogic Server provides two versions of Node Manager, Java-based and script-based, with similar functionality.  Java based NM provides more security capabilities than script based NM.

If you are installing WebLogic Server on a Windows system, you must use the Java version of Node Manager. The scripted version of Node Manager is not supported on Windows.

In order to use consensus leasing, you may see faster performance when using the Java version of Node Manager.

What You Can Do with Node Manager

• Start, Shut Down, and Restart an Administration Server

• Start, Shut Down, Suspend, and Restart Managed Servers

• Monitor Servers and View Log Data

Starting Java-based Node Manager

• Start scripts for Node Manager are installed in the WL_HOME\server\bin directory, where WL_HOME is the top-level installation directory for WebLogic Server. 

• Use startNodeManager.cmd on Windows systems and startNodeManager.sh on UNIX systems.

• The scripts set the required environment variables and start Node Manager in WL_HOME/common/nodemanager. Node Manager uses this directory as a working directory for output and log files. 

Important Node Manager Configuration Files

1. nodemanager.properties

This is the configuration file used by the Java-based version of Node Manager.

This file is located in WL_HOME/common/nodemanager.

2. nodemanager.hosts

This file contains a list of all the trusted hosts that can issue commands to Node Manager.

This file is located in WL_HOME/common/nodemanager.

3. nodemanager.domains

This file contains mappings between the names of domains managed by Node Manager and their corresponding directories. 

This file is located in WL_HOME/common/nodemanager.

4. nm_data.properties

This file stores the encryption data the Node Manager uses a symmetric encryption key. The data is stored in encrypted form.

This file is located in WL_HOME/common/nodemanager.

5. nm_password.properties

This file stores a username/password pair specific to the Node Manager server that is managing this domain. This is known as the Node Manager secret. The username and password are appended to a salt value (obtained from the SerializedSystemIni.dat of the domain) and SHA-hashed.

This file is located in DOMAIN_HOME/config/nodemanager.

6. boot.properties

Node Manager uses this file to specify a boot identity when starting a server. This file is located in domain-name/servers/server_name/data/nodemanager.

7. startup.properties

Each Managed Server instance has its own startup.properties file with properties that control how Node Manager starts up and controls the server. Node Manager automatically creates this file by using properties passed to Node Manager when the Administrative Server was last used to start the server. This allows a Node Manager client or startup scripts to restart a Managed Server using the same properties last used by the Administrative Server.

These properties correspond to the server startup attributes contained in ServerStartMBean and the health monitoring attributes in ServerStartMBean.

This file is located in domain-name/servers/server_name/data/nodemanager.

8. server_name.addr

server_name.addr stores the IP address added when a server starts or is migrated. This file is generated after the server IP address is successfully brought online during migration. server_name.addr is deleted when the IP address is brought offline. The server IP address is used to validate remove requests to prevent addresses being erroneously removed while shutting down the server.

This file is located in domain-name/servers/server_name/data/nodemanager.

9. server_name.lck

server_name.lck is generated by each server and contains an internally used lock ID.

This file is located in domain-name/servers/server_name/data/nodemanager

10. server_name.pid

server_name.pid is generated by each server and contains the process ID of the server. Node Manager checks the process ID generated by the server during crash recovery.

This file is located in domain-name/servers/server_name/data/nodemanager

11. server_name.state

server_name.state is generated by the server and contains the server's current state. Node Manager monitors the contents of this file to determine the current state of the server.

Note: Do not delete or alter this file. Without this file Node Manager cannot determine the current state of the server.

This file is located in domain-name/servers/server_name/data/nodemanager.

java.net.SocketException: Too many files in weblogic/EM

Root Cause:
Maximum number of file descriptors configured for Server is less than 65535.

Resolution:
1. verify the maximum number of file descriptors currently configured using below command
    cat /proc/sys/fs/file-max
    if the value is less than 65535 then perform the following steps,
2. Edit the file /etc/security/limits.conf with root permission
    sudo vi /etc/security/limits.conf
3. Append the following two lines, using a value of 65535 or greater
     *                soft    nofile          65535
     *                hard    nofile          65535
4. Start a new terminal session
5. Execute the limit descriptors command to verify that descriptors has been increased to the specified value (at least 65535).
    limit descriptors
6. Restart Weblogic server.

Difference between Development mode and Production mode

Development mode:

1. Default JDK for development domain is Sun Hotspot.
2. You can use the demo certificates for SSL.
3. Auto deployment is enabled.
4. Server instances rotate their log files on startup.
5. Admin Server uses an automatically created boot.properties during startup.
6. The default maximum capacity for JDBC Datasource is 15.
7. The debugFlag which is used to start the WebLogic Workshop Debugger is enabled.

Production mode:

1. The default JDK for production domain is JRockit.
2. If you use the demo certificates for SSL a warning is displayed.
3. Auto deployment is disabled.
4. Server instances rotate their log files when it reaches 5MB.
5. Admin Server prompts for username and password during startup.
6. The default maximum capacity for JDBC Datasource is 25.
7. The debugFlag which is used to start the WebLogic Workshop Debugger is disabled.

Ref: https://www.liferay.com/community and oracle docs.