How to Enroll a Node Manager in weblogic?

========================================================================
Applies to:
Weblogic: 10.3.6 and later
========================================================================
nmEnroll is used to enroll a machine or a domain with the node manager.
Few of us may forget to configure NodeManager after installing Weblogic server. Follow below steps to enroll nodemanager manually in weblogic.
1. Create a Machine:
     • Login to weblogic console
     • Click on Lock & Edit button in change center.
     • Go to Environment -> Machines -> click New
     • Provide name and select machine OS
     • Click Next
     • Select type, provide host and port details for node manager and click Finish
     • Click Activate Changes.

2. Configure node manager username and password:
     • Login to weblogic console and take lock & edit.
     • Under domain Structure click on domain_name on left hand side.
     • Go to security tab and click on Advanced.
     • Configure/update NodeManager Username and password.
     • Click save and click Active Changes.

3. Enroll Node Manager:
     • Make sure admin server is up and running.
     • Login to host
     • Go to $ORACLE_HOME/oracle_common/common/bin and run ./wlst.sh
     • Use Connect command to connect to admin server
        connect('<adminuser>','<password>','t3://<admin_host>:<admin_port>')
     • Run nmEnroll
        nmEnroll('<DOMAIN_HOME>', '<NODEMANAGER_HOME>')
        E.g.
        nmEnroll('/u01/Oracle/Middleware/user_projects/domains/base_domain',                '/u01/Oracle/Middleware/wlserver_10.3/common/nodemanager')
     • Once it is Successfully enrolled to machine with the domain directory check
        $WL_Home/common/nodemanager/nodemanager.domains file. It should have domain entry.
     • Set StartScriptEnabled=true and StopScriptEnabled=true in $WL_Home/common/nodemanager/nodemanager.properties file.
     • Restart the nodemanager and check the status in weblogic console if it is in active status or not.
     • Go to Environment -> Machines -> Click on Machine name
     • Go to Monitoring tab and verify Node Manager Status

Node Manager most commonly used commands

========================================================================
Applies to:
Weblogic: 10.3.6 and later
========================================================================
Below are the Node Manager commands which are most commonly uses in day to day admin activities for starting, stopping and restarting of weblogic servers.

1. nmConnect
Connects WLST to Node Manager for invoking Node Manager Commands via WLST. Node Manager must be running before we execute this command.
Syntax:
nmConnect(<username>, <password>, <host>, <port>, <domainName>, <domainDir>,<nmType>,<verbose>)
E.g.
nmConnect('weblogic','weblogic123','localhost','24500','demo_domain')

2. nmDisconnect
Disconnect WLST from a Node Manager session.
Syntax:
nmDisconnect()
E.g.
wls:/nm/demo_domain> nmDisconnect()
Successfully disconnected from Node Manager.
wls:/offline>

3. nm
Determine whether WLST is connected to Node Manager.
Syntax:
nm()
E.g.
wls:/offline> nm()
Not connected to Node Manager

4. nmKill
Kills the specified server instance that was started with Node Manager.
Syntax:
nmKill(<serverName>)
E.g.
nmKill(‘dev_poc_admin’)

5. nmServerStatus
Return the status of the server.
Syntax:
nmServerStatus(<serverName>)
E.g.
nmServerStatus(‘dev_poc_admin’)

6. nmStart
Start a server using Node Manager
Syntax:
nmStart(<serverName>)
E.g.
nmStart(‘dev_poc_admin’)

7. nmVersion
Return the Node Manager server version.
Syntax:
nmVersion()
E.g.
wls:/nm/ demo_domain> nmVersion()
The Node Manager version that you are currently connected to is 12.2.1.1.0.

8. nmLog
Return the Node Manager log
Syntax:
nmLog()
E.g.
wls:/nm/ demo_domain> nmLog()

Operation "exportMetadata" failure

========================================================================
Applies to:
BPEL : 12.2.1.1
Weblogic: 12.2.1.1
========================================================================
Problem Summary:
Unable to export MDS objects using wlst becuase of Exception MDS-91020/MDS-91009.

Error:
Case 1)
wls:/bpel_poc_dev/serverConfig/> exportMetadata(application='soa-infra', server='bpel_poc_dev_ms01',toLocation='/u01/puru/Practice/BPELHandsOn/12211/MDS/soa-infra_metadata.zip', docs='/**', remote=’true’)
Location changed to domainRuntime tree. This is a read-only tree
with DomainMBean as the root MBean.
For more help, use help('domainRuntime')

Executing operation: exportMetadata.

Traceback (innermost last):
  File "<console>", line 1, in ?
  File "/u01/puru/Practice/BPELHandsOn/bpel_poc_dev/mw_home/oracle_home/oracle_common/modules/internal/features/oracle.fmwshare.pyjar.jar!/wlstScriptDir/mdsWLSTCommands.py", line 251, in exportMetadata
  File "/u01/puru/Practice/BPELHandsOn/bpel_poc_dev/mw_home/oracle_home/oracle_common/modules/internal/features/oracle.fmwshare.pyjar.jar!/wlstScriptDir/mdsWLSTCommands.py", line 699, in executeAppRuntimeMBeanOperation
  File "/u01/puru/Practice/BPELHandsOn/bpel_poc_dev/mw_home/oracle_home/oracle_common/modules/internal/features/oracle.fmwshare.pyjar.jar!/wlstScriptDir/mdsWLSTCommands.py", line 661, in getMDSAppRuntimeMBean
UserWarning: MDS-91020: MDS Application runtime MBean for "soa-infra" is not available in server "bpel_poc_dev_ms01". Ensure server name and application name are correct. "exportMetadata" operation failure.
wls:/bpel_poc_dev/serverConfig/>

Case 2)
wls:/bpel_poc_dev/serverConfig/> exportMetadata(application='soa-infra', server='bpel_poc_dev_soa_ms01',toLocation='C:/PData_2Feb2015AfterCrash/Lo
anUnitBckup/Practice/BPELHandsOn/12211/MDS/', docs='/**', remote='true')

Executing operation: exportMetadata.
Exporting metadata from repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Metadata tranfer operation failed
java.io.FileNotFoundException: C:\PData_2Feb2015AfterCrash\LoanUnitBckup\Practice\BPELHandsOn\12211\MDS (Access is denied)
MDS-91009: Operation "exportMetadata" failure.

Traceback (innermost last):
  File "<console>", line 1, in ?
  File "/C:/oracle12211/Middleware/Oracle_Home/oracle_common/modules/internal/features/oracle.fmwshare.pyjar.jar!/wlstScriptDir/mdsWLSTCommands.py", l
ine 249, in exportMetadata
  File "/C:/oracle12211/Middleware/Oracle_Home/oracle_common/modules/internal/features/oracle.fmwshare.pyjar.jar!/wlstScriptDir/mdsWLSTCommands.py", l
ine 748, in executeMetadataTransferManagerOperation
  File "/C:/oracle12211/Middleware/Oracle_Home/oracle_common/modules/internal/features/oracle.fmwshare.pyjar.jar!/wlstScriptDir/mdsWLSTCommands.py", l
ine 759, in saveStackAndRaiseException
WLSTException: C:\PData_2Feb2015AfterCrash\LoanUnitBckup\Practice\BPELHandsOn\12211\MDS (Access is denied) MDS-91009: Operation "exportMetadata" failu
re. Use dumpStack() to view the full stacktrace.
wls:/bpel_poc_dev/serverConfig/>

========================================================================
Root Cause:
Case 1)
UserWarning: MDS-91020: MDS Application runtime MBean for "soa-infra" is not available in server "bpel_poc_dev_ms01". Ensure server name and application name are correct. "exportMetadata" operation failure.
Above warning clearly shows that either soa-infra isn't running or server name isn't correct.
In my case have used wrong server name bpel_poc_dev_ms01 in the wlst export metadata command.

Case 2)
Metadata export opration failed becuase of transfer failure.
Error says Access is denied on toLocation directory C:\PData_2Feb2015AfterCrash\LoanUnitBckup\Practice\BPELHandsOn\12211\MDS. We have verified permissions and ownership of directory. It has full access for user.
Error is kind of misguiding which says Access is denied but actually file name is missing in toLocation.
toLocation is expecting filename as well along with directory path where we would like to export MDS.
========================================================================
Resolution:
Case 1)
Use valid server name in server parameter of exportMetadata command.
Have changed server name from bpel_poc_dev_ms01 to bpel_poc_dev_soa_ms01 to fix MDS-91020: MDS Application error.
  exportMetadata(application='soa-infra',       server='bpel_poc_dev_soa_ms01',toLocation='/u01/puru/Practice/BPELHandsOn/12211/MDS/soa-infra_metadata.zip', docs='/**', remote=’true’)
Run above exportMetadata command. It's going to export MDS objects to location /u01/puru/Practice/BPELHandsOn/12211/MDS/soa-infra_metadata.zip

Case 2)
Change toLocation value to C:/PData_2Feb2015AfterCrash/LoanUnitBckup/Practice/BPELHandsOn/12211/MDS/soa-infra_metadata.zip
Rerun command,
wls:/bpel_poc_dev/serverConfig/> exportMetadata(application='soa-infra', server='bpel_poc_dev_soa_ms01',toLocation='C:/PData_2Feb2015AfterCrash/LoanUnitBckup/Practice/BPELHandsOn/12211/MDS/soa-infra_metadata.zip', docs='/**', remote=’true’)

Export MDS using EM console

========================================================================
Applies to:
BPEL : 12.2.1.1
Weblogic: 12.2.1.1
========================================================================
In below post we are going to see how to import, export and change/delete files from MDS using em console.
MDS is a repository to store shared resources like XSD, WSDL, DVM, XREF, FAULT-POLICIES etc.

Export:
1. Log in to em console using admin credentials.
2. Expand SOA folder and right click on soa-infra.
3. Go to Administration => MDS Configuration.
4. Select "Export metadata documents to an archive on the machine where this web browser is running" and click Export
5. It's going to take some time to fetch data from MDS store so please wait.
6. You will receive a pop-up for saving MDS.
7. Download/save soa-infra_metadata.zip to your local system.

bpel:21: error: There is no start activity in process

========================================================================
Applies to:
BPEL : 12.2.1.1
========================================================================
Problem Summary:
While trying to compile bpel code getting bpel:21: error: There is no start activity in process because of which build is failed.

Have used BPEL template defined service later for BPEL process.
It has a receive activity.

Error:
BPEL/FileBpelProcess.bpel:21: error: There is no start activity in process "FileBpelProcess"

Buildfile: C:\oracle12211\Middleware\Oracle_Home\soa\bin\ant-sca-compile.xml

scac:
    [input] skipping input as property scac.input has already been set.
     [scac] Validating composite "C:\JDeveloper\mywork\SOAPractice\fileAdapterDemo\SOA\composite.xml"
     [scac] soa.mds.consolidation is set to true
     [scac] Setting BPELC option 'classpath' to
.
.
.
.
     [scac] BPEL/FileBpelProcess.bpel:21: error: There is no start activity in process "FileBpelProcess"

BUILD FAILED
C:\oracle12211\Middleware\Oracle_Home\soa\bin\ant-sca-compile.xml:344: Java returned: 1 Check log file : C:\JDeveloper\mywork\SOAPractice\fileAdapterDemo\SOA\SCA-INF\classes\scac.log for errors

Total time: 5 seconds
========================================================================
Root Cause:
Above issue arises when we use receive activity in BPEL with createInstance as no.
By default it's value is no when we are using defined service later bpel template.

.bpel Code Snippet:
<receive name="ReceiveInput" partnerLink="pollFile" portType="ns1:Read_ptt" operation="Read"
             variable="ReceiveInput_Read_InputVariable" createInstance="no"/>

========================================================================
Resolution:
1. Go to project => BPEL => Open BPEL process (FileBpelProcess.bpel)
2. Double-click on the Receive activity.
3. Check the "Create Instance" checkbox and click on "OK".
4. Recompile project. This should fix your issue.

BPEL code snippet after change:
<receive name="ReceiveInput" partnerLink="pollFile" portType="ns1:Read_ptt" operation="Read"
             variable="ReceiveInput_Read_InputVariable" createInstance="yes"/>

While invoking HTTPS external webservice from OSB receiving "The invocation resulted in an error: General SSLEngine problem"

========================================================================
Applies to:
Weblogic Server: 12.2.1.1
OSB: 12.2.1.1
========================================================================
Problem Summary:
Unable to invoke HTTPS external webservice from OSB because of General SSLEngine problem.

Error:
Caused By: com.bea.wli.sb.transports.TransportException: General SSLEngine problem
        at com.bea.wli.sb.transports.TransportException.newInstance(TransportException.java:214)
        at com.bea.wli.sb.transports.http.HttpOutboundMessageContext.send(HttpOutboundMessageContext.java:527)
        at com.bea.wli.sb.transports.http.wls.HttpTransportProvider.sendMessageAsync(HttpTransportProvider.java:238)
        at sun.reflect.GeneratedMethodAccessor1632.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.bea.wli.sb.transports.Util$1.invoke(Util.java:80)
        at com.sun.proxy.$Proxy214.sendMessageAsync(Unknown Source)

After enabling DebugSecuritySSL flag able to get exact error in the logs,

The invocation resulted in an error: [Security:090497]HANDSHAKE_FAILURE alert received from <<3dnshost>> - <<ip>>. Check both sides of the SSL configuration for mismatches in supported ciphers, supported protocol versions, trusted CAs, and hostname verification settings..

The invocation resulted in an error: [Security:090477]Certificate chain received from <<3dnshost>> - <<ip>> was not trusted causing SSL handshake failure..
========================================================================
Root Cause:
Rectly external webservice host system has changed CERTIFICATES and external webservice certificates(root and intermediate) doesn't exists in public key store of weblogic $JAVA_HOME/jre/lib/security/cacerts.
========================================================================
How to verify:
1. Use keytool command to verify certificates in weblogic public key store.
$JAVA_HOME/bin/keytool -list -v -keystore $JAVA_HOME/jre/lib/security/cacerts
2. Provide password
========================================================================
Resolution:
1. Use https://<host>:<port>/<uri> to get external service root and intermediate certificates.
2. Open external webservice https://<host>:<port>/<uri> in IE browser.
3. Click on security report (lock icon) in url section
4. Click on view certificates.
5. Go to certificate tab => select root certificate and click view certificate.
6. Go to Details tab and click on Copy to file
7. Export cert wizard will open. Export cert as Base-64 encoded X.509 in .cer format.
8. Use unique name to save it.
9. Repeat step 5 to step 8 for intermediate certificate.
10. Once you have root and intermediate certificate, use keytool command to import certificates to cacert.
$JAVA_HOME/bin/keytool -import -trustcacerts -file /path/of/cert/<<name.cer>> -alias <<ALIAS NAME>> -keystore $JAVA_HOME/jre/lib/security/cacerts
11. Once import is done restart domain for changes to take effect and verify call.

Unable to start node manager because of java.io.IOException: Could not obtain exclusive lock with lockFile

========================================================================
Applies to:
Weblogic Server: 12.2.1.1
========================================================================
Problem Summary:
Unable to start node manager because of java.io.IOException: Could not obtain exclusive lock with lockFile.

Error Message:
Feb 11, 2017 12:38:05 PM weblogic.nodemanager.server.NMServer main
SEVERE: Fatal error in NodeManager server
java.io.IOException: Could not obtain exclusive lock with lockFile '/u01/domain/poc/osb/osb_poc_dit/nodemanager/nodemanager.process.lck'. This most likely indicates another NodeManager process is already running in this NodeManaagerHome.
        at weblogic.nodemanager.server.NMServer.getNMFileLock(NMServer.java:228)
        at weblogic.nodemanager.server.NMServer.<init>(NMServer.java:127)
        at weblogic.nodemanager.server.NMServer.main(NMServer.java:527)
        at weblogic.NodeManager.main(NodeManager.java:31)

========================================================================
Root Cause:
Error clearly states that NodeManager process is already running in this NodeManaagerHome.

========================================================================
How to verify:
Login to host and verify if node manager is running using below commands,
/usr/sbin/lsof -i:NM_PORT
OR
ps -ef | grep weblogic.NodeManager

========================================================================
Resolution:
1. Node manager is already running. No action needed and it's expected behaviour.
Try to perform intended activities like server restart/shutdown/start using node manager.
2. If you would like to restart node manager then shutdown node manager first and restart.