Weblogic server startup stuck at BEA-002014 IIOP subsystem enabled.

========================================================================
Applies to:
Weblogic: 10.3.6 and later
========================================================================
Problem Summary:
Weblogic admin server hangs at <BEA-002014> <IIOP subsystem enabled.> during restart.

Error:
<Dec 1, 2016 1:56:18 PM IST> <Info> <Server> <BEA-002609> <Channel Service initialized.>
<Dec 1, 2016 1:56:18 PM IST> <Info> <Socket> <BEA-000436> <Allocating 3 reader threads.>
<Dec 1, 2016 1:56:18 PM IST> <Info> <Socket> <BEA-000446> <Native IO Enabled.>
<Dec 1, 2016 1:56:18 PM IST> <Info> <IIOP> <BEA-002014> <IIOP subsystem enabled.>

========================================================================
Root Cause:
Issue can be due to corrupted LDAP. It's stuck at security/embedded LDAP start.

========================================================================
Resolution:
1. Login to admin server host
2. Take back up of $DOMAIN_HOME/servers/<SERVER_NAME>/data/ldap directory.
3. Restore ldap from working backup file $DOMAIN_HOME/servers/<SERVER_NAME>/data/ldap/backup
     OR
    We can start welogic admin server and ldap direcotry will get automatically created.
    Kindly note if you restart admin server without restoring ldap, all users access will get revoked except weblogic user.

Data Source test failed with Error ORA-01045: user SOA_DEMO lacks CREATE SESSION privilege

========================================================================
Applies to:
Weblogic: 10.3.6 and later
========================================================================
Problem Summary:
Application requires a XA data source to be configured on weblogic for DB connection using SOA_DEMO user.
Tried to configure it as per requirement but during Test configuration connection test failed with Error ORA-01045: user SOA_DEMO lacks CREATE SESSION privilege

Error:
Connection test failed.
Error ORA-01045: user SOA_DEMO lacks CREATE SESSION privilege; logon denied <br/>oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450)<br/>oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:392)<br/>oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:385)<br/>oracle.jdbc.driver.T4CTTIfun.processError(T4CTTIfun.java:1018)<br/>oracle.jdbc.driver.T4CTTIoauthenticate.processError(T4CTTIoauthenticate.java:501)<br/>oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:522)<br/>oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)<br/>oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:437)<br/>oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:954)<br/>oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:639)<br/>oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:666)<br/>oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)<br/>oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:566)<br/>oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:317)<br/>oracle.jdbc.xa.client.OracleXADataSource.getPooledConnection(OracleXADataSource.java:486)<br/>oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:174)<br/>oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:109)<br/>weblogic.jdbc.common.internal.DataSourceUtil.testConnection0(DataSourceUtil.java:356)<br/>weblogic.jdbc.common.internal.DataSourceUtil.access$000(DataSourceUtil.java:22)<br/>weblogic.jdbc.common.internal.DataSourceUtil$1.run(DataSourceUtil.java:254)<br/>.

========================================================================
Root Cause:
User SOA_DEMO doesn't have connect privileges.

========================================================================
Resolution:
1. Login to DB as SYS user.
2. Grant connet and resource to SOA_DEMO using below command.
    grant connect, resource to user_name identified by password;
    e.g. grant connect, resource to SOA_DEMO identified by Welcome1;
3. Once done, go back to weblogic console and test data source.

How to shutdown or restart IBM DataPower Appliance

========================================================================
Applies To:
IBM Datapower : XI52
IBM Datapower Gateways : 7.2.0
========================================================================
Purpose:
Shutdown or restart DataPower appliance.

Before we go to shutdown or restart procedure let's learn shutdown parameters which are needed during shutdown or restart of DataPower Appliance.

Shutdown DataPower parameters:
1. Reboot
    It will shutdown and restart the appliance.
2. Reload
    It will restart the appliance.
3. Halt
    It will shutdown the appliance.
4. Seconds
    Specifies the number of seconds delay before the appliance starts the shutdown operation.

Shutdown or restart using GUI/Console:
1. Login to DataPower WebGui
2. Under default domain go to Administration => Main => System Control
3. Scroll Down and go to Shutdown section
4. Select mode(e.g. Reboot System) and Delay (e.g. 10 Secs)
5. Click on Shutdown

Shutdown or restart using CLI:
1. Login to DataPower host
2. Go to default domain
3. Execute command
    shutdown reboot 10
    After 10 secs it will shutdown and reboot datapower appliance.
4. To reload use,
    shutdown reload 30
    To halt use,
    shutdown halt 30

oracle.mds.config.MDSConfigurationException: MDS-01330: unable to load MDS configuration document

========================================================================
Applies to:
BPEL : 12.2.1.1
JDEV: 12.2.1.1
========================================================================
Problem Summary:
Have created empty BPEL project MDSDemoProject which we are going to use as MDS project for shared artifacts.
During compilation of BPEL project facing oracle.mds.config.MDSConfigurationException: MDS-01330: unable to load MDS configuration document

Error:
oracle.fabric.common.FabricException: oracle.mds.config.MDSConfigurationException: MDS-01330: unable to load MDS configuration document  
oracle.xml.parser.v2.XMLParseException; systemId: file:/C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml; lineNumber: 28; columnNumber: 18; '--' is not allowed in comments.
at oracle.fabric.common.FabricMetadataManagerFactory.createMetadataManager(FabricMetadataManagerFactory.java:219)
at oracle.fabric.common.FabricMetadataManagerFactory.createMetadataManager(FabricMetadataManagerFactory.java:185)
at oracle.fabric.composite.model.ModuleConfigModel.initializeMetadataManager(ModuleConfigModel.java:116)
at oracle.fabric.composite.model.CompositeModel.getClientMetadataManager(CompositeModel.java:302)
at oracle.soa.scac.ValidateComposite.loadComposite(ValidateComposite.java:901)
at oracle.soa.scac.ValidateComposite.doValidation(ValidateComposite.java:571)
at oracle.soa.scac.ValidateComposite.run0(ValidateComposite.java:228)
at oracle.soa.scac.ValidateComposite.run(ValidateComposite.java:209)
at oracle.soa.scac.ValidateComposite.main(ValidateComposite.java:201)
Caused by: oracle.fabric.common.FabricException: oracle.mds.config.MDSConfigurationException: MDS-01330: unable to load MDS configuration document  
oracle.xml.parser.v2.XMLParseException; systemId: file:/C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml; lineNumber: 28; columnNumber: 18; '--' is not allowed in comments.: MDS-01330: unable to load MDS configuration document  
oracle.xml.parser.v2.XMLParseException; systemId: file:/C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml; lineNumber: 28; columnNumber: 18; '--' is not allowed in comments.
at oracle.integration.platform.common.MDSMetadataManagerImpl.init(MDSMetadataManagerImpl.java:463)
at oracle.integration.platform.common.MDSMetadataManagerImpl.<init>(MDSMetadataManagerImpl.java:277)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at oracle.fabric.common.FabricMetadataManagerFactory.createMetadataManager(FabricMetadataManagerFactory.java:215)
... 8 more
Caused by: oracle.mds.config.MDSConfigurationException: MDS-01330: unable to load MDS configuration document  
oracle.xml.parser.v2.XMLParseException; systemId: file:/C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml; lineNumber: 28; columnNumber: 18; '--' is not allowed in comments.
at oracle.mds.config.MDSConfig.<init>(MDSConfig.java:1016)
at oracle.mds.config.MDSConfig.<init>(MDSConfig.java:919)
at oracle.integration.platform.common.MDSUtil.getMDSConfig(MDSUtil.java:102)
at oracle.integration.platform.common.MDSUtil.getMDSInstance(MDSUtil.java:199)
at oracle.integration.platform.common.MDSMetadataManagerImpl.getMDSInstance(MDSMetadataManagerImpl.java:1053)
at oracle.integration.platform.common.MDSMetadataManagerImpl.init(MDSMetadataManagerImpl.java:418)
... 14 more
Caused by: oracle.xml.parser.v2.XMLParseException; systemId: file:/C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml; lineNumber: 28; columnNumber: 18; '--' is not allowed in comments.
at oracle.xml.parser.v2.XMLError.flushErrors1(XMLError.java:326)
at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:438)
at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:321)
at oracle.mds.internal.util.xml.OracleXDKXMLUtilImpl.parse(OracleXDKXMLUtilImpl.java:116)
at oracle.mds.internal.util.xml.DOMUtil.parse(DOMUtil.java:576)
at oracle.mds.internal.config.ConfigurationUtils.getConfigElement(ConfigurationUtils.java:145)
at oracle.mds.config.MDSConfig.<init>(MDSConfig.java:969)
... 19 more

Compilation of project 'MDSDemoProject.jpr' finished. Check 'C:\JDeveloper\mywork\MDSDemoApplication\MDSDemoProject\SOA\SCA-INF\classes\scac.log' for details.
[9:30:09 AM] Compilation complete: 1 errors, 0 warnings.
[9:30:09 AM] Build time 2,801 msec.
========================================================================
Root Cause:
Have done a silly mistake. Error clearly says that '--' is not allowed in comments and file which is having '--' in comments is file:/C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml;

oracle.xml.parser.v2.XMLParseException; systemId: file:/C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml; lineNumber: 28; columnNumber: 18; '--' is not allowed in comments.: MDS-01330: unable to load MDS configuration document

adf-config.xml code snippet:

         <!-- UNCOMMENT Only if you would like to use file based MDS
         <metadata-store-usage id="mstore-usage_2">
            <metadata-store class-name="oracle.mds.persistence.stores.file.FileMetadataStore">
            <!-- Not Needed For apps <property name="partition-name" value="seed"/>
              <property name="metadata-path" value="C:\oracle12211\Middleware\Oracle_Home\jdeveloper\integration"/>
            </metadata-store>
          </metadata-store-usage>-->

========================================================================
Resolution:
Remove <!-- from C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml file and recompile

adf-config.xml code Snippet after change:

         <!-- UNCOMMENT Only if you would like to use file based MDS
         <metadata-store-usage id="mstore-usage_2">
            <metadata-store class-name="oracle.mds.persistence.stores.file.FileMetadataStore">
            Not Needed For apps <property name="partition-name" value="seed"/>
              <property name="metadata-path" value="C:\oracle12211\Middleware\Oracle_Home\jdeveloper\integration"/>
            </metadata-store>
          </metadata-store-usage>-->

Compiling...
Compilation of project 'MDSDemoProject.jpr' finished. Check 'C:\JDeveloper\mywork\MDSDemoApplication\MDSDemoProject\SOA\SCA-INF\classes\scac.log' for details.
[9:30:34 AM] Successful compilation: 0 errors, 0 warnings.
[9:30:34 AM] Build time 5,219 msec.

How to Enroll a Node Manager in weblogic?

========================================================================
Applies to:
Weblogic: 10.3.6 and later
========================================================================
nmEnroll is used to enroll a machine or a domain with the node manager.
Few of us may forget to configure NodeManager after installing Weblogic server. Follow below steps to enroll nodemanager manually in weblogic.
1. Create a Machine:
     • Login to weblogic console
     • Click on Lock & Edit button in change center.
     • Go to Environment -> Machines -> click New
     • Provide name and select machine OS
     • Click Next
     • Select type, provide host and port details for node manager and click Finish
     • Click Activate Changes.

2. Configure node manager username and password:
     • Login to weblogic console and take lock & edit.
     • Under domain Structure click on domain_name on left hand side.
     • Go to security tab and click on Advanced.
     • Configure/update NodeManager Username and password.
     • Click save and click Active Changes.

3. Enroll Node Manager:
     • Make sure admin server is up and running.
     • Login to host
     • Go to $ORACLE_HOME/oracle_common/common/bin and run ./wlst.sh
     • Use Connect command to connect to admin server
        connect('<adminuser>','<password>','t3://<admin_host>:<admin_port>')
     • Run nmEnroll
        nmEnroll('<DOMAIN_HOME>', '<NODEMANAGER_HOME>')
        E.g.
        nmEnroll('/u01/Oracle/Middleware/user_projects/domains/base_domain',                '/u01/Oracle/Middleware/wlserver_10.3/common/nodemanager')
     • Once it is Successfully enrolled to machine with the domain directory check
        $WL_Home/common/nodemanager/nodemanager.domains file. It should have domain entry.
     • Set StartScriptEnabled=true and StopScriptEnabled=true in $WL_Home/common/nodemanager/nodemanager.properties file.
     • Restart the nodemanager and check the status in weblogic console if it is in active status or not.
     • Go to Environment -> Machines -> Click on Machine name
     • Go to Monitoring tab and verify Node Manager Status

Node Manager most commonly used commands

========================================================================
Applies to:
Weblogic: 10.3.6 and later
========================================================================
Below are the Node Manager commands which are most commonly uses in day to day admin activities for starting, stopping and restarting of weblogic servers.

1. nmConnect
Connects WLST to Node Manager for invoking Node Manager Commands via WLST. Node Manager must be running before we execute this command.
Syntax:
nmConnect(<username>, <password>, <host>, <port>, <domainName>, <domainDir>,<nmType>,<verbose>)
E.g.
nmConnect('weblogic','weblogic123','localhost','24500','demo_domain')

2. nmDisconnect
Disconnect WLST from a Node Manager session.
Syntax:
nmDisconnect()
E.g.
wls:/nm/demo_domain> nmDisconnect()
Successfully disconnected from Node Manager.
wls:/offline>

3. nm
Determine whether WLST is connected to Node Manager.
Syntax:
nm()
E.g.
wls:/offline> nm()
Not connected to Node Manager

4. nmKill
Kills the specified server instance that was started with Node Manager.
Syntax:
nmKill(<serverName>)
E.g.
nmKill(‘dev_poc_admin’)

5. nmServerStatus
Return the status of the server.
Syntax:
nmServerStatus(<serverName>)
E.g.
nmServerStatus(‘dev_poc_admin’)

6. nmStart
Start a server using Node Manager
Syntax:
nmStart(<serverName>)
E.g.
nmStart(‘dev_poc_admin’)

7. nmVersion
Return the Node Manager server version.
Syntax:
nmVersion()
E.g.
wls:/nm/ demo_domain> nmVersion()
The Node Manager version that you are currently connected to is 12.2.1.1.0.

8. nmLog
Return the Node Manager log
Syntax:
nmLog()
E.g.
wls:/nm/ demo_domain> nmLog()

Operation "exportMetadata" failure

========================================================================
Applies to:
BPEL : 12.2.1.1
Weblogic: 12.2.1.1
========================================================================
Problem Summary:
Unable to export MDS objects using wlst becuase of Exception MDS-91020/MDS-91009.

Error:
Case 1)
wls:/bpel_poc_dev/serverConfig/> exportMetadata(application='soa-infra', server='bpel_poc_dev_ms01',toLocation='/u01/puru/Practice/BPELHandsOn/12211/MDS/soa-infra_metadata.zip', docs='/**', remote=’true’)
Location changed to domainRuntime tree. This is a read-only tree
with DomainMBean as the root MBean.
For more help, use help('domainRuntime')

Executing operation: exportMetadata.

Traceback (innermost last):
  File "<console>", line 1, in ?
  File "/u01/puru/Practice/BPELHandsOn/bpel_poc_dev/mw_home/oracle_home/oracle_common/modules/internal/features/oracle.fmwshare.pyjar.jar!/wlstScriptDir/mdsWLSTCommands.py", line 251, in exportMetadata
  File "/u01/puru/Practice/BPELHandsOn/bpel_poc_dev/mw_home/oracle_home/oracle_common/modules/internal/features/oracle.fmwshare.pyjar.jar!/wlstScriptDir/mdsWLSTCommands.py", line 699, in executeAppRuntimeMBeanOperation
  File "/u01/puru/Practice/BPELHandsOn/bpel_poc_dev/mw_home/oracle_home/oracle_common/modules/internal/features/oracle.fmwshare.pyjar.jar!/wlstScriptDir/mdsWLSTCommands.py", line 661, in getMDSAppRuntimeMBean
UserWarning: MDS-91020: MDS Application runtime MBean for "soa-infra" is not available in server "bpel_poc_dev_ms01". Ensure server name and application name are correct. "exportMetadata" operation failure.
wls:/bpel_poc_dev/serverConfig/>

Case 2)
wls:/bpel_poc_dev/serverConfig/> exportMetadata(application='soa-infra', server='bpel_poc_dev_soa_ms01',toLocation='C:/PData_2Feb2015AfterCrash/Lo
anUnitBckup/Practice/BPELHandsOn/12211/MDS/', docs='/**', remote='true')

Executing operation: exportMetadata.
Exporting metadata from repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Metadata tranfer operation failed
java.io.FileNotFoundException: C:\PData_2Feb2015AfterCrash\LoanUnitBckup\Practice\BPELHandsOn\12211\MDS (Access is denied)
MDS-91009: Operation "exportMetadata" failure.

Traceback (innermost last):
  File "<console>", line 1, in ?
  File "/C:/oracle12211/Middleware/Oracle_Home/oracle_common/modules/internal/features/oracle.fmwshare.pyjar.jar!/wlstScriptDir/mdsWLSTCommands.py", l
ine 249, in exportMetadata
  File "/C:/oracle12211/Middleware/Oracle_Home/oracle_common/modules/internal/features/oracle.fmwshare.pyjar.jar!/wlstScriptDir/mdsWLSTCommands.py", l
ine 748, in executeMetadataTransferManagerOperation
  File "/C:/oracle12211/Middleware/Oracle_Home/oracle_common/modules/internal/features/oracle.fmwshare.pyjar.jar!/wlstScriptDir/mdsWLSTCommands.py", l
ine 759, in saveStackAndRaiseException
WLSTException: C:\PData_2Feb2015AfterCrash\LoanUnitBckup\Practice\BPELHandsOn\12211\MDS (Access is denied) MDS-91009: Operation "exportMetadata" failu
re. Use dumpStack() to view the full stacktrace.
wls:/bpel_poc_dev/serverConfig/>

========================================================================
Root Cause:
Case 1)
UserWarning: MDS-91020: MDS Application runtime MBean for "soa-infra" is not available in server "bpel_poc_dev_ms01". Ensure server name and application name are correct. "exportMetadata" operation failure.
Above warning clearly shows that either soa-infra isn't running or server name isn't correct.
In my case have used wrong server name bpel_poc_dev_ms01 in the wlst export metadata command.

Case 2)
Metadata export opration failed becuase of transfer failure.
Error says Access is denied on toLocation directory C:\PData_2Feb2015AfterCrash\LoanUnitBckup\Practice\BPELHandsOn\12211\MDS. We have verified permissions and ownership of directory. It has full access for user.
Error is kind of misguiding which says Access is denied but actually file name is missing in toLocation.
toLocation is expecting filename as well along with directory path where we would like to export MDS.
========================================================================
Resolution:
Case 1)
Use valid server name in server parameter of exportMetadata command.
Have changed server name from bpel_poc_dev_ms01 to bpel_poc_dev_soa_ms01 to fix MDS-91020: MDS Application error.
  exportMetadata(application='soa-infra',       server='bpel_poc_dev_soa_ms01',toLocation='/u01/puru/Practice/BPELHandsOn/12211/MDS/soa-infra_metadata.zip', docs='/**', remote=’true’)
Run above exportMetadata command. It's going to export MDS objects to location /u01/puru/Practice/BPELHandsOn/12211/MDS/soa-infra_metadata.zip

Case 2)
Change toLocation value to C:/PData_2Feb2015AfterCrash/LoanUnitBckup/Practice/BPELHandsOn/12211/MDS/soa-infra_metadata.zip
Rerun command,
wls:/bpel_poc_dev/serverConfig/> exportMetadata(application='soa-infra', server='bpel_poc_dev_soa_ms01',toLocation='C:/PData_2Feb2015AfterCrash/LoanUnitBckup/Practice/BPELHandsOn/12211/MDS/soa-infra_metadata.zip', docs='/**', remote=’true’)

Export MDS using EM console

========================================================================
Applies to:
BPEL : 12.2.1.1
Weblogic: 12.2.1.1
========================================================================
In below post we are going to see how to import, export and change/delete files from MDS using em console.
MDS is a repository to store shared resources like XSD, WSDL, DVM, XREF, FAULT-POLICIES etc.

Export:
1. Log in to em console using admin credentials.
2. Expand SOA folder and right click on soa-infra.
3. Go to Administration => MDS Configuration.
4. Select "Export metadata documents to an archive on the machine where this web browser is running" and click Export
5. It's going to take some time to fetch data from MDS store so please wait.
6. You will receive a pop-up for saving MDS.
7. Download/save soa-infra_metadata.zip to your local system.

bpel:21: error: There is no start activity in process

========================================================================
Applies to:
BPEL : 12.2.1.1
========================================================================
Problem Summary:
While trying to compile bpel code getting bpel:21: error: There is no start activity in process because of which build is failed.

Have used BPEL template defined service later for BPEL process.
It has a receive activity.

Error:
BPEL/FileBpelProcess.bpel:21: error: There is no start activity in process "FileBpelProcess"

Buildfile: C:\oracle12211\Middleware\Oracle_Home\soa\bin\ant-sca-compile.xml

scac:
    [input] skipping input as property scac.input has already been set.
     [scac] Validating composite "C:\JDeveloper\mywork\SOAPractice\fileAdapterDemo\SOA\composite.xml"
     [scac] soa.mds.consolidation is set to true
     [scac] Setting BPELC option 'classpath' to
.
.
.
.
     [scac] BPEL/FileBpelProcess.bpel:21: error: There is no start activity in process "FileBpelProcess"

BUILD FAILED
C:\oracle12211\Middleware\Oracle_Home\soa\bin\ant-sca-compile.xml:344: Java returned: 1 Check log file : C:\JDeveloper\mywork\SOAPractice\fileAdapterDemo\SOA\SCA-INF\classes\scac.log for errors

Total time: 5 seconds
========================================================================
Root Cause:
Above issue arises when we use receive activity in BPEL with createInstance as no.
By default it's value is no when we are using defined service later bpel template.

.bpel Code Snippet:
<receive name="ReceiveInput" partnerLink="pollFile" portType="ns1:Read_ptt" operation="Read"
             variable="ReceiveInput_Read_InputVariable" createInstance="no"/>

========================================================================
Resolution:
1. Go to project => BPEL => Open BPEL process (FileBpelProcess.bpel)
2. Double-click on the Receive activity.
3. Check the "Create Instance" checkbox and click on "OK".
4. Recompile project. This should fix your issue.

BPEL code snippet after change:
<receive name="ReceiveInput" partnerLink="pollFile" portType="ns1:Read_ptt" operation="Read"
             variable="ReceiveInput_Read_InputVariable" createInstance="yes"/>

While invoking HTTPS external webservice from OSB receiving "The invocation resulted in an error: General SSLEngine problem"

========================================================================
Applies to:
Weblogic Server: 12.2.1.1
OSB: 12.2.1.1
========================================================================
Problem Summary:
Unable to invoke HTTPS external webservice from OSB because of General SSLEngine problem.

Error:
Caused By: com.bea.wli.sb.transports.TransportException: General SSLEngine problem
        at com.bea.wli.sb.transports.TransportException.newInstance(TransportException.java:214)
        at com.bea.wli.sb.transports.http.HttpOutboundMessageContext.send(HttpOutboundMessageContext.java:527)
        at com.bea.wli.sb.transports.http.wls.HttpTransportProvider.sendMessageAsync(HttpTransportProvider.java:238)
        at sun.reflect.GeneratedMethodAccessor1632.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.bea.wli.sb.transports.Util$1.invoke(Util.java:80)
        at com.sun.proxy.$Proxy214.sendMessageAsync(Unknown Source)

After enabling DebugSecuritySSL flag able to get exact error in the logs,

The invocation resulted in an error: [Security:090497]HANDSHAKE_FAILURE alert received from <<3dnshost>> - <<ip>>. Check both sides of the SSL configuration for mismatches in supported ciphers, supported protocol versions, trusted CAs, and hostname verification settings..

The invocation resulted in an error: [Security:090477]Certificate chain received from <<3dnshost>> - <<ip>> was not trusted causing SSL handshake failure..
========================================================================
Root Cause:
Rectly external webservice host system has changed CERTIFICATES and external webservice certificates(root and intermediate) doesn't exists in public key store of weblogic $JAVA_HOME/jre/lib/security/cacerts.
========================================================================
How to verify:
1. Use keytool command to verify certificates in weblogic public key store.
$JAVA_HOME/bin/keytool -list -v -keystore $JAVA_HOME/jre/lib/security/cacerts
2. Provide password
========================================================================
Resolution:
1. Use https://<host>:<port>/<uri> to get external service root and intermediate certificates.
2. Open external webservice https://<host>:<port>/<uri> in IE browser.
3. Click on security report (lock icon) in url section
4. Click on view certificates.
5. Go to certificate tab => select root certificate and click view certificate.
6. Go to Details tab and click on Copy to file
7. Export cert wizard will open. Export cert as Base-64 encoded X.509 in .cer format.
8. Use unique name to save it.
9. Repeat step 5 to step 8 for intermediate certificate.
10. Once you have root and intermediate certificate, use keytool command to import certificates to cacert.
$JAVA_HOME/bin/keytool -import -trustcacerts -file /path/of/cert/<<name.cer>> -alias <<ALIAS NAME>> -keystore $JAVA_HOME/jre/lib/security/cacerts
11. Once import is done restart domain for changes to take effect and verify call.

Unable to start node manager because of java.io.IOException: Could not obtain exclusive lock with lockFile

========================================================================
Applies to:
Weblogic Server: 12.2.1.1
========================================================================
Problem Summary:
Unable to start node manager because of java.io.IOException: Could not obtain exclusive lock with lockFile.

Error Message:
Feb 11, 2017 12:38:05 PM weblogic.nodemanager.server.NMServer main
SEVERE: Fatal error in NodeManager server
java.io.IOException: Could not obtain exclusive lock with lockFile '/u01/domain/poc/osb/osb_poc_dit/nodemanager/nodemanager.process.lck'. This most likely indicates another NodeManager process is already running in this NodeManaagerHome.
        at weblogic.nodemanager.server.NMServer.getNMFileLock(NMServer.java:228)
        at weblogic.nodemanager.server.NMServer.<init>(NMServer.java:127)
        at weblogic.nodemanager.server.NMServer.main(NMServer.java:527)
        at weblogic.NodeManager.main(NodeManager.java:31)

========================================================================
Root Cause:
Error clearly states that NodeManager process is already running in this NodeManaagerHome.

========================================================================
How to verify:
Login to host and verify if node manager is running using below commands,
/usr/sbin/lsof -i:NM_PORT
OR
ps -ef | grep weblogic.NodeManager

========================================================================
Resolution:
1. Node manager is already running. No action needed and it's expected behaviour.
Try to perform intended activities like server restart/shutdown/start using node manager.
2. If you would like to restart node manager then shutdown node manager first and restart.

Unable to start managed server using node manager becuase of java.io.IOException: Invalid state file format

========================================================================
Applies to:
Weblogic Server: 10.3.6 and later
========================================================================
Problem Summary:
Trying to start managed server using node manager but couldn't able to because of java.io.IOException: Invalid state file format.
There was a power failure and managed server host restarted automatically.

Error:
<Mar 2, 2016 3:16:43 AM CST> <WARNING> <I/O error while reading domain directory>
java.io.IOException: Invalid state file format. State file contents:
at weblogic.nodemanager.common.StateInfo.load(StateInfo.java:135)
at weblogic.nodemanager.server.ServerMonitor.loadStateInfo(ServerMonitor.java:475)
at weblogic.nodemanager.server.ServerMonitor.isCleanupAfterCrashNeeded(ServerMonitor.java:139)
at weblogic.nodemanager.server.ServerManager.recoverServer(ServerManager.java:255)
at weblogic.nodemanager.server.DomainManager.initialize(DomainManager.java:103)
at weblogic.nodemanager.server.DomainManager.<init>(DomainManager.java:55)
at weblogic.nodemanager.server.NMServer.getDomainManager(NMServer.java:257)
at weblogic.nodemanager.server.Handler.handleDomain(Handler.java:224)
at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:108)
at weblogic.nodemanager.server.Handler.run(Handler.java:70)
at java.lang.Thread.run(Thread.java:619)

========================================================================
Root Cause:
managed_server.state file is invalid/corrupted which is located under $DOMAIN_HOME/servers/<managed_server>/data/nodemanager

========================================================================
Resolution:
1. Stop node manager of managed server.
2. Delete the below files,
   $DOMAIN_HOME/servers/<managed_server>/data/nodemanager/<managed_server>.state
   $DOMAIN_HOME/servers/<managed_server>/data/nodemanager/<managed_server>.lck
   $DOMAIN_HOME/servers/<managed_server>/data/nodemanager/<managed_server>.pid
3. Start node manager.
4. Start the managed server using admin.

Unable to start node manager becuase of java.net.BindException: Address already in use

========================================================================
Applies to:
Weblogic Server: 10.3.6 and later
========================================================================
Problem Summary:
Trying to start node manager but couldn't able to because of java.net.BindException: Address already in use

Error:
<Jan 8, 2017 10:43:12 PM CST> <SEVERE> <Fatal error in NodeManager server>
java.io.IOException: The server is unable to create a server socket for listening. The address puruhost might be incorrect or another process is using port 25000: java.net.BindException: Address already in use
        at weblogic.nodemanager.server.NMServer.start(NMServer.java:351)
        at weblogic.nodemanager.server.NMServer.main(NMServer.java:529)
        at weblogic.NodeManager.main(NodeManager.java:31)
Caused by: java.net.BindException: Address already in use
        at java.net.PlainSocketImpl.socketBind(Native Method)
        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
        at java.net.ServerSocket.bind(ServerSocket.java:375)
        at java.net.ServerSocket.<init>(ServerSocket.java:237)
        at javax.net.ssl.SSLServerSocket.<init>(SSLServerSocket.java:181)
        at sun.security.ssl.SSLServerSocketImpl.<init>(SSLServerSocketImpl.java:140)
        at sun.security.ssl.SSLServerSocketFactoryImpl.createServerSocket(SSLServerSocketFactoryImpl.java:95)
        at weblogic.nodemanager.server.SSLListener.privilegedInit(SSLListener.java:134)
        at weblogic.nodemanager.server.SSLListener.access$000(SSLListener.java:33)
        at weblogic.nodemanager.server.SSLListener$1.run(SSLListener.java:97)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.nodemanager.server.SSLListener.init(SSLListener.java:94)
        at weblogic.nodemanager.server.NMServer.start(NMServer.java:349)
        ... 2 more

========================================================================
Root Cause:
Port number used by node manager process is being already used by some other process.

========================================================================
How to verify:
1. Login to host
2. Verify process which is using node manager port using
 /usr/sbin/lsof -i:port_num
 netstat -an | grep port_num
 e.g. /usr/sbin/lsof -i:25000
netstat -an | grep 25000

========================================================================
Resolution:
1. Once you find a process which is using Node manager port and can be killed, kill it using,
 kill -9 PID
2. Restart node manager using $DOMAIN_HOME/bin/startNodeManager.sh

Not able to connect to node manager because of Connection refused: connect

========================================================================
Applies to:
Weblogic Server: 10.3.6 and later
========================================================================
Problem Summary:
We are trying to connect to node manager to perform restart using nmConnect but couldn't able to.

Error:
wls:/offline> nmConnect('weblogic','weblogic123','localhost','25000','weblogic_poc_dit')
Connecting to Node Manager ...
Traceback (innermost last):
  File "<console>", line 1, in ?
  File "<iostream>", line 123, in nmConnect
  File "<iostream>", line 648, in raiseWLSTException
WLSTException: Error occured while performing nmConnect : Cannot connect to Node
 Manager. : Connection refused: connect. Could not connect to NodeManager. Check
 that it is running at localhost:25000.
Use dumpStack() to view the full stacktrace
wls:/offline>
========================================================================
Root Cause:
Node manager isn't running at localhost:25000 becuase of which connection is refused.
========================================================================
How to verify:
1. Login to localhost
2. Use /usr/sbin/lsof -i:25000 or ps -ef | grep weblogic.NodeManager command to verify if node manager is up and running.
========================================================================
Resolution:
1. Go to $DOMAIN_HOME/bin and start node manager using startNodeManager.sh
<Feb 11, 2017 12:29:20 PM CST> <INFO> <Secure socket listener started on port 25000, host localhost/192.168.184.130>
2. Once it is started listening, try to connect to node manager using nmConnect
wls:/offline> nmConnect('weblogic','weblogic123','localhost','25000','weblogic_poc_dit')
Connecting to Node Manager ...
Successfully Connected to Node Manager.
wls:/nm/weblogic_poc_dit>

Couldn’t able to start Node Manager because of invalid java directory.

========================================================================

Applies to:

Weblogic Server: 10.3.6 and later

========================================================================

Problem Summary:

Couldn’t able to start Node Manager because of missing java directory.

Error:

C:\weblogic1036\Middleware\Oracle_Home\user_projects\domains\base_domain\bin>startNodeManager.cmd

NODEMGR_HOME is already set to C:\WEBLOG~1\MIDDLE~1\ORACLE~1\USER_P~1\domains\BASE_D~1\NODEMA~1

The JDK wasn't found in directory C:\oracle1221\Java\jdk1.8.0_66.

Please edit this script so that the JAVA_HOME

variable points to the location of your JDK.

C:\weblogic1036\Middleware\Oracle_Home\user_projects\domains\base_domain\bin>

========================================================================

Root Cause:

JAVA_HOME variable was set to invalid location C:\oracle1221\Java\jdk1.8.0_66 which is invalid location.

========================================================================

How to verify:

Use echo command to verify it.

C:\WINDOWS\system32>echo %JAVA_HOME%

C:\oracle1221\Java\jdk1.8.0_66

========================================================================

Resolution:

1. Go to Control Panel => System and Security => System => Advance System Settings => Environment Variables

2. Locate JAVA_HOME variable and change it’s value form invalid to valid location.

In our case from C:\oracle1221\Java\jdk1.8.0_66 to C:\oracle12211\Java\jdk1.8.0_66

3. Verify JAVA_HOME

C:\WINDOWS\system32>echo %JAVA_HOME%

C:\oracle12211\Java\jdk1.8.0_66

4. Start Node Manager,

Domain name mappings:

base_domain -> C:\weblogic1036\Middleware\Oracle_Home\user_projects\domains\base_domain

<Feb 11, 2017 1:16:36 PM IST> <INFO> <WebLogic Server 12.1.3.0.0  Wed May 21 18:53:34 PDT 2014 1604337 >

<Feb 11, 2017 1:16:36 PM IST> <INFO> <Secure socket listener started on port 5556, host localhost/127.0.0.1>

java.io.FileNotFoundException: Domain directory '/u01/oracle11g/mw1036/user_projects/domains/poc_domain' invalid (domain salt file not found)

========================================================================
Applies to:
Weblogic Server: 10.3.6 and later
========================================================================
Problem Summary:
Case 1: When we are trying to start managed server through the node manager facing java.io.FileNotFoundException: Domain directory '/u01/oracle11g/mw1036/user_projects/domains/poc_domain' invalid (domain salt file not found) error.

WebLogic server is complaining about a domain that is not correctly registered and enrolled for node Manager.

<Mar 2, 2016 3:16:43 AM CST> <WARNING> <There was a problem initializing the domain 'poc_domain' at '/u01/oracle11g/mw1036/user_projects/domains/poc_domain'. Please make sure that this domainName: 'poc_domain' is registered and is fully enrolled for this NodeManager at: '/u01/oracle11g/mw1036/user_projects/domains/poc_domain'.>

Case 2: Node Manager Status is Inactive on weblogic admin console,
Problem description:weblogic.nodemanager.NMException
Change: Node manager config was copied from box A to B of same domain, changed ListenAddress, Port and started nodemanager.

Error:
<Mar 2, 2016 3:16:43 AM CST> <WARNING> <I/O error while reading domain directory>
java.io.FileNotFoundException: Domain directory '/u01/oracle11g/mw1036/user_projects/domains/poc_domain' invalid (domain salt file not found)
        at weblogic.nodemanager.server.DomainManager.initialize(DomainManager.java:95)
        at weblogic.nodemanager.server.DomainManager.<init>(DomainManager.java:60)
        at weblogic.nodemanager.server.NMServer.findOrCreateDomainManager(NMServer.java:332)
        at weblogic.nodemanager.server.NMServer.getDomainManager(NMServer.java:316)
        at weblogic.nodemanager.server.Handler.handleDomain(Handler.java:277)
        at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:131)
        at weblogic.nodemanager.server.Handler.run(Handler.java:81)
        at java.lang.Thread.run(Thread.java:724)

========================================================================
Root Cause:
Case 1: Domain root directory(nodemanager.properties) was pointing to wrong directory.
Node manager was referring to wrong domain directory and looking for a SerializedSystemIni.dat file to decrypt the credentials required to start the server causing the exception.

Case 2: Domain isn't registered with node manager.
========================================================================
Resolution:
Case1:
1. Go to your nodemanager home,
$WL_HOME/common/nodemanager/
2. Edit nodemanager.domains file and make sure it has valid domain root directory.
3. Save your changes.
4. Restart Node Manager
5. Restart Server using Node manager.

Case2:
1. Enroll your nodemanager using NMEnroll command,
nmEnroll('$DOMAIN_HOME','$WL_HOME/common/nodemanager/')
2. Restart your node manager.
3. Check NM status on console.

java.lang.Exception: Failed to load url; https://host:port/tls12Tester/DIT?wsdl, 0 in SoapUI

========================================================================
Applies to:
SoapUI: 5.2.1
========================================================================
Problem Summary:
We aren't able to load a WSDL in SOAPUI which has TLS1.2 as minimum protocol version.

SOAPUI Error:
Error loading wsdl
There was something wrong with the wsdl you are trying to import
Error loading [https://host:port/tls12Tester/DIT?wsdl]: java.lang.Exception: Failed to load url; https://host:port/tls12Tester/DIT?wsdl, 0

2017-02-08 10:07:22,947 ERROR [errorlog] com.eviware.soapui.impl.support.definition.support.InvalidDefinitionException
com.eviware.soapui.impl.support.definition.support.InvalidDefinitionException
at com.eviware.soapui.impl.wsdl.support.wsdl.WsdlLoader.makeInvalidDefinitionException(WsdlLoader.java:119)
at com.eviware.soapui.impl.wsdl.support.wsdl.WsdlLoader.loadXmlObject(WsdlLoader.java:112)
at com.eviware.soapui.impl.wsdl.support.xsd.SchemaUtils.getDefinitionParts(SchemaUtils.java:488)
at com.eviware.soapui.impl.wsdl.support.xsd.SchemaUtils.getDefinitionParts(SchemaUtils.java:477)
at com.eviware.soapui.impl.support.definition.support.AbstractDefinitionCache.update(AbstractDefinitionCache.java:94)
at com.eviware.soapui.impl.support.definition.support.AbstractDefinitionContext$Loader.construct(AbstractDefinitionContext.java:209)
at com.eviware.soapui.support.swing.SwingWorkerDelegator.construct(SwingWorkerDelegator.java:46)
at com.eviware.soapui.support.swing.SwingWorker$2.run(SwingWorker.java:131)
at java.lang.Thread.run(Unknown Source)

========================================================================
Root Cause:
Minimum security protocol version for webservice is set to TLS1.2
SSL.minimumProtocolVersion=TLSv1.2
========================================================================
How to verify:
 1. Load wsdl in a browser to see if it's loading fine or not.
 2. use openssl to verify protocol,
openssl s_client -connect host:port -tls1_2
Result: Secure Renegotiation IS supported
SSL-Session:
Protocol  : TLSv1.2
========================================================================
Resolution:
Edit SoapUI-5.2.1.vmoptions file located at C:\SmartBear\SoapUI-5.2.1\bin\SoapUI-5.2.1.vmoptions

Add the following parameter, save and exit.
-Dsoapui.https.protocols=TLSv1.2

Kindly note it will enable only TLSv1.2 protocol

Close and reopen SoapUI for changes to take effect.

Retry to load wsdl in SoapUI. This time it should work fine.

Exception in HttpOutboundMessageContext.RetrieveHttpResponseWork.run: java.io.IOException: SocketMuxer detected socket closure

========================================================================
Applies To:
Linux Version:  OEL-6.5
WLS - 10.3.6
OSB - 11.1.1.7 and later
========================================================================
Problem Summary:
While calling buiness service which has 3DNS endpoint uri of a different domain which is hosted on the same box giving java.io.IOException: SocketMuxer detected socket closure while waiting for a response.

Error:
[WliSbTransports:381304]Exception in HttpOutboundMessageContext.RetrieveHttpResponseWork.run: java.io.IOException: SocketMuxer detected socket closure while waiting for a response
java.io.IOException: SocketMuxer detected socket closure while waiting for a response
 at weblogic.net.http.SocketClosedNotification.&lt;clinit>(SocketClosedNotification.java:10)
 at weblogic.net.http.AsyncResponseHandler$MuxableSocketHTTPAsyncResponse.handleError(AsyncResponseHandler.java:396)
 at weblogic.net.http.AsyncResponseHandler$MuxableSocketHTTPAsyncResponse.hasException(AsyncResponseHandler.java:504)
 at weblogic.socket.SocketMuxer.deliverExceptionAndCleanup(SocketMuxer.java:827)
 at weblogic.socket.SocketMuxer.deliverHasException(SocketMuxer.java:767)
 at weblogic.socket.EPollSocketMuxer.dataReceived(EPollSocketMuxer.java:222)
 at weblogic.socket.EPollSocketMuxer.processSockets(EPollSocketMuxer.java:177)
 at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
 at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
 at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
 at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

========================================================================
Root Cause:
Service is failing when using 3DNS where as working fine if we use individual managedServerHost:port url becuase of which suspect goes to 3DNS settings.
During investion found that 3DNS doesn't have SNAT settings enabled for members becuase of which local call which is travelling through network ( from domain1 to domain2 on same host) is failing.

========================================================================
How to verify:
1. Login to MS host and use wget with 3DNS url. You will recevie below error,
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.

2. Use OSB test console to test BS/PS

3. If we use SOAPUI to test service with 3DNS url should work fine.
========================================================================
Resolution:
Permanent Solution:
Configure SNAT for 3DNS members (LTM setting).
Work around:
Change business service endpoint URL from load balancer URL to individual server managedServerHost:port URL.