weblogic.nodemanager.common.ConfigException: Identity key store file not found

========================================================================
Applies To:
Weblogic: 12.2.1.3
Java: jdk-8u161

========================================================================
Problem Summary:
After a fresh install on 12.2.1.3, we are unable to start the nodemanger.

Error:
<Jan 9, 2019 1:20:02 PM IST> <INFO> <Upgrade> <Setting NodeManager properties version to 12.2.1.3.0>
<Jan 9, 2019 1:20:02 PM IST> <INFO> <Upgrade> <Saving upgraded NodeManager properties to '/u01/Oracle/Middleware/Oracle_Home/user_projects/domains/base_domain/nodemanager/nodemanager.properties'>
<Jan 9, 2019 1:20:02 PM IST> <INFO> <Loading domains file: /u01/Oracle/Middleware/Oracle_Home/user_projects/domains/base_domain/nodemanager/nodemanager.domains>
<Jan 9, 2019 1:20:02 PM IST> <INFO> <Loading identity key store: FileName=/u01/Oracle/Middleware/Oracle_Home/user_projects/domains/base_domain/security/DemoIdentity.jks, Type=jks, PassPhraseUsed=true>
<Jan 9, 2019 1:20:02 PM IST> <SEVERE> <Fatal error in NodeManager server>
weblogic.nodemanager.common.ConfigException: Identity key store file not found: /u01/Oracle/Middleware/Oracle_Home/user_projects/domains/base_domain/security/DemoIdentity.jks
        at weblogic.nodemanager.server.SSLConfig.loadKeyStoreConfig(SSLConfig.java:225)
        at weblogic.nodemanager.server.SSLConfig.access$000(SSLConfig.java:33)
        at weblogic.nodemanager.server.SSLConfig$1.run(SSLConfig.java:118)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.nodemanager.server.SSLConfig.<init>(SSLConfig.java:115)
        at weblogic.nodemanager.server.NMServer.<init>(NMServer.java:169)
        at weblogic.nodemanager.server.NMServer.getInstance(NMServer.java:134)
        at weblogic.nodemanager.server.NMServer.main(NMServer.java:589)
        at weblogic.NodeManager.main(NodeManager.java:31)

========================================================================
Root Cause:
DemoIdentity.jks is missing from $DOMAIN_HOME/security directory.

========================================================================
Resolution:
Generate DemoIdentity.jks
1. Open a command prompt and set WLS env variables.
. $WL_HOME/server/bin/setWLSEnv.sh
2. Run below command to generate identities
java utils.CertGen -keyfilepass DemoIdentityPassPhrase -certfile democert -keyfile demokey -strength 1024 -noskid
3. Run below to import generated files to keystore
java utils.ImportPrivateKey -keystore DemoIdentity.jks -storepass DemoIdentityKeyStorePassPhrase -keyfile demokey.pem -keyfilepass DemoIdentityPassPhrase -certfile democert.pem -alias demoidentity
4. Verify keystore using below command
keytool -list -v -keystore DemoIdentity.jks
5. Copy DemoIdentity.jks to $DOMAIN_HOME/security directory if it's generated in some other directory
6. Rerun ./startNodeManager.sh from $DOMAIN_HOME/bin directory.

VMware Player and Device/Credential Guard are not compatible. VMware Player can be run after disabling Device/Credential Guard.

VMware Player and Device/Credential Guard are not compatible. VMware Player can be run after disabling Device/Credential Guard.
========================================================================
Error:
VMware Player and Device/Credential Guard are not compatible. VMware Player can be run after disabling Device/Credential Guard. Please visit http://www.vmware.com/go/turnoff_CG_DG for more details.
========================================================================
Solution:
Open run and type gpedit.msc
Navigate to Local computer Policy => Computer Configuration => Administrative Templates => System => Device Guard => Double click on "Turn on virtualization"
select disable and click ok.

Now go to Control Panel => Uninstall a program => Click on Turn Windows features on or off => Deselect Hyper-V
Click on do not restart

Open command prompt as admin and delete the below EFI variables from BCD file.

C:\WINDOWS\system32>bcdedit /create {0cb3b571-2f2e-4343-a879-d86a476d7215} /d "DebugTool" /application osloader
The entry {0cb3b571-2f2e-4343-a879-d86a476d7215} was successfully created.

C:\WINDOWS\system32>bcdedit /set {0cb3b571-2f2e-4343-a879-d86a476d7215} path "\EFI\Microsoft\Boot\SecConfig.efi"
The operation completed successfully.

C:\WINDOWS\system32>bcdedit /set {bootmgr} bootsequence {0cb3b571-2f2e-4343-a879-d86a476d7215}
The operation completed successfully.

C:\WINDOWS\system32>bcdedit /set {0cb3b571-2f2e-4343-a879-d86a476d7215} loadoptions DISABLE-LSA-ISO,DISABLE-VBS
The operation completed successfully.

C:\WINDOWS\system32>bcdedit /set hypervisorlaunchtype off
The operation completed successfully.

C:\WINDOWS\system32>

Restart the system

Reopen VMware Player and start VM.

Weblogic forced Garbage collection for servers

========================================================================
Applies to:
Weblogic: 10.3.6 and later
OS: CentOS7
========================================================================
Procedure:
There are two ways to perform garbage collection.

Weblogic admin console:
1. Login to weblogic admin console using admin(usually weblogic) user.
2. From left pane of the Console, go to Environment => Servers.
3. On summary of servers page, select server for which you would like to perform Garbage collection.
4. Go to Monitoring => Performance
5. Click on Garbage Collect button.

Using WLST:
1. Invoke wlst.sh from $oracle_home/oracle_common/common/bin
e.g. /u02/oracle12213/middleware/oracle_home/oracle_common/common/bin
[oracle@purupoc bin]$ ./wlst.sh

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

wls:/offline>
2. Connect to admin server using below command,
connect('<<weblogic-username>>','<<weblogic-password>>','t3://<<adminserver-host>>:<<adminserver-port>>')

e.g. connect('weblogic','weblogic123','t3://localhost:7001')
wls:/offline> connect('weblogic','weblogic123','t3://localhost:7001')
Connecting to t3://localhost:7001 with userid weblogic ...
Successfully connected to Admin Server "AdminServer" that belongs to domain "wl_server".

Warning: An insecure protocol was used to connect to the server.
To ensure on-the-wire security, the SSL port or Admin port should be used instead.

wls:/wl_server/serverConfig/>

3. Enter into domainRuntime mode using domainRuntime()
wls:/wl_server/serverConfig/> domainRuntime()
Location changed to domainRuntime tree. This is a read-only tree
with DomainMBean as the root MBean.
For more help, use help('domainRuntime')

wls:/wl_server/domainRuntime/>

4. Go to server directory.
cd('/ServerRuntimes/<<Servername>>/JVMRuntime/<<Servername>>')
Use ls() command to check existing stats.

wls:/wl_server/domainRuntime/> cd('/ServerRuntimes/AdminServer/JVMRuntime/AdminServer')
wls:/wl_server/domainRuntime/ServerRuntimes/AdminServer/JVMRuntime/AdminServer> ls()

-r--   HeapFreeCurrent                              273970360
-r--   HeapFreePercent                              61
-r--   HeapSizeCurrent                              471633920
-r--   HeapSizeMax                                  518979584
-r--   JavaVMVendor                                 Oracle Corporation
-r--   JavaVendor                                   Oracle Corporation
-r--   JavaVersion                                  1.8.0_161
-r--   Name                                         AdminServer
-r--   OSName                                       Linux
-r--   OSVersion                                    3.10.0-693.el7.x86_64
-r--   Type                                         JVMRuntime
-r--   Uptime                                       1480421


wls:/wl_server/domainRuntime/ServerRuntimes/AdminServer/JVMRuntime/AdminServer>

5. Use cmo.runGC() to force garbage collection.
wls:/wl_server/domainRuntime/ServerRuntimes/AdminServer/JVMRuntime/AdminServer> cmo.runGC()
wls:/wl_server/domainRuntime/ServerRuntimes/AdminServer/JVMRuntime/AdminServer> ls()

-r--   HeapFreeCurrent                              311455952
-r--   HeapFreePercent                              69
-r--   HeapSizeCurrent                              471633920
-r--   HeapSizeMax                                  518979584
-r--   JavaVMVendor                                 Oracle Corporation
-r--   JavaVendor                                   Oracle Corporation
-r--   JavaVersion                                  1.8.0_161
-r--   Name                                         AdminServer
-r--   OSName                                       Linux
-r--   OSVersion                                    3.10.0-693.el7.x86_64
-r--   Type                                         JVMRuntime
-r--   Uptime                                       1691110


wls:/wl_server/domainRuntime/ServerRuntimes/AdminServer/JVMRuntime/AdminServer>

6. Disconnect from AdminServer using disconnect() command
wls:/wl_server/domainRuntime/ServerRuntimes/AdminServer/JVMRuntime/AdminServer> disconnect()
Disconnected from weblogic server: AdminServer
wls:/offline> 

java.lang.OutOfMemoryError: Java heap space error while Applying Oracle Weblogic Server Patch using BSU

========================================================================
Environment Details:
Weblogic Server : 10.3.6
Linux Version: CentOS7

========================================================================
Problem Summary:
We are unable to apply oracle wblogic 10.3.6 patch becuase of java.lang.OutOfMemoryError: Java heap space

Error:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.AbstractCollection.toArray(AbstractCollection.java:136)
at java.util.ArrayList.addAll(ArrayList.java:559)
at org.apache.xmlbeans.impl.schema.SchemaTypeImpl.getProperties(SchemaTypeImpl.java:705)
at com.bea.cie.common.dao.xbean.XBeanDataHandler.loadPropertyMap(XBeanDataHandler.java:775)
at com.bea.cie.common.dao.xbean.XBeanDataHandler.<init>(XBeanDataHandler.java:99)
at com.bea.cie.common.dao.xbean.XBeanDataHandler.createDataHandler(XBeanDataHandler.java:559)
at com.bea.cie.common.dao.xbean.XBeanDataHandler.getComplexValue(XBeanDataHandler.java:455)
at com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getPatchDependencies(PatchCatalogHelper.java:442)
at com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getPatchDependencies(PatchCatalogHelper.java:464)
at com.bea.plateng.patch.dao.cat.PatchCatalog.getPatchDependencies(PatchCatalog.java:56)
at com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getInvalidatedPatchMap(PatchCatalogHelper.java:1621)
at com.bea.plateng.patch.PatchSystem.updatePatchCatalog(PatchSystem.java:436)
at com.bea.plateng.patch.PatchSystem.refresh(PatchSystem.java:130)
at com.bea.plateng.patch.PatchSystem.<init>(PatchSystem.java:114)
at com.bea.plateng.patch.PatchSystem.<clinit>(PatchSystem.java:41)
at com.bea.plateng.patch.Patch.main(Patch.java:279)
Exception in thread "Thread-0" java.lang.NoClassDefFoundError: Could not initialize class com.bea.plateng.patch.PatchSystem
at com.bea.plateng.patch.PatchClientHelper.getAllPatchDetails(PatchClientHelper.java:74)
at com.bea.plateng.patch.PatchInstallationHelper.cleanupPatchSets(PatchInstallationHelper.java:130)
at com.bea.plateng.patch.PatchTarget.<init>(PatchTarget.java:272)
at com.bea.plateng.patch.PatchTargetFactory.create(PatchTargetFactory.java:30)
at com.bea.plateng.patch.ProductAliasTarget.constructPatchTargetList(ProductAliasTarget.java:88)
at com.bea.plateng.patch.ProductAliasTarget.<init>(ProductAliasTarget.java:46)
at com.bea.plateng.patch.ProductAliasTargetHelper.getProdAliasTargetList(ProductAliasTargetHelper.java:55)
at com.bea.plateng.patch.ProductAliasTargetHelper.getAllHomeToProdAliasesTargetMap(ProductAliasTargetHelper.java:32)
at com.bea.plateng.patch.ProductAliasTargetHelper.checkProfilesInProductAliases(ProductAliasTargetHelper.java:133)
at com.bea.plateng.patch.Patch$1.run(Patch.java:376)
at java.lang.Thread.run(Thread.java:745)


========================================================================
Root Cause:
Heap size configured for BSU is not sufficient

========================================================================
Resolution:
1. Go to $MIDDLEWARE_HOME/utils/bsu
2. Edit bsu.sh
3. Change the memory arguments from
MEM_ARGS="-Xms256m -Xmx512m"
To
MEM_ARGS="-Xms2048m -Xmx2048m"
4. Save the changes.
5. Try to apply patch again.

How to find Oracle WebLogic Server Version?

Today one of my customer asked me the way to find out weblogic server version which they have installed so thought to share details with you as well.

There are different ways to determine Oracle Weblogic Service Version. Use one of the following method to find weblogic server version.

1. We can determine weblogic server version using registry.xml file:
Location in WLS 12.2.1.3:
$ORACLE_HOME/inventory/registry.xml

<distribution status="installed" name="WebLogic Server" version="12.2.1.3.0">

e.g.
/u02/oracle12213/oracle_home/middleware/inventory/registry.xml

<?xml version = '1.0' encoding = 'UTF-8' standalone = 'yes'?>
<registry home="/u02/oracle12213/oracle_home/middleware" platform="226" sessions="5" xmlns:ns2="http://xmlns.oracle.com/cie/gdr/dei" xmlns:ns3="http://xmlns.oracle.com/
cie/gdr/nfo" xmlns="http://xmlns.oracle.com/cie/gdr/rgy">
   <distributions>
      <distribution status="installed" name="WebLogic Server" version="12.2.1.3.0">
         <sessions>
            <session id="1" date="2017-10-21T17:53:51.433+05:30" action="install"/>
         </sessions>
         <features>
            <feature status="installed" name="cieCfg_wls_shared_external" version="12.2.1.3.0">
               <sessions>

2. We can user weblogic.jar file (weblogic.version class) to find weblogic server version:
Go to $WLS_HOME/server/lib
e.g. For WLS 12.2.1.3
/u02/oracle12213/oracle_home/middleware/wlserver/server/lib

Use below command to find weblogic and it's component version,
java -cp weblogic.jar weblogic.version -verbose

e.g.
[oracle@demomachine lib]$ java -cp weblogic.jar weblogic.version -verbose

WebLogic Server 12.2.1.3.0 Thu Aug 17 13:39:49 PDT 2017 1882952 ImplVersion: 12.2.1.3.0
Oracle Security Developer Tools Security Engine ImplVersion: 3.1.0
Oracle Security Developer Tools Crypto ImplVersion: 3.1.0
Oracle Universal Connection Pool ImplVersion: 12.2.0.1.0
WebLogic Config External  Thu Aug 10 00:41:12 UTC 2017 ImplVersion: 8.6.0.0
WebLogic Application Descriptors ImplVersion: 12.2.1.3
WebLogic Application Descriptors Bindings ImplVersion: 12.2.1.3
WebLogic Application Container ImplVersion: 12.2.1.3
WebLogic Utility Classloader implementations ImplVersion: 12.2.1.3
WebLogic Descriptors for J2EE ImplVersion: 12.2.1.3
WebLogic Beangen ImplVersion: 12.2.1.3
WebLogic BeanInfo Caching and Discovery ImplVersion: 12.2.1.3
WebLogic Descriptor ImplVersion: 12.2.1.3
WebLogic Beangen ImplVersion: 12.2.1.3
WebLogic Coherence Descriptor ImplVersion: 12.2.1.3
WebLogic Coherence Api ImplVersion: 12.2.1.3
WebLogic Coherence App Descriptor ImplVersion: 12.2.1.3
WebLogic Coherence Integration ImplVersion: 12.2.1.3
WebLogic Workarea ImplVersion: 12.2.1.3
WebLogic Nodemanager Plugin ImplVersion: 12.2.1.3
WebLogic RMI ImplVersion: 12.2.1.3
WebLogic Timers ImplVersion: 12.2.1.3
WebLogic Channels API ImplVersion: 12.2.1.3
WebLogic Channels ImplVersion: 12.2.1.3
WebLogic Channels API ImplVersion: 12.2.1.3
WebLogic Socket Muxer API ImplVersion: 12.2.1.3
WebLogic java compiler utils package ImplVersion: 12.2.1.3
WebLogic Utils for working with Expressions ImplVersion: 12.2.1.3
WebLogic Utils ImplVersion: 12.2.1.3
WebLogic Utils ImplVersion: 12.2.1.3
WebLogic Utils for Dynamically Generated Class Wrappers ImplVersion: 12.2.1.3
WebLogic Work Manager ImplVersion: 12.2.1.3
WebLogic Datasource ImplVersion: 12.2.1.3
WebLogic RAC Module UCP ImplVersion: 12.2.1.3
WebLogic Resource Pool ImplVersion: 12.2.1.3
WebLogic Diagnostics Base ImplVersion: 12.2.1.3
WebLogic Diagnostics Core Interfaces ImplVersion: 12.2.1.3
WebLogic Diagnostics JRockit Flight Recorder Interfaces ImplVersion: 12.2.1.3
WebLogic i18n Build Support ImplVersion: 12.2.1.3
WebLogic i18n Runtime Support ImplVersion: 12.2.1.3
WebLogic I18N tools ImplVersion: 12.2.1.3
WebLogic Diagnostics Instrumentor Config Tool ImplVersion: 12.2.1.3
WebLogic Diagnostics Instrumentor Tool ImplVersion: 12.2.1.3
WebLogic Diagnostics Logging ImplVersion: 12.2.1.3
WebLogic Diagnostics Query Module ImplVersion: 12.2.1.3
WebLogic EJBGen ImplVersion: 12.2.1.3
WebLogic Messaging Kernel ImplVersion: 12.2.1.3
WebLogic JMS Pool ImplVersion: 12.2.1.3
WebLogic Security Provider Generation Tool ImplVersion: 12.2.1.3
WebLogic Domain Binding ImplVersion: 12.2.1.3
WebLogic Management Core Interfaces ImplVersion: 12.2.1.3
WebLogic Management EventBus Interfaces ImplVersion: 12.2.1.3
WebLogic Management JMX Interfaces ImplVersion: 12.2.1.3
Oracle Cooperative Memory Management Lower Tier ImplVersion: 12.2.1.3
WebLogic EclipseLink Integration ImplVersion: 12.2.1.3
WebLogic SCA Client ImplVersion: 12.2.1.3
WebLogic security network connection filter classes ImplVersion: 12.2.1.3
WebLogic security ssl classes ImplVersion: 12.2.1.3
WebLogic Spring Instrument Tool ImplVersion: 12.2.1.3
WebLogic Store ImplVersion: 12.2.1.3
WebLogic Store Admin Tool ImplVersion: 12.2.1.3
WebLogic STORE GXA ImplVersion: 12.2.1.3
WebLogic JDBC Store ImplVersion: 12.2.1.3
WebLogic JTA implementation ImplVersion: 12.2.1.3
WebLogic WebApp Container Public API ImplVersion: 12.2.1.3
WebLogic Servlet ImplVersion: 12.2.1.3
WebLogic Http Pub/Sub Module ImplVersion: 12.2.1.3
WebLogic SAAJ ImplVersion: 12.2.1.3
WebLogic STAX ImplVersion: 12.2.1.3
WebLogic Jersey Common Integration ImplVersion: 12.2.1.3
WebLogic Jersey Client Integration ImplVersion: 12.2.1.3
WebLogic JAX-RS 2.0 Portable Server / Jersey 2.x integration module ImplVersion: 12.2.1.3
WebLogic XML XPath Implementation ImplVersion: 12.2.1.3
OPatch Patches:
No patches installed

SERVICE NAME                    VERSION INFORMATION
============                    ===================
Web Services Execution Engine   1.0
Managed Beans Container         1.0
Post Admin Singleton Services   1.0
Pre Admin Singleton Services S  1.0
Singleton Services Batch Manag  1.0
Kernel                          Commonj WorkManager v1.1
CorbaService                    CORBA 2.3, IIOP 1.2, RMI-IIOP SFV2, OTS 1.2, CSIv2 Level 0 + Stateful
TimerService                    Commonj TimerManager v1.1
JDBCService                     JSR-221, JDBC 4.0
Enterpise Java Beans Container  EJB 3.2
CustomResourceServerService     1.0.0.0
Transaction Service             JTA 1.1
Transaction Stop Service        JTA 1.1
Servlet Container               Servlet 3.1, JSP 2.3
XMLService                      XML 1.1

[oracle@demomachine lib]$

3. We can find weblogic server version from check the AdminServer.log file:
Go to $DOMAIN_HOME/servers/AdminServer/logs
Check AdminServer.log file for weblogic server version

e.g.
####<Oct 24, 2017 3:39:16,238 PM IST> <Info> <WebLogicServer> <demomachine> <> <Thread-10> <> <> <> <1508839756238> <[severity-value: 64] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000214> <WebLogic Server "AdminServer" version: WebLogic Server 12.2.1.3.0 Thu Aug 17 13:39:49 PDT 2017 1882952 Copyright (c) 1995,2017, Oracle and/or its affiliates. All rights reserved.>

4. We can use WLST to check weblogic server version:
Go to $ORACLE_HOME/oracle_common/bin
Invoke wlst.sh
Use command print version

e.g.
wls:/offline> print version
WebLogic Server 12.2.1.3.0
wls:/offline>

5. We can use admin console to check weblogic server version:
Login to WebLogic Admin Console and navigate to Servers => AdminServer => Monitoring tab
This page will have weblogic version details.
OR
When you open Admin console, version will be displayed at the bottom of the welcome page.

e.g.
WebLogic Server Version: 12.2.1.3.0
Copyright (c) 1996,2017, Oracle and/or its affiliates. All rights reserved.

OHS instance failed to start because of (98)Address already in use: AH00072: make_sock: could not bind to address XXX.XXX.XXX.XX:7777

========================================================================
Environment Details:
OHS: 12.2.1.3.0

========================================================================
Problem Summary:
When we are trying to start OHS instance it's failing with an error (98)Address already in use: AH00072: make_sock: could not bind to address XXX.XXX.XXX.XX:7777

Have verified port 7777 if it's already used using lsof and netstat but nothing was listening on that port. It was free.
@machine1 config]# /usr/sbin/lsof -i:7777
@machine1 config]# netstat | grep 7777
@machine1 config]# netstat

Error:
Starting server ohs1 ...
Nov 05, 2017 12:56:48 PM oracle.ohs.plugin.nodemanager.OhsProcessManagementPlugin$ProcessImpl start
INFO: Starting server ohs1
Nov 05, 2017 12:56:48 PM oracle.ohs.plugin.nodemanager.OhsRunCommand execute
INFO: Running /u02/oracle/middleware/oracle_home/wlserver/../ohs/bin/launch httpd -DOHS_MPM_EVENT -d /u02/oracle/middleware/oracle_home/user_projects/domains/ohs_domain/config/fmwconfig/components/OHS/instances/ohs1 -k start -f /u02/oracle/middleware/oracle_home/user_projects/domains/ohs_domain/config/fmwconfig/components/OHS/instances/ohs1/httpd.conf
Nov 05, 2017 12:56:49 PM oracle.ohs.plugin.nodemanager.OhsRunCommand runGeneric
INFO: (98)Address already in use: AH00072: make_sock: could not bind to address XXX.XXX.XXX.XX:7777
Nov 05, 2017 12:56:49 PM oracle.ohs.plugin.nodemanager.OhsRunCommand runGeneric
INFO: no listening sockets available, shutting down
Nov 05, 2017 12:56:49 PM oracle.ohs.plugin.nodemanager.OhsRunCommand runGeneric
INFO: AH00015: Unable to open logs
Nov 05, 2017 12:56:49 PM oracle.ohs.plugin.nodemanager.OhsRunCommand execute
SEVERE: /u02/oracle/middleware/oracle_home/wlserver/../ohs/bin/launch httpd -DOHS_MPM_EVENT -d /u02/oracle/middleware/oracle_home/user_projects/domains/ohs_domain/config/fmwconfig/components/OHS/instances/ohs1 -k start -f /u02/oracle/middleware/oracle_home/user_projects/domains/ohs_domain/config/fmwconfig/components/OHS/instances/ohs1/httpd.conf: exit status = 1
Nov 05, 2017 12:56:49 PM oracle.ohs.plugin.nodemanager.OhsProcessHandler buildProcess
INFO: Check the instance log file for more information: /u02/oracle/middleware/oracle_home/user_projects/domains/ohs_domain/servers/ohs1/logs/ohs1.log
Nov 05, 2017 12:56:49 PM oracle.ohs.plugin.nodemanager.OhsProcessManagementPlugin$ProcessImpl buildIoException
SEVERE: Failed to start the server ohs1
<Nov 5, 2017 12:56:49 PM IST> <WARNING> <Server start command for OHS server 'ohs1' failed due to: [Failed to start the server ohs1
Check log file /u02/oracle/middleware/oracle_home/user_projects/domains/ohs_domain/system_components/OHS/ohs_nm.log
Check log file /u02/oracle/middleware/oracle_home/user_projects/domains/ohs_domain/servers/ohs1/logs/ohs1.log]. Please check Node Manager log and/or server 'ohs1' log for detailed information.>

This Exception occurred at Sun Nov 05 12:52:39 IST 2017.
weblogic.nodemanager.NMException: Received error message from Node Manager Server: [Server start command for OHS server 'ohs1' failed due to: [Failed to start the server ohs1


========================================================================
Root Cause:
There were 2 things which were creating above problem,
1. System wasn't connected to network. Network was OFF.
2. /etc/hosts was having wrong ip configured.

========================================================================
Resolution:
1. Have connected system to network. Network ON.
2. Have edited /etc/hosts and configured the correct IP.
3. Restarted OHS and it came up fine.
Starting server ohs1 ...
Nov 05, 2017 1:11:07 PM oracle.ohs.plugin.nodemanager.OhsProcessManagementPlugin$ProcessImpl start
INFO: Starting server ohs1
Nov 05, 2017 1:11:07 PM oracle.ohs.plugin.nodemanager.OhsRunCommand execute
INFO: Running /u02/oracle/middleware/oracle_home/wlserver/../ohs/bin/launch httpd -DOHS_MPM_EVENT -d /u02/oracle/middleware/oracle_home/user_projects/domains/ohs_domain/config/fmwconfig/components/OHS/instances/ohs1 -k start -f /u02/oracle/middleware/oracle_home/user_projects/domains/ohs_domain/config/fmwconfig/components/OHS/instances/ohs1/httpd.conf
Nov 05, 2017 1:11:07 PM oracle.ohs.plugin.nodemanager.OhsRunCommand runGeneric
INFO: Errors will be logged into /u02/oracle/middleware/oracle_home/user_projects/domains/ohs_domain/servers/ohs1/logs/ohs1.log
Nov 05, 2017 1:11:09 PM oracle.ohs.plugin.nodemanager.OhsRunCommand execute
INFO: /u02/oracle/middleware/oracle_home/wlserver/../ohs/bin/launch httpd -DOHS_MPM_EVENT -d /u02/oracle/middleware/oracle_home/user_projects/domains/ohs_domain/config/fmwconfig/components/OHS/instances/ohs1 -k start -f /u02/oracle/middleware/oracle_home/user_projects/domains/ohs_domain/config/fmwconfig/components/OHS/instances/ohs1/httpd.conf: exit status = 0
Nov 05, 2017 1:11:09 PM oracle.ohs.plugin.nodemanager.OhsProcessHandler buildProcess
INFO: Check the instance log file for more information: /u02/oracle/middleware/oracle_home/user_projects/domains/ohs_domain/servers/ohs1/logs/ohs1.log
<Nov 5, 2017 1:11:12 PM IST> <INFO> <ohs_domain> <ohs1> <The server 'ohs1' is running now.>
Successfully started server ohs1 ...

OEM 13c installer failed to start becuase of Failed prerequisite Checking monitor: must be configured to display at least 256 colors. Actual unknown. Failed

========================================================================
Environment Details:
OEM : 13c (13.2.0.0)
OS : Linux6

========================================================================
Problem Summary:
Am trying to invoke OEM installer for OEM installation but it's getting failed becuase of prerequisite failure.

Error:
-bash-4.1$ /u01/OEM13c_Soft/em13200_linux64.bin
0%.........................................................................100%
Launcher log file is /tmp/OraInstall2017-10-27_02-42-25AM/launcher2017-10-27_02-42-25AM.log.
Starting Oracle Universal Installer

Checking if CPU speed is above 300 MHz.   Actual 2295.051 MHz    Passed
Checking monitor: must be configured to display at least 256 colors.  Actual unknown.   Failed <<<<
Checking swap space: must be greater than 512 MB.   Actual 34287 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (64-bit not required)

Some system prerequisite checks failed.
You must fulfill these requirements before continuing.

Continue? (yes [y] / no [n]) [n]
n
You have confirmed that the product cannot be installed on this platform.
Quitting the installation.
The log is located here: /tmp/OraInstall2017-10-27_02-42-25AM/launcher2017-10-27_02-42-25AM.log.

========================================================================
Root Cause:
Was using root/superuser user to trigger installer instead of oracle which holds ownership of OEM installer.

========================================================================
Resolution:
1. Close the session which you have logged in with root user.
2. Login to the host as oracle user.
Kindly note if you login to host using root and then switch to oracle might not work and you may end up in same error so you have to start a new session and directly login as oracle user.
4. Go to OEM13c installer directory and invoke installer.

How to upgrade java/jdk used by Weblogic server 12.2.1.3 to different version

==============================================================================
Environment Details:
Weblogic : 12.2.1.3
OS : Linux
==============================================================================
In this case we are going to install new java/jdk to different location and change JAVA_HOME variable in all scripts using it.
So let's get started.

Have installed jdk1.8.0._151 in the directory /u02/oracle12213/custom/java/ and provided permissions 775 to it.

[oracle@demomachine ~]$ cd /u02/downloads/
[oracle@demomachine downloads]$ ll
total 1186004
-rwxrwxr-x. 1 oracle oracle 839208313 Aug 21 12:39 fmw_12.2.1.3.0_wls.jar
-rwxrwxr-x. 1 oracle oracle 185516505 Jul 25 21:06 jdk-8u141-linux-x64.tar.gz
-rw-rw-r--. 1 oracle oracle 189736377 Oct 24 01:35 jdk-8u151-linux-x64.tar.gz
[oracle@demomachine downloads]$ chmod -R 775 *
[oracle@demomachine downloads]$ ll
total 1186004
-rwxrwxr-x. 1 oracle oracle 839208313 Aug 21 12:39 fmw_12.2.1.3.0_wls.jar
-rwxrwxr-x. 1 oracle oracle 185516505 Jul 25 21:06 jdk-8u141-linux-x64.tar.gz
-rwxrwxr-x. 1 oracle oracle 189736377 Oct 24 01:35 jdk-8u151-linux-x64.tar.gz
[oracle@demomachine downloads]$ tar -xzf jdk-8u151-linux-x64.tar.gz -C /u02/oracle12213/custom/java/
[oracle@demomachine downloads]$ cd /u02/oracle12213/custom/java/
[oracle@demomachine java]$ ll
total 0
drwxrwxr-x. 8 oracle oracle 255 Jul 12 17:11 jdk1.8.0_141
drwxr-xr-x. 8 oracle oracle 255 Sep  6 08:02 jdk1.8.0_151
[oracle@demomachine java]$ chmod -R 775 jdk1.8.0_151/
[oracle@demomachine java]$

Oracle home and domain home is under /u02 so going to find out the files which are having jdk1.8.0_141 configred in it using below linux command,
grep -iworI 'jdk1.8.0_141' . | uniq | egrep -v "/logs/|/adr/|\.log|\.out|/.patch_storage|/inventory" | tr ':' ' ' |awk '{print $1}'

[oracle@demomachine u02]$ grep -iworI 'jdk1.8.0_141' . | uniq | egrep -v "/logs/|/adr/|\.log|\.out|/.patch_storage|/inventory" | tr ':' ' ' |awk '{print $1}'
./oracle12213/oracle_home/middleware/oui/bin/launch.sh
./oracle12213/oracle_home/middleware/oui/bin/viewInventory.sh
./oracle12213/oracle_home/middleware/oui/bin/compareInventory.sh
./oracle12213/oracle_home/middleware/oui/bin/setProperty.sh
./oracle12213/oracle_home/middleware/oui/.globalEnv.properties
./oracle12213/oracle_home/middleware/user_projects/domains/wls_domain/bin/setDomainEnv.sh
./oracle12213/oracle_home/middleware/user_projects/domains/wls_domain/bin/setNMJavaHome.sh
./oracle12213/oracle_home/middleware/user_projects/domains/wls_domain/init-info/domain-info.xml
./oracle12213/oracle_home/middleware/user_projects/domains/wls_domain/init-info/startscript.xml
./oracle12213/oracle_home/middleware/user_projects/domains/wls_domain/init-info/tokenValue.properties
./oracle12213/oracle_home/middleware/user_projects/domains/wls_domain/init-info/nodemanager-properties.xml
./oracle12213/oracle_home/middleware/user_projects/domains/wls_domain/nodemanager/nodemanager.properties


There are few files which need not to be changed,
./oracle12213/oracle_home/middleware/user_projects/domains/wls_domain/init-info/domain-info.xml
./oracle12213/oracle_home/middleware/user_projects/domains/wls_domain/init-info/startscript.xml
./oracle12213/oracle_home/middleware/user_projects/domains/wls_domain/init-info/tokenValue.properties
./oracle12213/oracle_home/middleware/user_projects/domains/wls_domain/init-info/nodemanager-properties.xml

Files which need to changed are,
./oracle12213/oracle_home/middleware/oui/bin/launch.sh
./oracle12213/oracle_home/middleware/oui/bin/viewInventory.sh
./oracle12213/oracle_home/middleware/oui/bin/compareInventory.sh
./oracle12213/oracle_home/middleware/oui/bin/setProperty.sh
./oracle12213/oracle_home/middleware/oui/.globalEnv.properties
./oracle12213/oracle_home/middleware/user_projects/domains/wls_domain/bin/setDomainEnv.sh
./oracle12213/oracle_home/middleware/user_projects/domains/wls_domain/bin/setNMJavaHome.sh
./oracle12213/oracle_home/middleware/user_projects/domains/wls_domain/nodemanager/nodemanager.properties

Shutdown all servers and nodemanager.
Take backup of domain and oracle home direcotries.

Edit all the below files and change JAVA_HOME location from /u02/oracle12213/custom/java/jdk1.8.0_141 to /u02/oracle12213/custom/java/jdk1.8.0_151
-----------------------------------------------------------
$oracle_home/oui/bin/launch.sh

# pre-install env: JAVA_HOME_LOCATION not stringsubbed
# post-install env: JAVA_HOME_LOCATION is stringsubbed
#JAVA_HOME_LOCATION="/u02/oracle12213/custom/java/jdk1.8.0_141"
JAVA_HOME_LOCATION="/u02/oracle12213/custom/java/jdk1.8.0_151"
------------------------------------------------------------
$oracle_home/oui/bin/viewInventory.sh

# OUI platform name used at install time
PLATFORM_NAME="Linux_AMD64"

#JAVA_HOME_LOCATION="/u02/oracle12213/custom/java/jdk1.8.0_141"
JAVA_HOME_LOCATION="/u02/oracle12213/custom/java/jdk1.8.0_151"
-----------------------------------------------------------------
$oracle_home/oui/bin/compareInventory.sh

# pre-install env: JAVA_HOME_LOCATION not stringsubbed
# post-install env: JAVA_HOME_LOCATION is stringsubbed
#JAVA_HOME_LOCATION="/u02/oracle12213/custom/java/jdk1.8.0_141"
JAVA_HOME_LOCATION="/u02/oracle12213/custom/java/jdk1.8.0_151"
----------------------------------------------------------------------
$oracle_home/oui/bin/setProperty.sh

#JAVA_HOME_LOCATION="/u02/oracle12213/custom/java/jdk1.8.0_141"
JAVA_HOME_LOCATION="/u02/oracle12213/custom/java/jdk1.8.0_151"
----------------------------------------------------------------------
$oracle_home/oui/.globalEnv.properties

#Sat Oct 21 17:55:26 IST 2017
#JAVA_HOME=/u02/oracle12213/custom/java/jdk1.8.0_141
#JAVA_HOME_1_8=/u02/oracle12213/custom/java/jdk1.8.0_141
JAVA_HOME=/u02/oracle12213/custom/java/jdk1.8.0_151
JAVA_HOME_1_8=/u02/oracle12213/custom/java/jdk1.8.0_151
---------------------------------------------------------------------
$domain_home/bin/setDomainEnv.sh

#SUN_JAVA_HOME="/u02/oracle12213/custom/java/jdk1.8.0_141"
SUN_JAVA_HOME="/u02/oracle12213/custom/java/jdk1.8.0_151"
export SUN_JAVA_HOME

#DEFAULT_SUN_JAVA_HOME="/u02/oracle12213/custom/java/jdk1.8.0_141"
DEFAULT_SUN_JAVA_HOME="/u02/oracle12213/custom/java/jdk1.8.0_151"
export DEFAULT_SUN_JAVA_HOME

else
                JAVA_VENDOR="Oracle"
                export JAVA_VENDOR
                #JAVA_HOME="/u02/oracle12213/custom/java/jdk1.8.0_141"
                JAVA_HOME="/u02/oracle12213/custom/java/jdk1.8.0_151"
                export JAVA_HOME

----------------------------------------------------------------------
$domain_home/bin/setNMJavaHome.sh

#SUN_JAVA_HOME="/u02/oracle12213/custom/java/jdk1.8.0_141"
SUN_JAVA_HOME="/u02/oracle12213/custom/java/jdk1.8.0_151"
export SUN_JAVA_HOME

#DEFAULT_SUN_JAVA_HOME="/u02/oracle12213/custom/java/jdk1.8.0_141"
DEFAULT_SUN_JAVA_HOME="/u02/oracle12213/custom/java/jdk1.8.0_151"
export DEFAULT_SUN_JAVA_HOME

 else
                JAVA_VENDOR="Oracle"
                export JAVA_VENDOR
                #JAVA_HOME="/u02/oracle12213/custom/java/jdk1.8.0_141"
                JAVA_HOME="/u02/oracle12213/custom/java/jdk1.8.0_151"
                VM_TYPE="HotSpot"

----------------------------------------------------------------------------
$domain_home/nodemanager/nodemanager.properties

#JavaHome=/u02/oracle12213/custom/java/jdk1.8.0_141
JavaHome=/u02/oracle12213/custom/java/jdk1.8.0_151

----------------------------------------------------------------------------
Restart nodemanager and servers.
Login to admin console. Click on Environment -> servers -> monitoring and verify the java version.

WebLogic Home:
/u02/oracle12213/oracle_home/middleware/wlserver
Middleware Home:
/u02/oracle12213/oracle_home/middleware
Java Vendor:
Oracle Corporation
Java Version:
1.8.0_151
OS Name:
Linux
OS Version:
3.10.0-693.el7.x86_64

WLST to decrypt Weblogic, jdbc and java keystore passwords offline

We can decrypt weblogic, jdbc and java keystore passwords offline using WLST.

Kindly follow below steps to decrypt them offline using WLST ( No Script/Online dependency ):
1.      Login to admin server host
2.      Go to $ORACLE_HOME/oracle_common/common/bin
3.      Invoke wlst.sh OR wlst.cmd
4.      wls:/offline> domain = "<$domain_home_path>"
5.      wls:/offline> service = weblogic.security.internal.SerializedSystemIni.getEncryptionService(domain)
6.      wls:/offline> encryption = weblogic.security.internal.encryption.ClearOrEncryptedService(service)
7.      wls:/offline> print encryption.decrypt("<encrypted password>")
Please note, in encrypted string we have to remove backward slash symbol.

Examples:
For Linux:
[oracle@demomachine ~]$ cd /u02/oracle12213/osb/oracle_home/oracle_common/common/bin
[oracle@demomachine u02]$./wlst.sh

wls:/offline> domain = "/u02/Oracle12213/osb/domain/dev_domain "
wls:/offline> service = weblogic.security.internal.SerializedSystemIni.getEncryptionService(domain)
wls:/offline> encryption = weblogic.security.internal.encryption.ClearOrEncryptedService(service)
wls:/offline> print encryption.decrypt("{AES}577uW8U1WXvE9Ep+9gw7V3ZfJiyjD5TDT3id5X9LtdE= ")
weblogic


For Windows:
C:\windows\system32>cd C:\Oracle12213\Middleware\Oracle_Home\oracle_common\common\bin
C:\Oracle12213\Middleware\Oracle_Home\oracle_common\common\bin>wlst.cmd

Initializing WebLogic Scripting Tool (WLST) ...

Jython scans all the jar files it can find at first startup. Depending on the system, this process may take a few minutes to complete, and WLST may not return a prompt right away.

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

wls:/offline> domain = "C:\\Oracle12213\\Middleware\\Oracle_Home\\user_projects\\domains\\dev_domain"
wls:/offline> service = weblogic.security.internal.SerializedSystemIni.getEncryptionService(domain)
wls:/offline> encryption = weblogic.security.internal.encryption.ClearOrEncryptedService(service)
wls:/offline> print encryption.decrypt("{AES}Rx9/lKeFMhllwNMYXWtC+DryFL9zU3MypEPLNtJ+W4w=")
weblogic123

=========================================================================================
Issues Faced:
You may face below issue while decrypting passwords if you don’t remove backward slash symbol.

Error:
wls:/offline> print encryption.decrypt("{AES}Rx9/lKeFMhllwNMYXWtC+DryFL9zU3MypEPLNtJ+W4w\=")
Traceback (innermost last):
  File "<console>", line 1, in ?
        at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptBytes(JSafeEncryptionServiceImpl.java:144)
        at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptString(JSafeEncryptionServiceImpl.java:192)
        at weblogic.security.internal.encryption.ClearOrEncryptedService.decrypt(ClearOrEncryptedService.java:99)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)

weblogic.security.internal.encryption.EncryptionServiceException: weblogic.security.internal.encryption.EncryptionServiceException: com.rsa.jsafe.JSAFE_InputException: Invalid input.

Resolution:
If you are facing above error, kindly delete backward slash symbols from the encrypted string and retry.
e.g. From {AES}Rx9/lKeFMhllwNMYXWtC+DryFL9zU3MypEPLNtJ+W4w\=
To {AES}Rx9/lKeFMhllwNMYXWtC+DryFL9zU3MypEPLNtJ+W4w=

Managed server failed to start becuase of Failed to invoke startup class "JRF Startup Class", java.lang.ClassNotFoundException: oracle.jrf.wls.JRFStartup

========================================================================
Environment Details:
Weblogic : 10.3.6
OSB : 11.1.1.7.0
OS : Windows 10

========================================================================
Problem Summary:
Am using weblogic admin console to start managed server but unable to start managed server because of Failed to invoke startup class "JRF Startup Class", java.lang.ClassNotFoundException: oracle.jrf.wls.JRFStartup

Error:
<Oct 16, 2017 8:10:24 PM IST> <Notice> <Log Management> <BEA-170019> <The server log file C:\OraclePS6\Middleware\user_projects\domains\osb_domain\servers\osb_server1\logs\osb_server1.log is opened. All server side log events will be written to this file.>
<Oct 16, 2017 8:10:28 PM IST> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Oct 16, 2017 8:10:30 PM IST> <Critical> <WebLogicServer> <BEA-000286> <Failed to invoke startup class "JRF Startup Class", java.lang.ClassNotFoundException: oracle.jrf.wls.JRFStartup
java.lang.ClassNotFoundException: oracle.jrf.wls.JRFStartup
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
Truncated. see log file for complete stacktrace
>

========================================================================
Root Cause:
StartScriptEnabled is set to false in nodemanager.properties.

========================================================================
Resolution:
1. Open C:\OraclePS6\Middleware\wlserver_10.3\common\nodemanager\nodemanager.properties on the managed server machine.
2. Set StartScriptEnabled=true.
3. Restart the node manager.
4. Stop the managed server.
5. Start the managed server from Admin console.

RCU fails while creating soainfra schema with ORA-01450: maximum key length (6398) exceeded

========================================================================
Applies to:
Weblogic : 12.2.1.2
OSB: 12.2.1.2

========================================================================
Problem Summary:
Creation of SOA INFRA schema using RCU is failing  becuase of ORA-01450: maximum key length (6398) exceeded

Error:
ORA-01450: maximum key length (6398) exceeded
File:/u01/poc/osb_home/fmw_12_2_1_2/soa/common/sql/soainfra/sql/oracle/createschema_soainfra_oracle_LARGE.sql
Statement:ALTER TABLE B2B_CONTROL_NUMBER
    ADD CONSTRAINT B2B_CONTROL_NUMBER_PK PRIMARY KEY (SENDER_NAME, RECEIVER_NAME, DOC_PROTOCOL_NAME, DIRECTION, NAME, IS_DEFAULT)

========================================================================
Root Cause:
We are facing above problem because of the NLS_LENGTH_SEMANTICS parameter which has been set to CHAR in database. It's not a supported setting for schema creation using RCU. Oracle OSB only supports schemas in a byte-mode(BYTE) database.

========================================================================
How to verify:
Connect to database and run below SQL to verify setting,
show parameters nls_length_semantics

========================================================================
Resolution:
nls_length_semantics initialization parameter on the database where the schemas going to reside must be set to BYTE.

To set the nls_length_semantics parameter to BYTE, login to the database as SYSDBA and issue the following command,
alter system set NLS_LENGTH_SEMANTICS=BYTE scope=both;

Once we change nls_length_semantics please restart the database for changes to take effect.

Now rerun RCU to create schemas.

javax.xml.ws.WebServiceException:javax.xml.soap.SOAPException: Message send failed: Received fatal alert: handshake_failure

========================================================================
Applies to:
BPEL: 12.1.3
Weblogic : 12.1.3/12.2.1.2
OSB: 12.2.1.2

========================================================================
Problem Summary:
BPEL (12.1.3) commposite is unable to call/invoke OSB(12.2.1.2) weservice becuase of handshake_failure.

Unable to invoke endpoint URI "https://puruhost:7004/osbHttpsTest" successfully due to: javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: Received fatal alert: handshake_failure

Error:
ExecuteThread: '2' for queue: 'weblogic.socket.Muxer', fatal error: 80: problem unwrapping net record
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
ExecuteThread: '2' for queue: 'weblogic.socket.Muxer', SEND TLSv1 ALERT:  fatal, description = internal_error
ExecuteThread: '2' for queue: 'weblogic.socket.Muxer', WRITE: TLSv1 Alert, length = 2
ExecuteThread: '2' for queue: 'weblogic.socket.Muxer', called closeOutbound()
ExecuteThread: '2' for queue: 'weblogic.socket.Muxer', closeOutboundInternal()
[Raw write]: length = 7

[2017-03-21T07:42:09.922-05:00] [bpel_poc_ms01] [ERROR] [] [oracle.integration.platform.blocks.soap] [tid: [ACTIVE].ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tunin
g)'] [userId: weblogic] [ecid: dab9a6ff-56f4-4909-a3da-888dd1c70014-000a3325,1:28719] [APP: soa-infra] [oracle.soa.tracking.FlowId: 590083] [oracle.soa.tracking.InstanceId: 680805] [oracle.
soa.tracking.SCAEntityId: 360005] [composite_name: bpelToOSBTest!1.0] [FlowId: 0000LfkssnqBT8ELN64Eye1OmvaI00001R] Unable to dispatch request to https://puruhost:7004/osbHttpsTest due to exception[[
javax.xml.ws.WebServiceException: javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: Received fatal alert: handshake_failure
        at oracle.j2ee.ws.client.jaxws.DispatchImpl.invoke(DispatchImpl.java:1381)
        at oracle.j2ee.ws.client.jaxws.OracleDispatchImpl.synchronousInvocationWithRetry(OracleDispatchImpl.java:237)
        at oracle.j2ee.ws.client.jaxws.OracleDispatchImpl.invoke(OracleDispatchImpl.java:108)
        at oracle.integration.platform.blocks.soap.AbstractWebServiceBindingComponent.dispatchRequest(AbstractWebServiceBindingComponent.java:629)
        at oracle.integration.platform.blocks.soap.WebServiceExternalBindingComponent.processOutboundMessage(WebServiceExternalBindingComponent.java:367)
        at oracle.integration.platform.blocks.soap.WebServiceExternalBindingComponent.sendSOAPMessage(WebServiceExternalBindingComponent.java:1508)
        at oracle.integration.platform.blocks.soap.WebServiceExternalBindingComponent.request(WebServiceExternalBindingComponent.java:957)
        at oracle.integration.platform.blocks.mesh.SynchronousMessageHandler.doRequest(SynchronousMessageHandler.java:151)
.
.
.
Caused by: javax.xml.soap.SOAPException: javax.xml.soap.SOAPException: Message send failed: Received fatal alert: handshake_failure
        at oracle.j2ee.ws.saaj.client.p2p.HttpSOAPConnection.call2(HttpSOAPConnection.java:234)
        at oracle.j2ee.ws.common.transport.HttpTransport.transmit(HttpTransport.java:75)
        at oracle.j2ee.ws.common.async.MessageSender.call(MessageSender.java:58)
        at oracle.j2ee.ws.common.async.Transmitter.transmitSync(Transmitter.java:105)
        at oracle.j2ee.ws.common.async.Transmitter.transmit(Transmitter.java:60)
        at oracle.j2ee.ws.common.async.RequestorImpl.transmit(RequestorImpl.java:258)
        at oracle.j2ee.ws.common.async.RequestorImpl.invoke(RequestorImpl.java:93)
        at oracle.j2ee.ws.client.jaxws.DispatchImpl.invoke(DispatchImpl.java:1315)
        ... 92 more
Caused by: javax.xml.soap.SOAPException: Message send failed: Received fatal alert: handshake_failure
        at oracle.j2ee.ws.saaj.client.p2p.HttpSOAPConnection.post2(HttpSOAPConnection.java:523)
        at oracle.j2ee.ws.saaj.client.p2p.HttpSOAPConnection$PrivilegedPost.run(HttpSOAPConnection.java:1346)
        at java.security.AccessController.doPrivileged(Native Method)
        at oracle.j2ee.ws.saaj.client.p2p.HttpSOAPConnection.call2(HttpSOAPConnection.java:232)
        ... 99 more
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
        at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1959)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1077)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:702)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
========================================================================
Root Cause:
BPEL(12.1.3) is invoking OSB(12.2.1.2) service over https and client is throwing handshake_failure becuase of mismatch in TLS protocol version.
BPEL is sending request using TLSv1 and osb server is expecting TLSv1.2

========================================================================
Resolution:
Configure -Dhttps.protocols=TLSv1.2 as a JAVA_OPTIONS in startup scripts or as a server start argument.

java.sql.SQLException: ORA-14300: partitioning key maps to a partition outside maximum permitted number of partitions

========================================================================
Applies to:
BPEL: 12.2.1.2
Weblogic : 12.2.1.2

========================================================================
Problem Summary:
BPEL composite is throwing below error on mediator fault processing instead of actual error.

Audit Level is set to PRODUCTION. We have seen same behavior with Audit Level = DEVELOPMENT.

Error:
####<Sep 6, 2016 1:43:38 AM IST> <Error> <EJB> <puruNode01> <bpel_poc_ms01> <[ACTIVE] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <03e82632-48e7-42c3-ba89-c0e44ee16691-0003f912> <1473144218489> <BEA-010026> <Exception occurred during commit of transaction Name=[EJB com.collaxa.cube.engine.ejb.impl.bpel.BPELEngineBean.syncCreateAndInvoke(oracle.soa.management.ComponentDN,java.lang.String,java.lang.String,oracle.fabric.common.NormalizedMessage)],Xid=BEA1-1BBC599C191E44C1322C(1400043235),Status=Rolled back. [Reason=javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
 Internal Exception: java.sql.SQLException: ORA-14300: partitioning key maps to a partition outside maximum permitted number of partitions

 Error Code: 14300
 Call: INSERT INTO AUDIT_DETAILS (CIKEY, DETAIL_ID, BIN, BIN_CSIZE, BIN_USIZE, DOC_REF, FLOW_ID, CI_PARTITION_DATE, SCA_PARTITION_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
 bind => [9 parameters bound]
 Query: InsertObjectQuery(AuditDetails -----
 flowId: 400225
 componentId: 431223
 detailId: 170201
 docRef: null
 binUsize: 0
 binCsize: 0
 partitionDate: null)],numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=1,seconds left=59,useSecure=false,XAServerResourceInfo[eis/MQ/TestOutbound]
 .
 .
 .
 weblogic.transaction.RollbackException: Unexpected exception in beforeCompletion: sync=org.eclipse.persistence.transaction.JTASynchronizationListener@4c6a7b73
 Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
 Internal Exception: java.sql.SQLException: ORA-14300: partitioning key maps to a partition outside maximum permitted number of partitions

 Error Code: 14300
 Call: INSERT INTO AUDIT_DETAILS (CIKEY, DETAIL_ID, BIN, BIN_CSIZE, BIN_USIZE, DOC_REF, FLOW_ID, CI_PARTITION_DATE, SCA_PARTITION_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
 bind => [9 parameters bound]
 Query: InsertObjectQuery(AuditDetails -----
 flowId: 400225
 componentId: 431223
 detailId: 170201
 docRef: null
 binUsize: 0
 binCsize: 0
 partitionDate: null)
 at weblogic.transaction.internal.TransactionImpl.throwRollbackException(TransactionImpl.java:1985)
 at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:369)
 at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:260)
 at weblogic.ejb.container.internal.BaseLocalObject.postInvoke1(BaseLocalObject.java:335)
 at weblogic.ejb.container.internal.BaseLocalObject.__WL_postInvokeTxRetry(BaseLocalObject.java:204)
 at weblogic.ejb.container.internal.SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:46)
 at com.collaxa.cube.engine.ejb.impl.bpel.BPELEngineBean_51369e_ICubeEngineLocalBeanImpl.syncCreateAndInvoke(Unknown Source)
 at com.collaxa.cube.engine.delivery.DeliveryHandler.callCreateAndInvoke(DeliveryHandler.java:990)

========================================================================
Root Cause:
Mediator fault isn't propagating expected tracking properties when mediator processing fails

========================================================================
Resolution:
Solution1: Temporary fix of this error is to set Audit Level = Off, in this case mediator fault will work as expected.

How to change audit level?
Login to EM console and go to SOA-Infra -> SOA Administration -> Common Properties
Audit Level select “Off” from drop down and click apply

Solution2: For permanent fix oracle advised us to apply below patch.            
Patch: 24794863 12.2.1.2.0 CI_PARTITION_DATE NULL. Error Code: 20000/ORA-14300 on Mediator fault processing.

Unable to access jenkins URL becuase of java.lang.NoSuchMethodError: org.jvnet.tiger_types.Types.bind

========================================================================
Applies to:
Weblogic: 12.2.1.2
JNEKINS : 2.32.3 (LTS release)
========================================================================
Problem Summary:
Have deployed jenkins (2.32.3 LTS release) application war file on weblogic 12.2.1.2 server for build and deploy automation.
Applicaton status is active and jenkins is fully up and running.

-------- Log snippet ----------------
Mar 13, 2017 4:10:01 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
-------------------------------------

Though jenkins is fully up and running , we are unable to access jenkins url http://localhost:7001/jenkins

Error:
--------------------- Browser Error -----------------------------
Error 500--Internal Server Error
From RFC 2068 Hypertext Transfer Protocol -- HTTP/1.1:
10.5.1 500 Internal Server Error

The server encountered an unexpected condition which prevented it from fulfilling the request.
--------------------------------------------------------

--------------------- Weblogic Server logs -----------------
<Mar 13, 2017, 4:10:52,250 PM IST> <Error> <HTTP> <BEA-101020> <[ServletContext@1185538388[app:jenkins module:jenkins.war path:null spec-version:3.1]] Servlet failed with an Exception
java.lang.NoSuchMethodError: org.jvnet.tiger_types.Types.bind(Ljava/lang/reflect/Type;Ljava/lang/reflect/GenericDeclaration;Ljava/lang/reflect/ParameterizedType;)Ljava/lang/reflect/Type;
at org.kohsuke.stapler.ClassDescriptor.findMethods(ClassDescriptor.java:155)
at org.kohsuke.stapler.ClassDescriptor.findMethods(ClassDescriptor.java:142)
at org.kohsuke.stapler.ClassDescriptor.<init>(ClassDescriptor.java:84)
at org.kohsuke.stapler.lang.KlassNavigator$1.getFunctions(KlassNavigator.java:216)
at org.kohsuke.stapler.lang.KlassNavigator$1.getFunctions(KlassNavigator.java:141)
Truncated. see log file for complete stacktrace
>
<Mar 13, 2017, 4:10:52,254 PM IST> <Error> <HTTP> <BEA-101107> <[ServletContext@1185538388[app:jenkins module:jenkins.war path:null spec-version:3.1]] Problem occurred while serving the error page.
javax.servlet.ServletException: java.lang.NoSuchMethodError: org.jvnet.tiger_types.Types.bind(Ljava/lang/reflect/Type;Ljava/lang/reflect/GenericDeclaration;Ljava/lang/reflect/ParameterizedType;)Ljava/lang/reflect/Type;
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:391)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:247)
at weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:625)
at weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:282)
at weblogic.servlet.internal.ForwardAction.run(ForwardAction.java:22)
Truncated. see log file for complete stacktrace
Caused By: java.lang.NoSuchMethodError: org.jvnet.tiger_types.Types.bind(Ljava/lang/reflect/Type;Ljava/lang/reflect/GenericDeclaration;Ljava/lang/reflect/ParameterizedType;)Ljava/lang/reflect/Type;
at org.kohsuke.stapler.ClassDescriptor.findMethods(ClassDescriptor.java:155)
at org.kohsuke.stapler.ClassDescriptor.findMethods(ClassDescriptor.java:142)
at org.kohsuke.stapler.ClassDescriptor.<init>(ClassDescriptor.java:84)
at org.kohsuke.stapler.lang.KlassNavigator$1.getFunctions(KlassNavigator.java:216)
at org.kohsuke.stapler.lang.KlassNavigator$1.getFunctions(KlassNavigator.java:141)
Truncated. see log file for complete stacktrace
>
------------------------------------------------------------
========================================================================
Root Cause:
Weblogic won't load the JARs in the jenkins WAR before the Weblogic installed JARs which is resulting an error javax.servlet.ServletException: java.lang.NoSuchMethodError: org.jvnet.tiger_types.Types.bind

tiger-types-2.2.jar exists in jenkins.war under WEB-INF/lib

========================================================================
Resolution:
1. Open weblogic deployment descriptor (weblogic.xml) file located under WEB-INF of jenkins.war
2. Add <wls:package-name>org.jvnet.*</wls:package-name> under prefer application packages in weblogic.xml which ensures package 'org.jvnet.*' contained in the Jenkins war is used ahead of the system class loader version. It contains a method required by Jenkins.
------------ weblogic.xml ------------

<?xml version="1.0" encoding="UTF-8"?>
<wls:weblogic-web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:wls="http://www.bea.com/ns/weblogic/90" xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd http://www.bea.com/ns/weblogic/90 http://www.bea.com/ns/weblogic/90/weblogic-web-app.xsd">
     <wls:weblogic-version>12.2.1.2</wls:weblogic-version>
     <wls:context-root>jenkins</wls:context-root>
     <wls:container-descriptor>
          <wls:prefer-web-inf-classes>false</wls:prefer-web-inf-classes>
      <wls:prefer-application-packages>
              <wls:package-name>com.google.common.*</wls:package-name>
 <wls:package-name>org.jvnet.*</wls:package-name>
         </wls:prefer-application-packages>
     </wls:container-descriptor>
</wls:weblogic-web-app>
----------------------------------
3. Save. That's it.
4. Now redeploy jenkins.war with a new change. Make sure jenkins application state is active( fully up and running)
5. Try to access jenkins URL.

Deployment is failing with an error : Composite with same revision ID already exists

========================================================================
Applies to:
Weblogic: 12.2.1.1
BPEL : 12.2.1.1
========================================================================
Problem Summary:
Deployment is failing with an error : Composite with same revision ID already exists

Error:
[12:47:10 PM] ----  Deployment started.  ----
[12:47:10 PM] Target platform is  (Weblogic 12.x).
[12:47:10 PM] Running dependency analysis...
[12:47:10 PM] Building...
[12:47:29 PM] Deploying profile...
[12:47:29 PM] Updating revision id for the SOA Project 'TestMultipleEndpointsPrj.jpr' to '1.0'..
[12:47:29 PM] Wrote Archive Module to C:\JDeveloper\mywork\SOAPractice\TestMultipleEndpointsPrj\deploy\sca_TestMultipleEndpointsPrj.jar
[12:47:33 PM] Deploying sca_TestMultipleEndpointsPrj.jar to partition "default" on server demo_bpel_ms01 [http://localhost:4811]
[12:47:33 PM] Processing sar=/C:/JDeveloper/mywork/SOAPractice/TestMultipleEndpointsPrj/deploy/sca_TestMultipleEndpointsPrj.jar
[12:47:33 PM] Adding sar file - C:\JDeveloper\mywork\SOAPractice\TestMultipleEndpointsPrj\deploy\sca_TestMultipleEndpointsPrj.jar
[12:47:33 PM] Preparing to send HTTP request for deployment
[12:47:33 PM] Creating HTTP connection to host:ausilwap002.us.dell.com, port:6811
[12:47:33 PM] Sending internal deployment descriptor
[12:47:33 PM] Sending archive - sca_TestMultipleEndpointsPrj.jar
[12:47:38 PM] Received HTTP response from the server, response code=500
[12:47:38 PM] Error deploying archive sca_TestMultipleEndpointsPrj.jar to partition "default" on server demo_bpel_ms01 [http://localhost:4811]
[12:47:38 PM] HTTP error code returned [500]
[12:47:38 PM] Error message from server:
There was an error deploying the composite on demo_bpel_ms01: Composite with same revision ID already exists: default/TestMultipleEndpointsPrj!1.0. Please set the overwrite flag or use different revision ID. Abort deployment..

[12:47:38 PM] Check server log for more details.
[12:47:38 PM] Error deploying archive sca_TestMultipleEndpointsPrj.jar to partition "default" on server demo_bpel_ms01 [http://localhost:4811]
[12:47:38 PM] Deployment cancelled.
[12:47:38 PM] ----  Deployment incomplete  ----.
[12:47:38 PM] Error deploying archive file:/C:/JDeveloper/mywork/SOAPractice/TestMultipleEndpointsPrj/deploy/sca_TestMultipleEndpointsPrj.jar
 (oracle.tip.tools.ide.fabric.deploy.common.SOARemoteDeployer)

========================================================================
Root Cause:
Composite with same revision ID already exists: default/TestMultipleEndpointsPrj!1.0

========================================================================
Resolution:
1. If you want to redeploy the same version of a SOA composite application on the server then you have to check the overwrite flag. It will over write the existing process with same version.
2. Kindly note that all the running threads with the existing process will be aborted and process will mark as status stale.

Weblogic server startup stuck at BEA-002014 IIOP subsystem enabled.

========================================================================
Applies to:
Weblogic: 10.3.6 and later
========================================================================
Problem Summary:
Weblogic admin server hangs at <BEA-002014> <IIOP subsystem enabled.> during restart.

Error:
<Dec 1, 2016 1:56:18 PM IST> <Info> <Server> <BEA-002609> <Channel Service initialized.>
<Dec 1, 2016 1:56:18 PM IST> <Info> <Socket> <BEA-000436> <Allocating 3 reader threads.>
<Dec 1, 2016 1:56:18 PM IST> <Info> <Socket> <BEA-000446> <Native IO Enabled.>
<Dec 1, 2016 1:56:18 PM IST> <Info> <IIOP> <BEA-002014> <IIOP subsystem enabled.>

========================================================================
Root Cause:
Issue can be due to corrupted LDAP. It's stuck at security/embedded LDAP start.

========================================================================
Resolution:
1. Login to admin server host
2. Take back up of $DOMAIN_HOME/servers/<SERVER_NAME>/data/ldap directory.
3. Restore ldap from working backup file $DOMAIN_HOME/servers/<SERVER_NAME>/data/ldap/backup
     OR
    We can start welogic admin server and ldap direcotry will get automatically created.
    Kindly note if you restart admin server without restoring ldap, all users access will get revoked except weblogic user.

Data Source test failed with Error ORA-01045: user SOA_DEMO lacks CREATE SESSION privilege

========================================================================
Applies to:
Weblogic: 10.3.6 and later
========================================================================
Problem Summary:
Application requires a XA data source to be configured on weblogic for DB connection using SOA_DEMO user.
Tried to configure it as per requirement but during Test configuration connection test failed with Error ORA-01045: user SOA_DEMO lacks CREATE SESSION privilege

Error:
Connection test failed.
Error ORA-01045: user SOA_DEMO lacks CREATE SESSION privilege; logon denied <br/>oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450)<br/>oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:392)<br/>oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:385)<br/>oracle.jdbc.driver.T4CTTIfun.processError(T4CTTIfun.java:1018)<br/>oracle.jdbc.driver.T4CTTIoauthenticate.processError(T4CTTIoauthenticate.java:501)<br/>oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:522)<br/>oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)<br/>oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:437)<br/>oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:954)<br/>oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:639)<br/>oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:666)<br/>oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)<br/>oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:566)<br/>oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:317)<br/>oracle.jdbc.xa.client.OracleXADataSource.getPooledConnection(OracleXADataSource.java:486)<br/>oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:174)<br/>oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:109)<br/>weblogic.jdbc.common.internal.DataSourceUtil.testConnection0(DataSourceUtil.java:356)<br/>weblogic.jdbc.common.internal.DataSourceUtil.access$000(DataSourceUtil.java:22)<br/>weblogic.jdbc.common.internal.DataSourceUtil$1.run(DataSourceUtil.java:254)<br/>.

========================================================================
Root Cause:
User SOA_DEMO doesn't have connect privileges.

========================================================================
Resolution:
1. Login to DB as SYS user.
2. Grant connet and resource to SOA_DEMO using below command.
    grant connect, resource to user_name identified by password;
    e.g. grant connect, resource to SOA_DEMO identified by Welcome1;
3. Once done, go back to weblogic console and test data source.

How to shutdown or restart IBM DataPower Appliance

========================================================================
Applies To:
IBM Datapower : XI52
IBM Datapower Gateways : 7.2.0
========================================================================
Purpose:
Shutdown or restart DataPower appliance.

Before we go to shutdown or restart procedure let's learn shutdown parameters which are needed during shutdown or restart of DataPower Appliance.

Shutdown DataPower parameters:
1. Reboot
    It will shutdown and restart the appliance.
2. Reload
    It will restart the appliance.
3. Halt
    It will shutdown the appliance.
4. Seconds
    Specifies the number of seconds delay before the appliance starts the shutdown operation.

Shutdown or restart using GUI/Console:
1. Login to DataPower WebGui
2. Under default domain go to Administration => Main => System Control
3. Scroll Down and go to Shutdown section
4. Select mode(e.g. Reboot System) and Delay (e.g. 10 Secs)
5. Click on Shutdown

Shutdown or restart using CLI:
1. Login to DataPower host
2. Go to default domain
3. Execute command
    shutdown reboot 10
    After 10 secs it will shutdown and reboot datapower appliance.
4. To reload use,
    shutdown reload 30
    To halt use,
    shutdown halt 30

oracle.mds.config.MDSConfigurationException: MDS-01330: unable to load MDS configuration document

========================================================================
Applies to:
BPEL : 12.2.1.1
JDEV: 12.2.1.1
========================================================================
Problem Summary:
Have created empty BPEL project MDSDemoProject which we are going to use as MDS project for shared artifacts.
During compilation of BPEL project facing oracle.mds.config.MDSConfigurationException: MDS-01330: unable to load MDS configuration document

Error:
oracle.fabric.common.FabricException: oracle.mds.config.MDSConfigurationException: MDS-01330: unable to load MDS configuration document  
oracle.xml.parser.v2.XMLParseException; systemId: file:/C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml; lineNumber: 28; columnNumber: 18; '--' is not allowed in comments.
at oracle.fabric.common.FabricMetadataManagerFactory.createMetadataManager(FabricMetadataManagerFactory.java:219)
at oracle.fabric.common.FabricMetadataManagerFactory.createMetadataManager(FabricMetadataManagerFactory.java:185)
at oracle.fabric.composite.model.ModuleConfigModel.initializeMetadataManager(ModuleConfigModel.java:116)
at oracle.fabric.composite.model.CompositeModel.getClientMetadataManager(CompositeModel.java:302)
at oracle.soa.scac.ValidateComposite.loadComposite(ValidateComposite.java:901)
at oracle.soa.scac.ValidateComposite.doValidation(ValidateComposite.java:571)
at oracle.soa.scac.ValidateComposite.run0(ValidateComposite.java:228)
at oracle.soa.scac.ValidateComposite.run(ValidateComposite.java:209)
at oracle.soa.scac.ValidateComposite.main(ValidateComposite.java:201)
Caused by: oracle.fabric.common.FabricException: oracle.mds.config.MDSConfigurationException: MDS-01330: unable to load MDS configuration document  
oracle.xml.parser.v2.XMLParseException; systemId: file:/C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml; lineNumber: 28; columnNumber: 18; '--' is not allowed in comments.: MDS-01330: unable to load MDS configuration document  
oracle.xml.parser.v2.XMLParseException; systemId: file:/C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml; lineNumber: 28; columnNumber: 18; '--' is not allowed in comments.
at oracle.integration.platform.common.MDSMetadataManagerImpl.init(MDSMetadataManagerImpl.java:463)
at oracle.integration.platform.common.MDSMetadataManagerImpl.<init>(MDSMetadataManagerImpl.java:277)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at oracle.fabric.common.FabricMetadataManagerFactory.createMetadataManager(FabricMetadataManagerFactory.java:215)
... 8 more
Caused by: oracle.mds.config.MDSConfigurationException: MDS-01330: unable to load MDS configuration document  
oracle.xml.parser.v2.XMLParseException; systemId: file:/C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml; lineNumber: 28; columnNumber: 18; '--' is not allowed in comments.
at oracle.mds.config.MDSConfig.<init>(MDSConfig.java:1016)
at oracle.mds.config.MDSConfig.<init>(MDSConfig.java:919)
at oracle.integration.platform.common.MDSUtil.getMDSConfig(MDSUtil.java:102)
at oracle.integration.platform.common.MDSUtil.getMDSInstance(MDSUtil.java:199)
at oracle.integration.platform.common.MDSMetadataManagerImpl.getMDSInstance(MDSMetadataManagerImpl.java:1053)
at oracle.integration.platform.common.MDSMetadataManagerImpl.init(MDSMetadataManagerImpl.java:418)
... 14 more
Caused by: oracle.xml.parser.v2.XMLParseException; systemId: file:/C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml; lineNumber: 28; columnNumber: 18; '--' is not allowed in comments.
at oracle.xml.parser.v2.XMLError.flushErrors1(XMLError.java:326)
at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:438)
at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:321)
at oracle.mds.internal.util.xml.OracleXDKXMLUtilImpl.parse(OracleXDKXMLUtilImpl.java:116)
at oracle.mds.internal.util.xml.DOMUtil.parse(DOMUtil.java:576)
at oracle.mds.internal.config.ConfigurationUtils.getConfigElement(ConfigurationUtils.java:145)
at oracle.mds.config.MDSConfig.<init>(MDSConfig.java:969)
... 19 more

Compilation of project 'MDSDemoProject.jpr' finished. Check 'C:\JDeveloper\mywork\MDSDemoApplication\MDSDemoProject\SOA\SCA-INF\classes\scac.log' for details.
[9:30:09 AM] Compilation complete: 1 errors, 0 warnings.
[9:30:09 AM] Build time 2,801 msec.
========================================================================
Root Cause:
Have done a silly mistake. Error clearly says that '--' is not allowed in comments and file which is having '--' in comments is file:/C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml;

oracle.xml.parser.v2.XMLParseException; systemId: file:/C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml; lineNumber: 28; columnNumber: 18; '--' is not allowed in comments.: MDS-01330: unable to load MDS configuration document

adf-config.xml code snippet:

         <!-- UNCOMMENT Only if you would like to use file based MDS
         <metadata-store-usage id="mstore-usage_2">
            <metadata-store class-name="oracle.mds.persistence.stores.file.FileMetadataStore">
            <!-- Not Needed For apps <property name="partition-name" value="seed"/>
              <property name="metadata-path" value="C:\oracle12211\Middleware\Oracle_Home\jdeveloper\integration"/>
            </metadata-store>
          </metadata-store-usage>-->

========================================================================
Resolution:
Remove <!-- from C:/JDeveloper/mywork/MDSDemoApplication/.adf/META-INF/adf-config.xml file and recompile

adf-config.xml code Snippet after change:

         <!-- UNCOMMENT Only if you would like to use file based MDS
         <metadata-store-usage id="mstore-usage_2">
            <metadata-store class-name="oracle.mds.persistence.stores.file.FileMetadataStore">
            Not Needed For apps <property name="partition-name" value="seed"/>
              <property name="metadata-path" value="C:\oracle12211\Middleware\Oracle_Home\jdeveloper\integration"/>
            </metadata-store>
          </metadata-store-usage>-->

Compiling...
Compilation of project 'MDSDemoProject.jpr' finished. Check 'C:\JDeveloper\mywork\MDSDemoApplication\MDSDemoProject\SOA\SCA-INF\classes\scac.log' for details.
[9:30:34 AM] Successful compilation: 0 errors, 0 warnings.
[9:30:34 AM] Build time 5,219 msec.

How to Enroll a Node Manager in weblogic?

========================================================================
Applies to:
Weblogic: 10.3.6 and later
========================================================================
nmEnroll is used to enroll a machine or a domain with the node manager.
Few of us may forget to configure NodeManager after installing Weblogic server. Follow below steps to enroll nodemanager manually in weblogic.
1. Create a Machine:
     • Login to weblogic console
     • Click on Lock & Edit button in change center.
     • Go to Environment -> Machines -> click New
     • Provide name and select machine OS
     • Click Next
     • Select type, provide host and port details for node manager and click Finish
     • Click Activate Changes.

2. Configure node manager username and password:
     • Login to weblogic console and take lock & edit.
     • Under domain Structure click on domain_name on left hand side.
     • Go to security tab and click on Advanced.
     • Configure/update NodeManager Username and password.
     • Click save and click Active Changes.

3. Enroll Node Manager:
     • Make sure admin server is up and running.
     • Login to host
     • Go to $ORACLE_HOME/oracle_common/common/bin and run ./wlst.sh
     • Use Connect command to connect to admin server
        connect('<adminuser>','<password>','t3://<admin_host>:<admin_port>')
     • Run nmEnroll
        nmEnroll('<DOMAIN_HOME>', '<NODEMANAGER_HOME>')
        E.g.
        nmEnroll('/u01/Oracle/Middleware/user_projects/domains/base_domain',                '/u01/Oracle/Middleware/wlserver_10.3/common/nodemanager')
     • Once it is Successfully enrolled to machine with the domain directory check
        $WL_Home/common/nodemanager/nodemanager.domains file. It should have domain entry.
     • Set StartScriptEnabled=true and StopScriptEnabled=true in $WL_Home/common/nodemanager/nodemanager.properties file.
     • Restart the nodemanager and check the status in weblogic console if it is in active status or not.
     • Go to Environment -> Machines -> Click on Machine name
     • Go to Monitoring tab and verify Node Manager Status