Monday 24 August 2015

RCU-6080:Global prerequisite check failed - Check requirement for specified database failed while running SOA 12c RCU - Connecting to a multitenant container database (CDB) is not supported

========================================================================
Applies To:
BPEL - version 12.1.3
WLS - 12.1.3
Oracle DB - 12.1.0.2.0
========================================================================
Problem:
While creating my local domain for POC work , faced the following error while running SOA 12c RCU for creating schemas,

Error:
RCU creation failed due to following error.

RCU-6002:The specified database does not meet the minimum requirement to load metadata repository.
RCU-6080:Global prerequisite check failed - Check requirement for specified database
The selected Oracle database is a multitenant container database (CDB). Connecting to a multitenant container database (CDB) is not supported. Instead, connect to a valid pluggble database (PDB).
========================================================================
Cause: 
The selected Oracle database is a multitenant container database (CDB)
========================================================================
Resolution:
1. Determine if your database has been created as CDB or not using below SQL
SELECT CDB FROM V$DATABASE;
2. Check if any PDBs are available in the Database.
SELECT CON_ID, NAME, OPEN_MODE FROM V$PDBS;
    If there is no PDB other than PDB$SEED, please work with your DBA to create a PDB for the RCU schema
3. If there is a PDB then get full service names using below sql,
select name from v$services;
4. Make sure PDB is open (Mounted doesn't mean it's open). To open PDB run below sql,
alter pluggable database all open;
5. Verify if PDB open by running,
SELECT CON_ID, NAME, OPEN_MODE FROM V$PDBS;
6. Once done you can verify service using "lsnrctl status"
7. Now DB is ready for schema creation using RCU. Connect and create schema.

Oracle document for reference: http://docs.oracle.com/database/121/ADMIN/cdb_plug.htm#CEGCEEAJ

Monday 22 June 2015

SOA Infrastructure Startup Failure in SOA 12c

========================================================================
Applies To:
BPEL - version 12.1.3
WLS - 12.1.3
========================================================================
Problem:
SOA Infra failed to start

Error Snippet:
Caused By: java.security.UnrecoverableKeyException: Password verification
failed  at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:769)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:38)
        at java.security.KeyStore.load(KeyStore.java:1185)
        at oracle.j2ee.ws.saaj.util.SSLUtil.loadKeyStore(SSLUtil.java:73)
        at oracle.j2ee.ws.saaj.util.SSLUtil.getKeyManagerFactory(SSLUtil.java:88)
        at oracle.j2ee.ws.saaj.util.SSLUtil.getKeyManagers(SSLUtil.java:97)
        at oracle.j2ee.ws.saaj.util.SSLUtil.createSSLSocketFactory(SSLUtil.java:50)
        at oracle.integration.platform.common.SSLSocketFactoryManagerImpl.getSSLSocketFac
        tory(SSLSocketFactoryManagerImpl.java:58)
        at oracle.fabric.common.wsdl.WSDLManager.init(WSDLManager.java:356)

======================================================================
Cause:
As per oracle documentation of SOA 12c- Administering Oracle SOA Suite and Oracle Business Process Management Suite, this is known issue and causes because of cwallet.sso has the SOA map
======================================================================
Resolution:
Perform one of the following actions
Delete the SOA map in cwallet.sso.
OR
Remove $DOMAIN_HOME/config/fmwconfig/default-keystore.jks
Restart the SOA Infrastructure

Thursday 18 June 2015

oracle.wsm.policymanager.PolicyManagerException: WSM-02141 : Unable to connect to the policy access service in SOA 12c(12.1.3)

========================================================================
Applies To:
BPEL - version 12.1.3
WLS - 12.1.3
========================================================================
Problem:
Today critical issue has been reported by user for SOA 12c (12.1.3) that policy manager has stopped working and while accessing WSM policy manager following error is seen,
oracle.wsm.policymanager.PolicyManagerException: WSM-02141 : Unable to connect to the policy access service

Services which uses security policies throwing below error
>>>InvalidSecurity : error in processing the WS-Security security header

Error Snippet:
oracle.wsm.policymanager.PolicyManagerException: WSM-02141 : Unable to connect to the policy access service.
at oracle.wsm.policymanager.BeanFactory.getInitialContext(BeanFactory.java:810)
at oracle.wsm.policymanager.BeanFactory.getJndiObj(BeanFactory.java:844)
at oracle.wsm.policymanager.BeanFactory.lookupJndiObj(BeanFactory.java:1175)
at oracle.wsm.policymanager.BeanFactory.getUsageTrackerBeanEJB(BeanFactory.java:1019)
at oracle.wsm.policymanager.BeanFactory.getBeanEJB(BeanFactory.java:544)

======================================================================
Cause:
As per oracle documentation of SOA 12c Fusion Middleware Securing Web Services and Managing Policies with Oracle Web Services Manager, by default WSM Policy Manager is auto-wired in the domain agents and its default targeted to the managed servers and not to the AdminServer. This means that AdminServer uses an agent to connect to the policy manager and we need to manually bind the OWSM Agent to the Policy Manager URL.
In my case listen address of MS has been changed recently and owsm-pm-connection-http and t3 shows the status as Out of Sync
======================================================================
Resolution:
1. Login to EM console
2. From the WebLogic Domain menu -> Cross Component Wiring -> Components
3. In the Components Table, select OWSM Agent
4. In the Client Configurations table, verify that the Client ID owsm-pm-connection-http reflects the correct Policy Manager URL in the Connection column and if the Status column displays Out of Sync, you need to bind the Agent to the Policy Manager.
To do so,
Select owsm-pm-connection-http in the Client Configurations table and click Bind.
In the Bind Client Configuration page, verify that the Service End Point contains the correct Policy Manager URL and click Yes.
Confirmation is displayed on the Components page and the status of the Agent is changed to Wired.
Repeat same for t3 entry as well.
5. Also verify OWSM Policy Manager if status is Published. If status is out of sync then select policy manager and click Publish.

Monday 20 April 2015

com.bea.wli.sb.context.BindingLayerException: General runtime error: Unknown error while processing message for ProxyService

========================================================================
Applies To:
OSB - version 11.1.1.7.0
WLS - 10.3.6
========================================================================
Problem:
Client was reporting that he is getting com.bea.wli.sb.context.BindingLayerException: General runtime error: Unknown error while processing message for ProxyService intermittently from few managed servers in a clustered environment.

Errors in log,

Caused By: com.bea.wli.sb.management.BrokerManagementException: java.lang.RuntimeException: com.bea.wli.sb.management.BrokerManagementException: com.bea.wli.sb.stages.StageException: The configured XQuery resource cannot be found.
                at com.bea.wli.sb.pipeline.RouterManager.getRouterRuntime(RouterManager.java:646)
                at com.bea.wli.sb.pipeline.RouterContext.getInstance(RouterContext.java:180)
                at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:583)   at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:398) at com.bea.wli.sb.transports.local.LocalTransportProvider.sendMessageAsync(LocalTransportProvider.java:322)     at sun.reflect.GeneratedMethodAccessor476.invoke(Unknown Source)   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
{{{{{{{{{{{{{{{ Truncating }}}}}}}}}}}}}}}}}}}}}}
at oracle.tip.adapter.fw.jca.messageinflow.MessageEndpointImpl.onMessage(MessageEndpointImpl.java:556)   at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent$AQDeliveryBean.sendInboundRecord(AbstractDequeueAgent.java:571)        at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.__AW_run(AbstractDequeueAgent.java:123)
                at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.run(AbstractDequeueAgent.java)
                at java.lang.Thread.run(Thread.java:662)
Caused By: weblogic.xml.query.exceptions.XQueryTypeException: line 15, column 10: {err}XP0004: Invalid static type: (anyAttribute|PI)
                at weblogic.xml.query.compiler.TypeMatchExpression.typeCheckNoCache(TypeMatchExpression.java:144)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.FirstOrderExpression.typeCheckNoCache(FirstOrderExpression.java:269)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.IfThenElseExpression.typeCheckNoCache(IfThenElseExpression.java:112)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.GeneralFLWGDORExpression.typeCheckNoCache(GeneralFLWGDORExpression.java:404)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.NodeConstructor.typeCheckNoCache(NodeConstructor.java:278)
========================================================================
Cause:
On investigation found that the root cause for com.bea.wli.sb.context.BindingLayerException: General runtime error: Unknown error while processing message for ProxyService is the configured XQuery resource which wasn't found at the run time. This was strange behavior because this is one of the server in cluster and has same copy like other where it's working fine. If we restart MS, it will resolve the issue temporarily but again will show up during peak load.
========================================================================
Resolution:
After some research found one oracle note Doc ID 1679044.1 which says it's a bug and there is patch 18427728 available to resolve the issue.
After applying patch, we haven't seen error anymore during run time.

Saturday 18 April 2015

Server failed during startup and moving to FAILED_NOT_RESTARTABLE

========================================================================
Applies To:
OSB - version 11.1.1.7.0
WLS - 10.3.6
========================================================================
Problem:
Today one of my client reported that he is not able to restart the managed sever and it's moving to FAILED_NOT_RESTARTABLE.

***************************************************
*  To start WebLogic Server, use a username and   *
*  password assigned to an admin-level user.  For *
*  server administration, use the WebLogic Server *
*  console at http://hostname:port/console        *
***************************************************
starting weblogic with Java version:
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Oracle JRockit(R) (build R28.2.7-7-155314-1.6.0_45-20130329-0641-linux-x86_64, compiled mode)
/u01/domains/poc/osb/bin/startWebLogic.sh: line 179: syntax error near unexpected token `fi'
/u01/domains/poc/osb/bin/startWebLogic.sh: line 179: `fi'
<Apr 1, 2015 12:40:15 AM> <FINEST> <NodeManager> <Waiting for the process to die: 2919>
<Apr 1, 2015 12:40:15 AM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>
<Apr 1, 2015 12:40:15 AM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>
========================================================================
Cause:
Logs clearly says there is problem with the startWebLogic.sh which is having invalid syntax.
On investigation found that it was modified recently and having invalid syntax for token "fi". 
If conditional statement wasn't properly used.
========================================================================
Solution:
Correct the syntax for if conditional statement used in startWebLogic.sh. 
Example is given below,

if [ condition ]; then 
............
fi

In my case, extra "fi" was used at the end of if statement so have to remove that extra "fi" and restart server.

Wednesday 1 April 2015

javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User weblogic denied during admin server restart

==============================================================================
Applies To:
SOA Product Type: BPEL - version 11.1.1.6.7
Oracle Database: Enterprise Edition - Version 11.2.0.3

==============================================================================
Problem:
Admin Server is not able to start because of javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User weblogic denied

####<Mar 31, 2015 12:14:51 AM MYT> <Critical> <WebLogicServer> <puru.com> <poc_bpel_admin> <Main Thread> <<WLS Kernel>> <> <> <1427732091824> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.
weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.   at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:960)     at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)   at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: javax.security.auth.login.FailedLoginException: [Security:090304]Authentication Failed: User weblogic javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User weblogic denied   at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:261)   at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)    At com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:579)
        at com.bea.common.security.internal.service.JAASLoginServiceImpl.login(JAASLoginServiceImpl.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
        at $Proxy39.login(Unknown Source)  at weblogic.security.service.internal.WLSJAASLoginServiceImpl$ServiceImpl.login(WLSJAASLoginServiceImpl.java:89)   at com.bea.common.security.internal.service.JAASAuthenticationServiceImpl.authenticate(JAASAuthenticationServiceImpl.java:82)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)  at java.lang.reflect.Method.invoke(Method.java:597)  at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
        at $Proxy57.authenticate(Unknown Source)  at weblogic.security.service.WLSJAASAuthenticationServiceWrapper.authenticate(WLSJAASAuthenticationServiceWrapper.java:40)  at weblogic.security.service.PrincipalAuthenticator.authenticate(PrincipalAuthenticator.java:338)  at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:930)  at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)  at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
> 
####<Mar 31, 2015 12:14:51 AM MYT> <Error> <WebLogicServer> <puru.com> <poc_bpel_admin> <Main Thread> <<WLS Kernel>> <> <> <1427732091887> <BEA-000383> <A critical service failed. The server will shut itself down>
==============================================================================
Cause:
The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created.
===============================================================================
Solution:
Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.
1.       Go to your AdminServer directory, $DOMAIN_HOME/servers/AdminServer
2.       Check the boot.properties file under security folder and data/nodemanager folder if they have been changed recently.
3.       If file is been changed recently then restore it from backup if you had taken it before change.
OR
If we don’t have backup for boot.properties file then we can follow below process,
·         Take a backup of "data" folder, i.e. just rename "data" directory to "data_bkp"
·         Inside AdminServer create a directory "security"
·         Inside the "security" directory create a file with the name "boot.properties"
·         Edit boot.properties and enter the username and password as shown below,
username=weblogic
password=weblogic1
·         Start webLogic server

java.lang.NoClassDefFoundError: weblogic/security/SecurityLogger during admin server restart.

=======================================================================
Applies To:
SOA Product Type: BPEL - version 11.1.1.6.7
Oracle Database: Enterprise Edition - Version 11.2.0.3

========================================================================
Problem:
Admin Server is not able to start because of java.lang.NoClassDefFoundError: weblogic/security/SecurityLogger.

[INFO ][mgmnt  ] JDP autodiscovery started at address XXX.XXX.X.XXX:7001
Exception in thread "Main Thread" java.lang.NoClassDefFoundError: weblogic/security/SecurityLogger
        at weblogic.security.utils.SecurityUtils.turnOffCryptoJDefaultJCEVerification(SecurityUtils.java:81)
        at weblogic.Server.main(Server.java:67)
<Mar 30, 2015 12:49:40 PM> <FINEST> <NodeManager> <Waiting for the process to die: 18009>
<Mar 30, 2015 12:49:40 PM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>
<Mar 30, 2015 12:49:40 PM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>
========================================================================
Cause:
Security logger Jar files from the modules directory is corrupted/missing. This can be caused because of improper file transfer.

com.bea.core.weblogic.security.logger_1.5.0.0.jar
com.bea.core.weblogic.security.logger.ja_1.5.0.0.jar

========================================================================
Solution:
1.       If this domain is shared across multiple host, copy the missing modules directory jars from another host to the corrupted host.
 Or
 If this domains isn't shared across multiple hosts, you can copy jar files from another BPEL domain.
2.       Location of modules directory will be $ORACLE_HOME/modules.
3.       Make sure you have all required files and with the same size.
4.       Once done, restart admin server and it should come up fine.

Tuesday 6 January 2015

Error enqueing event: ORA-25207: enqueue failed

==============================================================================
Applies To:
SOA Product Type: BPM - version 11.1.1.6.7
Oracle Database: Enterprise Edition - Version 11.2.0.3
==============================================================================
Problem: When you call BPM composite sometimes you will hit with enqueue/dequeue errors as below,

Non Recoverable System Fault :
ORAMED-03303:[Unexpected exception in case execution]Unexpected exception in request response operation "operation" on reference "RetrieveFraudCaseStatusWorkflow.RetrieveFraudCaseStatusWorkflow.service". Possible Fix:Check whether the reference service is properly configured and running or look at exception for analyzing the reason or contact Oracle Support Services. Cause:Error enqueing event: ORA-25207: enqueue failed, queue DEV_SOAINFRA.EDN_EVENT_QUEUE is disabled from enqueueing ORA-06512: at "SYS.DBMS_AQ", line 169 ORA-06512: at "DEV_SOAINFRA.EDN_ENQUEUE_BUSINESS_EVENT", line 52 ORA-06512: at "DEV_SOAINFRA.EDN_INTERNAL_PUBLISH_EVENT", line 4

Non Recoverable System Fault :
Error enqueing event: ORA-25207: enqueue failed, queue DEV_SOAINFRA.EDN_EVENT_QUEUE is disabled from enqueueing ORA-06512: at "SYS.DBMS_AQ", line 169 ORA-06512: at "DEV_SOAINFRA.EDN_ENQUEUE_BUSINESS_EVENT", line 52 ORA-06512: at "DEV_SOAINFRA.EDN_INTERNAL_PUBLISH_EVENT", line 4
============================================================================
Cause:
EDN_EVENT_QUEUE enqueue/dequeue is disabled.
============================================================================
Solution:
If you face above error perform the below steps to fix it. This can be performed by SOA Admin /DBA
1.  Login to the DB with SOAINFRA user (e.g. DEV_SOAINFRA) or use the SYS account and go to DEV_SOAINFRA schema.
2.  Run the below pl/sql statement to enable enqueue and dequeue operation,

begin dbms_aqadm.start_queue (queue_name => 'EDN_EVENT_QUEUE', enqueue => TRUE , dequeue => TRUE); end;

3. Check the queue table DEV_SOAINFRA.EDN_EVENT_QUEUE_TABLE. Once enqueue/dequeue is enabled, you will find updates on table EDN_EVENT_QUEUE_TABLE.