ADMIN server failed to start because of Internal Exception: java.sql.SQLException: ORA-28001: the password has expired for SOA Schemas

========================================================================

Applies To:

BPEL/OSB - 12.1.3

WLS - 12.1.3

Oracle DB - Enterprise Edition - Version 11.2.0.3

========================================================================

Problem:

ADMIN server failed to start because of Internal Exception: java.sql.SQLException: ORA-28001: the password has expired for SOA Schemas

Error:

Caused By: oracle.security.jps.JpsException: oracle.security.jps.service.policystore.PolicyStoreException: javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException

Internal Exception: java.sql.SQLException: ORA-28001: the password has expired

Error Code: 28001

        at oracle.security.jps.internal.config.OpssCommonStartup.start(OpssCommonStartup.java:211)

        at oracle.security.jps.wls.JpsWlsStartup.start(JpsWlsStartup.java:80)

        at oracle.security.jps.JpsStartup.start(JpsStartup.java:186)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.startJPS(CommonSecurityServiceManagerDelegateImpl.java:1499)

        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1422)

        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.preInitialize(CommonSecurityServiceManagerDelegateImpl.java:1090)

        at weblogic.security.service.SecurityServiceManager.preInitialize(SecurityServiceManager.java:925)

        at weblogic.security.PreSecurityService.start(PreSecurityService.java:139)

        at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:78)

        at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

========================================================================

Cause: 

Passwords used by SOA (with it’s different users) to access the Database have expired. This is the designed behaviour.

java.sql.SQLException: ORA-28001: the password has expired

========================================================================

Resolution:

1.Log in to SQLPLUS with system as sysdba

2.Execute the following sql command to see all the schemas that are used in your repository and account status,

SET linesize 120

SET pagesize 1000 

col owner FORMAT a20 

col comp_name FORMAT a40 

col version FORMAT a12

SELECT username, profile, account_status, expiry_date, default_tablespace from DBA_USERS where username IN (SELECT owner FROM schema_version_registry);

You should change the passwords of all schemas whose password is expired.

3.Change the SOA schema db password using below command,

ALTER USER user_name identified BY new_password;

Would like to suggest to set new password same as old so that we need not to do DS password update on console.

4.Set password expiry to unlimited using below command,

ALTER PROFILE default LIMIT PASSWORD_LIFE_TIME UNLIMITED;

It will set password expiry to default for all users under the default profile. In case you want to apply this just for some specific users, you can create a dedicated security profile and associate users with this profile,

CREATE PROFILE <profile_name_here> LIMIT PASSWORD_LIFE_TIME UNLIMITED;

ALTER USER <user_name_here> PROFILE <profile_name_here>;

5.Check the changes by logging to DB using SOA schema/password.

6.Log in to weblogic console and go to "Domain Structure/Services/Data Source" to change password in Connection Pool tab for required schema. You can ignore this step if you have set new password same as old.

7.Restart server.

RCU-6080:Global prerequisite check failed - Check requirement for specified database failed while running SOA 12c RCU - Connecting to a multitenant container database (CDB) is not supported

========================================================================
Applies To:
BPEL - version 12.1.3
WLS - 12.1.3
Oracle DB - 12.1.0.2.0
========================================================================
Problem:
While creating my local domain for POC work , faced the following error while running SOA 12c RCU for creating schemas,

Error:
RCU creation failed due to following error.

RCU-6002:The specified database does not meet the minimum requirement to load metadata repository.
RCU-6080:Global prerequisite check failed - Check requirement for specified database
The selected Oracle database is a multitenant container database (CDB). Connecting to a multitenant container database (CDB) is not supported. Instead, connect to a valid pluggble database (PDB).
========================================================================
Cause: 
The selected Oracle database is a multitenant container database (CDB)
========================================================================
Resolution:
1. Determine if your database has been created as CDB or not using below SQL
SELECT CDB FROM V$DATABASE;
2. Check if any PDBs are available in the Database.
SELECT CON_ID, NAME, OPEN_MODE FROM V$PDBS;
    If there is no PDB other than PDB$SEED, please work with your DBA to create a PDB for the RCU schema
3. If there is a PDB then get full service names using below sql,
select name from v$services;
4. Make sure PDB is open (Mounted doesn't mean it's open). To open PDB run below sql,
alter pluggable database all open;
5. Verify if PDB open by running,
SELECT CON_ID, NAME, OPEN_MODE FROM V$PDBS;
6. Once done you can verify service using "lsnrctl status"
7. Now DB is ready for schema creation using RCU. Connect and create schema.

Oracle document for reference: http://docs.oracle.com/database/121/ADMIN/cdb_plug.htm#CEGCEEAJ

SOA Infrastructure Startup Failure in SOA 12c

========================================================================
Applies To:
BPEL - version 12.1.3
WLS - 12.1.3
========================================================================
Problem:
SOA Infra failed to start

Error Snippet:
Caused By: java.security.UnrecoverableKeyException: Password verification
failed  at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:769)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:38)
        at java.security.KeyStore.load(KeyStore.java:1185)
        at oracle.j2ee.ws.saaj.util.SSLUtil.loadKeyStore(SSLUtil.java:73)
        at oracle.j2ee.ws.saaj.util.SSLUtil.getKeyManagerFactory(SSLUtil.java:88)
        at oracle.j2ee.ws.saaj.util.SSLUtil.getKeyManagers(SSLUtil.java:97)
        at oracle.j2ee.ws.saaj.util.SSLUtil.createSSLSocketFactory(SSLUtil.java:50)
        at oracle.integration.platform.common.SSLSocketFactoryManagerImpl.getSSLSocketFac
        tory(SSLSocketFactoryManagerImpl.java:58)
        at oracle.fabric.common.wsdl.WSDLManager.init(WSDLManager.java:356)

======================================================================
Cause:
As per oracle documentation of SOA 12c- Administering Oracle SOA Suite and Oracle Business Process Management Suite, this is known issue and causes because of cwallet.sso has the SOA map
======================================================================
Resolution:
Perform one of the following actions
Delete the SOA map in cwallet.sso.
OR
Remove $DOMAIN_HOME/config/fmwconfig/default-keystore.jks
Restart the SOA Infrastructure

oracle.wsm.policymanager.PolicyManagerException: WSM-02141 : Unable to connect to the policy access service in SOA 12c(12.1.3)

========================================================================
Applies To:
BPEL - version 12.1.3
WLS - 12.1.3
========================================================================
Problem:
Today critical issue has been reported by user for SOA 12c (12.1.3) that policy manager has stopped working and while accessing WSM policy manager following error is seen,
oracle.wsm.policymanager.PolicyManagerException: WSM-02141 : Unable to connect to the policy access service

Services which uses security policies throwing below error
>>>InvalidSecurity : error in processing the WS-Security security header

Error Snippet:
oracle.wsm.policymanager.PolicyManagerException: WSM-02141 : Unable to connect to the policy access service.
at oracle.wsm.policymanager.BeanFactory.getInitialContext(BeanFactory.java:810)
at oracle.wsm.policymanager.BeanFactory.getJndiObj(BeanFactory.java:844)
at oracle.wsm.policymanager.BeanFactory.lookupJndiObj(BeanFactory.java:1175)
at oracle.wsm.policymanager.BeanFactory.getUsageTrackerBeanEJB(BeanFactory.java:1019)
at oracle.wsm.policymanager.BeanFactory.getBeanEJB(BeanFactory.java:544)

======================================================================
Cause:
As per oracle documentation of SOA 12c Fusion Middleware Securing Web Services and Managing Policies with Oracle Web Services Manager, by default WSM Policy Manager is auto-wired in the domain agents and its default targeted to the managed servers and not to the AdminServer. This means that AdminServer uses an agent to connect to the policy manager and we need to manually bind the OWSM Agent to the Policy Manager URL.
In my case listen address of MS has been changed recently and owsm-pm-connection-http and t3 shows the status as Out of Sync
======================================================================
Resolution:
1. Login to EM console
2. From the WebLogic Domain menu -> Cross Component Wiring -> Components
3. In the Components Table, select OWSM Agent
4. In the Client Configurations table, verify that the Client ID owsm-pm-connection-http reflects the correct Policy Manager URL in the Connection column and if the Status column displays Out of Sync, you need to bind the Agent to the Policy Manager.
To do so,
Select owsm-pm-connection-http in the Client Configurations table and click Bind.
In the Bind Client Configuration page, verify that the Service End Point contains the correct Policy Manager URL and click Yes.
Confirmation is displayed on the Components page and the status of the Agent is changed to Wired.
Repeat same for t3 entry as well.
5. Also verify OWSM Policy Manager if status is Published. If status is out of sync then select policy manager and click Publish.

com.bea.wli.sb.context.BindingLayerException: General runtime error: Unknown error while processing message for ProxyService

========================================================================
Applies To:
OSB - version 11.1.1.7.0
WLS - 10.3.6
========================================================================
Problem:
Client was reporting that he is getting com.bea.wli.sb.context.BindingLayerException: General runtime error: Unknown error while processing message for ProxyService intermittently from few managed servers in a clustered environment.

Errors in log,

Caused By: com.bea.wli.sb.management.BrokerManagementException: java.lang.RuntimeException: com.bea.wli.sb.management.BrokerManagementException: com.bea.wli.sb.stages.StageException: The configured XQuery resource cannot be found.
                at com.bea.wli.sb.pipeline.RouterManager.getRouterRuntime(RouterManager.java:646)
                at com.bea.wli.sb.pipeline.RouterContext.getInstance(RouterContext.java:180)
                at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:583)   at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:398) at com.bea.wli.sb.transports.local.LocalTransportProvider.sendMessageAsync(LocalTransportProvider.java:322)     at sun.reflect.GeneratedMethodAccessor476.invoke(Unknown Source)   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
{{{{{{{{{{{{{{{ Truncating }}}}}}}}}}}}}}}}}}}}}}
at oracle.tip.adapter.fw.jca.messageinflow.MessageEndpointImpl.onMessage(MessageEndpointImpl.java:556)   at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent$AQDeliveryBean.sendInboundRecord(AbstractDequeueAgent.java:571)        at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.__AW_run(AbstractDequeueAgent.java:123)
                at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.run(AbstractDequeueAgent.java)
                at java.lang.Thread.run(Thread.java:662)
Caused By: weblogic.xml.query.exceptions.XQueryTypeException: line 15, column 10: {err}XP0004: Invalid static type: (anyAttribute|PI)
                at weblogic.xml.query.compiler.TypeMatchExpression.typeCheckNoCache(TypeMatchExpression.java:144)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.FirstOrderExpression.typeCheckNoCache(FirstOrderExpression.java:269)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.IfThenElseExpression.typeCheckNoCache(IfThenElseExpression.java:112)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.GeneralFLWGDORExpression.typeCheckNoCache(GeneralFLWGDORExpression.java:404)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.NodeConstructor.typeCheckNoCache(NodeConstructor.java:278)

========================================================================

Cause:

On investigation found that the root cause for com.bea.wli.sb.context.BindingLayerException: General runtime error: Unknown error while processing message for ProxyService is the configured XQuery resource which wasn't found at the run time. This was strange behavior because this is one of the server in cluster and has same copy like other where it's working fine. If we restart MS, it will resolve the issue temporarily but again will show up during peak load.

========================================================================

Resolution:

After some research found one oracle note Doc ID 1679044.1 which says it's a bug and there is patch 18427728 available to resolve the issue.

After applying patch, we haven't seen error anymore during run time.

Server failed during startup and moving to FAILED_NOT_RESTARTABLE

========================================================================

Applies To:

OSB - version 11.1.1.7.0

WLS - 10.3.6

========================================================================

Problem:

Today one of my client reported that he is not able to restart the managed sever and it's moving to FAILED_NOT_RESTARTABLE.

***************************************************

*  To start WebLogic Server, use a username and   *

*  password assigned to an admin-level user.  For *

*  server administration, use the WebLogic Server *

*  console at http://hostname:port/console        *

***************************************************

starting weblogic with Java version:

java version "1.6.0_45"

Java(TM) SE Runtime Environment (build 1.6.0_45-b06)

Oracle JRockit(R) (build R28.2.7-7-155314-1.6.0_45-20130329-0641-linux-x86_64, compiled mode)

/u01/domains/poc/osb/bin/startWebLogic.sh: line 179: syntax error near unexpected token `fi'

/u01/domains/poc/osb/bin/startWebLogic.sh: line 179: `fi'

<Apr 1, 2015 12:40:15 AM> <FINEST> <NodeManager> <Waiting for the process to die: 2919>

<Apr 1, 2015 12:40:15 AM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>

<Apr 1, 2015 12:40:15 AM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>

========================================================================

Cause:

Logs clearly says there is problem with the startWebLogic.sh which is having invalid syntax.

On investigation found that it was modified recently and having invalid syntax for token "fi". 

If conditional statement wasn't properly used.

========================================================================

Solution:

Correct the syntax for if conditional statement used in startWebLogic.sh. 

Example is given below,

if [ condition ]; then 

............

fi

In my case, extra "fi" was used at the end of if statement so have to remove that extra "fi" and restart server.

javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User weblogic denied during admin server restart

==============================================================================

Applies To:

SOA Product Type: BPEL - version 11.1.1.6.7

Oracle Database: Enterprise Edition - Version 11.2.0.3

==============================================================================

Problem:

Admin Server is not able to start because of javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User weblogic denied

####<Mar 31, 2015 12:14:51 AM MYT> <Critical> <WebLogicServer> <puru.com> <poc_bpel_admin> <Main Thread> <<WLS Kernel>> <> <> <1427732091824> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.

weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.   at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:960)     at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)   at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)

        at weblogic.security.SecurityService.start(SecurityService.java:141)

        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Caused By: javax.security.auth.login.FailedLoginException: [Security:090304]Authentication Failed: User weblogic javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User weblogic denied   at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:261)   at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)    At com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)

        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)

        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)

        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)

        at javax.security.auth.login.LoginContext.login(LoginContext.java:579)

        at com.bea.common.security.internal.service.JAASLoginServiceImpl.login(JAASLoginServiceImpl.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)

        at $Proxy39.login(Unknown Source)  at weblogic.security.service.internal.WLSJAASLoginServiceImpl$ServiceImpl.login(WLSJAASLoginServiceImpl.java:89)   at com.bea.common.security.internal.service.JAASAuthenticationServiceImpl.authenticate(JAASAuthenticationServiceImpl.java:82)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)  at java.lang.reflect.Method.invoke(Method.java:597)  at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)

        at $Proxy57.authenticate(Unknown Source)  at weblogic.security.service.WLSJAASAuthenticationServiceWrapper.authenticate(WLSJAASAuthenticationServiceWrapper.java:40)  at weblogic.security.service.PrincipalAuthenticator.authenticate(PrincipalAuthenticator.java:338)  at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:930)  at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)  at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)

        at weblogic.security.SecurityService.start(SecurityService.java:141)

        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

> 

####<Mar 31, 2015 12:14:51 AM MYT> <Error> <WebLogicServer> <puru.com> <poc_bpel_admin> <Main Thread> <<WLS Kernel>> <> <> <1427732091887> <BEA-000383> <A critical service failed. The server will shut itself down>

==============================================================================

Cause:

The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created.

===============================================================================

Solution:

Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.

1.       Go to your AdminServer directory, $DOMAIN_HOME/servers/AdminServer

2.       Check the boot.properties file under security folder and data/nodemanager folder if they have been changed recently.

3.       If file is been changed recently then restore it from backup if you had taken it before change.

OR

If we don’t have backup for boot.properties file then we can follow below process,

·         Take a backup of "data" folder, i.e. just rename "data" directory to "data_bkp"

·         Inside AdminServer create a directory "security"

·         Inside the "security" directory create a file with the name "boot.properties"

·         Edit boot.properties and enter the username and password as shown below,

username=weblogic

password=weblogic1

·         Start webLogic server