com.bea.wli.sb.context.BindingLayerException: General runtime error: Unknown error while processing message for ProxyService

========================================================================
Applies To:
OSB - version 11.1.1.7.0
WLS - 10.3.6
========================================================================
Problem:
Client was reporting that he is getting com.bea.wli.sb.context.BindingLayerException: General runtime error: Unknown error while processing message for ProxyService intermittently from few managed servers in a clustered environment.

Errors in log,

Caused By: com.bea.wli.sb.management.BrokerManagementException: java.lang.RuntimeException: com.bea.wli.sb.management.BrokerManagementException: com.bea.wli.sb.stages.StageException: The configured XQuery resource cannot be found.
                at com.bea.wli.sb.pipeline.RouterManager.getRouterRuntime(RouterManager.java:646)
                at com.bea.wli.sb.pipeline.RouterContext.getInstance(RouterContext.java:180)
                at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:583)   at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:398) at com.bea.wli.sb.transports.local.LocalTransportProvider.sendMessageAsync(LocalTransportProvider.java:322)     at sun.reflect.GeneratedMethodAccessor476.invoke(Unknown Source)   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
{{{{{{{{{{{{{{{ Truncating }}}}}}}}}}}}}}}}}}}}}}
at oracle.tip.adapter.fw.jca.messageinflow.MessageEndpointImpl.onMessage(MessageEndpointImpl.java:556)   at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent$AQDeliveryBean.sendInboundRecord(AbstractDequeueAgent.java:571)        at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.__AW_run(AbstractDequeueAgent.java:123)
                at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.run(AbstractDequeueAgent.java)
                at java.lang.Thread.run(Thread.java:662)
Caused By: weblogic.xml.query.exceptions.XQueryTypeException: line 15, column 10: {err}XP0004: Invalid static type: (anyAttribute|PI)
                at weblogic.xml.query.compiler.TypeMatchExpression.typeCheckNoCache(TypeMatchExpression.java:144)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.FirstOrderExpression.typeCheckNoCache(FirstOrderExpression.java:269)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.IfThenElseExpression.typeCheckNoCache(IfThenElseExpression.java:112)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.GeneralFLWGDORExpression.typeCheckNoCache(GeneralFLWGDORExpression.java:404)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.NodeConstructor.typeCheckNoCache(NodeConstructor.java:278)

========================================================================

Cause:

On investigation found that the root cause for com.bea.wli.sb.context.BindingLayerException: General runtime error: Unknown error while processing message for ProxyService is the configured XQuery resource which wasn't found at the run time. This was strange behavior because this is one of the server in cluster and has same copy like other where it's working fine. If we restart MS, it will resolve the issue temporarily but again will show up during peak load.

========================================================================

Resolution:

After some research found one oracle note Doc ID 1679044.1 which says it's a bug and there is patch 18427728 available to resolve the issue.

After applying patch, we haven't seen error anymore during run time.

Server failed during startup and moving to FAILED_NOT_RESTARTABLE

========================================================================

Applies To:

OSB - version 11.1.1.7.0

WLS - 10.3.6

========================================================================

Problem:

Today one of my client reported that he is not able to restart the managed sever and it's moving to FAILED_NOT_RESTARTABLE.

***************************************************

*  To start WebLogic Server, use a username and   *

*  password assigned to an admin-level user.  For *

*  server administration, use the WebLogic Server *

*  console at http://hostname:port/console        *

***************************************************

starting weblogic with Java version:

java version "1.6.0_45"

Java(TM) SE Runtime Environment (build 1.6.0_45-b06)

Oracle JRockit(R) (build R28.2.7-7-155314-1.6.0_45-20130329-0641-linux-x86_64, compiled mode)

/u01/domains/poc/osb/bin/startWebLogic.sh: line 179: syntax error near unexpected token `fi'

/u01/domains/poc/osb/bin/startWebLogic.sh: line 179: `fi'

<Apr 1, 2015 12:40:15 AM> <FINEST> <NodeManager> <Waiting for the process to die: 2919>

<Apr 1, 2015 12:40:15 AM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>

<Apr 1, 2015 12:40:15 AM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>

========================================================================

Cause:

Logs clearly says there is problem with the startWebLogic.sh which is having invalid syntax.

On investigation found that it was modified recently and having invalid syntax for token "fi". 

If conditional statement wasn't properly used.

========================================================================

Solution:

Correct the syntax for if conditional statement used in startWebLogic.sh. 

Example is given below,

if [ condition ]; then 

............

fi

In my case, extra "fi" was used at the end of if statement so have to remove that extra "fi" and restart server.

javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User weblogic denied during admin server restart

==============================================================================

Applies To:

SOA Product Type: BPEL - version 11.1.1.6.7

Oracle Database: Enterprise Edition - Version 11.2.0.3

==============================================================================

Problem:

Admin Server is not able to start because of javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User weblogic denied

####<Mar 31, 2015 12:14:51 AM MYT> <Critical> <WebLogicServer> <puru.com> <poc_bpel_admin> <Main Thread> <<WLS Kernel>> <> <> <1427732091824> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.

weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.   at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:960)     at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)   at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)

        at weblogic.security.SecurityService.start(SecurityService.java:141)

        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Caused By: javax.security.auth.login.FailedLoginException: [Security:090304]Authentication Failed: User weblogic javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User weblogic denied   at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:261)   at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)    At com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)

        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)

        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)

        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)

        at javax.security.auth.login.LoginContext.login(LoginContext.java:579)

        at com.bea.common.security.internal.service.JAASLoginServiceImpl.login(JAASLoginServiceImpl.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)

        at $Proxy39.login(Unknown Source)  at weblogic.security.service.internal.WLSJAASLoginServiceImpl$ServiceImpl.login(WLSJAASLoginServiceImpl.java:89)   at com.bea.common.security.internal.service.JAASAuthenticationServiceImpl.authenticate(JAASAuthenticationServiceImpl.java:82)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)  at java.lang.reflect.Method.invoke(Method.java:597)  at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)

        at $Proxy57.authenticate(Unknown Source)  at weblogic.security.service.WLSJAASAuthenticationServiceWrapper.authenticate(WLSJAASAuthenticationServiceWrapper.java:40)  at weblogic.security.service.PrincipalAuthenticator.authenticate(PrincipalAuthenticator.java:338)  at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:930)  at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)  at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)

        at weblogic.security.SecurityService.start(SecurityService.java:141)

        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

> 

####<Mar 31, 2015 12:14:51 AM MYT> <Error> <WebLogicServer> <puru.com> <poc_bpel_admin> <Main Thread> <<WLS Kernel>> <> <> <1427732091887> <BEA-000383> <A critical service failed. The server will shut itself down>

==============================================================================

Cause:

The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created.

===============================================================================

Solution:

Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.

1.       Go to your AdminServer directory, $DOMAIN_HOME/servers/AdminServer

2.       Check the boot.properties file under security folder and data/nodemanager folder if they have been changed recently.

3.       If file is been changed recently then restore it from backup if you had taken it before change.

OR

If we don’t have backup for boot.properties file then we can follow below process,

·         Take a backup of "data" folder, i.e. just rename "data" directory to "data_bkp"

·         Inside AdminServer create a directory "security"

·         Inside the "security" directory create a file with the name "boot.properties"

·         Edit boot.properties and enter the username and password as shown below,

username=weblogic

password=weblogic1

·         Start webLogic server

java.lang.NoClassDefFoundError: weblogic/security/SecurityLogger during admin server restart.

=======================================================================

Applies To:

SOA Product Type: BPEL - version 11.1.1.6.7

Oracle Database: Enterprise Edition - Version 11.2.0.3

========================================================================

Problem:

Admin Server is not able to start because of java.lang.NoClassDefFoundError: weblogic/security/SecurityLogger.

[INFO ][mgmnt  ] JDP autodiscovery started at address XXX.XXX.X.XXX:7001

Exception in thread "Main Thread" java.lang.NoClassDefFoundError: weblogic/security/SecurityLogger

        at weblogic.security.utils.SecurityUtils.turnOffCryptoJDefaultJCEVerification(SecurityUtils.java:81)

        at weblogic.Server.main(Server.java:67)

<Mar 30, 2015 12:49:40 PM> <FINEST> <NodeManager> <Waiting for the process to die: 18009>

<Mar 30, 2015 12:49:40 PM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>

<Mar 30, 2015 12:49:40 PM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>

========================================================================

Cause:

Security logger Jar files from the modules directory is corrupted/missing. This can be caused because of improper file transfer.

com.bea.core.weblogic.security.logger_1.5.0.0.jar

com.bea.core.weblogic.security.logger.ja_1.5.0.0.jar

========================================================================

Solution:

1.       If this domain is shared across multiple host, copy the missing modules directory jars from another host to the corrupted host.

 Or

 If this domains isn't shared across multiple hosts, you can copy jar files from another BPEL domain.

2.       Location of modules directory will be $ORACLE_HOME/modules.

3.       Make sure you have all required files and with the same size.

4.       Once done, restart admin server and it should come up fine.