com.bea.wli.sb.context.BindingLayerException: General runtime error: Unknown error while processing message for ProxyService

========================================================================
Applies To:
OSB - version 11.1.1.7.0
WLS - 10.3.6
========================================================================
Problem:
Client was reporting that he is getting com.bea.wli.sb.context.BindingLayerException: General runtime error: Unknown error while processing message for ProxyService intermittently from few managed servers in a clustered environment.

Errors in log,

Caused By: com.bea.wli.sb.management.BrokerManagementException: java.lang.RuntimeException: com.bea.wli.sb.management.BrokerManagementException: com.bea.wli.sb.stages.StageException: The configured XQuery resource cannot be found.
                at com.bea.wli.sb.pipeline.RouterManager.getRouterRuntime(RouterManager.java:646)
                at com.bea.wli.sb.pipeline.RouterContext.getInstance(RouterContext.java:180)
                at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:583)   at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:398) at com.bea.wli.sb.transports.local.LocalTransportProvider.sendMessageAsync(LocalTransportProvider.java:322)     at sun.reflect.GeneratedMethodAccessor476.invoke(Unknown Source)   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
{{{{{{{{{{{{{{{ Truncating }}}}}}}}}}}}}}}}}}}}}}
at oracle.tip.adapter.fw.jca.messageinflow.MessageEndpointImpl.onMessage(MessageEndpointImpl.java:556)   at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent$AQDeliveryBean.sendInboundRecord(AbstractDequeueAgent.java:571)        at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.__AW_run(AbstractDequeueAgent.java:123)
                at oracle.tip.adapter.aq.v2.database.AbstractDequeueAgent.run(AbstractDequeueAgent.java)
                at java.lang.Thread.run(Thread.java:662)
Caused By: weblogic.xml.query.exceptions.XQueryTypeException: line 15, column 10: {err}XP0004: Invalid static type: (anyAttribute|PI)
                at weblogic.xml.query.compiler.TypeMatchExpression.typeCheckNoCache(TypeMatchExpression.java:144)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.FirstOrderExpression.typeCheckNoCache(FirstOrderExpression.java:269)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.IfThenElseExpression.typeCheckNoCache(IfThenElseExpression.java:112)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.GeneralFLWGDORExpression.typeCheckNoCache(GeneralFLWGDORExpression.java:404)
                at weblogic.xml.query.compiler.Expression.typeCheck(Expression.java:330)
                at weblogic.xml.query.compiler.NodeConstructor.typeCheckNoCache(NodeConstructor.java:278)

========================================================================

Cause:

On investigation found that the root cause for com.bea.wli.sb.context.BindingLayerException: General runtime error: Unknown error while processing message for ProxyService is the configured XQuery resource which wasn't found at the run time. This was strange behavior because this is one of the server in cluster and has same copy like other where it's working fine. If we restart MS, it will resolve the issue temporarily but again will show up during peak load.

========================================================================

Resolution:

After some research found one oracle note Doc ID 1679044.1 which says it's a bug and there is patch 18427728 available to resolve the issue.

After applying patch, we haven't seen error anymore during run time.

Server failed during startup and moving to FAILED_NOT_RESTARTABLE

========================================================================

Applies To:

OSB - version 11.1.1.7.0

WLS - 10.3.6

========================================================================

Problem:

Today one of my client reported that he is not able to restart the managed sever and it's moving to FAILED_NOT_RESTARTABLE.

***************************************************

*  To start WebLogic Server, use a username and   *

*  password assigned to an admin-level user.  For *

*  server administration, use the WebLogic Server *

*  console at http://hostname:port/console        *

***************************************************

starting weblogic with Java version:

java version "1.6.0_45"

Java(TM) SE Runtime Environment (build 1.6.0_45-b06)

Oracle JRockit(R) (build R28.2.7-7-155314-1.6.0_45-20130329-0641-linux-x86_64, compiled mode)

/u01/domains/poc/osb/bin/startWebLogic.sh: line 179: syntax error near unexpected token `fi'

/u01/domains/poc/osb/bin/startWebLogic.sh: line 179: `fi'

<Apr 1, 2015 12:40:15 AM> <FINEST> <NodeManager> <Waiting for the process to die: 2919>

<Apr 1, 2015 12:40:15 AM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>

<Apr 1, 2015 12:40:15 AM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>

========================================================================

Cause:

Logs clearly says there is problem with the startWebLogic.sh which is having invalid syntax.

On investigation found that it was modified recently and having invalid syntax for token "fi". 

If conditional statement wasn't properly used.

========================================================================

Solution:

Correct the syntax for if conditional statement used in startWebLogic.sh. 

Example is given below,

if [ condition ]; then 

............

fi

In my case, extra "fi" was used at the end of if statement so have to remove that extra "fi" and restart server.

javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User weblogic denied during admin server restart

==============================================================================

Applies To:

SOA Product Type: BPEL - version 11.1.1.6.7

Oracle Database: Enterprise Edition - Version 11.2.0.3

==============================================================================

Problem:

Admin Server is not able to start because of javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User weblogic denied

####<Mar 31, 2015 12:14:51 AM MYT> <Critical> <WebLogicServer> <puru.com> <poc_bpel_admin> <Main Thread> <<WLS Kernel>> <> <> <1427732091824> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.

weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.   at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:960)     at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)   at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)

        at weblogic.security.SecurityService.start(SecurityService.java:141)

        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Caused By: javax.security.auth.login.FailedLoginException: [Security:090304]Authentication Failed: User weblogic javax.security.auth.login.FailedLoginException: [Security:090302]Authentication Failed: User weblogic denied   at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:261)   at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)    At com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)

        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)

        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)

        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)

        at javax.security.auth.login.LoginContext.login(LoginContext.java:579)

        at com.bea.common.security.internal.service.JAASLoginServiceImpl.login(JAASLoginServiceImpl.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)

        at $Proxy39.login(Unknown Source)  at weblogic.security.service.internal.WLSJAASLoginServiceImpl$ServiceImpl.login(WLSJAASLoginServiceImpl.java:89)   at com.bea.common.security.internal.service.JAASAuthenticationServiceImpl.authenticate(JAASAuthenticationServiceImpl.java:82)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)  at java.lang.reflect.Method.invoke(Method.java:597)  at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)

        at $Proxy57.authenticate(Unknown Source)  at weblogic.security.service.WLSJAASAuthenticationServiceWrapper.authenticate(WLSJAASAuthenticationServiceWrapper.java:40)  at weblogic.security.service.PrincipalAuthenticator.authenticate(PrincipalAuthenticator.java:338)  at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:930)  at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)  at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)

        at weblogic.security.SecurityService.start(SecurityService.java:141)

        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

> 

####<Mar 31, 2015 12:14:51 AM MYT> <Error> <WebLogicServer> <puru.com> <poc_bpel_admin> <Main Thread> <<WLS Kernel>> <> <> <1427732091887> <BEA-000383> <A critical service failed. The server will shut itself down>

==============================================================================

Cause:

The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created.

===============================================================================

Solution:

Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.

1.       Go to your AdminServer directory, $DOMAIN_HOME/servers/AdminServer

2.       Check the boot.properties file under security folder and data/nodemanager folder if they have been changed recently.

3.       If file is been changed recently then restore it from backup if you had taken it before change.

OR

If we don’t have backup for boot.properties file then we can follow below process,

·         Take a backup of "data" folder, i.e. just rename "data" directory to "data_bkp"

·         Inside AdminServer create a directory "security"

·         Inside the "security" directory create a file with the name "boot.properties"

·         Edit boot.properties and enter the username and password as shown below,

username=weblogic

password=weblogic1

·         Start webLogic server

java.lang.NoClassDefFoundError: weblogic/security/SecurityLogger during admin server restart.

=======================================================================

Applies To:

SOA Product Type: BPEL - version 11.1.1.6.7

Oracle Database: Enterprise Edition - Version 11.2.0.3

========================================================================

Problem:

Admin Server is not able to start because of java.lang.NoClassDefFoundError: weblogic/security/SecurityLogger.

[INFO ][mgmnt  ] JDP autodiscovery started at address XXX.XXX.X.XXX:7001

Exception in thread "Main Thread" java.lang.NoClassDefFoundError: weblogic/security/SecurityLogger

        at weblogic.security.utils.SecurityUtils.turnOffCryptoJDefaultJCEVerification(SecurityUtils.java:81)

        at weblogic.Server.main(Server.java:67)

<Mar 30, 2015 12:49:40 PM> <FINEST> <NodeManager> <Waiting for the process to die: 18009>

<Mar 30, 2015 12:49:40 PM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>

<Mar 30, 2015 12:49:40 PM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>

========================================================================

Cause:

Security logger Jar files from the modules directory is corrupted/missing. This can be caused because of improper file transfer.

com.bea.core.weblogic.security.logger_1.5.0.0.jar

com.bea.core.weblogic.security.logger.ja_1.5.0.0.jar

========================================================================

Solution:

1.       If this domain is shared across multiple host, copy the missing modules directory jars from another host to the corrupted host.

 Or

 If this domains isn't shared across multiple hosts, you can copy jar files from another BPEL domain.

2.       Location of modules directory will be $ORACLE_HOME/modules.

3.       Make sure you have all required files and with the same size.

4.       Once done, restart admin server and it should come up fine.

Error enqueing event: ORA-25207: enqueue failed

==============================================================================

Applies To:

SOA Product Type: BPM - version 11.1.1.6.7

Oracle Database: Enterprise Edition - Version 11.2.0.3

==============================================================================

Problem: When you call BPM composite sometimes you will hit with enqueue/dequeue errors as below,

Non Recoverable System Fault :

ORAMED-03303:[Unexpected exception in case execution]Unexpected exception in request response operation "operation" on reference "RetrieveFraudCaseStatusWorkflow.RetrieveFraudCaseStatusWorkflow.service". Possible Fix:Check whether the reference service is properly configured and running or look at exception for analyzing the reason or contact Oracle Support Services. Cause:Error enqueing event: ORA-25207: enqueue failed, queue DEV_SOAINFRA.EDN_EVENT_QUEUE is disabled from enqueueing ORA-06512: at "SYS.DBMS_AQ", line 169 ORA-06512: at "DEV_SOAINFRA.EDN_ENQUEUE_BUSINESS_EVENT", line 52 ORA-06512: at "DEV_SOAINFRA.EDN_INTERNAL_PUBLISH_EVENT", line 4

Non Recoverable System Fault :

Error enqueing event: ORA-25207: enqueue failed, queue DEV_SOAINFRA.EDN_EVENT_QUEUE is disabled from enqueueing ORA-06512: at "SYS.DBMS_AQ", line 169 ORA-06512: at "DEV_SOAINFRA.EDN_ENQUEUE_BUSINESS_EVENT", line 52 ORA-06512: at "DEV_SOAINFRA.EDN_INTERNAL_PUBLISH_EVENT", line 4

============================================================================

Cause:

EDN_EVENT_QUEUE enqueue/dequeue is disabled.

============================================================================

Solution:

If you face above error perform the below steps to fix it. This can be performed by SOA Admin /DBA

1.  Login to the DB with SOAINFRA user (e.g. DEV_SOAINFRA) or use the SYS account and go to DEV_SOAINFRA schema.

2.  Run the below pl/sql statement to enable enqueue and dequeue operation,

begin dbms_aqadm.start_queue (queue_name => 'EDN_EVENT_QUEUE', enqueue => TRUE , dequeue => TRUE); end;

3. Check the queue table DEV_SOAINFRA.EDN_EVENT_QUEUE_TABLE. Once enqueue/dequeue is enabled, you will find updates on table EDN_EVENT_QUEUE_TABLE.

SOA-Infra is not coming up beause of exception com.tangosol.net.RequestTimeoutException

SOA infra isn't coming up and below error message has been shown in the server logs.

Error:
Caused By: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
  MemberSet=MasterMemberSet(
    ThisMember=null
    OldestMember=null
    ActualMemberSet=MemberSet(Size=0
      )
    MemberId|ServiceVersion|ServiceJoined|MemberState
    RecycleMillis=1200000
    RecycleSet=MemberSet(Size=0
      )
    )
)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
        at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:58)

Root Cause:
Tangosole Configuration wasn't correct. Localhost was missing from WKA settings

Solution:
Corrected WKA/Tangosole settings by providing localhost and restarted server

Oracle Fusion Middleware Node manager

Node Manager is a WebLogic Server utility that enables you to start, shut down, and restart Administration Server and Managed Server instances from a remote location

A Node Manager process is not associated with a specific WebLogic domain but with a machine. Although Node Manager is optional but it's recommended by oracle.

WebLogic Server provides two versions of Node Manager, Java-based and script-based, with similar functionality.  Java based NM provides more security capabilities than script based NM.

If you are installing WebLogic Server on a Windows system, you must use the Java version of Node Manager. The scripted version of Node Manager is not supported on Windows.

In order to use consensus leasing, you may see faster performance when using the Java version of Node Manager.

What You Can Do with Node Manager

• Start, Shut Down, and Restart an Administration Server

• Start, Shut Down, Suspend, and Restart Managed Servers

• Monitor Servers and View Log Data

Starting Java-based Node Manager

• Start scripts for Node Manager are installed in the WL_HOME\server\bin directory, where WL_HOME is the top-level installation directory for WebLogic Server. 

• Use startNodeManager.cmd on Windows systems and startNodeManager.sh on UNIX systems.

• The scripts set the required environment variables and start Node Manager in WL_HOME/common/nodemanager. Node Manager uses this directory as a working directory for output and log files. 

Important Node Manager Configuration Files

1. nodemanager.properties

This is the configuration file used by the Java-based version of Node Manager.

This file is located in WL_HOME/common/nodemanager.

2. nodemanager.hosts

This file contains a list of all the trusted hosts that can issue commands to Node Manager.

This file is located in WL_HOME/common/nodemanager.

3. nodemanager.domains

This file contains mappings between the names of domains managed by Node Manager and their corresponding directories. 

This file is located in WL_HOME/common/nodemanager.

4. nm_data.properties

This file stores the encryption data the Node Manager uses a symmetric encryption key. The data is stored in encrypted form.

This file is located in WL_HOME/common/nodemanager.

5. nm_password.properties

This file stores a username/password pair specific to the Node Manager server that is managing this domain. This is known as the Node Manager secret. The username and password are appended to a salt value (obtained from the SerializedSystemIni.dat of the domain) and SHA-hashed.

This file is located in DOMAIN_HOME/config/nodemanager.

6. boot.properties

Node Manager uses this file to specify a boot identity when starting a server. This file is located in domain-name/servers/server_name/data/nodemanager.

7. startup.properties

Each Managed Server instance has its own startup.properties file with properties that control how Node Manager starts up and controls the server. Node Manager automatically creates this file by using properties passed to Node Manager when the Administrative Server was last used to start the server. This allows a Node Manager client or startup scripts to restart a Managed Server using the same properties last used by the Administrative Server.

These properties correspond to the server startup attributes contained in ServerStartMBean and the health monitoring attributes in ServerStartMBean.

This file is located in domain-name/servers/server_name/data/nodemanager.

8. server_name.addr

server_name.addr stores the IP address added when a server starts or is migrated. This file is generated after the server IP address is successfully brought online during migration. server_name.addr is deleted when the IP address is brought offline. The server IP address is used to validate remove requests to prevent addresses being erroneously removed while shutting down the server.

This file is located in domain-name/servers/server_name/data/nodemanager.

9. server_name.lck

server_name.lck is generated by each server and contains an internally used lock ID.

This file is located in domain-name/servers/server_name/data/nodemanager

10. server_name.pid

server_name.pid is generated by each server and contains the process ID of the server. Node Manager checks the process ID generated by the server during crash recovery.

This file is located in domain-name/servers/server_name/data/nodemanager

11. server_name.state

server_name.state is generated by the server and contains the server's current state. Node Manager monitors the contents of this file to determine the current state of the server.

Note: Do not delete or alter this file. Without this file Node Manager cannot determine the current state of the server.

This file is located in domain-name/servers/server_name/data/nodemanager.