Monday 6 June 2016

Weblogic admin fails to start because of javax.naming.ServiceUnavailableException [Root exception is java.rmi.NoSuchObjectException

========================================================================
Applies To:
WLS - 10.3.6.0.11
OSB - 11.1.1.7.4
========================================================================
Problem:
We are unable to start weblogic admin server becuase of javax.naming.ServiceUnavailableException [Root exception is java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.]

Error:
####<May 30, 2016 6:34:56 AM CDT> <Info> <messaging.interception> <purushottam> <poc_osb_admin> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1464608096553> <BEA-400000> <Initializing message interception service>
####<May 30, 2016 6:34:56 AM CDT> <Error> <Configuration Management> <purushottam> <poc_osb_admin> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1464608096834> <BEA-150000> <An error occurred while establishing a connection back to the adminstration server t3://192.168.184.130:6609 during startup. Since bootstrap succeeded check that t3://192.168.184.130:6609 uniquely identifies the administration server.
javax.naming.ServiceUnavailableException [Root exception is java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.]
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:70)
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:32)
        at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:792)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:431)
        at weblogic.jndi.Environment.getInitialReference(Environment.java:245)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:174)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:147)
        at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:88)
        at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:174)
        at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:154)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.
        at weblogic.rmi.internal.OIDManager.getServerReference(OIDManager.java:251)
        at weblogic.rmi.extensions.server.ServerHelper.getLocalInitialReference(ServerHelper.java:476)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:131)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:124)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:427)
        at weblogic.jndi.Environment.getInitialReference(Environment.java:245)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:174)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:160)
        at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:89)
        at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:175)
        at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:155)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
>
####<May 30, 2016 6:34:56 AM CDT> <Critical> <WebLogicServer> <purushottam> <poc_osb_admin> <Main Thread> <<WLS Kernel>> <> <> <1464608096842> <BEA-000362> <Server failed. Reason:

There are 1 nested errors:

javax.naming.ServiceUnavailableException [Root exception is java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.]
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:70)
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:32)
        at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:792)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:431)
        at weblogic.jndi.Environment.getInitialReference(Environment.java:245)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:174)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:147)
        at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:88)
        at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:174)
        at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:154)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.
        at weblogic.rmi.internal.OIDManager.getServerReference(OIDManager.java:251)
        at weblogic.rmi.extensions.server.ServerHelper.getLocalInitialReference(ServerHelper.java:476)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:131)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:124)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:427)
        at weblogic.jndi.Environment.getInitialReference(Environment.java:245)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:174)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:160)
        at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:89)
        at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:175)
        at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:155)
        ... 3 more

>
####<May 30, 2016 6:34:56 AM CDT> <Notice> <WebLogicServer> <purushottam> <poc_osb_admin> <Main Thread> <<WLS Kernel>> <> <> <1464608096863> <BEA-000365> <Server state changed to FAILED>
####<May 30, 2016 6:34:56 AM CDT> <Error> <WebLogicServer> <purushottam> <poc_osb_admin> <Main Thread> <<WLS Kernel>> <> <> <1464608096863> <BEA-000383> <A critical service failed. The server will shut itself down>
####<May 30, 2016 6:34:56 AM CDT> <Notice> <WebLogicServer> <purushottam> <poc_osb_admin> <Main Thread> <<WLS Kernel>> <> <> <1464608096867> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

========================================================================
Cause:
Weblogic admin server listener address was empty,
<listen-address></listen-address>

========================================================================
Resolution:
1. Go to $DOMAIN_HOME/config/config.xml file
2. Edit config.xml and set valid listen-address for admin server.
3. Restart the server.

Wednesday 1 June 2016

Weblogic Admin server fails to start Reason:java.security.AccessControlException: access denied (oracle.security.jps.JpsPermission idstore.config)

========================================================================
Applies To:
WLS - 10.3.6.0.11
OSB - 11.1.1.7.4
========================================================================
Problem:
We are unable to start weblogic admin server becuase of weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception.

Error:
####<May 30, 2016 6:25:36 AM CDT> <Critical> <WebLogicServer> <purushottam> <poc_osb_dit_admin> <Main Thread> <<WLS Kernel>> <> <> <1464607536237> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-03027: PDPService service instance failed during initialization due to lack of grant. If you are seeing this error after a new installation, make sure you have updated weblogic.policy as installation step suggested
weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-03027: PDPService service instance failed during initialization due to lack of grant. If you are seeing this error after a new installation, make sure you have updated weblogic.policy as installation step suggested
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1402)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)
        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: oracle.security.jps.JpsRuntimeException: JPS-03027: PDPService service instance failed during initialization due to lack of grant. If you are seeing this error after a new installation, make sure you have updated weblogic.policy as installation step suggested
        at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPDPService(PolicyUtil.java:2978)
        at oracle.security.jps.internal.policystore.PolicyUtil.getPDPService(PolicyUtil.java:3226)
        at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:167)
        at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:383)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at java.lang.Class.newInstance0(Class.java:357)
        at java.lang.Class.newInstance(Class.java:310)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1343)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)
        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.security.AccessControlException: access denied (oracle.security.jps.JpsPermission idstore.config)
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:374)
        at java.security.AccessController.checkPermission(AccessController.java:549)
        at oracle.security.jps.util.JpsAuth$AuthorizationMechanism$3.checkPermission(JpsAuth.java:463)
        at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:523)
        at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:549)
        at oracle.security.jps.internal.api.identitystore.IdentityStoreConfigurationUtil.checkPermission(IdentityStoreConfigurationUtil.java:548)
        at oracle.security.jps.internal.api.identitystore.IdentityStoreConfigurationUtil.getIgfLdapSpecifiedProperties(IdentityStoreConfigurationUtil.java:378)
        at oracle.security.jps.internal.api.identitystore.IdentityStoreConfigurationUtil.getLibOvdLdapPushData(IdentityStoreConfigurationUtil.java:526)
        at oracle.security.jps.internal.igf.ovd.OvdIGFServiceProvider$1.run(OvdIGFServiceProvider.java:232)
        at oracle.security.jps.internal.igf.ovd.OvdIGFServiceProvider$1.run(OvdIGFServiceProvider.java:229)
        at oracle.security.jps.internal.igf.ovd.OvdIGFServiceProvider.getConfigData(OvdIGFServiceProvider.java:228)
        at oracle.security.jps.internal.igf.ovd.OvdIGFServiceProvider.prepareDataPush2libOvd(OvdIGFServiceProvider.java:365)
        at oracle.security.jps.internal.idstore.ldap.LdapIdentityStoreProvider$NoLibOvd.getInstance(LdapIdentityStoreProvider.java:339)
        at oracle.security.jps.internal.idstore.ldap.LdapIdentityStoreProvider.getInstance(LdapIdentityStoreProvider.java:118)
        at oracle.security.jps.internal.idstore.ldap.LdapIdentityStoreProvider.getInstance(LdapIdentityStoreProvider.java:74)
        at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.findServiceInstance(ContextFactoryImpl.java:139)
        at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.getContext(ContextFactoryImpl.java:170)
        at oracle.security.jps.internal.core.runtime.ContextFactoryImpl.getContext(ContextFactoryImpl.java:191)
        at oracle.security.jps.internal.core.runtime.JpsContextFactoryImpl.getContext(JpsContextFactoryImpl.java:132)
        at oracle.security.jps.internal.core.runtime.JpsContextFactoryImpl.getContext(JpsContextFactoryImpl.java:127)
        at oracle.security.jps.internal.policystore.PolicyUtil$3.run(PolicyUtil.java:2957)
        at oracle.security.jps.internal.policystore.PolicyUtil$3.run(PolicyUtil.java:2950)
        at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPDPService(PolicyUtil.java:2950)
        at oracle.security.jps.internal.policystore.PolicyUtil.getPDPService(PolicyUtil.java:3228)
        at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:167)
        at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:383)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at java.lang.Class.newInstance0(Class.java:357)
        at java.lang.Class.newInstance(Class.java:310)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1343)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1024)
        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:894)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
>
####<May 30, 2016 6:25:36 AM CDT> <Notice> <WebLogicServer> <purushottam> <poc_osb_dit_admin> <Main Thread> <<WLS Kernel>> <> <> <1464607536482> <BEA-000365> <Server state changed to FAILED>
####<May 30, 2016 6:25:36 AM CDT> <Error> <WebLogicServer> <purushottam> <poc_osb_dit_admin> <Main Thread> <<WLS Kernel>> <> <> <1464607536482> <BEA-000383> <A critical service failed. The server will shut itself down>
####<May 30, 2016 6:25:36 AM CDT> <Notice> <WebLogicServer> <purushottam> <poc_osb_dit_admin> <Main Thread> <<WLS Kernel>> <> <> <1464607536498> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

========================================================================
Cause:
Weblogic Admin server file $DOMAIN_HOME/config/fmwconfig/system-jazn-data.xml wasn't having execute permissions.
-rw-rw-r--  1 oracle oracle        132K May 30  2016 system-jazn-data.xml

========================================================================
Resolution:
Kindly provide 775 permissions to $DOMAIN_HOME/config/fmwconfig/system-jazn-data.xml and restart the server.