Weblogic server fails to start with weblogic.store.PersistentStoreException: java.io.IOException: mmap, Resource temporarily unavailable error

Problem : WebLogic Managed Servers Fail During Startup with NullPointerException on the Persistent File Store

Applies to:

Oracle Weblogic Server - Version: 10.3 and later   [Release: and later ]
Information in this document applies to any platform.

Symptoms

WebLogic managed server fails to start with the error:

Error 1:

#### <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1306852872205>
#### <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1306852872303>
####

<> <> <> <1306852872315> java.lang.NullPointerException
at weblogic.store.io.file.StoreFile.close(StoreFile.java:432)
at weblogic.store.io.file.Heap.open(Heap.java:307)
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:104)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:413)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:404)
at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:118)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
at weblogic.diagnostics.lifecycle.DiagnosticSystemService.start(DiagnosticSystemService.java:128)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Error 2:

There are 1 nested errors:

weblogic.store.PersistentStoreException: java.io.IOException: mmap, Resource temporarily unavailable, errno=11

        at weblogic.store.io.file.Heap.getNextRecoveryFile(Heap.java:1204)

        at weblogic.store.io.file.Heap.recover(Heap.java:1029)

        at weblogic.store.io.file.FileStoreIO.recover(FileStoreIO.java:647)

        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:114)

        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:413)

        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:404)

        at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:118)

        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)

        at weblogic.diagnostics.lifecycle.DiagnosticSystemService.start(DiagnosticSystemService.java:128)

        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Caused by: java.io.IOException: mmap, Resource temporarily unavailable, errno=11

        at weblogic.store.io.file.direct.DirectIONative.mapFile(Native Method)

        at weblogic.store.io.file.direct.FileMapping.mapView(FileMapping.java:169)

        at weblogic.store.io.file.direct.FileMapping.remapViewInit(FileMapping.java:140)

        at weblogic.store.io.file.direct.FileMapping.openInternal(FileMapping.java:105)

        at weblogic.store.io.file.direct.FileMapping.open(FileMapping.java:76)

        at weblogic.store.io.file.StoreFile.mappedRecoveryInit(StoreFile.java:519)

        at weblogic.store.io.file.Heap.getNextRecoveryFile(Heap.java:1191)

        ... 11 more

Cause

The file $DOMAIN_HOME/servers//data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT is somehow corrupted.

The Archive component of the WebLogic Diagnostics Framework (WLDF) captures and persists all data events, log records, and metrics collected by WLDF from server instances and applications running on them and saves them into this file. So having this file corrupted, prevents the Server from writing the WLDF information, and hence it fails to start.

Solution

Backup and remove the WLS_DIAGNOSTICS000000.DAT file, then restart the server: a new file will be created automatically.

This should not affect the server performance. It will only cause the loss of the archived diagnostic information (which is likely already lost since the file is corrupt).

References

NOTE:1059924.1 - WebLogic Server Diagnostics File is Corrupt or Locked

WebLogic Managed Servers Fail During Startup with NullPointerException on the Persistent File Store [ID 1327538.1]

Soumya...

Can SOA 11g be Installed With an XE database?

Can SOA 11g be installed with an XE database?  While running rcu.bat (RCU) for SOA Suite Installation, the installer did not accept the default Oracle XE Service Name with error "Cannot find XE Service Name To Complete RCU Configuration". 


Although XE is not officially supported, many people use it without any issue for development or test environments or demonstrations. But before installing and running the RCU utility, you need to change the DB parameters as follows:


$ORACLE_HOME/bin/sqlplus /nolog

SQL> connect sys/oracle as sysdba;
SQL> EXEC DBMS_XDB.SETLISTENERLOCALACCESS(FALSE);
SQL> alter system set processes = 400 scope = spfile;
SQL> commit;
SQL> shutdown immediate

$ORACLE_HOME/bin/sqlplus /nolog

SQL> connect sys/oracle as sysdba;
SQL> startup
SQL> show parameter sessions
*** (make sure sessions is 225) ***
SQL> exit


For further reference please refer metalink id ID 1108487.1


Soumya...

Downloading of Patches Via BEA Smart Update no more supported

You need to apply a one-off patch to Weblogic. You run the bsu.sh script and try launching the BEA Smart Update. The Smart Update on launch recommends for an update. You go ahead and update it. The Smart Update now restarts and the you see the version of the utility to be 3.3.0.0. Yo try to find the Download patch tab, but surprisingly don't find it! You restart the BSU and wonder what went wrong!

Do not worry!

Smart Update patch and patch set download capability has been decommissioned by Oracle since June 10, 2011.Oracle says to better coordinate the distribution of patches to Oracle customers, they would consolidate on My Oracle Support as the single source for patch delivery([Metalink ID 1294294.1] ).

Hence you would now  need to download the Weblogic one-off patches from metalink from the patches and Updates section using your metalink ID.

How to Locate and Download Patches for WebLogic Server Using My Oracle Support

1. Log onto My Oracle Support.
2. Click on the "Patches & Updates" tab.
3. Click on "Product or Family" (Advanced Search)
4. For the "Product is" choose Oracle WebLogic Server.
5. For the "Release is" choose the WebLogic Server releases you want to search on
6. For "Description contains" type in your Smart Update Patch ID (in this case 8DNU)
7. Hit Search.

For step by step screen shots on how to download WLS patch please follow metalink note: 

Master Note: How to Locate and Download Patches for WebLogic Server Using My Oracle Support [ID 1302053.1]

Also you can refer to
WebLogic Net Installers and Downloading of Patches Via Smart Update To Be Decommissioned [ID 1294294.1] 

 Sanjay...

java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) error

Applies to:

Oracle Weblogic Server - Version: 10.3 to 10.3.3
Information in this document applies to any platform.

Symptoms

A stateless session bean (SLSB) is deployed in WebLogic 10.3.0. The SLSB only has one bean instance in the pool (max-beans-in-free-pool=1and initial-beans-in-free-pool=1).
After several hours and with over 100000 incoming requests the bean instance goes into waiting state.
Since there is only one bean instance in the pool, this effectively hangs all incoming calls.

The thread dump shows the bean instance goes into waiting state:

"[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0149e600 nid=48 lwp_id=5603464 waiting on condition [6c240000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <89861118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2054)
at weblogic.ejb.container.pool.StatelessSessionPool.waitForBean(StatelessSessionPool.java:269)
at weblogic.ejb.container.pool.StatelessSessionPool.getBean(StatelessSessionPool.java:111)
at weblogic.ejb.container.manager.StatelessManager.preInvoke(StatelessManager.java:148)
at weblogic.ejb.container.internal.BaseRemoteObject.preInvoke(BaseRemoteObject.java:227)
at weblogic.ejb.container.internal.StatelessRemoteObject.preInvoke(StatelessRemoteObject.java:52)
at com.tetrapak.ppm.external.xml.bean.UMGMessageLogFacade_n73y0z_EOImpl.isMachineValid(UMGMessageLogFacade_n73y0z_EOImpl.java:261)
at com.tetrapak.ppm.external.xml.bean.UMGMessageLogFacade_n73y0z_EOImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:589)
at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:230)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:473)
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Changes

None.

Cause

This is a known issue.
java.util.concurrent.locks.Condition.await(txTimeOutMS, TimeUnit.MILLISECONDS) is called by all threads except one thread (Thread A) which has got the bean from the pool.
When Thread A releases the bean, it acquires lock and call Condition.signal() which should wake up one waiting thread. But, none of thread actually comes out of waiting state unless one of the waiting thread times out and throw RuntimeException.

Solution

To implement the solution, please execute the following steps:
1. Download and install the patch using Smart Update
    Pass ID: QE7V
    Passcode: T5QP1CXU
2. Ensure that you have taken a backup of your system before applying the recommended patch.

3. Apply the patch in a test environment.

4. Retest the issue.

5. Migrate the solution as appropriate to other environments.

References

WLS 10.3.0 - Waiting For Available Stateless Bean Instance [ID 1086380.1]



Now you can folow us on facebook and post your comments/views and questions for expert advise. Check this out facebook


Find us on facebook here

Weblogic Server Development Mode Vs Production Mode

While configuring any Weblogic domain the there are two modes the domain can be configured with.

- Development Mode

- Production Mode

The question is how to decide which mode to use:

Let’s say you are developing your applications. The process will involve multiple redeployments to the server. In this case you would wish for a relaxed security configuration on the server. Also you would wish for features like auto deployment of your code. In such scenarios use the Development Mode.

Now consider you got your application working right, ready to go live. In this mode you would definitely wish for a full fledged security. Consider using the Production Mode here.

The differences between the two modes are stated below:

1.       SSL

In development mode you can use the demonstration digital certificates and the demonstration keystores provided by the WebLogic Server security services. With these certificates, you can design your application to work within environments secured by SSL where as in Production Mode you should not use the demonstration digital certificates and the demonstration keystores. If you do so, a warning message is displayed.

2.       Application Deployment

In Development Mode WebLogic Server instances can automatically deploy and update applications that reside in the domain_name/autodeploy directory (where domain_name is the name of a domain) where as in Production Mode the auto-deployment feature is disabled, so you must use the WebLogic Server Administration Console, the weblogic.Deployer tool, or the WebLogic Scripting Tool (WLST).

3.   

     Lock And Edit Feature

In Development Mode any configuration change being done by a user doesn’t need him to take a Lock and Edit session, where as in Production Mode the user needs to procure a Lock and Edit session before trying to make any configurational changes.

Important Considerations:

ü  It is recommended that this method be used only in a single-server development environment.

ü  You can always switch the domain startup mode from Development to Production and Vice Versa

ü  Oracle recommends usage of JRockit instead of Sun JDK in Production Mode. This recommendation depends on platform on which your domain runs. For platforms like Solaris SunJDK is recommended even in Production Mode. JRockit works well on OEL/Red hat Linux platforms.

Now you can folow us on facebook and post your comments/views and questions for expert advise. Check this out facebook

Find us on facebook here