Weblogic server fails to start with weblogic.store.PersistentStoreException: java.io.IOException: mmap, Resource temporarily unavailable error

Problem : WebLogic Managed Servers Fail During Startup with NullPointerException on the Persistent File Store

Applies to:

Oracle Weblogic Server - Version: 10.3 and later   [Release: and later ]
Information in this document applies to any platform.

Symptoms

WebLogic managed server fails to start with the error:

Error 1:

#### <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1306852872205>
#### <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1306852872303>
####

<> <> <> <1306852872315> java.lang.NullPointerException
at weblogic.store.io.file.StoreFile.close(StoreFile.java:432)
at weblogic.store.io.file.Heap.open(Heap.java:307)
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:104)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:413)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:404)
at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:118)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
at weblogic.diagnostics.lifecycle.DiagnosticSystemService.start(DiagnosticSystemService.java:128)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Error 2:

There are 1 nested errors:

weblogic.store.PersistentStoreException: java.io.IOException: mmap, Resource temporarily unavailable, errno=11

        at weblogic.store.io.file.Heap.getNextRecoveryFile(Heap.java:1204)

        at weblogic.store.io.file.Heap.recover(Heap.java:1029)

        at weblogic.store.io.file.FileStoreIO.recover(FileStoreIO.java:647)

        at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:114)

        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:413)

        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:404)

        at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:118)

        at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)

        at weblogic.diagnostics.lifecycle.DiagnosticSystemService.start(DiagnosticSystemService.java:128)

        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

Caused by: java.io.IOException: mmap, Resource temporarily unavailable, errno=11

        at weblogic.store.io.file.direct.DirectIONative.mapFile(Native Method)

        at weblogic.store.io.file.direct.FileMapping.mapView(FileMapping.java:169)

        at weblogic.store.io.file.direct.FileMapping.remapViewInit(FileMapping.java:140)

        at weblogic.store.io.file.direct.FileMapping.openInternal(FileMapping.java:105)

        at weblogic.store.io.file.direct.FileMapping.open(FileMapping.java:76)

        at weblogic.store.io.file.StoreFile.mappedRecoveryInit(StoreFile.java:519)

        at weblogic.store.io.file.Heap.getNextRecoveryFile(Heap.java:1191)

        ... 11 more

Cause

The file $DOMAIN_HOME/servers//data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT is somehow corrupted.

The Archive component of the WebLogic Diagnostics Framework (WLDF) captures and persists all data events, log records, and metrics collected by WLDF from server instances and applications running on them and saves them into this file. So having this file corrupted, prevents the Server from writing the WLDF information, and hence it fails to start.

Solution

Backup and remove the WLS_DIAGNOSTICS000000.DAT file, then restart the server: a new file will be created automatically.

This should not affect the server performance. It will only cause the loss of the archived diagnostic information (which is likely already lost since the file is corrupt).

References

NOTE:1059924.1 - WebLogic Server Diagnostics File is Corrupt or Locked

WebLogic Managed Servers Fail During Startup with NullPointerException on the Persistent File Store [ID 1327538.1]

Soumya...