Sunday, October 16, 2011

Weblogic Datasource pool disables by itself and doesn't recover


Applies to:

Oracle Weblogic Server - Version: 10.0 and later   [Release: and later ]
Information in this document applies to any platform.

Symptoms

After the database recovery from a network / database outrage, the data source was disabled and didn't recover when the connection restored.
In the logs, it shows the following error when the network outrage was occurred.

UncategorizedSQLException: Hibernate operation: Cannot open connection; uncategorized SQLException for SQL [???]; SQL state [null]; error code [0]; weblogic.common.resourcepool.ResourceDeadException: Could not create pool connection. The DBMS driver exception was: Io exception: The Network Adapter could not establish the connection; nested exception is weblogic.jdbc.extensions.ConnectionDeadSQLException: 

Cause

A connection pool may disable itself after a configurable number of consecutive failures to replace dead connections. There after it will periodically try to reconnect to the DBMS and when it succeeds, the pool will re-enable itself.

In above environment, it has been used the default jdbc pool settings as follows:

Connection Creation Retry Frequency, which is 0
Connection Reserve Timeout 10
Seconds to Trust an Idle Pool Connection 10
Test Frequency 120

The "Connection Creation Retry Frequency" setting is the number of seconds between attempts to establish connections to the database. If a value is not set for this, data source creation fails if the database is unavailable. If it being set and if the database is unavailable when the data source is created, WebLogic Server will attempt to create connections in the pool again after the number of second being specified, and will continue to attempt to create the connections until it succeeds.

As the Connection Creation Retry Frequency is 0, means it never retry. That would explain why the data source has been disabled after database failure.

Solution

Needs to set Connection Creation Retry Frequency to a desired seconds.



Now you can folow us on facebook and post your comments/views and questions for expert advise. Check this out facebook


Find us on facebook here