Weblogic Datasource pool disables by itself and doesn't recover

Applies to:

Oracle Weblogic Server - Version: 10.0 and later   [Release: and later ]
Information in this document applies to any platform.

Symptoms

After the database recovery from a network / database outrage, the data source was disabled and didn't recover when the connection restored.
In the logs, it shows the following error when the network outrage was occurred.

UncategorizedSQLException: Hibernate operation: Cannot open connection; uncategorized SQLException for SQL [???]; SQL state [null]; error code [0]; weblogic.common.resourcepool.ResourceDeadException: Could not create pool connection. The DBMS driver exception was: Io exception: The Network Adapter could not establish the connection; nested exception is weblogic.jdbc.extensions.ConnectionDeadSQLException: 

Cause

A connection pool may disable itself after a configurable number of consecutive failures to replace dead connections. There after it will periodically try to reconnect to the DBMS and when it succeeds, the pool will re-enable itself.

In above environment, it has been used the default jdbc pool settings as follows:
Connection Creation Retry Frequency, which is 0
Connection Reserve Timeout 10
Seconds to Trust an Idle Pool Connection 10
Test Frequency 120

The "Connection Creation Retry Frequency" setting is the number of seconds between attempts to establish connections to the database. If a value is not set for this, data source creation fails if the database is unavailable. If it being set and if the database is unavailable when the data source is created, WebLogic Server will attempt to create connections in the pool again after the number of second being specified, and will continue to attempt to create the connections until it succeeds.

As the Connection Creation Retry Frequency is 0, means it never retry. That would explain why the data source has been disabled after database failure.

Solution

Needs to set Connection Creation Retry Frequency to a desired seconds.



Now you can folow us on facebook and post your comments/views and questions for expert advise. Check this out facebook


Find us on facebook here

Configure Transaction Timeout for BPEL on SOA 11g

Applies to:

Oracle SOA Platform - Version: 11.1.1.1.0 to 11.1.1.5.0 - 

Goal

During runtime you may be seeing errors in the log similar to the following:

The transaction was rolled back
or
Transaction Rolledback.: weblogic.transaction.internal.TimedOutException: Transaction timed out

The solution is typically to increase the transaction timeout for the process.  

Solution

As a general rule, you should keep the following relation between the timeout parameters:

syncMaxWaitTime < BPEL EJB's transaction timeout < Global Transaction Timeout

Note: This recommendation are ONLY applicable to Sync Processes. Additionally the default Timeout setting that comes with SOA 11g installation does not comply with this rule. You might need to adjust the setting according to your particular business needs.

1. Setting syncMaxWaitTime:

This property controls the maximum time the process result receiver will wait for a result before returning for Sync processes.

For SOA 11g R1 PS1 (11.1.1.1.0 to 11.1.1.5):
* Login into EM
* Expand SOA and right click on "soa-infra" and select: SOA Administration -> BPEL Properties
* Click on "More BPEL Configuration Properties..." link
* Locate syncMaxWaitTime and change it.


-- Alternative Method -- For SOA 11g R1 (11.1.1.1.0) ONLY:
* Take backup of bpel-config.xml, located at: /user_projects/domains//config/soa-infra/configuration/.
* Open the bpel-config.xml file.
* Edit the value for the syncMaxWaitTime property.
* Save the changes.
* Restart Oracle WebLogic Server.

Note: Since 11.1.1.2, bpel-config.xml is no longer available into the file system and therefore the only chance for modification is through EM Console.

2. Setting the transaction timeout for BPEL EJBs:

The timeout properties for the EJBs control the particular timeout setting for the SOA application, overriding the global setting specified by the JTA timeout (See step 3).

Note: Prior implement next steps, ensure to shutdown SOA managed server. Otherwise you will get errors, see following document for details Note 1271369.1

* Log into Oracle WebLogic Administration Console.
* Click Deployments.
* Expand soa-infra -> EJBs.
* Following EJBs need to be updated:

BPELActivityManagerBean
BPELDeliveryBean
BPELDispatcherBean
BPELEngineBean
BPELFinderBean
BPELInstanceManagerBean
BPELProcessManagerBean
BPELSensorValuesBean
BPELServerManagerBean

* You can change the parameter in the Configuration tab for the Transaction Timeout setting.
* Click Save.
* Save the Plan.xml to some known location
* Start SOA Managed Server

3. Setting the global transaction timeout at Weblogic Domain Level:

This property controls the transaction timeout seconds for active transactions. If the transaction is still in the "active" state after this time, it is automatically rolled back.

   * Log into Oracle WebLogic Administration Console.

   * Click Services -> JTA.

   * Change the value of Timeout Seconds (the default is 30).

   * Click Save.

   * Restart Oracle WebLogic Server.

SOA 11g Configuration Files and Log Files Location

Applies to:

Oracle SOA Platform - Version: 11.1.1.1.0 and later   [Release: 11gR1 and later ]
Information in this document applies to any platform.

Goal

The intent of this note is to list the important configuration and log files used to diagnose SOA 11g problems.

Solution

Throughout this note we will refer to the following:
- $ MIDDLEWARE_HOME =
- $ DOMAIN_HOME = $ MIDDLEWARE_HOME/user_projects/domains/

Here is a list of files to look for:
1.Domain configuration under $DOMAIN_HOME/config  directory
   - Config.xml
   - jdbc/-Jdbc.xml
   - jms/-Jms.xml files under

2. Adapter configuration under $MIDDLEWARE_HOME//soa/connectors/   directory. ( Oracle_SOA1 is the default name)
  - AppsAdapter.rar
  - DbAdapter.rar
  - FtpAdapter.rar
  - MQSeriesAdapter.rar
  - SocketAdapter.rar
  - AqAdapter.rar
  - FileAdapter.rar
  - JmsAdapter.rar
  - OracleBamAdapter.rar

If you are experiencing problems related to the adapter, please send us the appropriate adapter's rar file.
Also provide the Adapter's deployment plan (Plan.xml).

3. Admin Server log files under $DOMAIN_HOME/servers//logs  directory
 - Access.log
 - -Diagnostic.log
 - .log

4. SOA Server log files under $DOMAIN_HOME/servers//logs  directory
 - . log
  - -Diagnostic.log
  - .log 


Soumya...

Error is: 'weblogic.management.DeploymentExceptiE01on: [J2E49]Error while pro cessing library references

Issue Descriprion:
The ADF code when deployed on an remote ADF server fails with below exception.


Error Log:

#### <[STANDBY] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <402a65ad9b43fa5a:69fd5e4b:12cff20c642:-8000-00000000000002f4> <1293010450094>
in the execution of deployment request with ID '1293010438085' for task '0'. Error is: 'weblogic.management.DeploymentException: [J2EE:160149]Error while pro
cessing library references. Unresolved application library references, defined in weblogic-application.xml: [Extension-Name: adf.oracle.domain, Implementatio
n-Version: 11.1.1.1.0, exact-match: false], [Extension-Name: oracle.jsp.next, exact-match: false].'
weblogic.management.DeploymentException: [J2EE:160149]Error while processing library references. Unresolved application library references, defined in weblog
ic-application.xml: [Extension-Name: adf.oracle.domain, Implementation-Version: 11.1.1.1.0, exact-match: false], [Extension-Name: oracle.jsp.next, exact-matc
h: false].


Cause:
The remote ADF server managed server on which you are trying to deploy your ADF code doesn't have the ADF run time libraries deployed on it. Ask your server admin to deploy it.


Solution:
There are primarily two ways to deploy the ADF run time libraries on a Weblogic managed server(ADF Runtime libraties are supported/installed in the WLS domain).


Method 1: Through EM Console
Method 2: WLST scripts
Method 3: While creating a domain the ADF libraries can be deployed on the managed server.


I will be describing Method 2 here:


You use the custom WLST command applyJRF to configure the Managed Servers or cluster with JRF. To use the custom WLST commands, you must invoke the WLST script from an Oracle home in which the Oracle Fusion Middleware component has been installed



The format of the applyJRF command is:


applyJRF(target={server_name | cluster_name | *}, domainDir=domain_path,
        [shouldUpdateDomain= {true | false}])
You can use the applyJRF command online or offline:


In online mode, the JRF changes are implicitly activated if you use the shouldUpdateDomain option with the value true (which is the default.) In online mode, this option calls the online WLST save() and activate() commands.


In offline mode, you must restart the Administration Server and the Managed Servers or cluster. (In offline mode, if you specify the shouldUpdateDomain option with the value true, this option calls the WLST updateDomain() command.)


To configure a Managed Server with JRF, use the following command:


applyJRF(target='server1', domainDir='/scratch/Oracle/Middleware/user_projects/domains/domain1')
To configure all Managed servers in the domain with JRF, specify an asterisk (*) as the value of the target option.


To configure a cluster with JRF, use the following command:


cd $FMW_HOME/oracle_common/common/bin (eg /orionsap_u01/app_ADF/oracle/product/fmw/oracle_common/common/bin)


applyJRF(target='cluster', domainDir='/scratch/Oracle/Middleware/user_projects/domains/domain1')


An Example:



cd $FMW_HOME/oracle_common/common/bin


sh wlst.sh


wls:/offline/test_domain>connect('weblogic','!Welc0me!','adf1.mycompany.com:8100')
Connecting to t3://adf1.mycompany.com:8100 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'test_domain'.


Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.


wls:/mmi_domain/serverConfig> applyJRF('test_enc_ms_1','/u01/app_ADF/oracle/admin/test_domain/aserver/test_domain')
Location changed to serverRuntime tree. This is a read-only tree with DomainMBean as the root.
For more help, use help(domainConfig)
Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root.
For more help, use help(domainRuntime)
Location changed to edit tree. This is a writable tree with
DomainMBean as the root. To make changes you will need to start
an edit session via startEdit().


For more help, use help(edit)


Starting an edit session ...
Started edit session, please be sure to save and activate your
changes once you are done.
Saving all your changes ...
Saved all your changes successfully.
Activating all your changes, this may take a while ...
The edit lock associated with this edit session is released
once the activation is completed.
Activation completed
Already in Edit Tree


After this is done,restart the managed server and redeploy your code.

Soumya....
   

ORA-01031: insufficient privileges JBO-28006: Could not create persistence table PS_TXN_seq

Applies to:

Oracle JDeveloper/ ADF server - Version: 11.1.1.0.0 and later   [Release: and later ]
Information in this document applies to any platform.

Symptoms

You developed an ADF BC ("ADF Business Components") application.
It was tested and was working correctly when run from JDeveloper, in the Integrated WebLogic Server.
However, in your Production environment, some users report intermittent errors like:
JBO-28006: Could not create persistence table PS_TXN_seq

When forcing passivation to occur with the parameter jbo.ampool.doampooling=false,
as explained in Note:417546.1 - How To Reproduce Problems Related to AM Passivation / Activation
you get the following errors occurring during the passivation:
(with the Java Options
"-Djbo.ampool.doampooling=false -Djbo.debugoutput=console -Djbo.logging.show.function=true"
set in the Project Properties)

[439] **createTable** created sequence error, sqlStmt=create sequence PS_TXN_seq increment by 50
[440] java.sql.SQLSyntaxErrorException: ORA-01031: insufficient privileges

at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:91)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:206)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:455)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:413)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:1035)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:183)
at oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:942)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1223)
at oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1775)
at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:1744)
at oracle.jdbc.driver.OracleStatementWrapper.execute(OracleStatementWrapper.java:299)
at oracle.jbo.pcoll.OraclePersistManager.createTable(OraclePersistManager.java:832)
at oracle.jbo.pcoll.OraclePersistManager.queryNextCollectionId(OraclePersistManager.java:1444)
at oracle.jbo.pcoll.PCollManager.register(PCollManager.java:562)
at oracle.jbo.pcoll.PCollection.(PCollection.java:99)
at oracle.jbo.pcoll.PCollManager.createCollection(PCollManager.java:461)
at oracle.jbo.server.DBSerializer.setup(DBSerializer.java:150)
at oracle.jbo.server.DBSerializer.passivateRootAM(DBSerializer.java:283)
at oracle.jbo.server.DBSerializer.passivateRootAM(DBSerializer.java:264)
at oracle.jbo.server.ApplicationModuleImpl.passivateStateInternal(ApplicationModuleImpl.java:5645)
at oracle.jbo.server.ApplicationModuleImpl.passivateState(ApplicationModuleImpl.java:5516)
...
[441] JBO-28006: Could not create persistence table PS_TXN_seq
[442] *** Stack trace for detail exception ***
[443] java.sql.SQLSyntaxErrorException: ORA-01031: insufficient privileges

at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:91)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:206)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:455)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:413)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:1035)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:183)
at oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:942)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1223)
at oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1775)
at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:1744)
at oracle.jdbc.driver.OracleStatementWrapper.execute(OracleStatementWrapper.java:299)
at oracle.jbo.pcoll.OraclePersistManager.createTable(OraclePersistManager.java:832)
at oracle.jbo.pcoll.OraclePersistManager.queryNextCollectionId(OraclePersistManager.java:1444)
at oracle.jbo.pcoll.PCollManager.register(PCollManager.java:562)
at oracle.jbo.pcoll.PCollection.(PCollection.java:99)
at oracle.jbo.pcoll.PCollManager.createCollection(PCollManager.java:461)
at oracle.jbo.server.DBSerializer.setup(DBSerializer.java:150)
at oracle.jbo.server.DBSerializer.passivateRootAM(DBSerializer.java:283)
at oracle.jbo.server.DBSerializer.passivateRootAM(DBSerializer.java:264)
at oracle.jbo.server.ApplicationModuleImpl.passivateStateInternal(ApplicationModuleImpl.java:5645)
at oracle.jbo.server.ApplicationModuleImpl.passivateState(ApplicationModuleImpl.java:5516)
...

Cause

Your DB schema hasn't got sufficient privileges to create the passivation table, sequence and/or
index in the database.

The passivation will store information in a table called PS_TXN and will use a DB sequence
PS_TXN_SEQ

These DB objects will be created

• in the same DB schema as specified in the Application Module Configuration
  (JDBC connection),
• or in the schema specified in the parameter jbo.server.internal_connection (recommended).

In order to create these, the DB schema needs to be granted DB privileges that are:

• CREATE TABLE
• CREATE INDEX
• CREATE SEQUENCE

For more information, see "Overview of Temporary Tables Created By BC4J"
and the Oracle Fusion Middleware Fusion Developer's Guide for Oracle Application Development Framework,
Chapter "39 Application State Management"
topic "39.5 What Model State Is Saved and When It Is Cleaned Up"

Solution

We recommend that you create a DB schema specific for the creation of these table.
Ask you DBA to create this schema and grant him the following privileges:

• CREATE TABLE
• CREATE INDEX
• CREATE SEQUENCE

When done, edit your Application Module Configuration, in the "Properties" tab and search for the
property called "jbo.server.internal_connection".
Specify there the value of your JDBC connection to connect to the schema you created.

A fully-qualified JDBC connection URL like:

jdbc:oracle:thin:/@:: 

or a JDBC datasource name like:

java:/comp/env/jdbc/YourJ2EEDataSourceName

The default CharSet ISO-8859-1 is not a unicode character set warning while starting SOA Managed server

ISSUE:

You get the following warning message while trying to start Admin/managed server for SOA Suite 11g.The message can be seen on the .out file:

Solution (Enabling Unicode Support)

Your operating system configuration can influence the behavior of characters supported by Oracle SOA Suite.On a UNIX operating system, Oracle highly recommends that you enable Unicode support by setting the LANG and LC_ALL environment variables to a locale with the UTF-8 character set. This enables the operating system to process any character in Unicode. SOA technologies are based on Unicode.If the operating system is configured to use a non-UTF-8 encoding, SOA components may function in an unexpected way. For example, a non-ASCII file name can make the file inaccessible and cause an error. Oracle does not support problems caused by operating system constraints.

In a design-time environment, if you are using Oracle JDeveloper, select Tools -> Preferences -> Environment -> Encoding -> UTF-8 to enable Unicode support.

Set locale to UTF-8

Login to the UNIX Box and type below commands

export LANG=”en_US.UTF-8”

export LC_ALL=”en_US.UTF-8”

Check the locale settings now by typing locale command>

bash-3.00$ locale

LANG=en_US.UTF-8

LC_CTYPE="en_US.UTF-8"

LC_NUMERIC="en_US.UTF-8"

LC_TIME="en_US.UTF-8"

LC_COLLATE="en_US.UTF-8"

LC_MONETARY="en_US.UTF-8"

LC_MESSAGES="en_US.UTF-8"

LC_ALL=en_US.UTF-8

Now restart the Admin/managed server.

Soumya....

UserWarning: MDS-91002: MDS Application runtime MBean for "soa-infra" is not available. "deleteMetadata" operation failure

You are trying to clean up the MDS using below steps:

1. Execute SOA_HOME/common/bin/wlst.sh
2. Connect to admin server by running connect()
3. Enter the domain config by running domainConfig()
4. Delete Metadata by running deleteMetadata('soa-infra', 'soa_server1','/apps/AIAMetaData/**')
   
   
   You encounter below error:
   
   
   UserWarning: MDS-91002: MDS Application runtime MBean for "soa-infra" is not available. "deleteMetadata" operation failure
   
   
   The SOA Managed Server is up and so s the SOAINFRA. What could be the possible reason?
   
   
   The reason for the above error is the name of the managed server used. In case your admin renamed the default managed server on which SOA is deployed (the default name is soa_server1) you would get the above stated error.
   
   
   So in case you get the above error, check with your admin the name of the SOA managed server. So in case your SOA managed server name is SOA_MS_1, the MDS purge command would be as follows:
   
   
   deleteMetadata('soa-infra', 'soa_ms_1','/apps/AIAMetaData/**')
   
   
   Hope this helps...
   
   Now you can folow us on facebook and post your comments/views and questions for expert advise. Check this out facebook
   
   
   Find us on facebook here

How to Check Fusion Middleware and WebLogic Component Versions

Applies to:
Oracle Fusion Middleware - Version: 11.1.1.1.0 to 11.1.1.5.0 - Release: to Oracle11g
Information in this document applies to any platform.

Goal
This document describes how to determine the versions of installed Fusion Middleware Components and Weblogic Server.

Note:
Fusion Middleware components including WebLogic Server, RCU, and components such as SOA, Identity Management, Portal-Forms-Reports-Discover, and JDeveloper must have concordant versions. This affects configuration, but not installation, and can cause a variety of failures, including ClassDefNotFound exceptions when trying to configure a new domain or managed server.

1. To determine the WebLogic Server version. There are several ways to confirm the version:

•  Enter this command with the weblogic environment enabled:

$ java weblogic.version

• Look in the file Middleware Home/registry.xml, and note the component tag

• The exact WLS version can be checked using the following commands:
  
  On Unix:

cat $MW_HOME/wlserver_10.3/.product.properties | grep WLS_PRODUCT_VERSION

On Windows:

type %MW_HOME%\wlserver_10.3\.product.properties | findstr WLS_PRODUCT_VERSION

• Use SmartUpdate to check the Product Version:
  
  On Unix:

$MW_HOME/utils/bsu/bsu.sh -view -status=applied -prod_dir=$MW_HOME/wlserver_10.3 | grep ProductVersion

On Windows:

%MW_HOME%\utils\bsu\bsu.sh -view -status=applied -prod_dir=$MW_HOME\wlserver_10.3 | findstr ProductVersion

You should see a result such as this:

ProductVersion: 10.3 MP5

2. For JDeveloper, look in the Middleware Home/registry.xml file for the following:

3. For Fusion Middleware products, use the command Oracle Home/Opatch/opatch lsinventory and note the output:

Installed Top-level Products (2):

Application Server 11g SOA Patchset 11.1.1.3.0
Oracle SOA Suite 11g 11.1.1.2.0
There are 2 products installed in this Oracle Home.

4. For the MDS version, look in the file RCU Home/rcu/log/logdir./mds.log:

REPOS_VERSION CONSTANT VARCHAR2(32) := '11.1.1.56.27' (this is for 11.1.1.3)

To get the schema version from the database, login to the database as sysdba and enter the command as shown (example here for 11.1.1.3):

SQL> select comp_id, comp_name, version from schema_version_registry;

COMP_ID
------------------------------
COMP_NAME
--------------------------------------------------------------------------------
VERSION
------------------------------
IAU
Audit Service
11.1.1.3.0

MDS
Metadata Services
11.1.1.2.0

OID
Oracle Internet Directory
11.1.1.3.0

OIF
Oracle Identity Federation Database Schema
11.1.1.2.0

ORAESS
Enterprise Scheduler Service
11.1.1.2.0

ORASDPLS
SIP Infrastructure Location Service
11.1.1.2.0

ORASDPSDS
SIP Infrastructure Subscriber Data Service
11.1.1.2.0

ORASDPXDMS
Presence
11.1.1.2.0

8 rows selected.

You can also get the schema prefix from the database using the following command:

SQL> select username, created from dba_users order by 2;

USERNAME CREATED
------------------------------ ---------
OE 12-MAY-10
SH 12-MAY-10
BI 12-MAY-10
PM 12-MAY-10
TEST01_MDS 21-JAN-11
TEST01_IAU 21-JAN-11
TEST01_IAU_APPEND 21-JAN-11
TEST01_IAU_VIEWER 21-JAN-11
TEST01_ORAESS 21-JAN-11
ODS 21-JAN-11
ODSSM 21-JAN-11

USERNAME CREATED
------------------------------ ---------
TEST01_OIF 21-JAN-11
TEST01_ORASDPXDMS 21-JAN-11
TEST01_ORASDPLS 21-JAN-11
TEST01_ORASDPSDS 21-JAN-11

The prefix given in the Repository Creation Utility (RCU) dialog is TEST01.