Unable to start weblogic admin server for a newly created domain on Weblogic 9.2 version due to authentication issue

While working on Weblogic 9.2 I could mark one of the issues faced during trying to start weblogic admin server for a newly created domain using startweblogic.sh script.On running the startmanagedserver script the script asks for authentication ie username.The moment you enter the admin username the script fails and so does the managed server fails to start.

The issue mentioned above is shown below:

***************************************************

* To start WebLogic Server, use a username and *
* password assigned to an admin-level user. For *
* server administration, use the WebLogic Server *
* console at http://hostname:port/console
*
***************************************************
starting weblogic with Java version:
java version "1.6.0_05"
Java(TM) SE Runtime Environment (build 1.6.0_05-b13)
BEA JRockit(R) (build R27.6.0-50_o-100423-1.6.0_05-20080626-2104-linux-ia32, compiled mode)
Starting WLS with line:
/u01/apps/oracle/product/bea/WLI103/jrockit_160_05/bin/java -jrockit -Xms256m -Xmx512m -da
-Dplatform.home=/u01/apps/oracle/product/bea/WLI103/wlserver_10.3
-Dwls.home=/u01/apps/oracle/product/bea/WLI103/wlserver_10.3/server
-Dweblogic.home=/u01/apps/oracle/product/bea/WLI103/wlserver_10.3/server
-Dweblogic.management.discover=true -Dwlw.iterativeDev=false
-Dwlw.testConsole=false -Dwlw.logErrorsToConsole=
-Dweblogic.ext.dirs=/u01/apps/oracle/product/bea/WLI103/patch_wlw1030/profiles/d
efault/sysext_manifest_classpath:/u01/apps/oracle/product/bea/WLI103/patch_wls10
30/profiles/default/sysext_manifest_classpath:/u01/apps/oracle/product/bea/WLI10
3/patch_wlp1030/profiles/default/sysext_manifest_classpath:/u01/apps/oracle/prod
uct/bea/WLI103/patch_cie670/profiles/default/sysext_manifest_classpath:/u01/apps/oracle/product/bea/WLI103/patch_alsb1031/profiles/default/sysext_manifest_classpath -Dweblogic.Name=oerServer
-Djava.security.policy=/u01/apps/oracle/product/bea/WLI103/wlserver_10.3/server/
lib/weblogic.policy weblogic.Server

Version R27.6.0-50_o-100423-1.6.0_05-20080626-2104-linux-ia32 from BEA Systems, Inc.>
2008
WebLogic Server Temporary Patch for 8408837 Fri Apr 03 17:01:18 EDT 2009
WebLogic Server Temporary Patch for CR378781, CR380313 Fri Sep 19 13:34:16 PDT 2008
WebLogic Server Temporary Patch for CR381056 Mon Oct 06 10:48:50 EDT 2008
WebLogic Server Temporary Patch for CR374413, CR378680 Tue Sep 02 09:55:36 PDT 2008
WebLogic Server Temporary Patch for CR378102 Wed Sep 10 23:28:48 PDT 2008
WebLogic Server Temporary Patch for CR378741 Tue Sep 09 13:08:51 PDT 2008
WebLogic Server Temporary Patch for CR376251 Wed Aug 06 09:19:34 PDT 2008
WebLogic Server Temporary Patch for CR371247 Sat Aug 09 20:10:38 PDT 2008
WebLogic Server Temporary Patch for CR377673 Tue Aug 12 20:39:50 EDT 2008
WebLogic Server Temporary Patch for CR377673 Tue Aug 12 20:39:50 EDT 2008
WebLogic Server Temporary Patch for CR376759 Thu Aug 14 14:53:02 PDT 2008
WebLogic Server 10.3 Fri Jul 25 16:30:05 EDT 2008 1137967 >
user.>
Enter username to boot WebLogic server:admin
-bash: admin: command not found
[1]+ Stopped ./startWebLogic.sh


The weblogic startup script prompts for a username. On typing the username the script fails.

Cause:



The weblogic startup script is expecting the weblogic admin username and password which it should be picking from the boot.properties file.The file should be present in the following location:
/u01/apps/oracle/product/bea/WLI103/user_projects/domains/oer_domain/servers/oerServer/security/
The file was not present here as the installation by some chance/error does not create it. So it was unable to start the weblogic server.


Resolution:


1. Go to directory /u01/apps/oracle/product/bea/WLI103/user_projects/domains/oer_domain/servers/oerServer/security/


2. Create a file using vi command named boot.properties


3. Enter the following to the file:
Username=weblogic #Weblogic admin username
Password=weblogic #Weblogic admin password

4. Save the file
5. Go to /u01/apps/oracle/product/bea/WLI103/user_projects/domains/wl_server/bin

6. Issue the command ./startWeblogic.sh &


This time the issue is resolved.You will be able to start the weblogic server without issues.

Let me know if you have any doubts...

Happy Learning

Soumya
Oracle Fusion middleware

Issues and Resolution during Oracle AIA 2.4 installation on Weblogic 9.2 MP3 and Oracle SOA 10.1.3.4 MLR 9

This year January I was assigned to a project where I was supposed to implement Oracle SOA suite on top of Weblogic 9.2.I have already posted the steps to do so in my earlier posts. Another challenge was to implement Oracle AIA 2.4 on top of this setup ie on Weblogic 9.2 and SOA 10.1.3.4.

I call it a challenge because as always Oracle document looks neat but when you actually try to implement AIA 2.4 on Weblogic as per the Oracle document expect to hit numerous roadblocks or issues. I had to work a lot with Oracle support and finally was able to install AIA 2.4 on Weblogic 9.2 (single as well as clustered environment).


In the below post I will be talking about some of the issues faced by us during Oracle AIA 2.4 installation on Weblogic 9.2 and their resolution.

Issue 1:


While starting the manual install of AIA 2.4 on Weblogic 9.2 and SOA 10.1.3.4 the following error was obtained:


jdbc:oracle:thin:@mytestdbserver.soumya.com:1521:DVEIP
[trycatch] Caught exception: java.sql.SQLException: ORA-01017: invalid username/password; logon denied
[sql] Executing commands
[sql] Failed to execute: create user aia identified by aia
[sql] java.sql.SQLException: ORA-28003: password verification for the specified password failed
[sql] ORA-20001: Password same as or similar to user

Cause:
The Database security team has implemented the security policies, due to which AIA install script was not able to create the aia user with the same password ie aia .This was verified by logging into the databse as sys user and then trying to create aia user with same password.Look below

SQL> create user aia identified by aia;
create user aia identified by aia
*
ERROR at line 1:
ORA-28003: password verification for the specified password failed

Resolution:
Work with the DBA team and disable such security restrictions. They can enable the restriction after AIA installation.

Issue 2:



While trying to install AIA Foundation Pack 2.4 on top of SOA+Weblogic 9.2 MP3 the FPWLInstall script was failing with the below error.

ant --noconfig -buildfile FPWLInstall.xml


CreateDatasources:
CreateCommonDatasources:
createConnectionPool:
echo creating ConnectionPool AIAConnectionPool


BUILD FAILED
/u07_00/aia_home/Infrastructure/install/wlscripts/FPWLInstall.xml:218: The following error occurred while executing this line:
/u07_00/aia_home/Infrastructure/install/wlscripts/FPWLInstall.xml:225: The following error occurred while executing this line:
/u07_00/aia_home/Infrastructure/install/wlscripts/FPWLConfiguration.xml:107: The following error occurred while executing this line:
/u07_00/aia_home/Infrastructure/install/wlscripts/FPWLConfiguration.xml:64: The following error occurred while executing this line:
/u07_00/aia_home/Infrastructure/install/wlscripts/FPWLCommonConfig.xml:187: The following error occurred while executing this line:
/u07_00/aia_home/Infrastructure/install/wlscripts/FPWLCommonConfig.xml:122: Unable to create mbean: Could not create provider JDBCConnectionPool


Cause:
The proxy value was set incorrectly in the arguments of the SOA Server.


Resolution:
To fix that, please do the following:

a. Log in to WebLogic Administration console http://:port/console.
b. Click Lock and Edit.
c. Navigate to Environments-Servers-OracleSOAServer(Managed server where SOA is hosted)-Server Start tab.
d. In case you have proxy server, please change the value of proxyhost to the correct URL.
e. In case you don't have proxy server(as in my case), just incluse -Dhttp.proxySet=false in the arguments and remove the proxyhost parameter.
f.Restart the managed server and restart installing AIA.


Sample:

Xbootclasspath/a:/u01/apps/oracle/product/soacore/integration/esb/lib/esbprotocol.jar -XX:MaxPermSize=512m -Dhttp.nonProxyHosts=*.oracle.com*.oraclecorp.com localhost 127.0.0.1
10.177.255.135 sraghura-pc sraghura-pc.idc.oracle.com -Dhttp.proxyPort=80 -Dhttp.proxySet=false -Doracle.home=/u01/apps/oracle/product/soacore -Dorabpel.home=/u01/apps/oracle/product/soacore/bpel -Djavax.xml.parsers.DocumentBuilderFactory=oracle.xml.jaxp.JXDocumentBuilderFactory -Djavax.xml.parsers.SAXParserFactory=oracle.xml.jaxp.JXSAXParserFactory -Djavax.xml.transform.TransformerFactory=oracle.xml.jaxp.JXSAXTransformerFactory -Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl -Djavax.xml.soap.SOAPFactory=oracle.j2ee.ws.saaj.soap.SOAPFactoryImpl -Djavax.xml.soap.SOAPConnectionFactory=oracle.j2ee.ws.saaj.client.p2p.HttpSOAPConnectionFactory -Djavax.xml.soap.SOAPElementFactory=oracle.j2ee.ws.saaj.soap.SOAPFactoryImpl -Dbpel.platform=weblogic_8 -Doraesb.home=/u01/apps/oracle/product/soacore/integration/esb -Dprimary_oc4j=false -DsoaPlatform=weblogic_8 -Dhttp.webdir.enable=false -Daq.jms.enabled=true -DTrackingEnabled=Y -Dweblogic.rjvm.enableRouterFix=true -Daia.home=/u01/apps/oracle/product/AIA_24 -Djavax.xml.rpc.ServiceFactory=oracle.j2ee.ws.client.ServiceFactoryImpl

Issue 3:


AIA 2.4 installation on Weblogic 9.2 fails with the following error:

[echo] --------------------------------------------------------------
[echo]
Deploying bpel process AIAErrorTaskAdministrationProcess on eipsoad1.biogenidec.com, port 8892
[echo] --------------------------------------------------------------
[echo]
[deployProcess] Deploying process


/u01/apps/oracle/product/AIA_24/Infrastructure/ErrorHandling/AIAErrorTaskAdministrationProcess/output/bpel_AIAErrorTaskAdministrationProcess_1.0.jar

BUILD FAILED
/u01/apps/oracle/product/AIA_24/Infrastructure/install/wlscripts/FPWLInstall.xml:218: The following error occurred while executing this line:
/u01/apps/oracle/product/AIA_24/Infrastructure/install/wlscripts/FPWLInstall.xml:226: The following error occurred while executing this line:
/u01/apps/oracle/product/AIA_24/Infrastructure/install/wlscripts/FPWLDeployment.xml:17: The following error occurred while executing this line:
/u01/apps/oracle/product/AIA_24/Infrastructure/install/wlscripts/FPWLDeployment.xml:29: The following error occurred while executing this line:
/u01/apps/oracle/product/AIA_24/Infrastructure/install/wlscripts/FPWLErrorHandlingInstall.xml:138: The following error occurred while executing this line:


/u01/apps/oracle/product/AIA_24/Infrastructure/ErrorHandling/AIAErrorTaskAdministrationProcess/build.xml:204: A problem occured while
connecting to server "eipsoad1.biogenidec.com" using port "8892": bpel_aiaerrortaskadministrationprocess_1.0.jar failed to deploy. Exception
message is: weblogic.transaction.internal.TimedOutException: Transaction timed out after 29 seconds
BEA1-003F9A96E4CE965ECCFA at com.oracle.bpel.client.util.ExceptionUtils.handleServerException(ExceptionUtils.java:94)


Cause and Solution:


786431.1 WebLogic Server 9.2 JTA Timeout values for OSM. The Note provide steps to increase the JTA default timeout.


Quite often the Java Transaction API (JTA) is set to the default timeout value for many OSM development/testing installations. The default value is 30 seconds and when there are bottlenecks in the system, many OSM java transactions cannot finish within 30 seconds. As a result, the user will see errors


Ideally the root cause of the bottleneck should be identified and resolved accordingly. However sometimes it is necessary to increase the JTA value, for example :


a) the nature of transaction will take longer than 30 seconds
b) it takes time to fix system bottleneck and a quick fix is needed.


1. Go to the BEA console
2. Expand Services under "Domain Structure"
3. Click JTA
4. Click "Lock & Edit"
5. Change "Timeout Seconds:" from value 30 to 300


This change will increase JTA timeout from 30 seconds to 300 seconds, if after this change you don't see the Transaction time out anymore, this means JTA is set to an adequate value ( you may consider lowering the JTA timeout to a smaller value).


300 is an arbitrary number, System Administrators can choose a number between 30 to 600. Please be aware that large JTA timeout values might make system stall when there are many hanging threads. The goal is find optimal value for JTA timeout so that the user will see no / very few JTA timeout exceptions in a day.

Restart FPWLInstall script to install AIA 2.4

Issue 4:




The FPWLInstall execution fails with the following error:

DeployESBSuitcase:
[echo]
[echo] --------------------------------------------------------------
[echo]
Registering ESB service
/u01/apps/oracle/product/AIA_24/Infrastructure/install/systems/AIASystemFoundation on eipsoad1.biogenidec.com, port 8892
[echo] --------------------------------------------------------------
[echo]
[deployESBSuitcase] Deployment Attempt Response :
[deployESBSuitcase]
[deployESBSuitcase]
[deployESBSuitcase] An unhandled exception has been thrown in the ESB system. The exception reported is: "oracle.tip.esb.console.exception.ConsoleTransactionException: File repository not initialized. May be ESB bootstrap failed
[deployESBSuitcase] Please review ESB prameters for their correctness.


or

DeployESBSuitcase:
[echo]
[echo] --------------------------------------------------------------
[echo]
Registering ESB service
$AIA_HOME/Infrastructure/install/systems/AIASystemFoundation on , port
[echo] --------------------------------------------------------------
[echo]
[deployESBSuitcase] Deployment Attempt Response :
[deployESBSuitcase]
[deployESBSuitcase]
[deployESBSuitcase] Designtime cache has not been initialized
[deployESBSuitcase] Please look in logs for following signs of failure. Fix them and restart. (a)
Database access errors (b) ESB Bootstrap errors (c) OC4J class load errors (d)
Product installation errors (e) Export ESB params and verify if host and port parameters are correct. Please contact Oracle Support if unable to fix the issue.
[deployESBSuitcase]
[deployESBSuitcase]
[deployESBSuitcase]

Cause:


The customer is facing the issue documented in an internal documentation bug. This has been found to be due to the fact that esbprotocol.jar is not found in the Weblogic Bootstrap classpath. When the esbprotocol.jar is not loaded by the Bootstrap classpath, the Slide initialization fails to complete as it cannot resolve the "esb:///" artifacts.
Fix:
1. Find out the jdk used by WebLogic, to do this run the startWeblogic.sh that is located under /project_user/domains/
2. Copy the esbprotocol.jar that is under the JAVA_HOME found on the previous step,
location:\jre\lib\ext directory.






Issue 5


Error faced during AIA 2.4 installation on weblogic 9.2

(Refer to FPWLInstall10282009-013503.log the Dev server AIA install log path:)

DeployESBSuitcase:


[echo]
[echo] --------------------------------------------------------------
[echo]
Registering ESB service /u01/apps/oracle/product/AIA_24/Infrastructure/CAVS/CAVSESB on eipsoad1.biogenidec.com, port 8892
[echo] --------------------------------------------------------------
[echo]
[deployESBSuitcase] Deployment Attempt Response :
[deployESBSuitcase]
[deployESBSuitcase] Entity Deployment Failed
[deployESBSuitcase]
[deployESBSuitcase]
[deployESBSuitcase] WSDL "http://eipsoad1.biogenidec.com:8892/AIAValidationSystemServlet/asyncrequestrecipient?wsdl" of service "AIASystem.ValidationSystem.AsyncRequestRecipient" is invalid or unaccessible. Following error is thrown by WSDL Parser : WSDL Parsing
Failed: Response: '404: Not Found' for url: 'http://eipsoad1.biogenidec.com:8892/AIAValidationSystemServlet/asyncrequestrecipient?wsdl'
[deployESBSuitcase] Ensure that the WSDL is accessible and valid. If the problem persists, Contact Oracle Support.
[deployESBSuitcase]
[deployESBSuitcase]
[deployESBSuitcase]


Cause:


The issue is caused because the AIAValidationSystemServlet was not deployed to the weblogic server. This servlet is part of the AIAApplication. The log file showed the following error to:


[wldeploy] Deployment Plan: null
[wldeploy] App root: /tmp/oracle/./config/deployments/AIAApplication
[wldeploy] App config: /tmp/oracle/./config/deployments/AIAApplication/plan
[wldeploy] Deployment Options:
{isRetireGracefully=true,isGracefulProductionToAdmin=false,isGracefulIgnoreSessions=false,rmiGracePeriod=-1,retireTimeoutSecs=-1,undeployAllVersions=false,archiveVersion=null,planVersion=null,isLibrary=false,libSpecVersion=null,libImplVersion=null,stageMode=null,clusterTimeout=3600000,
altDD=null,altWlsDD=null,name=AIAApplication,securityModel=null,securityValidationEnabled=false,versionIdentifier=null,isTestMode=false,forceUn
deployTimeout=0,defaultSubmoduleTargets=true,timeout=0}


[wldeploy] [BasicOperation.execute():421] : Initiating deploy operation for app, AIAApplication, on targets:
[wldeploy] [BasicOperation.execute():423] : OracleSOAServer
[wldeploy] Task 11 initiated: [Deployer:149026]deploy application AIAApplication on OracleSOAServer.
[wldeploy] Task 11 failed: [Deployer:149026]deploy application AIAApplication on OracleSOAServer.
[wldeploy] Target state: deploy failed on Server OracleSOAServer
[wldeploy] oracle/binding/DataControl


This shows that the AIADeployment application deployment failed. The weblogic server log file shows the error ClassNotFoundError: oracle/binding/DataControl. This class is part of the ADF installation. The cause of this issue is an improper installation of ADF.




Fix:
Make sure the ADF installation is performed properly and that the ADF jar files are added to the CLASSPATH of the server were you deploy the AIAApplication.The best way to add the ADF jar files using the WebLogic Console. Follow the steps below to do so:


1) Connect to the middleware server
2) Set your corrunt directory to /user_projects/
cd /user_projects//

3) Convert the file setupadf.sh from Windows format to Unix format by running the command below:


Copy setupadf.sh to setupadfTmp.sh
dos2unix setupadf.sh setupadfTmp.sh
4) Replace relative paths to the ADF jar files with absolute ones, you could use the bash shell command below:
baseDir=$(echo `pwd`
sed 's/\(^.*\)\/user_projects.*/\1/g'
sed 's/\//\\\//g');
sed "s/..\/..\/../$baseDir/g" setupadfTmp.sh > setupadfUnix.sh
5) Source the file setupadfUnix.sh that is located on the directory /user_projects//setupadfUnix.sh
. setupadfUnix.sh

6) Print the value for the CLASSPATH environment variable
i.e. echo $CLASSPATH
7) Connect and login to the WebLogic console:
i.e. Open a browser and navigate to: http://:/console


8) Expand the -> Environment -> Servers node on the Domain Structure section of the Web Logic Home page
9) Click on the Server where you are installing aia (i.e. OracleSoaServer)
10) Click on the Configuration -> Server Start tab
11) Click on the Lock & Edit Button that is located on the Change Center section of the console
12) Append the output of the command you executed in step 2 to the CLASSPATH environment variable.
13) Click on the Release Configuration button located on the Change Center section of the console.


For any troubleshooting please refer to document id 822831.1

JConsole setup for remote Oracle Application server 10.1.3.4 JVM monitoring

A lot of performance issues are seen as far as Oracle Application server instances are concerned with.The performance issues might be because of various factors like database,network,JVM etc.Troubleshooting Application level performance issues has got a lot to do with tuning the JVMs used by Oracle Application server oc4j containers.A lot of information about the JVMs like Heap size,Garbage Collection,memory usage,CPU usage etc can be found out using the Jconsole client provided by Sun.


The below steps explain how to setup Jconsole client on local system in order to monitor remote Oracle Application Server JVMs.The same setup can be used in order to monitor Weblogic server JVM too.Though a lot of document regarding the setup can be found on the internet none of those are as clear and precise like the steps mentioned below.The setup has been tested by me and should work flawlessly for others.


1. Install JDK 6 kit on your local system. The installer is named jdk-6u20-windows-i586 and can be downloaded from Sun site.


2. Install JDeveloper on your local system. The version should be 10.1.3.4. The JDeveloper comes with some library files required for JConsole to function remotely.


3. Set the $ORACLE_HOME environment variable to directory path where JDeveloper is installed. Set the JAVA_HOME to directory path where you have installed JDK 6.


D:\Software\jdevstudio10134\jdk\bin>echo %ORACLE_HOME%
D:\Software\jdevstudio10134
D:\Software\jdevstudio10134\jdk\bin>echo %JAVA_HOME%
D:\Software\jdevstudio10134\jdk


4. Login to Solaris server where Oracle Application Server 10.1.3.4 is installed server and edit the opmn.xml file present in $ORACLE_HOME/opmn/conf. The modification is adding up the start parameter ‘-Dcom.sun.management.jmxremote=true’ to the oc4j container you want to monitor.eg:


ias-component id="SOA_GROUP" status="enabled"

process-type id="oc4j_soa" module-id="OC4J" status="enabled"
module-data
category id="start-parameters">
data id="java-options" value="-value="-Dcom.sun.management.jmxremote=true -server -d64 -XX:PermSize=512m -XX:MaxPermSize=512m -XX:SurvivorRatio
=12 -Xmn3584m -Xms5120M -Xmx5120M -Dstdstream.filesize=10 -Dstdstream.filenumber=10 -Doc4j.userThreads=true -DHTTPClient.socket.staleCheck=true -Djava.securi
ty.policy=$ORACLE_HOME/j2ee/oc4j_soa/config/java2.policy -


5.Restart the oc4j container for which you modified the start parameter using opmnctl command.


6.On your local system(Windows) open the command prompt and check if the ORACLE_HOME and JAVA_HOME env variables are set properly using echo command.Make sure the JDK home/bin directory path is added to PATH.


7. On the local command prompt issue the following to start the Jconsole(Diagram shown below):


jconsole -J-Dcom.sun.management.jmxremote.ssl=false -J-Dcom.sun.management.jmxremote.authenticate=true -J-Djmx.remote.protocol.provider.pkgs=oracle.oc4j.admin.jmx.remote -J-Djava.class.path=;%JAVA_HOME%\lib\jconsole.jar;%JAVA_HOME%\lib\tools.jar;%ORACLE_HOME%\j2ee\home\lib\adminclient.jar;%ORACLE_HOME%\j2ee\home\oc4jclient.jar

You would get the below Jconsole login screen.

Select Remote Process radio button and enter the following URL:

service:jmx:rmi:///opmn://(HOSTNAME):(REQUEST_PORT)/(CONTAINER_NAME)

(HOSTNAME):(REQUEST_PORT): can be obtained by issuing the following command on the server where OAS is installed: opmnctl status –port


(CONTAINER_NAME) is the container name for the container which you want to monitor eg oc4j_soa


Eg: service:jmx:rmi:///opmn://mytestserver.soumya.com:34010/oc4j_soa
8. Click on connect and you should find the below screen:

Your remote setup for Jconsole is complete. Before trying to connect to the remote oc4j JVM make sure the rmi ports are opened. You can test this by using telnet command on your local desktop command prompt.




Eg: telnet mytestserver.soumya.com 12042


(12042 is the RMI port for oc4j_SOA container for my setup,please check yours using opmnctl status -l command)




Your remote setup for Jconsole is complete.Compared to the local Jconsole setup(will publish this setup soon) the remote setup is  abit slow on performance.The client takes a bit of time before it can show you all graphs.But once data is loaded from the remote server the performance is good enough to monitor the remote JVM. One thing you can be sure about this setup is that it wont affect the server performance in case of production instances.Remote Jconsole setups are reccomended for prod setups.The local Jconsole setups are reccomended for the dev instances.


Hoping we had a great learning.Please do write to me in case of issues.


Happy Learning !!!

Now you can folow us on facebook and post your comments/views and questions for expert advise. Check this out facebook


Find us on facebook here


Regards,
Sanjay
Oracle Fusion Middleware Admin