Tuesday, August 23, 2016

Faster Weblogic Server Startup on Linux VM


Problem


Weblogic server running on a Linux Virtual Machine is stuck while startup. For instance a Oracle SOA 12.2.1 weblogic manage server may take upto 12 minutes to startup on a Linux VM. A Weblogic 12C manage Server running nothing may take upto 4 minutes to start. The virtual machines are all equipped with enough CPUs and memory. 


Applies To 


Weblogic 12.1.x,12.2.x
RHEL X86,X86_64 Virtual Machines


Cause


Linux has two devices to provide random data at any time: /dev/random and /dev/urandom. Both ways should be secure enough to use them in generating PGP keys, ssh challenges, and other applications where secure random numbers are required. Starting on kernel 2.6, default entropy is 4096 bits and problem arises when the entropy available on the system is minimum (around 100 bits or less).


How to verify if you are encountering this issue?


1. Check the default system entropy

$ cat /proc/sys/kernel/random/poolsize 
4096

2. Check the available entropy.

$ cat /proc/sys/kernel/random/entropy_avail 
160

3. On previous example, entropy is too low.

Monitor the current entropy of the system by using the following command:


$ for i in $(seq 500); do cat /proc/sys/kernel/random/entropy_avail ; sleep 5; done

4. Start a WebLogic server instance. You should see that entropy decreases or stalls (use script in step 3)

Solution


1. Temporary Solution (Use for testing purpose)

Start the WLS Server with below startup arguments.

-Djava.security.egd=file:/dev/./urandom

Override the JAVA_OPTIONS environment variable before starting WebLogic Server via shell scripts.

export JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.security.egd=file:/dev/./urandom"

Start the Weblogic Server and note the timings!


2. Permanent Solution (Use if Step 1 works)

If the above solution works it is time to setup the fix permanently in the env. The fix can be applied in the JAVA_HOME the Weblogic server refers to.


i.   Edit the Java Security Properties file ($JAVA_HOME/jre/lib/security/java.security)

ii.  The securerandom.source property specifies the source of seed data for secure random.

Change
securerandom.source=file:/dev/random

To

securerandom.source=file:/dev/urandom

iii.  Save changes and start the WebLogic Server instances.

Observation


We could see the startup tiings for Weblogic server improved dramatically. For instance the SOA Manage Server now took only less than 3 minutes as against 12 minutes before the fix was applied! The bare Weblogic manage servers took less tha 20 seconds to start !



References


How to Diagnose a Linux Entropy Issue on WebLogic Server Instances (Doc ID 1574979.1)

Monday, August 15, 2016

Starting OHS 12c via startComponent takes several minutes on a Linux Virtual Machine - SOLVED


I recently started working with Oracle HTTP server on a Linux VM. On trying to start the OHS using the startComponent script I was surprised to observe the startup duration was over 8 minutes! The same took seconds in 11g version of the same software. After few research on metalink I could solve the issue. I am providing the solution below.

Issue Applies To

The issue described and solution offered applied to below Oracle FMW Components.

Oracle HTTP Server 12.2.1 Installed on RHEL 6 Virtual Machine. The same issue/solution applies to OHS 12.1. and later.

Issue Description

Starting from OHS 12C, the OHSservices are monitored/managed by Node Manager. Once the Node manager is started ,the OHS service is started using startComponent script.While trying to do the script gets stuck  for minutes (8 Minutes in my case!)

$DOMAIN_HOME/bin/startComponent.sh ohs1

    Starting system Component ohs1 ...

    Initializing WebLogic Scripting Tool (WLST) ...

    Welcome to WebLogic Server Administration Scripting Shell

    Type help() for help on available commands

    Reading domain from
     Here it just sits for several minutes 
     
    Connecting to Node Manager ...
    Successfully Connected to Node Manager.
    Starting server ohs1 ...

Cause 

The problem is due to random number generation (entropy) on the Linux VM. More on the entropy issues can be found in Metalink note NOTE:1574979.1.The vm was running out of entropy. After changing where Java got its random numbers from, the startup time came down manifold!

Solution 

1) Stop OHS
stopComponent.sh ohs1

2) Backup and edit java.security
$ORACLE_HOME/oracle_common/jdk/jre/lib/security/java.security

3) Change securerandom.source
From:
securerandom.source=file:/dev/urandom
(In 12.2.1 this is securerandom.source=file:/dev/random)
To:
securerandom.source=file:/dev/./urandom

4) Start OHS
startComponent.sh ohs1

Observation

Hurray! The startup time for OHS now takes close to 25 seconds compared to 8 minutes earlier!

References

NOTE:2006106.1 - Starting OHS 12c via startComponent takes several minutes on a Virtual Machine

NOTE:1574979.1 - How to Diagnose a Linux Entropy Issue on WebLogic Server Instances


Saturday, September 6, 2014

URL Redirection on Oracle HTTP Server

An application is supposed to be decommissioned now and a new Application needs to replace it. The new application has a new/different URL. However the business would want the old applications users to face less issues. The business people would like the old application users to be redirected to new Application URL on hitting the old Application URL automatically!


How can this be done? Well this involves a small change in the OHS level. Read below to know :

Old Application URL:
https://learning.oraclefusionfacts.com/welcome/name.jspx

Redirected to New Application URL:
https://focusonfusion.oraclefusionfacts.com/app/name.jspx

On the OHS server where we have the configuration for our old application the HTTP server Virtual Server entry look like this: You will find the below entry in the HTTPD.CONF file!

# Start Virtual Server Settings

NameVirtualHost *:7777

    ServerName https://learning.oraclefusionfacts.com:443
    ServerAdmin siddharth.mishra@oraclefusionfacts.in
    RewriteEngine On
    RewriteOptions inherit

 
    SetHandler weblogic-handler
    WebLogicCluster app.oraclefusionfacts.in:8001
 



# End Virtual Server Settings

In order to REDIRECT the request coming to the Old URL to our new URL make the following changes to the Virtual Host entry on the HTTPD.CONF file as shown below:

# Start Virtual Server Settings

NameVirtualHost *:7777

    ServerName https://learning.oraclefusionfacts.com:443
    Redirect 301 /welcome https://focusonfusion.oraclefusionfacts.com/app/name.jspx
    ServerAdmin siddharth.mishra@oraclefusionfacts.in
    RewriteEngine On
    RewriteOptions inherit

 
    SetHandler weblogic-handler
    WebLogicCluster app.oraclefusionfacts.in:8001
 



# End Virtual Server Settings

After making this change make sure you restart the OHS server using opmnctl HTTP utility!

Now open a new browser session. Make sure you clear the cache for the browser session. in case you do not do it the old URL will not redirect to the new URL! Once done now key in the old URL and Hit Enter. The old URL must automatically change to the new URL !

So go ahead and make your Boss proud !!! Do not forget to mention in comments if this helped you :)

Commands to find out Allocated CPU/Memory on a Solaris Zone

With times changing servers have changed. With the advent of virtualization we have see bigger servers being virtualized into smaller servers and provided to admins for use. Have you heard about Solaris T2/T3/T4 or the newer ones like T4-4 machines. All of those machines have found out acceptance these days in organizations these days. These are mostly CMT servers which offer huge resources in terms of Memory and CPU. The UNIX Admins generally slice and dice these servers into smaller servers known as Zones and allocate certain amount of Memory.CPU to each such zone created out of the master box.

For Admins it might get a bit tricky to find out the Memory/CPU allocated on such zones. I am providing below commands which might help find out the resources assigned to Solaris zones!

For Finding CPU Shares Assigned use below command:

bash-3.00$ prctl -n zone.cpu-shares $$
process: 4352: bash
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
zone.cpu-shares
        privileged         20       -   none                                 -
        system          65.5K     max   none                                 -


For Finding Memory Assigned to the Solaris Zone use below:

bash-3.00$
bash-3.00$ prctl -n zone.max-shm-memory $$
process: 4352: bash
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
zone.max-shm-memory
        privileged      2.73GB      -   deny                                 -
        system          16.0EB    max   deny                                 -

In case you want to be more descriptive with resources assigned to your zone use below command:

prctl $$

Now if this post helped you please say a Hi on the Comments Section Folks !!! Hope it helped :)

Sunday, March 11, 2012

ORA-01438:value larger than specified precision allowed error in SOA 11g logs

Issue:
SQLDataException: ORA-01438 Error Frequently in SOA Log Files (the SOA managed server out and diognostic log file)


Applies to:
Oracle SOA Suite 11.1.1.3.0 and later (11g R1)


What you see in logs:

Error while invoking bean "cube delivery": Exception not handled by the Collaxa Cube system.[[
an unhandled exception has been thrown in the Collaxa Cube systemr; exception reported is: "ORABPEL-00000

Exception not handled by the Collaxa Cube system.
an unhandled exception has been thrown in the Collaxa Cube systemr; exception reported is: "java.sql.SQLDataException: ORA-01438: value larger than specified precision allowed for this column

at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:83)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:135)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:210)

Effects:

Degraded performance of SOA Composites. Server spends additional processing power on logging errors, so performance of server is impacted. Also lots of log files are generated which might fill your disk system fast.

Cause:

This is an issue which has been identified by Oracle. The bug 12621337: ORA-01438 ON LIVE_INSTANCES COLUMN

The problem is with the precision of the column COMPOSITE_INSTANCE.LIVE_INSTANCES currently defined to be NUMBER(3) which can at the most hold a value up to 999. As there is no explicit limitation on this column, the correct way to handle this is to increase the precision of this column to accommodate more instances.


Solution:
The workaround is to modify the COMPOSITE_INSTANCE.LIVE_INSTANCES column's precision to be NUMBER(38) in the SOAINFRA schema.
To apply this workaround, follow the steps below:

1. Stop the SOA domain
2. Log in to the SOA infra repository database as SYSDBA
3. Modify the COMPOSITE_INSTANCE table as shown below

ALTER TABLE 
soa11g_soainfra.composite_instance 
MODIFY (live_instances NUMBER(38))
;


In the above sql statement I have assumed the dehydration prefix is soa11g. Please modify the statement according to your prefix.Restart the SOA domain. Check the logs.



Thursday, March 1, 2012

SOA 11g managed Server takes an hour to start on Oracle T Series machine

Recently I had some one asking me, that their SOA managed server took an hour to start. The instance in question had some 200 plus composites in it. On questioning about the platform they were on I came to know that they had the below configuration:

WLS=10.3.4
SOA Suite 11.1.1.4

Operating System: Solaris 10
(SunOS endenapp04 5.10 Generic_144488-07 sun4v sparc SUNW,SPARC-Enterprise-T5120)
Hardware: SUN SPARC T5120


Bingo !!! I got it right,

The machine they were on(T5120 or the T1 SUN Server) is a pathetic one and if your DB tier is on it it makes your situation even worse. I have worked on such machines and the experience are better forgotten. They are called The T Series machines belong to the CMT family of servers, which boast of supporting multi threaded applications well. Unfortunately single threaded applications like the DB run very poorly on them. For example on machines like the T5120 or T5240 running the RCU may take close to an hour, on M5000 it takes 10 minutes. With so many composites (remember the composites are loaded from Dehydrarion ie DB) your application will definitely take an hour or few minutes less than that to start

For tuning SOA 11g on T5140 check my post on my blog:
http://oraclefusionfacts.blogspot.in/2011/03/jvm-tuning-for-oracle-soa-suite-11g.html

Don't expect to fly like rocket after you follow my post since you cannot win a F1 race when you are driving a cart. You may get a better performance some how.

See below note from Oracle which will give you enough proof of what i am talking about:
Weblogic: Performance Problems After Migrating From WLS 8.1 To WLS 10.3 And Moving To SUN Sparc T5240 With CMT Chips [ID 860459.1]
Poor Performance of SOA/OIM/WL/EM on Solaris 10 - SPARC T3 Server [ID 1361701.1]

Solution:

Please propose a M5000 box for the Database Tier or upgrade the App and DB Tier to the newer version of T series ie T4. Oracle guarantees manifold performance improvements on T4 as compared to T1/T2/T3s.

Thursday, February 23, 2012

Oracle FMW 11.1.1.6 (Patch Set 5) Released

All,

This should be a good news to many, Oracle released its latest version of Oracle FMW 11g family.


The complete release of Oracle Fusion Middleware 11R1 (11.1.1.6.x). includes the following products:
  • Oracle WebLogic Server 11g R1 (10.3.6)
  • Oracle SOA Suite 11g R1 (11.1.1.6.0)
  • Oracle Business Process Management 11g R1 (11.1.1.6.0)
  • Oracle Complex Event Processing 11g R1 (11.1.1.6.0)
  • Oracle Service Bus 11g R1 (11.1.1.6.0)
  • Oracle Enterprise Repository 11g R1 (11.1.1.6.0)
  • Oracle Service Registry 11g R1 (11.1.1.6)
  • Oracle Application Integration Architecture Foundation Pack 11g R1 (11.1.1.6.0)
  • Oracle Data Integrator 11g R1 (11.1.1.6.0)
  • Oracle WebCenter Portal 11g R1 (11.1.1.6.0)  (previously known as Oracle WebCenter Suite)
  • Oracle WebCenter Content 11g R1 (11.1.1.6.0) (previously known as Oracle Enterprise Content Management)
  • Oracle Identity Management 11g R1 (11.1.1.6.0)
  • Oracle Forms, Reports, Portal & Discoverer 11g R1 (11.1.1.6.0)
  • Oracle Traffic Director 11g R1 (11.1.1.6)
  • Oracle Repository Creation Utility 11g R1 (11.1.1.6.0)
  • Oracle JDeveloper & Application Development Framework 11g R1 (11.1.1.6.0)
The 11.1.1.6 documentation is available here

To download the new release from OTN visit here.

I would soon try and post on some bug fixes we have on WLS/SOA Suite. For now you can go through the Product release document here