IBM SOA Vs Oracle SOA

All,

Here is the presentation link from Oracle:

Find out why Oracle SOA Suite beats IBM patchwork of products...

http://event.on24.com/r.htm?e=291702&s=1&k=27017EA5AD3A3AAE02921EA4AAEF2D86&partnerref=fbfmw

You would need to register first...

Soumya...

JVM Tuning for Oracle SOA Suite 11g

Now you can folow us on facebook and post your comments/views and questions for expert advise. Check this out facebook


Find us on facebook here

Last year I installed couple of 11g SOA Suite instances and delivered it to our development team to work on it. The SOA servers were being used by around 10 developers to build their application. For a week the server seemed to be working fine but then started my nightmare. The servers started becoming unstable. 

Following were the issues:

1.       The servers became too slow, especially the EM console hence making it difficult for  developers to continue working.

2.       EM Login page got struck at the user authentication page.

3.       Data sources went into suspended state hence bringing EM application to halt.

4.       Out of Memory issues

This is where I did have a look into the JVM settings for the SOA application. Till now I was going with default JVM settings as provided by Oracle. I generated some gc logs and used a GC analyzer to view the GC info. That is where I could see frequent GCs with unarguably high pause time. I needed to tune my JVM settings for sure.


JVM tuning is never a ready made solution.Yo would need to use some JVM profiling tools like the JVisualVM. Test your JVM performance using this post Monitoring FMW11g JVM using JVisualVM

With release of the Oracle Fusion Middleware 11g products like SOA suite, BAM, OER, OSB etc a lot has changed the way these products are built and work.  Let’s focus on the SOA Suite. The 11g SOA Suite unlike the 10g now runs on Oracle Weblogic server. The SOA suite application now grows bigger with addition of applications like B2B, BAM etc. In the past releases (10g) B2B and BAM used to be separate installations .On top of this there are two management consoles, the Weblogic Admin Console and the Enterprise Manager FMW console which the product needs to function. Hence as you see the new 11g SOA suite is not only new but also a big. The Application Sever 9Weblogic) has to be tuned appropriately in order to ensure a healthy SOA instance.

Below is my environment info:

Application and Database Server hardware Info


The SOA Application and the database servers both were installed on separate physical boxes. The  specifications of the boxes are mentioned below.


Server Hardware: SUN T 5240                                                                                                                          Operating System: Solaris 10                                                                                                                        Architecture: Sun Sparc 64 Bit                                                                                                                   
Number of CPU: 10                                                                                                                                             Available Memory: 13.6 GB


Application Installed and Version:


Application Server: Oracle Weblogic Server (Version 10.3.4 )                                                                              
FMW Product: Oracle SOA Suite (Version 11.1.1.4)                                                                                                         
JVM Used: Sun JDK 1.6 Update 23  
(Latest Sun JDK ,this version boasts of performance boost on Solaris servers)


Application Install Architecture: Stand Alone Install.    

 

Below is my JVM setting recommendation. Please note, below tuning might be a good one to start with.  As number of concurrent users, deployed applications, load increase the tuning parameter below might change.     

SOA Suite 11g ideally uses two JVMs to function.

1.       Admin Server JVM: This is the weblogic  server (JVM) on which the Weblogic Admin Console and the EM Fusion Middleware Control are deployed. The Weblogic Admin Console is used to manage and control the weblogic resources. The EM Fusion Middleware control mainly is used to work on the SOA suite. It enables application deployment, application monitoring etc.

2.       SOA Managed Server:  This is the   weblogic server (JVM) on which the   entire SOA Suite and B2B product stack is deployed. Hence you can expect it to be a bit heavier than   the Admin Server.

Below is the JVM settings I recommend:


On how to change JVM startup parameters in SOA 11g please check my post here

JVM Heap Recommendations for Development Managed Servers

-server –d64 –Xss256k –Xms4g –Xmx4g –XX:NewRatio=2 -XX:+AggressiveOpts -XX:PermSize=1g -XX:MaxPermSize=1g -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:ParallelGCThreads=8 -XX:InitialSurvivorRatio=10 -XX:SurvivorRatio=10 -XX:LargePageSizeInBytes=4m -Dweblogic.management.discover=false -Dweblogic.StuckThreadMaxTime=900 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/java.hprof -verbose:gc -Xloggc:/tmp/gc.log -Xnoclassgc -XX:TargetSurvivorRatio=90 -XX:ReservedCodeCacheSize=64m -XX:CICompilerCount=8 -XX:+AlwaysPreTouch -XX:+PrintReferenceGC -XX:+ParallelRefProcEnabled -XX:-UseAdaptiveSizePolicy -XX:+PrintAdaptiveSizePolicy -XX:+DisableExplicitGC

JVM Heap Recommendations for Production Managed Servers

-server –d64 –Xss256k –Xms6g –Xmx8g –XX:NewRatio=2 -XX:+AggressiveOpts -XX:PermSize=2g -XX:MaxPermSize=2g -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:ParallelGCThreads=16 -XX:LargePageSizeInBytes=4m -XX:InitialSurvivorRatio=10 -XX:SurvivorRatio=10 –XX:-UseTLAB -Dweblogic.management.discover=false -Dweblogic.StuckThreadMaxTime=900 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/java.hprof -verbose:gc -Xloggc:/tmp/gc.log -Xnoclassgc -XX:TargetSurvivorRatio=90 -XX:ReservedCodeCacheSize=64m -XX:CICompilerCount=8 -XX:+AlwaysPreTouch -XX:+PrintReferenceGC -XX:+ParallelRefProcEnabled -XX:-UseAdaptiveSizePolicy -XX:+PrintAdaptiveSizePolicy -XX:+DisableExplicitGC

JVM Heap Recommendations for AdminServer

Modify the AdminServer JVM since it's running both the WebLogic Console administration application and the Enterprise Manager Fusion Application Control: 

-server –Xms2g –Xmx2g –XX:NewRatio=3 -XX:+AggressiveOpts -XX:PermSize=512m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:ParallelGCThreads=16 -XX:InitialSurvivorRatio=10 -XX:SurvivorRatio=10 -Dweblogic.StuckThreadMaxTime=900 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/java.hprof -verbose:gc -Xloggc:/tmp/gc.log -Xnoclassgc -XX:TargetSurvivorRatio=90 -XX:ReservedCodeCacheSize=64m -XX:CICompilerCount=8 -XX:+AlwaysPreTouch -XX:+PrintReferenceGC -XX:+ParallelRefProcEnabled -XX:-UseAdaptiveSizePolicy -XX:+PrintAdaptiveSizePolicy -XX:+DisableExplicitGC

Tuning Explanations 

The production recommendation differs from Development by the size of the heap.  Since the Production environment will be hosting significantly more traffic it will need the additional heap space to grow and handle those requests.  Tuning will be required in that there’s a possibility the Production SOA applications may need more or less heap space as required by the following factors:

-          Size of SOA interfaces deployed                                                                                                                                      -          Frequency of SOA interface usage or # of instances per minute                                                                                -          Length of time through which each instance executes

Tuning the environment should be not speculative but a measured one. Using Memory and GC analyzing tools like Oracle Enterprise Manager Grid Control in conjunction with Performance Load Testing activities, you will be able to tune your production environment adequately to prevent any load related outages.Further tuning may need to be monitored on the JVM for Garbage Collection times.  If the time it takes to do partial or Full GC increases significantly then increase the number of ParallelGCThreads.  By default the ParallelGCThreads is set to what’s available at the system level.  Example on a 2xUltraSPARC T2+ [T5240] = 128 which is too high and can cause heap fragmentation.

Since garbage collection in the Old Space or Tenured Space can be costly requiring more pause time and cpu time to complete a full gc, you may need to size up the New, Nursery, or Eden Space.  This is controlled with the NewRatio=n directive.  This sets the Eden Space to 1 / n + 1 size of the Max heap space.  If you find that majority of objects are short lived meaning the heap grows to a high end with heavy load but then returns to a lower level, then you may benefit from a larger Eden space.  This may require using a different directive than NewRatio.  You may need to size your Eden space to 50 – 60% of the total heap size.  Try –XX:NewSize=5g –XX:MaxNewSize=5g where –Xmx8g.

The default 64bit thread stack size is 1024m under SPARCv9.  When defining a 64bit model [-d64] be sure to size down the thread stack size which by default is too large; the 32bit model defaults to 512k on SPARCv9; the Linux x86-64 the Java 32bit model is 256k.  Some performance benchmarks on spec.org for WebLogic set the thread stack size to 128k on the Sun T series servers.  Having a high thread stack size can waste a significant amount of stack space [heap space].  Consider setting it to –Xss128k or –Xss256k to free up heap space and thereby reducing the overall max heap the application may need under load.

Thread local portions of the heap in the young generation is free space on the thread stack.  This can be used as a cache and can offer “excellent speedups on smaller numbers of threads (100s)”.  However, this can become a burden to the JVM costing more gc time when the number of threads are in the thousands.  On the Solaris SPARC platform the directive –XX:+UseTLAB is on by default.  When testing an application under heavy load using thousands of threads and experiencing excessive gc, consider turning off TLABs:  -XX:UseTLAB

When sizing the JVM heap or internal heaps ensure you set both the min and max to the same size.  This reduces latency while the JVM is trying to size up or down the heap spaces.  Example NewSize, MaxNewSize or PermSize, MaxPermSize

-XX:+HeapDumpOnOutOfMemoryError This directive creates a Heapdump in case of a out of memory error in the JMV. This would allow you to diagnose the root cause of thr memory leak.

-XX:HeapDumpPath=/tmp/java.hprof : This creates the Heapdump in the specified location.

-verbose:gc -Xloggc:/tmp/gc.log : This option allows you to specify the gc log file location.

Above tuning recommendations could be applied to other FMW products as well. Again the above recommendations should give you a descent start. With growing load and usage you may reconsider the tuning.

• If after fair amount of usage you see performance issues in the EM console (timeouts,long time takn by EM console to show composite instance details check my EM console tuning post SOA 11g EM Console Tuning
• About How to change JVM tuning parameters for SOA Admin/Managed server check my post Change SOA 11g JVM tuning/startup parameters
• JVM tuning is never a ready made solution. Test your JVM performance using this post Monitoring FMW11g JVM using JVisualVM
• And finally I have few FMW 11g interview questions FMW 11g Interview Questions
• 
  

Now you can folow us on facebook and post your comments/views and questions for expert advise. Check this out facebook


Find us on facebook here

Looking forward to your thoughts and comments.

Sanjay...
Conatact for Oracle FMW 11g/Weblogic Trainings and Freelance Engagements
sanjay.soa@gmail.com
Call@ +91-9646515972

Understanding Out of Memory (OOM) in Weblogic

As Fusion or Weblogic  Admin one is supposed to encounter Out of Memory (OOM )issues now and then. Why talk about Fusion or Weblogic admins this issue can be encountered on multiple products like the Oracle Applications EBS, Hyperion, OTM, Oracle Application server and many other products .What seems common across all these products is the use of Application server.  All the above mentioned products use some or the other application server over which the applications are deployed.

For example the latest Oracle SOA Suite 11g uses Weblogic 10.3 Application Server while EBS Release 12 uses Oracle Application Server 10g. The point I am trying to make is the Out of Memory issue is not product specific but is related to Application Servers running Oracle-Sun JVMs. Application servers like Oracle Weblogic, IBM Websphere and Oracle Application Server facilitate running Java Virtual Machines on which the Java Applications can be deployed and executed. Out of memory issue is related to filled up space issues in these JVMs which makes it generic across all the above mentioned application servers.

Before understanding OOM let’s first understand the memory used by the JVM. The Java Virtual Machine (JVM) has the following types of memory:

 Heap memory is the runtime data area from which memory for all class instances and arrays is allocated

Non-heap memory includes the method area and memory required for the internal processing or optimization of the JVM. It stores per-class structures such as a runtime constant pool, field and method data, and the code for methods and constructors.

Native memory is the virtual memory managed by the operating system.

When the above mentioned memory is insufficient for a deployed application, a java.lang.OutOfMemoryError is thrown. There will be different error messages for OutOfMemoryErrors in each type of memory. They are listed below:

-          Heap memory error. When an application creates a new object but the heap does not have sufficient space and cannot be expanded further, an OutOfMemoryError will be thrown with the following error message:

java.lang.OutOfMemoryError: Java heap space

-          Non-heap memory error:When the permanent generation is full, the application will fail to load a class or to allocate an interned string, and an OutOfMemoryError will be thrown with the following error message:

java.lang.OutOfMemoryError: PermGen space

-          Native memory error: The Java Native Interface (JNI) code or the native library of an application and the JVM implementation allocate memory from the native heap. An OutOfMemoryError will be thrown when an allocation in the native heap fails. For example, the following error message indicates insufficient swap space, which could be caused by a configuration issue in the operating system or by another process in the system that is consuming much of the memory:

java.lang.OutOfMemoryError: request bytes for .

Out of swap space?

After having understood the OOM issue the question is why would the code encounter an OOM at all. This could be for more than one reason.

An insufficient memory problem could be due

1.       Either to a problem with the configuration ie the application really needs that much memory

2.       Or to a performance problem in the application that requires you to profile and optimize to reduce the memory use. Configuring memory settings and profiling an application to reduce the memory use are beyond the scope of this post and I would cover them in my forth coming posts.

The issues are mentioned below:

a.       Memory Leaks

The JVM is responsible for automatic memory management, which reclaims the unused memory for the application. However, if an application keeps a reference to an object that it no longer needs, the object cannot be garbage collected and will occupy space in the heap until the object is removed. Such unintentional object retention is referred to as a memory leak. If the application leaks large amounts of memory, it will eventually run out of memory, and an OutOfMemoryError will be thrown. In addition, garbage collection may take place more frequently as the application attempts to free up space, thus causing the application to slow down.

b.      Finalizers

Another possible cause of an OutOfMemoryError is the excessive use of finalizers. The java.lang.Object class has a protected method called finalize. A class can override this finalize method to dispose of system resources or to perform cleanup before an object of that class is reclaimed by garbage collection. The finalize method that can be invoked for an object is called a finalizer of that object. There is no guarantee when a finalizer will be run or that it will be run at all. An object that has a finalizer will not be garbage collected until its finalizer is run. Thus, objects that are pending for finalization will retain memory even though the objects are no longer referenced by the application, and this could lead to a problem similar to a memory leak.

c.       Deadlocks

A deadlock occurs when two or more threads are each waiting for another to release a lock. The Java programming language uses monitors to synchronize threads. Each object is associated with a monitor, which can also be referred as an object monitor. If a thread invokes a synchronized method on an object, that object is locked. Another thread invoking a synchronized method on the same object will block until the lock is released. Besides the built-in synchronization support, the java.util.concurrent.locks package that was introduced in J2SE 5.0 provides a framework for locking and waiting for conditions. Deadlocks can involve object monitors as well as java.util.concurrent locks.

Typically, a deadlock causes the application or part of the application to become unresponsive. For example, if a thread responsible for the graphical user interface (GUI) update is deadlocked, the GUI application freezes and does not respond to any user action.

d.      Looping Threads

Looping threads can also cause an application to hang. When one or more threads are executing in an  infinite loop, that loop may consume all available CPU cycles and cause the rest of the application to be unresponsive.

e.      High Lock Contention

Synchronization is heavily used in multithreaded applications to ensure mutually exclusive access to a shared resource or to coordinate and complete tasks among multiple threads. For example, an application uses an object monitor to synchronize updates on a data structure. When two threads attempt to update the data structure at the same time, only one thread is able to acquire the object monitor and proceed to update the data structure. Meanwhile, the other thread blocks as it waits to enter the synchronized block until the first thread finishes its update and releases the object monitor. Contended synchronization impacts application performance and scalability.

Hope the above article helps you to understand the basics of OOM issues in Application Servers. OOM issues could be a night mare for any admin given the cause is memory leak. But we will discuss methods to resolve such scenarios soon. Also I plan to post on ways to alter JVM settings on Weblogic server.

 Below are URLs to guides/documents that has been used by me for compilation of above article:

Monitoring and Managing Java SE 6 Platform Applications (http://java.sun.com/developer/technicalArticles/J2SE/monitoring/)

Now you can folow us on facebook and post your comments/views and questions for expert advise. Check this out facebook


Find us on facebook here

Looking forward to your comments. Please let me know if the post was useful.

Understanding Data Sources and Connection Pool in Weblogic

Terms like datasources and connection pools are very familiar to weblogic developers and admins. Most of the times developer ask for datasource creation on weblogic servers. Developers  or admins who have worked on Oracle Application server also will be familiar with these terms. Newbies often get confused about these and the difference between the two.


In simple terms Applications deployed on weblogic servers(managed servers) use the datasources( created on Weblogic server level) to connect to the databases.


Though the connection can be made at the deployed code level(JDBC connect strings) Data sources and their connection pools provide connection management processes that help keep your system running and performant.You can set options in the data source to suit your applications and your environment. 


I explain the concepts in more details below:

Data Sources


In WebLogic Server, you configure database connectivity by adding data sources to your WebLogic domain. WebLogic JDBC data sources provide database access and database connection management. Each data source contains a pool of database connections that are created when the data source is created and at server startup. Applications reserve a database connection from the data source by looking up the data source on the JNDI tree or in the local application context and then calling getConnection(). When finished with the connection, the application should call connection.close() as early as possible, which returns the database connection to the pool for other applications to use.


Connection Pool


Each JDBC data source has a pool of JDBC connections that are created when the data source is deployed or at server startup. Applications use a connection from the pool then return it when finished using the connection. Connection pooling enhances performance of the application server or deployed application response time by eliminating the costly task of creating database connections for the application.


Hope the above post helps. In future posts I would write on how to create connection pools in Weblogic and more importantly tuning them.

Gathering Solaris system information

As a fusion admin/architect most of the times one needs to gather basic information about the server on which the fusion application runs. Below are some solaris commands that can be used by an admin for gathering system information on a Solaris machine.The below commands work well on Solaris 7 and above. The same commands can be used on solaris zones(virtualized servers).


Processor Info


The psrinfo utility displays processor information. When run in verbose mode, it lists the speed of each processor and when the processor was last placed on-line (generally the time the system was started unless it was manually taken off-line).


/usr/sbin/psrinfo -v
Status of processor 1 as of: 12/12/02 09:25:50
  Processor has been on-line since 11/17/02 21:10:09.
  The sparcv9 processor operates at 400 MHz,
        and has a sparcv9 floating point processor.
Status of processor 3 as of: 12/12/02 09:25:50
  Processor has been on-line since 11/17/02 21:10:11.
  The sparcv9 processor operates at 400 MHz,
        and has a sparcv9 floating point processor.


The psradm utility can enable or disable a specific processor.


To disable a processor:
/usr/sbin/psradm -f processor_id


To enable a processor:
/usr/sbin/psradm -n processor_id


The psrinfo utility will display the processor_id when run in either standard or verbose mode.


RAM Info


The prtconf utility will display the system configuration, including the amount of physical memory.


To display the amount of RAM:


/usr/sbin/prtconf | grep Memory
Memory size: 3072 Megabytes


Disk space Info


Although there are several ways you could gather this information, the following command lists the amount of kilobytes in use versus total kilobytes available in local file systems stored on physical disks. The command does not include disk space usage from the /proc virtual file system, the floppy disk, or swap space.


df -lk | egrep -v "Filesystem|/proc|/dev/fd|swap" | awk '{ total_kbytes += $2 } { used_kbytes += $3 } END { printf "%d of %d kilobytes in use.\n", used_kbytes, total_kbytes }'
19221758 of 135949755 kilobytes in use.


You may want to convert the output to megabytes or gigabytes and display the statistics as a percentage of utilization.


The above command will list file system usage. If you are interested in listing physical disks (some of which may not be allocated to a file system), use the format command as the root user, or the iostat -En command as a non-privileged user.


Processor and kernel bits Info
If you are running Solaris 2.6 or earlier, you are running a 32-bit kernel.


Determine bits of processor:
isainfo -bv


Determine bits of Solaris kernel:
isainfo -kv


Hope the above info helps. I would be blogging more on unix level commands/scripts which would aid you manage and troubleshoot fusion instances.Looking forward to your valuable suggestions...