Common issues in weblogic

Common issues in weblogic

1.Issue in Wls while starting the services:

Unable to get file lock, will retry;
Unable to get file lock, will retry;
Unable to get file lock, will retry;
Unable to get file lock, will retry;

Solution:

Kill the process in background using kill -9 command.


2.High CPU Utilization Finding Cause?

1)jps: to know How to use JPS utility
$ jps -v

2)top: This is an Operating System Utility
$ top -H -b -p 5398

3)jstack: utility
$ jstack -l (pid) >> ThreadDumps.log

Note: The jps and stack are stored in JAVA_HOME/bin.

JDBC Tips

JDBC Tips

JDBC related issues in your WebLogic Environment the Please Keep the following tips always in mind.

1) Set the Maximum Capacity of the connection pool at least equal to the Execute Thread Count.

2) Configure Inactive Connection Timeout to specify how long (in seconds) a connection can be inactive before it is reclaimed to the pool.

3) The connection leak profiling option shows leaked connections from the connection pool. BEA recommends that you do not use this option in production; it uses extra resources and typically slows connectionpool operations.

4) If possible, size database connection pools so that they never increase the number of connections: set Initial Capacity equal to Maximum Capacity.

5) Use Test Reserved Connections only if you can afford the overhead of testing connections as part of normal request processing.

6) Avoid using a production table for “Test Table Name”, use a dummy table (ex. dual).

7) Use the statement cache to improve performance of prepared and callable statements.

8) Select the least-recently-used (LRU) algorithm for the cache; this removes rarely-used statements from the cache.

9) Connection Creation Retry Frequency can be used to retry for establishing connections to the database, if database is not reachable when creating connection pool or starting WebLogic Server.

10) If database is restarted when WebLogic Server is running, Test Frequency can be increased from 0, so that all the connections are closed and reopened to re-establish valid physical connection. After all the connections are recreated changing it back to 0 will disable the testing.

11) When using DataSource objects for a connection pool, use the Honors Global Transaction option to create a TxDataSource.

12) The only time you should use a non-Tx Data Source is when you want to do some work on the database that you do not want to include in the current transaction.

13) When configuring a connection pool to use with WebLogic JMS JDBC Store, use non-XA database drivers.

OutOfMemory Causes In Weblogic Server

OutOfMemory Causes In Weblogic Server

What is OutOfMemory ?
An OutOfMemory is a condition in which there is not enough space left for allocating required space for the new objects or libraries or native codes. OutOfMemory can be divided in categories:

1) OutOfMemory in Java Heap
2) Native OutOfMemory
3) OutOfMemory in PermGen Space

1) OutOfMemory in Java Heap ?

This happens when the JVM is not able to allocate the required memory space for a Java Object. There may be many reasons behind.

a)Very Less Heap Size allocation. Means setting the MaxHeapSize (-Xmx) parameter to a very less value.
b)The Leaking of Objects. Either the Application is not unreferencing the unused Objects or the Third part frameworks (Hibernate/Spring/Seam…etc) might not be releasing the references of the objects due to some inaccurate configurations.
c)In Many cases it may be the reason that Application codes are getting the JDBC connections objects from the DataSource are not being released back to the Connection Pool.
d)Garbage Collection strategy may be in correct according to the environmental/application requirements.
e)In-accurate setting of Application/Frameworks Cache.

Example:
Exception in thread "Thread-10" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(Abs tractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStr ingBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java: 119)
at java.lang.Throwable.toString(Throwable.java:344)

What to do in case of OutOfMemory In JavaHeap?
Whenever we see an OutOfMemory in the server log or in the stdout of the server. We must try to do the following things as first aid steps:

a)If possible enable the following JAVA_OPTIONS in the server start Scripts to get the informations of the Garbage Collection status.
-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails  -Xloggc:/opt/app/GCLogsDirectory/gc.log
b)It is always needed to see what all objects were present when the OutOfMemory error occured to identify whether those objects belongs to the Application Code/ Application Framework Codes/ The Application Server APIs. Sothat we can isolate the issue. In order to get the details of the Heap Objects collect “HeapDump” either using JHat (not a better tool) or JMap (Much Better compared to the Jhat tool).
c)Once we collected the Heap Dump we can easily monitor the Heap Details using best GUI toold like “Jhat Web Browser” or using “Eclipse Memory Analyzer”.

2) Native OutOfMemory ?

Native OutOfMemory is a scenario when the JVM is not able to allocate the required Native Libraries and JNI Codes in the memory.
Native Memory is an area which is usually used by the JVM for it’s internal operations and to execute the JNI codes. The JVM Uses Native Memory for Code Optimization and for loading the classes and libraries along with the intermediate code generation.
The Size of the Native Memory depends on the Architecture of the Operating System and the amount of memory which is already commited to the Java Heap. Native memory is an Process Area where the JNI codes gets loaded or JVM Libraries gets loaded or the native Performance packs and the Proxy Modules gets loaded…
Native OutOfMemory can happen due to the following main reasons:

a) Setting very small StackSize (-Xss). StackSize is a memory area which is allocated to individual threads where they can place their thread local objects/variables.
b) Usually it may be seen because of Tuxedos incorrect setting. WebLogic Tuxedo Connectors allows the interoperability between the Java Applications deployed on WebLogic Server and the Native Services deployed on Tuxedo Servers. Because Tuxedos uses JNI code intensively.
c) Less RAM or Swap Space.
d) Usually it may occur is our Application is using a very large number of JSPs in our application. The JSPs need to be converted into the Java Code and then need to be compiled. Which reqires DTD and Custom Tag Library resolution as well. Which usually consumes more native memory.

Example:
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at TestXss.main(TestXss.java:18)

What to do in case of Native OutOfMemory?
a) Usually Native OutOfMemory causes Server/JVM Crash. So it is always recommended to apply the following JAVA_OPTIONS flags in the Server Start Script to instruct the JVM to generate the HeapDump  “-XX:+HeapDumpOnOutOfMemoryError“
By default the heap dump is created in a file called java_pidpid.hprof in the working directory of the VM, as in the example above. You can specify an alternative file name or directory with the “-XX:HeapDumpPath=C:/someLocation/“

Note: Above Flags are also suitable to collect HeapDump in case of JavaHeap OutOfMemory as well. But these flags never gurantees that the JVM will always generate the Heap Dump in case of any OutOfMemory Situation.

b) Usually in case of Native OutOfMemory a “hs_err_pid.log” file is created in case of Sun JDK and “xxxx.dump” file is created in case of JRockit JDK. These log files are usually Text Files and tells about the Libraries which caused the Crash. These files need to be collected and analyzed to find out the root cause.
c) Make Sure that the -XX:MaxHeapSize is not set to a Very Large Space…because it will cause a very less Native Space allocation. Because as soon as we increase the HeapSize, the Native Area decreases.
d) Keep Monitoring the process’s memory using the Unix utility ‘ps’ like following:
ps -p <PID> -o vsz
Here you need to pass the WebLogic Server’s PID (Process ID) to get it’s Threading Details with respect to the Virtual Memory Space.
e) If the Heap Usages is less Or if you see that Your Application usages less Heap Memory then it is always better to reduls the MaxHeapSize so that the Native Area will automatically gets increased.
f) Sometimes the JVMs code optimization causes Native OutOfMemory or the Crash…So in this case we can disable the Code Optimization feature of JVM.
(Note: disabling the Code Optimization of JVM will decrease the Performance of JVM)
For JRockit JVM Code Optimization can be disabled using JAVA_OPTION  –Xnoopt
For Sun JDK Code Optimization can be disabled using   JAVA_OPTION  -Xint

3) OutOfMemory in PermGen Space

Permanent Generation is a Non-Heap Memory Area inside the JVM Space. Manytimes we see OutOfMemory in this Area. PermGen Area is NOT present in JRockit JVMs.
The PermGen Area is measured independently from the other generations because this is the place where the JVM allocates Classes, Class Structures, Methods and Reflection Objects. PermGen is a Non-Heap Area.It means we DO NOT count the PermGen Area as part of Java Heap.
The OutOfMemory in PermGen Area can be seen because of the following main reasons:

a) Deploying and Redeploying a very Large Application which has many Classes inside it.
b) If an Application is getting deployed/Updated/redeployed repeatedly using the Auto Deployment feature of the Containers. In that case the Classes belonging to the application stays un cleaned and remains in the PermGen Area without Class Garbage Collection.
c) If  “-noclassgc” Java Option is added while starting the Server. In that case the Classes instances which are not required will not be Garbage collected.
d) Very Less Space for allocated the “=XX:MaxPermGen”

Example: you can see following kind of Trace in the Server/Stdout Logs:
<Notice> <Security> <BEA-090171> <Loading the identity certificate and private key stored under the alias DemoIdentity from the jks keystore file D:ORACLEMIDDLE~1WLSERV~1.3serverlibDemoIdentity.jks.>
Exception in thread "[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

What to do in case of OutOfMemory In PermGen?
a) Make Sure that the PermGen Area is not set to a very less value.
b) Usually if an Application has Many JSP Pages in that case every JSP will be converted to a *.class file before JSP Request Process. So a large number of JSPs causes generation of a Large number of *.class files all these classes gets loaded in the PermGen area.
c) There is no standard formula to say which value of MaxPermSize will suit your requirement. This is because it completely depends on the kind of framework,APIs, number of JSPs…etc you are using in your application. The number of class which has to be loaded will vary based on that. but if you want to really tune the MaxPermSize then you should first start with some base value like 512M or 256M and then If you still get the OutOfMemory then please follow below instruction to troubleshoot it.
d) If you are repeatedly getting the OutOfMemory in PermGen space then it could be a Classloader leak….
May be some of the classes are not being unloaded from the permgen area of JVM . So please try to increase the -XX:MaxPermSize=512M  or little more and see if it goes away.
If not then add the following JAVA_OPTIONS to trace the classloading and unloading to find out the root cause :
-XX:+TraceClassloading and -XX:+TraceClassUnloading
e)If users want to investigate which kind of classes are consuming more PermGen space then we can use the “$JAVA_HOME/bin/jmap” utility as following:
$JAVA_HOME/bin/jmap -permstat $PID  >& permstat.out
Above utility will dump the list of classes loaded in that JVM process Process (we are passing the processID to this command as $PID). This helps us in understanding if there is any classloader leak or if a particular class is consuming more memory in PermGen…etc Collecting HeapDump also gives a good idea on this.

Troubleshooting Application Slowness Issue In Weblogic

Troubleshooting Application Slowness Issue In Weblogic

Linux performance related commands

top - to check cpu and memory usage

top - 03:32:44 up 1 day, 23:09,  6 users,  load average: 0.00, 0.05, 0.32
Tasks: 157 total,   1 running, 156 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.1%us,  0.3%sy,  0.0%ni, 97.7%id,  0.8%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   1011508k total,  1001500k used,    10008k free,     3528k buffers
Swap:  2097148k total,   333960k used,  1763188k free,    69648k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
10 root      20   0     0    0    0 S  3.7  0.0   0:19.80 rcu_sched
13415 oracle    20   0 1286m 293m 4116 S  1.8 29.7   0:29.02 java
1 root      20   0 19408  376  200 S  0.0  0.0   0:04.57 init

System activity report
sar -r 2 5
sar -r 2

[oracle@OEL6 ~]$ sar -r 2 2
Linux 3.8.13-98.2.2.el6uek.x86_64 (OEL6)        02/10/2016      _x86_64_        (1 CPU)

03:33:57 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
03:33:59 AM     11020   1000488     98.91      2692     69448   2032144     65.37
03:34:01 AM     11020   1000488     98.91      2692     69448   2032144     65.37
Average:        11020   1000488     98.91      2692     69448   2032144     65.37

Input and output statistics

iostat 2 5
iostat 2

[oracle@OEL6 ~]$ iostat 2 1
Linux 3.8.13-98.2.2.el6uek.x86_64 (OEL6)        02/10/2016      _x86_64_        (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          1.12    0.00    0.37    0.85    0.00   97.66

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
scd0              0.00         0.00         0.00        544          0
sda               4.91       160.92        43.61   27073268    7336356
fd0               0.00         0.00         0.00         16          0

[oracle@OEL6 ~]$ free
             total       used       free     shared    buffers     cached
Mem:       1011508     999864      11644       2328       3064      60936
-/+ buffers/cache:     935864      75644
Swap:      2097148     334344    1762804

df -h - Disk usage

[oracle@OEL6 ~]$ df -h /home/oracle/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        57G   23G   32G  42% /

jmap -heap will display the snap shot of heap usage

[oracle@OEL6 ~]$ /home/oracle/jdk1.7.0_21/bin/jmap -heap 12659
Attaching to process ID 12659, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 23.21-b01

using thread-local object allocation.
Mark Sweep Compact GC

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 268435456 (256.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 21757952 (20.75MB)
   MaxPermSize      = 268435456 (256.0MB)
   G1HeapRegionSize = 0 (0.0MB)

Heap Usage:
New Generation (Eden + 1 Survivor Space):
   capacity = 80543744 (76.8125MB)
   used     = 75933440 (72.415771484375MB)
   free     = 4610304 (4.396728515625MB)
   94.27602471521563% used
Eden Space:
   capacity = 71630848 (68.3125MB)
   used     = 71377800 (68.07117462158203MB)
   free     = 253048 (0.24132537841796875MB)
   99.6467332063415% used
From Space:
   capacity = 8912896 (8.5MB)
   used     = 4555640 (4.344596862792969MB)
   free     = 4357256 (4.155403137207031MB)
   51.112904268152576% used
To Space:
   capacity = 8912896 (8.5MB)
   used     = 0 (0.0MB)
   free     = 8912896 (8.5MB)
   0.0% used
tenured generation:
   capacity = 178978816 (170.6875MB)
   used     = 101403672 (96.7060775756836MB)
   free     = 77575144 (73.9814224243164MB)
   56.65680121607241% used
Perm Generation:
   capacity = 130940928 (124.875MB)
   used     = 130640608 (124.58859252929688MB)
   free     = 300320 (0.286407470703125MB)
   99.7706446681056% used

69269 interned Strings occupying 8985680 bytes.

Validation

Welogic Server check
Check for the weblogic server status
Number of free vs available threads and stuck threads
Timeouts in logs ( Application logs and server logs)
Check java Heap memory size ( Free memory)

Unix server Check
Check I/O statistics -   iostat
Check the memory,swap Bottleneck - free
To check top process CPU utilizations -top
Check the disk space - df -h

DB Server check
Check for hung sessions
Check for long running queries
Check for CPU usage – If beyond a certain threshold need to take an action
Stale Stats

Weblogic Troubleshooting Issues

Weblogic Troubleshooting Issues

Deployment issues:

code issues: we will send error log to the application team for modification.
Caused By: weblogic.utils.ErrorCollectionException:
There are 1 nested errors:
weblogic.j2ee.dd.xml.AnnotationProcessException: Duplicate ejb name 'BDAccountEjbBean' found: annotation 'Stateless' on bean
failed due to connection pool issue: we will fix connection pool issues and then redeploy the application

Out of memory issue during the deployment:
error: java.lang.outofmemory.permgenspace
this error occured due to space in perm area.
setDomainEnv.sh
xx:permsize 64m
xx:maxpermsize 128m
we have set intial permsize=maxpermsize then restarted the servers, redeployed the application
If one or two application faile
d when we are triggering through scipt. we will fix that issue and do a deployment using console

jdbc issues:

1) DB down (raise a ticket to db team)
2) Incorrect hostname or port number ( raise a ticket to network team)
3) Data base connection lost ( telnet ipaddress port )
4) Data base user_acc lock ( raise a ticket to db team for unlocking user_acc)
5) Invalid pakage error (raise a ticket to db team)
6) TNS listener error (raise a ticket to db team)
7) Schema does not exist (raise a ticket to db team)
8) Cannot allocate resource error
Intial capacity : 5
max      : 15 increase max to 25
9) connection leaks ( send error to application team)
10) Connection time out ( raise a tickect to db team for long running quries)

jms issues:

stuck message issues
Check whether dest queue is available, check message format, check queue name.
rolling message issues (messages will run continuously in the loop)
delete those messages in the queue.

diskspace issues:

If the disk space usage is 95%-100% then we will delete old log files
[root@localhost ~]# df -kh
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             3.8G  1.9G  1.8G  52% /
/dev/sda1              46M  9.2M   35M  22% /boot
tmpfs                 506M     0  506M   0% /dev/shm
/dev/sda3              14G  14G    0G  100% /home

du -kh (disk usage)

[root@localhost ~]# du -sh /home
1.8G    /home

[root@localhost bea10.3]# du -sh *
181M    jdk160_05
211M    jrockit_160_05
28K     logs
100M    modules
24K     registry.dat
8.0K    registry.xml
19M     user_projects
556K    utils
429M    wlserver_10.3

delete old log files
/home/bea10.3/user_projects/domains/sherkhan/servers/AdminServer/logs

rm -rf Adminserver.log00001  Adminserver.log00002 Adminserver.log00003
rm -rf Adminserver.out00001 Adminserver.out00002 Adminserver.out00003
rm -rf access.log00001 access.log00002 access.log00003

/home/bea10.3/user_projects/domains/sherkhan/servers/ms1/logs

rm -rf ms1.log00001
rm -rf ms1.out00001
or zip the log files
/home/bea10.3/user_projects/domains/sherkhan/servers/AdminServer/logs
gzip -r *
/home/bea10.3/user_projects/domains/sherkhan/servers/AdminServer
gzip -r logs


High cpu utilization:

top (linux)
prstat (solaris)

top - 07:45:22 up  3:03,  3 users,  load average: 0.16, 0.33, 0.17
Tasks: 113 total,   2 running, 109 sleeping,   0 stopped,   2 zombie
Cpu(s):  0.0%us,  0.7%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1035400k total,  1020348k used,    15052k free,    77688k buffers
Swap:  2040212k total,        0k used,  2040212k free,   483724k cached

ps -ef | grep 9523

If you find any zombie process count >50 raise a ticket to solaris admins
If any java processes are occupying 95-100% cpu usage then check the log files for any continuous looping messages or jdbc transaction time outs.
fix the problem and kill manged  server using kill -9 pid and restart the service instance.

Application logs files not rotating issue:

check the diskspace if it is full then delete old logs
check whether log4j properties file set in classpath

404 error:

404 error: page can't be displayed.
10.4.5 404 Not Found

The server has not found anything matching the Request-URI. No indication is given of whether the condition is temporary or permanent.

sol:

1) check whether they are using correct url
2) check whether apache server is running ( ps -ef | grep httpd) ( ps -ef | grep -i apache)

Apache2.2/bin

httpd -k start
httpd -k stop

apachectl -k start
apachectl -k stop

3) check the diskspace of Apache server if it is full then delete the log files (df -kh)

goto Apache2.2/logs

delete old logs

4) Check whether the deployed application is in active state
5) If the deployed application is failed then fix the issue and redeploy the application

500 error:

Service unavailable
this error is due to server down
check apache or weblogic service instance is the server is down then start the server.

403 error:

Access forbidden
check whether the proxy mapping is correct
check syngrants and synanyms run properly in data base side
check whether the user having access to the application
issue:replicas.prop file corrupted

<BEA-000386> Server subsystem failed. Reason: java.lang.NumberFormatException: null
java.lang.NumberFormatException: null
        at java.lang.Integer.parseInt(Integer.java:417)
        at java.lang.Integer.parseInt(Integer.java:499)
        at weblogic.ldap.EmbeddedLDAP.validateVDEDirectories(EmbeddedLDAP.java:1097)
        at weblogic.ldap.EmbeddedLDAP.start(EmbeddedLDAP.java:242)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)

This mostly happens when LDAP files are corrupted under the ../domain-name/server/AdminServer/data/ldap/ directory. A possible cause of corruption is when space on server is full. When the associated volume is full (100%) weblogic server will corrupt these files.

sol:
rm -rf /home/bea10.3/user_projects/domains/sherkhan/servers/AdminServer/data/ldap/conf/replicas.prop

To fix the above error tried the below:
Remove the ../domain-name/server/AdminServer/data/ldap/conf/replicas.prop file and restart the Admin server. It should work now.

Error: unable to obtain lock file

<May 13, 2012 8:34:54 PM IST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

There are 1 nested errors:
weblogic.management.ManagementException: Unable to obtain lock on C:\bea10.3\user_projects\domains\sherkhan\servers\AdminServer\tmp\AdminServer.lok. Server may already be running
        at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:159)
        at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:58)
        at weblogic.management.internal.DomainDirectoryService.start(DomainDirectoryService.java:73)
        at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:459)
        at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:164)
        at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:711)
        at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:482)
        at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:440)
        at weblogic.Server.main(Server.java:67)
<May 13, 2012 8:34:54 PM IST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<May 13, 2012 8:34:54 PM IST> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<May 13, 2012 8:34:54 PM IST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

rm -rf \bea10.3\user_projects\domains\sherkhan\servers\AdminServer\tmp\AdminServer.lok
if the server is already running then ignore this error
if you are unable to start the server delete lok file and restart the server.


Error: Users are getting 404 error some times and they are able to access the application sometimes.

1) check whether all managed servers are in running state.
if one of the managed server is in shutdown state then bring up the server.
check the http requests in access.log file for all managed server
if you are getting 404 error in one of the managed server log. then check server log for any errors
i got the below error in log file:
java.lang.socket exception: address or port already in use
netstat -anp | grep 8002
if the port is listened on any other instance. restat managed server.
if the issue still persists then raise a reqest to network team..

servers are running in admin mode:

server will run in admin mode due to deployment or connection pool issue

fix deployment or jdbc problems and resume servers.

Failed_not_restartable mode:

if the disk space is full then servers will go to failed not restable mode

stack overflow error:

If you get stack overflow error in log file.
we need to restart the server or increase the stack size using XSS:1024 (2048)

Weblogic.nodemanager.NMException / java.rmi.NoSuchObjectException While Starting AdminServer

Weblogic.nodemanager.NMException / java.rmi.NoSuchObjectException While Starting AdminServer 

Issue:

javax.naming.ServiceUnavailableException [Root exception is java.rmi.NoSuchObjectException: The object identified by: '31' could not be found. Either it was has not been exported or it has been collected by the distributed garbage collector.]
at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:70)
weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:174)
at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:154)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: java.rmi.NoSuchObjectException: The object identified by: '31' could not be found. Either it was has not been exported or it has been collected by the distributed garbage collector.
at weblogic.rmi.internal.OIDManager.getServerReference(OIDManager.java:251)
at weblogic.rmi.extensions.server.ServerHelper.getLocalInitialReference(ServerHelper.java:455)
at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:131)
at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:124)
at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:427)

Cause:

Under DOMAIN_HOME/servers/AdminServer/nodemanager directory, there is a file called startup-copy.properties file.

This file has ADMIN_URL parameter. Adminserver does not need ADMIN_URL parameter defined in start-up.properties. If it is present, then it results in the error.

Solution:

Make sure nodemanager.properties has StartScriptEnabled=true and StartScriptName=startWebLogic.sh
Under DOMAIN_HOME/servers/AdminServer/nodemanager directory, make sure none of the files have ADMIN_URL parameter.

Referene metalink Doc ID 2335354.1

Unlocking Weblogic Admin User Account

Unlocking Weblogic Admin User Account 

Error:
<Notice> <Security> <BEA-090078> <User xxxxxxx in security realm myrealm has had 5 invalid login attempts, locking account for 30 minutes.>

Cause:
By default WebLogic Server enables the user lockout mechanism which locks a particular user account for 30 minutes, if more than 5 attempts are made to login with wrong password.

Solution:
This behaviour of WebLogic Server can be  modified  and the WLS Admin can disable the user account lockout on invalid login attempts mechanism by using the steps mentioned below.

Solution 1

1)- Login to the WLS console as an admin user.
2)- In the left hand side tree go to Security => Realms => myrealm.
3)- In the right hand pane select User Lockout tab => Uncheck Lockout Enabled option.
4)- Click Apply.
5)- Restart the server.

Solution 2

Unlock the admin user by following below example:

In the following example, an administrator named "Admin" with a password of "welcome1" requests the unlocking of the WebLogic Server listening on port "7001" on machine "localhost"

$java weblogic.Admin -url <host_name:<port> -username <username> -password <adminpasswd> UNLOCK

How to check whether user is unlocked or not.

Administration Console: Select Admin server -> Expand the Monitoring-->Security tab for the server.
This page allows you to monitor user lockout management statistics for this server.
Total Invalid Logins:       14
User Lockout Total Count: 1
Total Login Attempts while Locked:8
Invalid Logins High: 1
Locked Users:    0
Total Users Unlocked:   1

Decrypt The Passwords In Weblogic

Decrypt The Passwords In Weblogic

step.1

source the $DOMAIN_HOME/bin/setDomainEnv.sh

step.2

cd $DOMAIN_HOME/security
create a file called plainpassword.py and save the file.
vi plainpassword.py

from weblogic.security.internal import *
from weblogic.security.internal.encryption import *

encryptionService = SerializedSystemIni.getEncryptionService(".")
clearOrEncryptService = ClearOrEncryptedService(encryptionService)

passwd = raw_input("Enter encrypted password of one which you wanted to decrypt : ")

plainpwd = passwd.replace("\\", "")

print "Plain Text password is: " + clearOrEncryptService.decrypt(plainpwd)

step.3

Run the setWLSEnv.sh script located in $WL_HOME/server/bin.

step.4

Run the below command to decrypting the password. Enter the encrypted password when prompted.
java weblogic.WLST plainpassword.py

step.5

Provide the encrypted password .

Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
Enter encrypted password of one which you wanted to decrypt : {AES}LsGaddassssvQDyibmejXFkf1tWxyndNArAhZ3M5GcnjXWUpJs=
Plain Text password is: Welcome1234

Note:
username and password both can be decrypted with the above method.

Generating Thread Dumps In Weblogic Server Using OS commands

Generating Thread Dumps:

1. Start the server from command line script (using nohup). Let us take managed server as an example for which we need to generate the thread dumps so start the server using script as shown below.
cd $DOMAIN_HOME/bin
nohup ./startManagedWeblogic.sh <Server_name> &

2. Now identify the PID (java Process ID) for the managed server using the below command:
ps -ef|  grep –i <server_name>
ps -ef|grep -i ms1

3. Now run the below command to create the thread dump.
kill -3 <PID>
kill -3 220221

(This will send a signal to the process whose dump we require. This signal causes the Java Virtual Machine to generate a stack trace of the process.). This command will alos create thread dump in the nohup.out file (where we started the managed server)

4. Open the nohup.out file to see generated thread dumps.

WebLogic Server Cannot Be Started And Throws Following Exception "com.bea.wcp.sip.engine.server.setup.SipAnnotationParsingException"

Applies To:

Oracle WebLogic Server - Version 10.3.6 and later
Information in this document applies to any platform.
***Checked for relevance on28/01/2014***

Symptoms:

After successfully installing WebLogic 10.3.6, WebLogic Admin server cannot be started. The following exception is thrown and can be seen the the Admin Server log file when trying to start the WebLogic server :

####<Mar 19, 2012 2:41:33 PM EDT> <Error> <WLSS.Setup> <np-wlssb1.erp.temple.edu> <AdminServer> <[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <0000JOgFnm3EoIVX21m3Vj1FPrmf000002> <1332182493341> <BEA-331210> <Skip SIP related logic, because error occurs when parsing sip related annotations of "em"
com.bea.wcp.sip.engine.server.setup.SipAnnotationParsingException:
at com.bea.wcp.sip.engine.server.setup.SipAnnotationData.<init>(SipAnnotationData.java:155)
at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:96)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:179)
at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:43)
at com.bea.wcp.sip.engine.server.setup.SipAnnotationData.loadClassFromName(SipAnnotationData.java:430)
at com.bea.wcp.sip.engine.server.setup.SipAnnotationData.jarAnnotationParsing(SipAnnotationData.java:287)
at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:671)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:52)
at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:212)
at weblogic.application.internal.EarDeployment.activate(EarDeployment.java:59)
at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:161)
at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:79

Cause:

This issue is due to Out of memory situation in Perm Gen space.

The JVM has run out of space in the permanent generation.

Solution:

1)Increase the Perm Gen space in the java options of the Admin Server in setDomainEnv.sh.

It can be resized according to the requirement using -XX:MaxPermSize and -XX:PermSize JVM Options.

The following needs to be set :

USER_MEM_ARGS=-Xms2048m -Xms2048m -XX:PermSize=512 -XX:MaxPermSize=1024

The values can be set per the requirement

2) Try starting the server after that.

Reference metalink Doc ID 1439171.1

Password Not Supplied Error Occurs When Trying to Start WebLogic Using nohup ./startWeblogic.sh 

Applies To:

Oracle WebLogic Server - Version 8.1 and later
Information in this document applies to any platform.
***Checked for relevance on 19-Nov-2015***

Symptoms:

When attempting to start WebLogic Server using the command nohup ./startWeblogic.sh, the following error occurs:

weblogic.security.SecurityInitializationException: Authentication for user denied
  at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:966)
  at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)
  at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
  at weblogic.security.SecurityService.start(SecurityService.java:141)
  at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
  Truncated. see log file for complete stacktrace
Caused By: javax.security.auth.login.FailedLoginException: [Security:090304]Authentication Failed: User javax.security.auth.login.LoginException: [Security:090301]Password Not Supplied
  at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:261)
  at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
  at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  Truncated. see log file for complete stacktrace

Cause:

When using nohup there is not a chance to enter the username and password at the server startup command prompt. For that reason, if the boot.properties file does not exist in the security folder of the AdminServer, this error is shown in the nohup.out file and the server fails to start.

Solution:

To resolve the issue, follow these steps:

Go to the WL_HOME/user_projects/domains//servers/AdminServer folder.
Create a folder called "security" if it does not already exist.
Create a new file called boot.properties and put the following information inside it:
username=WLS_username
password=WLS_password
Save the changes and restart the servers using nohup startWeblogic.sh again.
Note that the values of boot.properties are automatically encrypted if you are running in development mode. If you are running in production mode, you will need to encrypt these values before starting the server. Use the weblogic.security.Encrypt utility to accomplish this.

Reference metalink Doc ID 1504258.1

Weblogic 10.3.6 Patching Error Using BSU - "java.lang.OutOfMemoryError: GC overhead limit exceeded"

Applies To:
Oracle WebLogic Server - Version 10.3.6 to 12.1.1.0 Information in this document applies to any platform.

Symptoms:
When attempting to apply a patch, the following error occurs:
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
  at java.lang.Class.getDeclaredMethods0(Native Method)
  at java.lang.Class.privateGetDeclaredMethods(Class.java:2521)
  at java.lang.Class.getMethod0(Class.java:2764)
  at java.lang.Class.getMethod(Class.java:1653)
  at com.bea.cie.common.dao.xbean.XBeanDataHandler.isValueSet(XBeanDataHandler.java:958)
 com.bea.cie.common.dao.xbean.XBeanDataHandler.getValueFromObject(XBeanDataHandler.java:589)com.bea.cie.common.dao.xbean.XBeanDataHandler.getSimpleValue(XBeanDataHandler.java:431
  at com.bea.plateng.patch.dao.cat.PatchDetail.getId(PatchDetail.java:42)
  at com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getPatchDetails(PatchCatalogHelper.java:700)
  at com.bea.plateng.patch.dao.cat.Version.getPatchDetails(Version.java:41)
 com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getFilteredPatchDetailMap(PatchCatalogHelper.java:1202) com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getPatchDetails(PatchCatalogHelper.java:1398)
 com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getEveryPatchDetail(PatchCatalogHelper.java:76com.bea.plateng.patch.dao.cat.PatchCatalogHelper.getInvalidatedPatchMap(PatchCatalogHelper.java:1622)  at com.bea.plateng.patch.PatchSystem.updatePatchCatalog(PatchSystem.java:436)
  at com.bea.plateng.patch.PatchSystem.refresh(PatchSystem.java:130)
  at com.bea.plateng.patch.PatchSystem.setCacheDir(PatchSystem.java:201)
  at com.bea.plateng.patch.Patch.main(Patch.java:306)

Changes:

None

Cause:

The heap size specified for BEA Smart Update (BSU) is not sufficient.

Solution:

Follow these steps to resolve the issue:
Go to $FMW_Home/utils/bsu.
Edit bsu.sh (for UNIX) or bsu.cmd (for Windows).
Here you will find the following: MEM_ARGS="-Xms256m -Xmx512m"
Increase these values as needed: for example, to "-Xms2048m -Xmx2048m".

Reference metalink Doc ID 2007492.1

EM Console Slowness and Stuck Thread Issue

EM Console Slowness and Stuck Thread Issue

Let me share my experience with EM (Enterprise Manager) console which got installed as part of Oracle Fusion Middleware 11g Installation.
Before I start discussing the problem and solution in details, here are some brief about the environment which we were running –

Item	Description
Domain	1 Admin Server and 22 Managed servers (includes 8 SOA mservers)
Machine	sun4v, 2 physical T4 servers having 12 logical LDOMs, 30 GB Memory and 4 CPU each LDOM. 2 Managed servers installed on each LDOM just for 11g application.
OS	SunOS 5.10 64 bit Installation
Database	11.2.0.2.0
FMW	11.1.1.6.

1  Problem Statement: 

In production when multiple users 15-20 start accessing the EM console, then it’s start behaving badly, response was too slow, login was not happening properly, was taking long time around 6-10 minutes, login time in all attempt was not consistent. In few of attempts we were not able to login at all, it was hanging during page loading process etc etc, for few attempts we were able to login but then internal links e.g. Dashboard tab page, Instance tab page, Fault and recovery tab page, most of the link were extremely slow.

At JVM level the threads which was getting initiated for EM console was getting declared as Stuck threads in AdminServer.out file and after few hours e.g. 4 hrs or so Admin server was getting crash since we have overload configuration implemented for all the server in our environment where we have defined if the Stuck thread counts increase more than 60 then server should stop itself.

All the time, only one error message which we were able to see in .out file is this –

<Jul 31, 2012 7:47:42 PM EST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "635" seconds working on the request "Workmanager: default, Scheduled=true, Started=true, Started time: 635793 ms[GET /em/faces/as/as/wlFarmHome?target=Farm_PROD1-SOA-domain&type=oracle_ias_farm&_afrLoop=2532392215126770&_afrWindowMode=0&_afrWindowId=null HTTP/1.1User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:8.0) Gecko/20100101 Firefox/8.0Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip, deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Referer: http://<HostName.domainName>:<PortNumber>/em/faces/as/as/wlFarmHome?target=Farm_PROD1-SOA-domain&type=oracle_ias_farm&_afrLoop=2532193472836150&_afrWindowMode=0&_afrWindowId=null

Cookie: JSESSIONID=1nfKQXdBkkPY4nNTxfM4ZLtX19BcsbpFy2nQ2KJ01t4KwpR4NQyb!-86916873; s_pers=%20s_evar41%3D%255B%255B'e%253A21813dm%253Av%253Apers%253Acus%253Ambb%253Aredeemmodem%253A%253Aredeem-mega-modem-short-lnk'%252C'1303350152411'%255D%255D%7C1461202952411%3B%20s_prop19%3Df429da9da84831ca7aea17ae80f486d0%7C1334887942700%3B%20s_visit%3D1%7C1303353770759%3B%20s_prop25%3Dv%253Abus%253Acorporate-offer%7C1322794757110%3B%20s_nr%3D1322792957112%7C1325384957112%3B; s_vi=[CS]v1|26D7C5CA85149F1B-600001670000530B[CE]; PRX-LB=185106698.36895.0000; ADMINCONSOLESESSION=JxZXQXYXBnL2lpXCQZLphk56rm2Hj4jLTZx4yPnftNQPvkY4Lv9q!-86916873

Cache-Control: max-stale=0

Connection: Keep-Alive

X-BlueCoat-Via: C540871B14123F46

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace: null>

Above error message was not giving detailed information that at what steps EM is hanging, we have took several round of thread dump but were not able to conclude root cause of failure.

We have raised this concern with Oracle, also we took the help from all other accounts wherever FMW11g got implemented, SR run almost 4 month to resolve this issue, multiple changes we have done in various deployment windows and at last EM was stable, no login problem, always get logged in, all the internal tab pages were working fine and fetching the data in expected time.

2   Solution:

A series of changes has been implemented which I am going to list out here –

2.1       Change1: Enable the “Data Display Option”

Enable the checkbox for below options-

Disabled fetching of instance and fault matrices count.

Restrict display of instance and fault to the last 30 minute.

Initially we have reduced time for  “restrict display of instance and fault” from 24 hrs to 30min, later from 30min to 10min, and then finally 10min to 0min. So during initial login it will not fetch any records but instance can be pulled out on demand.

2.2       Change2: Increased cache timeout for discovery

During login process in Em console, EM does three things 1) Authentication 2) Discovery of targets and 3) loading the page.

Discovery of target step was taking too long around 10 minute since Domain was quite big and having a big list of targets.

Oracle note 1423893.1 has been implemented to cache the discovery result, so the sub sequent login attempt will be fast - Following given Mbeans attributes has been added into to improve performance – 

oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_MAX_CACHE_AGE	28800000
oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_MAX_WAIT_TIME	30000
oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_USE_CACHED_RESULTS	True
LargeRepository	True

All timing for mbeans in milliseconds.

After implementing above change, very first user login will take time after restart of Admin and the discovery result will be stored in cache and sub sequent login attempt will be fast.

 

2.3       Change3: increased the PermSize of Admin server JVM

Max PERM size has been increased from 512M to 1024M

MEM_PERM_SIZE_64BIT="-XX:PermSize=1024m”

2.4       Change4: Decrease the frequency of DMS application

EM has one inbuilt application called Dynamic Monitoring System (DMS) which does the status collection for all the targets wherever DMS got deployed, if the frequency of DMS stats collection will be too fast then it makes EM console to slow down. To increase the frequency here are the steps-

Increased below parameters in dms_config.xml file given at this location –

$MiddlewareHome/fmw/soa11.1.1.6/oracle_common/modules/oracle.dms_11.1.1/server_config

prefetch intervalSeconds from 15 second to 120,

Discover intervalSeconds from 180 second to 300 seconds

### snap of last few line of dms_config.xml

  <collectorConfiguration>

    <prefetch intervalSeconds="120" removeCycle="2" isDefault="true"/>

    <prefetch intervalSeconds="300" removeCycle="3"/>

    <discover intervalSeconds="300"/>

    <queue maxSizePerServer="4" timeoutSeconds="240"/>

  </collectorConfiguration> 

N.B.

Above highlighted configuration was also suggested by Oracle but its was causing issue a lot and got rollbacked after some time.

2.5       Change5: Un-Target DMS application from unwanted servers.

In thread dumps which we took we realized that most of threads are getting stuck at DMS application level and decided to disabled DMS application for all servers.

In Weblogic Admin Console >> Domain>> Deployment >> DMS >> Target tab page >> uncheck all servers.

Initially we have disabled the DMS application for entire SOA cluster and OSB cluster, and observed huge significant improvement in logging time, instead of taking 2min or 4 min, now EM was hardly taking 30 seconds to login but internal links response time stand same as it was previously, but login time got improved a lots.

But with this configuration we have notice lots of other new issues where most of configuration in SOA-Infrastructure was not working, we were not able to make any configuration changes e.g. under FARM>>SOA>>SOA-INFRA>>SOA Administration>>Common Properties, BPEL properties etc.

All the components e.g. BPEL, Mediator, human workflow, business rule, fault and rejected message process everything got impacted once we disabled the DMS application for SOA cluster.

Due to above impact we have to again enable the DMS application for SOA servers alone.

Since DMS is quite bulky application which does stats collection from all the targets where it got deployed, so it was advisable to disable DMS for some of the targets where monitoring was not required to improve login response in EM console.

Weblogic Admin Console >> Domain>> Deployment >> DMS >> Target tab page >> check all SOA servers alone.

2.6       Change6: Unicast mode of communication has been implemented for SOA and OSB cluster

Oracle note id 1437883.1 has been followed to implement the same-

If you have large domain, having multiple clusters e.g. Weblogic cluster, coherence cluster etc then preferred way for communication between all the members in cluster should be unicast.

In our environment Weblogic clustering was already enabled in Unicast mode but coherence cluster was using multicast mode and we have identified this using string which was present in setDomain.env file –

-Dtangosol.coherence.clusteraddress=227.7.7.9 -Dtangosol.coherence.clusterport=9778 -Dtangosol.coherence.log=jdk

Above entries got removed at domain level and for each managed server new configuration for coherence cluster got added –

e.g.  for OSB servers and SOA servers–

-Dtangosol.coherence.wka1=hostname1

-Dtangosol.coherence.wka2=hostname2

-Dtangosol.coherence.localhost= hostname1-app

-Dtangosol.coherence.localport=hostname1Port

-Dtangosol.coherence.wka1.port=hostname1Port

-Dtangosol.coherence.wka2.port=hostname2Port

Note: You can use same port number for all hostname, also if you need more logging for coherence component add this line in your setDomain.env file under JAVA_Option

-Dtangosol.coherence.log=stdout  # this will print details logs for members joining coherence cluster etc.

2.7       Change7: Disabled “BPEL recovery console” option from Dashboard

Once you logged in into EM console, during dashboard page loading, EM will try to fetch the data from “DLV_MESSAGE” tables for “invoke” and “callback” activities which take longer time since DLV_ MESSAGE table normally is quite huge.

In order to disable the same please perform this –

EM console>>Farm>>soa-infra>>Administration>>System Mbeam Browser>> Filter>> Type bean name "oracle.as.soainfra.config:name=soa-infra,*", >> In result click on "AduitConfig" attribute and change the "bpelRecoveryStatus=Off", default value for bpelRecoveryStatus is "on"

N.B.

In above configuration “O” should be capital letter of “Off”, small ‘o’ in ‘off’ will not work.

2.8       Change8: Added JVM parameter at domain level “-XX:+UseMembar”

In threads dump Oracle has identified that few of threads concurrently getting locked to one object, this mainly happening because they observed whenever thread state is getting change its not notifying to others waiting threads, to get rid from this situation above JVM parameter got added

###Snap of stack trace which we have noticed in our thread dump

"[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x00000001067c4800 nid=0x47 waiting on condition [0xffffffff5b4f9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xfffffffeca281c30> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at oracle.adf.model.dcframe.DataControlFrameImpl.lock(DataControlFrameImpl.java:482)

2.9       Change9: Added JVM parameter at domain level “-Dweblogic.management.disableManagedServerNotifications=true”

Above JVM parameter has been suggested by Oracle, this parameter will help to reduce JMX notification which occur between Admin and Managed servers whenever any new component get added, any state get changes etc.

This configuration got added only for Admin Server Instance, not at Manage server level, and we have notice huge benefit in EM after this change, now EM was not at all hanging during login time, internal link were slow but at least multiple users were able to login into EM.

2.10  Change10: Modified JVM parameter to print more details information in Thread dump at Domain level

Added the following Java option at domain level for all servers. This will help to provide more representative information in thread dumps which help during investigation..

“-XX:+PrintConcurrentLocks”

2.11  Change11: Patch applied in Environment specific to EM Console Slowness Problem

1. Patch Number 12972254, download file name “p12972254_1036_Generic”
2. Patch Number 14653405, download file name “fmw_diagnostics_14653405”
3. Patch Number 14750864, download file name “p14750864_111160_Generic”
4. Patch Number 14812586, download file name “p14812586_111160_Generic”
5. Patch Number 15853874, download file name “p15853874_111160_Generic”
6. Patch Number 15847755, download file name “p15847755_111160_Generic”

N.B.

You might not get above patches on Oracle Support website, since we got these patches directly uploaded into SR level. Also, please confirm with Oracle Support before deploying any PATCH into your environment, this given information is just for reference and does not be supported by any Vendor.

2.12  Change12: Identification of EM Queries for tunning purpose

Now after doing all sort of tunning at application level, login problem got resolved but still internal links to instance tab page, fault and rejected tab page and BPEL recovery console were taking long time to respond, then we started looking into AWR report and found that couple of queries are taking huge time around 20 minute as result EM is extremely slow for internal links and was keep spinning.

We manage to get the EM queries which EM runs in the background once users click on Instance, dashboard, fault and rejected message and BPEL recovery console tab pages.

Here are the queries –

2.12.1DASHBOARD - BPEL Recovery console queries

SELECT * FROM (SELECT /*+ FIRST_ROWS */ a.*, ROWNUM rnum  FROM (SELECT MESSAGE_GUID AS a1, DLV_TYPE AS a2, CIKEY AS a3, CLUSTER_NODE_ID AS a4, CLUSTER_NODE_KEY AS a5, COMPONENT_NAME AS a6, COMPONENT_TYPE AS a7, COMPOSITE_LABEL AS a8, COMPOSITE_NAME AS a9, COMPOSITE_REVISION AS a10, CONV_ID AS a11, DOMAIN_NAME AS a12, ECID AS a13, EVENT_NAME AS a14, EXT_INT1 AS a15, EXT_STRING1 AS a16, EXT_STRING2 AS a17, HEADER_PROPERTIES_BIN_FORMAT AS a18, HEADERS_REF_ID AS a19, OPERATION_NAME AS a20, PARTNER_LINK AS a21, PRIORITY AS a22, PROPERTIES AS a23, RECEIVE_DATE AS a24, RECOVER_COUNT AS a25, STATE AS a26, TENANT_ID AS a27, CONV_TYPE AS a28, RES_SUBSCRIBER AS a29 FROM DLV_MESSAGE WHERE ((((COMPONENT_TYPE = ?) AND (STATE IN (?, ?))) AND (RECEIVE_DATE <= ?)) AND (DLV_TYPE = ?)) ORDER BY RECEIVE_DATE DESC) a WHERE ROWNUM <= ?) WHERE rnum > ?

        bind => [bpel, 0, 1, 2012-10-31 12:07:36.044, 2, 1, 0]

SELECT * FROM (SELECT /*+ FIRST_ROWS */ a.*, ROWNUM rnum  FROM (SELECT MESSAGE_GUID AS a1, DLV_TYPE AS a2, CIKEY AS a3, CLUSTER_NODE_ID AS a4, CLUSTER_NODE_KEY AS a5, COMPONENT_NAME AS a6, COMPONENT_TYPE AS a7, COMPOSITE_LABEL AS a8, COMPOSITE_NAME AS a9, COMPOSITE_REVISION AS a10, CONV_ID AS a11, DOMAIN_NAME AS a12, ECID AS a13, EVENT_NAME AS a14, EXT_INT1 AS a15, EXT_STRING1 AS a16, EXT_STRING2 AS a17, HEADER_PROPERTIES_BIN_FORMAT AS a18, HEADERS_REF_ID AS a19, OPERATION_NAME AS a20, PARTNER_LINK AS a21, PRIORITY AS a22, PROPERTIES AS a23, RECEIVE_DATE AS a24, RECOVER_COUNT AS a25, STATE AS a26, TENANT_ID AS a27, MASTER_CONV_ID AS a28 FROM DLV_MESSAGE WHERE ((((COMPONENT_TYPE = ?) AND (STATE IN (?, ?))) AND (RECEIVE_DATE <= ?)) AND (DLV_TYPE = ?)) ORDER BY RECEIVE_DATE DESC) a WHERE ROWNUM <= ?) WHERE rnum > ?

        bind => [bpel, 0, 1, 2012-10-31 12:07:36.044, 1, 1, 0]

2.12.2Dashboard - Recent fault and rejected messages

SELECT * FROM (SELECT /*+ FIRST_ROWS */ a.*, ROWNUM rnum from (SELECT f.CIKEY, f.NODE_ID, f.SCOPE_ID, f.COUNT_ID, f.FAULT_NAME, f.FAULT_TYPE, f.POLICY_NAME, f.POLICY_VERSION, f.POLICY_CATEGORY, f.POLICY_ECID, f.CREATION_DATE, f.MODIFY_DATE, f.MESSAGE, wi.LABEL, wi.CREATOR, wi.MODIFIER, wi.STATE, ci.ECID, ci.CMPST_ID, ci.DOMAIN_NAME, ci.COMPOSITE_NAME, ci.COMPONENT_NAME, ci.COMPOSITE_REVISION, ci.COMPOSITE_LABEL FROM CUBE_INSTANCE ci, WI_FAULT f LEFT JOIN WORK_ITEM wi ON f.CIKEY = wi.CIKEY AND f.NODE_ID = wi.NODE_ID AND f.SCOPE_ID = wi.SCOPE_ID AND f.COUNT_ID = wi.COUNT_ID WHERE ci.CIKEY =  f.CIKEY AND ci.COMPONENTTYPE = ? AND (f.FAULT_TYPE is null OR f.FAULT_TYPE = ?) AND ci.STATE != 9 AND (wi.STATE is null OR wi.STATE IN (9, 4, 13, 3))  ORDER BY f.CREATION_DATE DESC) a  where ROWNUM < ? ) where rnum > ?

        bind => [bpel, {http://schemas.oracle.com/bpel/extension}RuntimeFaultMessage, 5, 0]

2.12.3Dashboard - Recent composite Instance queries 

SELECT /*+ FIRST_ROWS(40) */ ID, CONVERSATION_ID, HAS_ASSOC, PARENT_ID, UPDATED_TIME, CREATED_BY, ECID, TITLE, TEST_RUN_NAME, INDEX5, TEST_RUN_ID, INDEX3, TEST_SUITE, INDEX1, TEST_CASE, BATCH_INDEX, SOURCE_NAME, COMPOSITE_DN, SOURCE_TYPE, CREATED_TIME, SOURCE_ACTION_TYPE, INDEX6, SOURCE_ACTION_NAME, INDEX2, STATE, BATCH_ID, LIVE_INSTANCES, TAGS, STATE_COUNT, BUSINESS_STATUS, VERSION, INDEX4, PARTITION_DATE, UPDATED_BY, TENANT_ID FROM COMPOSITE_INSTANCE WHERE (CREATED_TIME >= :1 ) ORDER BY CREATED_TIME DESC

We got this query from AWR report and our guess is it’s being used for “recent instance” section,

2.12.4Instance tab page 

Again our guess is same query which is getting used for Dashboard tab page instance section is getting used for "Instance" tab page as well

SELECT /*+ FIRST_ROWS(40) */ ID, CONVERSATION_ID, HAS_ASSOC, PARENT_ID, UPDATED_TIME, CREATED_BY, ECID, TITLE, TEST_RUN_NAME, INDEX5, TEST_RUN_ID, INDEX3, TEST_SUITE, INDEX1, TEST_CASE, BATCH_INDEX, SOURCE_NAME, COMPOSITE_DN, SOURCE_TYPE, CREATED_TIME, SOURCE_ACTION_TYPE, INDEX6, SOURCE_ACTION_NAME, INDEX2, STATE, BATCH_ID, LIVE_INSTANCES, TAGS, STATE_COUNT, BUSINESS_STATUS, VERSION, INDEX4, PARTITION_DATE, UPDATED_BY, TENANT_ID FROM COMPOSITE_INSTANCE WHERE (CREATED_TIME >= :1 ) ORDER BY CREATED_TIME DESC

2.12.5Instance tab page search query based on name vs title

SELECT /*+ FIRST_ROWS(50) */ ID, CONVERSATION_ID, HAS_ASSOC, PARENT_ID, UPDATED_TIME, CREATED_BY, ECID, TITLE, TEST_RUN_NAME, INDEX5, TEST_RUN_ID, INDEX3, TEST_SUITE, INDEX1, TEST_CASE, BATCH_INDEX, SOURCE_NAME, COMPOSITE_DN, SOURCE_TYPE, CREATED_TIME, SOURCE_ACTION_TYPE, INDEX6, SOURCE_ACTION_NAME, INDEX2, STATE, BATCH_ID, LIVE_INSTANCES, TAGS, STATE_COUNT, BUSINESS_STATUS, VERSION, INDEX4, PARTITION_DATE, UPDATED_BY, TENANT_ID FROM COMPOSITE_INSTANCE WHERE (TITLE LIKE :1 ) ORDER BY CREATED_TIME DESC

2.12.6Fault and Rejected message tab page 

For this tab page we could see one parent query runs which fetch initial 40 records and then for each records its run another child sql query-

2.12.6.1  Parent query-

SELECT * FROM (SELECT /*+ FIRST_ROWS */ a.*, ROWNUM rnum from (SELECT f.CIKEY, f.NODE_ID, f.SCOPE_ID, f.COUNT_ID, f.FAULT_NAME, f.FAULT_TYPE, f.POLICY_NAME, f.POLICY_VERSION, f.POLICY_CATEGORY, f.POLICY_ECID, f.CREATION_DATE, f.MODIFY_DATE, f.MESSAGE, wi.LABEL, wi.CREATOR, wi.MODIFIER, wi.STATE, ci.ECID, ci.CMPST_ID, ci.DOMAIN_NAME, ci.COMPOSITE_NAME, ci.COMPONENT_NAME, ci.COMPOSITE_REVISION, ci.COMPOSITE_LABEL

FROM CUBE_INSTANCE ci, WI_FAULT f LEFT

JOIN WORK_ITEM wi ON f.CIKEY = wi.CIKEY AND f.NODE_ID = wi.NODE_ID AND f.SCOPE_ID = wi.SCOPE_ID AND f.COUNT_ID = wi.COUNT_ID WHERE ci.CIKEY =  f.CIKEY AND ci.COMPONENTTYPE = ? AND ci.STATE != 9 AND (wi.STATE is null OR wi.STATE IN (9, 4, 13, 3))  ORDER BY f.CREATION_DATE DESC) a  where ROWNUM < ? ) where rnum > ?

        bind => [bpel, 40, 0]

2.12.6.2  Child query –

SELECT F.CIKEY, F.NODE_ID, F.SCOPE_ID, F.COUNT_ID, F.FAULT_NAME,

                F.FAULT_TYPE,F.CREATION_DATE,F.MODIFY_DATE,

                F.MESSAGE, WI.LABEL, WI.CREATOR,WI.MODIFIER,WI.STATE,CI.ECID,CI.CMPST_ID

                FROM WI_FAULT F LEFT JOIN WORK_ITEM WI ON F.CIKEY = WI.CIKEY

                AND F.NODE_ID = WI.NODE_ID AND F.SCOPE_ID = WI.SCOPE_ID AND F.COUNT_ID = WI.COUNT_ID

                LEFT JOIN CUBE_INSTANCE CI ON F.CIKEY = CI.CIKEY WHERE F.CIKEY = ? AND F.NODE_ID = ?

                AND F.SCOPE_ID = ? AND F.COUNT_ID = ?

        bind => [2050981, BpInv1, BpSeq3.17, 2]

                               

 SELECT F.CIKEY, F.NODE_ID, F.SCOPE_ID, F.COUNT_ID, F.FAULT_NAME,

                F.FAULT_TYPE,F.CREATION_DATE,F.MODIFY_DATE,

                F.MESSAGE, WI.LABEL, WI.CREATOR,WI.MODIFIER,WI.STATE,CI.ECID,CI.CMPST_ID

                FROM WI_FAULT F LEFT JOIN WORK_ITEM WI ON F.CIKEY = WI.CIKEY

                AND F.NODE_ID = WI.NODE_ID AND F.SCOPE_ID = WI.SCOPE_ID AND F.COUNT_ID = WI.COUNT_ID

                LEFT JOIN CUBE_INSTANCE CI ON F.CIKEY = CI.CIKEY WHERE F.CIKEY = ? AND F.NODE_ID = ?

                AND F.SCOPE_ID = ? AND F.COUNT_ID = ?

        bind => [2050969, BpInv6, BpSeq19.39, 2]

2.13  Change13: Created following Index to improve the EM queries execution response time

Index Creation
Table Name	Column name	Index type
COMPOSITE_INSTANCE	created_time	Normal Index
CUBE_INSTANCE	CPMST_ID	Normal Index
COMPOSITE_INSTANCE	TITLE STATE	Normal Index Normal Index
DLV_MESSAGE	STATE DLV_TYPE	Composite Index

Change14: Explored direct links to access the EM console Instance tab page, fault and recovery tab page, dashboard tab etc.

If we login into EM console via normal login procedure, it takes quite a long time to navigate from Dashboard tab page to Instance tab page etc. To avoid this delay we have provide direct link for accessing these page based on user need-

2.13.1Instance Tab Page

http://<HostName.domainName>:<PortNumber>/em/faces/ai/soa/infra?target=<targetName>

&type=oracle_soainfra&selectedTab=instancesTab

N.B.

In above string replace hostname, domain name, port number and target value to prepare the URL for your environment, target name can be pulled out from this path- right click on “soa-infra(mservername) >> click to general >> a dialogue box will appear, copy the target name from there and replace in above string. Refer the above screen shot for path.

2.13.2Fault and Recovery Tab page

http://<HostName.domainName>:<PortNumber>/em/faces/ai/soa/infra?target=<targetName>

&type=oracle_soainfra&selectedTab=faultsTab

2.13.3Dashboard Tab Page

http://<HostName.domainName>:<PortNumber>/em/faces/ai/soa/infra?target=<targetName>

&type=oracle_soainfra&selectedTab=compositeTabs

2.14     Change 15: Manually execution of Stats pack at Database level for SOA_INFRA partition tables.

One more major impacting solution we have adapted to auto schedule stats collection process for partition tables.

During our analysis of AWR report, we have identified the long running queries and also gone through the explain plan of those query and concluded that SQL optimizer is not working as expected and possible reason behind that automatic database stats collection process updates the latest partition of SOA_INFRA tables where as the old partitions SOA_INFRA tables remains unchanged.

When EM runs the queries it gets two different explain plans according to the data exist into separate partition tables as result it take longer time to response.

In order to overcome this situation we have implemented manual stats collection on Tables level, rather than having stats collection at partition level for table. A cron job now run every mid night which does table scan to identify data changes %, if more than 5% data got changed in table it does the stats collection for that table. SOA_INFRA table list where stats collection process been schedule is given below-

2.14.1     Table List where DB Manual Stats collection is required 

TABLES PARTITIONED

AUDIT_COUNTER

AUDIT_DETAILS

AUDIT_TRAIL

CI_INDEXES

COMPOSITE_INSTANCE

COMPOSITE_INSTANCE_ASSOC

COMPOSITE_INSTANCE_FAULT

CUBE_INSTANCE

CUBE_SCOPE

DLV_MESSAGE

DLV_SUBSCRIPTION

DOCUMENT_CI_REF

DOCUMENT_DLV_MSG_REF

HEADERS_PROPERTIES

REFERENCE_INSTANCE

WI_FAULT

WLI_QS_REPORT_ATTRIBUTE

WORK_ITEM

XML_DOCUMENT

2.14.2     Call the below SQL statement to forcefully initiate DB stats collection for a table 

EXEC DBMS_STATS.gather_table_stats('SOAP_SOAINFRA', 'AUDIT_COUNTER',estimate_percent => 5);