Troubleshooting Application Slowness Issue In Weblogic

Troubleshooting Application Slowness Issue In Weblogic

Linux performance related commands

top - to check cpu and memory usage

top - 03:32:44 up 1 day, 23:09,  6 users,  load average: 0.00, 0.05, 0.32
Tasks: 157 total,   1 running, 156 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.1%us,  0.3%sy,  0.0%ni, 97.7%id,  0.8%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   1011508k total,  1001500k used,    10008k free,     3528k buffers
Swap:  2097148k total,   333960k used,  1763188k free,    69648k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
10 root      20   0     0    0    0 S  3.7  0.0   0:19.80 rcu_sched
13415 oracle    20   0 1286m 293m 4116 S  1.8 29.7   0:29.02 java
1 root      20   0 19408  376  200 S  0.0  0.0   0:04.57 init

System activity report
sar -r 2 5
sar -r 2

[oracle@OEL6 ~]$ sar -r 2 2
Linux 3.8.13-98.2.2.el6uek.x86_64 (OEL6)        02/10/2016      _x86_64_        (1 CPU)

03:33:57 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
03:33:59 AM     11020   1000488     98.91      2692     69448   2032144     65.37
03:34:01 AM     11020   1000488     98.91      2692     69448   2032144     65.37
Average:        11020   1000488     98.91      2692     69448   2032144     65.37

Input and output statistics

iostat 2 5
iostat 2

[oracle@OEL6 ~]$ iostat 2 1
Linux 3.8.13-98.2.2.el6uek.x86_64 (OEL6)        02/10/2016      _x86_64_        (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          1.12    0.00    0.37    0.85    0.00   97.66

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
scd0              0.00         0.00         0.00        544          0
sda               4.91       160.92        43.61   27073268    7336356
fd0               0.00         0.00         0.00         16          0

[oracle@OEL6 ~]$ free
             total       used       free     shared    buffers     cached
Mem:       1011508     999864      11644       2328       3064      60936
-/+ buffers/cache:     935864      75644
Swap:      2097148     334344    1762804

df -h - Disk usage

[oracle@OEL6 ~]$ df -h /home/oracle/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        57G   23G   32G  42% /

jmap -heap will display the snap shot of heap usage

[oracle@OEL6 ~]$ /home/oracle/jdk1.7.0_21/bin/jmap -heap 12659
Attaching to process ID 12659, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 23.21-b01

using thread-local object allocation.
Mark Sweep Compact GC

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 268435456 (256.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 21757952 (20.75MB)
   MaxPermSize      = 268435456 (256.0MB)
   G1HeapRegionSize = 0 (0.0MB)

Heap Usage:
New Generation (Eden + 1 Survivor Space):
   capacity = 80543744 (76.8125MB)
   used     = 75933440 (72.415771484375MB)
   free     = 4610304 (4.396728515625MB)
   94.27602471521563% used
Eden Space:
   capacity = 71630848 (68.3125MB)
   used     = 71377800 (68.07117462158203MB)
   free     = 253048 (0.24132537841796875MB)
   99.6467332063415% used
From Space:
   capacity = 8912896 (8.5MB)
   used     = 4555640 (4.344596862792969MB)
   free     = 4357256 (4.155403137207031MB)
   51.112904268152576% used
To Space:
   capacity = 8912896 (8.5MB)
   used     = 0 (0.0MB)
   free     = 8912896 (8.5MB)
   0.0% used
tenured generation:
   capacity = 178978816 (170.6875MB)
   used     = 101403672 (96.7060775756836MB)
   free     = 77575144 (73.9814224243164MB)
   56.65680121607241% used
Perm Generation:
   capacity = 130940928 (124.875MB)
   used     = 130640608 (124.58859252929688MB)
   free     = 300320 (0.286407470703125MB)
   99.7706446681056% used

69269 interned Strings occupying 8985680 bytes.

Validation

Welogic Server check
Check for the weblogic server status
Number of free vs available threads and stuck threads
Timeouts in logs ( Application logs and server logs)
Check java Heap memory size ( Free memory)

Unix server Check
Check I/O statistics -   iostat
Check the memory,swap Bottleneck - free
To check top process CPU utilizations -top
Check the disk space - df -h

DB Server check
Check for hung sessions
Check for long running queries
Check for CPU usage – If beyond a certain threshold need to take an action
Stale Stats