Cluster Management (Starting and stopping clusterware and resources)

Cluster Management (Starting and stopping clusterware and resources)

Check status of Cluster Ready Services on Specific Node

[oracle@racnode1 ~]$ ./crsctl check crs

CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

Check status of running services on all nodes

[oracle@racnode1 ~]$ ./srvctl status nodeapps

VIP racnode1-vip is enabled
VIP racnode1-vip is running on node: racnode1
VIP racnode2-vip is enabled
VIP racnode2-vip is running on node: racnode2
VIP racnode3-vip is enabled
VIP racnode3-vip is running on node: racnode3
VIP racnode4-vip is enabled
VIP racnode4-vip is running on node: racnode4
Network is enabled
Network is running on node: racnode1
Network is running on node: racnode2
Network is running on node: racnode3
Network is running on node: racnode4
GSD is disabled
GSD is not running on node: racnode1
GSD is not running on node: racnode2
GSD is not running on node: racnode3
GSD is not running on node: racnode4
ONS is enabled
ONS daemon is running on node: racnode1
ONS daemon is running on node: racnode2
ONS daemon is running on node: racnode3
ONS daemon is running on node: racnode4
eONS is enabled
eONS daemon is running on node: racnode1
eONS daemon is running on node: racnode2
eONS daemon is running on node: racnode3
eONS daemon is running on node: racnode4

Check status of complete clusterware stack on all nodes:

[oracle@racnode1 ~]$ ./crsctl status resource -t

--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS  
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMDATA1.dg
               ONLINE  ONLINE       racnode1                                
               ONLINE  ONLINE       racnode2                                
               ONLINE  ONLINE       racnode3                                
               ONLINE  ONLINE       racnode4                                
ora.LISTENER.lsnr
               ONLINE  ONLINE       racnode1                                
               ONLINE  ONLINE       racnode2                                
               ONLINE  ONLINE       racnode3                                
               ONLINE  ONLINE       racnode4                                
ora.asm
               ONLINE  ONLINE       racnode1                 Started        
               ONLINE  ONLINE       racnode2                 Started        
               ONLINE  ONLINE       racnode3                 Started        
               ONLINE  ONLINE       racnode4                 Started        
ora.eons
               ONLINE  ONLINE       racnode1                                
               ONLINE  ONLINE       racnode2                                
               ONLINE  ONLINE       racnode3                                
               ONLINE  ONLINE       racnode4                                
ora.gsd
               OFFLINE OFFLINE      racnode1                                
               OFFLINE OFFLINE      racnode2                                
               OFFLINE OFFLINE      racnode3                                
               OFFLINE OFFLINE      racnode4                                
ora.net1.network
               ONLINE  ONLINE       racnode1                                
               ONLINE  ONLINE       racnode2                                
               ONLINE  ONLINE       racnode3                                
               ONLINE  ONLINE       racnode4                                
ora.ons
               ONLINE  ONLINE       racnode1                                
               ONLINE  ONLINE       racnode2                                
               ONLINE  ONLINE       racnode3                                
               ONLINE  ONLINE       racnode4                                
ora.registry.acfs
               ONLINE  ONLINE       racnode1                                
               ONLINE  ONLINE       racnode2                                
               ONLINE  ONLINE       racnode3                                
               ONLINE  ONLINE       racnode4                                
--------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       racnode2                                
ora.oc4j
      1        OFFLINE OFFLINE                                              
ora.racdb.db
      1        OFFLINE OFFLINE                                              
      2        OFFLINE OFFLINE                                              
      3        OFFLINE OFFLINE                                              
      4        OFFLINE OFFLINE                                              
ora.racnode1.vip
      1        ONLINE  ONLINE       racnode1                                
ora.racnode2.vip
      1        ONLINE  ONLINE       racnode2                                
ora.racnode3.vip
      1        ONLINE  ONLINE       racnode3                                
ora.racnode4.vip
      1        ONLINE  ONLINE       racnode4                                
ora.scan1.vip
      1        ONLINE  ONLINE       racnode2                                
[oracle@racnode1 ~]$


Check status of servers in clusterware

[oracle@racnode1 ~]$ ./crsctl status server -f
NAME=racnode1
STATE=ONLINE
ACTIVE_POOLS=Generic ora.racdb
STATE_DETAILS=

NAME=racnode2
STATE=ONLINE
ACTIVE_POOLS=Generic ora.racdb
STATE_DETAILS=

NAME=racnode3
STATE=ONLINE
ACTIVE_POOLS=Generic ora.racdb
STATE_DETAILS=

NAME=racnode4
STATE=ONLINE
ACTIVE_POOLS=Generic ora.racdb
STATE_DETAILS=

Stop CRS on all nodes in clusterware

[root@racnode1 bin]# ./crsctl stop cluster -all

CRS-2673: Attempting to stop 'ora.crsd' on 'racnode2'
CRS-2673: Attempting to stop 'ora.crsd' on 'racnode3'
CRS-2673: Attempting to stop 'ora.crsd' on 'racnode4'
CRS-2673: Attempting to stop 'ora.crsd' on 'racnode1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'racnode3'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'racnode3'
CRS-2673: Attempting to stop 'ora.ASMDATA1.dg' on 'racnode3'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'racnode3'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'racnode4'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'racnode4'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'racnode4'
CRS-2673: Attempting to stop 'ora.ASMDATA1.dg' on 'racnode4'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'racnode1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'racnode1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'racnode1'
CRS-2673: Attempting to stop 'ora.ASMDATA1.dg' on 'racnode1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'racnode3' succeeded
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'racnode2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'racnode2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'racnode2'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'racnode2'
CRS-2673: Attempting to stop 'ora.racnode3.vip' on 'racnode3'
CRS-2677: Stop of 'ora.racnode3.vip' on 'racnode3' succeeded
CRS-2673: Attempting to stop 'ora.ASMDATA1.dg' on 'racnode2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.racnode1.vip' on 'racnode1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.racnode1.vip' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.racnode4.vip' on 'racnode4'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.racnode2.vip' on 'racnode2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'racnode2'
CRS-2677: Stop of 'ora.racnode4.vip' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.racnode2.vip' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'racnode3' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.ASMDATA1.dg' on 'racnode3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racnode3'
CRS-2677: Stop of 'ora.asm' on 'racnode3' succeeded
CRS-2673: Attempting to stop 'ora.eons' on 'racnode3'
CRS-2673: Attempting to stop 'ora.ons' on 'racnode3'
CRS-2677: Stop of 'ora.ASMDATA1.dg' on 'racnode4' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racnode4'
CRS-2677: Stop of 'ora.asm' on 'racnode4' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'racnode4'
CRS-2673: Attempting to stop 'ora.eons' on 'racnode4'
CRS-2677: Stop of 'ora.ons' on 'racnode3' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'racnode3'
CRS-2677: Stop of 'ora.net1.network' on 'racnode3' succeeded
CRS-2677: Stop of 'ora.ons' on 'racnode4' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'racnode4'
CRS-2677: Stop of 'ora.net1.network' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.ASMDATA1.dg' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racnode2'
CRS-2677: Stop of 'ora.ASMDATA1.dg' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racnode1'
CRS-2677: Stop of 'ora.asm' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.asm' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.eons' on 'racnode1'
CRS-2673: Attempting to stop 'ora.ons' on 'racnode1'
CRS-2677: Stop of 'ora.ons' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'racnode1'
CRS-2677: Stop of 'ora.net1.network' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.eons' on 'racnode3' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'racnode3' has completed
CRS-2677: Stop of 'ora.eons' on 'racnode4' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'racnode4' has completed
CRS-2677: Stop of 'ora.crsd' on 'racnode3' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode3'
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode3'
CRS-2673: Attempting to stop 'ora.evmd' on 'racnode3'
CRS-2673: Attempting to stop 'ora.asm' on 'racnode3'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode3' succeeded
CRS-2677: Stop of 'ora.crsd' on 'racnode4' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode4'
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode4'
CRS-2673: Attempting to stop 'ora.evmd' on 'racnode4'
CRS-2673: Attempting to stop 'ora.asm' on 'racnode4'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.evmd' on 'racnode3' succeeded
CRS-2677: Stop of 'ora.eons' on 'racnode1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'racnode1' has completed
CRS-2673: Attempting to stop 'ora.eons' on 'racnode2'
CRS-2673: Attempting to stop 'ora.ons' on 'racnode2'
CRS-2677: Stop of 'ora.evmd' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.ons' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'racnode2'
CRS-2677: Stop of 'ora.net1.network' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.crsd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode1'
CRS-2673: Attempting to stop 'ora.evmd' on 'racnode1'
CRS-2673: Attempting to stop 'ora.asm' on 'racnode1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'racnode3' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.eons' on 'racnode2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'racnode2' has completed
CRS-2677: Stop of 'ora.asm' on 'racnode3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode3'
CRS-2677: Stop of 'ora.asm' on 'racnode4' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode4'
CRS-2677: Stop of 'ora.crsd' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode2'
CRS-2673: Attempting to stop 'ora.evmd' on 'racnode2'
CRS-2673: Attempting to stop 'ora.asm' on 'racnode2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.cssd' on 'racnode3' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'racnode3'
CRS-2677: Stop of 'ora.cssd' on 'racnode4' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'racnode4'
CRS-2677: Stop of 'ora.asm' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode1'
CRS-2677: Stop of 'ora.evmd' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.cssd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'racnode1'
CRS-2677: Stop of 'ora.diskmon' on 'racnode3' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'racnode4' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.asm' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode2'
CRS-2677: Stop of 'ora.cssd' on 'racnode2' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'racnode2'
CRS-2677: Stop of 'ora.diskmon' on 'racnode2' succeeded
[root@racnode1 bin]#


Stop crs on specific node (Run this command as root user)

[root@racnode1 bin]# ./crsctl stop has

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racnode1'
CRS-2673: Attempting to stop 'ora.crsd' on 'racnode1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'racnode1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'racnode1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'racnode1'
CRS-2673: Attempting to stop 'ora.ASMDATA1.dg' on 'racnode1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.racnode1.vip' on 'racnode1'
CRS-2677: Stop of 'ora.racnode1.vip' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.racnode1.vip' on 'racnode2'
CRS-2677: Stop of 'ora.registry.acfs' on 'racnode1' succeeded
CRS-2676: Start of 'ora.racnode1.vip' on 'racnode2' succeeded
CRS-2677: Stop of 'ora.ASMDATA1.dg' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racnode1'
CRS-2677: Stop of 'ora.asm' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.eons' on 'racnode1'
CRS-2673: Attempting to stop 'ora.ons' on 'racnode1'
CRS-2677: Stop of 'ora.ons' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'racnode1'
CRS-2677: Stop of 'ora.net1.network' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.eons' on 'racnode1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'racnode1' has completed
CRS-2677: Stop of 'ora.crsd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode1'
CRS-2673: Attempting to stop 'ora.evmd' on 'racnode1'
CRS-2673: Attempting to stop 'ora.asm' on 'racnode1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'racnode1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'racnode1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.asm' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.cssd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'racnode1'
CRS-2673: Attempting to stop 'ora.diskmon' on 'racnode1'
CRS-2677: Stop of 'ora.gpnpd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'racnode1'
CRS-2677: Stop of 'ora.gipcd' on 'racnode1' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'racnode1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'racnode1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@racnode1 bin]#


Start crs on specific node (Run this command as root user)

[root@racnode1 bin]# ./crsctl start has

CRS-4123: Oracle High Availability Services has been started.

Check status of a specific instance (INSTANCE NAME IS CASE SENSITIVE)

[oracle@racnode1 ~]$ srvctl status instance -d racdb -i racdb1
Instance racdb1 is running on node racnode1

Check status of all instances in a Cluster (INSTANCE NAME IS CASE SENSITIVE)

[oracle@racnode1 ~]$ ./srvctl status instance -d racdb -i racdb1,racdb2,racdb3,racdb4

Instance racdb1 is running on node racnode1
Instance racdb2 is running on node racnode2
Instance racdb3 is running on node racnode3
Instance racdb4 is running on node racnode4

Stop specific Instance in a cluster (INSTANCE NAME IS CASE SENSITIVE)

[oracle@racnode1 ~]$ ./srvctl stop instance -d racdb -i racdb4

Now check status of instance

[oracle@racnode1 ~]$ ./srvctl status instance -d racdb -i racdb1,racdb2,racdb3,racdb4
Instance racdb1 is running on node racnode1
Instance racdb2 is running on node racnode2
Instance racdb3 is running on node racnode3
Instance racdb4 is not running on node racnode4
[oracle@racnode1 ~]$

Start specific Instance in a cluster

[oracle@racnode1 ~]$ ./srvctl start instance -d racdb -i racdb4

Now check status of instance

[oracle@racnode1 ~]$ ./srvctl status instance -d racdb -i racdb1,racdb2,racdb3,racdb4

Instance racdb1 is running on node racnode1
Instance racdb2 is running on node racnode2
Instance racdb3 is running on node racnode3
Instance racdb4 is running on node racnode4
[oracle@racnode1 ~]$

Check the status of a cluster database

[oracle@racnode1 ~]$./srvctl status database -d racdb

Instance racdb1 is running on node racnode1
Instance racdb2 is running on node racnode2
Instance racdb3 is running on node racnode3
Instance racdb4 is running on node racnode4

Stop a cluster database

[oracle@racnode1 ~]$ ./srvctl stop database -d racdb

Check status of database

[oracle@racnode1 ~]$ srvctl status database -d racdb

Instance racdb1 is not running on node racnode1
Instance racdb2 is not running on node racnode2
Instance racdb3 is not running on node racnode3
Instance racdb4 is not running on node racnode4

Start a cluster database

[oracle@racnode1 ~]$ ./srvctl stop database -d racdb

Check status of database

[oracle@racnode1 ~]$ ./srvctl status database -d racdb
Instance racdb1 is running on node racnode1
Instance racdb2 is running on node racnode2
Instance racdb3 is running on node racnode3
Instance racdb4 is running on node racnode4