RAC Installation Step By Step

RAC Installation Step By Step

All steps will be carried out on one of the node (node1) and later we will clone to other nodes or machines:

Install the system Linux prerequisites:

The package oracle-database-preinstall-19c contains all the prerequisites on Oracle Linux using the Oracle Unbreakable Enterprise Kernel (UEK).

yum install -y oracle-database-preinstall-19c
yum install -y oracleasm-support

sysctl -p
crosscheck >>> cat /etc/sysctl.conf

Group Creation: –

groupadd -g 5001 oinstall
groupadd -g 5002 dba
groupadd -g 5003 oper
groupadd -g 5004 asmadmin
groupadd -g 5005 asmdba
groupadd -g 5006 asmoper

User Creation: –

useradd -u 5007 -g oinstall -G dba,oper,asmdba oracle
useradd -u 5008 -g oinstall -G asmadmin,asmdba,asmoper,dba grid
usermod -g oinstall -G dba,oper,asmdba oracle
usermod -g oinstall -G asmadmin,asmdba,asmoper,dba grid

passwd oracle
passwd grid

Add below entries in /etc/security/limits.conf file which will detine limits

oracle soft nofile 1024
oracle hard nofile 65536
oracle soft nproc 16384
oracle hard nproc 16384
oracle soft stack 10240
oracle hard stack 32768
oracle hard memlock 134217728
oracle soft memlock 134217728

grid soft nofile 1024
grid hard nofile 65536
grid soft nproc 16384
grid hard nproc 16384
grid soft stack 10240
grid hard stack 32768
grid hard memlock 134217728
grid soft memlock 134217728

Create the directories in which the Oracle / Grid software will be installed.

mkdir -p /u01/app/grid
mkdir -p /u01/app/19c/gridhome_1
mkdir -p /u02/app/oracle
mkdir -p /u02/app/oracle/product/19c/dbhome_1
mkdir -p /u01/app/oraInventory
mkdir -p /u01/LATEST_PATCH/
chown -R grid:oinstall /u01
chown -R oracle:oinstall /u02
chown -R grid:oinstall /u01/app/oraInventory
chmod -R 775 /u01
chmod -R 775 /u02

Set bash_profile for grid and oracle user

login to grid user and set .bash_profile :

export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/19c/gridhome_1
export ORACLE_SID=+ASM1
export ORACLE_TERM=xterm
export BASE_PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

login to oracle user and set .bash_profile :

export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_BASE=/u02/app/oracle
export ORACLE_HOME=/u02/app/oracle/product/19c/dbhome_1
export ORACLE_SID=racdb1
export ORACLE_TERM=xterm
export BASE_PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib


Add the following entry in /etc/hosts file and crosscheck node reachability

Primary Site:

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

# Private

192.168.10.1 node1-priv.localdomain node1-priv
192.168.10.2 node2-priv.localdomain node2-priv

# Public

192.168.56.71 node1.localdomain node1
192.168.56.72 node2.localdomain node2

# Virtual

192.168.56.81 node1-vip.localdomain node1-vip
192.168.56.82 node2-vip.localdomain node2-vip

# SCAN

192.168.56.91 node-scan.localdomain node-scan
192.168.56.92 node-scan.localdomain node-scan
192.168.56.93 node-scan.localdomain node-scan

Firewall Stop and Disable

We can open firewall after installation

systemctl stop firewalld.service
systemctl disable firewalld.service

Chrony NTP Configuration

systemctl enable chronyd.service
systemctl restart chronyd.service


Verify the network reachability using ping command:

From Node1:
ping node2
ping node2-priv

From Node2:
ping node1
ping node1-priv

Create shared ASM Disk FOR OCR, DATAFILE and ARCHIVE on Both Sites.

Add new shared disk to virtual box machines it should be sharable

Login to system and run ls /dev/sd*
/dev/sda /dev/sda1 /dev/sda2 /dev/sdb
fdisk /dev/sdb

The sequence of answers is "n", "p", "1", "Return", "Return" and "w".

Set ASM disks >>>> oracleasm configure -i
Create ASM disks >>>> oracleasm createdisk ASM1 /dev/sdb1
Scan ASM disks >>>> oracleasm scandisks
List ASM disks >>>> oracleasm listdisks

######### On Primary Site ##########

Copy software to node1 on primary site and unzip software at grid home location and rdbms home locations by login to respective users.

Setup passwordless ssh configuration

cd /u01/app/19c/gridhome_1/deinstall
./sshUserSetup.sh -user grid -hosts "node1 node2" -noPromptPassphrase -confirm -advanced

Similarly do it for oracle user

cd /u02/app/oracle/product/19c/dbhome_1/deinstall

./sshUserSetup.sh -user oracle -hosts "node1 node2" -noPromptPassphrase -confirm –advanced
./sshUserSetup.sh -user root -hosts "node1 node2" -noPromptPassphrase -confirm –advanced

Set profile and go to ORACLE_HOME location from grid user and run following command and follow the steps.

cd $ORACLE_HOME
./gridSetup.sh -applyRU /u01/LATEST_PATCH/32545008

Once Grid Installation done, Install RDBMS Software through oracle user and create RAC database on production (PROD).

./runInstaller -applyRU /u01/LATEST_PATCH/32545008
Use DBCA to create database PROD.

Oracle Single Instance Vs RAC Vs RAC One Node

Oracle Single Instance Vs RAC Vs RAC One Node

There are three types of deployment options you have :

1.Oracle Single Instance (Single Instance)

2.Oracle Real Application Cluster (RAC)

3.Oracle RAC One Node

Below are the overview of these databases :

Single Instance: 

Single Instance is a simple and basic option of deploying oracle database. In single instance database there is only one instance of database running on one node. Single Instance is simple to install and maintain. This could be a good choice for small production and development environments.

Oracle RAC Database: 

Oracle RAC database is a clustered database. It can have more than one instance running on one or more hosts. To make is database RAC , there is another software needs to be installed called Oracle Grid infrastructure which takes care of multinode sync, ASM, Oracle restart etc. Oracle GI is necessary as a base of Oracle RAC. In Oracle RAC, the load is balanced across multiple instance of databases running on different nodes. This is best for Production database. Failure of any instance is abstracted from user and forwarded to another instance from the database server pool. No of instances can be controlled using Policy Managed or Admin Managed.

Oracle RAC One Node: 

Some time we need to have RAC databases but don't need many instances, why? Simple answers to it is leverage the high availability of RAC. The Oracle RAC One node is a RAC DB runs on one host. The advantage is it saves your resource, can to converted to full RAC any time, Cluster fail over i.e. the Database instance is moved to new instance in case of any failure on host known as Instance Relocation, easy to upgrade etc. This type of database is well suited for small production and dev/test environments.

RAC Manual Patching Steps On 12c

RAC Manual Patching Steps On 12c

On NODE.1

1. Stop the database resources from DB home. 

[oracle@test01 ~]$ ps -ef|grep pmon

oracle   15294     1  0 00:35 ?        00:00:00 asm_pmon_+ASM1

oracle   18171     1  0 00:39 ?        00:00:00 ora_pmon_RUMPROD1

oracle   19097  9655  0 00:40 pts/2    00:00:00 grep pmon

SQL> shut immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

[oracle@test01 ~]$ ps -ef|grep pmon

oracle   15294     1  0 00:35 ?        00:00:00 asm_pmon_+ASM1

oracle   28749  9655  0 01:16 pts/2    00:00:00 grep pmon

2. On GI Home, as the root user or sudo execute below:

[root@test01 install]# ps -ef|grep pmon

oracle   15294     1  0 00:35 ?        00:00:00 asm_pmon_+ASM1

root     29836  6253  0 01:20 pts/1    00:00:00 grep pmon

[root@test01 install]# $ORACLE_HOME/crs/install/rootcrs.pl -prepatch

Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params

Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [0].

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'test01'

CRS-2673: Attempting to stop 'ora.crsd' on 'test01'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'test01'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'test01'

CRS-2673: Attempting to stop 'ora.OCR.dg' on 'test01'

CRS-2677: Stop of 'ora.OCR.dg' on 'test01' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'test01'

CRS-2677: Stop of 'ora.asm' on 'test01' succeeded

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'test01' succeeded

CRS-2673: Attempting to stop 'ora.test01.vip' on 'test01'

CRS-2677: Stop of 'ora.test01.vip' on 'test01' succeeded

CRS-2672: Attempting to start 'ora.test01.vip' on 'test02'

CRS-2676: Start of 'ora.test01.vip' on 'test02' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'test01'

CRS-2677: Stop of 'ora.ons' on 'test01' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'test01'

CRS-2677: Stop of 'ora.net1.network' on 'test01' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'test01' has completed

CRS-2677: Stop of 'ora.crsd' on 'test01' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on 'test01'

CRS-2673: Attempting to stop 'ora.evmd' on 'test01'

CRS-2673: Attempting to stop 'ora.storage' on 'test01'

CRS-2673: Attempting to stop 'ora.crf' on 'test01'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'test01'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'test01'

CRS-2673: Attempting to stop 'ora.gpnpd' on 'test01'

CRS-2677: Stop of 'ora.storage' on 'test01' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'test01'

CRS-2677: Stop of 'ora.drivers.acfs' on 'test01' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'test01' succeeded

CRS-2677: Stop of 'ora.evmd' on 'test01' succeeded

CRS-2677: Stop of 'ora.crf' on 'test01' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'test01' succeeded

CRS-2677: Stop of 'ora.gpnpd' on 'test01' succeeded

CRS-2677: Stop of 'ora.asm' on 'test01' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'test01'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'test01' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'test01'

CRS-2677: Stop of 'ora.cssd' on 'test01' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'test01'

CRS-2677: Stop of 'ora.gipcd' on 'test01' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'test01' has completed

CRS-4133: Oracle High Availability Services has been stopped.

2021/04/14 01:22:01 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.

2021/04/14 01:22:37 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.

2021/04/14 01:22:37 CLSRSC-347: Successfully unlock /u01/app/12.1.0/grid

[root@test01 install]# ps -ef|grep pmon

root      3037  6253  0 01:27 pts/1    00:00:00 grep pmon

3. Check the patch Conflicts

4. Patch GI home [as GI home owner]

Goto patch directory and apply the patch.

opatch apply

5. Run the post script (as the root or sudo user)

[root@test01 install]# $ORACLE_HOME/rdbms/install/rootadd_rdbms.sh

[oracle@test01 ~]$ ps -ef|grep pmon

oracle   21867  9655  0 01:30 pts/2    00:00:00 grep pmon

[root@test01 install]# $ORACLE_HOME/crs/install/rootcrs.pl -postpatch

Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params

2021/04/14 01:30:33 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2021/04/14 01:30:49 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

CRS-4123: Starting Oracle High Availability Services-managed resources

CRS-2672: Attempting to start 'ora.mdnsd' on 'test01'

CRS-2672: Attempting to start 'ora.evmd' on 'test01'

CRS-2676: Start of 'ora.mdnsd' on 'test01' succeeded

CRS-2676: Start of 'ora.evmd' on 'test01' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'test01'

CRS-2676: Start of 'ora.gpnpd' on 'test01' succeeded

CRS-2672: Attempting to start 'ora.gipcd' on 'test01'

CRS-2676: Start of 'ora.gipcd' on 'test01' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'test01'

CRS-2676: Start of 'ora.cssdmonitor' on 'test01' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'test01'

CRS-2672: Attempting to start 'ora.diskmon' on 'test01'

CRS-2676: Start of 'ora.diskmon' on 'test01' succeeded

CRS-2676: Start of 'ora.cssd' on 'test01' succeeded

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'test01'

CRS-2672: Attempting to start 'ora.ctssd' on 'test01'

CRS-2676: Start of 'ora.ctssd' on 'test01' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'test01' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'test01'

CRS-2676: Start of 'ora.asm' on 'test01' succeeded

CRS-2672: Attempting to start 'ora.storage' on 'test01'

CRS-2676: Start of 'ora.storage' on 'test01' succeeded

CRS-2672: Attempting to start 'ora.crf' on 'test01'

CRS-5013: Agent "ORAROOTAGENT" failed to start process "/u01/app/12.1.0/grid/bin/osysmond" for action "start": details at "(:CLSN00008:)" in "/u01/app/oracle/diag/crs/test01/crs/trace/ohasd_orarootagent_root.trc"

CRS-2674: Start of 'ora.crf' on 'test01' failed

CRS-2679: Attempting to clean 'ora.crf' on 'test01'

CRS-2681: Clean of 'ora.crf' on 'test01' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'test01'

CRS-2676: Start of 'ora.crsd' on 'test01' succeeded

CRS-6017: Processing resource auto-start for servers: test01

CRS-2672: Attempting to start 'ora.net1.network' on 'test01'

CRS-2676: Start of 'ora.net1.network' on 'test01' succeeded

CRS-2672: Attempting to start 'ora.ons' on 'test01'

CRS-2673: Attempting to stop 'ora.test01.vip' on 'test02'

CRS-2677: Stop of 'ora.test01.vip' on 'test02' succeeded

CRS-2672: Attempting to start 'ora.test01.vip' on 'test01'

CRS-2676: Start of 'ora.test01.vip' on 'test01' succeeded

CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'test01'

CRS-2676: Start of 'ora.ons' on 'test01' succeeded

CRS-2676: Start of 'ora.LISTENER.lsnr' on 'test01' succeeded

CRS-6016: Resource auto-start has completed for server test01

CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources

CRS-4123: Oracle High Availability Services has been started.

Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [0].

PRCC-1010 : _mgmtdb was already enabled

PRCR-1002 : Resource ora.mgmtdb is already enabled

[oracle@test01 ~]$ ps -ef|grep pmon

oracle    1099  9655  0 01:36 pts/2    00:00:00 grep pmon

oracle    1502     1  0 01:32 ?        00:00:00 asm_pmon_+ASM1

6. Start the database resources from DB home.

[oracle@test01 ~]$ ps -ef|grep pmon

oracle    1099  9655  0 01:36 pts/2    00:00:00 grep pmon

oracle    1502     1  0 01:32 ?        00:00:00 asm_pmon_+ASM1

SQL> startup;

ORACLE instance started.

Total System Global Area  754974720 bytes

Fixed Size                  2928968 bytes

Variable Size             385879736 bytes

Database Buffers          360710144 bytes

Redo Buffers                5455872 bytes

Database mounted.

Database opened.

[oracle@test01 ~]$ ps -ef|grep pmon

oracle    1502     1  0 01:32 ?        00:00:00 asm_pmon_+ASM1

oracle   16180     1  0 01:37 ?        00:00:00 ora_pmon_RUMPROD1

oracle   19651 19507  0 01:38 pts/2    00:00:00 grep pmon

Perform all the above steps in same manner on NODE.2

Note:

Always read the patch readme to understand the patching instructions, general procedure to apply the database prepatch and postpatch steps.

7. Run the postpatch datapatch apply (In rac only datapatch will be run only one node)

[oracle@test01 ~]$ cd $ORACLE_HOME/OPatch

[oracle@test01 OPatch]$ ./datapatch -verbose

SQL Patching tool version 12.2.0.0.0 on Wed Apr 14 01:41:32 2021

Copyright (c) 2014, Oracle.  All rights reserved.

Connecting to database...OK

Determining current state...done

Current state of SQL patches:

Adding patches to installation queue and performing prereq checks...

Installation queue:

Nothing to roll back

Nothing to apply

SQL Patching tool complete on Wed Apr 14 01:43:17 2021

Creating a Services In Oracle RAC

Creating a Services In Oracle RAC

To check the configuration:

srvctl config database -d RUMPROD

To create a new service:

srvctl add service -d RUMPROD -s QSHO -r RUMPROD1,RUMPROD2

To start a service on one node:

srvctl start service -d RUMPROD -s QRUM -n test02 -o open

To start a service on all nodes:

srvctl start service -d RUMPROD -s QSHO -o open

To stop service in one node/ instance:

srvctl stop service -d RUMPROD -s QSHO -n test01

To stop service in all nodes/ instances:

srvctl stop service -d RUMPROD -s QSHO

To remove a service:

Oracle Best practices is to frirst stop the service, then drop it.

srvctl stop service -d RUMPROD -s QSHO

srvctl remove service -d RUMPROD -s QSHO

To check status of database and list services:

srvctl status database -d db_unique_name -v

Or using View :

SQL> select * from gv$active_services;

How to Stop and Start Oracle RAC Services on Single Node

How to Stop and Start Oracle RAC Services on Single Node

1) Disable the automatic startup of the Oracle Clusterware software:

crsctl disable has

2) Stop the Oracle clusterware stack:

crsctl stop has

3) Once clusterware Services has been stopped we will restart those Services:

crsctl enable has

4) Enable the automatic startup of the Oracle Clusterware software:

crsctl start has

Check the status:

ps -ef|grep d.bin|wc -l

ps -ef|grep pmon

5) Start the Oracle clusterware stack:

crsctl start crs

6)Check the status of Oracle Clusterware Resources:

crsctl stat res -t

How to check Grid Infrastructure version

How to check Grid Infrastructure version

# crsctl query crs activeversion
# crsctl query crs releaseversion
# crsctl query crs softwareversion

What does crsctl query crs activeversion mean ?
If some nodes become unreachable in the middle of an upgrade, then you cannot complete the upgrade, because the upgrade script (rootupgrade.sh) did not run on the unreachable nodes. Because the upgrade is incomplete, Oracle Clusterware remains in the previous release version.

What does crsctl query crs releaseversion mean ?
Use the crsctl query crs releaseversion command to display the version of the Oracle Clusterware software stored in the binaries on the local node.

What does crsctl query crs softwareversion mean ?
Use the crsctl query crs softwareversion command to display latest version of the software that has been successfully started on the specified node.

Enable Archive Log Mode In Oracle RAC 11g

Enable Archive Log Mode In Oracle RAC 11g

To check the status:

[oracle@rac1 ~]$ srvctl status database -d PROD
Instance PROD1 is running on node rac1
Instance PROD2 is running on node rac2

To stop the database:

[oracle@rac1 ~]$ srvctl stop database -d PROD -o immediate

To check the status:

[oracle@rac1 ~]$ srvctl status database -d PROD
Instance PROD1 is not running on node rac1
Instance PROD2 is not running on node rac2

To mount the database:

[oracle@rac1 ~]$ srvctl start database -d PROD -o mount

To enable archivelog:

SQL> alter database archivelog;

To restart the database:

[oracle@rac1 ~]$ srvctl stop database -d PROD -o immediate
[oracle@rac1 ~]$ srvctl status database -d PROD
Instance PROD1 is not running on node rac1
Instance PROD2 is not running on node rac2

To start the database:

[oracle@rac1 ~]$ srvctl start database -d PROD
[oracle@rac1 ~]$ srvctl status database -d PROD
Instance PROD1 is running on node rac1
Instance PROD2 is running on node rac2

To set the archive destination to a ASM DISK:

alter system set log_archive_dest_1='LOCATION=+DATA/ARCH/' scope=both sid='*';

For disabling archive mode:

srvctl stop database -d PROD
srvctl start database -d PROD -o mount
alter database noarchivelog;
srvctl stop database -d PROD
srvctl start database -d PROD

Node Eviction Overview

Node Eviction Overview

The Oracle Clusterware is designed to perform a node eviction by removing one or more nodes from the cluster if some critical problem is detected.  A critical problem could be a node not responding via a network heartbeat, a node not responding via a disk heartbeat, a hung or severely degraded machine, or a hung ocssd.bin process.  The purpose of this node eviction is to maintain the overall health of the cluster by removing bad members.

Starting in 11.2.0.2 RAC or above (or if you are on Exadata), a node eviction may not actually reboot the machine.  This is called a rebootless restart.  In this case we restart most of the clusterware stack to see if that fixes the unhealthy node.

1.0 - PROCESS ROLES FOR REBOOTS

OCSSD (aka CSS daemon) - This process is spawned by the cssdagent process. It runs in both
vendor clusterware and non-vendor clusterware environments.  OCSSD's primary job is internode health monitoring and RDBMS instance endpoint discovery. The health monitoring includes a network heartbeat and a disk heartbeat (to the voting files).  OCSSD can also evict a node after escalation of a member kill from a client (such as a database LMON process). This is a multi-threaded process that runs at an elevated priority and runs as the Oracle user.

Startup sequence: INIT --> init.ohasd --> ohasd --> ohasd.bin --> cssdagent --> ocssd --> ocssd.bin

CSSDAGENT - This process is spawned by OHASD and is responsible for spawning the OCSSD process, monitoring for node hangs (via oprocd functionality), and monitoring to the OCSSD process for hangs (via oclsomon functionality), and monitoring vendor clusterware (via vmon functionality).  This is a multi-threaded process that runs at an elevated priority and runs as the root user.

Startup sequence: INIT --> init.ohasd --> ohasd --> ohasd.bin --> cssdagent

CSSDMONITOR - This proccess also monitors for node hangs (via oprocd functionality), monitors the OCSSD process for hangs (via oclsomon functionality), and monitors vendor clusterware (via vmon functionality). This is a multi-threaded process that runs at an elevated priority and runs as the root user.

Startup sequence: INIT --> init.ohasd --> ohasd --> ohasd.bin --> cssdmonitor

2.0 - DETERMINING WHICH PROCESS IS RESPONSIBLE FOR A REBOOT
Important files to review:

Clusterware alert log in
The cssdagent log(s)
The cssdmonitor log(s)
The ocssd log(s)
The lastgasp log(s) in /etc/oracle/lastgasp or /var/opt/oracle/lastgasp
CHM or OS Watcher data
opatch lsinventory -detail' output for the GRID home
Messages files

* Messages file locations:

Linux: /var/log/messages
Sun: /var/adm/messages
HP-UX: /var/adm/syslog/syslog.log
IBM: /bin/errpt -a > messages.out

11.2 Clusterware evictions should, in most cases, have some kind of meaningful error in the clusterware alert log.  This can be used to determine which process is responsible for the reboot.  Example message from a clusterware alert log:

[ohasd(11243)]CRS-8011:reboot advisory message from host: sta00129, component: cssagent, with timestamp: L-2009-05-05-10:03:25.340
[ohasd(11243)]CRS-8013:reboot advisory message text: Rebooting after limit 28500 exceeded; disk timeout 27630, network timeout 28500, last heartbeat from CSSD at epoch seconds 1241543005.340, 4294967295 milliseconds ago based on invariant clock value of 93235653

This particular eviction happened when we had hit the network timeout.  CSSD exited and the cssdagent took action to evict. The cssdagent knows the information in the error message from local heartbeats made from CSSD.

If no message is in the evicted node's clusterware alert log, check the lastgasp logs on the local node and/or the clusterware alert logs of other nodes.

3.0 - TROUBLESHOOTING OCSSD EVICTIONS

If you have encountered an OCSSD eviction review common causes in section 3.1 below.

3.1 - COMMON CAUSES OF OCSSD EVICTIONS
Network failure or latency between nodes. It would take 30 consecutive missed checkins (by default - determined by the CSS misscount) to cause a node eviction.
Problems writing to or reading from the CSS voting disk.  If the node cannot perform a disk heartbeat to the majority of its voting files, then the node will be evicted.
A member kill escalation.  For example, database LMON process may request CSS to remove an instance from the cluster via the instance eviction mechanism.  If this times out it could escalate to a node kill.
An unexpected failure or hang of the OCSSD process, this can be caused by any of the above issues or something else.
An Oracle bug.
3.2 - FILES TO REVIEW AND GATHER FOR OCSSD EVICTIONS

All files from section 2.0 from all cluster nodes.  More data may be required.

Example of an eviction due to loss of voting disk:

CSS log:

2012-03-27 22:05:48.693: [ CSSD][1100548416](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 3 configured voting disks available, need 2
2012-03-27 22:05:48.693: [ CSSD][1100548416]###################################
2012-03-27 22:05:48.693: [ CSSD][1100548416]clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread

OS messages:

Mar 27 22:03:58 choldbr132p kernel: Error:Mpx:All paths to Symm 000190104720 vol 0c71 are dead.
Mar 27 22:03:58 choldbr132p kernel: Error:Mpx:Symm 000190104720 vol 0c71 is dead.
Mar 27 22:03:58 choldbr132p kernel: Buffer I/O error on device sdbig, logical block 0

4.0 - TROUBLESHOOTING CSSDAGENT OR CSSDMONITOR EVICTIONS

If you have encountered a CSSDAGENT or CSSDMONITOR eviction review common causes in section 4.1 below.

4.1 - COMMON CAUSES OF CSSDAGENT OR CSSDMONITOR EVICTIONS
An OS scheduler problem.  For example, if the OS is getting locked up in a driver or hardware or there is excessive amounts of load on the machine (at or near 100% cpu utilization), thus preventing the scheduler from behaving reasonably.
A thread(s) within the CSS daemon hung.
An Oracle bug.
4.2 - FILES TO REVIEW AND GATHER FOR CSSDAGENT OR CSSDMONITOR EVICTIONS

All files from section 2.0 from all cluster nodes. More data may be required.

Reference metalink Doc ID 1050693.1

Enable archivelog mode in RAC

Enable archivelog mode in RAC

1. stop the database service.
srvctl stop database -d MAPROD

2. start the database in mount state.
srvctl start database -d MAPROD -o mount

3. enable archive log mode.
alter database archivelog;

4. Restart the database service (using srvctl)
srvctl stop database -d MAPROD
srvctl start database -d MAPROD

Oracle RAC Upgrade From 11.2.0.4 to 12.1.0.2

Oracle RAC Upgrade From 11.2.0.4 to 12.1.0.2

Main steps:

GRID_HOME:

1.check all the services are up and running of GRIDHOME (11gr2)
crsctl check crs
crsctl status resource -t
2.check the GRID infrastructure software version
crsctl query crs activeversion
3.check database status and configuration
srvctl status database -d PROD
srvctl config database -d PROD
4.perform the local backup of OCR, GRID and DATABASE.
./ $GRID_HOME/bin/ocrconfig -export /u01/ocrmanulbkp
use tar command for GRID and DATABASE backup
5.Run the cluvfy utility from 12cr1 to pre-check any errors
./runclufy.sh stage -pre crsinst -upgrade -rolling -scr_crshome <11g loc> -dest_crshome <12c loc> -dest_version 12.1.0.2 -
versbose (-fixup) optionally
6.stop the running 11g database
srvctl stop database -d PROD
6.Unzip the 12c software, change the ownership and permissions
7.start the 12cr1 upgrade by executing ./runInstaller
-select upgrade GRID infrastructure with ASM
-check the existing grid homes
-provide the location of 12c grid home
8.execute the rootupgrade.sh script in both nodes from root user
9.check the clusterware upgrade version
crsctl query crs activeversion
10.start the 11g database
srvctl start database -d PROD
srvctl status database -d PROD

RDBMS_HOME:

1.backup the database before upgrade(important)
take level zero backup or cold backup
2.database pre-checks
-run cluvfy.sh stage -pre dbinst -upgrade -src_dbhome <11g> -dest_dbhome <12c> dest_version 12.1.0.2.0
-unzip the database software
-execute runInstaller
-select the RAC cluster database
-provide the 12c binaries home location
-run the root.sh script on both the nodes with root user

DATABASE:

1.perform the pre-checks
-check the invalid objects
select count(*) from dba_objects where status='INVALID';
-check the duplicate objects owned by sys and system
other than 4 we need to cleanup
2.run the preupgrade tool
copy the prequprd.sql and utluppkg.sql from 12c and run on 11g home
preupgrd.sql will generate 3 files, prequpgrade.log, preupgrade_fixups.sql and postupgrade_fixups.sql
3.execute the preupgrade_fixups.sq
it will set few parameters like processess, job_queue_processes, execute gather status and purge recyclebin
4.check the timezone version, for 12c database timezone is 18 and 11g is 14
select version from v$timezone_file;
5.make sure no files in begin backup mode and no files in media recovery
select * from v$recover_file;
6.disable cronjob and dbms_scheduler jobs
execute dbms_scheduler.disable;
7.remove EM repository because it is superseded in 12c
emctl stop dbcontrol
@?/rdbms/admin/emremove.sql
8.enable flashback by setting 2 parameters
db_recovery_file_dest_size and db_recovery_file_dest
alter database flashback on;
9.create restore point
create restore point bef_upgrade guarantee flashback database;
10.stop the 11g listener and database.
lsnrctl stop PROD
shut immediate;
11.update theORACLE_HOME paths pointing to 12c home
export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2
export PATH=$ORACLE_HOME/bin:$PATH
export ORACLE_SID=PROD
12.move spfile and password file from 11g home dbs directory to 12c home dbs directory
13.start the 12c database with startup upgrade option
startup upgrade;
14.run catupgrade script from os level with parallel=6
goto cd $ORACLE_HOME/rdbms/admin
$ORACLE_HOME/perl/bin/perl catctl.pl -n 6 -1 $ORACLE_HOME/diagnostics catupgrd.sql
check the logfiles for diagnosis the upgrde process
15.run the post upgrade status tool
startup
@utlu121s.sql
16.run the catuppst.sql script
$ORACLE_HOME/rdbms/admin/catuppst.sql
16.upgrade DST timezone
downlod the dst upgrade script from metalink
17.run the postupgrade_fixups.sql script
18.change the ORACLE_HOME to 12c in listener.ora file
19.uncomment the crontab and enable the dba_scheduler_jobs
18.run utlrp.sql to compile to invalid objects
19.update the compatible parameter and bounce the db
alter system set compatibe='12.1.0.2' scope=spfile;
Once compatible is set to higher version, downgrade is not possible

IF UPGRADE FAILS:

If you have done the prechecks properly, then chances are very less that, upgrade willl fail. but enven if it is fails for any other reasons like server crash during upgrade, then follow below steps to revert back to 11g version.

1.shutdown immediate;
2.set ORACLE_HOME to 11g
3.startup mount (with the 11g spfile)
4.select * from v$restore_point;
5.flashback database to restore point bef_upgrade (this restore point was created before upgrade)
6.alter database open resetlogs;

Oracle RAC Patching Steps In 11gr2

Oracle RAC Patching Steps In 11gr2

Patch is a software code.

-To fix the bugs
-Defects in software
-It delivers new features/tools

Disadvantages:

-Downtime

Classifications of patches:

-Security patche (CPU/SPU)
-PSU

Pre-requisite before patch (PPA):

-check the invalid objets
-patch conflicts
-patch dependencies
-patch status (applied or not)
-check opatch version (6880880)

GRID patching:

While applying patches on GRID home, bydefault the GRID home is locked

cd $GRID_HOME/crs/install

#./rootcrs.pl -unlock

-it will stop the grid services.
-unlock the GRID home.

$Opatch apply -local
after applying the patch

cd $GRID_HOME/crs/install
#rootcrs.pl -patch

-it will start the GRID services
-it will lock the GRID home.

Apply the GRID patch on node2 with above steps

Patching Steps:

GRID_HOME patching:

-export PATH=$ORACLE_HOME/OPatch:$PATH:.
-check the OPatch version
-check the patch conflicts
-check the invalid objects
-Run rootcrs.pl -unlock script which is located in $GRID_HOME/crs/install with root privileges
-finally apply with OPatch
-OPatch apply -local

RDBMS_HOME patching:

-set the paths
-srvctl stop instance -d PROD -i PROD1
-OPatch apply -local
-srvctl start instance -d PROD -i PROD1
-Run the catbundle.sql script

Note:
catbundle.sql script will update the dba_registry_history table
root.sh will be located in $GRID_HOME/crs/install

How to connect to DATABASE and ASM instance in RAC ?

How to connect to DATABASE and ASM instance in RAC ?

Database for 2 node RAC:

set the environment variable for rdbms.
export ORACLE_SID=ORCLPROD1
sqlplus '/as sysdba'

or

export ORACLE_SID=ORCLPROD2
sqlplus '/as sysdba'


ASM for 2 node RAC:

set the environment variable for ASM instance.
export ORACLE_SID=+ASM1
sqlplus / as sysasm

or

export ORACLE_SID=+ASM2
sqlplus /as sysasm

How to add a new disk to ASM Diskgroup

How to add a new disk to ASM Diskgroup 

[root@rac10 ~]# fdisk /dev/sdb

The number of cylinders for this disk is set to 12446.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
Partition 1 is already defined.  Delete it before re-adding it.

Command (m for help): ^C
[root@rac10 ~]# fdisk /dev/sdb

The number of cylinders for this disk is set to 12446.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
3
Invalid partition number for type `3'
Command action
   e   extended
   p   primary partition (1-4)
^C
[root@rac10 ~]# fdisk /dev/sdb

The number of cylinders for this disk is set to 12446.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (4866-12446, default 4866):
Using default value 4866
Last cylinder or +size or +sizeM or +sizeK (4866-12446, default 12446): +15g

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
[root@rac10 ~]#
[root@rac10 ~]#
[root@rac10 ~]#
[root@rac10 ~]#
[root@rac10 ~]# oracleasm exit
Unmounting ASMlib driver filesystem: failed
Unable to unmount ASMlib driver filesystem
[root@rac10 ~]# oracleasm init
[root@rac10 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@rac10 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@rac10 ~]# oracleasm listdisks
DATAQ
OCRVD
[root@rac10 ~]# fdisk -l

Disk /dev/sda: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        3187    25599546   83  Linux
/dev/sda2            3188        4717    12289725   83  Linux
/dev/sda3            4718        5737     8193150   82  Linux swap / Solaris
/dev/sda4            5738       13054    58773802+   5  Extended
/dev/sda5            5738       13054    58773771   83  Linux

Disk /dev/sdb: 102.3 GB, 102374572032 bytes
255 heads, 63 sectors/track, 12446 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1217     9775521   83  Linux
/dev/sdb2            1218        4865    29302560   83  Linux
/dev/sdb3            4866        6690    14659312+  83  Linux

[root@rac10 ~]# oracleasm createdisk FRA /dev/sdb3  ------ only on one node
Writing disk header: done
Instantiating disk: done

Below steps on all nodes:

[root@rac10 ~]# oracleasm exit
Unmounting ASMlib driver filesystem: failed
Unable to unmount ASMlib driver filesystem
[root@rac10 ~]# oracleasm init
[root@rac10 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@rac10 ~]# oracleasm listdisks
DATAQ
FRA
OCRVD

[root@rac10 ~]# cat /proc/partitions

major minor  #blocks  name

   8        0  104857600 sda
   8        1   25599546 sda1
   8        2   12289725 sda2
   8        3    8193150 sda3
   8        4          1 sda4
   8        5   58773771 sda5
   8       16   99975168 sdb
   8       17    9775521 sdb1
   8       18   29302560 sdb2
   8       19   14659312 sdb3

[oracle@rac10 ~]$ export ORACLE_SID=+ASM1
[oracle@rac10 ~]$ asmca

Convert Single Instance to RAC using RCONFIG

Convert Single Instance to RAC using RCONFIG

In $ORACLE_HOME/assistants/rconfig/sampleXMLs directory, there are two templates ConvertToRAC_AdminManaged.xml and ConvertToRAC_PolicyManaged.xml that are used to convert a single instance database to RAC admin managed or policy managed database respectively.

Below are the prerequisites that should be completed.

1-> Ensure the clusterware is configured, up and running on the nodes.
2-> Oracle RDBMS binaries for RAC database must be installed.
3-> Active ASM instance across the nodes.
4-> Create and mount required ASM diskgroups.
5-> Stand alone database running on the local node.
6-> Stand alone DB and RAC must be of same db versions.

All the steps to be performed from RAC1 Node(cluster node)
Note: It is assumed that we are also having ORACLE_HOME for single instance (i.e dbhome_1) at RAC1

1-> #su - oracle
2-> $. oraenv
3-> $export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
4-> $export PATH=$ORACLE_HOME/bin:$PATH
5-> $dbca => for creating a database for single instance
6-> $export ORACLE_HOME=/u01/home/oracle/product/11.2.0/dbhome_1
7->  $export PATH=$ORACLE_HOME/bin:$PATH
8-> $cp $ORACLE_HOME/assistants/rconfig/sampleXMLs/ConvertToRAC_Adminmanged.xml  /convert1.xml

Convert parameters:

– Source $OH = OracleHome of non-rac database for SourceDBHome
– Destination $OH = OracleHome where the RAC database should be configured.
– SourceDBInfo SID
– List of nodes that should have rac instances running for the Admin Managed Cluster Database. Local Node should be the first node.
– Instance Prefix tag is optional starting with 11.2.
– Specify the type of storage to be used by RAC database (ASM or Cluster File system).
– Specify Database Area Location to be configured for RAC database.
– Specify Fast Recovery Area to be configured for RAC database.

Example:

9-> $vi convert1.xml
:se nu
7=> verify="ONLY" (for verify and after give NO for convert)
9=> /u01/app/oracle/product/11.2.0/db_1   (source db home location)
11=>/u01/app/oracle/product/11.2.0/db_1  (target db home location)
13=> SID="nonrac" (sid of non-rac db)
16=>password=manager
22=> name="rac1"    (cluster(rac) nodes)
23=> name="rac2"
28=> prefix>nonrac<    (after migration rconfig creates nonrac1,nonrac2 intances)
31=> type="ASM"      (storage type (only two are there (ASM|CFS)))
33=> +DATA (storage location)
35=> +DATA (FRA location)
:wq!

10-> $rconfig convert1.xml (Only for verifying whether successful migrate or not)

11-> Set ONLY->NO in convert1.xml and run the below again:

$rconfig convert1.xml (Now For Migration after giving NO or YES option)

Note:

The Convert verify option in the ConvertToRAC.xml file has three options:

Convert verify="YES": rconfig performs checks to ensure that the prerequisites for single-instance to Oracle RAC conversion have been met before it starts conversion
Convert verify="NO": rconfig does not perform prerequisite checks, and starts conversion
Convert verify="ONLY" rconfig only performs prerequisite checks; it does not start conversion after completing prerequisite checks

Post conversion steps:

1-> Adjust the SGA for the RAC database, at least 15% for buffer cache and shared pool.
2-> Create SERVICE and apply TAF & Load balancing options, if required.
3-> Backup the RAC database.
4-> Backup OCR.
5-> Remove single-instance database.

How to resume from a failed conversion:

1-> Simply rerun the command if the rconfig exits in the mid-of the conversion for any fatal reasons.
2-> Perform clean-up operations on converted instance before reinitializing the conversion procedure.
3-> You can manually add the resources to CRS, in case rconfig utility fails to perform the final steps.

Logs to monitor:

1-> Single instance database alert log file.
2-> RAC database alert log file.
3-> $ORACLE_BASE/cfgtools/rconfig or /u01/app/oracle/cfgtoollogs/rconfig [records entire operation step-by-step]
4-> For RMAN log and SQL log, check the respective directories under /u01/app/oracle/cfgtoollogs/rconfig/db


Rconfig FAQ's:

What is the best scenario for using RCONFIG?

RCONFIG invokes RMAN internally to back up  the database to proceed with converting non-ASM to ASM.  This is time consuming task if the DB is in Terabyte size  and it will cause longer outage.
Hence this is good option only for small database and customer allows significant amount downtime.

How do we improve the conversion time with RCONFIG?

We can increase RMAN backup channel to reduce the backup time. Again it depends on how many number of CPU you have in your node. Here is the command to adjust the parallelism.  The  channel can be adjusted according to your server capacity.

RMAN> CONFIGURE DEVICE TYPE DISK PARALLELISM 6;

What does RCONFIG do during the conversion?

Migrate the database to ASM storage if we specified in XML file
Create Database Instances on all nodes in the cluster
Configure Listener and NetService entries
Configure and register CRS resources
Start the instances on all nodes in the cluster.
What are the Pre-requisite for RCONFIG utility?
Cluster should  installed and all the cluster process is up and running on all the nodes.
The oracle RAC RDBMS is installed on all nodes.
ASM disk must be mounted across all nodes if we want to migrate from file system to ASM.
The source and target should be in same home and same OS version.
Make sure, ASM disk has enough storage to migrate all the data files
How do we test RCONFIG before we do real conversion?

We have convert verify parameter in XML file and it has below options. Please choose ONLY.

Convert verify="YES"
Runs through verification checks to ensure that the
prerequisites for single-instance to RAC db
conversion have been met before it acutually starts
the conversion.

Convert verify="NO"
Bypass the prerequisite verification checks, and
starts conversion straight away

Convert verify="ONLY"
Do not perform the acutal conversion process.
Rather, just run through the prerequisite verification
checks and ends.

What can do if it fails middle of the conversion?

Don't get panic.  This will perform the clean-up operation on converted Instance or will delete the files created by earlier run.

How do we monitor conversion process during the migration?

There are few logs and we can tail the log during the process. We can monitor (tail -f)
single instance alert log, RAC database alert log, rconfig alert log, rman logs. Log info will give better idea and we can see what is going on.

How do we convert using RCONFIG if the stand alone DB is in different host?

You would accomplish this task by first restoring single instance database  in one of RAC node and , then you would use RCONFIG  to convert to RAC option.

Cluster Verification Utility (CLUVFY )

Cluster Verification Utility (CLUVFY )

The Cluster Verification Utility (CVU) performs system checks in preparation for installation, patch updates, or other system changes. Using CVU ensures that you have completed the required system configuration and preinstallation steps so that your Oracle Grid Infrastructure or Oracle Real Application Clusters (Oracle RAC) installation, update, or patch operation completes successfully.


1. Pre-check for CRS installation


Use the cluvfy stage -pre crsinst command to check the specified nodes before installing Oracle Clusterware. CVU performs additional checks on OCR and voting disks if you specify the -c and -q options.

cluvfy stage -pre crsinst -n node1,node2 -verbose

2. Post-Check for CRS Installation


Use the cluvfy stage -post crsinst command to check the specified nodes after installing Oracle Clusterware.

cluvfy stage -post crsinst -n node1,node2 -verbose

3. Post-check for hardware and operating system


-- Use the cluvfy stage -post hwos stage verification command to perform network and storage verifications on the specified nodes in the cluster before installing
   Oracle software. This command also checks for supported storage types and checks each one for sharing.

cluvfy stage -post hwos -n node_list [-s storageID_list] [-verbose]
cluvfy.sh stage -post hwos -n node1,node2 -verbose

4. Pre-check for ACFS Configuration


-- the cluvfy stage -pre acfscfg command to verify your cluster nodes are set up correctly before configuring Oracle ASM Cluster File System (Oracle ACFS).

cluvfy stage -pre acfscfg -n node_list [-asmdev asm_device_list] [-verbose]
cluvfy stage -pre acfscfg -n node1,node2 -verbose

5. Post-check for ACFS Configuration


-- Use the cluvfy stage -post acfscfg to check an existing cluster after you configure Oracle ACFS.

cluvfy stage -post acfscfg -n node_list [-verbose]
cluvfy stage -post acfscfg -n node1,node2 -verbose

6. Pre-check for OCFS2 or OCFS


-- Use the cluvfy stage -pre cfs stage verification command to verify your cluster nodes are set up correctly before setting up OCFS2 or OCFS for Windows.

cluvfy stage -pre cfs -n node_list -s storageID_list [-verbose]
cluvfy stage -pre cfs -n node1,node2 -verbose

7. Post-check for OCFS2 or OCFS


-- Use the cluvfy stage -post cfs stage verification command to perform the appropriate checks on the specified nodes after setting up OCFS2 or OCFS for Windows.

cluvfy stage -post cfs -n node_list -f file_system [-verbose]
cluvfy stage -post cfs -n node1,node2 -verbose

8. Pre-check for database configuration


-- Use the cluvfy stage -pre dbcfg command to check the specified nodes before configuring an Oracle RAC database to verify whether your system meets all of the
   criteria for creating a database or for making a database configuration change.

cluvfy stage -pre dbcfg -n node_list -d Oracle_home [-fixup [-fixupdir fixup_dir]] [-verbose]
cluvfy stage -pre dbcfg -n node1,node2 -d Oracle_home -verbose

9. Pre-check for database installation


-- Use the cluvfy stage -pre dbinst command to check the specified nodes before installing or creating an Oracle RAC database to verify that your system meets all of
   the criteria for installing or creating an Oracle RAC database.

cluvfy stage -pre dbinst -n node_list [-r {10gR1 | 10gR2 | 11gR1 | 11gR2}]  [-osdba osdba_group] [-d Oracle_home] [-fixup [-fixupdir fixup_dir] [-verbose] 

10. Pre-check for configuring Oracle Restart


-- Use the cluvfy stage -pre hacfg command to check a local node before configuring Oracle Restart.

cluvfy stage -pre hacfg [-osdba osdba_group] [-orainv orainventory_group] [-fixup [-fixupdir fixup_dir]] [-verbose]
cluvfy stage -pre hacfg -verbose

11. Post-check for configuring Oracle Restart


-- Use the cluvfy stage -post hacfg command to check the local node after configuring Oracle Restart.

cluvfy stage -post hacfg [-verbose]
cluvfy stage -post hacfg -verbose

12. Pre-check for add node.


/*Use the cluvfy stage -pre nodeadd command to verify the specified nodes are configured correctly before adding them to your existing cluster, and to verify the integrity of the cluster before you add the nodes.

This command verifies that the system configuration, such as the operating system version, software patches, packages, and kernel parameters, for the nodes that you want to add, is compatible with the existing cluster nodes, and that the clusterware is successfully operating on the existing nodes. Run this node on any node of the existing cluster.
*/

cluvfy stage -pre nodeadd -n node_list [-vip vip_list]  [-fixup [-fixupdir fixup_dir]] [-verbose]
cluvfy stage -pre nodeadd -n node1,node2 -verbose

13. Post-check for add node.


/*
Use the cluvfy stage -post nodeadd command to verify that the specified nodes have been successfully added to the cluster at the network, shared storage, and clusterware levels.
*/

cluvfy stage -post nodeadd -n node_list [-verbose]
cluvfy stage -post nodeadd -n node1,node2 -verbose

14. Post-check for node delete.


/*
Use the cluvfy stage -post nodedel command to verify that specific nodes have been successfully deleted from a cluster. Typically, this command verifies that the node-specific interface configuration details have been removed, the nodes are no longer a part of cluster configuration, and proper Oracle ASM cleanup has been performed.
*/

cluvfy stage -post nodedel -n node_list [-verbose]
cluvfy stage -post nodedel -n node1, node2 -verbose


15. Check ACFS integrity


-- Use the cluvfy comp acfs component verification command to check the integrity of Oracle ASM Cluster File System on all nodes in a cluster.

cluvfy comp acfs [-n [node_list] | [all]] [-f file_system] [-verbose]
cluvfy comp acfs -n node1,node2 -f /acfs/share -verbose

16. Checks user accounts and administrative permissions


/*
Use the cluvfy comp admprv command to verify user accounts and administrative permissions for installing Oracle Clusterware and Oracle RAC software, and for creating an Oracle RAC database or modifying an Oracle RAC database configuration.
*/

cluvfy comp admprv [-n node_list]
{ -o user_equiv [-sshonly] |
 -o crs_inst [-orainv orainventory_group] |
 -o db_inst [-osdba osdba_group] [-fixup [-fixupdir fixup_dir]] |
 -o db_config -d oracle_home [-fixup [-fixupdir fixup_dir]] }
 [-verbose]

17. Check ASM integrity


Use the cluvfy comp asm component verification command to check the integrity of Oracle Automatic Storage Management (Oracle ASM) on all nodes in the cluster. This check ensures that the ASM instances on the specified nodes are running from the same Oracle home and that asmlib, if it exists, has a valid version and ownership.

cluvfy comp asm [-n node_list | all ] [-verbose]
cluvfy comp asm -n node1,node2 -verbose

18. Check CFS integrity


Use the cluvfy comp cfs component verification command to check the integrity of the clustered file system (OCFS for Windows or OCFS2) you provide using the -f option. CVU checks the sharing of the file system from the nodes in the node list.

cluvfy comp cfs [-n node_list] -f file_system [-verbose]
cluvfy comp cfs -n node1,node2 -f /ocfs2/share -verbose

19. Check Clock Synchronization


Use the cluvfy comp clocksync component verification command to clock synchronization across all the nodes in the node list. CVU verifies a time synchronization service is running (Oracle Cluster Time Synchronization Service (CTSS) or Network Time Protocol (NTP)), that each node is using the same reference server for clock synchronization, and that the time offset for each node is within permissible limits.

cluvfy comp clocksync [-noctss] [-n node_list [all]] [-verbose]
cluvfy comp clocksync [-noctss] [-n node_list [all]] [-verbose]

-noctss
If you specify this option, then CVU does not perform a check on CTSS. Instead, CVU checks the platform's native time synchronization service, such as NTP.

20. Check cluster integrity


Use the cluvfy comp clu component verification command to check the integrity of the cluster on all the nodes in the node list.

cluvfy comp clu [-n node_list] [-verbose]
cluvfy comp clu -n node1,node2 -verbose

21. Check cluster manager integrity


Use the cluvfy comp clumgr component verification command to check the integrity of cluster manager subcomponent, or Oracle Cluster Synchronization Services (CSS), on all the nodes in the node list.

cluvfy comp clumgr [-n node_list] [-verbose]
cluvfy comp clumgr -n node1, node2 -verbose

22. Check CRS integrity


Run the cluvfy comp crs component verification command to check the integrity of the Cluster Ready Services (CRS) daemon on the specified nodes.

cluvfy comp crs [-n node_list] [-verbose]
cluvfy comp crs -n node1, node2 -verbose

23. Check DHCP


Starting with Oracle Database 11g release 2 (11.2.0.2), use the cluvfy comp dhcp component verification command to verify that the DHCP server exists on the network and is capable of providing a required number of IP addresses. This verification also verifies the response time for the DHCP server. You must run this command as root.

# cluvfy comp dhcp -clustername cluster_name [-vipresname vip_resource_name] [-port dhcp_port] [-n node_list] [-verbose]

-clustername cluster_name
The name of the cluster of which you want to check the integrity of DHCP.

-vipresname vip_resource_name
The name of the VIP resource.

-port dhcp_port
The port on which DHCP listens. The default port is 67.

24. Check DNS

Starting with Oracle Database 11g release 2 (11.2.0.2), use the cluvfy comp dns component verification command to verify that the Grid Naming Service (GNS) subdomain delegation has been properly set up in the Domain Name Service (DNS) server.

Run cluvfy comp dns -server on one node of the cluster. On each node of the cluster run cluvfy comp dns -client to verify the DNS server setup for the cluster.

25. Check HA integrity

Use the cluvfy comp ha component verification command to check the integrity of Oracle Restart on the local node.

cluvfy comp ha [-verbose]
cluvfy comp ha -verbose

26. Check space availability

Use the cluvfy comp freespace component verification command to check the free space available in the Oracle Clusterware home storage and ensure that there is at least 5% of the total space available. For example, if the total storage is 10GB, then the check ensures that at least 500MB of it is free.

cluvfy comp freespace [-n node_list | all]
cluvfy comp freespace -n node1, node2

27. Check GNS

Use the cluvfy comp gns component verification command to verify the integrity of the Oracle Grid Naming Service (GNS) on the cluster.

cluvfy comp gns -precrsinst -domain gns_domain -vip gns_vip [-n node_list]  [-verbose]

cluvfy comp gns -postcrsinst [-verbose]

28. Check GPNP

Use the cluvfy comp gpnp component verification command to check the integrity of Grid Plug and Play on all of the nodes in a cluster.

cluvfy comp gpnp [-n node_list] [-verbose]
cluvfy comp gpnp -n node1,node2 -verbose

29. Check healthcheck

Use the cluvfy comp healthcheck component verification command to check your Oracle Clusterware and Oracle Database installations for their compliance with mandatory requirements and best practices guidelines, and to ensure that they are functioning properly.

cluvfy comp healthcheck [-collect {cluster|database}] [-db db_unique_name]
 [-bestpractice|-mandatory] [-deviations] [-html] [-save [-savedir directory_path]]

30. Checks node applications existence

Use the component cluvfy comp nodeapp command to check for the existence of node applications, namely VIP, NETWORK, ONS, and GSD, on all of the specified nodes.

cluvfy comp nodeapp [-n node_list] [-verbose]
cluvfy comp nodeapp -n node1, node2 -verbose

31. Check node connectivity

Use the cluvfy comp nodecon component verification command to check the connectivity among the nodes specified in the node list. If you provide an interface list, then CVU checks the connectivity using only the specified interfaces.

cluvfy comp nodecon -n node_list [-i interface_list] [-verbose]
cluvfy comp nodecon -i eth2 -n node1,node2 -verbose
cluvfy comp nodecon -i eth3 -n node1,node2 -verbose

32. Checks reachability between nodes

Use the cluvfy comp nodereach component verification command to check the reachability of specified nodes from a source node.

cluvfy comp nodereach -n node_list [-srcnode node] [-verbose]

-srcnode node
The name of the source node from which CVU performs the reachability test. If you do not specify a source node, then the node on which you run the command is used as the source node.

33. Check OCR integrity

Use the cluvfy comp ocr component verification command to check the integrity of Oracle Cluster Registry (OCR) on all the specified nodes.

cluvfy comp ocr [-n node_list] [-verbose]
cluvfy comp ocr -n node1,node2 -verbose

34. Check OHASD integrity

Use the cluvfy comp ohasd component verification command to check the integrity of the Oracle High Availability Services daemon.

cluvfy comp ohasd [-n node_list] [-verbose]
cluvfy comp ohasd -n node1,node2 -verbose

35. Check OLR integrity

Use the cluvfy comp olr component verification command to check the integrity of Oracle Local Registry (OLR) on the local node.

cluvfy comp olr [-verbose]
cluvfy comp olr -verbose

36. Check node comparison and verification

Use the cluvfy comp peer component verification command to check the compatibility and properties of the specified nodes against a reference node. You can check compatibility for non-default user group names and for different releases of the Oracle software. This command compares physical attributes, such as memory and swap space, as well as user and group values, kernel settings, and installed operating system packages.

cluvfy comp peer -n node_list [-refnode node]  [-r {10gR1 | 10gR2 | 11gR1 | 11gR2}] [-orainv orainventory_group]  [-osdba osdba_group] [-verbose]

-refnode
The node that CVU uses as a reference for checking compatibility with other nodes. If you do not specify this option, then CVU reports values for all the nodes in the node list.

37. Checks SCAN configuration

Use the cluvfy comp scan component verification command to check the Single Client Access Name (SCAN) configuration.

cluvfy comp scan -verbose

38. Checks software component verification

Use the cluvfy comp software component verification command to check the files and attributes installed with the Oracle software.

cluvfy comp software [-n node_list] [-d oracle_home] [-r {10gR1 | 10gR2 | 11gR1 | 11gR2}] [-verbose]

39. Checks space availability

Use the cluvfy comp space component verification command to check for free disk space at the location you specify in the -l option on all the specified nodes.

cluvfy comp space [-n node_list] -l storage_location -z disk_space {B | K | M | G} [-verbose]

cluvfy comp space -n all -l /u01/oracle -z 2g -verbose

40. Checks shared storage accessibility

Use the cluvfy comp ssa component verification command to discover and check the sharing of the specified storage locations. CVU checks sharing for nodes in the node list.

cluvfy comp ssa [-n node_list] [-s storageID_list] [-t {software | data | ocr_vdisk}] [-verbose]

cluvfy comp ssa -n node1,node2 -verbose
cluvfy comp ssa -n node1,node2 -s /dev/sdb

41. Check minimum system requirements

Use the cluvfy comp sys component verification command to check that the minimum system requirements are met for the specified product on all the specified nodes.

cluvfy comp sys [-n node_list] -p {crs | ha | database}  [-r {10gR1 | 10gR2 | 11gR1 | 11gR2}] [-osdba osdba_group]  [-orainv orainventory_group] [-fixup [-fixupdir fixup_dir]] [-verbose]

cluvfy comp sys -n node1,node2 -p crs -verbose
cluvfy comp sys -n node1,node2 -p ha -verbose
cluvfy comp sys -n node1,node2 -p database -verbose

42. Check Voting Disk Udev settings

Use the cluvfy comp vdisk component verification command to check the voting disks configuration and the udev settings for the voting disks on all the specified nodes.

cluvfy comp vdisk [-n node_list] [-verbose]
cluvfy comp vdisk -n node1,node2 -verbose

43. Run cluvfy before doing an upgrade

runcluvfy stage -pre crsinst -upgrade -n  -rolling -src_crshome  -dest_crshome  -dest_version  -verbose
runcluvfy stage -pre crsinst -upgrade -n rac1,rac2 -rolling -src_crshome /u01/app/grid/11.2.0.1 -dest_crshome /u01/app/grid/11.2.0.3 -dest_version 11.2.0.4.0 -verbose

44. Strace the command

Strace the command to get more details
eg: strace -t -f -o clu.trc cluvfy comp olr -verbose
/*
[oracle@rac1 ~]$ strace -t -f -o clu.trc cluvfy comp olr -verbose

Verifying OLR integrity

Checking OLR integrity...

Checking OLR config file...

OLR config file check successful


Checking OLR file attributes...

OLR file check successful


WARNING:
This check does not verify the integrity of the OLR contents. Execute 'ocrcheck -local' as a privileged user to verify the contents of OLR.

OLR integrity check passed

Verification of OLR integrity was successful.
[oracle@rac1 ~]$ ls -ltr clu.trc
-rw-r--r-- 1 oracle oinstall 4206376 Jun 12 01:15 clu.trc
[oracle@rac1 ~]$

Oracle RAC 11gr2 Installation Steps

Oracle RAC 11gr2 Installation Steps

Basic Requirements and Assumptions:

Min 2/3 nodes
storage


Node1
- Linux Machine
Partitions
-- 100g
/ - 25g
swap - 10g
/tmp - 10g
/u01 - 50g
Network card -2 NIC
Ram  - 4gb
cpu - 1


Node2
- Linux Machine
Partitions
-- 100g
/ - 25g
swap - 10g
/tmp - 10g
/u01 - 50g
Network card -2 NIC
Ram  - 4gb
cpu - 1



Node3
- Linux Machine
Partitions
-- 100g
/ - 25g
swap - 10g
/tmp - 10g
/u01 - 50g
Network card -2 NIC
Ram  - 4gb
cpu - 1

Storage

openfiler(Linux)
starwind

Openfiler
10g storage
Partitions
- / - 8g
- swap - 102

ram - 512M
NIC - 1
cpu - 1
+ Extra storage 100g

1) OpenFiler Configuration:

http://mammqm.blogspot.com/2019/07/openfiler-installation-for-rac-setup.html

2) Define your IQN:

iscsiadm -m discovery -t st -p 147.43.0.10 (on all nodes)
service iscsi restart
service iscsi status

3) setting up host file:

Setting up host file:
vi /etc/hosts

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
##-- Public-IP
192.168.0.1    rac10.oracle.com   rac10
192.168.0.2    rac20.oracle.com   rac20
192.168.0.3    rac30.oracle.com   rac30
##-- Private-IP
147.43.0.1 rac10-priv.oracle.com rac10-priv
147.43.0.2 rac20-priv.oracle.com rac20-priv
147.43.0.3 rac30-priv.oracle.com rac30-priv
##-- Virtual-IP
192.168.0.4 rac10-vip.oracle.com rac10-vip
192.168.0.5 rac20-vip.oracle.com rac20-vip
192.168.0.6 rac30-vip.oracle.com rac30-vip
##-- SCAN IP
192.168.0.7 oracle-scan.oracle.com oracle-scan
192.168.0.8 oracle-scan.oracle.com oracle-scan
192.168.0.9 oracle-scan.oracle.com oracle-scan
##-- Storage-IP
192.168.0.10    san.oracle.com    oracle-san

4) Download and Install ASMlib

rpm -ivh oracleasmlib-2.0.4-1.el5.i386.rpm --force --nodeps
rpm -ivh oracleasmlib-2.0.4-1.el5.x86_64.rpm --force --nodeps
rpm -ivh oracleasm-support-2.1.7-1.el5.i386.rpm --force --nodeps

5) Deletion of bydefault users and groups:

userdel -r oracle
groupdel dba
groupdel oinstall

6) User and groups creation:

groupadd -g 501 oinstall
groupadd -g 502 dba
groupadd -g 503 asmadmin
useradd -u 504 -g oinstall -G dba,asmadmin -m oracle
chown -R oracle:dba /u01
chmod -R 775 /u01
passwd oracle

7) Stop the ntpd services on all the nodes:

[root@rac10 ~]# mv /etc/ntp.conf /etc/ntp.conf_bkp
[root@rac10 ~]# service ntpd restart
Shutting down ntpd:                                        [FAILED]

8) Disk Partitioning using fdisk

[root@rac10 u01]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

The number of cylinders for this disk is set to 12446.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-12446, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-12446, default 12446): +10g

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

# partprobe /dev/sdb --------------on all nodes

[root@rac10 u01]# oracleasm configure -i ------ on all nodes
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done

[root@rac10 u01]# oracleasm exit ----- stopping the services
[root@rac10 u01]# oracleasm init ----- starting the services

Example:

[root@rac10 u01]# oracleasm exit
[root@rac10 u01]# oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Mounting ASMlib driver filesystem: /dev/oracleasm
[root@rac10 u01]# oracleasm createdisk OCR /dev/sdb1 (only on one node)
Writing disk header: done
Instantiating disk: done

below commands should be executed on all nodes:

#oracleasm exit
#oracleasm init
#oracleasm scandisks
#oracleasm listdisks

8) Check the newly created partitions:

[root@rac10 u01]# cat /proc/partitions
major minor  #blocks  name

   8        0  104857600 sda
   8        1   25599546 sda1
   8        2   12289725 sda2
   8        3    8193150 sda3
   8        4          1 sda4
   8        5   58773771 sda5
   8       16   99975168 sdb
   8       17    9775521 sdb1 ---- new

9) Run the cluvfy utility:

[oracle@rac10 grid]$ ./runcluvfy.sh stage -pre crsinst -n rac10,rac20,rac30  -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "rac10"
  Destination Node                      Reachable?           
  ------------------------------------  ------------------------
  rac30                                 yes                   
  rac20                                 yes                   
  rac10                                 yes                   
Result: Node reachability check passed from node "rac10"


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Status               
  ------------------------------------  ------------------------
  rac10                                 failed               
  rac20                                 failed               
  rac30                                 failed               
Result: PRVF-4007 : User equivalence check failed for user "oracle"

ERROR:
User equivalence unavailable on all the specified nodes
Verification cannot proceed

Pre-check for cluster services setup was unsuccessful on all the nodes.

Above error is the expected error, please proceed with Grid installation.

Oracle Grid Infrastructure Installation Steps

[root@rac10 u01]#  xhost +
[root@rac10 u01]# su - oracle
[oracle@rac10 u01] /u01/installation/grid
[oracle@rac10 u01]./runInstaller

Installation is failed with SCAN listeners, this is expected as we are not using DNS, please proceed with next.

Oracle Database Installation Steps

Oracle Database Creation Using DBCA

Homes and path sets for RDBMS and ASM instance :

RDBMS:

[oracle@rac1 ~]$ cat rdbms.env
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1
export PATH=$ORACLE_HOME/bin:$PATH:.

ASM:

[oracle@rac1 ~]$ cat grid.env
export ORACLE_HOME=/u01/app/11.2.0/grid
export PATH=$ORACLE_HOME/bin:$PATH:.