Showing posts with label DBA Issue's. Show all posts
Showing posts with label DBA Issue's. Show all posts

Tuesday, April 19, 2022

How to check the block corruptions in database ?

How to check the block corruptions in database ?



RMAN Level:

RMAN> backup validate check logical datafile <<<file_number>>>;
RMAN> backup validate check logical database;

SQLPLUS Level:

sqlplus / as sysdba
select * from v$database_block_corruption;

OS Level:

The following example shows a sample use of the command-line interface to this mode of DBVERIFY.

% dbv FILE=t_db1.dbf feedback=100 >replace the file

dbv FILE=/u05/oradb/TEST/db/apps_st/data/eqddasm.dbf FEEDBACK=100

Friday, October 9, 2020

Resize Operation Completed For File#

Resize Operation Completed For File# 


Symptoms:

File Extension Messages are seen in alert log.There was no explicit file resize DDL as well.

Resize operation completed for file# 45, old size 26M, new size 28M

Resize operation completed for file# 45, old size 28M, new size 30M

Resize operation completed for file# 45, old size 30M, new size 32M

Resize operation completed for file# 36, old size 24M, new size 26M


Changes:

NONE


Cause:

These file extension messages were result of diagnostic enhancement through unpublished to record automatic datafile resize operations in the alert log with a message of the form:

"File NN has auto extended from x bytes to y bytes"

This can be useful when diagnosing problems which may be impacted by a file resize. 


Solution:

In busy systems, the alert log could be completely flooded with file extension messages. A new Hidden parameter parameter "_disable_file_resize_logging" has been introduced through bug 18603375 to stop these messages getting logged into alert log.

(Unpublished) Bug 18603375 - EXCESSIVE FILE EXTENSION MESSAGE IN ALERT LOG 

Set the below parameter along with the fix.

SQL> alter system set "_disable_file_resize_logging"=TRUE ; (Its default value is FALSE)

The bug fix 18603375 is included in 12.1.0.2 onwards.


Reference metalink Doc ID 1982901.1

Monday, October 5, 2020

ORA-01940: Cannot Drop A User that is Currently Connected

ORA-01940: Cannot Drop A User that is Currently Connected


While dropping a user in oracle database , you may face below error. ORA-01940

Problem:

SQL> drop user SCOTT cascade

2 /

drop user SCOTT cascade

*

ERROR at line 1:

ORA-01940: cannot drop a user that is currently connected


Solution:

1. Find the sessions running from this userid:

SQL> SELECT SID,SERIAL#,STATUS from v$session where username='SCOTT';

SID SERIAL# STATUS

---------- ---------- --------

44 56381 INACTIVE

323 22973 INACTIVE

2. Kill the sessions:

SQL> ALTER SYSTEM KILL SESSION '44,56381' immediate;

System altered.

SQL> ALTER SYSTEM KILL SESSION '323,22973' immediate;

System altered.

SQL> SELECT SID,SERIAL#,STATUS from v$session where username='SCOTT';

no rows selected

3. Now the drop the user:

SQL> drop user SCOTT cascade

user dropped.


Tuesday, November 7, 2017

How To Drop An UNDO Tablespace

Following error while dropping the undo tablespace:

SQL> select tablespace_name,file_name from dba_data_files;

TABLESPACE_NAME                FILE_NAME
------------------------------ ---------------------------------------------------------------------
USERS                               D:\ORACLE\ORADATA\NOIDA\USERS01.DBF
UNDOTBS1                       D:\ORACLE\ORADATA\NOIDA\UNDOTBS01.DBF
SYSAUX                             D:\ORACLE\ORADATA\NOIDA\SYSAUX01.DBF
SYSTEM                            D:\ORACLE\ORADATA\NOIDA\SYSTEM01.DBF
EXAMPLE                          D:\ORACLE\ORADATA\NOIDA\EXAMPLE01.DBF

SQL> drop tablespace undotbs1;
drop tablespace undotbs1
*
ERROR at line 1:
ORA-30013: undo tablespace 'UNDOTBS1' is currently in use

As the error indicate that the undo tablespace is in use so i issue the following command.

SQL> alter tablespace undotbs1  offline;
alter tablespace undotbs1  offline
*
ERROR at line 1:
ORA-30042: Cannot offline the undo tablespace.

Therefore, to drop undo  tablespace, we have to perform following steps:

1.) Create new undo tablespace
2.) Make it defalut tablepsace and undo management manual by editting parameter file and restart it.
3.) Check the all segment of old undo tablespace to be offline.
4.) Drop the old tablespace.
5.) Change undo management to auto by editting parameter file and restart the database

Step 1 : Create Tablespace   :  Create undo tablespace undotbs2 

SQL> create undo tablespace UNDOTBS2 datafile  'D:\ORACLE\ORADATA\NOIDA\UNDOTBS02.DBF'  size 100M;
Tablespace created.

Step 2 : Edit the parameter file

SQL> alter system set undo_tablespace=UNDOTBS2 ;
System altered.

SQL> alter system set undo_management=MANUAL scope=spfile;
System altered.

SQL> shut immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup
ORACLE instance started.
Total System Global Area  426852352 bytes
Fixed Size                  1333648 bytes
Variable Size             360711792 bytes
Database Buffers           58720256 bytes
Redo Buffers                6086656 bytes
Database mounted.
Database opened.
SQL> show parameter undo_tablespace
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
undo_tablespace                      string      UNDOTBS2

Step 3: Check the all segment of old undo tablespace to be offline

SQL> select owner, segment_name, tablespace_name, status from dba_rollback_segs order by 3;

OWNER  SEGMENT_NAME                   TABLESPACE_NAME                STATUS
------ ------------------------------ ------------------------------ ----------------
SYS                 SYSTEM                                     SYSTEM                            ONLINE
PUBLIC       _SYSSMU10_1192467665$          UNDOTBS1                       OFFLINE
PUBLIC        _SYSSMU1_1192467665$           UNDOTBS1                       OFFLINE
PUBLIC        _SYSSMU2_1192467665$           UNDOTBS1                       OFFLINE
PUBLIC        _SYSSMU3_1192467665$           UNDOTBS1                       OFFLINE
PUBLIC        _SYSSMU4_1192467665$           UNDOTBS1                       OFFLINE
PUBLIC        _SYSSMU5_1192467665$           UNDOTBS1                       OFFLINE
PUBLIC        _SYSSMU6_1192467665$           UNDOTBS1                       OFFLINE
PUBLIC        _SYSSMU7_1192467665$           UNDOTBS1                       OFFLINE
PUBLIC        _SYSSMU8_1192467665$           UNDOTBS1                       OFFLINE
PUBLIC        _SYSSMU9_1192467665$           UNDOTBS1                       ONLINE
PUBLIC      _SYSSMU12_1304934663$          UNDOTBS2                        OFFLINE
PUBLIC      _SYSSMU13_1304934663$          UNDOTBS2                        OFFLINE
PUBLIC      _SYSSMU14_1304934663$          UNDOTBS2                        OFFLINE
PUBLIC      _SYSSMU15_1304934663$          UNDOTBS2                        OFFLINE
PUBLIC      _SYSSMU11_1304934663$          UNDOTBS2                        OFFLINE
PUBLIC      _SYSSMU17_1304934663$          UNDOTBS2                        OFFLINE
PUBLIC      _SYSSMU18_1304934663$          UNDOTBS2                        OFFLINE
PUBLIC      _SYSSMU19_1304934663$          UNDOTBS2                        OFFLINE
PUBLIC      _SYSSMU20_1304934663$          UNDOTBS2                        OFFLINE
PUBLIC      _SYSSMU16_1304934663$          UNDOTBS2                        OFFLINE

21 rows selected.

If any one the above segment is online then change it status to offline by using below command .
SQL>alter rollback segment "_SYSSMU9_1192467665$" offline;

Step 4 : Drop old undo tablespace

SQL> drop tablespace UNDOTBS1 including contents and datafiles;
Tablespace dropped.

Step  5 : Change undo management to auto and restart the database

SQL> alter system set undo_management=auto scope=spfile;
System altered.

SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area  426852352 bytes
Fixed Size                  1333648 bytes
Variable Size             364906096 bytes
Database Buffers           54525952 bytes
Redo Buffers                6086656 bytes
Database mounted.
Database opened.

SQL> show parameter undo_tablespace
NAME                                       TYPE        VALUE
------------------------------------   ----------- ------------------------------
undo_tablespace                      string      UNDOTBS2

Sunday, October 8, 2017

Not able to connect as rman catalog "RMAN-04004: error from recovery catalog database: ORA-12154: TNS:could not resolve the connect identifier specified"


If you have created catalog user and you want to connect to rman with that user and you are getting the following error-

RMAN-04004: error from recovery catalog database:
ORA-12154: TNS:could not resolve the connect identifier specified

see below errors--

rman catalog = catalog/metalog@catdb
Recovery Manager: Release 11.2.0.3.0 - Production on Tue Feb 23 15:02:03 2016
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00554: initialization of internal recovery manager package failed
RMAN-04004: error from recovery catalog database: ORA-12154: TNS:could not resolve the connect identifier specified

You are trying with connect identifier on same machine on which you have database like @catdb in this command.

Action - Don't use @catdb because if you are using same server to connect there is no need to give connect identifier.

export ORACLE_SID=catdb

-bash-4.1$ rman catalog = catalog/metalog

by giving this you will easily connect to your rman.

If you are trying to connect to catalog database on different host then you have to create tns entry in tnsnames.ora file .

like  ----

catdb =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = hostname )(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SID = DB_NAME)

Sunday, October 1, 2017

Terminating Instance Due To Error 472/ORA-00472

Terminating Instance Due To Error 472/ORA-00472

Cause:

When  user killed pmon process using kill -9 <pid> you will get this error in the alert log file.

MMAN: terminating instance due to error 472
ORA-00472: PMON process terminated with error
Instance terminated by MMAN, pid = 26700.

Terminating Instance Due To Error 474/ORA-00474

Terminating Instance Due To Error 474/ORA-00474

Cause:

When user killed smon process using kill -9 <pid> you will get this error in the alert log file.

Errors in file /u01/home/oracle/product/10.2.0/db_1/admin/PROD/bdump/prod_pmon_26800.trc:
ORA-00474: SMON process terminated with error
Mon Feb 25 15:51:26 2013
PMON: terminating instance due to error 474
Instance terminated by PMON, pid = 26800

Wednesday, August 3, 2016

ORA-609 : opiodr aborting process unknown ospid (8327_47148946930848)

ORA-609 : opiodr aborting process unknown ospid (8327_47148946930848)

As a general error, the ORA-609 error indicates that a client connection failed to complete.  This can be an ORA-609 from an abort or killing an Oracle session.

To diagnose any error, you start by using the OERR UTILITY to display the ORA-609 error:

Example :

bash-3.2$ oerr ora 609
00609, 00000, "could not attach to incoming connection"
// *Cause:  Oracle process could not answer incoming connection
// *Action: If the situation described in the next error on the stack
// can be corrected, do so; otherwise contact Oracle Support.

Cause:

The ORA-609 error is thrown when a client connection of any kind failed to complete or aborted the connection
process before the server process was completely spawned.
Beginning with 10gR2, a default value for inbound connect timeout has been set at 60 seconds.

This is also triggered, when a DB session is killed/aborted manually from the OS prompt.

Solution:

Increase the values for INBOUND_CONNECT_TIMEOUT at both listener and server side sqlnet.ora file as a preventive measure.
If the problem  is due to connection timeouts,an increase in the following parameters should eliminate or reduce the occurrence of the ORA-609s.
Sqlnet.ora: SQLNET.INBOUND_CONNECT_TIMEOUT=180
Listener.ora: INBOUND_CONNECT_TIMEOUT_listener_name=120

Reference metalink Doc ID 1121357.1

Monday, June 20, 2016

ORA-19802 Error: "Cannot Use Db_recovery_file_dest Without DB_RECOVERY_FILE_DEST_SIZE

ORA-19802 Error: "Cannot Use Db_recovery_file_dest Without DB_RECOVERY_FILE_DEST_SIZE

Error:

ORA-19802 error: "cannot use DB_RECOVERY_FILE_DEST without DB_RECOVERY_FILE_DEST_SIZE."

Cause:

The problem is caused by the fact that these two settings are interdependent and they must both be set to valid values.

Solution:

Ensure both parameters are properly set.

Use:

ALTER SYSTEM set DB_RECOVERY_FILE_DEST_SIZE

and/or

ALTER SYSTEM set DB_RECOVERY_FILE_DEST

Reference metalink (Doc ID 749595.1) 

Wednesday, May 11, 2016

ORA-1654: unable to extend index SYS.WRH$_BG_EVENT_SUMMARY_PK by 1024 in tablespace SYSTEM

ORA-1654: unable to extend index SYS.WRH$_BG_EVENT_SUMMARY_PK by 1024 in tablespace SYSTEM

While doing Datapump Import operation on RHEL 5.4 server everything went good but at the end of the job import operation experienced resumable wait with an error mentioned below.

ERROR:
ORA-01654: unable to extend index SYS.I_HH_OBJ#_COL# by 128 in tablespace SYSTEM
ORA-39171: Job is experiencing a resumable wait.

Solution :
Check the size and maxsize of the data files in the SYSTEM tablespace

SQL> select banner from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
PL/SQL Release 11.2.0.2.0 - Production
CORE    11.2.0.2.0      Production
TNS for Linux: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 – Production

SQL> select file_name, bytes, autoextensible, maxbytes from dba_data_files where tablespace_name='SYSTEM';

SQL> select sum(bytes)/1024/1024 MB from dba_free_space  where TABLESPACE_NAME='SYSTEM';

The SYSTEM tablespace has no space to allocate any more in it then we increased the size of the SYSTEM’s data file then our problem got solved.

SQL> alter database datafile '<Path to datafile>' resize <larger size> ;

or

SQL> alter tablespace system add datafile '<path of datafile>' size 2g;

Sunday, May 1, 2016

Error 'PLS-00801: internal error [1401]' received when recompiling packages

Error 'PLS-00801: internal error [1401]' received when recompiling packages

Error:

SQL> alter package OE_DEFAULT_LINE_PATTR compile body;

Warning: Package Body altered with compilation errors.

SQL> show errors
Errors for PACKAGE BODY OE_DEFAULT_LINE_PATTR:

LINE/COL ERROR
-------- -----------------------------------------------------------------
0/0 PLS-00801: internal error [1401]
0/0 PLS-00801: internal error [1401]
0/0 PLS-00801: internal error [1401]
0/0 PLS-00801: internal error [1401]
0/0 PLS-00801: internal error [1401]
1604/17 PL/SQL: Statement ignored
 -- Steps To Reproduce:
1. Open database SQL*Plus session
2. Run the below command to recompile package

alter package OE_DEFAULT_LINE_PATTR compile body;

Cause:

This issue is caused by inconsistencies in underlying application codelines and binaries as a result of recent patching and (potential) database upgrades.

Solution:

To implement the solution, please execute the following steps:

1. Review the comments in the 2 scripts below:

$ORACLE_HOME/rdbms/admin/utlirp.sql
$ORACLE_HOME/rdbms/admin/utlrp.sql

NOTE: Please note the requirements under "USAGE"to restart the database in UPGADE-mode before running script.

NAME
utlirp.sql - UTiLity script to Invalidate Pl/sql modules

DESCRIPTION
This script can be used to invalidate and all pl/sql modules
(procedures, functions, packages, types, triggers, views)
in a database.

This script must be run when it is necessary to regenerate the
compiled code because the PL/SQL code format is inconsistent with
the Oracle executable (e.g., when migrating a 32 bit database to
a 64 bit database or vice-versa).

Please note that this script does not recompile invalid objects
automatically. You must restart the database and explicitly invoke
utlrp.sql to recompile invalid objects.

USAGE
To use this script, execute the following sequence of actions:
1. Shut down the database and restart in UPGRADE mode
(using STARTUP UPGRADE or ALTER DATABASE OPEN UPGRADE)
2. Run this script
3. Shut down the database and restart in normal mode
4. Run utlrp.sql to recompile invalid objects. This script does
not automatically recompile invalid objects.

NOTES
* This script must be run using SQL*PLUS.
* You must be connected AS SYSDBA to run this script.
* This script expects the following files to be available in the
current directory:
standard.sql
dbmsstdx.sql
* There should be no other DDL on the database while running the
script. Not following this recommendation may lead to deadlocks.


NAME
utlrp.sql - Recompile invalid objects

DESCRIPTION
This script recompiles invalid objects in the database.

When run as one of the last steps during upgrade or downgrade,
this script will validate all remaining invalid objects. It will
also run a component validation procedure for each component in
the database. See the README notes for your current release and
the Oracle Database Upgrade book for more information about
using utlrp.sql

Although invalid objects are automatically re-validated when used,
it is useful to run this script after an upgrade or downgrade and
after applying a patch. This minimizes latencies caused by
on-demand recompilation. Oracle strongly recommends running this
script after upgrades, downgrades and patches.

2. Run these scripts using SQL*Plus as SYSDBA

3. Review your invalid listing for resolution

Reference metalink Doc ID 949222.1

Wednesday, April 20, 2016

ORA-01920 and ORA-02303 error occurs while running catmgdidcode.sql script

ORA-01920 and ORA-02303 error occurs while running catmgdidcode.sql script

Error:

CREATE USER mgdsys IDENTIFIED by mgdsys
  *
ERROR at line 1:
ORA-01920: user name 'MGDSYS' conflicts with another user or role name

CREATE OR REPLACE TYPE MGD_IDCOMPONENT TIMESTAMP '2005-05-26:10:23:10' OID 'F805ACEFD7883DD1E030578CBB054A48' wrapped
*
ERROR at line 1:
ORA-02303: cannot drop or replace a type with type or table dependents

No errors.
CREATE OR REPLACE TYPE MGD_IDCOMPONENT TIMESTAMP '2005-05-26:10:23:10' OID 'F805ACEFD7883DD1E030578CBB054A48' wrapped
*
ERROR at line 1:
ORA-02303: cannot drop or replace a type with type or table dependents

Cause:
The schema created by the script was already in existence.
This is evident from the error messages

Solution:

Use this workaround for this issue.

1. Run this script (this drops the MGDSYS)

@?/md/admin/catnomgdidcode.sql
2. Then run the below script:

@?/md/admin/catmgdidcode.sql

Reference metalink Doc ID 1544490.1

Sunday, April 17, 2016

ORA-00600: internal error code, arguments: [ktsxtffs2]

ORA-00600: internal error code, arguments: [ktsxtffs2]

ERROR:
Format: ORA-600 [ktsxtffs2]

When tried to search this error on the metalink with ORA lookup tool. It redirected to the following note.

ORA-600 [ktsxtffs2] [ID 1229336.1]

Its available in the Note ID: 153788.1 (Lookup Tool)

entered the following details in respective fields:

after entering these details, just click the Lookup Error button.

The above note reports the following bug.

Bug 8223165 – ORA-600 [ktsxtffs2] During Startup When Using Temporary Tablespace Group [ID 8223165.8]

The bug says:

Fixed:

This issue is fixed in * 12.1 (Future Release)

Symptoms:

-Internal Error May Occur (ORA-600)
-ORA-600 [ktsxtffs2]
-ORA-600 [kghstack_free1]
-Instance Startup

SMON will report ORA-600 [ktsxtffs2] and/or ORA-600 [kghstack_free1] when starting the database.

Solution:

Checked and found that a temporary tablespace group was being used.

SQL> select GROUP_NAME,TABLESPACE_NAME from DBA_TABLESPACE_GROUPS;

ROUP_NAME                     TABLESPACE_NAME
------------------------------ ------------------------------
TEMP                           TEMP1
TEMP                           TEMP2

SQL> select property_name, property_value from database_properties where PROPERTY_NAME='DEFAULT_TEMP_TABLESPACE';

PROPERTY_NAME                                          PROPERTY_VALUE
-------------                                          --------------
DEFAULT_TEMP_TABLESPACE                                TEMP

Please note that you can either have a “default tablespace group” or a “default tablespace” at a time. You can not have both at the same time.

There is no theoretical maximum limit to the number of tablespaces in a tablespace group, but it must contain at least one. The group is implicitly dropped when the last member is removed.

SQL> ALTER TABLESPACE TEMP1 TABLESPACE GROUP '';

Tablespace altered.

SQL> ALTER TABLESPACE TEMP2 TABLESPACE GROUP '';

ALTER TABLESPACE TEMP2 TABLESPACE GROUP ''

*
ERROR at line 1:

ORA-10919: Default temporary tablespace group must have at least one tablespace

The last member of a group cannot be removed if the group is still assigned as the default temporary tablespace. So we need to change the “default” from tablespace group “TEMP” to the tablespace “TEMP1”.

SQL> ALTER DATABASE DEFAULT TEMPORARY TABLESPACE TEMP1;
Database altered.

SQL> ALTER TABLESPACE TEMP2 TABLESPACE GROUP '';
Tablespace altered.

Once all the members have been removed from the group, the group is automatically dropped from the database.

SQL> select GROUP_NAME from DBA_TABLESPACE_GROUPS;
no rows selected

Reference metalink Doc
Note ID: 153788.1
Note ID: 8223165.8

Thursday, April 14, 2016

Shutdown Active Processes Prevent Shutdown Operation

Shutdown Active Processes Prevent Shutdown Operation

When shutdown immediate hanged for long, found in user dump a trace file containing messgaes

Trace file showing:

Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - Production
With the Partitioning, Oracle Label Security, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /oracle/app/product/11.1.0/db_1
...
*** 2010-03-30 09:25:58.372
*** SESSION ID:(170.15) 2010-03-30 09:25:58.372
*** CLIENT ID:() 2010-03-30 09:25:58.372
*** SERVICE NAME:(SYS$USERS) 2010-03-30 09:25:58.372
*** MODULE NAME:(sqlplus@piorovm.localdomain (TNS V1-V3)) 2010-03-30 09:25:58.372
*** ACTION NAME:() 2010-03-30 09:25:58.372
...
ksukia: Starting kill, force = 0
ksukia: killed 57 out of 57 processes.

*** 2010-03-30 09:26:03.421
ksukia: Starting kill, force = 0
ksukia: Attempt 1 to re-kill process OS PID=19840.
ksukia: Attempt 2 to re-kill process OS PID=19840..
.
.
.
$ ps -ef grep 19840
...
oracle 7426 11402 0 09:34 pts/1 00:00:00 sqlplus @a.sql
oracle 19840 7426 0 09:34 ? 00:00:00 [oracle] defunct

kill -9 19840

This process 19840 was defunct (dead) as it appeared defunct. So this could not be killed directly with kill -9, instead its parent process sqlplus was killed and shutdown immediate proceeded

kill -9 7426

Tuesday, April 12, 2016

EMCA Failing with Invalid Password for Dbsnmp

EMCA Failing with Invalid Password for Dbsnmp 

Error:

Running: <ORACLE_HOME>/bin/emca -config dbcontrol db -repos create

fails with:

Invalid username/password.
Password for DBSNMP user:

A sqlplus connenction with the same user and password works correctly.

SQL> connect dbsnmp/<password>
Connected.

Cause:

There are (at least) 2 possible causes:

1) The password contains special characters or local country character set characters
2) The ORACLE_HOME has a trailing slash.

Solution:

1. Wrap the password in double quotes " " (ie. if the password is dbnmp, use in emca input "dbsnmp")

or

2) Verify the ORACLE_HOME value to match the one in oratab:

ie.:
echo $ORACLE_HOME
/d01/oracle/product/10.2.0/db_1/  ->   notice the extra trailing  slash /

remove the training slash from ORACLE_HOME

set ORACLE_HOME=/d01/oracle/product/10.2.0/db_1
or
export ORACLE_HOME=/d01/oracle/product/10.2.0/db_1

and retry the EMCA command.

Reference metalink Doc ID 1264024.1

EMCA Fails With "Invalid username/password" for DBSNMP or SYSMAN user when Creating DBConsole

EMCA Fails With "Invalid username/password" for DBSNMP or SYSMAN user when Creating DBConsole 

Error:

The command emca fails with Invalid username/password for DBSNMP or for SYSMAN.

When you enter the password for SYSMAN or DBSNMP, you get the error message Invalid username/password.

[oracle@ldb009 10.2.0]$ emca -config dbcontrol db -repos create
STARTED EMCA at Sep 27, 2005 2:08:57 PM
EM Configuration Assistant, Version 10.2.0.1.0 Production
Copyright (c) 2003, 2005, Oracle.  All rights reserved.
Enter the following information:
Database SID: ldb009
Listener port number: 1523
Password for SYS user:
Password for DBSNMP user:
Invalid username/password.
Password for DBSNMP user:
Invalid username/password.
Password for DBSNMP user:
Invalid username/password.
Password for DBSNMP user:
Invalid username/password.

LOG FILE
----------
emca_2005-09-28_10-09-54-AM.log

Sep 28, 2005 10:13:40 AM oracle.sysman.emcp.util.GeneralUtil initSQLEngine
CONFIG: SQLEngine connecting with SID: ldb009, oracleHome: /oracle/app/10.2.0, and user: SYS
Sep 28, 2005 10:13:40 AM oracle.sysman.emcp.util.GeneralUtil initSQLEngine
CONFIG: SQLEngine created successfully and connected
Sep 28, 2005 10:13:40 AM oracle.sysman.emcp.DatabaseChecks validateUserCredentials
CONFIG: Failed to update account status.
oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-01034: ORACLE not available

 at oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeImpl(SQLEngine.java:1467)
 at oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeQuery(SQLEngine.java:694)
 at oracle.sysman.emcp.DatabaseChecks.updateAccountStatus(DatabaseChecks.java:1040)
 at oracle.sysman.emcp.DatabaseChecks.validateUserCredentials(DatabaseChecks.java:1013)
 at oracle.sysman.emcp.ParamsManager.validatePassword(ParamsManager.java:2694)
 at oracle.sysman.emcp.EMConfigAssistant.promptForData(EMConfigAssistant.java:583)
 at oracle.sysman.emcp.EMConfigAssistant.promptForParams(EMConfigAssistant.java:2231)
 at oracle.sysman.emcp.EMConfigAssistant.displayWarnsAndPromptParams(EMConfigAssistant.java:2257)
 at oracle.sysman.emcp.EMConfigAssistant.getDisplayAndPromptWarnsParms(EMConfigAssistant.java:2284)
 at oracle.sysman.emcp.EMConfigAssistant.performConfiguration(EMConfigAssistant.java:928)
 at oracle.sysman.emcp.EMConfigAssistant.statusMain(EMConfigAssistant.java:463)
 at oracle.sysman.emcp.EMConfigAssistant.main(EMConfigAssistant.java:412)

Similar errors about SYSMAN account can be encountered when running emca only for building the configuration files (SYSMAN and MGMT_VIEW exists and only "emca -config dbcontrol db" is being used).

Cause:

This error is mainly due to profile limits which prevent emca to reset the password for any of the following:

SYSMAN
DBSNMP
MGMT_VIEW
In Release 10.1, emca resets the password with the one provided at prompt:

SYSMAN => Always
DBSNMP=> Always
MGMT_VIEW=> Always
In Release 10.2, emca resets the password with the one provided at prompt:

SYSMAN => If password expired
DBSNMP=> If password expired
MGMT_VIEW=> Always

If the user has a profile limit for the resource type PASSWORD which is violated by the password provided, then emca will fail with the error "Invalid username/password".

In Release 11G, emca will prompt for the password only if the account is EXPIRED & LOCKED or if the account does not exist.
If the password is required, and if the password provided does not meet the password policy for the user profile, the error message will clearly state that there is a policy violation.

Solution:

In order to fix the Invalid username/password message

-Check the profile limit for the users SYSMAN, DBSNMP and MGMT_VIEW

select u.username, u.profile, p.resource_name, p.limit
from dba_profiles p, dba_users u
where p.profile=u.profile
and u.username in ('SYS', 'SYSMAN','DBSNMP','MGMT_VIEW')
and p.resource_type = 'PASSWORD'
order by u.username, p.resource_name;

-If the value of LIMIT for PASSWORD_VERIFY_FUNCTION is not NULL:
-Change the password so that it meets the function requirements.
-If the value of LIMIT for PASSWORD_REUSE_MAX is not UNLIMITED:

Change the password so that it is different from a password that has already been used the number of times set in PASSWORD_REUSE_MAX.
Or
Change the value of LIMIT for PASSWORD_REUSE_MAX to UNLIMITED for the profile.
Note: If SYSMAN and MGMT_VIEW user do not exist yet (repository was not created with -repos create option in emca), ensure that the password policies described above are met for those as well.

Reference metalink Doc ID 337260.1

Sunday, April 10, 2016

ORA-12537: TNS:connection closed' Errors Connecting To Oracle11g R1 on Linux via Oracle Net

ORA-12537: TNS:connection closed' Errors Connecting To Oracle11g R1 on Linux via Oracle Net

Suddently unable to connect to Oracle11g R1 (11.1.0.6) on Linux x86 via Oracle Net (i.e. using the TNS Listener).  The Oracle11g database instance is up but client cannot connect as listener service refuses connections.
sqlplus system/<password>@11gtest
SQL*Plus: Release 11.1.0.6.0 - Production on Fri Aug 22 11:59:54 2008
Copyright (c) 1982, 2007, Oracle. All rights reserved.
ERROR:
ORA-12537: TNS:connection closed
Corresponding error stack in the listener.log:

TNS-12518: TNS:listener could not hand off client connection
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
Linux Error: 32: Broken pipe

Cause:
Oracle user's stacksize was set to unlimited.

Solution:
To implement the solution, please execute the following steps:

1) Check the stacksize for the Oracle user to see if it is set to 'unlimited'
$ ulimit -a
For example:
ulimit -a
time(cpu-seconds) unlimited
file(blocks) unlimited
coredump(blocks) 0
data(kbytes) unlimited
stack(kbytes) unlimited <lockedmem(kbytes) 32
memory(kbytes) unlimited
nofiles(descriptors) 1000
processes 2047

2) If the stacksize for the Oracle user is set to 'unlimited', set the stacksize for the Oracle owner to 10240 kbytes.

For example:

ulimit -s 10240

Reference metalink Doc ID 733737.1

Saturday, April 9, 2016

Oracle Shutdown Immediate Waiting On Active Process

Oracle Shutdown Immediate Waiting On Active Process

Usually whenever you trying to shutdown an Oracle database with immediate option, it will wait for some active process to get terminate. This may take long time and it also depends on the number of processes.

use below command to find out the active sessions on database:

bash-3.00$ ps -ef | grep LOCAL=NO

oracle 17862     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17882     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17854     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17890     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17906     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17848     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17978     1   0   Jun 13 ?           0:03 oracletest (LOCAL=NO)
oracle 17908     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17918     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17910     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17852     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17922     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17982     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17970     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17866     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17930     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17914     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17874     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17932     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17986     1   0   Jun 13 ?           0:03 oracletest (LOCAL=NO)
oracle 17952     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17936     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17836     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17944     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17974     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17948     1   0   Jun 13 ?           0:05 oracletest (LOCAL=NO)
oracle 17898     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17878     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17876     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17900     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17924     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17902     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17892     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17894     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17956     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17966     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17940     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17834     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17960     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17934     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17884     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17916     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17860     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17868     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)
oracle 17856     1   0   Jun 13 ?           0:00 oracletest (LOCAL=NO)


You need to kill the process only with option LOCAL=NO. The below is the very useful UNIX command which can be used for killing these idle active session on database which are avoiding the database to shutdown with immediate option.

kill -9 `ps -ef | grep LOCAL=NO | grep <INSTANCE NAME> | grep -v grep | awk '{print $2}'`

Now, verify the processes again with ps -ef | grep LOCAL=NO it should not return any list, If these processes are killed successfully then the database will shutdown smoothly.

Note: It is strongly recommended to do on Test Instances before executing on Producion.










Troubleshooting Internal Errors (Error Look-up Tool)

Troubleshooting Internal Errors (Error Look-up Tool)

Assessing and resolving ORA-600 and ORA-7445 errors

If you’re an Oracle DBA, you’re likely to have come across an error message in your Oracle Database alert.log files prefixed by either ORA-600 or ORA-7445.

Example:
ORA 600 [ktfbtgex-7], [1015817]
ORA 600 [3600]
ORA 7445 [kewa_dump_time_diff()+157]

The internal error messages include no attached explanation in the way that external error messages do (for example, “ORA-00942: table or view does not exist”), it is difficult to assess the seriousness of the error and whether it is cause for concern.

Difference between ORA-600 and ORA-7445.

ORA-600 is a catchall message that indicates an error internal to the database code. The key point to note about an ORA-600 error is that it is signaled when a code check fails within the database. At points throughout the code, Oracle Database performs checks to confirm that the information being used in internal processing is healthy, that the variables being used are within a valid range, that changes are being made to a consistent structure, and that a change won’t put a structure into an unstable state. If a check fails, Oracle Database signals an ORA-600 error and, if necessary, terminates the operation to protect the health of the database.

The first argument to the ORA-600 error message indicates the location in the code where the check is performed; in the example above, that is ktfbtgex-7 (which indicates that the error occurred at a particular point during tablespace handling). The subsequent arguments have different meanings, depending on the particular check.

An ORA-7445 error, on the other hand, traps a notification the operating system has sent to a process and returns that notification to the user. Unlike the ORA-600 error, the ORA-7445 error is an unexpected failure rather than a handled failure.

The Oracle function in which that notification signal is received is usually, from Oracle Database 10g onward, contained in the ORA-7445 error message itself. For example, in the error message

ORA-07445: exception encountered:
core dump [kocgor()+96] [SIGSEGV]
[ADDR:0xF000000104] [PC:0x861B7EC]
[Address not mapped to object] []

the failing function is kocgor, which is associated with the handling of user-defined objects. The trapped signal is SIGSEGV (signal 11, segmentation violation), which is an attempt to write to an illegal area of memory. Another common signal is SIGBUS (signal 10, bus error), and there are other signals that occur less frequently, with causes that range from invalid pointers to insufficient OS resources.

Both ORA-600 and ORA-7445 errors will Write the error message to the alert.log, along with details about the location of a trace containing further information

In Oracle Database 11g Release 1 onward, create an incident and place the relevant files in the incident directory in the location defined by the diagnostic_dest initialization file parameter

Write the error message to the user interface if the server process is not terminated or signal ORA-3113 if it is Often you will see multiple errors reported within the space of a few minutes, typically starting with an ORA-600. It is usually, but not always, the case that the first is the significant error and the others are side effects.

Resolution For This Errors:

ORA-600 and ORA-7445 errors, you can either identify the cause and resolve the error on your own or find ways to avoid the error. The information provided in this section will help you resolve or work around some of the more common errors.

ORA-600 [729]. The first argument to this ORA-600 error message, 729, indicates a memory-handling issue. The error message text will always include the words space leak, but the number after 729 will vary:

ORA-00600: internal error code,
arguments: [729], [800],
[space leak], [], [],

A space leak occurs when some code doesn't completely release the memory it used while executing. In this example, when that process disconnected from the database, it discovered that some memory was not cleaned up at some point during its life and reported ORA-600 [729]. The number in the second set of brackets (800) is the number of bytes of memory discovered.

You cannot determine the cause of the space leak by checking your application code, because the error is internal to Oracle Database. You can, however, safely avoid the error by setting an event in the initialization file for your database in this form:

event="10262 trace name
context forever, level xxxx"

or by executing the following:

SQL>alter system set events '10262 trace name context forever, level xxxx' scope=spfile;

You will also need to shut the database down and restart it to enable the event to take effect.

Replace xxxx with a number greater than the value in the second set of brackets in the ORA-600 [729] error message. In the example above, you could set the number to 1000, in which case the event instructs the database to ignore all user space leaks that are smaller than 1000 bytes.

ORA-600 [kddummy_blkchk]. The kddummy_blkchk argument indicates that checks on the physical structure of a block have failed. This error is reported with three additional arguments: the file number, the block number, and an internal code indicating the type of issue with that block. The following is an alert.log excerpt for an ORA-600 [kddummy_blkchk] error:

Errors in file/u01/oracle/admin/
PROD/bdump/prod_ora_11345.trc:
ORA-600: internal error code, arguments:
[kddummy_blkchk], [2], [21940], [6110],
[], [], [], []
         
The ORA-600 [kddummy_blkchk] error message usually reports a corruption, so to identify the object involved, first use the file number (&afn) and the block number (&bn) reported in the error message in the SQL query:

select * from dba_extents where file_id=&afn and &bn between block_id and block_id + blocks -1;

&afn is the first additional argument (2 in this example); &bn is the second additional argument (21940 in this example).

If the query returns a table, confirm the corruption by executing

SQL>analyze table <tablename>
validate structure;

If the query returns an index, confirm the corruption by executing

SQL>analyze index <indexname>
validate structure;

If there is definitely a corruption in the object, the error returned will be

ORA 1498 "block check failure -
see trace file"

The best way to resolve the corruption is to restore a copy of the affected datafile from before the error and to recover the database so these changes are applied to the restored datafile and brought forward to the current time. This action is possible only if the archivelog feature for your database is enabled. (Enabling this feature ensures that all changes to the database are saved in archived redo logs.)

ORA-600 [6033]. This error is reported with no additional arguments, as shown in the following alert.log file excerpt:

Errors in file/u01/oracle/admin/PROD/
bdump/prod_ora_2367574.trc:
ORA-600: internal error code, arguments:
[6033], [], [], [], [], [], [], []

The ORA-600 [6033] error often indicates an index corruption. To identify the affected index, you’ll need to look at the trace file whose name is provided in the alert.log file, just above the error message. In this alert.log excerpt, the trace file you need to look at is called prod_ora_2367574.trc and is located in /u01/oracle/admin/PROD/bdump.

There are two possible ways to identify the table on which the affected index is built:

Look for the SQL statement that was executing at the time of the error. This statement should appear at the top of the trace file, under the heading “Current SQL Statement.” The affected index will belong to one of the tables accessed by that statement.

ORA-7445 [xxxxxx] [SIGBUS] [OBJECT SPECIFIC HARDWARE ERROR]. This ORA-7445 error can occur with many different functions (in place of xxxxxx). For example, the following alert.log excerpt shows the failing function as ksxmcln.

/u01/app/oracle/admin/prod/bdump/
prod_smon_8201.trc:
ORA-7445: exception encountered:
core dump [ksxmcln()+0] [SIGBUS]
[object specific hardware error]
[6822760] [] []

The important part of this error is the ”object specific hardware error” argument, which indicates that there were insufficient operating system resources to complete the action. The most common resources involved are swap and memory.

To diagnose the cause of an ORA-7445 error, you should first check the operating system error log; for example, in Linux this error log is /var/log/messages. Within the error log, look for information with the same time stamp as the ORA-7445 error (this will be in the alert.log next to the error message). You will often find an error message similar to

Jun 9 19:005:05 PRODmach1 genunix:
[ID 470503 kern.warning]
WARNING: Sorry, no swap space to grow
stack for pid 9632

If no errors are reported in the operating system error log with the same time stamp as the ORA-7445 error, check the ORA-7445 trace file. If there is a statement in the trace file under the heading “Current SQL Statement,” execute that statement again to try to reproduce the error. If the error is reproduced, run the statement again while monitoring OS resources with standard UNIX monitoring tools such as sar or vmstat (contact your system administrator if you are not sure which to use).
Once you’ve identified the resource that affects the running of the statement, increase the amount of that resource available to Oracle Database.

You should be able to resolve some errors that are caused by underlying physical issues such as file corruption or insufficient swap space. But because ORA-600 and ORA-7445 errors are internal, many cannot be resolved by user-led troubleshooting.

Oracle Database users with Oracle support introduces support for the ORA-600/ORA-7445 lookup tool Knowledge on this Article 153788.1 it enables you to enter the first argument to an ORA-600 or ORA-7445 error message and use that information to identify known defects, workarounds, and other knowledge targeted specifically to that error/argument combination.


Thursday, April 7, 2016

Hang-analyze In Oracle Database

Hang-analyze In Oracle Database

Collecting Hanganalyze and Systemstate Dumps

Logging in to the system
Using SQL*Plus connect as SYSDBA using the following command:
sqlplus '/ as sysdba'

If there are problems making this connection then in 10gR2 and above, the sqlplus "preliminary connection" can be used :
sqlplus -prelim '/ as sysdba'

Collection commands for Hanganalyze and Systemstate: Non-RAC:
Sometimes, database may actually just be very slow and not actually hanging. It is therefore recommended, where possible to get 2 hanganalyze and 2 systemstate dumps in order to determine whether processes are moving at all or whether they are "frozen".

Hanganalyze
sqlplus '/ as sysdba'
oradebug setmypid
oradebug unlimit
oradebug hanganalyze 3
-- Wait one minute before getting the second hanganalyze
oradebug hanganalyze 3
oradebug tracefile_name
exit

Systemstate
sqlplus '/ as sysdba'
oradebug setmypid
oradebug unlimit
oradebug dump systemstate 266
oradebug dump systemstate 266
oradebug tracefile_name
exit

Collection commands for Hanganalyze and Systemstate: RAC
There are 2 bugs affecting RAC that without the relevant patches being applied on your system, make using level 266 or 267 very costly. Therefore without these fixes in place it highly unadvisable to use these level

For information on these patches see:

Document 11800959.8 Bug 11800959 - A SYSTEMSTATE dump with level >= 10 in RAC dumps huge BUSY GLOBAL CACHE ELEMENTS - can hang/crash instances

Document 11827088.8 Bug 11827088 - Latch 'gc element' contention, LMHB terminates the instance

Collection commands for Hanganalyze and Systemstate: RAC with fixes for bug 11800959 and bug 11827088
sqlplus '/ as sysdba'
oradebug setorapname reco
oradebug unlimit
oradebug -g all hanganalyze 3
oradebug -g all hanganalyze 3
oradebug -g all dump systemstate 266
oradebug -g all dump systemstate 266
exit

Collection commands for Hanganalyze and Systemstate: RAC without fixes for Bug 11800959 and Bug 11827088
sqlplus '/ as sysdba'
oradebug setorapname reco
oradebug unlimit
oradebug -g all hanganalyze 3
oradebug -g all hanganalyze 3
oradebug -g all dump systemstate 258
oradebug -g all dump systemstate 258
exit

In RAC environment, a dump will be created for all RAC instances in the DIAG trace file for each instance

Explanation of Hanganalyze and Systemstate Levels-

Hanganalyze levels:
Level 3: In 11g onwards, level 3 also collects a short stack for relevant processes in hang chain
Systemstate levels:
Level 258 is a fast alternative but we'd lose some lock element data
Level 267 can be used if additional buffer cache / lock element data is needed with an understanding of the cost

Other Methods

If connection to the system is not possible in any form, then please refer to the following article which describes how to collect systemstates in that situation:

Document 121779.1 Taking a SYSTEMSTATE dump when you cannot CONNECT to Oracle.

On RAC Systems, hanganalyze, systemstates and some other RAC information can be collected using the 'racdiag.sql' script, OR see:
Document 135714.1 Script to Collect RAC Diagnostic Information (racdiag.sql)

$wait_chains

Starting from 11g release 1, the dia0 background processes starts collecting hanganalyze information and stores this in memory in the "hang analysis cache". It does this every 3 seconds for local hanganalyze information and every 10 seconds for global (RAC) hanganalyze information. This information can provide a quick view of hang chains occurring at the time of a hang being experienced.

More information:

Document 1428210.1 Troubleshooting Database Contention With V$Wait_Chains

waiting Session details -
SELECT chain_id, num_waiters, in_wait_secs, sid, sess_serial#, osid, blocker_osid, substr(wait_event_text,1,30) FROM v$wait_chains;

Blocking session details -
set pages 1000
set lines 120
set heading off
column w_proc format a50 tru
column instance format a20 tru
column inst format a28 tru
column wait_event format a50 tru
column p1 format a16 tru
column p2 format a16 tru
column p3 format a15 tru
column Seconds format a50 tru
column sincelw format a50 tru
column blocker_proc format a50 tru
column waiters format a50 tru
column chain_signature format a100 wra
column blocker_chain format a100 wra

SELECT *
FROM (SELECT 'Current Process: '||osid W_PROC, 'SID '||i.instance_name INSTANCE,
'INST #: '||instance INST,'Blocking Process: '||decode(blocker_osid,null,'<none>',blocker_osid)||
' from Instance '||blocker_instance BLOCKER_PROC,'Number of waiters: '||num_waiters waiters,
'Wait Event: ' ||wait_event_text wait_event, 'P1: '||p1 p1, 'P2: '||p2 p2, 'P3: '||p3 p3,
'Seconds in Wait: '||in_wait_secs Seconds, 'Seconds Since Last Wait: '||time_since_last_wait_secs sincelw,
'Wait Chain: '||chain_id ||': '||chain_signature chain_signature,'Blocking Wait Chain: '||decode(blocker_chain_id,null,
'<none>',blocker_chain_id) blocker_chain
FROM v$wait_chains wc,
v$instance i
WHERE wc.instance = i.instance_number (+)
AND ( num_waiters > 0
OR ( blocker_osid IS NOT NULL
AND in_wait_secs > 10 ) )
ORDER BY chain_id,
num_waiters DESC)
WHERE ROWNUM < 101;

Below is the sql to get BLOCKING Session -
set pages 1000
set lines 120
set heading off
column w_proc format a50 tru
column instance format a20 tru
column inst format a28 tru
column wait_event format a50 tru
column p1 format a16 tru
column p2 format a16 tru
column p3 format a15 tru
column Seconds format a50 tru
column sincelw format a50 tru
column blocker_proc format a50 tru
column fblocker_proc format a50 tru
column waiters format a50 tru
column chain_signature format a100 wra
column blocker_chain format a100 wra

SELECT *
FROM (SELECT 'Current Process: '||osid W_PROC, 'SID '||i.instance_name INSTANCE,
'INST #: '||instance INST,'Blocking Process: '||decode(blocker_osid,null,'<none>',blocker_osid)||
' from Instance '||blocker_instance BLOCKER_PROC,
'Number of waiters: '||num_waiters waiters,
'Final Blocking Process: '||decode(p.spid,null,'<none>',
p.spid)||' from Instance '||s.final_blocking_instance FBLOCKER_PROC,
'Program: '||p.program image,
'Wait Event: ' ||wait_event_text wait_event, 'P1: '||wc.p1 p1, 'P2: '||wc.p2 p2, 'P3: '||wc.p3 p3,
'Seconds in Wait: '||in_wait_secs Seconds, 'Seconds Since Last Wait: '||time_since_last_wait_secs sincelw,
'Wait Chain: '||chain_id ||': '||chain_signature chain_signature,'Blocking Wait Chain: '||decode(blocker_chain_id,null,
'<none>',blocker_chain_id) blocker_chain
FROM v$wait_chains wc,
gv$session s,
gv$session bs,
gv$instance i,
gv$process p
WHERE wc.instance = i.instance_number (+)
AND (wc.instance = s.inst_id (+) and wc.sid = s.sid (+)
and wc.sess_serial# = s.serial# (+))
AND (s.final_blocking_instance = bs.inst_id (+) and s.final_blocking_session = bs.sid (+))
AND (bs.inst_id = p.inst_id (+) and bs.paddr = p.addr (+))
AND ( num_waiters > 0
OR ( blocker_osid IS NOT NULL
AND in_wait_secs > 10 ) )
ORDER BY chain_id,
num_waiters DESC)
WHERE ROWNUM < 101;

Provide AWR/Statspack snapshots of General database performance

Hangs are a visible effect of a number of potential causes, this can range from a single process issue to something brought on by a global problem.
Collecting information about the general performance of the database in the build up to, during and after the problem is of primary importance since these snapshots can help to determine the nature of the load on the database at these times and can provide vital diagnostic information. This may prove invaluable in identifying the area of the problem and ultimately resolving the issue.

To do this, please take and upload snapshot reports of database performance (AWR (or statspack) reports) immediately before, during and after the hang..

Please refer to the following article for details of what to collect:
Document 781198.1 Diagnostics for Database Performance Issues

Gather an up-to date RDA

An up to date current RDA provides a lot of additional information about the configuration of the database and performance metrics and can be examined to spot background issues that may impact performance.
See the following note on My Oracle Support:
Document 314422.1 Remote Diagnostic Agent (RDA) 4 - Getting Started

PROACTIVE METHODS TO GATHER INFORMATION ON A HANGING SYSTEM

On some systems a hang can occur when the DBA is not available to run diagnostics or at times it may be too late to collect the relevant diagnostics. In these cases, the following methods may be used to gather diagnostics:
As an alternative to the manual collection method notes above, it is also possible to use the HANGFG script as described in the following note to collect the information:
Document 362094.1 HANGFG User Guide
Additionally, this script can collect information with lower impact on the target database.

LTOM

The Lite Onboard Monitor (LTOM) is a java program designed as a real-time diagnostic platform for deployment to a customer site.LTOM proactively provides real-time automatic problem detection and data collection.
For more information see:
Document 352363.1 LTOM - The On-Board Monitor User Guide
Procwatcher
Procwatcher is a tool that examines and monitors Oracle database and/or clusterware processes at a specific interval
The following notes explain how to use Procwatcher:
Document 459694.1 Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware Processes
Document 1352623.1 How To Troubleshoot Database Contention With Procwatcher
OS Watcher Black Box OSWatcher Black Box contains a built in analyzer that allows the data that has been collected to be automatically analyzed, pro-actively looking for cpu, memory, io and network issues. It is recommended that all users install and run OSWbb since it is invaluable for looking at issues on the OS and has very little overhead. It can also be extremely useful for looking at OS performance degradation that may be seen when a hang situation occurs.

Refer to the following for download, user guide and usage videos on OSWatcher Black Box:

Document 301137.1 OSWatcher Black Box User Guide .

ORACLE ENTERPRISE MANAGER 12C REAL-TIME ADDM

Real-Time ADDM is a feature of Oracle Enterprise Manager Cloud Control 12c that allows you to analyze database performance automatically when you cannot logon to the database because it is hung or performing very slowly due to a performance issue. It analyzes current performance when database is hanging or running slow and reports sources of severe contention.

Oracle Enterprise Manager 12c Real-Time ADDM

RETROACTIVE INFORMATION COLLECTION

Sometimes we may only notice a hang after it has occurred. In this case the following information may help with Root Cause Analysis:

A series of AWR/Statspack reports leading up to and during the hang
ASH reports - one can obtain more granular reports during the time of the hang - even up to
one minute in time.

Raw ASH information. This can be obtained by issuing an ashdump trac.

To See more:

Document 243132.1 10g and above Active Session History (Ash) And Analysis Of Ash Online And Offline
Document 555303.1 ashdump* scripts and post-load processing of MMNL traces
Alert log and any traces created at time of hang
On a RAC specifically check the following traces files as well: dia0, lmhb, diag and lmd0 traces
RDA as above