Showing posts with label Errors & Fixes. Show all posts
Showing posts with label Errors & Fixes. Show all posts

Tuesday, April 19, 2022

How to check the block corruptions in database ?

How to check the block corruptions in database ?



RMAN Level:

RMAN> backup validate check logical datafile <<<file_number>>>;
RMAN> backup validate check logical database;

SQLPLUS Level:

sqlplus / as sysdba
select * from v$database_block_corruption;

OS Level:

The following example shows a sample use of the command-line interface to this mode of DBVERIFY.

% dbv FILE=t_db1.dbf feedback=100 >replace the file

dbv FILE=/u05/oradb/TEST/db/apps_st/data/eqddasm.dbf FEEDBACK=100

Standby RFS & MRP Process

Standby RFS & MRP Process

select process, status,sequence#,block#,blocks, delay_mins from v$managed_standby;


PROCESS STATUS SEQUENCE# BLOCK# BLOCKS DELAY_MINS

--------- ------------ ---------- ---------- ---------- ----------

ARCH CONNECTED 0 0 0 0

ARCH CLOSING 23760 813056 1925 0

ARCH CLOSING 23713 567296 1119 0

ARCH CLOSING 23759 958464 656 0

ARCH CLOSING 23715 1 116 0

RFS IDLE 0 0 0 0

RFS IDLE 0 0 0 0

RFS IDLE 23761 200510 1 0

MRP0 APPLYING_LOG 23761 200510 1024000 0

RFS IDLE 0 0 0 0

10 rows selected.


Here RFS process is idle its mean there is no archive log generation on primary.



SQL> select thread#, process,status,sequence#, block#,blocks from v$managed_standby where process='RFS';


THREAD# PROCESS STATUS SEQUENCE# BLOCK# BLOCKS

---------- --------- ------------ ---------- ---------- ----------

0 RFS IDLE 0 0 0

0 RFS IDLE 0 0 0

1 RFS IDLE 23761 204221 1

0 RFS IDLE 0 0 0



SQL> select process,pid,status from v$managed_standby;


PROCESS PID STATUS

--------- ------------------------ ------------

ARCH 9352 CONNECTED

ARCH 9354 CLOSING

ARCH 9356 CLOSING

ARCH 9358 CLOSING

ARCH 9360 CLOSING

RFS 1468 IDLE

RFS 1462 IDLE

RFS 1464 IDLE

MRP0 9376 APPLYING_LOG

RFS 1466 IDLE

10 rows selected.


SQL> !ps -ef|grep 9376

oraprod 7178 7176 0 12:09:19 pts/1 0:00 grep 9376

oraprod 9376 1 0 Mar 06 ? 1:30 ora_mrp0_QASKDR

oraprod 7176 7149 0 12:09:19 pts/1 0:00 /usr/bin/bash -c ps -ef|grep 9376


SQL> !ps -ef|grep 1466

oraprod 7181 7179 0 12:09:49 pts/1 0:00 grep 1466

oraprod 1466 1 0 Apr 15 ? 0:00 oracleQASKDR (LOCAL=NO)

oraprod 7179 7149 0 12:09:49 pts/1 0:00 /usr/bin/bash -c ps -ef|grep 1466

SQL>

Friday, October 9, 2020

Resize Operation Completed For File#

Resize Operation Completed For File# 


Symptoms:

File Extension Messages are seen in alert log.There was no explicit file resize DDL as well.

Resize operation completed for file# 45, old size 26M, new size 28M

Resize operation completed for file# 45, old size 28M, new size 30M

Resize operation completed for file# 45, old size 30M, new size 32M

Resize operation completed for file# 36, old size 24M, new size 26M


Changes:

NONE


Cause:

These file extension messages were result of diagnostic enhancement through unpublished to record automatic datafile resize operations in the alert log with a message of the form:

"File NN has auto extended from x bytes to y bytes"

This can be useful when diagnosing problems which may be impacted by a file resize. 


Solution:

In busy systems, the alert log could be completely flooded with file extension messages. A new Hidden parameter parameter "_disable_file_resize_logging" has been introduced through bug 18603375 to stop these messages getting logged into alert log.

(Unpublished) Bug 18603375 - EXCESSIVE FILE EXTENSION MESSAGE IN ALERT LOG 

Set the below parameter along with the fix.

SQL> alter system set "_disable_file_resize_logging"=TRUE ; (Its default value is FALSE)

The bug fix 18603375 is included in 12.1.0.2 onwards.


Reference metalink Doc ID 1982901.1

Thursday, September 12, 2019

Unable to create Patch Object Exception occured : PatchObject constructor: Input file does not exist.

Unable to create Patch Object Exception occured : PatchObject constructor: Input file does not exist.



Version:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

Issue:
Error found on invoking patch conflict check using opatch for a PSU patch on database Home.

Error:
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail"not executed
Unable to create Patch Object.
Exception occured : PatchObject constructor: Input file "/u01/stage/etc/config/actions" or "/u01/stage/etc/config/inventory" does not exist.

OPatch succeeded.

Reason:

Error is due to the location (directory) given for the Patch folder given using -ph.

I used the command, "opatch prereq CheckConflictAgainstOHWithDetail -ph ./". Since I gave -ph . the command should have been executed from the directory where the patch was extracted.

In this case the patch folder after extraction was /u01/stage/14275605.

But I invoked the patch conflict check from /u01/stage and specified -ph ./ where it will look for patch files in the current directory and resulted in the error.

Fix:

Move to the folder created after extracting the patch zip file and execute the command again from there.

[oracle@node1 stage]$ cd  /u01/stage/14275605
[oracle@node1 14275605]$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./

or specify the full directory to -ph command from current directory

opatch prereq CheckConflictAgainstOHWithDetail -ph /u01/stage/14275605

Thursday, February 28, 2019

Basic Performance Tunning in Oracle Database

Basic Performance Tunning in Oracle Database


Method.1


How will you find the performance issues ?

Mostly, when any users intimates us that:
This query is running slowly ?
Daily, this query will retrieve data in 1 minute. But, today it is running since 2 hours ?
Load average is high ?
CPU utilization of a process is high ?
Users experiencing slowness ?

a. Get the top consuming PID’s (Works in almost all Linux related OS). And if it is Windows, please look into task manager.

ps -eo pcpu,pid,user,args | sort -k 1 -r |head -10

b.  Now, you need to pick the SQL_ID mainly for proceeding further. You will be prompted for PID which you picked in above command.

set linesize 2000;
select s.sid,s.serial#, s.inst_id,p.spid, s.SQL_ID, t.SQL_TEXT, s.machine from gv$process p, gv$session s, gv$sqltext t where s.paddr = p.addr and p.spid=&processid and s.SQL_HASH_VALUE = t.HASH_VALUE;

Now we have two ways:

1. Using the sql_id, pick the tables involved in it. We are using explain plans.
select * from table(dbms_xplan.display_cursor(sql_id => ‘&SQL_ID’, format => ‘+ALLSTATS’));

2. Run below Oracle Provided script to get details. This is Sql Tuning Report. This report will give all the recommendations available.
@?/rdbms/admin/sqltrpt.sql

Now, you will be having the tables list. Check when was latest time stamp of the table analyzed.

SELECT OWNER, TABLE_NAME, NUM_ROWS, BLOCKS, AVG_ROW_LEN, TO_CHAR(LAST_ANALYZED, 'MM/DD/YYYY HH24:MI:SS') FROM DBA_TABLES WHERE TABLE_NAME ='&TABLE_NAME';

What is the use of last_analyzed column ?
This will give the information when the stats were gathered.

If the stats are not up-to-date, we need to gather stats.

Find whether the table is partitioned table or normal table.

select table_name, subpartition_name, global_stats, last_analyzed, num_rows from dba_tab_subpartitions where table_name='&Tablename' and table_owner='&owner' order by 1, 2, 4 desc nulls last;

select TABLE_OWNER, TABLE_NAME, PARTITION_NAME, LAST_ANALYZED from DBA_TAB_PARTITIONS where TABLE_OWNER=’&owner’ and TABLE_NAME=’&Tablename’ order by LAST_ANALYZED;

If table has NO partitions and if stats need to be gathered, use this

EXEC dbms_stats.gather_table_stats(‘&Owner’,’&Table’,cascade=>TRUE);

If table has partitions, we can gather stats on whole table. But, its better to gather stats for recently created ten partitions, use below

begin
SYS.DBMS_STATS.gather_table_stats
(ownname => ‘&Table_Owner’,
tabname => ‘&Table_Name’,
PARTNAME => ‘&partition’,
estimate_percent => 0.001,
DEGREE => 24,
granularity => ‘PARTITION’,
CASCADE => TRUE,
stattab => NULL,
statid => NULL,
statown => NULL
);
END;
/

We can also run this SQL Tunning Advisor from OEM:

1.Go to home
2.Click on Advisor Central
3.Click on SQL Advisor
4.Click on SQL Tunning Advisor
5.Click on Top Activity
6.Choose a SQL_ID for which you want to run tune
7.Click on Actions button go
8.click Submit Button


Method.2


How to tune a SELECT query ?

Step.1
Find the SQL_ID of the slow running query

There could be two possibilities:
1) Query is still running: If the slow query is still running then we can find the sql_id of the query by using v$session view.

2) Query is completed: It might be query is completed but application team got to know it later that the query was slow and it did not finish in its usual time. Since this query is not running right now, we can’t query v$session to get its information.

So for this we use AWR/ASH report. We ask application team that at what time the query was running and for that duration we generate the AWR. In ASH report we find all kind of information regarding the top SQL’s. Basically we see SQL STATISTIS section of the AWR report. In this section there is SQL ORDERED BY ELAPSED TIME which matters most to us. If the query which is reported by application team is present in this section then we note down the sql_id of the query. Otherwise we generate ASH report for that particular time and get the sql_id.

Step.2
Run the SQL Tuning advisor for that SQL_ID
After finding the sql_id we can run sql tuning advisor on this sql_id.

@$ORACLE_HOME/rdbms/admin/sqltrpt.sql) by providing SQL_ID as the input for generating the findings and recommendations. SQL Tuning Advisor seems to be doing logical optimization mainly by checking your SQL structure and statistics.

Based on the tuning advisor recommendation we have to take corrective actions. These recommendation could be and many more:

1) Gather Statistics
2) Create Index
3) Drop Index
4) Join orders
5) Accept sql profile
6) Create baseline and many more recommendations

Before applying any corrective action on production, we need to test that in DEV/QA/UAT/TEST environment or we can ask and tuning expert that this is the recommendation. After all the analysis we should apply in the production database.

After corrective action from tuning advisor run the SQL again and see the improvement.

Step.3
Use TOP command to check the CPU usages by various processes:

TOP command is also useful for performance tuning. Many times, in a single server multiple databases are running. It may happen that one database is consuming more server resources than others. So we have to find out which oracle process is consuming more resources and it is related to which database. For this we use  TOP command. If we see there is CPU used by an oracle process is very high then this a matter of subject to worry about.

If it is a DML statement then we have check the locking in the database

Step.4
Find the locking in the database:

The very first step is to find out if there is any locking in the database. Sometime due to locking a session does not get the required resources and the session gets slow.

We can find below command to check locking in the database:

sql>select s1.username || '@' || s1.machine || ' ( SID=' || s1.sid || ') is blocking'
|| s2.username || '@' || s2.machine || '( SID=' || s2.sid || ')' from
v$lock l1, v$session s1, v$lock l2, v$session s2
where s1.sid=l1.sid and s2.sid=l2.sid
and l1.block=1 and l2.request > 0
and l1.id1=l2.id1
and l2.id2=l2.id2;

Query Output:

S1.USERNAME||'@'||S1.MACHINE||'(SID='||S1.SID||')ISBLOCKING'||S2.USERNAME||'@'||
----------------------------------------------------------------------------------------
SYS@host1.abc.com ( SID=229) is blockingSYS@host1.abc.com( SID=226)

If we see the locking from above query then we can simply inform to user/application. And if they suggest to kill this blocking session then after killing we can get rid of this slowness.


Step.5
Check for the wait events:

There could be some wait events on the database. Check for the particular user and session.

Query for displaying sessions, session state, and wait details

col "Description" format a50
select sid,
        decode(state, 'WAITING','Waiting',
                'Working') state,
        decode(state,
                'WAITING',
                'So far '||seconds_in_wait,
                'Last waited '||
                wait_time/100)||
        ' secs for '||event
        "Description"
from v$session
where username = 'ARUP';

Output:

SID   STATE       Description
————— ——————————  ———————————————————————————————————————————————————————
2832  Working     Last waited 2029 secs for SQL*Net message from client
3346  Waiting     So far 743 secs for enq: TX - row lock contention
4208  Waiting     So far 5498 secs for SQL*Net message from client

It clearly shows the state of the sessions: whether they are working or waiting; if they are working, what they were waiting for earlier and for how long; and if they are waiting, what for and for how long.

In many troubleshooting situations, just knowing the SID of each session is not enough. We may need to know other details, such as the client machine the session is connecting from, the user (of both the database and the operating system), and the service name. All of this information is also readily available in the same V$SESSION view we have been using. Let’s briefly examine the columns that provide that information, by running the below query

select SID, osuser, machine, terminal, service_name,
       logon_time, last_call_et
from v$session
where username = 'ARUP';

SID   OSUSER  MACHINE   TERMINAL  SERVICE_NAME  LOGON_TIME LAST_CALL_ET
————— ——————  ———————   ————————  ————————————  —————————— ————————————
3346  oradb   prodb1    pts/5     SYS$USERS     05-FEB-12          6848
2832  oradb   prodb1    pts/6     SERV1         05-FEB-12          7616
4408  ANANDA  ANLAP     ANLAP     ADHOC         05-FEB-12             0

OSUSER. The operating system user as which the client is connected. The output indicates that session 4408 is connected from the ANLAP machine, where a Windows user, ANANDA, has logged in.
MACHINE. The name of the machine where the client is running. This could be the database server itself. For two of the sessions, the machine name shows up as “prodb1.” Session 4408 runs on a different machine—ANLAP—presumably a laptop.
TERMINAL. If the session is connected from a UNIX server, this is the terminal where it runs.
LOGON_TIME. This shows when the session was first connected to the Oracle Database instance.

More:

1.Run TOP command in Linux to check CPU usage.
2.Run VMSTAT, SAR, PRSTAT command to get more information on CPU, memory usage and possible blocking.
3.Enable the trace file before running your queries,then check the trace file using tkprof create output file.
According to explain plan check the elapsed time for each query,then tune them respectively.

What is the use of iostat/vmstat/netstat command in Linux?

iostat – reports on terminal, disk and tape I/O activity.
vmstat – reports on virtual memory statistics for processes, disk, tape and CPU activity.
netstat – reports on the contents of network data structures.

Sunday, January 27, 2019

Backup and Recovery Guidelines For Online Patching (adop) Cutover Failed

Backup and Recovery Guidelines For Online Patching (adop) Cutover Failed


If a cutover error occurs, you should first check the error message and try to determine if the problem can be fixed easily, or (as is true in many cases) cutover can be made to succeed simply by running the command again. Restoring to a point before cutover via Flashback recovery should only be done when the error cannot easily be fixed, and continues to fail on subsequent cutover attempts.

Before proceeding further with the instructions in this document:

1.Review failure messages and cutover logs, identify problems, and make corrections as applicable. Issues such as running out of disk space can be corrected easily, whilst issues such as timeouts, deadlocks, and network issues may prove to be transient.
2.Retry the cutover command.
3.If cutover still fails, follow the instructions in the rest of this document to restore system availability while you take further diagnostic and corrective actions.

If after cutover you want to revert to the state of the system before the patching cycle was started, you can use the Oracle Database Flashback feature to go back to a designated point in time (a restore point). You should create the restore point just before running the cutover phase. Depending on exactly when the failure occurred, you may also need to restore the application tier file systems.

Note: Before creating the restore point, it is advisable to issue a suitable downtime notification and shut down the web services. This will ensure you do not lose any transactional data, and in effect simply extends slightly the cutover downtime.

Setting Up Flashback:

1.Set ARCHIVELOG mode

2.Enable Fast/Flash Recovery Area
You enable the Fast Recovery Area (FRA) by setting two database initialization parameters:
DB_RECOVERY_FILE_DEST_SIZE - Specifies the size of the Fast Recovery Area.
DB_RECOVERY_FILE_DEST - Specifies the physical location of the Flashback recovery files.

3.Specify maximum flashback time
You enable the Flashback time by setting below parameter:
alter system set db_flashback_retention_target=120;

Note: The amount of retention time and space needed will be governed by the amount of time required for cutover. Setting the flashback retention target too high may result in issues if DB_RECOVERY_FILE_DEST_SIZE is set to a large value.

4. Activate Flashback

5.Create restore point
Create a restore point called BEFORE_CUTOVER. As shown in the example below, it is also recommended to force a logfile switch both before and after the restore point is created.
SQL>alter system switch logfile;
System altered.
SQL>create restore point BEFORE_CUTOVER guarantee flashback database;
Restore point created.
SQL>alter system switch logfile;
System altered.

Note: As noted under the FRA description, the Online Patching cutover phase should be scheduled for a time when there are few online transactions and batch processing is minimal. You should confirm that critical concurrent requests are not executing during cutover.  You should also consider putting scheduled concurrent requests on hold prior to creating the BEFORE_CUTOVER flashback restore point.

Scenario
You are running an Online Patching cycle:
$ adop phase=prepare
...
$ adop phase=apply patches=11111111,22222222
...
$ adop phase=finalize
...
$ adop phase=cutover
Cutover fails, and you need to go back to the state of the system before you ran the cutover phase.

Note: If you had not run the cutover phase, you would have been able to roll back the patch application process by running the adop abort phase. However, this is not possible once cutover has been run.

There are two main parts to the restore procedure:

1.You will at least need to restore the database using the Flashback feature.
2.Depending on when cutover failed, you may also need to restore the application tier file systems.

Flashing Back the Database
----------------------------------:

1.First, shut down the database, then start it up in mount state:
SQL>shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>startup mount
ORACLE instance started.

2.Restore the flashback to the specified restore point:
SQL>flashback database to restore point BEFORE_CUTOVER;
Flashback complete.

3.Start the database in read-only mode:
SQL>alter database open read only;
Database altered.
Check all looks as expected.

4.Shut down the database, start it up in mount state, then open it with the resetlogs option:
SQL>shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>startup mount
ORACLE instance started.
Total System Global Area 2142679040 bytes
Fixed Size 1346140 bytes
Variable Size 520095140 bytes
Database Buffers 1593835520 bytes
Redo Buffers 27402240 bytes
Database mounted.
SQL>alter database open resetlogs;
Database altered.

5.Disable flashback:
SQL>alter database flashback off;
Database altered.

6.Drop the restore point:
SQL>drop restore point BEFORE_CUTOVER;
Restore point dropped.

7.Set recovery file destination:
SQL>alter system set db_recovery_file_dest='';
System altered.

8.Confirm that Flashback has been deactivated:
SQL>select FLASHBACK_ON from v$database;
FLASHBACK_ON
------------
NO

Restoring the File Systems
---------------------------------:

Whether you need to perform this step is conditional, depending on whether cutover failed before the file systems were switched. You can identify which of these cases applies by referring to the cutover logs in $NE_BASE/EBSapps/log/adop/<current_session_id>/cutover_<timestamp>/ for your current session id.

Case 1 - If the log messages indicate that cutover failed before the file systems were switched, do a clean shutdown of any services that are running. Then restart all the services using the normal startup script, and go to Section 6.

Section 6. Options and Next Steps
After the restore is complete, you have two basic options for proceeding:

* Abort the current patching cycle, if the issue that required you to restore was caused by the patches you were attempting to apply.
* Identify and fix any other issues in the current patching cycle, and proceed with patching.

Case 2 - If the log messages indicate that cutover failed after the file systems were switched, follow Step 5.1 to shut down any services that have started from the new run file system, then follow Step 5.2 to switch the file systems back. After that, go to Section 6.

Section 5.1 Shut down services started from new run file system

* Source the environment on the new run file system.
* From $ADMIN_SCRIPTS_HOME, shut down all the services (using adstpall.sh on UNIX).
* In a multi-node environment, repeat the preceding two steps on all nodes, leaving the admin node until after all the slave nodes.

Section 5.2 Switch file systems back

* On all nodes where file systems have been switched, run the following command to switch the file systems back:
$ perl $AD_TOP/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl \
-action=ctxupdate \
-contextfile=<full path to new run context file> \
-patchcontextfile=<full path to new patch file system context file> \
-outdir=<full path to out directory>
* Start up all services from the old run file system (using adstrtal.sh on UNIX).
* In a multi-node environment, repeat the preceding two steps on all nodes, starting with the admin node and then proceeding to the slave nodes.

Section 6. Options and Next Steps
After the restore is complete, you have two basic options for proceeding:

* Abort the current patching cycle, if the issue that required you to restore was caused by the patches you were attempting to apply.
* Identify and fix any other issues in the current patching cycle, and proceed with patching.

Reference metalink Doc ID 1584097.1

Thursday, January 24, 2019

How to Resolve ORA-00257:Archiver Error

How to Resolve ORA-00257:Archiver Error ?


Solution:

1) Check if there are any errors for the archive destination(s). Make sure that the number of VALID archive destinations is greater than or equal to the value specified by LOG_ARCHIVE_MIN_SUCCEED_DEST initialization parameter.

SELECT dest_id, dest_name, binding, status, destination, error FROM v$archive_dest;

SHOW PARAMETER log_archive_min_succeed_dest;

2) If space is full in one or more of the archive destinations (or if destination is not available), take any of the following steps:

2.a) Manually move the archives to another location and delete them from archive destination.

(OR)

2.b) Change the archive destination to an alternate archive destination which has space available.
SQL> alter system set log_archive_dest_1='LOCATION=<alternate location path>';
 
(OR)

2.c) If there are VALID archive destinations but are less than the value specified by LOG_ARCHIVE_MIN_SUCCEED_DEST, then set this parameter to a lower value:
SQL> alter system set log_archive_min_succeed_dest=<>;
 
(OR)

2.d) Backup the archives and delete them using RMAN command - BACKUP ARCHIVELOG with DELETE INPUT; (Doc ID 388422.1)

3) In case, archiving does not resume even after freeing up space in archive destination, then probably archiver is stuck.In such case, issue the following command as per (Doc ID 121927.1), for each archive destination to resume automatic archiving:

alter system set LOG_ARCHIVE_DEST_.. = 'location=/<archive log path> reopen';

When Flash Recovery Area is in use

i) Check if flash_recovery_area is in use.

archive log list;
Database log mode Archive Mode
Automatic archival Enabled
Archive Destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 384
Next log sequence to archive 386
Current log sequence 386
ii) To immediately resolve this issue, allow more space in the DB_RECOVERY_FILE_DEST with the DB_RECOVERY_FILE_DEST_SIZE parameter :

SQL> alter system set db_recovery_file_dest_size=<>G ;
OR

iii) To avoid the situation once DB_RECOVERY_FILE_DEST_SIZE is reached , specify and alternate location
(archiving is automatically performed to the alternate dest2) :

log_archive_dest_1='LOCATION=use_db_recovery_file_dest NOREOPEN ALTERNATE=LOG_ARCHIVE_DEST_2'
log_archive_dest_2='LOCATION=/other_dest_for_archiving'
log_archive_dest_state_1='enable'
log_archive_dest_state_2='alternate'
db_recovery_file_dest='/u01/app/oracle/product/10.1.0/db_1/flash_recovery_area'
db_recovery_file_dest_size=2G

Reference metalink Doc ID 2014425.1

Tuesday, June 26, 2018

OPP: Output Post Processor TroubleShooting

OPP: Output Post Processor TroubleShooting


Issue 1

Output Post Processor is Down with Actual Process is 0 And Target Process is 1

1. Shutdown Concurrent Managers
2. To ensure concurrent manager down; check there is no FNDLIBR process running.
ps -ef | grep FNDLIBR
3. Run adadmin to relink FNDSVC executable.
a. Invoke adadmin from command prompt
b. Choose option 2 (2. Maintain Applications Files menu)
c. Choose option 1 (1. Relink Applications programs )
d. Then type FND When prompted; ( Enter list of products to link (all for all products) [all] : FND )
e. Ensure adrelink is exiting with status 0
4. Start Concurrent Managers
5. Check the Output Post Processor

Issue 2

Concurrent Processing  R12 Output Post Processor Service Not Coming Up 

Reason :
If Service Manager for the node is not running.  Possible cause might be service manager definition is missing under Concurrent ->Manager ->Define form. If the Service Manager is not present/defined for a particular node,then this causes all the services provided by Service Manager like OPP,WF etc.. not to work.

1. Shutdown all the services.
Below Step 2 will create  Service Manager FNDSM
2. Log in as applmgr
cd to $FND_TOP/patch/115/sql
Run the script: afdcm037.sql
3. Relink FNDSM and FNDLIBR executables as mentioned below:
$ adrelink.sh force=y link_debug=y "fnd FNDLIBR"
$ adrelink.sh force=y link_debug=y "fnd FNDSM"
4. Run cmclean.sql
5. Start up the managers/services

Issue 3

How to restart OOP

Using System Administrator responsibility  navigate to
Concurrent-->Manager-->Administer ->query for the Output Post Processor
The Actual should equal  to the Target processes (eg: Actual=2 Target =2)

Deactivate and Activate

Alternatively you can click on Restart.
If not stopped/deactivated please check the processes and kill  and try to start
ps -ef |grep $LOGNAME | grep -v grep | grep FNDOPP

Issue 4

Output Post Processor (OPP) Log Contains Error "java.lang.OutOfMemoryError: Java heap space

- Determine what the heap size per OPP process is currently:

select DEVELOPER_PARAMETERS from FND_CP_SERVICES
where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = 'FNDCPOPP');

- The default should be:
J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx512m
- Increase the Heap Space per Process to 1024:
update FND_CP_SERVICES
set DEVELOPER_PARAMETERS =
'J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx1024m'
where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = 'FNDCPOPP');
- Bring the Concurrent managers down.
-Run cmclean.sql script from Note 134007.1 - CMCLEAN.SQL Non-Destructive Script to Clean Concurrent Manager Tables.
-Bring the managers up again.

OR

-Log into applications with the System Administrator responsibility.
- Navigate to Concurrent -> Program -> Define
-Query the XML Publisher Template Re-Generator program
-Set the following value for the Executable Options: -Xmx1024m
- Save changes.
- Retest the program.

Issue 5

Output Post Processor is Down with Actual Process is 0 And Target Process is 1

This can happen on a cloned instance.

- FNDSVC should exist under FND_TOP/bin
- Bring down all application services and relink the FNDSVC through adadmin or using the below command:
adrelink.sh force=y ranlib=y "FND FNDSVC"
- Restart all applications services and restest the issue.

Issue 6

Output Post Processing Fails Due To java.lang.ThreadDeath

-Increase the value of the Concurrent:OPP Timeout profile option to 10800 seconds.
-Enable the scalability feature of XML Publisher:
a. Login as SYSADMIN
b. Responsibility: XML Publisher Administrator
c. Function: Administration
d. Set the following properties:
e. Temporary Directory
f. Use XML Publisher's XSLT processor: True
g. Enable scalable feature of XSLT processor: True
h. Enable XSLT runtime optimization: True
- Restart the Concurrent Managers so that changes take effect

Issue 7

Timeout issue with OPP

Solution:

There are 2 new profiles options that can be used to control the timeouts.

Profile Option : Concurrent:OPP Response Timeout
Internal Name : CONC_PP_RESPONSE_TIMEOUT
Description : Specifies the amount of time a manager waits for OPP to respond to its request for post processing.

Profile Option : Concurrent:OPP Process Timeout
Internal Name : CONC_PP_PROCESS_TIMEOUT
Description : Specifies the amount of time the manager waits for the OPP to actually process the request.
The value for the above profile options can be increased to avoid timeouts.
The number of processes/threads for OPP can also be increased; however the concurrent manager has to be restarted for the changes to take effect.

Some use full Note IDs

BI Publisher - Troubleshooting Oracle Business Intelligence (XML) Publisher For The Oracle E-Business Suite (Doc ID 364547.1) ****
How to Determine the Version of Oracle XML Publisher for Oracle E-Business Suite 11i and Release 12 (Doc ID 362496.1)
Concurrent Requests Fail Due to Output Post Processing (OPP) Timeout (Doc ID 352518.1)
OPP Manager Error 'ORA-600 [kqludp2]' Unable To Publish XML Reports (Doc ID 1083510.1)
Problem with OPP Where Getting XML Output Instead of PDF (Doc ID 1178457.1)
EEO Output Post Processor errors: OPP Process Timeout and OPP Response Timeout (Doc ID 458242.1)
R12 - Output Post Processor (OPP) is Down With Actual Process is 0 And Target Process is 1 (Doc ID 1075598.1)
Output Post Processor (OPP) Manager Not Working Error: APPLTMP Environment Variable is not Set (Doc ID 1381474.1)
Concurrent Processing - CP Analyzer for E-Business Suite (Doc ID 1411723.1)

Sunday, May 20, 2018

How to Unlock OIM Schema

How to Unlock OIM Schema.


To view schema name and status:

SELECT username, account_status FROM dba_users;

Unlock Schema:
ALTER USER <Schema_Name>_OIM ACCOUNT UNLOCK;
ALTER USER <Schema_Name>_MDS ACCOUNT UNLOCK;
ALTER USER <Schema_Name>_OPSS ACCOUNT UNLOCK;
ALTER USER <Schema_Name>_ORASDPM ACCOUNT UNLOCK;
ALTER USER <Schema_Name>_SOAINFRA ACCOUNT UNLOCK;
ALTER USER <Schema_Name>_BIPLATFORM ACCOUNT UNLOCK;

12.2 ADOP Prepare Phase Fails with message " [ERROR]: Domain might be locked by some other WLS user process"

12.2 ADOP Prepare Phase Fails with message " [ERROR]: Domain might be locked by some other WLS user process"


Error:
On EBS 12.2 When running adop, prepare phase fails with the below error:

Validating configuration on node: [MYSERV]...
    Log: /u01/app/EBSR12/fs_ne/EBSapps/log/adop/14/prepare_20/<Context_Name>
    [WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.
    [WARNING]: Either some of the required entries in /etc/hosts file might be missing (e.g. localhost or hostname) OR the file /etc/hosts could not be read.
    [ERROR]: Domain might be locked by some other WLS user process
    [UNEXPECTED]Error occurred while executing "perl /u01/app/EBSR12/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl  -contextfile=/u01/app/EBSR12/fs2/inst/apps/<Context_Name>/appl/admin/<Context_Name>.xml -patchctxfile=/u01/app/EBSR12/fs1/inst/apps/<Context_Name>/appl/admin/<Context_Name>.xml -phase=prepare -logloc=/u01/app/EBSR12/fs_ne/EBSapps/log/adop/14/prepare_20150811_185137/<Context_Name> -promptmsg=hide"
    [UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script on MYSERV
Reviewing the file $INST_TOP/apps/<Context_Name>/logs/appl/rgf/TXK/txkValidateDomainInRCOut.txt, the below error message is recorded:
cmd : java weblogic.WLST /u01/app/EBSR12/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkValidateDomainInRC.py --adminuser=weblogic --verify=domainEditModeEnabled --adminurl=t1://hostname.domain:adminport> /u01/app/EBSR12/fs2/inst/apps/<Context_Name>/logs/appl/rgf/TXK/txkValidateDomainInRCOut.txt
ERROR: Domain is in edit mode.

Cause:
Someone with administrator privileges on WLS Admin Console may have locked the Weblogic domain for making changes, and has not released the checkout.
This leaves the domain in edit mode, resulting in the issue.

Solution:
To resolve the issue test the following steps in a development instance and then migrate accordingly:

1. Start the admin server on the run file system (if not started already).
2. Login in to WLS Admin Console (http(s)://hostname.domain:<adminport>/console)
3. Click on the "Activate Changes" or the "Release Configuration" button (depending on the version) in the "Change Center" region on the top left corner.
4. Logout of the WLS console.
5. Rerun the adop phase=prepare command and confirm adop completes successfully.

Tuesday, May 15, 2018

Users can not Login to EBS with Correct UserName Password And Login Page Loops Back

Users can not Login to EBS with Correct UserName Password And Login Page Loops Back 


Customer is Logging In with correct ID and password, but he can't log in to EBS. The web page returns to Login page again in a Loop back.

The issue can be reproduced at will with the following steps:
1. Open IE with EBS URL
2. Input correct ID and Password
3. User can't log in to EBS and return to log in page.

Changes:
no change

Cause:
In the alert.log file, you may find bellow error message.
ORA-1653: unable to extend table APPLSYS.FND_LOGINS by 16 in tablespace APPS_TS_ARCHIVE

Solution:
To implement the solution please execute following:

1. Add space to tablespace APPS_TS_ARCHIVE
2. Retest it

Reference metalink Doc ID 1929675.1

Saturday, March 10, 2018

ADOP Patching Fails Due To ETCC Error During The Apply Phase ' [ERROR] ETCC not run in the database node'


Applies To:

Oracle Applications DBA - Version 12.2 and later
Information in this document applies to any platform.

Symptoms:

E-Business Suite 12.2.x, Online Patching ADOP Utilities issues ETCC errors when applying a patch in Apply phase.

Receiving an ETCC error while executing the following ADOP command:

$ adop phase=apply apply_mode=downtime patches=21865610

ERROR
-----------:
Validating credentials.

Initializing.
Run Edition context :/d01/oracle/inst/fs1/inst/apps/odbdev_oadev/appl/admin/odbdev_oadev.xml
Patch edition context: /d01/oracle/inst/fs2/inst/apps/odbdev_oadev/appl/admin/odbdev_oadev.xml
Reading driver file (up to 50000000 bytes).
Patch file system free space: 45.86 GB

Validating system setup.

Node registry is valid.
[ERROR] ETCC not run in the database node odbdev
The EBS Technology Codelevel Checker needs to be run on the database node.
It is available as Patch 17537119.

Encountered the above errors when performing database validations.
Resolve the above errors and restart adop.


STEPS
------:

The issue can be reproduced at will with the following steps:

1. Running the Apply phase.

Cause:

The ETCC scripts had not been successfully run prior to applying the patch.

1. Patch error:

[ERROR] ETCC not run in the database node odbdev
The EBS Technology Codelevel Checker needs to be run on the database node.
It is available as Patch 17537119.

2. ETCC error:

[oracle@odbdev ~]$ checkMTpatch.sh

+===============================================================+
| Copyright (c) 2005, 2016 Oracle and/or its affiliates. |
| All rights reserved. |
| Oracle E-Business Suite Release 12.2 |
| Middle Tier Technology Codelevel Checker |
+===============================================================+

Using context file from currently set applications environment:
/d01/oracle/odbdev/db/tech_st/12.1.0/appsutil/odbdev_odbdev.xml
Unable to read the value of s_tools_oh from the context file
Verify location and rerun this script.

Solution:

To resolve the issue test the following steps in a development instance and then migrate accordingly:

1. Stop the instance and run AutoConfig.

2. After AutoConfig completes successfully, then run both the ETCC scripts according to Sections 2 & 3 Note 2008451.1 - How To Run The 12.2 EBS Technology Code Level Checker (ETCC):
checkDBpatch.sh
checkMTpatch.sh

3. Reapply the patch with ADOP:
$ adop phase=apply patches=xxxxxxxx

(Note: If the ETCC reports show that some patches need to be applied, at some point apply all of the patches identified in the ETCC reports.)

Reference metalink Doc ID 2190263.1

Wednesday, February 7, 2018

FNDCPASS Cause: AFPCOA Failed Due To ORA-28001: The Password Has Expired


Error:

FNDCPASS Cause: AFPCOA failed due to ORA-28001: the password has expired

Solution:

fndcpass or afpasswd cannot be used to change APPS password because they require login with APPS which fails because password has expired (vicious circle).

To solve the issue, please do the following:

Login as sqlplus apps/<OLD Apps Password> it prompts to enter new password so account could be opened ->>> re-enter the OLD password.

If this works the password expire issue will be fixed and then fndcpass or afpasswd can be used to change the password to a new one.

If an error is received because the same password can't be used, it's needed to check and if necessary change the following profile resources for APPS schema to be as below:

PASSWORD_REUSE_TIME UNLIMITED
PASSWORD_REUSE_MAX UNLIMITED
PASSWORD_VERIFY_FUNCTION NULL

Check the current values:

select profile from dba_users where username='APPS';
SELECT * FROM dba_profiles WHERE profile = 'result_from_previous_query' AND resource_type = 'PASSWORD';

If any of the resources need to be changed, do it as in the below example:

SQL> alter profile default limit PASSWORD_VERIFY_FUNCTION null;

Then try again to set the old password via sqlplus. "

Please review this input and make any needed updates.

Reference metalink Doc ID 2051299.1

Monday, December 25, 2017

R12 E-Business Suite Users Report Responsibilities Are Missing From The EBS Home Page, After Login The Responsibility Page Is Blank 


Symptoms:

-Intermittently responsibilities are missing from the Personal Home Page, or after user login the responsibilities homepage is blank.
(or)
-Missing responsibilities for a user in the navigator page during logon.
-After upgrading and patching, the System Administrator responsibility no longer appears in the Navigator.
-Some users cannot see/receive their open notifications due to missing responsibilities.
-Some users get notification e-mails even though their responsibilities are end-dated
-If a user is end-dated, a user's responsibility creates duplicate row
-Newly added responsibility is not displayed until Apache is bounced
-Intermitent issue: After resetting user passwords using the "Login Assistance " > "Forgot Password" link, users do not see their responsibilities in home page and get the following message:
"There are no active responsibilities available for this user"
"System Administrator" responsibility missing

Cause:

Workflow Roles are out of sync.

Solution:

To resolve the issue test the following steps in a development instance and then migrate accordingly:

1. Run the sync responsibility data into the workflow tables concurrent request via the following steps:

a. Navigate -> System Administrator > Concurrent > Request: Submit a New Request
b. Select request "Sync responsibility role data into the WF table"

2. Retest the user login and confirm the responsibilities now appear as expected.

3. If the issue still occurs, test the following set of additional steps to see if the problem is resolved:

a. Run request -> "Compile Security" - For parameter Everything, select Yes
b. Run request -> "Workflow Directory Services User/Role Validation" with: 'Fix dangling user/roles', 'Add missing user/role assignments' and Update WHO columns in WF tables set to "yes"
c. Run request -> "Synchronize WF LOCAL tables"
d. Run request -> "Create FND_RESP WF ROLES"
e. Run request -> "Sync responsibility role data into the WF table."
f. Follow MOS Doc ID 759038.1 How To Clear The Cache Using Functional Administrator.

4. Retest the user login and confirm the responsibilities now appear.

Reference metalink Doc ID 2259375.1

Thursday, December 21, 2017

ORA-20001: Invalid Column Name or Duplicate Columns/Column groups/expressions In method_opt


Symptoms:

Gather Schema Statistics" program reported following errors in request log files:

ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt

Statistics Errors:
stats on table FND_CP_GSM_OPP_AQTBL is locked
Error #1: ERROR: While GATHER_TABLE_STATS:
object_name=GL.JE_BE_LINE_TYPE_MAP******
Error #2: ERROR: While GATHER_TABLE_STATS:
object_name=GL.JE_BE_LOGS***ORA-20001: invalid column name or duplicate columns/column
groups/expressions in method_opt***
Error #1: ERROR: While GATHER_TABLE_STATS:
object_name=GL.JE_BE_LINE_TYPE_MAP***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***
Error #2: ERROR: While GATHER_TABLE_STATS:
object_name=GL.JE_BE_LOGS***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***
Error #3: ERROR: While GATHER_TABLE_STATS:
object_name=GL.JE_BE_VAT_REP_RULES***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***
Error #4: ERROR: While GATHER_TABLE_STATS:  object_name=FII.FII_FIN_ITEM_HIERARCHIES***ORA-20001: invalid column name or duplicate columns/column groups/expressions in method_opt***
groups/expressions in method_opt***

Cause:

1 ) There are duplicate rows on FND_HISTOGRAM_COLS table for JE_BE_LINE_TYPE_MAP table. Because of this problem, FND_STATS tries to gather histogram information using wrong command and it fails with ora-20001 errors.

The following SQL should return one row, not two:
SQL> show user
USER is "APPS"

SQL>select column_name, nvl(hsize,254) hsize from FND_HISTOGRAM_COLS where table_name = 'JE_BE_LINE_TYPE_MAP' order by column_name;

COLUMN_NAME HSIZE
------------------------------ ----------
SOURCE 254
SOURCE 254

SQL>
Since there are two rows in histograms table, FND_STATS creates following command to gather statistics on table 'JE_BE_LINE_TYPE_MAP' :


Solution:                                                                                                                                         

Find out all duplicates and/or obsolete rows in FND_HISTOGRAM_COLS and delete one of them logged in as the applsys user. Remember to take backup of the FND_HISTOGRAM_COLS table before deleting any data.

SQL> select table_name, column_name, count(*) from FND_HISTOGRAM_COLS group by table_name, column_name having count(*) > 1;

TABLE_NAME                     COLUMN_NAME                      COUNT(*)
------------------------------ ------------------------------ ----------
JE_BE_LOGS                     DECLARATION_TYPE_CODE                   2
JE_FR_DAS_010                  TYPE_ENREG                              2
JE_FR_DAS_010_NEW              TYPE_ENREG                              2
JE_BE_LINE_TYPE_MAP            SOURCE                                  2
JE_BE_VAT_REP_RULES            SOURCE                                  2
JE_BE_VAT_REP_RULES            LINE_TYPE                               2
JE_BE_VAT_REP_RULES            VAT_REPORT_BOX                          2
JG_ZZ_SYS_FORMATS_ALL_B        JGZZ_EFT_TYPE                           2

-- Use above results on the following SQL to delete duplicates

SQL> delete from FND_HISTOGRAM_COLS
where table_name = '&TABLE_NAME'
and  column_name = '&COLUMN_NAME'
and rownum=1;
Enter value for table_name: JE_BE_LOGS
old   2: where table_name = '&TABLE_NAME'
new   2: where table_name = 'JE_BE_LOGS'
Enter value for column_name: DECLARATION_TYPE_CODE
old   3: and  column_name = '&COLUMN_NAME'
new   3: and  column_name = 'DECLARATION_TYPE_CODE'

1 row deleted.

SQL> /
Enter value for table_name: JE_FR_DAS_010
old   2: where table_name = '&TABLE_NAME'
new   2: where table_name = 'JE_FR_DAS_010'
Enter value for column_name: TYPE_ENREG
old   3: and  column_name = '&COLUMN_NAME'
new   3: and  column_name = 'TYPE_ENREG'

1 row deleted.

SQL> /
Enter value for table_name: JE_FR_DAS_010_NEW
old   2: where table_name = '&TABLE_NAME'
new   2: where table_name = 'JE_FR_DAS_010_NEW'
Enter value for column_name: TYPE_ENREG
old   3: and  column_name = '&COLUMN_NAME'
new   3: and  column_name = 'TYPE_ENREG'

1 row deleted.

SQL> /

Repeat same till you delete all the duplicate rows and then commit.
SQL> select table_name, column_name, count(*)
from FND_HISTOGRAM_COLS
group by table_name, column_name
having count(*) > 1;

no rows selected
SQL>
SQL> select column_name, nvl(hsize,254) hsize
from FND_HISTOGRAM_COLS
where table_name = 'JE_BE_LINE_TYPE_MAP'
order by column_name;

COLUMN_NAME                         HSIZE
------------------------------ ----------
SOURCE                                254

SQL>
SQL> commit;

Commit complete.
SQL>


RC-40000: Unable To Startup The Database


Error:

RC-40000 will occur when you gave wrong password for adpreclone

[oracletest@jancln]$ adpreclone.pl dbTier

                     Copyright (c) 2002 Oracle Corporation
                        Redwood Shores, California, USA

                        Oracle Applications Rapid Clone

                                 Version 12.0.0

                      adpreclone Version 120.20.12010000.2

Enter the APPS User Password:*******
Running:
perl /u02/oracle/PROD/db/tech_st/11.1.0/appsutil/bin/adclone.pl java=/u02/oracle/PROD/db/tech_st/11.1.0/appsutil/jre mode=stage stage=/u02/oracle/PROD/db/tech_st/11.1.0/appsutil/clone component=dbTier method=CUSTOM dbctx=/u02/oracle/PROD/db/tech_st/11.1.0/appsutil/PROD_ruh-erpdb01.xml showProgress
APPS Password :

Beginning database tier Stage - Sat Apr  8 10:05:53 2017

/u02/oracle/PROD/db/tech_st/11.1.0/appsutil/jre/bin/java -Xmx600M -DCONTEXT_VALIDATED=false  -Doracle.installer.oui_loc=/u02/oracle/PROD/db/tech_st/11.1.0/oui -classpath /u02/oracle/PROD/db/tech_st/11.1.0/lib/xmlparserv2.jar:/u02/oracle/PROD/db/tech_st/11.1.0/jdbc/lib/ojdbc6.jar:/u02/oracle/PROD/db/tech_st/11.1.0/appsutil/java:/u02/oracle/PROD/db/tech_st/11.1.0/oui/jlib/OraInstaller.jar:/u02/oracle/PROD/db/tech_st/11.1.0/oui/jlib/ewt3.jar:/u02/oracle/PROD/db/tech_st/11.1.0/oui/jlib/share.jar:/u02/oracle/PROD/db/tech_st/11.1.0/oui/jlib/srvm.jar:/u02/oracle/PROD/db/tech_st/11.1.0/jlib/ojmisc.jar   oracle.apps.ad.clone.StageDBTier -e /u02/oracle/PROD/db/tech_st/11.1.0/appsutil/PROD_ruh-erpdb01.xml -stage /u02/oracle/PROD/db/tech_st/11.1.0/appsutil/clone -tmp /tmp -method CUSTOM    -showProgress
APPS Password :
Log file located at /u02/oracle/PROD/db/tech_st/11.1.0/appsutil/log/PROD_ruh-erpdb01/StageDBTier_04081005.log

AutoClone needs to start the database up to generate cloning information
  -     50% completed       RC-40000: Fatal: Unable to start up the database

ERROR while running Stage...
Sat Apr  8 10:06:19 2017

ERROR while running perl /u02/oracle/PROD/db/tech_st/11.1.0/appsutil/bin/adclone.pl java=/u02/oracle/PROD/db/tech_st/11.1.0/appsutil/jre mode=stage stage=/u02/oracle/PROD/db/tech_st/11.1.0/appsutil/clone component=dbTier method=CUSTOM dbctx=/u02/oracle/PROD/db/tech_st/11.1.0/appsutil/PROD_ruh-erpdb01.xml showProgress ...
Sat Apr  8 10:06:19 2017
[oracletest@jancln]$


adpreclone logfile message:


    Connection could not be obtained; returning null

-------------------ADX Database Utility Finished---------------

RC-40000: Unable to start up the database
Raised by oracle.apps.ad.clone.StageDatabase
RC-40000: Unable to start up the database
Raised by oracle.apps.ad.clone.StageDatabase


Cause:

Because of wrong credentials, apps user account has locked. You cannot connect apps until you have unlocked apps user account.

[oracletest@jancln]$ sqlplus / as sysdba

SQL*Plus: Release 11.1.0.7.0 - Production on Sat Apr 8 10:14:50 2017

Copyright (c) 1982, 2008, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> connect apps/apps
ERROR:
ORA-28000: the account is locked


Warning: You are no longer connected to ORACLE.
SQL> exit


Solution:


[oracletest@jancln]$ sqlplus / as sysdba

SQL*Plus: Release 11.1.0.7.0 - Production on Sat Apr 8 10:15:33 2017

Copyright (c) 1982, 2008, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL>
SQL> select username, account_status from dba_users where username like 'APPS';

USERNAME                       ACCOUNT_STATUS
------------------------------ --------------------------------
APPS                           LOCKED(TIMED)

SQL> alter user apps account unlock;

User altered.

SQL> select username, account_status from dba_users where username like 'APPS';

USERNAME                       ACCOUNT_STATUS
------------------------------ --------------------------------
APPS                           OPEN

SQL> exit

[oracletest@jancln]$ adpreclone.pl dbTier

                     Copyright (c) 2002 Oracle Corporation
                        Redwood Shores, California, USA

                        Oracle Applications Rapid Clone

                                 Version 12.0.0

                      adpreclone Version 120.20.12010000.2

Enter the APPS User Password:*******
Running:
perl /u02/oracle/PROD/db/tech_st/11.1.0/appsutil/bin/adclone.pl java=/u02/oracle/PROD/db/tech_st/11.1.0/appsutil/jre mode=stage stage=/u02/oracle/PROD/db/tech_st/11.1.0/appsutil/clone component=dbTier method=CUSTOM dbctx=/u02/oracle/PROD/db/tech_st/11.1.0/appsutil/PROD_ruh-erpdb01.xml showProgress
APPS Password :

Beginning database tier Stage - Sat Apr  8 10:21:14 2017

/u02/oracle/PROD/db/tech_st/11.1.0/appsutil/jre/bin/java -Xmx600M -DCONTEXT_VALIDATED=false  -Doracle.installer.oui_loc=/u02/oracle/PROD/db/tech_st/11.1.0/oui -classpath /u02/oracle/PROD/db/tech_st/11.1.0/lib/xmlparserv2.jar:/u02/oracle/PROD/db/tech_st/11.1.0/jdbc/lib/ojdbc6.jar:/u02/oracle/PROD/db/tech_st/11.1.0/appsutil/java:/u02/oracle/PROD/db/tech_st/11.1.0/oui/jlib/OraInstaller.jar:/u02/oracle/PROD/db/tech_st/11.1.0/oui/jlib/ewt3.jar:/u02/oracle/PROD/db/tech_st/11.1.0/oui/jlib/share.jar:/u02/oracle/PROD/db/tech_st/11.1.0/oui/jlib/srvm.jar:/u02/oracle/PROD/db/tech_st/11.1.0/jlib/ojmisc.jar   oracle.apps.ad.clone.StageDBTier -e /u02/oracle/PROD/db/tech_st/11.1.0/appsutil/PROD_ruh-erpdb01.xml -stage /u02/oracle/PROD/db/tech_st/11.1.0/appsutil/clone -tmp /tmp -method CUSTOM    -showProgress
APPS Password :
Log file located at /u02/oracle/PROD/db/tech_st/11.1.0/appsutil/log/PROD_ruh-erpdb01/StageDBTier_04081021.log

  |      0% completed

Completed Stage...
Sat Apr  8 10:21:26 2017
[oracletest@jancln]$

Automatic Tuning Of Undo_retention Causes Space Problems

Automatic Tuning Of Undo_retention Causes Space Problems


Check Whether the undo is automatically managed by the database by checking the following instance parameter: UNDO_MANAGEMENT=AUTO

Whether the undo tablespace is fixed in size:

SQL> SELECT autoextensible FROM dba_data_files WHERE tablespace_name='<UNDO_TABLESPACE_NAME>'

This returns "NO" for all the undo tablespace datafiles.
The undo tablespace is already sized such that it always has more than enough space to store all the undo generated within the undo_retention time, and the in-use undo space never exceeds the undo tablespace warning alert threshold (see below for the query to show the thresholds).
The tablespace threshold alerts recommend that the DBA add more space to the undo tablespace:

SQL> SELECT creation_time, metric_value, message_type, reason, suggested_action FROM dba_outstanding_alerts WHERE object_name='<UNDO_TABLESPACE_NAME>';

This returns a suggested action of: "Add space to the tablespace".

Or,

This recommendation has been reported in the past but the condition has now cleared:
SQL> SELECT creation_time, metric_value, message_type, reason, suggested_action, resolution FROM dba_alert_history WHERE object_name='<UNDO_TABLESPACE_NAME>';

The undo tablespace in-use space exceeded the warning alert threshold at some point in time. To see the warning alert percentage threshold, issue:

SQL> SELECT object_type, object_name, warning_value, critical_value FROM dba_thresholds WHERE object_type='TABLESPACE';

To see the (current) undo tablespace percent of space in use:

SQL> SELECT
         ((SELECT (NVL(SUM(bytes),0))
           FROM dba_undo_extents
           WHERE tablespace_name='<UNDO_TABLESPACE_NAME>'
           AND status IN ('ACTIVE','UNEXPIRED')) * 100)/       
         (SELECT SUM(bytes)
          FROM dba_data_files
          WHERE tablespace_name='<UNDO_TABLESPACE_NAME>')
         "PCT_INUSE"
     FROM dual;

Cause:

The cause of this problem has been identified in:
Bug:5387030 - AUTOMATIC TUNING OF UNDO_RETENTION CAUSING SPACE PROBLEMS
It is caused by a wrong calculation of the tuned undo retention value.
Bug:5387030 is fixed in RDBMS 11.1.

Solution:

To implement a solution for Bug:5387030, please execute any of the below alternative solutions:

Upgrade to 11.1 in which Bug:5387030 is fixed

OR

Apply patchset release 10.2.0.4 or higher in which Bug:5387030 is fixed.

OR

Download and apply interim Patch:5387030, if available for your platform and RDBMS release. To check for conflicting patches, please use the MOS Patch Planner Tool. If no patch is available, file a Service Request through My Oracle Support for your specific Oracle version and platform.

OR

Use any of the following workarounds:

1. Set the AUTOEXTEND and MAXSIZE attributes of each datafile of the undo tablespace in such a way that they are autoextensible and the MAXSIZE is equal to the current size (so the undo tablespace now has the AUTOEXTEND attribute but does not autoextend):
SQL> ALTER DATABASE DATAFILE '<datafile_flename>' AUTOEXTEND ON MAXSIZE <current_size>

With this setting, V$UNDOSTAT.TUNED_UNDORETENTION is not calculated based on a percentage of the undo tablespace size. Instead it is set to the maximum of (MAXQUERYLEN secs + 300) and UNDO_RETENTION.

2. Set the following instance parameter:
_smu_debug_mode=33554432

With this setting, V$UNDOSTAT.TUNED_UNDORETENTION is not calculated based on a percentage of the fixed size undo tablespace. Instead it is set to the maximum of (MAXQUERYLEN secs + 300) and UNDO_RETENTION.

3. Set the following instance parameter:
_undo_autotune = false

With this setting, V$UNDOSTAT (and therefore V$UNDOSTAT.TUNED_UNDORETENTION) is not maintained and the undo retention used is based on the UNDO_RETENTION instance parameter.

NOTE: This means you loose all advantages in having automatic undo management and is not an ideal long term fix.
NOTE: Even with the patch fix installed, the autotuned retention can still grow under certain circumstances. The fix attempts to throttle back how aggressive that autotuning will be. Options 2 and 3 may be needed to get around this aggressive growth in some environments.

Refernce metalink Doc ID 420525.1

Clearing Cache In Oracle E-Business Suite


Apache / iAS Cache clear for 11.5.9 or 11.5.10.x

- shutdown iAS server
- go to $OA_HTML (for 11.5.9) or $COMMON_TOP (for 11.5.10.x) directory
- backup the directory _pages and delete its contents by running for instance:

rm -rf $COMMON_TOP/_pages/*

- for modplsql caches remove contents of $IAS_ORACLE_HOME/Apache/modplsql/cache directory
- restart iAS server


Cache clear on middle tier for R12


Note: Clearing _pages directory is no longer a recommended solution.
Clearing the _pages in R12 creates blank login page issue, as in R12 the jsp files does not get compiled automatically.

From Forntend
--------------

- go to "Functional Administrator" responsibility

- select Core Services => Caching Framework => Global Configuration => Clear cache

From backend
------------

- adopmnctl.sh stopall

- rm -fr $INST_TOP/ora/10.1.3/j2ee/oacore/persistence/*

- rm -fr $INST_TOP/ora/10.1.3/j2ee/oafm/persistence/*

- rm -fr $INST_TOP/ora/10.1.3/j2ee/forms/persistence/*

- adopmnctl.sh startall


Cabo
-----

Images and style sheets can be corrupted or out of sync in the cabo caches,you may need to clear the related directories after backup:

- $OA_HTML/cabo/images/cache
- $OA_HTML/cabo/styles/cache

ERROR: ORA-00257: Archiver Error. Connect Internal Only, Until Freed.

ERROR: ORA-00257: Archiver Error. Connect Internal Only, Until Freed.


Cause:

The archiver process received an error while trying to archive a redo log. If the problem is not resolved soon, the database will stop executing transactions. The most likely cause of this message is the destination device is out of space to store the redo log file.

Action:

Check archiver trace file for a detailed description of the problem. Also verify that the device specified in the initialization parameter ARCHIVE_LOG_DEST is set up properly for archiving.alter database flashback on

There is two possible way to solution.
1. without increasing DB_RECOVERY_FILE_DEST_SIZE.
2. by increasing DB_RECOVERY_FILE_DEST_SIZE.

Without increasing DB_RECOVERY_FILE_DEST_SIZE.
1. Check whether the database is in archive log mode and automatic archiving is enabled.

 SQL> archive log list;
 Database log mode              Archive Mode
 Automatic archival             Enabled
 Archive destination            USE_DB_RECOVERY_FILE_DEST
 Oldest online log sequence     29
 Next log sequence to archive   31
 Current log sequence           31

2. If archive destination is defined by USE_DB_RECOVERY_FILE_DEST, find the archive destination by:

SQL> show parameter db_recovery_file_dest;
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string      C:\oracle\product\10.2.0/flash_recovery_area
db_recovery_file_dest_size           big integer 2G
Check what the value for db_recovery_file_dest_size.

3. Find the space used in flash recovery area by using following SQL:

select name, round(space_limit/1048576) As space_limit,round(space_used/1048576) As space_used
from v$RECOVERY_FILE_DEST;

4. If SPACE_USED is equal to SPACE_LIMIT of db_recovery_file_dest, move the archive logs to different destination.

5. Archive all the log files

SQL> alter system archive log all;
6. Just switch the logs to verify

SQL> alter system switch logfile;
7. DB_RECOVERY_FILE_DEST_SIZE is to delete (archive log) files from DB_RECOVERY_FILE_DEST if you are sure you have backups and the archived logs are no longer necessary.

$rman target /
RMAN>delete archivelog until time 'SYSDATE-1';
or,
RMAN>delete archivelog all;
By increasing DB_RECOVERY_FILE_DEST_SIZE.
1. See the path of flash recovery area.

SQL> show parameter db_recovery_file_dest;
2. Increase the Flash Recovery Area

SQL> ALTER SYSTEM SET db_recovery_file_dest_size='10G' SCOPE=BOTH;
Sytem Altered.

If User Complain For Long Running Request's, Do The Following.

If User Complain For Long Running Request's, Do The Following.


- Check the Statistics,History Runs of that program & Based on this timelines we need to decide the long running Request.

- Check The load on DB node to ensure that High Resource Usage is not the Main cause.i troubleshoot

- Check DB locks in The Databse To Ensure that This Session is not blocked by any other session.

- Check the CM status and ICM status.

- Check any Locks on DB particularly related to that program.

- Check any INVALID objects related to that program.

- Check any incompatible requests/programs are running for that program.

-Login as system administrator responsibility and query the long running request /program. Navigate to concurrent program define query that long running request in the same form select incompatible requests .

-If any incompatible request is running then find the user name who is running that program inform them regarding the incompatibilities