Monday, November 23, 2015

Logfiles In Oracle Applicaitons

Logfiles In Oracle Applicaitons

Log files are useful in troubleshooting issues in Oracle Applications. Here is the list of Log file location in Oracle Applications for various kinds of tasks

To locate Alert Log Files :

SQL> Select value from V$DIAG_INFO where name=’Diag Alert’;

Network Logs :

$ORACLE_HOME/network/admin/$SID.log

Cloning Log files :

Adpreclone :

i) Apps Tier and Database Tier:

$ORACLE_HOME/appsutil/scripts/context_name

$ADMIN_SCRIPTS_HOME

Adcfgclone :

i) Apps Tier and Database Tier:

$ORACLE_HOME/appsutil/clone/bin

$COMMON_TOP/clone/bin

Preclone (adpreclone.pl) log files in source instance

i) Database Tier:

$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/

(StageDBTier_<time>.log)

ii) Apps Tier:

$INST_TOP/apps/$CONTEXT_NAME/admin/log/

(StageAppsTier_<time>.log)

Post-Clone (Adcfgclone.pl) log files in target instance

i) Database Tier:

$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/

ApplyDBTier_<time>.log

ii) Apps Tier:

$INST_TOP/apps/$CONTEXT_NAME/admin/log/

ApplyAppsTier_<time>.log

Patching log files :

Application Tier:

adpatch log – $APPL_TOP/admin/$SID/log/

Developer (Developer/Forms & Reports 10.1.2):
Patch – $ORACLE_HOME/.patch_storage

Web Server (Apache):
patch – $IAS_ORACLE_HOME/.patch_storage

Database Tier:
opatch log – $ORACLE_HOME/.patch_storage

Autoconfig log files :

$ORACLE_HOME/appsutil/scripts/context_name

$ADMIN_SCRIPTS_HOME

On the database tier:

RDBMS $ORACLE_HOME/appsutil/log/<context>/

<timestamp>/adconfig.log
<timestamp>/NetServiceHandler.log

On the application tier:

$INST_TOP/admin/log/<timestamp>/

adconfig.log
NetServiceHandler.log

Startup/Shutdown scripts:

$COMMON_TOP/admin/scripts/context_name (11i)

$ADMIN_SCRIPTS_HOME (R12)

Startup/Shutdown Log files :

i) Startup/Shutdown error message text files like adapcctl.txt, adcmctl.txt… :

$INST_TOP/apps/$CONTEXT_NAME/logs/appl/admin/log
 or
$LOG_HOME/appl/admin/log

ii) Startup/Shutdown error message related to tech stack (10.1.2, 10.1.3 forms/reports/web) :

R12Apps Installation Logs:

Other log files in R12:

1) Database Tier

1.1) Relink Log files :

$ORACLE_HOME/appsutil/log/$CONTEXT_NAME // make.log

1.2) Alert Log Files :

$ORACLE_HOME/admin/$CONTEXT_NAME/bdump/alert_$SID.log

1.3) Network Logs :

$ORACLE_HOME/network/admin/$SID.log

1.4) OUI Inventory Logs :

$ORACLE_HOME/admin/oui/$CONTEXT_NAME/oraInventory/logs

APPS LOG FILE and File LOCATION

$LOG_HOME= $INST_TOP/logs

Apache Logs in R12

Access Log :  $LOG_HOME/ora/10.1.3/Apache
Error Log :  $LOG_HOME/ora/10.1.3/Apache

Concurrent Manager logs and out files in R12

$INST_TOP/logs/appl/conc
OR
$LOG_HOME/appl/conc

Note : There are four directory on these locations :
Inbound
Outbound
Log
Out

Forms and Reports related logs (10.1.2 O_H in R12 is equivalent to 806 O_H in 11..5.10.2)

Forms log file : $LOG_HOME/ora/10.1.2/forms
Reports Log file : $LOG_HOME/ora/10.1.2/reports

OPMN Log Files
$INST_TOP/logs/ora/10.1.3/opmn

OC4J Log Files (Text)
$INST_TOP/logs/ora/10.1.3/j2ee/oacore/

OC4J Log Files (ODL)
$INST_TOP/logs/ora/10.1.3/j2ee/oacore/log/oacore_default_group

SSL CERTIFICATES

$INST_TOP/certs
/Apache/ewallet.p12
/opmn/cwallet.sso
/opmn/ewallet.p12

Certs is the default directory for SSL or any other certificate files
used by this instance. To use a centrally staged certificate, change
the appropriate context file and run autoconfig

AD Scripts log files (adstrall,adstpall)
$INST_TOP/logs/appl/admin/log

AD tools log files (adpatch,actrl)
$APPL_CONFIG_HOME/admin/$TWO_TASK/log

Apps Environment file Location : $APPL_TOP
name will be APPSSID_HOSTNAME.env e.g APPSPROD_DEMO.env

Application startup & shutdown scripts Location :

$INST_TOP/admin/scripts
or
$ADMIN_SCRIPTS_HOME

Applicaton context file i.e .xml file : $INST_TOP/appl/admin/
e.g SID_HOSTNAME.xml

adpreclone.pl script location : $INST_TOP/admin/scripts
adcfgclone.pl script location : $COMMON_TOP/clone/bin

Database log file and File Location

adpreclone.pl script location : $RDBMS_O_H/appsutil/scripts/SID_HOSTNAME
adcfgclone.pl script location : $RDBMS_O_H/appsutil/clone/bin

Database listener and database startup and shutdown scripts :

$RDBMS_O_H/appsutil/scripts/SID_HOSTNAME

Database alert.log file location in 11g:
$O_H/admin/SID_HOSTNAME/diag/rdbms/sid/sid/trace
alertlog file format alter_sid.log

All .sql scripts location :

RDBMSO_H/rdbms/admin
Parameter file location i.e Init.ora or pfile or spfile file :
$RDBMS_O_H/dbs

Adpatch Log file / Worker log file / adpatch informative log file i.e adpatch.lgi

$APPL_TOP/admin/SID/log/ e.g SID=PROD

Forms.fmb Location : $AU_TOP/forms/US/

(All fmb’s reside in AU_TOP/forms/US)

R12.2. Startup and Shutdown Procedure

 Node Manager
$adnodemgrctl.sh start Enter Weblogic Admin Password:

 Weblogic Admin Server
 $adadminsrvctl.sh start Enter Weblogic Admin Password:

Application Listener
$adalnctl.sh start

Oracle Process Manager
$adopmnctl.sh start

Apache Services
$adapcctl.sh start

Managed Server for OACORE Services
$admanagedsrvctl.sh start oacore_server1 Enter Weblogic Admin Password:

Managed Server for FormsServices
$admanagedsrvctl.sh start forms_server1 Enter Weblogic Admin Password:

Managed Server for Fusion MiddleWare  Services
 $admanagedsrvctl.sh start oafm_server1 Enter Weblogic Admin Password:

Managed Server for Forms web  Services
$admanagedsrvctl.sh start forms-c4ws_server1 Enter Weblogic Admin Password:

Concurrent Manager Service
$adcmctl.sh start apps/apps

Fullfillment Serer Services
$jtffmctl.sh start

Individual Components:  Application(Middle Tier)
$ADMIN_SCRIPTS_HOME

When we want to stop all application services using script adstpall.sh , we provide apps password in R12.1.3.But in R12.2 it will ask weblogic admin password in addition to bring down to all services .

About Concurrent Managers

About Concurrent Managers

Concurrent managers are used to run concurrent jobs or programs. Concurrent programs are processed by concurrent managers.

There are four main concurrent managers. They are:

1) Internal Concurrent Manager

ICM stands for internal concurrent manager.
The role is ICM is, first it will start and monitor all other managers.
Internal concurrent manager monitor all  other managers like Standard Manager, MRP manager, Conflict Resolution Manager and etc , if it finds any other manager is down it will bring it up.
             
The main function of ICM are to startup and shutdown the individual concurrent manager and reset the other manager after one of them has failure.
If ICM goes down all other manager keeps working.
ICM will not process/Execute any request.

The concurrent manager log files can be located in one of the following:

1.If the environment variable $APPLCSF is set,  the default location
           is $APPLCSF/$APPLLOG
2.If the environment variable $APPLCSF is not set, the logs go to
           $FND_TOP/$APPLLOG

Log naming convention for ICM.

Internal manager : SID_<TIMESTAMP>.mgr

 ls -tlrh <SID>*.mgr|tail  ==>check the logfile to trubleshoot if any errors.

2) Standard Manager

SM stands for standard manager.
The role is SM is, by default all concurrent requests were handled by this, until and unless that request is assigned to particular manager.
Standard managers are used to run any kind of request.
All the unassigned and unattached jobs will be taken care by standard managers but We can assign a specific program to specific manager.

Log naming convention for SM.
=======================
W<request id>.mgr

ls -tlrh W<request id>.mgr|tail  ===> to see the current request processing by SM.

3) Conflict Resolution Manager

CRM stands for conflict resolution manager.
The role of CRM is, it will handle all the concurrent requests which are incompatible with other.
It resolves the conflict such as request incompatibilities.
When a concurrent request is being process and another manager is trying to process same kind of request ,there is an incompatibility with the concurrent request and resolved by CRM.
CRM checks what's that program and whether that program has incompatibilities (or) it is already running.

Log naming convention for CRM.
========================
c<request id>.mgr

4)Transaction Manager

TM stands for transactional manager.
The role is TM is, it will handle all online transactions, this will work synchronously, where as all other managers work asynchronously.
A transaction manager started on concurrent processing server and periodically reads the pipe for incoming transaction.

Log naming convention for TM.
=======================
t<requesd id>.mgr

What is the meaning of Concurrent ?
Parallel .

How to run a concurrent program?
In oracle apps we have concurrent program submission screen .
we can sumbit the Concurrent program/Concurrent Request from that screen.

What happens when you submit a concurrent program?
purpose of a Concurrent manager is to processes Concurrent program/jobs.

When I submit a concurrent program( or call it concurrent request), how does concurrent manager pick this up?
 Concurrent manager will be running in the background waiting for a concurrent program to be submitted. As soon as a concurrent program is submitted. Concurrent Manager put the program in execution queue .

Why does the Concurrent manager put a concurrent program into a queue? Why doesn't the manager simply let the program run?

Suppose my concurrent manager will run 10 request at a given time . First the manager puts a submitted program into a queue, next the manager checks if there is a slot available (i.e. Less than 10 programs
are currently running). If a slot is found available, the concurrent manager then runs the program, or else it keeps the concurrent program in a queue with status Pending.

How to stop the concurrent manager at OS level?

Stopping the concurrent managers

You stop the concurrent managers from the command line:

 CONCSUB apps/apps SYSADMIN 'System Administrator' SYSADMIN CONCURRENT FND
 DEACTIVATE or ABORT

 DEACTIVATE tells the managers to complete processing of any running
 requests before terminating.  This is a cleaner way of bringing down the
 managers.

 ABORT tells the managers to terminate the managers immediately.
 When the concurrent managers are terminated through the abort command,
 the jobs that they were running will be returned to pending status.

 adcmctl.sh apps/password stop

Let see the scenario:
==============

CONCSUB apps/apps SYSADMIN 'System Administrator' SYSADMIN CONCURRENT FND DEACTIVATE or ABORT

[applmgr@apps scripts]$ ps -fu applmgr|grep -i fndlibr
applmgr    607   602  0 19:09 pts/1    00:00:00 FNDLIBR
applmgr    838   764  0 19:10 ?        00:00:00 FNDLIBR
applmgr    839   764  0 19:10 ?        00:00:00 FNDLIBR
applmgr    845   764  0 19:10 ?        00:00:00 FNDLIBR
applmgr    855   764  0 19:10 ?        00:00:00 FNDLIBR
applmgr    856   764  0 19:10 ?        00:00:00 FNDLIBR
applmgr    861   764  0 19:10 ?        00:00:00 FNDLIBR
[applmgr@apps scripts]$ CONCSUB apps/apps SYSADMIN 'System Administrator' SYSADMIN CONCURRENT FND DEACTIVATE
Submitted request 390260 for CONCURRENT FND DEACTIVATE

[applmgr@apps scripts]$

[applmgr@apps scripts]$ ps -fu applmgr|grep -i fndlibr|wc -l
1
You have new mail in /var/spool/mail/applmgr

Phase status Description

PENDING/Normal -Request is waiting for the next available manager.
PENDING/Standby-Program to run request is incompatible with other program currently running.
PENDING/Scheduled-Request is scheduled to start at a future time or date.
PENDING/Waiting-A child request is waiting for its Parent request to mark it ready to run. For example, a request in a request set that runs sequentially must wait for a prior request to complete.
RUNNING/Normal-Request is running normally.
RUNNING/Paused-Parent request pauses for all its child requests to finish running. For example, a request set pauses for all requests in the set to complete.
RUNNING/Resuming -All requests submitted by the same parent request have completed running. The Parent request resumes running.
RUNNING/Terminating-Request is terminated by choosing the Cancel Request button in Requests window.
COMPLETED/Normal-Request completed successfully.
COMPLETED/Error-Request failed to complete successfully.
COMPLETED/Warning-Request completed with warnings. For example, a request is generated successfully but fails to print.
COMPLETED/Cancelled-Pending or Inactive request is cancelled by choosing the Cancel Request button in the Requests window.
COMPLETED/Terminated-Request is terminated by choosing the Cancel Request button in the Requests window.
INACTIVE/Disabled-Program to run request is not enabled. Contact your system administrator.
INACTIVE/On Hold-Pending request is placed on hold by choosing the Hold Request button in the Requests window.
INACTIVE/No Manager-No manager is defined to run the request. Check with your system administrator. A status of No Manager is also given when all managers are locked by run-alone requests.

DIAGNOSTIC SCRIPTS
===================

1.  afimchk.sql  Tells the status of the ICM and PMON method

2.  afcmstat.sql  Lists active manager processes

3.  afrqrun.sql  Lists all the running, waiting and terminating  requests
4.  afrqwait.sql  Lists requests that are constrained and waiting for the ICM to release them

5.  afrqscm.sql  Prints log file name of managers that can run a given request.  It can be used to check for possible errors when a request stays in pending status.  It requires a request id value.

6.  afcmcreq.sql  Prints the log file name of the manager that processed the request.

7.  afrqstat.sql  Summary of completed concurrent requests grouped by completion status and execution type.  It requires number of days prior to today on which to report parameter.

8.  afimlock.sql  Lists locks that the ICM is waiting to get
9.  afcmrrq.sql  Lists managers that currently are running a request


1: What is concurrent manager?

(I) When an Oracle Applications user submits a request to run a program, it's called concurrent request. Concurrent manager are the programs, which are responsible for running the concurrent requests. When a user submits a report to be run as a concurrent request, the job enters in a request queue. Concurrent managers continously read request from this master queue and run the requests based on the request's schedule, priority, and compatibility rules. Concurrent managers run in background and they take care of initiating and completing the concurrent requests.

(I) When an Oracle Applications user submits a request to run a program, it's called concurrent request.
(2) When a user submits a report to be run as a concurrent request, the job enters in a request queue
(3) Concurrent managers continuously read request from this master queue and run the requests based on the request's schedule
(4)  and compatibility rules. Concurrent managers run background and they take care of initiating and completing the concurrent requests.

2: What are the different types of concurrent manager?

Oracle Applications consist of several types of concurrent managers.
(1) The important ones are internal concurrent manager (icm)
(2)  standard concurrent manager
(3) and conflict resolution manager. (crm)
Apart from these, you can define your own custom concurrent manager.

3: What is an internal concurrent manager?

(1)The internal manager is the one which is responsible for controlling all other concurrent managers.
(2)Its main task is to ensure that all other concurrent managers are up and running.
(3)The internal concurrent manager starts, sets the number of active processes, monitors and terminates all other concurrent (4)processes through requests made to the service manager, including restarting and failed processes.
(5)The internal concurrent manager also starts and stops, and restarts the service manager for each node.

4: What is conflict resolution manager?

(1)The conflict resolution manager takes care of resolving the program incompatibilities and checks if a request in queue
(2)can be run in parallel with the running request.
(3) It also takes care of resolving the program incompatibilities.
(4) If a program is identified as run alone,
(5)then the conflict resolution manager prevents the concurrent managers from starting other programs in the same conflict domain.
(6)When a program lists other programs as being incompatible with it,
(7) the conflict resolution manager prevents the program from starting until any incompatible programs in the same domain have completed running.

5: What is a standard manager?
Ans:

(1)The standard manager is the master concurrent manager.
(2) This manager is always running and it can take care of processing any concurrent request.
(3)It has no specialization rules.
(4)This manager runs 24 hours a day for the whole year.
(5)The definition of this manager should never be altered.
(6)In case if you alter the definition of the standard manager and you have not defined additional managers to run your requests, some of the programs may not run in a proper way.

6: How do I enable/disable the conflict resolution manager?

(1)This is a system profile option "Concurrent: Use ICM".
(2) The default value of this profile option is No which allows the conflict resolution manager to be started.
(3)Setting the same to Yes will cause the conflict resolution manager to be shutdown and the internal concurrent manager will take care of the conflict resolution duties.
(4)Using the internal concurrent manager to resolve the conflicts is not recommended.

7: What are the different ways to stop concurrent manager?

(1)Concurrent manager can be stopped using the script adcmctl.sh. It can also be stopped using the Concsub utility.
(2)From the operating system, the concurrent manager can be stopped by querying the FNDLIBR process and killing the same.

8: What are the different ways to start concurrent manager?

Concurrent manager can be started using the script adcmctl.sh located at the locations of the scripts or with the startmgr utility located at $FND_TOP/bin.

9: In administer concurrent manager form there are two columns labelled as actual and target. What are these columns and what is the thier significance?

(1)Target column lists the number of processes that should be running for each manager for this particular workshift.
(2)Actual column lists the number of processes that are actually running.
(3) If the actual column is zero, there are no processes running for this manager.
(4)If the target column is zero, then either a workshift has not been assigned to this manager,
(5)or the current workshift does not specify any target processes. If the target column is not zero, then the manager processes have either failed to start up, or has gone down.

10: How do I run/schedule a concurrent request from operating system level without logging into the applications?

A concurrent request can be scheduled/run from the operating system using the CONCSUB utility. CONCSUB means Concurrent Submit.

11: What are the different ways to check if concurrent manager (CM) is running or not?

There are a couple of ways through which one can check if the CM is running or not. From the operating system, it can be checked by querying the FNDLIBR process. From the forms, it can be checked from the Navigation > Concurrent > Manager > Administer. It can be also checked using the scripts adcmctl.sh status and finally it can also be checked from Oracle Applications Manager.

12: What is the default location of the concurrent manager logfiles?

The concurrent manager log files can be located in one of the following places:
(i) If the environment variable $APPLCSF is set, the default location is $APPLCSF/$APPLLOG.
(ii) If the environment variable $APPLCSF is not set, the logs go to $FND_TOP/$APPLLOG.

13: I have submitted a request and it's showing the status inactive/no manager. Concurrent manager is up and running and the request are being picked after some time. What could be the reason for the same?
Ans:

If the concurrent manager is up and running and the request goes to the status inactive/no manager for some time it means that cache cycle is less. Cache size is set on the Concurrent> Manager> Define form. Basically, this regulates how many requests a manager will pick up for each sleep cycle. The solution is either to increase the cache size of the manager or increase the actual number of the manager process. The manager could be standard manager or any other manager for which the issue is coming.

14: I have submitted a request, it has gone to pending standby status for a long time whereas other requests are getting completed normally without any issues. What could be the reasons?

If any particular request is going to pending standby status and others are getting completed, it means that either it is waiting for the output of some other request or is conflicting with some other request. If the request is conflicting, check the queue of the conflict resolution manager for troubleshooting.

15: How do I process more concurrent requests in parallel?

If you want to process more requests simultaneously, there are two ways for the same-one, increase the number of the target process for the manager and second, change the cache size of the concurrent manager.

16: When do the tables FND_CONCURRENT_REQUESTS and FND_CONCURRENT_PROCESS need to be purged?

When the tables reach 20,000 rows, the performance begins to diminish. You may want to run purge concurrent request on a regular basis, depending on the amount of requests being run.

17: What are the concurrent request log file and output file naming conventions?

Request log files: l<request id>.req

Output files: If $APPCPNAM is not set: <username>.<request id>

If $APPCPNAM = REQID: o<request id>.out

If $APPCPNAM = USER: <username>.out

Where: <request id> = The request id of the concurrent request

And: <username> = The id of the user that submitted the request

Manager log files:

ICM log file: Default is std.mgr, can be changed with the mgrname startup parameter

Concurrent manager log: w<XXXXXX>.mgr

Transaction manager log: t<XXXXXX>.mgr

Conflict Resolution manager log: c<XXXXXX>.mgr

Where: <XXXXXX> is the concurrent process id of the manager.

18: What happens when the internal concurrent manager dies? Are all the managers also killed immediately after it?

No, if the internal manager dies, the request continues to run normally except for queue control requests.

19: Does the internal manager run or schedule any request for itself?

No, the internal manager does not run or schedule any requests. It has nothing to do with scheduling requests, or deciding which manager will run a particular request. Its function is only to run 'queue control' requests, which are requests to startup or shutdown other managers. It is responsible for startup and shutdown of the whole concurrent processing facility, and it also monitors the other managers periodically, and restarts them if they should go down. It can also take over the conflict resolution manager's job, and resolve incompatibilities.

20: If the internal manager goes down, do I need to kill all the managers before restarting the internal manager?

No, if the internal manager goes down you need not kill all the managers. You can simply start the internal manager using startmgr.

21: Can I delete concurrent manager?

Yes, you can delete any concurrent manager. For deleting, query for the manager in the defined concurrent manager form and then delete the row.

Deleting the predefined concurrent managers is not recommended and it should never be done. Deletion may cause instability in the system.

22: What is internal monitor?

This manager is used to implement parallel concurrent processing. It monitors whether the ICM is still running, and if the ICM crashes, it will restart in another node.

23: How do I clean out concurrent manager tables?

For cleaning concurrent manager tables, Oracle provides a script called cmclean.sql.

24: I hit the restart button to start the standard manager but it still doesn't start?

Asking a manager to restart sets the status to restart. The internal concurrent manager will start in the next process monitor session or the next time internal concurrent manager starts. Use activate, to start a manager immediately. Also, when a manager is deactivated manually, the internal concurrent manager will not restart it. You will need to set it to restart, or activate it manually.

25: I tried to stop the concurrent manager using the script adcmctl.sh. I can still see from the operating system that a few FNDLIBR processes are still running and adcmctl.sh is not able to stop the concurrent manager completely. What do I do in this situation?

If you are not able to stop the concurrent manager using the script, query for the FNDLIBR process using the command

ps -ef | grep FNDLIBR

And then kill the process using the command

Kill -9 <process id>

If there are more than one process of the FNDLIBR, you can kill all of them in one go using the command
Ps -ef | grep FNDLIBR | awk '{ print $2}' | xargs kill -9

26: What are the circumstances in which you need to bounce the concurrent manager?

The following are the situations in which one may need to bounce the concurrent manager.
- When you modify the definition of the printers
- When you modify the environment variables. Say you have changed the APPLTMP and APPLPTMP variable.
- When all the requests are pending and hanging

27: What are the reasons a concurrent manager hangs?

The concurrent manager hangs due to many reasons. A few of them are:
- Long running jobs
- The internal manager was activated by someone other then owner of the application system
- The operating system files system is full
- It's not able to create the log file
- You've shut down the internal manager, but actual has a number in it
- The database is hanging may be because the archive log files have filled
- Pending/standby requests are too many

28: How can you stop concurrent manager using the CONSCUB utility?

Concurrent manager can be stopped using the CONSCUB utility by the following command:

CONCSUB apps/apps@<dbname> SYSADMIN 'System Administrator' SYSADMIN CONCURRENT FND SHUTDOWN

29: What are the different parameters of the startmgr utility?

The parameters of the startmgr utility:

Parameter: sysmgr
Description: Sqlplus username/password that owns the foundation tables
Default: Applsys/<passwd>
Parameter: Mgrname
Description: The name of the Manager
Default: Internal Manager

Parameter: Log file
Description: The log file of the Manager
Default:
$FND_TOP/$APPLLOG/$mgrname.mgr
or
$APPLCSF/$APPLLOG/$mgrname.mgr

Parameter: Sleep
Description: The number of seconds the ICM should wait before checking new request from the table FND_CONCURRENT_REQUESTS
Default: 60 seconds

Parameter: Restart
Description: If the CM goes down abnormally, it will automatically restart the manager. Y = the number of minutes the ICM waits before restarting the manager
Default: N=not to restart after abnormal termination

Parameter: mailto
Description: MAILTO is a list of users who should receive mail whenever the manager terminates
Default: Current user

Parameter: printer
Description: The default printer for sending the output files
Default:

Parameter: diag
Description: This is used for diagnosis. If the CM is started with the parameter diag=y then full diagnostic output is produced in the log file
Default: N

Parameter: Pmon
Description: The number of sleep cycles ICM will wait before checking failed managers
Default: 20

Parameter: Quesiz
Description: Number of pmon cycles the ICM waits between times it checks for normal changes in concurrent manager operation. Normal changes include the start or end of a work shift and changes to the concurrent manager definitions entered in the Define Concurrent Manager form. (Default 1)
Default: 1

30: What exactly happens when a concurrent request is submitted?

Once a concurrent request is submitted by the user, the table FND_CONCURRENT_REQUESTS is automatically updated with the details of the request. The table is also updated with the information about the schedule of the concurrent request whether it's immediately scheduled or scheduled at a fixed fixed time. Once the request is scheduled to run the concurrent manager checks the FND_CONCURRENT_REQUESTS table to find out if the request is incompatible with any other request. If the request is incompatible then the conflict resolution manager takes care of the request and finds out what are the incompatibilities, it's checked whether any special manager is there to take care of this request. If there is any special manager to take care of this request then it goes to the queue of that manager else the standard manager takes care of the same. Once the request is processed, the FND_CONCURRENT_REQUESTS table is updated with the status.

31: In the administer concurrent manager form, what is the significance of the terminate button?

The terminate button is used to terminate any concurrent manager. When you terminate internal manager, all the managers automatically get deactivated and all the running requests are terminated. If you want to terminate a particular manager, select the manager and click the terminate button. The status of the manager changes to deactivate after a few seconds and all the requests processed by that manager are immediately terminated. Once a manager is terminated, it doesn't restart automatically. You have to manually restart it using the restart button.

32: In administer concurrent manager form, what is the significance of the deactive button and how can you deactivate a manager from there?

For deactivating a particular manager, select the manager and press the deactivate button. In case of deactivation, all the requests processed by the manager are allowed to complete before the manager shuts down. If you deactivated the internal manager, all the managers automatically get deactivated but all the running requests are allowed to complete before the manager is shut down. This is the only difference between termination and deactivation. In termination, all the running requests are terminated immediately whereas in case of deactivation all the running requests are allowed to complete first.

33: In administer concurrent manager form, what is the significance of the verify button and for which manager's it's available?

The verify button becomes enable only when you select the internal manager. One of the functions of the internal manager is to monitor the processes of each concurrent manager. The process of monitoring the other concurrent manager by internal manager is known as the PMON cycle. When you click the verify button you can force the process monitoring or the PMON activity to occur. The verify button is also available for the conflict resolution manager which checks for the program incompatibilities.

34: What is parallel concurrent processing and what is the significance of the same?

Parallel concurrent processing is the way to distribute concurrent managers across multiple nodes in a cluster, massively parallel, or networked environment. It helps in distributing the load across multiple nodes thereby fully utilizing the hardware resource.

The following are the advantages of the parallel concurrent processing.

Load Distribution:
Since the concurrent processing is distributed among multiple servers, as a result the load is distributed across various nodes which results in high performance.

Fault Tolerance:
When a node fails, the concurrent processes continues to run on other nodes, as a result the work is not hampered.

Single Point of Control:
The ability to administer concurrent managers running on multiple nodes from any node in a cluster, massively parallel, or networked environment.

35: Explain briefly how concurrent processing works?

In case of parallel concurrent processing, all the managers are assigned a primary and a secondary node. The managers are started in their primary node by default. In case of node failure or Oracle instance failure, all the concurrent managers on that node are switched to their secondary nodes. Once the primary node is available again the concurrent managers on the secondary nodes are migrated back to the primary node. During the migration process, a manager may be spread across both primary and secondary nodes.

In case of parallel concurrent processing, it may happen that in a node where parallel concurrent processing is configured, the Oracle instance may or may not be running. The node which is not running Oracle, the concurrent managers connects via Net8 to a node which is running Oracle.

The internal concurrent manager can run on any node, and can activate and deactivate concurrent managers on all nodes. Since the internal concurrent manager must be active at all times, it needs high fault tolerance. To provide this fault tolerance, parallel concurrent processing uses internal monitor processes. The job of the internal monitor process is to constantly monitor the internal manager and start it when it fails. Only one internal monitor process can be active on a single node. You decide which nodes have an internal monitor process when you configure your system. You can also assign each internal monitor process a primary and a secondary node to ensure fail over protection. Internal monitor processes, like concurrent managers, can be assigned work shifts, and are activated and deactivated by the internal concurrent manager.

The concurrent log and output files from requests that run on any node are accessible online from any other node. Users need not log onto a node to view the log and output files from requests running on that node.

36: Where can I define the primary and the secondary nodes for the concurrent manager form?

For defining the primary and secondary nodes of each manager, you need to launch forms with system administer and need to navigate the Concurrent > Manager > Define form. Query for the manager in which you want to define the primary and secondary node. In this screen, put the values for the primary and the secondary nodes and save.

37: I have defined for nodes of the concurrent manager. Now do I need to start the concurrent manager from all the nodes?

No, even if you have defined the concurrent manager in four different nodes you need not start the concurrent manager from all the nodes. You just need to start the concurrent manager from the primary node and GSM takes care of starting the concurrent manager from all the other nodes.

38: My requests are making error out with the error-unable to create temporary files xxxxx.tmp. How do I fix it?

This issue normally comes if the values of $APPLTMP, $APPLPTMP in the APPL_TOP and the utl_file_dir parameter of the database are not in sync. All the three variables should be exactly the same. If these issues come, change the values in the APPSORA.env, you need to bounce the concurrent manager for the changes to get effected. In case if you change the values of the init.ora, you need to bounce the database to reflect the changes. (Of course you need to bounce the application tier also if you are bouncing the database.)

39: The user comes to you saying that the request is taking a lot of time to complete. What will be your approach for debugging it?

You can do the following to debug the same.
- You can run a trace on the request id to find the expensive sql's and then tell the developer to fix the same.
- You can check the program incompatibilities in the concurrent request.
- You can check the query which the concurrent program is executing and see if it is creating any locks in the database.
- Many times the users schedule the request to run at a later time.

You can check the parameters with which the request is run. (For example, once a user came saying the request is not printing the output. On Checking the possible things, it was realized that he scheduled the request with print copies = 0.)

40: What are the things that need to be taken care when you define a concurrent program?

When defining a concurrent program the following things need to be taken care.
- Selecting an executable file to run the program
- Choosing the execution method for the program (when defining your executable in define concurrent program executable)
- Defining parameters for the program, if any
- Defining printing information
- Specifying any incompatible program that must not run while the program runs
- Choosing whether to allow users to run this report from the run reports form or from within a form. If the latter option is chosen, the form from which you want to kick-off your program needs to be modified. If the first option is chosen, the program needs to be added to a report security group.

41: How do you schedule concurrent requests?

For scheduling the concurrent requests, you need to click the schedule button while submitting the request. The concurrent program can be scheduled only once, periodically or on some specific days. You can also save this schedule for future reference and can use the same schedule for a different concurrent program by using the option apply a saved schedule. If you don't schedule the request then by default the concurrent requests are submitted immediately.

43: What does the completion option mean at the time of submitting a request?

The completion option refers to what Oracle Applications will do once the request is completed. It can notify people via email, can save the output in a file, can take a print out of the same or simply won't do anything.

44: What is a work shift?

The work shift defines the time for which the concurrent manager is active. You can define some fixed date or time for manager or can make the manager run 24*7 making it active all the times. The work shifts are defined by using the work shift form from the following navigation > Concurrent > Manager > Work Shifts.

45: What are the important scripts related to the concurrent managers and what are their locations?

The following SQL scripts located under $FND_TOP/sql are useful when diagnosing concurrent manager problems.

(i) afimchk.sql: Informs about the status of the ICM and PMON method.

(ii) afcmstat.sql: Lists active manager processes.

(iii) afrqrun.sql: Lists all the running, waiting and terminating requests.

(iv) afrwait.sql: Lists requests that are constrained and waiting for the ICM to release them.

(v) afrqscm.sql: Prints log file name of managers that can run a given request. It can be used to check for possible errors when a request stays in pending status. It requires a request id value.

(vi) afcmcreq.sql: Prints the log file name of the manager that processed the request.

(vii) afrqstat.sql: Summary of completed concurrent requests grouped by completion status and execution type. It requires number of days prior to the current date, when to report parameter.

(viii) afimlock.sql: Lists locks that the internal concurrent manager is waiting to get.

(ix) afcmrrq.sql: Lists managers that currently are running a request.

46: What are the things you need to check if you are not able to view the logs of the concurrent manager?

- You need to cross check the TNS entries.
- You need to check the DBC file.
- You need to check if the Apache/Jserv is running properly.
- You need to check if the connect descriptor is correct.
What are the meaning of the codes in the status_code and phase_code columns of the FND_CONCURRENT_REQUESTS table?
STATUS_CODE Column:
A Waiting
B Resuming
C Normal
D Cancelled
E Error
F Scheduled
G Warning
H On Hold
I Normal
M No Manager
Q Standby
R Normal
S Suspended
T Terminating
U Disabled
W Paused
X Terminated
Z Waiting
PHASE_CODE column
C Completed
I Inactive
P Pending
R Running

Post Cloning Activites

Post Cloning Activites


1)Change profile option values

2)dba directories

3)change password of custom user,standard user(FNDCPASS),SYSTEM USER

3)change parameters in xml file like 
s_applcsf/s_applout/s_appllog/s_cphost/s_jsp_main_mode/s_appltmp/s_appltmp/s_login/s_lock_pid/s_web_pid/ and many more.

4)archive log disabled for non prod and it depends on the requiement

5)SSO registration

6)workflow mailer configuration

7)db links

8)export of any schema

9)color change

10) third party tools

11)softlinks that are pointing to production.

12)printer settings

13)Upload Responsibilities if any

14)Do sanity and release instance for user access.

Applying Application Patches to Oracle EBS 11i/R12 Applications

Applying Application Patches to Oracle EBS 11i/R12 Applications

Step 1: Before applying a patch you must check whether the patch is applied or not:

Method 1:

For this we query the database:

Sqlplus apps/<apps_password>

Sql> select bug_id, bug_number from ad_bugs where bug_number='&num';

Sql> select patch_type, patch_name from ad_applied_patches where patch_name='&num';

Method 2:

Checked with OAM
a) Connect to OAM
b) Go to System Administrator --> OAM-->Dashboard --> Site map --> Maintenance --> Applied patches
c) Enter patch id and press ‘Go’
d) See if patch was returned

Step 2: Download the patch:

Example:

www.support.oracle.com

Step 3: Unzip the patch:

Exapmple: Command: 

Unzip < patch.zip>

Step 4: Before applying patch:(check invalid objects in DB) and Backup the invalids object before applying the patch:

Example:

Sql> select name from v$database;

Sql>select owner,count(*) from dba_objects where status='INVALID' group by owner; 

Backup the invalids with CTAS:

create table dba_objects_18apr2013 AS select * from dba_objects where status='INVALID';

Note:
Send outage communication/put mail to business

Step 5: Stop all application services (If we don’t want to close application services, we have hot patch option:

Note: If it is shared application file system you have to bring down admin tier service i.e, concurrent node it depends on business requirement.

ps -fu applmgr|grep -i FND|wc -l

If processes not went down, do kill processes and proceed ahead

Step 6: Enable Maintenance mode

Example:

$ adadmin

5. change maintenance mode

To check maintenance mode enable or not use below query

select fnd_profile.value('APPS_MAINTENANCE_MODE') from dual;

If  the status:

“MAINT” = MAINTENANCE MODE HAS BEEN ENABLED AND THE USERS WILL NOT BE ABLE TO LOGIN.

“NORMAL” = MAINTENANCE MODE HAS BEEN DE-ACTIVATED AND THE USERS WILL BE ABLE TO LOGIN.

Reasons For Enabling Maintenance Mode:

To ensure optimal performance and reduce downtime during patching sessions, AutoPatch requires that you enable Maintenance mode when you apply a patch. Enabling this feature shuts down the Workflow Business Events System and sets up function security so that Oracle Applications functions are unavailable to users. This provides a clear separation between normal runtime operation and system downtime for patching.

Note:
Now fire adpatch from VNC server as its process will be created on the server.
You can also use putty session but this process will run at client side. Putty
session will be inactive after 30 mins. if your patch taking more than 30 mins
your session will gone. You have to start adpatch from the start. So it's better practice to use VNC.

Step 7: Applying patch using ‘adpatch’ (auto patch utility):

FIRE :adpatch
as adpatch will ask for some questions like logfile name,system,apps,ORACLE_HOME,number of worker,driver file, once the patch is applied successfully. Please check the logfile for errors and warnings.
Use adpatch by going to patch directory
Go to the patch top directory,where all driver and required ldt files are present with the application file system owner,makes sure all files have read,write and execute permission.

As described below:

Note:It is very important to review the readme.txt in unix vi editor before applying applying patch and follow the instruction as given in the readme.txt file and apply any pre-requistics patch if required. Autoconfig run also not required unless it is specified after patch installation in readme.txt file. After this answer the questions of autopatch. As autopatch finishes its tasks, it writes timing information to the AD timing report for jobs running in parallel (if any ) and reminds you to run the log files for any errors.
If you don’t see the “autopatch is complete” message at the end of the Autopatch log file, Autopatch did not complete successfully.
The most important step after autopatch completes is to check the log files for any errors that may occurred during the patching process. Check the main Autopatch log file first,then additional log files as necessary.
The default name of main autopatch log file is adpatch.log .The file is located in
$APPL_TOP/admin/<SID>/log

Step 8 : Please check whether patch is applied or not.

Step 9: Disable Maintenance mode again by using 'adadmin' utility:

Step 10: Start application services :

Step 11:After applying PATCH:

step a:Verify the patch is applied successfully:

SQL> select name from v$database;

NAME
---------
TESTDB

SQL> select bug_number,creation_date from ad_bugs where bug_number='5522470';

BUG_NUMBER                     CREATION_DATE
------------------------------ ---------------
5549427                        12-FEB-13

Step b:Run cmclean.sql

Run cmclean.sql from application node by going to $COMMON_TOP/admin/scipts/TESTDB_ebstest in 11i,where as in R12 goto $ADMIN_SCRIPTS_HOME or $INST_TOP/admin/scripts
commit;
Note:We run cmclean after clonning also to make sure the node name is updated in the FND_NODES Table correctly

SQL> select owner,count(*) from dba_objects where status='INVALID' group by owner;

OWNER                            COUNT(*)
------------------------------ ----------
SYS                                     1
TEST_USER1                    2
TEST_USER2                    5
APPS                                 22

Step c:Check the file versions got changed successfully after applying patch:

strings -a POXWARMB.pls|grep Header =>Can be used to check the file version.

Step d:Do the Health Check of Oracle EBS Application

Note :We can get HOME page URL by using below query:
SQL>Select Home_URL from icx_parameter;

Health check completed successfully by submitting active user Concurrent request REQUEST ID 28758820.

Step e:Intimate end User 

Make sure you intimate the end User or release your application to the end User 

do sanity check and release instance for user access .
In sanity check you will submit below two requests:

Go to System adminstrator --->Request--->Run

1)Active User =====>Check View LOG/vIEW OUT
2)Active Responsiblity ===>Check View LOG/vIEW OUT


In Oracle, to apply database patch we use ‘opatch‘ ($ORACLE_HOME/OPatch) where as in order to apply applications patch we use ad utility ‘adpatch‘ ($AD_TOP/bin).
Patch Zipped File Details: When you unzip a patch you will see the following files:readme.txt - This file contain steps to apply the patch, list of prerequisites should be there on apps Instance (If not apply that patch ) cXXXXXXX.drv c stand for copy driver file , this copies patch content to respective patch location , driver is like bus driver which provides instruction on work adpatch need to perform. dXXXXXXX.drv d stand for Database driver & contain content related to database like creating packages, tables, adding column…. gXXXXXXX.drv This contain files related to forms , reports, graphics or messages uXXXXXXX.drv Sometime these three types of files are bundled together into single driver file called Unified driver file

Maintenance Mode/ Hot Patch: Maintenance Mode is mode of operation introduced with AD.I.2, in which the oracle application system is made accessible only for patching activities. Greatly improves performance by minimizing downtime. If you wish to apply patch without putting applications in maintenance mode (for small patches) use options=hotpatch with adpatch.

Ways of Reducing Patching Downtime

Ways of Reducing Patching Downtime

1.Use a staged applications system

This major time-saver hinges on a key principle:  all of your applications filesystem patches are applied to a clone of your production Apps environment.  This can be done while your production system is still running.  Your production system is down only for the time needed to apply database patches.

2.Use a shared application-tier file system

If you have a pool of application-tier servers set up for load-balancing, make sure that all of the individual servers share a single application filesystem.  Patches applied to this central shared filesystem are instantly available to all application-tier servers.

3.Distribute worker processes across multiple servers

When applying a patch that includes a large number of processes, you can reduce the downtime even further by distributing the worker processes across multiple servers on multiple nodes. Using the Distributed AD feature of AutoPatch and AD Controller, you can assign workers to run on the primary node and on other nodes that share the filesystem.

4.Merge multiple patches using AD Merge Patch

Merging patches saves time because the AutoPatch overhead of starting a new session is eliminated for those patches that are consolidated.  Duplicate linking, generating or database actions are run once only.  If two patches update the same file, AD Merge Patch will save time by applying only the latest one.  Patches can -- and should -- be merged with their listed prerequisite patches.

5.Run AD Patch in non-interactive mode

Applying a set of patches using AD Patch in non-interactive mode eliminates the delay between successive tasks.

6.Defer system-wide database tasks until the end

Using adpatch options=nocompiledb,nomaintainmrc defers system-wide database tasks such as "Compile APPS schema" and "Maintain MRC" until after all patches have been applied.  As of AD.H, AutoPatch automatically compiles the APPS schema and maintains MRC when applying standard patches.

7.Avoid resource-related bottlenecks

Patching can grind to a halt if you bump into the ceiling on your system.  Before patching, make sure that you've enabled automatic tablespace management, and that you have sufficient hardware and free disk and temp space.

Oracle EBS Patching

Oracle EBS Patching 

A Patch which fixes a bug (or) enhance/add a particular feature in existing program.

Patch is a program to fix a particular problem or enhance/add a particular feature in existing program/product/software.

Why patches require in oracle Ebs?

Patches may be required to resolve problem with application code, to fix production issues, to install new feature to upgrade components of the technology stack.

Patches is of two types

1) (Opatch) to apply database patches
2) (Adpatch) to apply application patches.

MODES IN PATCHING:

We can apply adpatch in four modes.They are

1)Interactive
2)Non - Interactive
3)Pre-install mode
4)Test  mode

1) Interactive :-

Providing all the inputs at the time of applying a patch.


2) Non - Interactive :-

Providing  the inputs for adpatch in a file is said to be Non-interactive mode.

It is a text file.

Location of the defaults file is $APPL_TOP/admin/<SID>

While working with defaults file for the first time we need to provide all the inputs. From next time onwards we have to provide only limited information like Patch location, driver file, patch log file and number of works.

Example:

$adpatch defaultsfile=/u01/erpapp/appl/admin/ebsofdb/adpatchdef.txt

From next time onwards no need to provide all the inputs,its enough to provide below inputs.
1)Logfile
2)Patch directory location
3)Driver file
4)Workers

Advantage: Reduce patch timing.

3) Pre-install mode :-

In this mode, Only copy portion is applied.('C' Driver is applied)
Database portion and generate portion will not be executed.('D' and 'G' Driver will  not be executed). Default it is set to no adpatch preinstall=n
This would be done normally during an upgrade or consolidated update. When a patch is applied in a preinstall mode then all the AD utilities are updated before the upgrade or update. If you wish to apply a patch in preinstall mode then use below syntax.

$adpatch preinstall=y

4) Test  mode :-

By using this mode, We can see the effects of applying this patch will have on your system before applying the patch.

The default is adpatch apply=y.

If you want to apply a patch in test mode follow below syntax.

Syntax: adpatch apply=no

in this mode just file comparision will be done between PATCH_TOP and APPL_TOP and that will be reported in .lgi file.
It is located under $APPL_TOP/adminn/<SID>/log/adpatch.lgi

Advantage: Will get to know what patch is doing before applying the patch on our system.

Adpatch options:

To change the default behavior of patching ,we can use below options.

Non default

nocopyportion
nodatabaseportion
nogenerateportion
nocompiledb
nocompilejsp
novalidate
nomaintainancemrc
noautoconfig
hotpatch

default

copyportion
databaseportion
generateportion
compiledb
compilejsp
validate
maintainancemrc
autoconfig
nohotpatch

Checkfile :-

The checkfile option of adpatch tells adpathc to check for already executed exec, SQL, and exectier commands. this can cause performance overheds so should be used only when specified.

$adpatch options=nocheckfile

Compiledb :-

Purpose: To compile invalid objects in the database after running actions in the database driver.
Default: compiledb (use ‘nocompiledb’ to skip)

Compilejsp :-

Purpose: To compile out-of-date JSP files, if the patch has copy actions for at least one JSP file.
Default: compilejsp (use’nocompilejsp’ to skip)

Copyportion :-

Purpose: To run commands found in a copy driver. This will copy the higher version files from patch to product top.
Default: copyportion (Use ‘nocopyportion’ to skip. Use it only when mentioned in readme of patch)

databaseportion :-

Purpose: To run commands found in a database driver. This portion includes applying the files (like sql, pls etc) to database.
Default: databaseportion (use ‘nodatabaseportion’ to skip. Use it only when mentioned in readme of patch).

generateportion :-

Purpose: To run commands found in a generate driver. This portion will generate new executable files from the copied code of patch. For example if will generate new forms files (fmx) from new .fmb files.
Default: generateportion (use ‘nogenerateporation’ to skip).

$adpatch options=nogenform
$adpatch options=nogenrep

integrity :-
Purpose: To perform patch integrity checking. Tells adpatch whether to perform patch integrity checking, which verifies that the version of each file referenced in a copy action matches the version present in the patch.
Default: nointegrity (By default the integrity is not checked)

$adpatch options=integrity

maintainmrc :-
Purpose: To maintain the MRC schema after running actions found in the database driver.
Default: maintainmrc (use ‘nomaintainmrc’ to skip)

autoconfig :-
Purpose: Tells adpatch to run Autoconfig after patch installation.
Default: autoconfig (use ‘noautoconfig’ to skip)

parallel :-
Purpose: To run actions that update the database or actions (like SQL) that generate files in parallel (like genform).

prereq :-
Purpose: Tells adpatch whether to perform prerequisite patch checking prior to running patch driver files that contain actions normally found in the copy driver.
Default: prereq (use ‘noprereq’ to skip)

$adpatch options=noprereq

validate :-
Purpose: To connect to all registered Oracle Applications schemas at the start of the patch. Adpatch validates the passwords for each schema.
Default: novalidate (use ‘validate’ to validate schema passwords)
Default: parallel (use ‘noparallel’ to skip)

Java Classes :-
If you wish adpatch not to copy new java classes from the patch .By default java classes are copied.

$adpatch options=nojcopy

Relinking :-

If you wish adpatch not do perform relinking you can use options=nolink.

Force Copy :-
By default adpatch copies the files without check the version of the existing files already present on the system.If you do  not wish the newer version of the file to be replaced by the older version contained in the patch use options=noforcecopy.

$adpatch options=noforcecopy.

You could specify multiple options at the command line using the , delimiter.

$adpatch options=nocompiledb,nocompilejsp,nojcopy
Following flags can be passed to adpatch
1) hidepw
Purpose: This argument is used to hide the passwords in log files
Default: nohidepw
2) trace
Purpose: Tells the adpatch utility whether to log all database operations to a trace file
Default: notrace
3) logging
Purpose: Tells the adpatch utility whether to create indexes using the logging or nologging mode.

What two tables adpatch creates when you apply or run adpatch session ?


FND_INSTALL_PROCESSES 
AD_DEFERRED_JOBS

Naming convention of patch

1)<p><number>_11i_<platform>

 <number>=7 digits

example: p1234567_r12_LINUX.Zip

2)P1234567_11i_GENERIC.Zip

Note: Generic indicates on all platform we can apply this patch.


Types of patch in oracle applications

One of patch

A patch that doesn't depends upon another patch. One of patches fixes one or more bugs for a single product. One of patches may or may not have the pre requisites.
These patches are small in size. They often called as standalone patches.

Mini Pack:

A mini pack is one which will upgrade any product patch set level to next level .

or

Collection of one -off patches and enhancement related to particular module.
All bug fixes related to one product which will change the patchset level of
specific product.

Minipack version is denoted by alphabetic characters.

patch+patch=Minipack

 <11i>_<product>_<minipack level>

Example: 11i.FND.H
         11I.GL.I
         11i.GL.H.1


Rollup patches

A rollup patch is one which will deliver bug fixes identified after the release of any major application versions like 11.5.8/11.5.9.

Family pack

A Family pack is one which will upgrade the patch set level of all the products in that family to particular patchsetlevel.

(OR)

Collection of minipack patches for a particular family group of application modules.

Some of the families in oracle Apps are:

 FIN == GL,AR,AP,FA

 ATG == AD,FNDAU,WF

Family pack patches will change entire product family.

product Minipack+product Minipack= Family pack.

Naming convention of family pack patch:
=========================

<Family name>_PF.<CODE LEVEL>

Example:HR_PF.J
        GL_PF.K
Maintenance pack patch

Group of family pack is called Maintenance pack patch.

A maintenance pack will upgrade applications from one version to another like 11.5.8 to 11.5.9


Consolidated  patches.

Consolidated patches will come into pictures after upgrades from one version of applications to another, all post upgrade patches will a consolidated and given as consolidated patch.


PATCH ARCHTECTURE
================
Suppose this is the patch number 1234567.
Example:P<1234567>_R12_LINUX.zip

A patch contains three things.

1)Directory with the patch number like here in this case 1234567.inside you will find below files.

2)README.txt or README.html

3)Driver files
c<patch number>.drv
d<patch number>.drv
g<patch number>.drv

u<patch number>.drv  it is ther in updated versions.


What happens when we unzip a patch?
===========================

unzip P<1234567>_R12_LINUX.zip

When we unzip a patch it create a base directory with the patch number like 1234567.
it contains all patch related files .in this directory there is a file README.txt or README.html
which contains instruction how to apply a patch and also it contain patch number,product,release
and platform.if patch is too long it contain document id which helps us to apply a patch.
it also contain driver files.

c<patch number>.drv
====================
Copies the files that are there in patch to the APPL_TOP
Extracts the appropriate files from each product c library.
Re-links the oracle application product.

Compares the files in the patch with those in the APPL_TOP,if the files in the patch are a higher
version, then adpatch copies the files from patch to APPL_TOP

d<patch number>.drv
===================
The database driver contains all commands to change the database objects. In multi node installation d driver runs
from the admin server. The database driver applies all script copied by the copy driver to the database.

==> Make a list of all invalid objects in the database.
==>Run sql scripts,which make changes to the database.(Packages,procedures and etc).
==>Compile all the invalid objects that are there in the database.

g<patch number>.drv
==================

Regenerate all forms,reports and PL/SQL libraries that have been effected by the patch.

u<patch number>.drv
===================
It is combination of all three drivers c,d and g.The action of each driver is performed by u driver.


Q) Who will created the patches?

Patches are created by Oracle whenever there are enhancement in its application or there are any problem with the exists ones.


Q) After Patches are Applied Successfully, it will update the patch history in AD_SNAPSHOTS.


what is a snapshot?

There are two types of snapshosts, APPL_TOP and global. APPL_TOP snapshots contain version of files and patches applied within that APPL_TOP. A global snapshot contains the same information but for the entire environment, ie. all APPL_TOPS.

The global view snapshot is used when you use Patch Wizard to determine whether or not a patch has been applied. APPL_TOP snapshots are using by autopatch (adpatch) to determine if all prerequisite patches have been applied. Each time you apply a patch AutoPatch updates the current view snapshot. I believe it may even create a new current view snapshot and just replace the existing one.

Additionally there are two types of snapshots, current view and named. A named snapshot is just a copy of the current view snapshot at a given point in time. Patch wizard and AutoPatch use current view snapshots.

To access snapshot information launch adadmin, select option 2 - Maintain Applications Files menu, select 5 - Maintain snapshot information and you will see the menu below:


Maintain Snapshot Information
-------------------------------------------

1. List snapshots

2. Update current view snapshot

3. Create named snapshot

4. Export snapshot to file

5. Import snapshot from file

6. Delete named snapshot(s)

7. Return to Maintain Applications Files menu

Back to the problem of adpatch seeming to hang while instantiating a current-view snapshot. Since this is a cloned environment, a snapshot doesn't exist yet for the APPL_TOP. So before AutoPatch can check if prerequistite patches have been applied, it must create a snapshot. This process can take 1-2 hours depending on how fast your servers are. You can avoid this by running "Update current view snapshot" via adadmin after you clone.

I should add, that in my experience I've only encountered this problem a few times. Most of the patches I apply to a cloned environment are quick one-offs with no pre-reqs. Large patching efforts such as family packs or patches with pre-reqs may experience this problem. I haven't tried, but if your crunched for time you may be able to bypass adpatch updating the current view snapshot by specifying "adpatch options=noprereq" to skip the prerequisite check.

Patching Best Practices

As an Oracle Applications DBA we tend to spend a considerable amount of time applying patches. A newbie Applications DBA recently asked me the duties of an Applications DBA besides patching .At that moment i refrained myself just to a smile. But the question does highlight the significance of patching in an AppsDba's routine.

Patching, though not a very complicated process, inefficient or inappropriate patching can seriously jeopardize the  functioning of your system. Most of the patching for oracle applications is done using adpatch tools and there are many more which have other methods of application like through a shell script. However in my current post i will talking only about the adpatch patches. The 'best practices' mentioned here are the ones which I have felt to be of use and have made my patching  life a bit less complicated.

Patching Methodology
Having a proper patching process and methodology is always helpful. The ideal one for me is the PATCH -> TEST -> DEVELOPMENT  -> PRODUCTION one. Under this the patch is first applied to a patch instance then it is propagated to the test environment where the testing is done after which it is applied on the development instance where the development team tests the patch against there customizations if any. Upon passing these stages the patch is finally applied to the production system. If you cannot afford these many instances at minimum you must ensure the patch is tested on TEST environment before you actually go 
ahead and apply the patch to your production.


Consistency
There should be consistency in the patch application process. That is you should use the same steps to apply the patch in your  production environment that you used in your TEST/PATCH environment for example if you applied a pre requisite patch A as part 
of your main patch application, you must ensure the same pre requisite patch is also applied on the production environment and  not a superseded version of patch A.
Needless to mention there must be also consistency among the all the different environments. That is your PATCH/TEST/DEVELOPMENT and PRODUCTION environments should have the same setups configuration and patchset levels.


Schedule Patch Application
Schedule the patch application as far as possible. There are multiple benefits you realize out of scheduling. The most  important one being that the downtimes will be scheduled and will have help the business to be prepared for them and not be  taking them by surprise. Also if your patching downtimes are scheduled you can plan to have the downtime following your backup schedule thereby eliminating the need for having a separate backups for each patch application. However there will be high priority patches that will have applied outside this schedule.


A Patch in Time
It is always advisable to be at the latest patch set level at most of the times. I have always felt that it is one of your  duties as an Apps DBA to take the fear of patching out of the business's mind. My experience sys that being on the latest  Consolidated Update or the latest Roll up patch helps in avoiding most of the pre requisite in case of applying any one off patch which might be required. Also you should apply the Critical Patch Updates that are released by oracle every quarter.


The Readme
Before applying a patch make sure you have gone through the readme file of the patch. This file might contain some special or additional steps that you might have to follow as a part of the patch application. Also it will tell you if there are any pre requisite patches that are require. Downloading and having the pre requisites ready saves you a lot of time.


Pre requisite Patch
In case you are required to apply a pre requisite patch before you can apply the main patch. it is better to apply the pre requisite patch only and not the superseded version of the pre requisite patch. In case you choose to apply the super seeded version of the of the pre requisite patch you must doubly make sure that the superseded patch qualifies as a pre requisite patch for the main patch.


Patch Impact Analysis
A patch impact analysis should be carried out as far as possible before going ahead with the actual pat6ch application. The simplest way of doing it would be to apply the path with apply=no option along with adpatch. You can then go through the actions the patch would have done either from the logfile or view it through the Oracle Applications manager.


NLS Environment
In case you are patching an NLS environment. Check if you have a translation patch also available for the main patch. if one is available make sure to download and keep it ready. You must install the US language patch first and immediately follow it by applying the translation patch.


Merge Patches
In cases where you are doing a bulk patch application like during a production setup the patch application time is reduced  greatly by merging the patches using AD Merge Patch. However per requisite patches should not be merged. Also in an NLS environment its is not advisable to merge multiple language translation patches.


The Logfile
I most patching scenarios the patch log file is either over written by accepting the default logfile name adpatch.log or in other cases these are not maintained properly. I have found myself digging up patch logfiles for a variety of reasons, though you have a lot of this information available to you via Oracle Applications Manager's Patch Reports.


Maintenance Mode
With the latest AD.I patch set there is a pre requisite to put the system on maintenance mode before applying a patch. I have come across DBA's by passing this option by using options=hotpatch for reducing the patching related downtime. There is a reason for having the maintenance mode feature out there, it offers certain performance benefits and reduces the chances of any conflicts. There are other ways to reduce the patching related down times, which i will touch upon briefly next.


Reduce Your Patch Downtime
As discussed earlier merging your patches is one of the ways to reduce your patching related downtimes. There are also certain options you could use with adpatch which could reduce your downtime significantly. You must however understand each of there options clearly before using them with adpatch. For e.g. choosing not to compile invalid objects might reduce your patching downtime but you must be sure of the invalid objects or you must manually run the compilation script once you are through with all your patch application.
You can also implement the concept of a Shared Application Tier File System and a Staged APPL_TOP to further bring down your patching downtime and effort.


Automate If Possible
In case your business can afford it you should opt for automating the patch application and management process using third party tools. My favorite ones have been Ringmaster APM and Kintana. You might have to still wait a little bit more to have all these features inbuilt through your oracle applications manager.

Difference between a multi-node and a shared environment & how is patching impacted

In a multi-node (distributed) system, servers are installed on more than one node.  The core technology directories (admin, ad, au, and fnd) and all product directories are installed under the APPL_TOP on all nodes, except for any node that contains only a database.  You can patch the APPL_TOPs separately, or you can set up a shared environment that will enable you to perform patching and other maintenance procedures only once. This type of system shares not only the APPL_TOP and COMMON_TOP file systems, but the Applications node technology stack as well. All Applications nodes are installed on a single, shared disk resource that is mounted under the same name from each Applications node machine. Any Applications node can be used to provide standard services, such as serving forms or Web pages, concurrent processing, or applying patches. Maintenance procedures are performed on a single Applications node.

If you have not configured your system as a shared environment, you apply the unified driver to all APPL_TOPs. AutoPatch sorts out which driver actions it must perform on each APPL_TOP. You apply the driver first to the node where the administration server is located. Then, you apply it to the node where the concurrent processing server is located. Start the concurrent managers, and apply the driver to the remaining nodes in any order.

How is the maintenance of Oracle Applications simpler in Release 12?  Are patches easier to apply?

In Release 12, Oracle keeps major new features and bug fixes separate. Significant improvements to maintaining Applications in Release 12 include:
  • Major new functionality is reserved for point releases
  • A new maintenance branch will be created for each point release
  • New feature introduction into maintenance branches is limited and requires executive approval
  • Codelevels make checking for prerequisites easier
  • Patch Application Assistant (PAA) enhances tracking of manual steps during patch application
What are codelines? Codelevels?

In Release 12, Oracle Applications patches are grouped into codelines. A codeline begins with a point release that delivers a unique set of features, such as Release 12.0, and progresses to include all the patches required to maintain that point release. The initial Release 12.0 point release introduces codeline A. Any future point releases introduce new codelines, each identified by a new letter.  For example, Release 12.1 would introduce codeline B and Release 12.2 would introduce codeline C.

Patches associated with codelines not only implement a set of product features for that point release, but also provide fixes to that set of features.  We describe this unique set of product features for a point release as a codeleveland assign it a unique number. Codelevels also identify patches for individual products. For example, codelevel R12.AD.A.1 is the first set of fixes to codelevel R12.AD.A, R12.AD.A.2 is the second set, and so on. Codelevels are cumulative - each one contains the initial set of features plus all the fixes created to date for that product or product family.

All codelevels created after the initial point release are aggregated into cumulative release update packs (RUPs). RUP1 is equivalent to R12.0.1, RUP2 is equivalent to R12.0.2, and so on.  In addition, they may also provide feature enhancements, which provide new functionality that has a limited impact on your system.

A new point release contains new features that will substantially impact your system and may change its operation.  It starts a new codeline (for example, codeline B).  At that point, you can choose to upgrade to the new codeline and adopt the new features, or stay on your existing codeline, where bug fixes and enhancements will continue to be provided for your existing features.

How has patch prereq checking improved in Release 12 compared to Release 11i?

In Release 11i, patches could require other individual patches as prerequisites. This process was flexible, but it was difficult to determine if a new patch included all the functionality of an older patch.

In Release 12, patches can only require a codelevel as a prerequisite and sometimes based on the technical reasons may require other individual patches as prerequisite. That means if a patch requires R12.GL.A.3 as a prerequisite, it is clear that R12.GL.A.4 satisfies the prerequisite and R12.GL.A.2 does not.

What is a recommended patch?

Recommended patches include high-priority patches or patches that update your system to a new codelevel. Use the Patch Wizard utility in Oracle Applications Manager (OAM) to determine which recommended patches you have not yet applied and what the effect of applying an individual patch will be on your system.

What is a patch driver file?

AutoPatch uses a driver file to direct the installation of a patch. This unified driver is named u<patchnumber>.drv. It contains all the driver actions (copy, database, and generate) that the patch requires, and it performs these actions in the stated order. Typically, you run the driver on all APPL_TOPs and AutoPatch determines which actions are required for the current APPL_TOP and runs only those actions.

The driver actions are as follows:
  • Copy: Contains commands to change Oracle Applications files. The commands include directives to copy and update files, libraries, and/or Java, and commands for generating JAR files and/or C executables. In a multi-node system, the copy portion runs on all application tier APPL_TOPs.
  • Database: Contains commands to change Oracle Applications database objects, such as PL/SQL and table definitions, or to update or migrate data. In a multi-node system, the database portion runs only on the application tier APPL_TOP that implements the administration server.
  • Generate: Contains commands to generate forms, reports, messages, and/or graphics files. In a multi-node system, the generate portion runs on all application tier APPL_TOPs, unless the APPL_TOP only implements the administration server.
What is the AutoPatch checkfile feature?

The checkfile feature reduces patch application downtime by checking to see if a given database action has been performed previously for the associated file contained in the patch. If an action has been performed using the current (or higher) version of a file, AutoPatch omits the action from the current patch application.

What are the Oracle Applications patch types?

Applications patches are organized by aggregation level. 

Individual Bug Fix: A patch that fixes an existing issue. Individual bug fixes are released only when there is an immediate need for a fix or enhancement that cannot wait until a release update pack (RUP) is available. Although individual bug fixes are intended to be as small as possible, they usually include any dependent files that have changed since the base release in order to form a complete patch that can be applied by any customer. The actual number of files changed will depend on the current code level on the system to which the patch is being applied.

Product Family Release Update Pack (Product Family RUP): An aggregation of patches on a given codeline created for all products in a specific product family for a specific point release. 

Release Update Pack (RUP):
 An aggregation of product family RUPs on a given codeline created across Oracle E-Business Suite after the initial release. RUPs are cumulative and include all one-off bug fixes previously released to customers as well as legislative updates and less critical fixes. New functionality is also allowed in RUPs with executive approval.

Pre-upgrade patch: All upgrade-related, high priority patches consolidated from all the products within a product family.  Pre-upgrade patches are released as needed.  The 
Oracle Applications Release Notes lists the most recent pre-upgrade patches.

Consolidated Upgrade Patch:
 All upgrade-related patches consolidated from all the products within a product family. These patches are released as needed and are only available for upgrading a Release 12 system from one point release to another. Release 11.5.10. The Oracle Applications Release Notes 
Note 405293.1 lists the most recent patches.

Documentation Patch: Updates online help.   

What are AutoPatch restart files?

Restart files store information about completed processing in the event of a patch or system failure. They allow AutoPatch and AD Administration to continue processing at the point where they stopped. Do not modify or delete restart files unless specifically told to do so by Oracle Support Services.

The restart files reside in $APPL_TOP/admin/<SID>/restart (UNIX) or in %APPL_TOP%\admin\<SID>\restart (Windows). 

What is the difference between a multi-node and a shared environment and how is patching impacted?

In a multi-node (distributed) system, servers are installed on more than one node.  The core technology directories (admin, ad, au, and fnd) and all product directories are installed under the APPL_TOP on all nodes, except for any node that contains only a database.  You can patch the APPL_TOPs separately, or you can set up a shared environment that will enable you to perform patching and other maintenance procedures only once. This type of system shares not only the APPL_TOP and COMMON_TOP file systems, but the Applications node technology stack as well. All Applications nodes are installed on a single, shared disk resource that is mounted under the same name from each Applications node machine. Any Applications node can be used to provide standard services, such as serving forms or Web pages, concurrent processing, or applying patches. Maintenance procedures are performed on a single Applications node.

If you have not configured your system as a shared environment, you apply the unified driver to all APPL_TOPs. AutoPatch sorts out which driver actions it must perform on each APPL_TOP. You apply the driver first to the node where the administration server is located. Then, you apply it to the node where the concurrent processing server is located. Start the concurrent managers, and apply the driver to the remaining nodes in any order.


Can I apply multiple patches in one operation?

Before you run AutoPatch, use AD Merge Patch to merge multiple patches into a single, integrated patch so that the required patching tasks and processes are performed only once. In general, you can safely merge any Oracle Applications patch with another Oracle Applications patch. Patches should be merged with their listed prerequisite patches to make the application of the patch easier. However, patches that affect the Applications DBA (AD) product may change the AutoPatch utility itself. So, they can be merged only with other AD patches and must be applied separately, before you apply any non-AD patches.

Note: AD Merge Patch cannot merge patches of different releases, different parallel modes, or different platforms. However, it can merge patches for a specific platform with a generic patch, or patches with different source character sets. The utility notifies you if you try to merge incompatible patches.
How do I apply multiple translation patches?

If an Oracle Applications system contains multiple languages other than American English (US), and you are applying multiple patches for each language, the recommended method is to merge all US patches into a single patch and all patches for every non-US language into a single patch. Then, apply the merged US patch followed by the merged language patch.

You can also merge US patches with the additional language patches or merge each language in separate language-specific patches. Depending on your downtime window and your system topology, it may be necessary to keep the US and non-US patches separate.

Can I run multiple AutoPatch sessions at the same time?

You cannot run multiple sessions simultaneously (concurrently). However, patches can be merged and applied in a single patching session.

Do patches need to be applied in a particular order? What is a prerequisite patch?

AD patches are the only patches that must be applied in a specific order. This is necessary because you may need to patch the patching utility itself so that it works properly when you use it to apply subsequent patches. It is not necessary to apply non-AD patches in a particular order, even though a readme may state a specific order is required.

A prerequisite patch fulfills a dependency for another patch.  In Release 12, only codelevels can be prereq patches. Strictly speaking, they are co-requisites and can be applied in any order before using the system. We recommend that you merge a patch with its required prerequisites, with the exception of prerequisite patches for the AD product.


Do I have to perform all the manual steps associated with a patch?

Oracle Patch Application Assistant (PAA) helps you track and perform manual steps during patching. For patches that have manual steps, the patch readme file contains the generic instructions for all systems.  PAA generates a customized set of instructions specific to your installation and displays the relevant manual steps.  For merged patches, PAA automatically merges the contents of the individual patch readme files. After successfully completing each step, you can record that step as completed for future reference.


Can I automate the patching process?

Non-interactive patching allows you to save time by automating the patching process and avoiding some of the prompts. You can store the responses to the patching prompts in a defaults file.  Then, when you run AutoPatch, you specify the name of the defaults file, the location of the patch top directory, the name of the driver file and other parameters in the command line.
How can I track my customizations? What happens to my customizations during patching?

You should apply patches first on a test system. Then, review the changes in the test system and identify the best way to re-integrate customizations affected by the patch.

You need to register flagged files through Register Flagged Files tool in Oracle Applications Manager (OAM). Before applying AutoPatch, you need to run Patch Impact Analysis from OAM to know the list of customizations that would be overridden with the patch application. The Register Flagged Files tool replaces the applcust.txt file that was used in 11i and is described in 
Oracle Applications Patching Procedures.

Note: Registering customized files does not prevent the object or the patch from being applied. It only makes them available to OAM for review. See Customization Standards in 
Oracle Applications Developer's Guide for more details.
ASSESSING THE EFFECTS OF PATCHING

Can I determine ahead of time how a patch will affect my system?
You can analyze the actions a patch will take by reviewing patch log files without applying a patch to production or you can access a Patch Impact Analysis report through the Patch Wizard in Oracle Applications Manager (OAM) to see how a patch will affect the files on your system.

If you want to review log files, you can apply a patch on a test system. Alternatively, you can apply the patch in production using the AutoPatch test mode. Applying a patch in test mode requires that you use the AutoPatch optionapply=no.  The resulting log file shows all the actions that AutoPatch will take.

To determine how a patch will affect the files on your system, you can request a Patch Impact Analysis report for a specific patch through the Patch Wizard in OAM. The Patch Impact Analysis feature of Patch Wizard provides links to details about a patch including the following information:
  • The total number of files in the patch
  • The number and type of files the patch will install
  • The products that will have updated files
  • The files that will be introduced by the patch
  • The files on the target system that will be changed by the patch
How do I know what patches have been applied to my system?

The best way to review patching history is to use the Applied Patches utility provided by Oracle Applications Manager (OAM). From the Applied Patches interface, you can perform a simple search by querying on the patch number, the number of days or date range during which patches were applied and/or the patch language.  An advanced search provides additional search criteria.  The search results display useful information including patch name, description, a list of merged patches, location of applied patch, language, files changed or copied, bug fixes in each driver file, whether patch application was successful and timing information.
     
How do I know what files have been updated by a patch?

One of the features of the Applied Patches utility in Oracle Applications Manager (OAM) is the File History option. Using this feature, you can search for files that have been updated by a patch.  You can view file history information such as the APPL_TOP and directory where the file resides, the product that owns the file, the file name and version, the date on which the file was changed, the patch details report and the action summary report for updates to the file.

How can I find out how much time a patch took to apply? How can I monitor the progress of a patch?

The Timing Reports utility in Oracle Applications Manager (OAM) allows you to monitor a job that is running or view statistics of completed AutoPatch and AD Administration sessions.  You can view information such as task name, task status, time taken to complete the task, start time and end time, and log files.
TROUBLESHOOTING

If I am applying a patch and it fails, should I simply re-run it from the beginning after fixing the issue?

If a patch driver fails, fix the issue and restart AutoPatch. AutoPatch will allow you to continue where the patch left off. Re-running the patch from the beginning may result in it being applied incorrectly.

Can I restart a failed patch non-interactively?

Use the restart argument to continue the old session from where it failed.  Use the same command line options that you used initially to start the non-interactive session, but add restart=yes.  

If AutoPatch fails during the copy portion of the unified driver, what should I do?

In many cases, the issue can be resolved and the patching process can be restarted at the point of failure.  If there is no feasible method of resolving the issue, review the log files and the copy portion of the unified driver to determine the files copied by the patch and the update actions performed. Depending on the number of files that were copied prior to the failure, you must decide the best method for restoring the files. Based on the actions recorded in the log file, you must decide which files, if any, need to be relinked, restored, and generated. 

If a worker fails during the database portion of the unified driver, what should I do?

Attempt to correct the problem and restart the failed job.
To restart a failed job, run AD Controller and choose Option 2 to tell the worker to restart a failed job. Enter the worker number when prompted.
What should I do if a worker process is hanging?

When running AutoPatch, there may be situations when a worker process appears to hang or stop processing.  If this occurs, you may need to terminate the process manually.  If you do, you  must also restart the process manually.  A process that appears to be hanging could be a valid long-running job.  Be careful when terminating processes. 

Can I restore my system after the database portion of the patch fails?

There is no general method of backing out changes that a patch makes to the Oracle Application database.  To avoid the need to restore a database, you should always test the application of the patch several times on a test system, particularly if the patch is a release update pack (RUP) or pre-upgrade patch.  Do not apply the patch in production until the patch has successfully applied in a test system.

What happens in pre-install mode ?

It updates the adutilities

Unified APPL_TOP in Oracle Apps R12

Unified APPL_TOP is different from separate APPL_TOP in Oracle Applications 11i. Unified APPL_TOP make more sense if you are using multi node Oracle Applications R12.
In Oracle Applications 11i , in multi node installation each APPL_TOP have its different name and files in each APPL_TOP will depend on type of Node (i.e. Forms Node will have fmx or forms related files where as APPL_TOP belonging to CM only node will have rdf or files required to run CM node)

Starting from R12, it will use Unified APPL_TOP which means all files required for all middle tier services are included in all nodes of Multi Node installation (irrespective of services running on that node)

What are few changes because of Unified APPL_TOP ?

— During Cloning of multi node Oracle Application install, only one copy (any one appl_top) of Application Node files need to copied to target instance.
— During Cloning of Multi Node to Single Node you don’t have to merge APPL_TOP’s in R12 as required in 11i .
—Services start/stop If you are using adstrtall.sh to start services on a node then it will start services which were configured to start during install on that node (You can still start a specific service even though it was not suppose to configure/start by calling startup script of that specific services).
To explain this better, lets assume you installed multi node R12 instance with Node 1 as Forms & Web Server where as Node 2 was installed with Concurrent Manager. Now on Node1 when you use adstrtall.sh script to start services as expected it will start Forms and Web services but unlike 11i, You can still start Concurrent Manager on Node1 by running adcmctl.sh
—FND_NODES From R12 all nodes will have Y against all services (For multi-node) in FND_NODES table.