Workflow Logfiles And Troubleshooting

Workflow Logfiles And Troubleshooting

Workflow Log file location:

Logs on CM Node at $APPLCSF/$APPLLOG/FNDC*.txt

To identify the 4 more recent log files for Workflow Mailer and Agent Listener services, run the

following command:

# ls -lrt $APPLCSF/$APPLLOG/FNDCPGSC*.txt

Current workflow mailer logfile location with sql:

select fl.meaning,fcp.process_status_code, decode(fcq.concurrent_queue_name,'WFMLRSVC','maile r container','WFALSNRSVC','listener container',fcq.concurrent_queue_name), fcp.concurrent_process_id,os_process_id, fcp.logfile_name from fnd_concurrent_queues fcq, fnd_concurrent_processes fcp , fnd_lookups fl where fcq.concurrent_queue_id=fcp.concurrent_queue_id and fcp.process_status_code='A' and fl.lookup_type='CP_PROCESS_STATUS_CODE' and fl.lookup_code=fcp.process_status_code and concurrent_queue_name in('WFMLRSVC','WFALSNRSVC') order by fcp.logfile_name;

Check workflow mailer service current status:
 
select running_processes from apps.fnd_concurrent_queues where concurrent_queue_name = 'WFMLRSVC';

Number of running processes should be greater than 0


To check current mailer status:

select component_status from apps.fnd_svc_components where component_id = (select component_id from apps.fnd_svc_components where component_name = 'Workflow Notification Mailer');

Possible values:

RUNNING
STARTING
STOPPED_ERROR
DEACTIVATED_USER
DEACTIVATED_SYSTEM


How to determine if the workflow services are up and running:

$FND_TOP/sql/wfver.sql output also shows the status of the workflow services:

Service Instance Status
______________________________________________________________________________
Workflow Agent Listener Service is enabled -> Actual: 1, Target: 1
Workflow Mailer Service is enabled -> Actual: 1, Target: 1
Workflow Document Web Services Service is enabled -> Actual: 1, Target: 1

The following query also returns the status of the workflow services:

select fcq.USER_CONCURRENT_QUEUE_NAME Container_Name, DECODE(fcp.OS_PROCESS_ID,NULL,'Not
Running',fcp.OS_PROCESS_ID) PROCID,
fcq.MAX_PROCESSES TARGET,
fcq.RUNNING_PROCESSES ACTUAL,
fcq.ENABLED_FLAG ENABLED,
fsc.COMPONENT_NAME,
fsc.STARTUP_MODE,
fsc.COMPONENT_STATUS
from APPS.FND_CONCURRENT_QUEUES_VL fcq, APPS.FND_CP_SERVICES fcs, APPS.FND_CONCURRENT_PROCESSES
fcp, fnd_svc_components fsc
where fcq.MANAGER_TYPE = fcs.SERVICE_ID
and fcs.SERVICE_HANDLE = 'FNDCPGSC'
and fsc.concurrent_queue_id = fcq.concurrent_queue_id(+)
and fcq.concurrent_queue_id = fcp.concurrent_queue_id(+)
and fcq.application_id = fcp.queue_application_id(+)
and fcp.process_status_code(+) = 'A'
order by fcp.OS_PROCESS_ID, fsc.STARTUP_MODE;


The TARGET and ACTUAL columns should be 1. If one of these columns is 1 and the other one zero then the workflow service is not running


What to do when the workflow services go down:

Make sure the Service Component Containers are started from OAM


Login as SYSADMIN in OAM then navigate to:

1. Sitemap > Generic Services > Status Overview > View All > Generic Service Component Container >

select Workflow Agent Listener Service > Start

2. In the same page select Workflow Mailer Service > Start

Now the components should start according with the start mode that is set:

- Automatic
- On-Demand
- Manual

Make sure Concurrent GSM Enable profile option is set to Yes:

select DECODE(b.profile_option_value, 'Y', 'Enabled', 'Disabled') DETAILS
from fnd_profile_options a, fnd_profile_option_values b
where a.APPLICATION_ID = b.APPLICATION_ID
and a.PROFILE_OPTION_ID = b.PROFILE_OPTION_ID
and a.PROFILE_OPTION_NAME = 'CONC_GSM_ENABLED';

When the workflow services are down, this is usually solved by running the Concurrent Manager Recovery:

a. Ensure the Concurrent GSM Enable profile option is set to Yes otherwise this action plan won't work

b. Stop the concurrent managers, workflow mailer and wf services

c. Make sure there are no orphan processes
NOTE: The recovery wizard (which is the program you will run in the next steps) They may need to be killed.
This works on Linux for find the Workflow Service processes ps -ef|grep -i FNDCPGSC*.

d. Go into OAM

e. Navigate to Site Map > Concurrent Manager Recovery (under Diagnostics)

f. Click Run Wizard button
If the OAM User Interface isn't the same or if you have problems then use the Help/Search screen to

locate the Concurrent Manager Recovery Wizard. The
location of this wizard changes almost with every new version of OAM.

g. Retest the issue by starting the workflow services from OAM

Make sure the Service Service Manager:<NODE> is up and running in the concurrent manager tier otherwise the workflow services won't start:

select CONCURRENT_QUEUE_NAME, ENABLED_FLAG, MAX_PROCESSES, RUNNING_PROCESSES
from FND_CONCURRENT_QUEUES where CONCURRENT_QUEUE_NAME like 'FNDSM%';

If the Service Service Manager:<NODE> is up and running, max_processes and running_processes will return

1. If one of them is zero then it is not running.

This can be done also in the command line by running the following commands:

- ps -ef | grep FNDSM
- record the PPID for the FNDSM process
- ps -ef | grep <PPID # for FNDSM>


The FNDSM PPID will be the parent process of the other concurrent manager services in the environment

Run the CP Java Regression Test concurrent program to identify if there are dbc problems:

a. System ADministrator Responsibility
b. Requests menu option
c. Run
d. Select from the list of values: CP Java Regression Test concurrent program
Parameters:
Run mode = FULL


Run the following to confirm that the required jar files exist or will exist in appsborg2.zip:

$JAVA_TOP/appsborg2.zip is the file that should contain the jar files used by the concurrent manager and workflow services as the work. They are added when you regenerate your jar files and adjborg2.txt is read and used to rebuild appsborg2.zip.

grep jar $APPL_TOP/admin/adjborg2.txt

unzip -l $JAVA_TOP/appsborg2.zip | grep -i jms | wc -l
For R12:

$ unzip -l $COMMON_TOP/java/lib/appsborg2.zip | grep -i jms | wc -l


If this returns 0, then regenerate the jar files using ADADMIN and run the command again on your concurrent manager node to see if the count changes.

The $JAVA_TOP/appsborg2.zip is also used for making HTTP(S) connections for the framework region to the iAS web server. It uses the http_client.jar file that is on the CCM node server. Although, you may not have a web server running on the machine, you will have an$IAS_ORACLE_HOME that contains the files required by the CCM to make HTTP requests. The $APPL_TOP/admin/adjborg2.txt will lists all the jar and zip archives that will be used to build the $JAVA_TOP/appsborg2.zip archive. jdbc14.zip for example is the version of the jdbc zip file used be JDK 1.4.2 and above. JDK1.3.1 uses jdbc12.zip.

$APPL_TOP/admin/adjborg2.txt and $APPL_TOP/admin/adovars.env servers the same purpose as the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties file does for the jserv engine for the web server.

Reviewing the output of $FND_TOP/sql/wfver.sql in the Service Component Control Event Subscriptions section:

Reviewing the output of $FND_TOP/sql/wfver.sql script it might seen following:

Service Component Control Event Subscriptions Name
--------------------------------------------------
Subscription Rule Function Subscription Out Agent Status
------------------------------ ------------------------------ ----------
oracle.apps.fnd.cp.gsc.bes.control.group
WF_RULE.ERROR_RULE@INSTANCE WF_CONTROL@INSTANCE DISABLED

or

oracle.apps.fnd.cp.gsc.bes.control.group

WF_RULE.ERROR_RULE@INSTANCE WF_CONTROL@INSTANCE Subscription Not Defined

The first example shows shows the subscription is not enabled and second example shows that no subscription exists for this control group / event.

Increase the Workflow Mailer's Log Level to Statement in order to get additional diagnostic information for the Workflow Mailer:

Changing the Workflow Mailer Log Level:

a. Login to E-Business as a user assigned the System Administrator responsibility
b. Navigate to System Administrator responsibility
c. Oracle Applications Manager
d. Workflow
e. Notification Mailers
f. Edit
g. Advanced


After trying to start the Workflow Services, the mailer log files show the following error:

java.lang.AbstractMethodError: getTypeMap
at oracle.AQ.AQOracleSession.<init>(AQOracleSession.java:84)
at oracle.jms.AQjmsSession.getTopic(AQjmsSession.java:2743)
at oracle.apps.fnd.wf.bes.ControlConnection.<init>(ControlConnection.java:114)
at oracle.apps.fnd.wf.bes.ConnectionManager$1.run(ConnectionManager.java:112)
at oracle.apps.fnd.wf.bes.Utilities$1.run(Utilities.java:570)
at java.lang.Thread.run(Thread.java:512)
Could not start Service Component Container ->
oracle.apps.fnd.cp.gsc.SvcComponentContainerException: BES system could not establish connection to the control queue after 180 seconds

This means there are incorrect JDK entries and missing reference to appsborg2.zip file in AF_CLASSPATH.

WF Mailer Service cannot be started, mailer log file is not created and there are no errors in the ICM and FNDSM log files:

Make sure the number of processes for the WF Mailer service in the Define Concurrent Manager form is set to 1 or greater

A. Log in as SYSADMIN

B. Go to Define Concurrent manager and query your "Workflow Mailer Service"

Navigation path:
a. System administrator responsibility
b. Concurrent> Manager> Define
c. Query the Workflow Mailer Service
d. Click on work shifts button
e. Do you have zero in the processes column?
If so, please change it to 1.
f. Save the changes
g. Wait for sometime to find out if the Workflow Mailer Service starts automatically.