Process Scheduler All Processing Suspended If you are stuck running an application engine program through the process scheduler with a message like this: All Processing Suspended: Restart OPRID=PS, RUNID=PS, PI=12345 (108,503) Navigate to: PeopleTools > Application Engine > Manage Abends To clear out the abended app engine process - matching the relevant process instance.Scheduled Jobset going to Completed If you have a scheduled jobset that is going to completed, then it could be because there is a restartable app engine in the job definition. To fix, find and restart the app engine that failed through the restart request option in process monitor. If the app engine is failing due to an error, you'll need to fix this first. Once it runs to success it will complete the job and also set the jobset status back to Active from Complete.Process Groups Process groups are used in PeopleSoft to implement security to access processes and jobs through the process scheduler. Process groups are associated with Permission Lists. They can be accessed through the following portal navigation: PeopleTools > Security > Permissions & Roles > Permission Lists Then click on the Process Group Permissions hyperlink located in the Process page. To find which permission lists have access to which process groups use the following query: select * from PSAUTHPRCS order by CLASSID, PRCSGRP; Not Authorized To View Report The following error occurs when you attemp to view a report either through report manager, process scheduler or using a direct URL. This is caused by the report distribution node being incorrectly configured. To check your settings: PeopleTools -> Process Scheduler -> Report Nodes Check that the URL for the distribution node correctly matches the report node definition. Also if you have more than one report node definition, check that the correct one is being used. Also make sure that your servers are now pointing to the updated distribution node: PeopleTools -> Process Scheduler -> Servers -> Distribution Behind the scenes: The distribution settings are stored in: PS_CDM_DIST_NODE The server settings are stored in PS_SERVERDEFN. Process Sequence Numbers The table, PS_PRCSSEQUENCE stores the last sequence numbers for the process scheduler and report manager. In PeopleTools 8.48.15 there are 5 process sequence keys (PRCSSEQKEY) which you can view yourself by looking at the translates on this field: 0 = Process Instance 1 = Report Instance 2 = Transfer Instance 3 = Report ID 4 = Folder ID Its probably a good health check to ensure that the numbers in PS_PRCSSEQUENCE match those in the relevant tables: select max(PRCSINSTANCE) as PRCSSEQKEY0 from PSPRCSRQST; select max(CONTENTID) PRCSSEQKEY1 from PS_CDM_AUTH; select max(TRANSFERINSTANCE) PRCSSEQKEY2 from PS_CDM_LIST; select max(PSRF_REPORT_ID) PRCSSEQKEY3 from PSRF_RINFO_TBL; select max(PSRF_FOLDER_ID) PRCSSEQKEY4 from PSRF_FINFO_TBL; Especially if you happen to be clearing out data from these tables manually. If you do clear out processes/reports you'll want to update the PS_PRCSSEQUENCE accordingly.Report Manager Default File Type End users normally use report manager to view process output. One thing you can do to make things easier form them is to default the file type that opens when a user clicks on process hyperlink in the Administration tab in report manager. For example, if your process creates a log file, output file and a CSV file, chances are you want the CSV (spreadsheet) file to open by default. To make this happen, you need to change the Output Destination Options in your process definition under the Destination tab. For a CSV file, the settings would be: Type = Web Format = Comma delimited (*.csv) If you aren't sure what combination of Type/Format you need, set both of them to Any in your process definition, then schedule your process with different combinations until you find the right one: This should help you find the appropriate combination.Message Log The message log is used by a number of processes to provide debugging and troubleshooting information. You can view the message log from the process monitor after clicking on the details hyperlink for a process and the by clicking on the message log hyperlink in the actions area. You may occasionally want to query the information in the message log. To do this there two tables that you need to use: PS_MESSAGE_LOG PS_MESSAGE_LOGPARM You can use the following query to get the message log data for a given process instance: select ML.DTTM_STAMP_SEC, ML.JOBID, ML.PROGRAM_NAME, ML.MESSAGE_SET_NBR, ML.MESSAGE_NBR, MLP.PARM_SEQ, MLP.MESSAGE_PARM from PS_MESSAGE_LOG ML inner join PS_MESSAGE_LOGPARM MLP on ML.PROCESS_INSTANCE = MLP.PROCESS_INSTANCE and ML.MESSAGE_SEQ = MLP.MESSAGE_SEQ where ML.PROCESS_INSTANCE = 'PROCESS_INSTANCE' You can write to the message log with an Application Engine by simply using the log message action or by using the MessageBox function in PeopleCode.Not Authorized To Run Process Type If you get the following messaging stating: You are not authorized to run process type {Process Type} and process name {Process Name}. (65,8) It means that your user profile does not have a permission list that has access to the process group for the relevant process definition. For example, I received this message when attempting to run the dynamic role publish process DYNROLE_PUBL. If you open this process through: PeopleTools > Process Scheduler > Processes Then look in the process groups tab, you will find it is only associated with the INALL process group: None of the permission lists that I belonged to had access to this group. Here's a bit of SQL to check this quickly: select OC.OPRID, OC.OPRCLASS, AP.PRCSGRP from PSOPRCLS OC inner join PSAUTHPRCS AP on OC.OPRCLASS = AP.CLASSID where OC.OPRID = 'OPRID' and AP.PRCSGRP = 'PRCSGRP' ; Replace OPRID with the appropriate operator ID (e.g. PS) and PRCSGRP with the appropriate process group (e.g. INALL). The solution is to either add a permission list with access to this process group to your user profile OR to add the process group to a permission list you already have access to through the permission list set up pages.Catch Up Recurrences Trap Recurrences are a key part of batch scheduling. They also have a default feature (or trap depending on your perspective) that makes processes perform a catch up if they have been missed by the recurrence pattern. What does this mean? Well for example if you use the delivered Daily recurrence and your scheduled jobset is stopped for a week and then restarted, the 6 days that were missed when the process should have run will all be scheduled and run at once on the process scheduler. This can lead to havoc on your process scheduler and generally ends in contention issues as there will be more processes scheduled than can be run. The longer the catch up period and the greater the frequency of the recurrence the worse the problem gets! This feature is almost always a bad idea and it should be turned off on all your recurrences by checking the Do not schedule any processes missed form the recurrence pattern check box. If you ever do want to perform catch up, do it a controlled manner (i.e one at a time). Most processes are also designed to simply process what was missed and will catch up automatically by running just once. NOTE: if you want to script this, this is configured in the table, PS_PRCSRECUR on the field RECUR_USECURRDATE, a 1 means this is set, a 0 means it is turned off. To turn off the catch up trap, you want this field to be set to 1 (do not schedule any process missed form recurrence pattern is true). Long Running Processes The following query can be used to identify any long running processes from the process request table PSPRCSRQST. It identifies any processes that took more than 300 seconds (5 minutes) to run using the difference of the process begin date/time and process end date/time that were run over the past 7 days. You can easily change these values to whatever parameters are most suitable for your environment. Having this information can help identify processes that are taking a long time to run and may be holding up other processes. If you have more than one process scheduler available, it may be worth moving such long running processes (if they can't be optimised) to another process scheduler to improve performance for other users. If you only have the one process scheduler in production, then the information from this query may be grounds for justifying for another process scheduler to be put in place. select PQ.SERVERNAMERUN, PQ.PRCSINSTANCE, PQ.PRCSTYPE, PQ.PRCSJOBNAME, PQ.PRCSNAME, PD.DESCR, PQ.OPRID, PQ.RUNCNTLID, ( select XLATSHORTNAME from PSXLATITEM where FIELDNAME = 'RUNSTATUS' and FIELDVALUE = PQ.RUNSTATUS ) as RUNSTATUS, PQ.RUNDTTM, PQ.RQSTDTTM, round((PQ.ENDDTTM - PQ.BEGINDTTM) * 24 * 60 * 60) || ' seconds' as PROCESSING_TIME from PSPRCSRQST PQ inner join PS_PRCSDEFN PD on PQ.PRCSNAME = PD.PRCSNAME where round((ENDDTTM - BEGINDTTM) * 24 * 60 * 60) >= 300 and trunc(PQ.RUNDTTM) >= trunc(sysdate - 7) order by PRCSINSTANCE desc; NOTE: this SQL is written for an Oracle database, but it wouldn't be hard to modify for other platforms. SMTP Failed on Process Scheduler If you are seeing messages like this in your message log when running a process through the process scheduler and distributing the information via email: SMTP sendMail failed (server yourserver.com:25). Cannot send email to example@test.com Failed to connect to the SMTP server at :25 It indicates that the SMTP settings have not been configured correctly on your process scheduler servers. What you need to do is open your process scheduler configuration file (%PS_HOME%\appserv\prcs\\psprcs.cfg) and find the following section: [SMTP Settings] ;========================================================================= ; Settings for SMTP mail ; All controls under SMTP Settings can be dynamically changed ;========================================================================= SMTPServer= SMTPPort=25 SMTPServer1= SMTPPort1=0 SMTPSender=PeopleSoft@peoplesoft.com SMTPSourceMachine= SMTPCharacterSet= SMTPEncodingDLL= SMTPTrace=0 SMTPSendTime=0 SMTPUseSSL=N SMTPSSLPort=465 SMTPClientCertAlias= SMTPUseSSL1=N SMTPSSLPort1=465 SMTPClientCertAlias1= Change the following parameters: SMTPServer= the fully qualified address of your SMTP server. Make sure this server is accessible from the process scheduler server. SMTPPort= default SMTP port is 25. If you are planning on using SSL, consult PeopleBooks. Make sure you can reach the target port (e.g. through telnet) from your process scheduler server. SMTPSender= the send address to use for emails, e.g. noreply@yourcompany.com SMTPSourceMachine= make sure you set this to the fully qualified name of your process scheduler. You shouldn't need a restart if you have dynamic changes enabled in your configuration. Otherwise you will need a restart. There is a flag in the file psprcs.cfg that will tell you this: [Process Scheduler] ... Allow Dynamic Changes=N ... Reauthenication Required to View Reports I came across a scenario where the user would need to re-authenticate with PeopleSoft in order to view reports through view/log trace in the Process Monitor or through the report manager. When clicking on the report, a new window would appear with the PeopleSoft Sign-on screen and message, Your User ID and/or Password are invalid: If the user re-authenticates through this screen, then the message no longer appears again for their session. However, if they close their browser, login to PeopleSoft again and then try to view a report, the message re-appears. This turns out to be an important fact, as the problem is in fact related to session cookies. The issue occurs when the Secure Cookie with SSL option is set in the web profile, but the environment is not using SSL. For example, it is cloned from an SSL-enabled environment. To tell which web profile you are using, check your web profile history under: PeopleTools > Web Profile > Web profile History Or use the viewconfig portal servlet directive. Then, open that web profile through: PeopleTools > Web Profile > Web Profile Configuration Then check the security tab and SSL group box: As described towards the end of the help text (click on the ? mark): NOTE: if you select this check box, you are effectively disabling single signon to any non-SSL servers. After changing this setting, you might need to reload the web profile configuration by either restarting your web server or using the reloadconfig portal servlet directive in order for it to take effect. Another thing to check if you are seeing a signon screen when viewing a report is the URL specified in your report node. If the URL is fully qualified with a domain, and you are not using an authentication domain, this might be causing the signon screen to appear. Try removing the domain from the URL and (just leaving the hostname) in your report node if you are not using an authentication domain to see if this helps.Check Process Override Options This SQL checks all process definitions where the parameter list, command line or working directory has been changed in the process definition override options. You might find that the majority of the processes returned are Crystal Reports so you can amend the SQL to skip this process type if required. select PRCSTYPE, PRCSNAME, DESCR, ( select XLATSHORTNAME from PSXLATITEM XI where FIELDNAME = 'PARMLISTTYPE' and FIELDVALUE = PD.PARMLISTTYPE and EFFDT = ( select max(EFFDT) from PSXLATITEM where FIELDNAME = XI.FIELDNAME and FIELDVALUE = XI.FIELDVALUE ) and EFF_STATUS = 'A' ) as PARMLISTTYPE, PARMLIST, ( select XLATSHORTNAME from PSXLATITEM XI where FIELDNAME = 'CMDLINETYPE' and FIELDVALUE = PD.CMDLINETYPE and EFFDT = ( select max(EFFDT) from PSXLATITEM where FIELDNAME = XI.FIELDNAME and FIELDVALUE = XI.FIELDVALUE and EFFDT <= sysdate) and EFF_STATUS = 'A' ) as CMDLINETYPE, CMDLINE, ( select XLATSHORTNAME from PSXLATITEM XI where FIELDNAME = 'WORKINGDIRTYPE' and FIELDVALUE = PD.WORKINGDIRTYPE and EFFDT = ( select max(EFFDT) from PSXLATITEM where FIELDNAME = XI.FIELDNAME and FIELDVALUE = XI.FIELDVALUE and EFFDT <= sysdate) and EFF_STATUS = 'A' ) as WORKINGDIRTYPE, WORKINGDIR from PS_PRCSDEFN PD where (PARMLISTTYPE != '0' OR CMDLINETYPE != '0' OR WORKINGDIRTYPE != '0') order by PRCSTYPE, PRCSNAME ; Error while loading shared libaries libsqrbtunicode.so If you get the error: $PS_HOME/bin/{DBTYPE}/bin/sqr: error while loading shared libraries: libsqrbtunicode.so: cannot open shared object file: No such file or directory. The process scheduler cannot find the libsqrbtunicode.so file which lives under $PS_HOME/bin/{DBTYPE}/bin. To fix, append to LD_LIBRARY_PATH in the file psprcsrv.env under the /appserv/prcs/{domain}/ folder. NOTE: this file doesn't seem to like environment variables so you might have to hard code the path. Error while loading shared libraries libcobrts64.so If you get the error: PSRUN: error while loading shared libraries: libcobrts64.so: cannot open shared object file: No such file or directory You need to append to LD_LIBRARY_PATH in the file psprcsrv.env under the /appserv/prcs/{domain}/ folder. Add the $COBDIR/lib location (base directory of your Microfocus server installation). NOTE: don't use environment variables, hardcode the location of $COBDIR in this file. Users with Recurrences It can be quite useful to know which users have recurrences configured in the process monitor. These are processes that users have scheduled to run on a regular basis at a scheduled time. The following SQL will return this information from the process request table (PSPRCSRQST): select p.PRCSINSTANCE, p.PRCSTYPE, p.PRCSNAME, p.RUNDTTM, p.RECURNAME, p.OPRID, o.OPRDEFNDESC, o.EMPLID, o.EMAILID, p.RUNCNTLID, p.LASTUPDDTTM from PSPRCSRQST p left outer join PSOPRDEFN o on p.OPRID = o.OPRID where p.RECURNAME != ' ' and p.RUNSTATUS = '5' ; You might be a bit curious as to why I have a left outer join to the operator definition table. Every user that has a reoccurrence in the system should exist right? Well, turns out if the operator has been deleted, and the recurrence is left behind it can cause a bit of a problem. For instance if you are seeing the following in your message log: 1:45:40PM You are not authorized to run process type SQR Report and process name PRCS1. 1:45:40PM You are not authorized to run process type SQR Report and process name PRCS2. And also seeing unique constraint errors on the PS_MESSAGE_LOG table like this (this is for an SQR running on Oracle): PRCSAPI.SQC,Update-Process-Status,Update,PSPrcsRqst SQL Status = 1, SQL Error = ORA-00001: unique constraint (SYSADM.PS_MESSAGE_LOG) violated Error on line 56: (SQR 3301) Program stopped by user request. SQR for PeopleSoft: Program Aborting. Then go and check your process scheduler logs for further information (SCHDLR_MMYY.LOG) If you find something like the following, it will tell you the user that was deleted. Go back and check if this user has an recurrences configured in the system using the SQL provided at the start of this post. =================================Error=============================== Database error encountered Info: Section: Info: SQL Stmt: SELECT A.ACCESSID ,A.ACCESSPSWD ,A.ENCRYPTED FROM PSOPRDEFN O ,PSACCESSPRFL A WHERE O.OPRID = :1 AND A.SYMBOLICID = O.SYMBOLICID ===================================================================== =================================Error=============================== Msg Set: 65 Msg #: 67 Message: Unable to retrieve the access profile for this user ID (65,67) ===================================================================== To fix, I recommend recreating the operator ID and then cancelling the recurrences from the process monitor. NOTE: you can't see recurrences in the process monitor for users that have been deleted! You could also try manually changing the RUNSTATUS field in the process request table from a value of 5 (scheduled) to say 8 (cancelled). However, proceed with caution if you go down this road.Processes Stuck at Queued There are a number of reasons why a process might be stuck at queued. The most obvious is that the process scheduler is down (check the Servers tab in the process monitor). If that's not the issue, check the following tables: PSPRCSRQST PSPRCSQUE PSPRCSPARMS The row count should be the same in both tables. If one is out of sync with the other, then it can help to remove orphaned instances in of the tables. A few other things to check: Another process may be queued and blocking subsequent processes from running. That process will need to be fixed first. Any database errors e.g. Tablespace full or disk full Check Restarting the process scheduler (and master scheduler if you have) and clearing the process scheduler cache will also fix a number of issues. Remember too that restartable Application Engines will abended with an All Processing Suspended message. The following query will give you a summary of the requested processes by process status for further troubleshooting. select RQST.RUNSTATUS, RQST.PRCSTYPE, ( select XLAT.XLATLONGNAME from PSXLATITEM XLAT where XLAT.EFFDT = ( select max(XLAT_ED.EFFDT) from PSXLATITEM XLAT_ED where XLAT_ED.FIELDNAME = XLAT.FIELDNAME and XLAT_ED.FIELDVALUE = XLAT.FIELDVALUE ) and XLAT.FIELDNAME = 'RUNSTATUS' and XLAT.FIELDVALUE = RQST.RUNSTATUS ) as RUNSTATUS_XLAT, count(RQST.PRCSINSTANCE) as TOTAL_PROCESSES, min(RUNDTTM) as FIRST_OCCURRED, max(RUNDTTM) as LAST_OCCURRED from PSPRCSRQST RQST group by RQST.RUNSTATUS, RQST.PRCSTYPE order by RUNSTATUS_XLAT, RQST.PRCSTYPE Only Application Engine Programs Stuck at Queued You may find that only application engine programs are stuck at queued while other processes (SQRs, crystals etc) run to success. This typically happens due to processes blocking the process scheduler queue. Check the process scheduler/master process scheduler logs. You might see something like this: Checking Process cancels... (NET.113): Client ChkAeStatus3 service request succeeded Process 2319373 is still running as Session ID 19955 (NET.113): Client ChkAeStatus1 service request succeeded Process 2319395 is still running as Session ID 19946 (NET.113): Client ChkAeStatus2 service request succeeded Process 2319404 is still running as Session ID 19950 Application Engine : 3:3 Requests found in Process Request table 3 This indicates that the three process instances, 2319373, 2319395 and 2319404 are all running. As there is a maximum of 3 application engine programs that can run at any one time and there are currently 3 running, all other application engine programs requested will remained at queued. However, the three process instances may not actually be running. If this is the case, they will need to be manually stopped, for example with a process scheduler restart and perhaps by manually killing the processes on the process scheduler server if required.Active Processes Ever wondered where the active processes value on the server list tab comes from? Well it actually uses a view - PS_PMN_PRCSACTV_VW. This view uses the underlying records PS_SERVERMONITOR and PS_SERVERCLASS. It is the ITEMCOUNT field from PS_SERVERMONITOR that gives you the active processes count by process type (SQR, COBOL, Application engine etc)..Process Run Status In the PeopleTools look at the translates on the field RUNSTATUS or use the query. select FIELDVALUE, XLATLONGNAME from PSXLATITEM where FIELDNAME = 'RUNSTATUS' Here's a summary of the run status translates. Note that not all of these are active. Value Status 1 Cancel 2 Delete 3 Error 4 Hold 5 Queued 6 Initiated 7 Processing 8 Cancelled 9 Success 10 Not Successful 11 Posted 12 Unable to Post 13 resend 14 Posting 15 Content Generated 16 Pending 17 Success with Warning 18 Blocked 19 Restart The following query will give you a summary of the process run statuses in your process request table: select RUNSTATUS, ( select XLATSHORTNAME from PSXLATITEM where FIELDNAME = 'RUNSTATUS' and FIELDVALUE = RUNSTATUS ) as RUNSTATUS_DESCR, count(PRCSINSTANCE) from PSPRCSRQST group by RUNSTATUS order by RUNSTATUS; As a PeopleSoft Administration it is worth running this query regularly to check if there any unusual processes statuses. Suppressing Files from the Report Repository I once wrote an application engine to extract photographs from the PeopleSoft database (BLOBs) and to put the photos into a common location on the application server. This was an application engine program. However, the only problem was that the photos being outputted (in .jpg format) were also going to the report repository. This was a hard one to troublesehoot as the .jpg extension had never been configured in the Process Scheduler system settings under distribution file options: PeopleTools > Process Scheduler > Process Scheduler > System Settings > Distribution File Options This meant that to the end user, the links were never created to the files on the report repository but they were indeed going there. It was only when the application engine batch processed a lot of photos and the report repository came to a grinding halt that this became apparent. The fix required suppressing output to the report repository. Here's how: PeopleTools > Process Scheduler > Processes > {Select the Process} > Destination On the destination page, change the output destination options from the default to: Type = File Format = Other (or select appropriate file format if available) Destination Source = User Specified User specified is the default (seee PeopleSoft Process Scheduler PeopleBooks) and means that the output destination is determined by the process run control designation. SQRs must use this as their setting as well as any application engine programs that output files. Distribution The following tables store information about the distribution of process scheduler output to users or roles. PS_PRCSDEFNCNTDIST Distribution settings for process definitions. PS_PRCSJOBCNTDIST Distribution settings for jobs. PS_PRCSRQSTDIST Distribution settings by process instance. Join to PS_CDM_FILE_LIST on the process instance to get the file names. PS_PRCSRUNCNTLDIST Distribution by operator ID and run control. System Process Requests System Process Requests is a component (PRCSMULTI) located under: PeopleTools > Process Scheduler > System Process Requests That allows you to run a number of system level administration processes, jobs and tests. For developer's this is a great place to find out how to do something as it provides a number of different ways of using the process scheduler. I believe one of the best ways to get better at PeopleSoft development is to understand the system side of things -- there may be options out there that you didn't even know about! After you create a run control, there are three pages in the component: Process Request Dialog for running system processes Component Interface to test running a process (XRFMENU) through the PROCESSREQUEST component interface ProcessRequest PeopleCode to test running a process through PeopleCode Press the run button on the run control and you'll be give a list of system processes. You can select one or more of these and run them. The processes provided are a combination of system processes, example/test processes, and jobs. Some of the useful ones include: All Process Type (ALLTYPES) job which runs test COBOL, Crystal and SQR programs. Database Designer/Database Audit (DDDAUDIT) which checks for inconsistencies between PeopleSoft records/indexes and the equivalent database records/indexes. Export User Tables (EXPRTUSR) which runs a data mover script located at PS_HOME\scripts\userexport.dms and exports all users (operators) and out of your system and into an output file called USEREXPORT.DAT. This is also a good example of how to run datamover scripts from the process scheduler. Process scheduler server clean (PRCSSRVCLN) which seems to be for clearing the process scheduler cache? Process scheduler system purge (PRCSYSPURGE) for purging process scheduler requests, archiving report manager tables, and it also calls PSXPARCHATTR for archiving XML publisher reports. XML Publisher File Cleanup (PSXPCLEAN) for cleaning up working tables associated with XML Publisher. XML Publisher Purge (PSXPDELATTR) which publishes the message for the PSXP_CLEANATTR through Integration Broker. This appears to be for purging XML publisher report search data. This message has a handler, PurgeAttributes which lives in the application package class PSXP_REPORTMGR.AttributeDelAsync and this is the code that does the cleanup work. Swap Audit Report (SWPAUDIT) which checks for data integrity problems that may occur when swapping language settings. System Audit (SYSAUDIT) which is a very powerful tool for checking for inconsistencies in PeopleTools definitions. A number of cross reference (XRF*) reports. These are useful for impact analysis - for example if you want to see what fields are referenced by PeopleCode (in case you want to modify that field) then run XRFFLPC. The PeopleCode that runs the processes through the PROCESSREQUEST component interface is located in Component Record Field PeopleCode: PRCSMULTI.GBL.PRCSSAMPLEREC.RUNCCNTLCOMPINTF.FieldChange. This code serves as another good example of how to use a component interface through PeopleCode. The PeopleCode that runs the processes through the ProcessRequest PeopleCode is located in Component Record Field PeopleCode: PRCSMULTI.GBL.PRCSSAMPLEREC.RUNCCNTLPRCSRQST.FieldChange. If you ever want to fire another process in your PeopleCode program, then this is how to do it (or you can use the component interface approach).