Mainframe Testing – Complete Tutorial

Before learning Mainframe Testing concepts, let us first look at the platform the tests run on.

What is a Mainframe?

The mainframe is a high performance and high-speed computer system. It is used for large scale computing that requires high availability and strong security. It is mostly used in sectors like finance, insurance, retail and other critical areas where huge volumes of data are processed many times a day.

Mainframe Testing

Mainframe Testing is a process of testing software applications and services based on Mainframe Systems. The purpose of Mainframe Testing is to ensure the performance, reliability and quality of a software application or service through verification and validation methods, and to check whether it is ready to deploy.

While performing Mainframe Testing, the tester mainly needs to know the navigation of the CICS screens. These screens are custom built for specific applications. When changes are made to the code in COBOL, JCL and similar languages, the tester does not have to worry about the emulator set up on the machine, because changes that work on one terminal emulator will work on the others.

• The Mainframe application (otherwise called job batch) is tested against the test cases developed using requirements.
• Mainframe Testing is usually performed on the deployed code using various data combinations set into the input file.
• Applications that run on the mainframe can be accessed through a terminal emulator. The emulator is the only software that needs to be installed on the client machine.

Because the platform behaves differently from a web stack, it helps to know which mainframe characteristics drive the test design. Mainframe Testing therefore sits alongside the other types of software testing rather than replacing any of them.

Mainframe Attributes

1. Virtual Storage
   • It is a technique that lets a processor simulate main storage that is larger than the actual amount of real storage.
   • It is a technique to use memory effectively to store and execute various sized tasks.
   • It uses disk storage as an extension of real storage.
2. Multiprogramming
   • The computer executes more than one program at the same time. But at any given moment only one program can have control of the CPU.
   • It is a facility provided to make efficient use of the CPU.
3. Batch Processing
   • It is a technique by which any task is accomplished in units known as jobs.
   • A job may cause one or more programs to execute in a sequence.
   • The Job scheduler makes a decision about the order in which the jobs should be executed. To maximize the average throughput, jobs are scheduled as per their priority and class.
   • The necessary information for batch processing is provided through JCL (JOB CONTROL LANGUAGE). JCL describes the batch job — programs, data and resources needed.
4. Time Sharing
   • In a time-sharing system, each user has access to the system through the terminal device. Instead of submitting jobs that are scheduled for later execution, the user enters commands that are processed immediately.
   • Hence this is called “Interactive Processing”. It enables the user to interact directly with the computer.
   • Time-share processing is known as “Foreground Processing” and the batch job processing is known as “Background Processing.”
5. Spooling
   • SPOOLing stands for Simultaneous Peripheral Operations Online.
   • SPOOL device is used to store the output of a program or application. The spooled output is directed to output devices like a printer (if needed).
   • It is a facility exploiting the advantage of buffering to make efficient use of the output devices.

Classification of Manual Testing in Mainframe

These attributes split manual test work on the mainframe into two clearly separated streams.

Mainframe Manual Testing can be classified into two types:

1. Batch Job Testing —

• The testing process involves execution of batch jobs for the functionality implemented in the current release.
• The test results extracted from the output files and the database are verified and recorded.

2. Online Testing —

• Online Testing refers to testing of CICS screens, which is similar to testing a web page.
• The functionality of the existing screens could be changed, or new screens could be added.
• Various applications can have enquiry screens and update screens. The functionality of these screens needs to be checked as part of the online testing.

How to Do Mainframe Testing

1. The Business team prepares requirement documents, which determine how a particular item or process is going to be modified in the cycle of release.
2. The testing team and the development team receive the requirement document. They figure out how many processes will be affected by the change. Usually, in a release, only 20-25% of the application is affected directly by the customized requirement. The remaining 75-80% of the release effort goes to the out-of-the-box functionality, such as testing the surrounding applications and processes.
3. So, a Mainframe application has to be tested in two parts:
   • Testing Requirements — Testing the application for the functionality or the change mentioned in the requirement document.
   • Testing Integration — Testing the whole process or other applications which receive or send data to the affected application. Regression Testing is the primary focus of this testing activity.

Mainframe Automation Testing Tools

Below is the list of tools which can be used for mainframe Automation Testing.

• REXX — the scripting language shipped with z/OS, widely used to drive repetitive job submission and output checks.
• Excel — used with macros to build, compare and report on test data and output files.
• OpenText UFT One — the current name of the tool the industry still calls QTP or QuickTest Professional; it automates 3270 terminal screens.
• Galasa — an open-source, Open Mainframe Project deep integration test framework that drives 3270 screens, JCL batch jobs and Db2 from a CI/CD pipeline.
• Vendor z/OS test suites — IBM Test Accelerator for Z and BMC AMI DevX Total Test cover COBOL unit testing and virtualised test environments.

Whichever tool is chosen, it only pays back when it sits inside a maintained test automation framework rather than a loose pile of scripts.

Methodology in Mainframe Testing

Let us consider an example: An XYZ insurance company has a member enrollment module. It takes data both from the member enrollment screen and from offline enrollment. As discussed earlier, it takes two approaches for Mainframe Testing, online testing and batch testing.

• Online testing is done on the member enrollment screen. Just like a web page, the database is validated with data entered through the screens.
• Offline enrollment can be paper enrollment or enrollment on a third party website. The offline data (also referred to as batch) will be entered into the company database through batch jobs. An input flat file is prepared as per the prescribed data format and fed to the sequence of batch jobs. So for mainframe application testing we can use the following approach.
  • The first job in the line of batch jobs validates the data entered — for example special characters, or alphabets in number-only fields.
  • The second job validates the consistency of data based on business conditions. For example, a child enrollment should not contain dependent data, or a member zip code which is not available for service by the enrolled plan.
  • The third job modifies the data into the format that can be entered into the database. For instance, deleting the plan name (the database will store only the plan ID and insurance plan name), appending the date of entry, and similar changes.
  • The fourth job loads the data into the database.
• Batch job testing is done on this process in two phases —
  • Each job is validated separately, and
  • Integration between the jobs is validated by providing the input flat file to the first job and validating the database. (Intermediary results have to be validated for extra caution.)

The following is the method followed for Mainframe Testing:

Step 1) Shakedown/Smoke Testing

The main focus in this stage is to validate whether the code deployed is in the right test environment. It also ensures that there are no critical issues with the code. This is the mainframe equivalent of Smoke Testing on any other platform.

Step 2) System Testing

Below are the types of testing done as part of System Testing.

1. Batch Testing — This testing is done by validating the test results on output files and the data changes made by the batch jobs under testing scope, and recording them.
2. Online Testing — This testing is done on the front end of the mainframe application. Here the application is tested for correct entry fields like an insurance plan, interest on the plan, and similar values.
3. Online-Batch Integration testing — This testing is done on systems having both batch processes and an online application. The data flow and interaction between the online screens and the batch jobs is validated.

   (Example for this type of testing — Consider an update on plan details such as an increase of interest rate. The change of interest is done on an update screen, and the balance details on the affected accounts will be modified only by a nightly batch job. Testing in this case is done by validating the plan details screen and the batch job run for updating all the accounts.)

4. Database Testing — The databases where the data from the mainframe application is held (IMS, IDMS, Db2, VSAM/ISAM, sequential datasets, GDGs) are validated for their layout and data storage.

Step 3) System Integration Testing

The primary purpose of this testing is to validate the functionality of the systems which are interacting with the system under test.

These systems are not directly affected by the requirements. However, they use data from the system under test. It is important to test the interface and the different types of messages (like Job Successful, Job Failed, Database updated) that can flow between the systems, and the resulting actions taken by the individual systems.

Types of testing done in this stage are

1. Batch Testing
2. Online Testing
3. Online — Batch Integration Testing

Step 4) Regression Testing

Regression Testing is a common phase in any type of testing project. This testing in Mainframes ensures that batch jobs and online screens which do not directly interact with the system under test (or do not come in the scope of requirements) are not affected by the current project release.

In order to have effective regression testing, a particular set of test cases should be shortlisted depending on their complexity and a regression bed (test case repository) should be created. This set should be updated whenever new functionality is rolled out into the release. Where the regression bed is too large to run in full, Risk Based Testing is used to decide which jobs and screens are re-run first.

Step 5) Performance Testing

This testing is done to identify the bottlenecks in high hit areas like front end data entry and online database updates, and to project the scalability of the application. Long-running batch windows are usually examined with Stress Testing against peak volumes.

Step 6) Security Testing

This testing is done to evaluate how well the application is designed and developed to counter anti-security attacks.

Twofold security testing should be done on the system — mainframe security and network security.

The features which need to be tested are

1. Integrity
2. Confidentiality
3. Authorization
4. Authentication
5. Availability

Steps Involved in Batch Testing

1. After the QA team receives the approved package (the package contains procedures, JCL, control cards, modules and similar items), the tester should preview and retrieve the contents into PDS as required.
2. Convert the production JCL or development JCL into QA JCL, otherwise called JOB SETUP.
3. Copy the production file and prepare the test files.
4. For every functionality, there will be a job sequence defined (as explained in the example in the Methodology in Mainframe Testing section). The jobs should be submitted using the SUB command with the test data files.
5. Check the intermediate file in order to identify the reasons for missing or errored-out data.
6. Check the final output file, the database and the Spool to validate the test results.
7. If the job fails, the spool will have the reason for the job failure. Address the error and resubmit the job.

Test Reporting — A defect should be logged if the actual result deviates from the expected result.

Steps Involved in Online Testing

1. Select the Online screen in a Test Environment.
2. Test each field for the acceptable data.
3. Test the Test Scenario on the screen.
4. Verify the database for the data updates from the online screen.

Test Reporting — A defect should be logged if the actual result deviates from the expected result.

Steps Involved in Online — Batch Integration Testing

1. Run the job in a Test Environment and validate the data on the online screens.
2. Update the data on the online screens and validate whether the batch job runs correctly with the updated data.

Commands Used in Mainframe Testing

These steps are driven from the terminal, so a small command vocabulary covers most of a tester’s day.

1. SUBMIT — Submit a background job.
2. CANCEL — Cancel a background job.
3. ALLOCATE — Allocate a dataset.
4. COPY — Copy a dataset.
5. RENAME — Rename a dataset.
6. DELETE — Delete a dataset.
7. JOB SCAN — Bind the JCL with the program, libraries, files and other resources without executing it.

There are many other commands used when required, but they are not that frequent.

Prerequisites to Start Mainframe Testing

Basic details needed for Mainframe Testing are:

• Login ID and password for logging into the application.
• Brief knowledge of ISPF commands.
• Names of the files, the file qualifier and their types.

Before starting Mainframe Testing, the below aspects should be verified.

1. Job
   • Do a job scan (command — JOBSCAN) to check for errors before executing it.
   • The CLASS parameter should be pointed to the test class.
   • Direct the job output into spool or a JHS, or as required, by using the MSGCLASS parameter.
   • Reroute the email in the job to spool or to a test mail ID.
   • Comment out the FTP steps for initial testing and then point the job to a test server.
   • In case an IMR (Incident Management Record) is generated in the job, add the comment “TESTING PURPOSE” in the job or param card.
   • All the production libraries in the job should be changed and pointed to test libraries.
   • The job should not be left unattended.
   • To prevent the job from running in an infinite loop in case of any error, the TIME parameter should be added with a specified time.
   • Save the output of the job including the spool. The spool can be saved using XDC.
2. File
   • Create a test file of the needed size only. Use GDGs (Generation Data Groups — files with the same name but with sequential version numbers, such as MYLIB.LIB.TEST.G0001V00 and MYLIB.LIB.TEST.G0002V00) when necessary to store data into consecutive files with the same name.
   • The DISP (Disposition — tells the system whether to keep or delete the dataset after normal or abnormal termination of the step or job) parameter for the files should be coded correctly.
   • Ensure that all the files used for job execution are saved and closed properly to prevent the job from going into HOLD.
   • While testing using GDGs, make sure that the right version is pointed at.
3. Database
   • While executing the job or online program, ensure that unintended data is not inserted, updated or deleted.
   • Also, ensure that the correct Db2 region is used for testing.
4. Test cases
   • Always test for boundary conditions such as an empty file, first record processing and last record processing.
   • Always include both positive and negative test conditions.
   • In case standard procedures are used in the program, such as checkpoint restart, abend modules or control files, include Test Cases to validate whether the modules have been used correctly.
5. Test Data
   • Test data setup should be done before the beginning of the testing.
   • Never modify the data on the test region without notifying others. There may be other teams working with the same data, and their tests would fail.
   • In case the production files are needed during the execution, proper authorization should be obtained before copying or using them.

Best Practices

1. In case of a batch job run, MAX CC 0 is an indicator that the job has run successfully. It does not mean that the functionality is working fine. The job will run successfully even when the output is empty or not as per expectation. So it is always expected to check all the outputs before declaring the job successful.
2. It is always a good practice to do a dry run of the job under test. A dry run is done with empty input files. This process should be followed for the jobs which are impacted by the changes made for the test cycle.
3. Before the test cycle begins, the test job set up should be done well in advance. This helps in finding out any JCL error in advance, hence saving time during execution.
4. While accessing Db2 tables through SPUFI (an option on the emulator to access Db2 tables), always set auto commit to “NO” in order to avoid accidental updates.
5. Test data availability is the primary challenge in batch testing. Required data should be created well in advance of the test cycle and should be checked for completeness. Tracking that preparation in a shared test management repository keeps the regression bed and the data setup aligned.
6. Some online transactions and batch jobs may write data into MQs (Message Queues) for transmitting data to other applications. If the data is not valid, it may disable or stop the MQs, and this will affect the whole testing process. It is good practice to check that MQs are working fine after testing.

Mainframe Testing Challenges and Troubleshooting

Even with those practices in place, a few problems recur on almost every mainframe release. The table below pairs each one with the approach that resolves it.

Challenges	Approach
Incomplete / Unclear Requirements	There may be access to a user manual or training guide, but those are not the same as documented requirements. Testers should be involved in the software testing life cycle from the requirements phase onwards. This helps to verify whether the requirements are testable.
Data Setup / Identification	There may be situations where existing data should be reused as per the requirement. It is sometimes difficult to identify the required data from the existing data. For data setup, homegrown tools can be used as per the need. For fetching existing data, queries should be built in advance. In case of any difficulty, a request can be placed with the data management team for creating or cloning the required data.
Job Setup	Once the jobs are retrieved into PDS, the job needs to be set up in the QA region, so that the jobs are not submitted with a production qualifier or path detail. Job setup tools should be used to overcome human errors made during setup.
Ad-hoc Request	There may be situations when end to end testing needs to be supported due to a problem in upstream or downstream applications. These requests increase the time and effort in the execution cycle. Use of automation scripts, regression scripts and skeleton scripts can help in reducing the time and effort overhead.
On-Time Releases for scope change	There may be a situation where the code impact completely changes the look and feel of the system. This may require a change to test cases, scripts and data. A scope change management process and impact analysis should be in place.

Common Abends Encountered

When a job fails, the spool reports an abend code. The list below covers the codes a mainframe tester meets most often, along with the usual cause.

1. S001 — An I/O error occurred.

   Reason — Reading at the end of the file, file length error, or an attempt to write into a read-only file.

2. S002 — Invalid I/O record.

   Reason — Attempt to write a record longer than the record length.

3. S004 — Error occurred during OPEN.

   Reason — Invalid DCB.

4. S013 — Error opening a dataset.

   Reason — PDS member does not exist, or the record length in the program does not match the actual record length.

5. S0C1 — Operation Exception.

   Reason — Unable to open file, or missing DD card.

6. S0C4 — Protection exception / storage violation.

   Reason — Trying to access storage not available to the program.

7. S0C7 — Program check exception, data.

   Reason — Change in record layout or file layout.

8. Sx22 — Job has been cancelled.

   Reason — The job was ended before completion; the middle digit identifies who or what cancelled it.

9. S222 — Job cancelled by the user without a dump.
10. S322 — Job or step time exceeded the specified limit, or the program is in a loop, or the TIME parameter is insufficient.
11. S522 — TSO session timeout.
12. S806 — Unable to link or load.

    Reason — The job is unable to find the specified load module.

13. S80A — Not enough virtual storage to satisfy GETMAIN or FREEMAIN requests.
14. S913 — Trying to access a dataset which the user is not authorized to use.
15. Sx37 — Unable to allocate enough storage to the dataset.

Error Assist — A very popular tool to get detailed information on various types of abends.

Common Issues Faced During Mainframe Testing

• Job Abends — For successful completion of the job, you should check the data, the input file and whether the modules are present at the specific location. Abends can occur for multiple reasons, the most common being invalid data, an incorrect input field, a date mismatch or environmental issues.
• Output file empty — Though the job might run successfully (MaxCC 0), the output might not be as expected. So before passing any test case, the tester has to make sure that the output is cross verified. Only then should testing proceed further.
• Input file empty — In some applications, files are received from upstream processes. Before using the received file for testing the current application, the data should be cross verified to avoid re-execution and rework.

FAQs