An Open Source Approach for Modernizing Message-Processing and Transactional COBOL Applications by Integration in Java EE Application Servers

Abstract

Modernization of monolithic “legacy” mainframe COBOL applications to enable them for modern service- and cloud-centric environments is one of the ongoing challenges in the context of digital transformation for many organizations. This challenge has been addressed for many years by different approaches. However, the possibility of using a pure Open Source Software (OSS)-based approach to run existing transactional COBOL code as part of Java EE-based web applications has just recently been demonstrated by the author. Therefore, in this paper, an overview of the previously proposed Quick Web-Based Interactive COBOL Service (QWICS) is given and its new extension to run message-processing COBOL applications via JMS is described. QWICS runs on Un*x-like operating systems such as Linux, and therefore on most platforms, but in particular on the mainframe itself. This enables a mainframe-to-mainframe re-hosting, preserving the unique features of the mainframe platform like superior availability and security.

1 Introduction

In the recent and ongoing discussions about digital transformation and its impact for companies, in particular in the banking and financial service industries, it has been recently pointed out by various authors that the mainframe is by no means an outdated technology and probably will remain an important part of the entrprise IT landscape for a long time [16, 28, 42].

However, the traditional monolithic “legacy” COBOL enterprise applications are too inflexible and not open enough to fulfill the requirements of the digital age [1, 38]. And despite its age, COBOL still plays a major role in enterprise application development on the mainframe [1, 20, 40] (with “mainframe” in this paper denoting IBM’s S/390 platform and its descendants) and will continue to do so for a long time due to various reasons [1, 9, 18, 30, 36].

Therefore, the challenge of modernizing the existing COBOL applications is to convert them into service-oriented backends with well-defined APIs and using open technologies [17, 19] while preserving their inherent value, making them accessible for cloud-based “Systems of Engagement” like mobile apps and web frontends [12] or distributed big data processing applications [27, 39].

Migrating to Open Source Software (OSS) has recently been suggested as one approach for the banking industry to reach this goal [11]. Therefore, the recently proposed Quick Web-Based Interactive COBOL Service (QWICS) [6] discussed in this paper follows this direction. It is built on top of well-established, enterprise-ready OSS components such as e.g. the PostgreSQL relational database¹ [15].

Most of these legacy mainframe COBOL programs are online transaction-processing (OLTP) applications, which require a so-called transaction processing monitor (TPM) middleware to run in, like e.g. IBM’s CICS² [25] or Fujitsu’s openUTM³. Therefore, any modernization approach needs to take into account the inherent dependency of the transactional COBOL code on these TPM environments [6].

Most previous approaches [6] thus focus either on adding modern features such as RESTful APIs or Java EE support to the mainframe TPM middleware itselve, thus making use of the unique features of modern mainframes [9, 30, 40], or on providing a complete (mostly proprietary) replacement or emulation for the traditional mainframe TPM on other, non-mainframe platforms⁴ [37, 41].

In contrast, QWICS adopts another perspective on modernizing transactional COBOL applications, as it provides an open framework to run COBOL code in the context of any modern Java Enterprise Edition (EE) application server⁵, using the latter to provide the TPM functionality [6]. QWICS is built using established OSS components and runs on Un*x and Linux derivates, in particular on the mainframe itself. Therefore, it combines the unique features of the modern mainframe with the platform independence and openess of Linux and OSS [6].

To further extend the capabilities of QWICS, in this paper its extension to support transactional message-processing COBOL programs is described. This is an important feature, as asynchronous, message-oriented data processing is an important concept for many enterprise applications [4]. Consistent with the original idea of QWICS, this is achieved by using the Java Message Service (JMS)⁶ functionality of the Java EE application server.

The rest of this paper is organized as follows: In Sect. 2 the related work is analyzed in detail, while the software architecture of QWICS proposed in [6] is summarized in Sect. 3. Section 4 describes its new extension to support message-oriented transaction processing in COBOL using JMS. The experimental evaluation of QWICS and its results are illustrated in Sect. 5. We conclude with a summary of our findings.

2 Related Work

As described in [6], over the years the challenge of modernizing transactional “legacy” COBOL applications by modularization and encapsulation [31] has been addressed by numerous approaches, which my be classified into three major categories:

• Modernization on the mainframe itself [32]: Making use of new technologies such as web services and Java EE supported by the current versions of the “classical” mainframe TPM products to wrap COBOL transaction programs by web service facades or web user interfaces [19, 22, 35] and integrate them into service-oriented architectures [7, 10, 26, 29]. This is well supported by various software tools from the mainframe vendors [3] as well as by third parties, and allows to make use of the unique features of modern mainframes such as extremely high availability and outstanding transaction throughput [9, 40].

• Migration to non-mainframe platforms (including cloud services) [17, 21]: Usually driven by the expectation to reduce the perceived high operating costs of the mainframe platform and the related vendor lock-in [17, 21], this typically requires to use either the original mainframe TPM middleware [25] on these non-mainframe platforms (which is the case for the major mainframe TPM products) or an emulation mimicking the functionality of the TPM [2, 37]. In particular, the latter has been addressed over the years by various hobbyist approaches⁷ as well as professional commercial offerings (See footnote 4). However, these approaches have different drawbacks, since they frequently do not achieve the same transactional throughput as the original mainframes, rely on on proprietary emulation techniques (thereby creating a new vendor lock-in) or may suffer from patent-related and licensing issues [2, 23, 37, 41].

• Conversion of the program code to other languages and platforms: This involves either the (automatic) conversion of the COBOL code to other, more modern programming languages and/or the extraction of the business rules and logic from the existing code (e.g. by using special analysis tools) and their subsequent re-implementation (either manually or by code generators) using other languages and platforms [5, 8, 13, 21, 24, 34,35,36, 43]. Being closely related to model-driven development, this approach has gained wide interest in the scientific community on legacy systems modernization, but has been rarely used in practice since it may be too expensive or riskful for companies in many cases [36].

Despite numerous attempts for re-writing [14] or re-hosting mainframe “legacy” applications on other platforms, mainframe-based organizations after various failed migration projects [1, 9, 40] realized that the mainframe platform offers unique features like the support for high availability, vertical scalability and security [40], which could not always be recovered on other platforms [28, 42]. Therefore, to enable the digital transformation [38], in the last years the focus shifted again to the first approach, namely the modernization on the mainframe itself [42].

Since Java has overtaken the role of COBOL in enterprise application development to a large extend, Java Enterprise Edition (EE)⁸ application servers for the Enterprise Java Bean (EJB) components provide functionalities similar to those classical TPM middleware does for COBOL [3, 21]. These includes support for distributed transactions and the 2-phase-commit (2PC) protocol through the Java Transaction API (JTA)⁹ as well as transactional message processing via Message-driven Beans (MDB) and JMS.

Due to this analogy between Java EE application servers and classical TPM middleware, the previously proposed QWICS framework [6] demonstrated an approach to execute transactional COBOL programs integrated in a Java EE application server as part of a JTA transaction, such that the required TPM functions (such as transaction handling, resource access, access to message queues, user interfaces, etc.) are realized by the Java EE application server out of the box.

QWICS has been implemented as pure OSS, built using mature, enterprise-ready OSS components, since openess and OSS have been identified recently as cornerstones for enterprise application modernization in the age of digital transformation [11, 15]. Since QWICS adds only a thin “glue component layer” to the established OSS components to manage the native COBOL execution for the Java EE application server [6], it is fully portable among various Un*x- and Linux derivatives, with a strong focus on “re-hosting” COBOL applications to Linux on the mainframe itself using OSS¹⁰ [6].

While the feasibility of this approach has been demonstrated already in [6], QWICS so far lacked the support for transactional message-processing. Since this is an important feature [4], in this paper together with a summary of its previously described architecture and functionality the question is addressed, how an extension of QWICS to support message processing in COBOL via JMS could be designed and implemented.

3 Design of the Software Architecture

Figure 1 shows an overview of the software architecture of QWICS, which has been presented in [6] and will be summarized in this section. Its full source code is available as OSS on GitHub¹¹.

As described in [6], QWICS is based on well-established, mature and enterprise-ready OSS components selected following a “best of breed” strategy: GnuCOBOL¹² for compiling the COBOL sources on the respective target platform, the PostgreSQL relational database system (See footnote 1) replacing the original mainframe DBMS and the JBoss WildFly application server¹³ as the Java EE runtime. Here, in particular PostgreSQL has a wide recognition by practitioners as a reliable, scalable enterprise-quality OSS database management system [15].

Fig. 1.

Overview of the software architecture of QWICS. The arrows denote usage/invocation relationships. Yellow boxes describe the QWICS-specific components and white boxes the application-specific COBOL or Java code. The integration between the Java and the COBOL code is achieved using a specific JDBC driver calling the COBOL server via its own protocol over a TCP connection. Reprinted from [6]. (Color figure online)

Fig. 2.

Necessary modification to termio.c of GnuCOBOL’s libcob runtime library. Only the lines shown between // BEGIN... and // END ... need to be added, no further modifications are necessary. This code adds an interception to DISPLAY statements of the form DISPLAY" TPMI:..., which are used to execute the EXEC-macros in the original COBOL source. Reprinted from [6].

Fig. 3.

Process of preprocessing the original COBOL source files and map definitions for use by QWICS. Reprinted from [6].

To execute the transactional COBOL programs in the context of the Java EE container, as the core concept the integration by a special Java Database Connectivity(JDBC)¹⁴-compliant driver is used. This QWICS JDBC driver provides Non-XA and XA datasources to handle distributed transactions. It acts as a TCP client for the COBOL Transaction Server, which actually loads and executes the COBOL binary programs, the so-called load modules. Thus, the execution of these COBOL programs is invoked and controlled from the Java code (e.g. from an EJB) by special callable statements and result sets implemented by the QWICS JDBC driver. Thus, in QWICS always a Java EE application (e.g. consisting of EJB, servlets, JPA, Web Services, ...) is needed to call the COBOL programs via the JDBC driver [6].

The COBOL Transaction Server itself is a separate program implemented in C, which dynamically loads and executes the COBOL load modules created by the GnuCOBOL compiler. It also serves as a client for the PostgreSQL database to execute SQL statements embedded in the COBOL programs as well as those directly send from the Java side via the JDBC driver, being able to mix both in one transaction. The transaction server uses a slightly modified version of GnuCOBOL’s libcob library, which has an intercept added to handle special DISPLAY statements of the form DISPLAY "TPMI:... and call a function in the transaction server for these. The necessary modification is shown in Fig. 2. This mechanism allows to handle the original EXEC ... END-EXEC macros in the code [25], which therefore have to be converted to these special DISPLAY statements in a subsequent preprocessing step [6].

To keep track of the state of all running transactions, a TPM needs to handle all input/output (I/O) operations performed by the programs it executes. Therefore, transactional COBOL programs running inside a TPM may not use the normal COBOL I/O statements. Instead, all necessary I/O operations (like e.g. executing SQL statements, sending or receiving data from the screen, etc.) are usually coded in COBOL (or that of any other supported language) by using TPM- and SQL-specific EXEC ... END-EXEC macros. These macros are then translated into TPM API calls in an additional preprocessing step before actually compiling the COBOL code [25]. In addition, the terminal UI screen definitions (so-called maps) referenced by these macros also need to be translated to stored COBOL code fragments (called copybooks) containing the necessary variable declarations. These copybooks are then inserted in the COBOL code during preprocessing as well [6].

This preprocessing needs to be mimicked by QWICS to allow to re-use the unmodified COBOL source codes. In Fig. 3, the overall process implemented in QWICS by two preprocessors written in C, cobprep for COBOL and mapprep for the map definitions, is illustrated. After the preprocessing, the resulting COBOL code is compiled to an executable load module using the GnuCOBOL compiler [6].

Last but not least a JavaScript Library has been implemented for QWICS, which supports the (optional) implementation of web user interfaces by converting the original TPM’s map defintions [25] to JavaScript Object Notation (JSON) structures beforehand using the preprocessor [6].

Fig. 4.

Necessary modification to call.c of GnuCOBOL’s libcob runtime library. Only the lines shown between // BEGIN... and // END ... need to be added, no further modifications are necessary. This code adds an interception to COBOL CALL statements, which executes the function denoted by the pointer resolveCALL(..) if it is set.

4 Extension to Transactional Message-Processing

Asynchronous, transactional message-processing is an important concept of large-scale enterprise applications [4]. In traditional mainframe applications this functionality is typically provided by a message-oriented middleware invoked from COBOL programs running in a TPM. A similar concept exists for Java EE application servers through the Java Message Service (JMS).

Following the general approach of QWICS, transactional message-processing was therefore implemented by using JMS from the COBOL code by extending the QWICS JDBC driver. To do so, the COBOL program calls for getting and putting messages from and into queues or topics [4] need to be intercepted by the QWICS transaction server. Therefore, again a patch has been added to the GnuCOBOL libcob library to intercept and redirect the respective CALL statements to corresponding C functions provided by the transaction server. Figure 4 shows the respective modification to the libcob library.

Fig. 5.

Conceptual overview of the integration between COBOL and JMS via the QWICS framework. The actual access to JMS is implemented within an EJB application and not inside the QWICS JDBC driver to keep the different Java EE subsystems independent from each other. A message-driven bean listens at a queue or topic and upon reception of a message triggers the corresponding COBOL program using the JDBC driver. The COBOL program accesses the JMS queues via the JDBC driver and the EJB application.

The transaction server communicates with the QWICS JDBC driver to send and receive messages via JMS. Figure 5 shows a conceptual overview of the interplay between the components involved in this process. Since JDBC datasource management, JTA and JMS are independent subsystems of Java EE, a JDBC driver should not depend on any JMS API or call it directly. Therefore, the integration of COBOL with JMS requires a custom Java EE application using message-driven beans (MDB).

While the JDBC driver only offers interfaces representing abstract wrappers for queues or topics (QueueWrapper) and a factory for creating them (QueueManager), the Java EE application implements these interfaces to actually access the JMS functionality. It registers a QueueManager implementation with the JDBC driver, so the latter could create instances of these classes. The UML class diagram in Fig. 6 illustrates the relevant Java classes of the JDBC driver and the example Java EE application and there relations.

Fig. 6.

UML class diagram describing the Java-side implementation of an example Message-Driven Bean (QwicsMDB) using the newly added message-queueing functionality of the QWICS JDBC driver. The MDB class listens for and reacts to received messages from a JMS queue or topic and provides the driver with appropriate QueueWrapper implementation classes (labeled QwicsQueue and QwicsTopic here) for accessing the respective JMS objects.

Fig. 7.

Excerpt of the example Java EE message-driven bean (MDB) listening for messages on a JMS queue or topic. Upon reception of a respective message, the method onMessage(Message message) is invoked by the container, wrapped into a distributed XA transaction. The message is only removed from the queue of this transaction is commited successfully. The method invokes a sample message-processing COBOL program via the QWICS JDBC driver.

For every JMS queue or topic for which messages should be processed within a transaction by a COBOL program, a corresponding MDB needs to be implemented, for which the example QwicsMDB may serve as a blueprint. When such a MDB receives a message, its method onMessage(Message message) is invoked by the EJB container, wrapped into a distributed JTA transaction. This method now uses the QWICS JDBC driver via a XA datasource to trigger the respective COBOL program.

Figure 7 shows an excerpt from the source code of the example QwicsMDB to illustrate this. The COBOL program invoked here is a demo program called QPUBCBL. By the statement maps.updateObject(“QMGR”, this);, the MDB registers itself as the QueueManager implementation with the JDBC driver.

The invoked COBOL load module now may access the JMS queues itself by executing CALL statements to the (emulated) routines “MQOPEN”, “MQCLOSE”, “MQGET” and “MQPUT”. Intercepted by the transaction server as described above, these calls are forwarded to the JDBC driver, which again uses the implementation classes QwicsTopic or QwicsQueue provided by the Java EE application to access the respective JMS artifacts.

With this mechanism, analogous to the previously described QWICS functionality, COBOL programs will be triggered by JMS messages are able to send and receive messages via JMS.

5 Experimental Evaluation

To evaluate the functionality and usefulness of QWICS, in a first step an existing transactional COBOL application was ported to the QWICS evironment [6]. To avoid a bias, a representative COBOL application written for training purposes using the IBM CICS TPM a consulting company was used for this [33], since its developers are not related or known to the author [6].

The original sources were passed through the preprocessors as described above, compiled using GnuCOBOL and then run in the QWICS environment. The screenshot shown in Fig. 8 illustrates how this COBOL application running in QWICS may appear to user in a web browser window. This original evaluation was repeated two times, first on an Apple MacBook Air laptop running MacOS X 10.11.6 (thus, a BSD Un*x- derivative), and second on a IBM zBC12 mainframe computer running Linux on Z. Both tests worked smoothly, delivered identical results regarding the functionality and thus demonstrate the feasibilty of the approach in principle [6].

Subsequent to the original publication of these evaluation results [6], in a second step a public community website (https://qwics.org) including a free demonstration and testing environment for QWICS has been set up, to further evaluate the QWICS framework and move it towards production readiness.

This website provides further information on QWICS, links to the source code repository and after a free online registration offers everyone the possibilty to run a personal QWICS environment in a Virtual Machine (VM) based on a Docker container¹⁵. Besides the actual QWICS framework, this environment offers the user a web-based administration console for the PostgreSQL database (using the OSS phpPgAdmin¹⁶) and a web-based source code editor to write COBOL code and edit UI screens online (using the Codiad IDE¹⁷). Figure 9 shows screenshots of the dashboard and the online code editor of this environment. As can also be seen from the screenshot, the QWICS online environment currently still runs on an Intel x86-based server running Ubuntu Linux. It is planned also to provide a demo running on the mainframe under Linux on Z in the future.

The QWICS server VM offered on https://qwics.org to the public already contains a simple but ready-to-run transactional COBOL example written by the author for demonstration purposes, in the form of a small guestbook web app. This online version of QWICS not only allows to demonstrate and explore the framework, but also may serve as an easy to use and free opportunity to learn and practice transactional COBOL programming online.

Fig. 8.

Screenshot of a map screen converted to JSON and displayed in web page by the JavaScript library. Reprinted from [6].

Fig. 9.

Screenshots from the free QWICS online demonstration and testing environment: Admin Dashboard showing a running QWICS VM instance (above) and the online source code editor showing a COBOL example (below).

The availability of this QWICS online trial has been promoted via a press release and internationally via social media so far. Until now, a small number of interested persons from different countries have registered, but it is yet too early to obtain results from the analysis of user feedback.

It has been pointed out before that QWICS is different from other mainframe modernization approaches with respect to its focus on using OSS to integrate existing COBOL code into Java EE applications to use modern web technologies with COBOL [6]. Therefore, it requires a partial adaption and recompilation of the existing sources instead of achieving full binary¹⁸ or source-level compatibilty [2, 37] on other (commodity), non-mainframe platforms [6]. Instead, it enables customers to modernize transactional COBOL applications on the mainframe itself using Linux.

The next research steps will focus on evaluating the potentials of QWICS in real-world case studies, on the one hand by an extended Proof-of-Concept using a real “legacy” COBOL application in a company, on the other hand by further exploring the use of the QWICS online platform for COBOL programming education. The first will also need include an extended analysis of the performance and scalability of the approach compared to the original mainframe TPM. As described in [6], there definitely will be a performance tradeoff due to the design of the used OSS components, but it remains an open issue how big it will be in practice.

6 Conclusion

In conclusion, in this paper the previously proposed Quick Web-Based Interactive COBOL Service (QWICS) [6] was presented and its extension to support asynchronous, transactional, message-oriented communication in COBOL via the Java Message Service (JMS) API was introduced. The latter is in particular important for building large-scale enterprise applications.

Therefore, QWICS now allows to run the most relevant types of transactional COBOL applications (synchronous and asynchronous) within the context of a Java EE application server using a pure OSS stack. Therefore, it allows to mix COBOL and Java code for extending and converting “legacy” applications into web services. Its pure OSS stack runs on most Un*x and Linux platforms, allowing in particular to modernize “legacy” applications on the mainframe itself using Linux.

The feasibility of the approach has already been evaluated by porting a semi-realistic third-party COBOL application to QWICS [6], as well as by making it available for public use via an online demonstration and testing environment at https://qwics.org. Also, meanwhile the community feedback received via the GitHub source code repository of QWICS (See footnote 12) has been taken into account as far as possible.

However, since QWICS is still a proof-of-concept implementation and thus not feature-complete, or its production readiness further extensions and adaptations may be necessary, which should be detected and addressed during migration of a real-world application. Therefore, further research is needed to explore the approach in a real industry case study.