Keywords

1 Introduction

Information and Communication Technologies (ICT) penetrated steadily into all spheres of life in past decades. In social communication, the given process has occurred quite spontaneously [1]. However, in the field of communication between a government, citizens and business the management component of this ensemble plays a huge role [2].

In scientific literature at first much attention has been paid to e-services delivery [3, 4] and the development of “good governance” [5] with ICT. Gradually, scientists came to the conclusion about the impact of digital transformation influence on public values formation of various types [6,7,8]. Electronic interaction includes not only the interface interaction with users of public services portals but also rather complex connections of the participating information systems. Moreover, researchers already proposed a list of public values components counting: efficiency, effectiveness, intrinsic enhancements, transparency, participation, collaboration [8]. Viale Pereira [9] revealed value generating the mechanisms pointing out a significant potential of open data initiatives in creating values.

Undoubtfully, the created values themselves are significant indicators of the quality of information systems [10], but their effectiveness, often understood as the ability to realize demanded public values by optimally using resources and balancing the associated risks [11] is also crucial. Particularly, precise measurement and management of the information systems effectiveness becomes important in the face of limited resources and the need to early obtain the necessary benefits.

At the same time, considering examples of countries with a highly centralised management system (like Russia), it is necessary to consider the existing peculiarities of government information systems management.

This paper describes an attempt to assess the effectiveness of e-participation portals in Russia from the perspectives of such stakeholders’ as Federal Government, Regional Authorities and Citizens.

The paper has the following structure: Literature review provides a review of appropriate government information systems effectiveness assessment approaches and indicators. Methodology section demonstrates the author’s framework for information system evaluation. Section “Findings” illustrates the results of Russian projects in the field of e-participation based on the methodology proposed. Section “Conclusion and discussion” concentrates on the key research results and the future steps.

2 Literature Review

We started our research from the analysis of government information systems effectiveness assessment approaches in the studied area.

In one of the early works devoted to the goal we studied it was said that Effectiveness is determined by comparing performance to a goal and the way to assess system effectiveness is first to determine the task objectives of the system, or of the organizational units utilizing the system, and then to develop criterion measures to assess how well the objectives are being achieved [12]. This point of view remains relevant up to our time. The following main problems mentioned in Scott’s work remain important so far: (1) Objectives and measures of accomplishments are often not defined adequately at the outset of an IS implementation effort [13]; (2) Efficiency-oriented and easily quantified objectives and measures are typically employed while effectiveness-oriented and qualitative objectives and measures are ignored [14]; (3) Objectives and measures used to evaluate the system are not the same as those defined initially [15]. In more recent works, discussion of these problems is rare, although our experience shows that they still exist and have a significant impact on the achievement of the necessary effectiveness of the information systems being created and on the ability to evaluate effectiveness by comparing goals and outcomes.

Bozeman and Moore are the founders of methods for assessing the effectiveness of public services (and the information systems providing them) using the concept of the public value created by them. It was they who proposed to measure “context-specific preferences of individuals concerning, on the one hand, the rights, obligations, and benefits to which citizens are entitled, on the other hand, obligations expected of citizens and their designated representatives” [16, 17]. Research into the further development and application of this approach has shown its usefulness for improving the quality of government decisions in the field of application of information technologies, including the area of improving communication between the government and citizens [18,19,20]. Indeed, a number of studies has proven that the focus on measurement and achievement of public values leads to an increase in the effectiveness of government agencies and their information systems: Effectiveness of public organisations itself creates public values [16]. Citizens expect efficiency, openness, and responsiveness from public organisations [21, 22]. E-government can be used for improving the efficiency of public organisations by cutting processing costs, and making strategic connections between and among government agencies [23] through developing better ICT infrastructures, re-designing public functions [24], sharing public information and empowering public staff [25]. Since public organisations run on taxpayers’ money, citizens value the improved efficiency of public organisations through e-government [26].

The studies surveyed revealed that more than 100 indicators of public value and effectiveness were proposed and used, among them: ensuring environmental sustainability, transforming citizens’ interests into political decisions, openness, internal efficiency, etc. Unlike the traditional model of public administration, the concept of social value emphasises that the list of social values is formulated not by those who provide public services, but by the citizens who can express their preferences through various instruments of interaction [27]. It also argues that public perceptions of values such as trust and democracy should play a leading role and be complemented by other values such as cost-efficiency and effectiveness [28].

Judging by the large number of publications on research on the effectiveness of state information systems, one of the most popular is the PVIT (Public Value of Information Technology) technique [29]. Perhaps, one of the most extensive and carefully implemented studies on the effectiveness of e-government using this method was conducted in 2012 by Professor K. Karunasena in Sri Lanka [30]. Many subsequent studies, for example, in 2016 in South Africa [31], in 2017 in Turkey [32] and in Taiwan [33] were based on the modification of Karunasena’s methodology.

An early analysis of the declared objectives of IT projects and their relationship to content and target indicators is critical, since the expected benefits of implementing state-owned IT projects can be realized only if they are clearly identified already in the design process and are reflected in the key documentation of the project being launched [34]. In response to this need for government authorities, the Department of Public Administration of the University of Albany (USA) in 2012 developed and proposed for free use a methodology and tool for the evaluation of the PVAT (Public Value Assessment Tool) project portfolio [8], which is further logical development of the PVIT methodology. The PVAT methodology was applied to the formation of IT project portfolios in several US government organizations and showed good results that are of practical importance for increasing the effectiveness of their outcomes [35].

Returning to the definition of the effectiveness of information systems given in the CobiT 5 standard [11] and temporarily refusing the impact of risks, we can calculate the overall effectiveness of an information system using indicators characterising the goals set, the goals achieved, and the costs incurred. However, considering the need to take these indicators into account in the set of organisational subsystems and in a variety of dimensions, such simple calculations become complicated. A practical model for solving the task of calculating the complex efficiency was justified and proposed in the form of the Global Organizational Effectiveness Index Subsystem (GOEIS) meta-model [36].

3 Research Methodology

This research aims to investigate the effectiveness of government information systems, understood as their ability to realise demanded public values by using resources optimally. To adequately accomplish this aim, a theoretical framework is required for providing the foundation for the implementation of both the quantitative and qualitative studies.

The theoretical framework is developed based on such theoretical concepts as the theory of public value, the sources of public value creation, inventories of public value and information systems effectiveness measurement described in Literature review section. In summary, the effectiveness of public value creation greatly depends on public bodies and their information systems organisations, various stakeholders, and their interactions [20, 22].

Based on these theoretical perspectives and the indicators derived from various e-government performance evaluation methodologies discussed in Literature review, a theoretical framework for evaluating the effectiveness of government information systems in Russia is hypothesised and shown in Fig. 1.

Fig. 1.
figure 1

The theoretical framework for evaluating the effectiveness of government information systems

Full size image

The relationships stemming from the stakeholder’s conscious demands to the effectiveness of government information systems, that are created to meet these demands, became the basis for the formulation of a series of reflective research hypotheses as shown in Fig. 1 and summarised in Table 1.

Table 1. An overview of the hypotheses
Full size table

This research is going to be both confirmatory and exploratory. The confirmatory nature of the research is reflected by its objective to test a hypothesised theoretical framework for evaluating the public value of government information systems in Russia. The exploratory nature of the research is characterised in its pursuit of investigating the sources of effectiveness, how government information systems create public value for stakeholders in Russia, and how the existing practices in implementing IT initiatives can be improved for delivering better public value to its stakeholders [37].

A mixture of qualitative and quantitative approaches was chosen to implement the research, since it allows the researcher to test a theory by understanding the various factors in the theory and establishing relationships between the factors, and also to explore the reasons behind the relationships [38].

To test and validate the theoretical framework, the data were collected from open sources (regional state information systems development plans, Public Procurement Portal, descriptions of the public IS introduction results submitted to the all-Russia competition Prof-IT, state statistics portal, etc.), some of which provide data in digitalised forms, but some in the form of analytical and descriptive texts that are to be pre-processed for further analysis.

To calculate the effectiveness of the studied government information systems, taking into account their number of dimensions and sub-systems, the GOEIS meta-model has been chosen. This meta-model is not limited and gives the opportunity to add or remove elements or vectors, depending on the system characteristics to be evaluated. Table 2 shows the meta-model and its components.

Table 2. GOEIS meta-model (Páscoa 2012).
Full size table

To narrow the scope of research, government information systems supporting e-participation in Russia were selected as the object for pilot research. Taking into account the specifics of a rigid vertical state management system in Russia, three major stakeholder groups were selected for the study: the parent authorities that dictate the development goals of state information systems development are mandatory for execution at local level; local authorities responsible for identifying and satisfying the citizen’s needs; citizens - users of state information e-participation systems.

Following the recommendations of GOEIS methodology selected, for initial analysis and verifying its applicability, the generalised specific needs of the three stakeholder groups (Parent Authorities, Local Authorities and Citizens) were used as vectors, and five primary functions of e-participation government information systems (Stakeholders’ demands formulation; Identification and prioritisation of demands and formulation of goals; Formation of programs and projects to achieve goals; e-participation IS operation and realisation of public values) were taken as analysable elements. This narrowing of the research scope is due to the limited availability of open information needed for analysis, sources of which are: analysed e-participation portals, portals of authorities responsible for their functioning, detailed information on the processes of these portals creating, presented at the annual All-Russian contest of the best government IT-projects. Several independent e-participation portals were included in the study for comparison. For vectors, it was assumed that their effecting values are measured for each of the elements by the degree of their conformity to the corresponding hypothesis, and their planned values are assumed to be 100% consistent with the hypothesis. The numerical values of the effecting vectors (where 0% is the absolute discrepancy and 100% - full compliance between the appraised portal and the hypothesis formulation) were obtained by averaging the expert assessments of 8 experienced analysts - employees of the e-Governance Center of the ITMO University, the Committee for Informatization of St. Petersburg and the Information and Analytical Center of the Administration of St. Petersburg. Additional evaluation of the validity of expert assessments are not performed at this stage.

Hypotheses H6-H8 were not included in the model at this stage and their analysis was done separately.

4 Results

The results of expert decisions on the numerical values of vectors for the analysed elements are given in Table 3.

Table 3. Initial data obtained as a result of e-participation sites expert evaluation
Full size table

For each of the analysed portals, calculations were made of the effectiveness, both global and in sections of hypotheses and individual vectors using the GOEIS technique. The results of calculations for all portals are summarised in Table 4.

Table 4. Consolidated settlement results for all analysed e-participation portals
Full size table

Based on the results of the surveyed e-participation portals effectiveness calculations, supporting organisational structures descriptions and related information systems, it became possible to draw preliminary conclusions on the hypotheses formulated at the beginning of the research. Since the obtained results indicate an unsatisfactory situation, the conclusions given in this paper are concentrated on the problems found. These conclusions are summarised in Table 5.

Table 5. Main conclusions on hypotheses
Full size table

Some differences between government and independent e-participation portals are of interest. Government portals are more successful in identifying the needs of stakeholders and in formulating the goals of development programs than independent portals which are more successful in these goals implementation. This finding is illustrated by the data presented in Fig. 2.

Fig. 2.
figure 2

The degree of the hypotheses fairness for state and independent portals

Full size image

However, it is possible to observe significant differences between state and independent portals in their striving to satisfy various stakeholder groups. This is evident from Fig. 3.

Fig. 3.
figure 3

Different focus of efforts to meet the needs of stakeholders

Full size image

In general, the results of this study showed the suitability of the method chosen to assess the overall effectiveness of public information systems and to assess the contribution of some organisational factors to overall effectiveness, which may be useful for more effective management of these systems.

5 Discussion

The study contributes to understanding on management factors in IT projects development and implementation. The framework proposed could be of interest for GCIOs and IT-managers facing some issues of IT effectiveness and its measurement indicators. The model proposed involves the assessment of external environment, upper-level management system, goal setting, planning and organisation of government information systems operation, operational management, outcome parameters, and collection of data on the status and results, as well as integrated effectiveness.

The author’s approach to combine a management cycle and public values types (social, political, economic) was first applied on e-participation cases since these portals provide new ways to solve citizens’ needs and demands not just in an operational way (as e-services) but in some part of qualitative decision-making.

The limitations of the study are connected with the limited number of assessed projects as well as the orientation on very specific Russian projects context. The study counted data from the contest, at the same time some interviews with portals developers could shed light on internal management operations.

The following studies will be focused on further improving the described model of effectiveness evaluation. Among our immediate tasks we can mention the following: searching for and testing source data more objective than those used in this study, application this model to other IT project types, the comparison of calculated effectiveness levels of different managerial systems.