Keywords

1 Introduction and Objectives

In management literature, performance evaluation systems have been proposed as fundamental tools to improve rationality in the decision-making process as well as organizational mechanisms to align an individual behaviour to a firm objective and, consequently, to improve strategic and operational management [1].

Performance management has also been gaining momentum in the public sector. This follows new managerial paths of reform, mainly in the Western countries, which have broadened the responsibilities of public managers and have emphasized the concepts of efficiency, effectiveness and long-term economic performance [2].

In Italy, in the early 1990s, while the public sector in general was undergoing a profound reform, which was inspired by New Public Management (NPM), public universities delayed the introduction of managerial practices and tools [3]. This was most likely due to the large organization, management and accounting changes that were introduced during the 1980s (DPR 382/1980; DPR 371/1982). Such changes conditioned the acceptance of a new process of change. Consequently, the innovations that were inspired by NPM were barely implemented by Italian public universities. Furthermore, the principle of autonomy, which was established by Law no. 537/1993, was not accompanied by an adequate assumption of responsibility [4]. The public finance problems of Italy, as well as the general financial crisis, determined budget-cutting policies and a season of deep changes in the governance, organization, management and accounting of universities, started with Law no. 240/2010 [5]. Performance-based principles and reward systems in resource allocation, competitive mechanisms among public universities and a new accounting model were introduced [6].

In order to survive and succeed, Italian public universities must develop advanced information systems at all levels (central administration, departments and degree courses). This will improve decision-making qualities and, consequently, performances that are related to their activities (research, didactics, services for students and transfer of technology to the local territory). In particular, didactic performance plays a central role due to the effects it has had and will have in the future on the public financing system of universities (Decree no. 893/2014).

The aim of this paper is to design a performance management system to improve student success that is the degree of attainment within the regular duration of the degree course. It also aims to highlight the conditions and peculiarities that an IT system should have in order to effectively serve its purpose. This paper is theoretical but with an explorative nature, as the proposed system is built on a literature review and on what the Italian law requires, and is currently being tested in a bachelor’s degree course of an Italian university. The level of analysis is the degree course of a department of an Italian public university, which is responsible for the organization, planning and results of didactics. The complex concept of student success [7] is investigated at three specific moments in time: the precollege and entrance phase, duration of degree course and final phase and postcollege outcomes. The rationale behind the model is that student success must be planned in advance and operationalized into concrete actions and indicators. It must also be evaluated along a student’s career through a continuous monitoring system and must rely on a sound and complete IT system.

This paper is organized as follows. Section 2 presents a literature review on performance management in the public sector and in public universities. Meanwhile, Sect. 3 outlines the Italian situation, with a particular focus on the evolution of legislation on university and didactic evaluations. In Sect. 4, a performance management system that improves didactic outcomes is proposed, as well as an assessment of the characteristics that an IT system should have in order to make it properly work. Finally, Sect. 5 is devoted to discussion and conclusion.

2 Performance Management Systems in the Public Sector and in Public Universities

Over the last 25 years, performance measurement systems have been one of the main tools that have been introduced by the reforms of Western countries, following the principles of New Public Management (NPM) [8]. Since the 1980s, the attention that has been paid to the development of result-based management mechanisms has increased. This has led some scholars to label the current era as that of the “audit society” [9].

However, the accountability purpose of a performance measurement system, which is to inform citizens and funders on resource use, has often prevailed over its primary purpose of giving public managers and policy-makers timely and useful information to improve service outcome [10]. Furthermore, in the public sector, performance management, which is the concrete use of performance information in decision-making, has been barely achieved. This is due to some critical factors, which have been highlighted in literature. The first problem is that the adoption of a performance measurement system has usually been seen as a fulfilment of the law. This issue is related to the poor managerial culture, which has traditionally affected the public sector. This can be seen, for instance, by the common underestimation of developing accompanying mechanisms to enhance performance management. In fact, while major efforts have been made to collect and report data by public administrations, much more commitment is required to link long-term strategic objectives to short-term activities. Although environmental variables unavoidably have an impact on public sector performance [11], literature agrees on the fact that supportive leadership, completeness of a management cycle and organizational culture are key elements to promote performance information use [12].

Furthermore, some authors argue that the public sector’s uniqueness sometimes imposes upon the success of performance management. The concept of performance itself, with reference to public services, seems to be rather controversial. On the contrary, the approach that is used to develop performance management systems in the public context has often assumed that outputs can be easily measured and counted [13] through a narrow range of indicators. This has given rise to unintended consequences [14] and, in the worst cases, to what has been defined as a “performance paradox” [15].

Additionally, since the early 1990s, public universities have been subjected to an increasing emphasis on management by objectives that are developed following a sort of goal-directed and institutional approach. They strictly adhere to an instrumental and technically rational paradigm, which leads to a lack of coupling between goals and performance indicators. Modell contrasts such an approach in his study on the development of performance measurement by the government in order to control the university sector in Sweden [16]. He highlights the need for performance measurement systems that are more tailored to objectives, targets and standards, as well as being capable of providing information to a broad range of constituencies.

Following a macro perspective, scholarly attention has mainly focused on the implementation of performance evaluations in the public sector. However, the subject of their outcome has almost never been evaluated [17]. In particular, at different stages of the reforms of public universities, several concerns regarding the potentially negative effects of performance measurement for the quality of teaching and student success have risen on an international level. In addition, the student perspective has often been disregarded by academic research on performance measurement. Actually, among the multiple perspectives under which didactic performance can be evaluated, the degree attainment within the regular duration of a university course by a student is obviously the main outcome of the didactic activity. This perspective has never been more important, considering the complex social, political and cultural issues of modern society and the current financial crisis. Furthermore, the decision level of the units that are directly involved in the organization, planning and monitoring of student careers, such as university departments and degree courses, has also been disregarded.

The construct of student success is very complex. This is because, over time, multiple definitions have been proposed, for example, considering degree attainment as the definitive measure of success [18]. Traditional measures of student success are included in the category of academic achievement, such as scores on entry exams, college grades and credits that are earned in academic years or terms. These represent progress towards a degree. Otherwise, the category of post-graduation achievement comprises graduate school admission test scores, professional school enrolment and postcollege employment rates and income. To evaluate success, student satisfaction with his/her learning experience must also be taken into account [19], as well as the plurality of outcomes that are related to the benefits for individuals and society [20].

In the array of measures of student success that are explored in literature, there is wide agreement on the multidimensional nature of the concept, as well as on the different meaning it has in relation to at least three specific moments in time. These moments are during the precollege and entrance phase, along the degree course duration and in the final and postcollege phase. Finally, some external variables that are conducive to good student outcomes, such as parental encouragement, support of friends, finances, economic trends and workforce development needs, are typically beyond the direct control of organizations [21]. Consequently, student engagement, which has a considerable impact on didactic performance as it is conducive to student success, can be greatly influenced by universities and their didactic structures [22]. In fact, decision-makers at department and degree course level can reasonably affect the behaviour of students and create fruitful institutional conditions. Student behaviours include the time and effort that students put into their studies and the interaction with faculty and peers. Meanwhile, institutional conditions include didactic resources, educational polices, programmes and structural features [7].

3 Introducing Performance Budgeting in Italian Public Universities: Lights and Shadows

Recent reforms of the Italian university sector have aimed at improving performance, which is evaluated from the double perspective of educational quality improvement and efficiency in service delivery. This is achieved through the gradual introduction of a quasi-market framework [23] and giving increased autonomy to universities [24].

In particular, since the 1990s, several acts have changed the resource allocation model from the state to universities in order to abandon an incremental financing system that is based on historical expenditure and supply.

Nevertheless, in Italy, the evolution towards performance budgeting financing models followed a rough path. In fact, the frequency and speed of the changes, which affected the ministerial parameters that were used to allocate the share of rewards of the state funding (Fondo di funzionamento ordinario or FFO) in the last 15 years, did not let universities align their behaviours to the incentives that were set by the financing system. In other words, the retroactive effects of the models led to a sort of “schizophrenia” in university and department decision-making. Initially, Law no. 537/1993 considerably increased the degree of financial autonomy of universities, giving them the possibility of managing resources from the state without a purpose bond. This promoted the transition to a lump sum budget model [25]. Furthermore, in order to remedy the situation of lack of balance, which was provoked by the use of historical expenditure as the main resource allocation principle, FFO was shared in a basic share (linked to historical expenditure) and a (even smaller) share to restore equilibrium (8 % of FFO in 1999).

In the second phase, which started with the Ministerial Decree no. 146/2004, a performance budgeting model was introduced for the first time. DM no. 146 set new criteria for the “restoring equilibrium share”. This is now based on educational demand (full-time equivalent students, for 30 %), educational results (number of university credits—CFU—earned by current students, for 30 %), research results (30 %) and specific incentives (as they were not identified, this 10 % was spread on the other shares). In this phase, for the allocation of reward shares (66.6 %), didactic performance seems to prevail over research performance. This means that university competition is mainly influenced by attractiveness (number of full-time students regularly enrolled) and educational quality (CFU earned and annual number of graduated students).

The third phase started with Law no. 1/2009. From 2009 (with a retroactive effect), this established the allocation of a reward share (no less than 7 % of FFO) on the basis of two variables. These were (a) educational supply quality and educational results and (b) research quality. Under this framework, resources were allocated for 34 % and 66 %, respectively, in relation to didactic and research performance. Prior incentives were changed and competition became dramatically oriented towards research quality. With regard to didactic performance, the model was simplified, passing from the original five to two weighted indicators: A1, the number of “active” students (at least five CFU earned) shared and weighted for each category, and A2—CFU earned/CFU expected ratio (DM no. 71/2012).

Nevertheless, as soon as universities began to assimilate the model, it was changed by DM no. 893/2014, following the principles of Law no. 240/2010. It introduced a demand-driven mechanism in resource allocation. In relation to the basic share of the FFO (for about 20 %), this entered into force in 2015. However, by 2019, it is going to become the only criterion for the whole basic share of the FFO. Under the new framework, didactic performance plays a key role and is measured with just two essential indicators. These are the number of current students and the standard cost per student for each university. A current student is defined as a “student enrolled within the regular duration of the degree course” (art. 1, c. 1., DM no. 893/2014), irrespective of the number of CFU earned. Meanwhile, the standard cost is an economic measure that defines an efficiency target in service delivery. In theory, it shows how much it should cost a university to educate a student within the expected time and considering the different socio-economic and structural contexts. In brief, the product of standard cost for the number of current students, in relation to the National standard cost, determines a portion of the basic share (20 % in the FFO 2014) that is given to each university, without any consideration of qualitative didactic performance. This is in line with the basic hypothesis of quasi-market theory. This argues that the free choice of service users rewards the best performers, drives supply towards higher levels of need satisfaction, increases efficiency and improves resource allocation in the market [26].

The Law no. 240/2010 also established the introduction of an accreditation system for university departments and degree courses, based on specific indicators defined in advance by a National agency (Agenzia nazionale di valutazione del sistema universitario e della ricerca, or ANVUR). The following Decree no. 19/2012 disciplined the implementation of the system of self-evaluation, periodic evaluation and accreditation (autovalutazione, valutazione periodica e accreditamento, or AVA), started in the academic year 2012/2013. Table 1 summarizes all the different laws and reforms about the financial system and the overall evaluation criteria set for public universities.

Table 1 Reforms, laws and interventions in the Italian public universities
Full size table

The new financing model has strongly impacted universities, as it tends to reward those with a high number of current students and penalize those with a low level of attractiveness. Otherwise, the propensity to increase the number of enrolled students seems to depend not only on service quality but also on context variables, which remarkably affect the demand characteristics. Thus, the hypothesis that the choice of users rewards the best producers is mostly unrealistic. This is because information asymmetry and other factors that influence demand should be considered. The choice of a certain university, for instance, is largely conditioned by context variables such as quality of life and services of the city in which the university is located, rather than income, logistical reasons and prestige. The evaluation of didactic quality shows the traditional ambiguity of relational services. Evidently, the number of enrolled—or graduated—students, the number of CFU earned or that of out-of-course students (those who have not completed the degree course within its regular duration) may also depend on the ease of graduating in countries where the educational qualification has the same legal force. Parameters that are used by the financing system drive the behaviour of service deliverers. This is because they naturally try to draw as many resources as possible, sometimes creating distortive effects on service quality or incentives to overproduction. On the other hand, it is certainly simpler to achieve a better performance in a favourable environment and vice versa. Demand quality in the entrance phase, which is related to socio-economic factors, also has an impact on didactic performance.

Financing mechanisms that are based on rewards tend to increase the gap between the best performers, which will gather extra resources, and the worst performer, whose funds will be progressively cut. Although this is an intended consequence of such a competitive model, in cases where a university is strongly affected by territoriality or socio-economic handicaps, resource cut increases unfairness between universities.

4 A Performance Management System to Improve Didactic Performance of a Bachelor Degree Course

A bachelor degree course, which usually lasts three academic years in Italian universities, is considered. The rationales behind the model are that the improvement of didactic performance in an organization derives from student success [7] and that the latter should be planned in advance and explained in terms of strategic goals and operational results, which are measured by appropriate indicators. In addition, specific actors must be appointed as responsible for achieving such goals through a set of actions that are taken at scheduled times. During the whole educational cycle, a continuous monitoring of student activities, as well as a report that relies on an information system that includes all of the useful performance information, should be developed. For this purpose, a performance management system is a powerful tool that can be used to increase rationality in decision-making at a degree course level.

The functioning and effectiveness of such a system are strongly related to the characteristics of the information system. The operational complexity of an organization and the multitude of information that is gathered from the outside to depict the context in which the educational offer will be delivered need advanced systems of data storage and integration. In fact, a balanced planning and control system [27], including a plurality of objectives for each phase of the didactic path (precollege and entrance phase, degree course duration and entrance in the job world), must rely on a double-purpose information system. Not only should it be useful to collect and archive internal and external data, but also, it should select and aggregate the data to inform decisions [28]. It can be defined as a “strategic intelligence system” that is able to continuously store data, regardless of the time of the decision-making [29, 30].

After defining the strategic goals, they must be translated into strategic and operational actions that are to be entrusted to a specific responsible actor and performed within a scheduled time. The responsible actor, scheduled time, actors involved and indicators to be reported and evaluated must be clearly defined for each operational phase (Table 2).

Table 2 The information system to monitor and evaluate didactic performance
Full size table

For example, in the following section, a mapping of strategic goals, actions and indicators of student success for each key phase of a university student’s career is shown. The law that is related to each indicator (reference), as well as its impact (low–medium–high) on the financing system of universities, is also shown.

4.1 Phase A: The Precollege Phase and Entrance to Academic Year “t”

The central purpose of this phase is to protect a student’s interest and support him/her in making the right choice. In this regards, the actions to be taken should not be oriented towards increasing attraction rate (more enrolled students in a certain degree course). Instead, they should be oriented towards enrolling students who are really motivated to that specific course and have the right basic skills.

During this phase, a wrong degree choice compromises a student’s whole educational path and often leads to student failure. It is also the main reason for the presence of “accidental students” that express an improper demand. In turn, this significantly worsens a degree course’s performance, as it negatively impacts on abandonment rates and inactive students.

The information system supports decision-making by giving useful information about the context in order to frame the potential characteristics of students and prefigure enrolment policies that foster their future performance. The collection of such external data is crucial. This can be done through simple questionnaires that are submitted during student orientation programmes. They should permit a full mapping of the student in terms of educational provenience, place of residence, part-time or full-time status, score average, attitude and so on.

Strategic Goals

  1. 1.

    Attracting high-quality and motivated students

  2. 2.

    Enhancing the consistency between enrolment alternatives and student status (part time or full time)

  3. 3.

    Improving the quality of students entering university (Table 3)

    Table 3 Indicators for strategic reporting and evaluation
    Full size table

Some strategic actions that need to be taken in order to achieve the strategic goals are shown below. Such actions, in turn, are translated into operational actions that specific actors must take in due times. These are measured through a set of indicators (for space reasons operational actions are not reported). Indicators that are used at this moment do not have a direct impact on financing. Nonetheless, attracting high-quality and motivated students consequently leads to good results along all of student careers, thus increasing the number of graduated students and their entrance into the work force.

Strategic Action 1

Orientation policies and programmes for the entrance phase, in collaboration with high schools of selected territories, which are aimed at attracting motivated and skilled students for a certain degree course.

Strategic Action 2

Communication plan of the degree course.

4.2 Phase B: The Educational Path (Duration of the Degree Course)

4.2.1 Phase B.1: The First Year of the Degree Course

The first year is probably the most delicate phase of a student’s career, as he/she has to get used to a new way of living and studying. During this phase, student behaviour must be monitored and continuously supported, especially in the case of problems. Above all, during the first months of the first year, a student may feel confused or just realize that he/she has made the wrong choice of degree course or university studies. In the worst case, he/she may decide to leave university.

For this reason, it is essential to guide and go after him/her, to understand his/her problems and help him/her to cope with them. Expected results include a reduction in the abandonment rate between first and second year and an increase in the average number of university credits that are earned by a student.

Strategic Goals

  1. 1.

    Increasing the rate of enrolment to second year (equal to reducing the abandonment rate)

  2. 2.

    Increasing the number of university credits that are earned in relation to expected credits (those shown in the degree course plan)

  3. 3.

    Increasing the average number of university credits that are earned by a student (Table 4)

    Table 4 Indicators for strategic reporting and evaluation
    Full size table

Strategic Action 1

Monitoring freshmen career and evaluating the critical factors at the end of terms.

Strategic Action 2

Reviewing teaching programmes and coordinating professors and lecturers.

4.2.2 Phase B.2: The Second and Third Year of the Degree Course

During this phase, it is important for decision-makers to be informed about the study delay of the students who enrolled 1 and 2 years before. A critical factor of an information system is the timeliness in taking note of passed exams, which can be fostered by leaving the hardcopy archive in favour of online systems. Data stored in this way should be promptly made available by the information system for reporting selected indicators in order to evaluate a student’s career and take specific action to remove any hindrances.

Strategic Goals

  1. 1.

    Increasing the average number of university credits that are earned by a second- and third-year student

  2. 2.

    Increasing the monitoring activity of teachings

  3. 3.

    Promoting participation in international exchange and mobility programmes

  4. 4.

    Increasing the number of internships (Table 5)

    Table 5 Indicators for strategic reporting and evaluation
    Full size table

Strategic Action 1

Monitoring student careers and also making evaluations term by term in order to promote participation in international mobility programmes.

4.2.3 Phase B.3: The Second Term of the Third Year of the Degree Course

In the second term of the third year, students must be oriented towards the successful completion of their degree courses (acquisition of all university credits and graduation in due time).

Strategic Goals

  1. 1.

    Increasing the annual percentage of graduates in due time

  2. 2.

    Increasing the satisfaction level of students who are about to graduate (Table 6)

    Table 6 Indicators for strategic reporting and evaluation
    Full size table

Strategic Action 1

Monitoring students to enhance the frequency and outcome of remaining exams and assigning degree thesis with a didactic weight that corresponds to the university credits that are set for the degree course.

4.3 Phase C: Post-degree Phase and Entrance into the Job World

After graduating students must be supported in their choice of the next best path. It is important to encourage the best students to continue their study with a master’s degree or a first-level master. However, attention has also got to be paid to help a student choose the best way in relation to his/her own needs and peculiarities. Otherwise, those who decide to not continue the studies should be oriented towards a post-degree internship experience in order to promote their entrance into the work force.

In this phase, the information system must include a multitude of external data on the labour market. The main critical factor is the cost of gathering such information. This can be effectively reduced by taking operational actions that are aimed at involving the firms and companies of the territory.

Strategic Goals

  1. 1.

    Increasing the annual percentage of graduates with a post-degree internship

  2. 2.

    Increasing the percentage of graduates who find a job within a year from their degree (Table 7)

    Table 7 Indicators for strategic reporting and evaluation
    Full size table

Strategic Action 1

Updating data and linking graduate registry to the business world.

Strategic Action 2

Organizing events aimed at promoting demand–supply matching and training students to enter into the work force.

The balanced set of the above-mentioned indicators is useful to express, control and evaluate the achievement of didactic performance objectives. In summary, two simple indicators can show the performance improvement of a didactic structure. The rate of graduates in due time (number of graduates of an academic year/total enrolled students of 2 years before ratio) expresses the output of the educational process. It is also a measure of efficiency and internal effectiveness. This is because it accounts for the ability of a didactic organization to graduate students within the time expected for a degree course. On the other hand, the student employment rate within 1 and 3 years from their degree is an extraordinary measure of external effectiveness of the degree course. This is because it expresses the real outcome of the educational process.

5 Conclusion

For university management, the development of advanced information systems to cope with the increased competition and progressive lack of resources is challenging. Competing and succeeding in such a complex environment seems to be related to the improvement of the decision-making quality, followed by performances that are related to university activity. Since the 1990s, several reforms have changed the resource allocation model from the state to universities in order to abandon a financing system based on historical expenditure and supply, in favour of reward systems, which are based on didactic and research performance. Following Law no. 240/2010, the recent DM no. 893/2014 introduced a demand-driven mechanism in resource allocation. In particular, didactic performance is extremely important due to the effects it has on the public financing system of universities and the goal of quality assurance in higher education that are set by the state.

Starting with the complex concept of student success, this paper was aimed to design a performance management system to improve didactic performance. It also aimed to highlight the conditions and peculiarities that a university IT system must have in order to be effective. Considering the key role it has on didactic organization, planning and outcomes, the degree course was viewed as a privileged decision-making level. Meanwhile, a bachelor’s degree course was chosen as the time interval over which the system is to be implemented. In fact, degree course policy can reasonably affect the behaviour of students and create fruitful institutional conditions to foster student engagement. Furthermore, the multidimensional nature of didactic success led to the identification of three specific moments of a student’s career to be separately evaluated: precollege and entrance phase, degree course duration and final phase and postcollege outcomes.

For each moment in time, a set of strategic goals, which are then translated into strategic and operational actions and finally measured by performance indicators, were identified. Responsible and involved actors, as well as the scheduled times, were also identified. The rationales behind the model are that improving didactic performance derives from student success and that this can be rationally planned in advance and explained in terms of strategic goals, operational results and indicators. A key point of this study is the importance of a sound and complete IT system. The multitude of internal and external information that needs to be gathered to monitor a student’s career along its different phases requires advanced systems of data storage and integration. The possibility of selecting and aggregating archived data to inform decisions at the right time marks the evolution towards a “strategic intelligence system”. This is a significant difference for university didactic performance. The proposed system is currently being tested in a bachelor’s degree course of a department of an Italian university. Future research avenues are related to the results of the biennial test and include the refinement of the model and its extension to other degree courses.