Quality of Higher Education: Key Components of Stakeholders’ Assessment

Abstract

The article describes the findings of the research into the Russian and international practices of building the higher education quality assessment system. The issue of building of national higher education quality assessment system based on the modern challenges faced by the contemporary higher education system and the best international practices gained in external and internal quality assessment system development are examined. The article is part of the commissioned research conducted by the Financial University in accordance with the order of the Government in 2018.

1 Introduction

Higher education systems play an important role in the development of any society and economy. The goal of the Global Education 2030 Agenda developed by the UNESCO is to build a knowledge economy that will ensure sustainable employment and high living standards. It stresses the importance of reliable quality assessment and quality assurance systems.

Today, the higher education quality assessment and quality assurance systems development gains momentum worldwide. The key drivers of this are the increasing public demand for high-quality educational services, a solid stakeholders’ demand for the education system efficiency improvement, the need of highly-qualified personnel who could manage national economies and the need of better using the public resources for the education system development purposes.

Quality control and quality management are the priorities in the Russian education system modernization process. The current understanding of the Russian national education quality assessment system is based on combination of institutions that ensure use of a single methodology and concept-based assessment of academic achievements and identification of personal children’s and adults’ qualities and factors that make an impact upon the learning outcomes.

The internationalization and globalization of higher education requires the harmonization of quality assessment and quality assurance systems. The comparative analysis of the Russian and international practice of higher education assessment systems underlines the possible ways for contingence of different approaches that should have a positive impact on the development of global learning environment.

2 Methodology

Authors conducted a survey of conference participants to identify the key components of higher education quality during the annual Education Foresight: Accreditation Bounds VS Academic Freedom International Financial University Research Conference that was held on 20–22 March 2019. The survey respondents were the 1300 representatives of the academic and industry community and 580 students from Moscow, St. Petersburg, Sevastopol, the Republic of Bashkortostan, the Republic of Dagestan, Altai Territory, Nenets Autonomous Area, Krasnodar Territory, Moscow, Omsk, Oryol, Penza, Chelyabinsk, Yaroslavl, Vladimir, Saratov, Novgorod, Tula, Lipetsk, and Novosibirsk Regions.

When answering the question “What is the most objective procedure to be used to assess and recognize higher education quality”, the respondents identified the key indicator of education quality was the graduate employability. According to them, the accreditation performed by the government accreditation agencies is as important as that performed by the independent accreditation agencies and the rankings have virtually no significance as far as higher education quality is concerned (Table 1).

Table 1 Higher education quality assessment survey results

The higher education quality assessment system in the Russian Federation includes the following: assessment by the government institutions made through the use of the state accreditation procedures, education system monitoring procedures and assessment performed by the independent stakeholders via industry accreditation, accreditation by accreditation agencies, international comparative analysis and other education quality assessment and monitoring procedures (Fig. 1).

Fig. 1

Russian higher education quality assessment system and its components

Operations in the field of education in the Russian Federation are regulated by the federal state educational standards (except for the further education programs and pre-school activities) [4]. Federal state higher educational standards (hereinafter referred to as FSHES) that govern the educational program development process have been updated during the past two years and the education quality assessment standards have been changed.

For instance, an obligatory internal operation quality assessment standard has been included into the list of the FSHES. Part 4.6 of the FSHES describes internal assessment made by the teaching staff, employers, their associations and students and the external assessment:

• Accreditation performed by the government institutions is ‘the confirmation of the fact that the operations performed in accordance with the program curriculum meet the FSHES and the curriculum requirements’;

• Accreditation by accreditation agencies is ‘the confirmation of the fact that the quality of graduates’ training meets the professional standards’ requirements (if any) and labor market requirements’.

The higher education quality assessment procedures diverse depending on the type of organizations conducting the assessment.

• Accreditation performed by the government institutions. This is the program accreditation performed by the government institutions, supervision of education system operations (HEI efficiency monitoring) by the Ministry of Science and Higher Education of the Russian Federation.

Accreditation performed by the government institutions means determining the compliance of the program content and education institution’s students’ training quality with the federal state educational standards. Nationally accredited education institutions have certain special characteristics, incl. the following: male students have a right to have deferment from military service in the Armed Forces of the Russian Federation for the period of study; the institution has a right to issue diplomas formatted in accordance with the standards determined by the state; the students have a right to use the maternity capital funds when paying the tuition fee, and the students have a right to have their tuition fee expenses covered by the federal budget funds.

Supervision of the operations in the field of education (monitoring of HEI efficiency) by the Ministry of Science and Higher Education of the Russian Federation is a systemic supervision of the learning environment, student cohort composition, students’ academic achievement and graduates’ achievement in the workplace. Monitoring is conducted in order to provide detailed objective information on the education system to the Russian nationals and the government institutions. The Ministry of Science and Higher Education of the Russian Federation uses the information in order to prepare an annual report to be presented to the Government of the Russian Federation. As of 2014, reports on the regional and municipal education systems have been posted on the official websites of agencies monitoring the educational systems’ operations and are freely accessible to the public.

• Independent higher education quality assessment means an assessment process that is aimed at obtaining information on the educational activity conducted by the education institution, students’ training quality and the program design [3]. Independent higher education quality assessment can be external and internal.

The detailed independent education quality assessment description is contained in the Methodological Recommendations for Making Independent Educational Organizations’ Educational Activity Quality Assessment approved on 1 April 2015 by the Ministry of Education and Science of the Russian Federation (circular letter No. AP-512/02 of the Ministry of Education and Science of 3 April 2015 ‘On Distribution of Methodological Recommendations for Making Independent Educational Activity Quality Assessment’) [7].

• External assessment is assessment made by the experts (accreditation made by the accreditation agencies, incl. industry accreditation and accreditation be internationally recognized agencies).

In accordance with Article 96 of Law No. 273-FZ ‘On Education in the Russian Federation’ [3]:

• 1. Education institutions can be accredited by various Russian, foreign and international organizations;

• 2. Accreditation means evaluating the education institution’s operations and confirming that the operations are in conformity with the criteria and requirements set by the Russian, foreign and international organizations;

• 3. Employers, their associations and agencies authorized by them have a right to perform accreditation of educational programs (hereinafter PA) offered by the education institution;

• 4. Industry program accreditation means confirming that the education institution’s graduates’ training quality meets the professional standards’ requirements, and the requirements of the labor market set for the employees who are experts in a certain area.

Industry and international accreditation agencies operate in accordance with the following principles when making independent quality assessment:

• Participation is voluntary;

• The examination process is conducted and accreditation decisions are made independently; no discrimination is allowed and no biased decisions are made;

• The information on the procedures and criteria for assessment and accreditation is open for the public;

• Assessment and accreditation procedure participants’ rights are protected;

• The information used by experts is complete and reliable;

• Accreditation agencies are responsible for the results gained by the accreditation agencies during the examination and the decisions made by the accreditation agencies;

• Accreditation procedure results are accessible and open to the public.

National Qualifications Council (NQC) under the President of the Russian Federation founded in accordance with Decree No. 249 of the President of the Russian Federation of 16 April 2014 consolidated the efforts of employers, government institutions, and trade unions in order to develop a national qualifications framework. The NQC examines draft laws and by-laws, drafts of professional standards and coordinates efforts aimed at improving quality of higher education, incl. improvement of industry accreditation of programs of study. Sectoral qualifications councils under the national employers’ associations founded by the NQC are engaged in this on the sectoral level.

It should be emphasized that the PA is the recognition of graduates’ training quality and the confirmation of the fact that the program meets professional standards. Conformity with the labor market requirements is at the core of the industry accreditation. It should be taken into account the fact that the PA does not coincide with the accreditation performed by the government institutions as different assessment criteria and procedures are used in the PA. The Russian PA can be compared to the international program accreditation procedure. Qualifications councils founded by the NQC, All-Russia associations and All-Russia industry employers’ associations and major employers have a right to conduct the PA procedure.

• Internal assessment is performed through the internal (university) independent higher education quality assessment system. The key goal is to assess quality of training of the students in the most objective way possible. Each education institution has its own internal quality management system in place (an ISO standards-based quality management system, European standards-based quality assurance system, etc.) However, it is important to emphasize that the independent university higher education quality assessment system should include a certain independent component into each of the assessment events. If such an independent component is lacking, the system cannot be considered to be independent.

The goals of the internal independent assessment system are the following:

• to improve the structure and update the content of the educational programs offered by the education institution;

• to improve the resources used to support the teaching process;

• to improve the competence and professionally develop the teaching staff involved in the teaching process;

• to improve student motivation to learn;

• to improve the interaction between the education institution and the social partners, incl. the employers when discussing program improvement issues;

• to counter corruption in the teaching process.

When building an internal university independent higher education quality assessment system, the universities embed independent elements into the existing higher education quality monitoring procedures. Students’ assessment of the higher education quality is undoubtedly an important and significant internal independent higher education quality assessment system element.

Examination of the education systems of other countries shows similar trends in the development of approaches to higher education quality assessment, incl.:

• Student engagement as a basis for standards and assessment methods development. The American national research project NSSE findings prove that the student engagement in the teaching process and their learning outcomes highly correlate (NSSE) [2]. To fulfill their supervisory obligations, the educational institutions’ competent authorities should be aware of the meaning of such concepts as the student engagement in the teaching process, they should learn about the students’ expectations associated with the higher education quality, and identify the best ways to disseminate information about the learning outcomes [6].

• New formats of flexible learning. A number of new teaching formats and practices emerge as there is a demand for using a more flexible approach to learning on the educational services market. It includes the use of competences-based learning modes, recognition of the learning outcomes obtained by the students of the Massive Open Online Courses (MOOCs), modular programs, recognition of informal learning outcomes, etc. Without doubt, the quality assurance system should change accordingly to the new reality.

• A focus on skills and competencies. Majority of national (incl. Japanese, British, Canadian, and US) governments express concern over the fact that the universities do not produce graduates with the job-ready skills that the economies need in order to provide opportunities for growth, create incentives for innovation, and build wealth. Such university graduates’ soft skills as critical thinking, teamwork, efficient communicative skills, ability to learn, problem identification and solving skills are becoming increasingly important for employers. These skills are developed during the period of study of certain disciplines that contemporary universities include in the program curricula [5]. Therefore, the higher education quality assessment procedures should also be flexible, capable of assessing professional competencies and non-academic skills required by the employers.

The globalization process and the increasingly fierce competition have facilitated launch of international higher education quality assurance organizations and processes (Figs. 2 and 3).

Fig. 2

The key global higher education quality assessment system participants

Fig. 3

International organizations engaged in higher education quality assurance and assessment

• the International Network for Quality Assurance Agencies in Higher Education (INQAAHE) [1]. Section 1: The External Quality Assurance Agency (EQAA): Accountability, Transparency and Resources; Sect. 2: Institutions of Higher Education and the EQAA: Relationship, Standards and Internal Reviews; Sect. 3: EQAA Review of Institutions: Evaluation, Decision and Appeals; Sect. 4: External Activities: Collaboration with Other Agencies and Transnational/Cross-Border Education

• the European Association for Quality Assurance in Higher Education (ENQA): The Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG) (Part 1: Standards and guidelines for internal quality assurance; Part 2: Standards and guidelines for external quality assurance; Part 3: Standards and guidelines for quality assurance agencies).

• the Council for Higher Education Accreditation (CHEA). Standard 1. Promotes academic quality and advances student achievement; Standard 2. Demonstrates accountability for performance and transparency; Standard 3. Encourages, where appropriate, self-scrutiny and planning for change and for needed improvement; Standard 4. Employs appropriate and fair procedures in decision making; Standard 5. Demonstrates ongoing review of accreditation practices; Standard 6. Possesses sufficient resources;

• Ibero-American Network for Quality Assurance in Higher Education (RIACES);

• Asia Pacific Quality Network (APQN): institutional internal and external higher education quality assessment and quality assurance in accreditation agencies’ operations.

The approach offered by the ENQA is the most trusted approach to quality assurance and quality assessment in higher education for the Russian Federation, a Bologna Process country participant.

While wishing to harmonize teaching and research, the Bologna Process country participants join their efforts to develop a single approach to higher education quality assurance and assessment. Established in 2000, the European Association for Quality Assurance in Higher Education (ENQA) is a non-profit organization engaged in independent monitoring of education institutions’ operations.

In 2015, ENQA presented the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG) (hereinafter referred to as the ESG Standards) to the academic community members [5]. Two years of intense work of all the stakeholders, incl. European accreditation agencies, education institutions, student associations, professional and academic communities were needed to develop the ESG Standards. The document was adopted by the Ministers responsible for higher education in the European Higher Education Area in May 2015.

The ESG Standards consist of three parts [9]:

Part 1: European standards and guidelines for internal quality assurance within higher education institutions;

Part 2: European standards and guidelines for the external quality assurance of higher education;

Part 3: European standards and guidelines for external quality assurance agencies (ENQA, 2015).

All three parts of ESG Standards are closely connected with one another and are the basis upon which the European quality assurance institutions are built.

Internal quality assurance standards include examination of the quality policy, the educational program development and approval process, student-centered approach to training, teaching and assessment, student admission procedure, learning outcomes recognition procedure, academic performance and certification, the teaching staff qualifications; learning resources and student support infrastructure, data management and information distribution procedure, and monitoring and periodic program review.

According to the ENQA recommendations, external quality assessment procedures should be conducted in education institutions in accordance with the ESG Standards on a regular basis. Guidelines for external quality assessment of higher education are described in the second part of the ESG Standards. The following principles and tools form the basis of the standards:

• external quality assessment procedures should take into account the internal quality assurance procedure efficiency;

• external quality assurance procedures should be fit for specific purposes and comply with the by-laws;

• external quality assessment procedures should be reliable and well-defined;

• the external quality assessment procedure should include 3 main stages: analysis of a self-examination report, the external assessment event proper, which, as a rule, includes a visit of a group of experts to a higher education institution, and preparation of a report based on the results of the external assessment;

• subsequent quality assessment procedures should be uniform;

• students should be included into the group of external experts that makes the external assessment;

• decisions made and conclusions drawn following the external quality assessment should be based on published strict criteria, regardless of whether the procedure leads to an official resolution or not.

The third part of the ESG Standards defines the operations of external higher education quality assessment agencies. The agencies should be independent and autonomous, should have a legal status and be officially recognized as quality assurance agencies. The periodic external review of agencies’ operations should be made at least once every five years to confirm their compliance with the ESG standards.

The 2015 Conference of Ministers of Higher Education of the countries participating in the Bologna Process has also developed the European Quality Assurance Register for Higher Education (EQAR). It officially registers higher education quality assurance agencies that comply with the ESG Standards [10].

Another important contribution made by the Yerevan Communiqué signed in 2015 by the Ministers of Higher Education of the countries participating in the Bologna Process was a document titled European Approach for Quality Assurance of Joint Programs. The world’s leading universities have been offering the joint programs for a long time as such programs provide an opportunity to students and staff to become engaged in academic mobility programs which contribute to the international cooperation development and harmonization of the EHEA. In accordance with the European approach, the joint program quality assurance is also based on the use of the ESG standards. Partner universities should jointly select an agency that offers external quality assessment services and is registered in the EQAR. The Agency shall in due course interact with the competent national authorities of the countries where the partner universities are located.

In most countries where objective higher education quality assessment systems are developed, such systems are based on a combination of quantitative parameters and good expert opinions.

Quantitative parameters, as a rule, are based on the collected statistical data on all the key university operations. Much attention is paid to teaching, research, and the quality of services provided to the local or regional community. China has gained the best practices here. Using the latest information technologies, it builds its own assessment system based on the generated and examined Big Data.

Universities are placed into quality clusters based on their qualitative indicators. Qualitative assessment is made on the basis of, first of all, self-examinations where immediate and long-term development goals are taken into account; a combination of internal and external university assessments is used. External assessment goes beyond the academic community and takes into account the opinion of employers, industry representatives, and users of university’s services.

3 Results

Table 2 systemically shows the characteristics and compares the indicators used in the PA made by the Council for Professional Qualifications of Financial Market Experts and the internationally recognized accreditation made by the European Council for Business Education (ECBE).

Table 2 A comparative analysis of indicators of international program accreditation offered by the ECBE (based on the ENQA principles) and industry accreditation offered by the Council for Professional Qualifications of Financial Market Experts (based on the principles of the Presidential National Council for Professional Qualifications)

4 Recommendations

Despite the fact that directly correlate the criteria-based quality assessment indicators using the methods offered by the ECBE and the Council for Professional Qualifications of Financial Market Experts, generally, the authors observe a certain conceptual similarity of parts of the international and Russian public assessment systems. In general, the goals of international and Russian non-profit accreditation agencies can be described as follows:

• 1. To assess the actual learning outcomes achieved by the graduates.

• 2. To assess the quality assurance policy that the education institutions demonstrate to the students.

• 3. To identify and examine the educational program advantages and disadvantages.

• 4. To assess how the programs relate to the international education system development trends, incl. program development based on the expected learning outcomes; program development based on the research findings, the maximal use of the research findings in the learning process; employment of highly-qualified and competent teaching staff, etc. [8].

• 5. To assess the compliance between the training offered to the graduates and the industries’ standards requirements, the international, national and regional labor market requirements set for experts in a certain field; national and international qualifications frameworks’ requirements.

• 6. To assess whether the program directors track and take into account the changes in the program content and higher education quality requirements set by the employers and the labor market.

• 7. To assess the program efficiency in terms of the graduate employability, employment and career development.

The key difference between the approach used by the ECBE and the Council for Professional Qualifications of Financial Market Experts is that in the ECBE methodology, the emphasis is placed on the student satisfaction with the higher education quality and student participation in the assessment. Industry accreditation methodology proposed by the Council for Professional Qualifications of Financial Market Experts, first of all, stresses the necessity of taking into account the employers’ (labor market) satisfaction with the quality of graduates’ training. The employers’ important involvement in all the stages of the teaching process (from the program design stage to the learning outcomes assessment) is in focus here.