The National Assessment of Courses in Brazil

Abstract

In 1996, the Brazilian Ministry of Education introduced a National Assessment of Courses for Brazilian higher education. The assessment consisted of a test applied to all students graduating from specific course programs in the country. The objective of the test was to provide information to the public on the quality of higher education courses, helping the students and their families to choose where to study, and to provide the Ministry of Education with information that could be used in the accreditation and reaccredidation of higher education institutions. The exam generated an intensive process of discussions and consultations among academics about the contents and standards of the different careers, which supposedly helped to improve the quality of Brazilian higher education throughout. While the exam faced strong opposition from student associations, teachers’ unions and many higher education institutions, it was received with support in public opinion and in the press. The results became widely used by students in their choice of institutions and by institutions themselves, particularly in the private sector, which either publicized their results or tried to improve them.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In 1996, the Brazilian Ministry of Education introduced a National Assessment of Courses for Brazilian higher education. The exam – which became known as Provão, the big exam, or ENC – consisted of a national test applied to all the students graduating in each specific course program in the country. The results were published in a five-point scale, from A to E, according to their distribution in each field. In the first year, the test was applied to students graduating in Law, Administration, and Civil Engineering – the careers with the largest attendance. In 2003, the exam included 470,000 students graduating in 26 different fields in 6,500 course programs in the whole country.

The objective of the test was to provide information to the public on the quality of higher education courses, helping the students and their families to choose where to study, and to provide the Ministry of Education with information that could be used in the accreditation and reaccreditation of higher education institutions. Besides, the exam generated an intensive process of discussion and consultations among academics about the contents and standards of the different careers, which is supposed to have helped to improve the quality of Brazilian higher education throughout.

The exam was introduced without previous consultation, and was received with strong opposition from student associations, teachers’ unions, and many higher education institutions. However, from the beginning, it received strong support in public opinion and in the press. The criticism ranged from specific objections to the way the tests were conceived and the results presented – a uniform test for the whole country, a national rank of outcomes without consideration of existing conditions and explicit standards – to broad objections to any kind of measurement of education outcomes. However, once in place, the results became widely used as references for students in their choice of institutions, and for the institutions themselves, particularly in the private sector, to publicize their results, or to try to improve them. Bad results, when persistent and associated with other indications of low quality, were supposed to lead to the closing down of the course programs by the education authorities, but, in practice, this has seldom happened.

In 2002, the opposition Laborer’s Party won the presidential elections, and the candidate’s program for education announced the end of the National Exam. Once in power, the new Minister of Education established a Commission to examine the issue and to propose a new approach to higher education assessment. The Commission published its conclusions in September 2003, and, in December the Government announced its own proposal for higher education assessment, which changed the previous system very substantially. At the end, the new assessment system kept the national exam with a new denomination, the National Exam of Student Achievement (Exame Nacional de Desempenho de Estudantes – ENADE) and very substantial modifications which, in this author’s view, has very serious methodological and conceptual flaws, and makes it much less significant than before. This chapter provides a detailed analysis of ENC, and also an overview of ENADE, implemented in the years 2004, 2005, and 2006 (Table 15.1). The last ENC took place in 2003.¹

Table 15.1 Evolution of the National Assessments of Course Programs in Brazil

Context

Brazilian higher education developed late, and was based on the European, mostly French and Italian models. Until the early nineteenth century, Brazil was a colony of Portugal, and no higher education institutions existed – it was necessary to go to Coimbra in Portugal or perhaps France to get a degree. In 1808, the Portuguese King and his court moved to Brazil, fleeing from the invading Napoleonic troops, and Rio de Janeiro became, for several years, the capital of the Portuguese Empire, to become later an independent country. The first higher education institutions were established in those years – one military academy, later to become a school of engineering; two medical schools; and two law schools. They were all owned, financed, controlled, and supervised by the royal government. In the late nineteenth and early twentieth century, as the old Brazilian Empire was replaced by a decentralized Republic, other institutions were added. Some states – notably the state of São Paulo – started to create their own institutions, and private institutions began to appear. Until 1889, only 24 higher education faculties existed; between 1889 and 1918, 56 new, mostly private, faculties were established.² New fields, like pharmacy, dentistry, agriculture, and accounting, were introduced side by side with the old learned professions.

The first universities were established in the 1930s, and they were, mostly, a collection of old schools or faculties, with one important innovation, a new Faculty of Philosophy, Sciences, and Letters, which was to be, at the same time, the place for scientific and academic research, and for the preparation of secondary school teachers. The first University, the University of São Paulo, was established by the State government in 1934, and the Universidade do Brasil, now the Federal University of Rio de Janeiro, was established in 1939 by the National government. In the early 1940s, the Catholic Church created the first private university in Rio de Janeiro, and they all introduced course programs in the natural sciences, mathematics, history, geography, social sciences, philosophy, and language and literature, which did not exist before.

Throughout the nineteenth century, holders of higher education degrees strived to assert their exclusive rights to practice their respective professions, and, after the 1930s, the principle that a university degree was tantamount to a professional license became firmly entrenched (see Coelho 1999). This created, at once, a problem of regulation, which was never fully solved. To deal with this, a new Ministry of Education was created together with a National Education Council formed by public personalities. The new ministry tried to establish a “model university” in the country’s capital, based on a detailed description of the course contents of all disciplines, down to the assignation of textbooks and time tables, which all other institutions had to follow (Schwartzman et al. 2000). At the same time, the government created a complex system of professional councils, which, together with the business associations and the trade unions, were supposed to organize the country into a neat and coherent corporatist structure, integrating the professions, the entrepreneurs, the unions, and the education institutions (Malloy 1977; Schmitter 1974; Schwartzman 1988).

This tightly conceived system never worked in practice, and its limitations became all too obvious as higher education began to expand and new professions started to emerge after the Second World War. However, the basic assumptions established in the 1930s – that all higher education degrees should be equivalent to a professional certification, that all professions had to be regulated by law, controlled and supervised by a legally established professional council or association, and that it was the role of the Federal government to make sure that all course programs provided equivalent contents – remained and are still in place. The National Education Law of 1996 introduced more flexibility, and the legal requirement that each career should have a national “minimum curriculum” was replaced by more general “curriculum guidelines” (Ranieri 2000). In a sense, the National Assessment of Courses of the 1990s could be seen as a step backward in terms of centralization, although, in many cases, there was a genuine effort to limit the assessment to very central skills and competencies, allowing for local experimentation and variations.

The Policy Problem

In the 1940s and 1950s, the Federal government created a network of Federal Universities, established usually by the absorption, through legislation, of existing private and state-based institutions, based on political considerations, without any mechanisms of quality assurance. At the same time, new private institutions emerged, first as religious and community-based institutions, and later, predominantly, as profit-oriented endeavors. In 1968, there was an important university reform, introducing several innovations taken from the American context – graduate degrees, the credit system, departments, and institutes – with the assumption that all higher education should evolve toward a university model, based on academic research and a full-time academic profession. Simultaneously, however, the government responded to the growing demand for higher education by making it easier for private institutions to open up and offer degrees, without too much control and oversight. By the 1990s, higher education in Brazil had expanded very rapidly. The number of students doubled in 10 years, from 1.5 to more than 3 million, two thirds of them in private institutions. Some of these institutions tried to follow the 1968 model of university organization. Most of them, however, provided just one or a few course programs, particularly in business administration or law, without postgraduate education, and based on part-time lecturers, drawn from the professions or from retired or moonlighting academics from the public sector.

The pressures for and against opening up new institutions and controlling their quality comes from many sides (Schwartzman 1998). Brazil’s higher education coverage, at about 11% of the 18–24 age cohort³, is still very limited, and the social and economic benefits of higher education degrees and the entrance in the learned professions are very high, creating a growing demand for more places. In recent years, the provision of private higher education became a multibillion dollar business, employing about 200,000 people, among lecturers and administrative workers (Schwartzman and Schwartzman 2002). Side by side with small institutions, there are now very large private universities, with tens of thousands of students in many different locations, with considerable ability to lobby the government and Congress for freedom from control and regulation. Opposition to the expansion comes from the professional organizations, particularly in Medicine and Law, who are concerned about the watering down of their professional standards and job market privileges; and from the academics and students in public institutions, for similar reasons.

Quality assurance is not, however, a problem limited to the private sector. The Brazilian legislation grants full academic autonomy to universities, many of them public, which includes the right to create new course programs and to define the number of students admitted each year. The assumption is that universities are established according to strict academic standards, but in fact public universities can be created by Federal or state legislative acts. In principle, private institutions need to be accredited to get university status and be granted the same autonomy, but, in practice, accreditation has been granted case by case, without any systematic assessment. A new type of institution has been officially recognized in recent years, the “university centers”, which are (mostly private) institutions dedicated solely to teaching, expected to be of good quality, which have almost the same autonomy as the universities.⁴ Thus, the authority of the Ministry of Education is limited to the approval for the creation of new universities and university centers in the private sector, and to the minute oversight of non-university institutions, which have to apply for each new career they want to establish, and for the number of students they expect to admit.

The demands for a system of quality assurance, beyond the bureaucratic and ineffective procedures of the Ministry and the National Council of Education, on their different incarnations, has been clear since at least the Presidential paper on Higher Education of 1985 (Brasil Ministério da Educação 1985), and has led to several initiatives since then. They included a program to provide universities with resources to develop their self-evaluation (Ministério da Educacão and Secretaria de Educação Superior 1997) and the establishment of National Commissions of Specialists to define and revise the minimum core curricula of the different careers.⁵ In the late 1995, under Minister of Education Paulo Renato de Souza, a comprehensive system of assessment of higher education was created. It included the development of a yearly census to provide quantitative information on the sector by region, state, fields of knowledge, and type of institution; qualitative assessments of each institution, large and small, looking at their installations, institutional development plans, research performance, and other indicators of quality; and assessment of individual course programs or careers, with two components. The first was an assessment of their resources in terms of academic personnel, infrastructure, and internal organization if they had clearly defined missions, self-assessment, and coherent pedagogical projects. This assessment was carried on by peers, who visited each course program to get the information and process them according to predefined template. The second was the National Assessment of Courses, an exam to which all the students had to present themselves before graduation. Postgraduate education (Masters and Doctoral programs), in the meantime, have been subject to a well-established assessment procedure which remained in place (Ministério da Educacão 2002).⁶

Implementation

The authority for the Ministry of Education to implement the assessment was established by federal law⁷, which made it mandatory for students to complete the test if it is applied to their field in their last year of graduation, as a precondition to obtain their degrees. This was possible because higher education degrees in Brazil, to be legally valid, have to be registered with the Ministry of Education, usually through the office of a Federal university. However, there is no minimum pass grade for the students, since the goal is to assess the course program, not the student. In the first years, the National Student Union asked their members to boycott the exam, and, in some institutions, the students would just sit without answering the questions. This, however, led to a low ranking to their courses, which reflected badly among their colleagues who did participate, and this practice was abandoned almost completely in the following years.

The implementation for the National Assessment was carried on by an agency within the Ministry of Education, the National Institute for Education Research (INEP) following a very elaborate procedure.⁸ First, an assessment committee was established for each field of knowledge. Members were chosen from lists prepared by professional associations, teaching and scientific associations, and by the Brazilian Council of Rectors and the Ministry of Education. They had to be also representative of Brazil’s different regions, and different types of institutions – public and private, large and small. From these lists, the Ministry of Education would choose seven names in each area. Thus, for the year 2002, there were 24 such commissions, in Administration, Law, Civil Engineering, Chemical Engineering, Veterinary, Dentistry, Electric Engineering, Journalism, Language and Literature, Mathematics, Economics, Mechanical Engineering, Medicine, Agronomy, Biology, Physics, Psychology, Chemistry, Pharmacy, Pedagogy, Architecture, Accounting, Nursing, and History. They met in Brasilia, and their task was to define the general contents, scope, and goals of the assessment of their fields. For their work, the Ministry obtains all the course descriptions, pedagogical projects and teaching programs adopted by all institutions in the country, and organize this material in terms of their goals, objectives, basic bibliography, teaching procedures, and so on, identifying eventual differences in these orientations and goals. The Commissions work also with reports of the assessment of previous years, prepared by the Ministry and through seminars held with the participation of course coordinators and professors in each field. Based on this information, it is the task of the Commissions, each year, to revise and improve on the guidelines of the previous year in an interactive and continuous learning process.

Once ready, the guidelines prepared by the Committee are passed on to an external contractor, who has the responsibility of developing the tests, administering them, and tabulating its results. The choice of this external contract is made through open, competitive bids. In practice, two institutions working together, the Fundação Carlos Chagas in São Paulo and Fundação Cesgranrio in Rio de Janeiro, have won all these bids since 1995. They are experienced in administering large-scale assessments, having started with the entrance examinations for public institutions in São Paulo and Rio de Janeiro. They also recruit academics in the universities to develop the instruments, and persons in different institutions to deliver, control, and oversee the exams.

Before the exam, the institutions have to provide a list of all students likely to conclude their course programs in a given year. The exam takes place on the same day throughout the country, and is widely announced in the press. Observers from professional associations, teaching associations, and other entities are regularly invited to be present in the different locations where the exam takes place.

Just before the exam, the students receive from the Ministry of Education a magazine, explaining the purposes of the exam, the description of the procedures, and other materials. The core instrument is a written exam, which can be either of a multiple-choice test or open-ended questions, or both, according to the Committee’s recommendations. The general orientation is to put emphasis on the mastery of key concepts, on the ability to think independently and to apply knowledge to new situations; rote learning and the accumulation of information for its own sake are discouraged. Another instrument is a survey questionnaire, in which the students are asked to provide socioeconomic information on themselves and their families, and their views and perceptions about their course programs. A third instrument is the student’s assessment of the assessment – if they like the instrument, if they considered it too easy or too simple, inappropriate, etc.

The correction of the multiple-choice tests is done through the use of optical scan technology, and grades are provided after an assessment, by the Commission, of each item’s discrimination, level of difficulty and reliability. For open-ended questions, a sample of the responses is used to develop an assessment protocol, which is then applied to the universe of respondents. The grades received by each student are established according to their relative place in the distribution of results for the whole country. According to the mean results of their students, each course program receives a grade – from A to E. The student’s individual results are made available confidentially to each student, in a bulletin with information about his relative placement regarding his class, region, and country. The course’s mean score, however, is made public.

There are several follow-ups, besides the establishment of the grades. Immediately after the exam, the correct answers to the questions are made public, so that the students can see what they did right or wrong, and the professionals in the field can assess the quality of the exam. Then, the aggregate results of the students’ assessment of the assessment for each course are made available on the Internet to the course coordinators.

The next step is a series of national seminars, for each field of knowledge, to discuss the results of the last exam, with the cooperation of professional associations, course coordinators, and university. In these seminars, the results are presented, the Commissions share their views, complaints are aired, and the officers from INEP in charge of the whole process have an opportunity to hear the views of the academic community and express their perceptions of the whole process.

Meanwhile, the Ministry prepares a series of technical reports for each exam, and also a summary of the main statistics obtained with the socioeconomic questionnaire, which helps clarify the characteristics and attitudes of the students. There is a report with a synthesis of all the results, reports for each field of knowledge, and individual reports sent to the persons in charge of each course program.

Finally, some research institutions and independent researchers are asked to make more in depth analysis of the data, which may be disseminated by the Ministry, published as academic papers, or remain as technical reports of limited circulation.⁹

There is no estimation of the total cost of the operation. In 2002, the cost paid to the external contractor was about 36 million reais, or 12 million dollars (according to the prevailing exchange rate at the time). With these resources, they were required to prepare 24 different exams to be applied to 361,000 students graduating from 5,000 course programs in 627 municipalities. The per capita cost was, therefore, 100 reais or 33 dollars per student. There are many more course programs in the country, but these 24 accounted for about 90% of the students graduating in that year. There is no information about the internal costs for the Ministry of Education, which includes travel of the 168 members of the academic commissions to meetings in Brasilia, the time of the staff working in the preparation of the materials for the Commissions to work, the organization of seminars and other events, and contracts with external consultants for the analysis of the data. It is a sizeable effort, but not out of proportion, if one considers that the Ministry of Education spends about 5 billion reais – 1.6 billion dollars – a year in higher education alone.

Impact

The Brazilian legislation gives to the Minister of Education, with the support of the National Council of Education, the authority to accredit new higher education institutions and to renew their accreditation periodically. In practice, however, once a higher education institution is allowed to function, only in extreme cases will it lose its authorization or accreditation, and the process of periodical accreditation and reaccreditation of universities was never fully implemented. The government has intervened in a few private institutions in recent years, but never in a public university, and never because of a negative assessment of their academic quality (in some cases, attempts by the Ministry to close down bad quality courses and institutions were stopped by the judiciary or by appeals to the National Council of Education). The National Assessment of Courses was meant to be influential information for such decisions, but, since it refers to course programs, and not to whole institutions, it can be at most one element of information in a much broader assessment procedure, still to be implemented.

Because of this, the direct contribution of the National Assessment for the regulation of higher education has been minimal. Its indirect impact, however, is considered very important. One such impact was to encourage students to search for better-ranked course programs. According to a study done by the Ministry of Education, the number of new applicants for courses in Administration, Law, Civil Engineering, Chemical Engineering, and Dentistry, who received “D” and “E” in the assessment, went from 35,000 to 18,000 between 1997 and 2001, while the demand for courses rated “A” increased by 6%. Another finding was that new courses in private institutions, established after the assessment was introduced, tend to be better than many old ones. Thus, both students and academic officers are taking the concepts into account, and changing their behavior, looking for better course programs, and trying to work according to higher standards (Ministério da Educação and Instituto Nacional de Estudos e Pesquisas Educacionais 2002).

The professional and academic associations in Administration carried out a detailed survey among course coordinators on the impact of the assessment in their institutions (Conselho Federal de Administração and Associação Nacional de Cursos de Graduação em Administração 2003). They asked whether the institutions introduced changes in their course programs in the last 3 years, and whether these changes were induced by the assessment or not. They found that about 65% of the course programs reported changes in the period, half of which attributed directly to the national assessment. In general, private institutions reacted more to the assessment than public ones, but the difference is not large – 38–30% of all the course programs in the sample. Not surprisingly, the most frequent innovation was to prepare the students to take the exam, followed by changes in pedagogical and teaching practices of different kinds (Table 15.2). Changes involving investments, infrastructure, and salary raises were much less frequent.

Table 15.2 Main changes in administration of courses induced by the National Assessment of Courses

The few studies that exist on the socioeconomic characteristics of the students, with the information produced by the exams socioeconomic questionnaire, provide very interesting information, some of it unexpected (Meneguello et al. 2000). In general, achievement has to do much more with the characteristics of the institutions than with the characteristics of the students, and the correlations between socioeconomic status and achievement are not high. Part of the reason is that course programs in the private sector tend to be of lower quality than those in the public sector, but students in the private sector come from families with higher income than those in public institutions. The other reason is that, once the students are able to reach higher education, they have already overcome most of the disadvantages that would usually affect their academic performance. There are, however, important differences in careers choice: the parents of more than half of the students in journalism, law, engineering, and medicine have a higher education degree against less than 20% for those in teaching careers: mathematics, language, and pedagogy, where the percentage is under 10%. There are some differences among public and private institutions, but they are much less significant than those among careers. Finally, detailed regression analysis confirms that achievement depends, above all, on whether the student is in public or in private institutions, and on factors like age, knowledge of English, hours dedicated to study, work, and whether the student attended public or private secondary education (with best results for those coming from private schools).

One of the most important contributions of the National Assessment, not readily documented but very clear in the minds of those responsible for its implementation, was the opportunity it provided for course coordinators, academics, and professional associations to come together in a continuous process of discussion and negotiation about the quality standards of their respective fields. Beyond the efforts of many institutions to “learn the tricks” of the exam to get better grades, there are many stories of institutions looking for help to improve their courses, and others closing down because of lack of student demand.

Opposition and Criticism

From the onset, the National Assessment was received with strong opposition from the National Students Union (UNE) and some public universities. The Student Union asked the students to boycott the exam, and tried to disrupt its implementation. Both the Student Union and the Federal University of Rio de Janeiro went to Court trying to stop the assessment from taking place. The students argued, among other things, that the assessment would hurt the students from the institutions receiving lower ratings. The eventual shortcomings, however, were not the responsibility of the students, but of their institutions, or the government, which did not provide the institutions with the support they needed. The arguments coming from public universities were similar. If they did not perform well, it was because they were not getting the necessary support, and should not be punished for that. There were other criticisms, from general statements about the impossibility to measure and quantify quality, to a principled stand against establishing comparisons and competition among institutions and students, bringing a market mentality to the realm of culture and education.

These criticisms have to be placed in the Brazilian political context of the time. Both the National Student Union and the higher education teachers’ association were in the opposition to the Fernando Henrique Cardoso government, and strong critics of whatever initiative came from the Ministry of Education, for good or bad reasons. Cardoso, a renowned sociologist and former professor at the University of São Paulo, had a history of strong opposition to the Brazilian military regime that lasted until 1985, and was elected President in 1994 after being able, as Brazil’s economic minister, to bring the country’s inflation under control. His mandate, which lasted until 2002, was characterized by very significant efforts to bring order to the economy and reduce the runaway expenses of the public sector. It was a period of economic stagnation, his government was accused of obeying the neoliberal orientations of the International Monetary Fund (IMF), and one of the strongholds of the opposition was the organized civil servant unions, including those working in public universities.¹⁰

Partisan reasons aside, several criticisms to the assessment are reasonable. The adoption of a single, unified exam in each field for all course programs in the country led all the institutions to adjust to the same mold, and may have thwarted their freedom to experiment and to diversify. By selecting a group of specialists to write up the exam, the Ministry made the particular biases of this group the national standard. This policy was coherent with the traditional view that all higher education course programs in a given field should provide the same contents and equivalent professional certifications. In areas with well-established academic and professional paradigms, this is not controversial; but this is the exception, rather than the rule, in a highly differentiated mass higher education system, with different types of students, institutions, and visions about what the contents of higher education should be.

The decision to make public the place of each course program in a five-point scale, based on the distribution of results, was related to a conscious decision not to establish clear references, or cutting points, in relation to which a given course program could be considered acceptable or not acceptable. So, in a field where all course programs are of very good quality, 12% of them or so would be ranked as “E”, while in another, where all course programs are bad, 12% would receive an “A”¹¹. In other words, all courses are ranked by uniform criteria, and the public is informed about their relative position in the rank, but not if they are of good quality or substandard. The reason for this was never spelled out very clearly, but it is not difficult to understand. The establishment of cutting points would be very controversial, and the official information that many, perhaps the majority, of the course programs in many fields are substandard – a very likely result – would create a crisis situation the Ministry could not possibly handle.

A third criticism is that the assessment may be measuring the cultural capital the students bring to the university, rather than the education value added to them by their courses. Prestigious institutions attracting very good students would have good results even if the courses were bad; hardworking and dedicated institutions accepting students with poor backgrounds would not be able to get higher marks, regardless of their effort. It would be possible to estimate the value added by the courses by taking into account the student’s achievements on their entrance examinations to the university, or their achievements in a another national voluntary test, applied to students at the end of secondary school. A statistical analysis using information from the student’s university entrance examinations in the state of Minas Gerais shows that, indeed, previous conditions affect the final outcome, but that, in general, this information would not change the final rankings in the national assessment, except in a few isolated cases (Soares et al. 2001).

A fourth criticism was that, by looking only at the student’s results, without considering other variables related to the academic staff, installations, computer facilities, library resources, and so on, the National Assessment was not an complete assessment instrument, but at most a partial one. In fact, together with the Exam, the Ministry of Education developed another assessment procedure of these input variables with heavy weight given to the academic degrees of the faculty (the percentage holding doctoral and master degrees) and the percentage with full-time contracts, plus an assessment of their physical installations and their pedagogical project, if any. Initially, the Ministry ranked the course programs according to a combination of these instruments. The information on inputs is necessary and useful, but there are good reasons not to combine input and output effects in the same scale, since it is important to know, for instance, which inputs are more effective than others in producing the outcomes. Besides, most lecturers in public institutions are nominally full-time, while most in the private sector are not, and this introduced a bias in favor of public against private institutions.

The Commission established by the Ministry of Education in 2003 to propose a new national assessment system for higher education in Brazil presented a detailed criticism of NEC, and suggested a different path. Some of the criticism was technical, like the ones mentioned above, related to the lack of clear standards and the measurement of the education value added to the courses, and the lack of comparability of results through time. Others were more political and ideological, like the statement that the exams responded to “motivations coming from outside, rather than inside the institutions, leading to isolated distorted and wrong representation of the academic world”, or that “its rationality was much more market-oriented (“mercadológica”) and regulatory than academic and pedagogic.” Other criticisms, finally, were related to the growing cost of the assessments. According to the report, the current costs are likely to grow, as higher education expands and new fields and disciplines were included in the assessment (Comissão Especial da Avaliação da Educação Superior 2003).

ENADE – The New Higher Education Assessment

With the change of government in early 2003, the original team responsible for the establishment and implementation of the National Assessment within the Ministry of Education was disbanded, and most of the institutional memory and experiences accumulated in recent years was lost. In 2003, the Ministry of Education still went ahead with the implementation of ENC, following the standing legislation, but without carrying on the usual procedures of analyzing the results with the participation of the academic committees. In December of 2003, at last, the government issued a “provisional act” (Brasil Presidencia da República 2003)¹² changing the legislation regulating the whole higher education assessment system, while the Ministry of Education issued another document spelling out how it intended to proceed. The provisional act created a new system for the assessment of higher education, based on two new National Commissions, one to provide guidelines and another to implement the new procedures. The members of both institutions are to be nominated by the government, the first among persons with recognized competencies and representatives of the “organized civil society” of students, teaching and administrative staff, and the second among civil servants from the Ministry of Education.

The new system is supposed to rank the “institutional quality” of higher education establishments on three levels, satisfactory, regular, and not satisfactory. The five-level ranking system disappears, and the new ranking will combine the results of four different assessments: institutional capabilities, teaching, knowledge production processes (presumably research), and social responsibility. In another document, the Ministry of Education spelled out the broad outlines of the new assessment it expects to undertake (Ministério da Educacão 2003). There will be an “Index of Development of Higher Education” (inspired in the Index of Human Development of the United Nations Development Program) which will combine the results of the four assessments. The National Assessment of Courses remains, with a different name, to assess the learning process. But, instead of a yearly universal assessment of all graduating students and course programs in specific fields, the assessments will be done now every 3 years through sampling procedures. And, instead of just one assessment, there will be two, one at the beginning, the other at the end of the course program.

Apparently, the new system was more comprehensive than the previous one, and more friendly to the institutions being evaluated. In practice, it was too ambitious, and impossible to implement. The former Minister of Education, Paulo Renato de Souza, in a press conference, indicated some potential problems, stating that, in practice, the government was shutting the assessment system down, and other observers also raised questions (Souza 2003). By making the participation in the assessment voluntary for the students, the Ministry would not be able to get them to participate; the proposed sampling procedures were not spelled out; it was not clear how the assessment of a sample of course programs could be combined with the assessment of institutions; it was not clear whether the assessments to be published would refer to course programs or to institutions as a whole, which seems to be the case; the new legislation bypasses the National Council of Education; and the new evaluation committees are likely to represent the existing unions of students, lecturers, and civil servants, rather than the country’s existing academic and professional communities. Finally, by combining the results of the assessment of outputs with three other assessments, supposedly with the same weight, the new procedure was likely to obscure, for society, the main information it wants, the quality of the education provided in specific course programs, which may vary widely within the same institution. In spite of these criticisms, the new system was presented as an important improvement over the past.

The new assessment, now called the National Exam for the Assessment of Students – ENADE, was carried on in the years of 2004, 2005, and 2006. The 2005 results are available in the website of the Ministry of Education for each institution and as a very comprehensive technical report (SINAES – Sistema Nacional de Avaliação da Educação Superior 2006). The full availability of results was part of a new policy of the Brazilian Institute for Education Research, INEP, to provide detailed information on assessment results at all levels, to allow for comparisons and stimulate competition for quality, an important departure from the previous stand of the Ministry of Education against quantitative assessments and comparisons.¹³ Another important function of this policy is to allow for a critical assessment, and perhaps improvement, of the assessment instruments in place. Looking at the data and the methodological explanations published in the official documents, it is possible to say that, so far at least, the new assessment for higher education did not overcome the flaws pointed out by observers when it was announced (Schwartzman 2005; Verhine et al. 2006).

The first flaw was the adoption of sampling, instead of universal coverage, for course programs with more than 20 graduating students. The justification was that it would be cheaper. However, the sample fraction of ENAD 2005 was about 50%. The additional cost of applying the test to all students in the same class in course program is probably not higher than the procedures of selecting one in two randomly; in any case, it is impossible to know, since no information is available on costs. One problem with the sampling is the possibility of selection bias, with only the best students showing up for the test. The Ministry of Education states that participation in ENADE is mandatory for the selected students, as a condition for receiving their degrees, but presents no information about the number of selected students who did not participate, and what happened with them. Two of the leading universities in Brazil, the Universidade de São Paulo and the Universidade e Campinas, both state institutions, refused to participate in ENADE, arguing that they have their own assessment system, and it is doubtful that the Ministry of Education will sanction them or their students.

One criticism of ENC was that it did not measure the added value of education in the course programs, since it did not take into account the student levels of competence at the start. ENADE tried to overcome this limitation by applying the same tests for students both in the first and last year of their courses. It also developed a new test to measure the “general competence” of students, to be applied side by side with the specific assessments of each field. However, instead of using these tests to measure the value added, considering the differences, they decided to add them in one composite measure, to establish the final grade of the course program in a five-point scale. This final grade is a combination of the results of the graduating students in the specific test, weighting 60%; the results of the entering students in the same test, weighting 15%; and the combined results of entering and graduating students in the general test, weighting 25%. The rationale for this strange procedure is nowhere to be found, but the consequence is that it makes the final result more biased in favor of institutions that get the best students in the first place.¹⁴ The consequence of this procedure was to raise the scores of the public institutions, which usually get the best students, even if they did not add much to their previous knowledge. A comparison between the results of ENADE 2004 and ENC 2003 for medical schools (Table 15.3) showed that their correlation was small (r ² = 0.16), and that ENADE’s results tend to be fairly homogeneous and much higher than those of ENC, which may please the institutions, but does not make the assessments more reliable.

Table 15.3 Results of ENC 2003 and ENAD 2004, 78 medical schools

Another problem is the validity of the tests. It is impossible to know what the test of “general capabilities” actually measures. In the legislation, and in the official document, it says that the test should measure “how far the student is being educated as a professional who behaves ethically, is competent and committed with the society in which he lives. The test should also measure the student’s ability to analyze, synthesize, deduct, develop hypothesis, establish relations, make comparisons, detect contradictions and organize their ideas” (SINAES – Sistema Nacional de Avaliação da Educação Superior 2006, p. 12). However, the test is made up of just ten questions, of which seven are multiple-choice, and three require written answers. It is clearly impossible to measure all the expected dimensions with so few questions, and more so without very strong validation procedures, which did not exist. The validation of the tests for the specific areas did not exist either, except for the item’s correlation and discrimination, a flaw that ENADE shares with the previous ENAC.

The 2005 report presents an extensive and laudable effort by INEP’s statisticians to analyze the results in different ways, combining the test results with the answers to a large socioeconomic questionnaire the students also had to fill in. One of the most intriguing efforts was the development of a measure comparing the expected and actual achievement of the course programs, based on the general characteristics of the entering students. However, the meaning of the figures presented is impossible to ascertain. For instance, in a scale of 100 points (presumably before standardization), students in physics gained 1 point in the scale of general capabilities, while students in computer science gained 5.9 points; in the scale of specific capabilities, the gains were 7 and 10.4, respectively; but there is no way to know what these differences actually mean.

There are other conceptual and technical flaws with ENADE, but these are enough to draw the general picture. The Ministry of Education presents ENADE as just one component of a larger system of assessment, which should not be considered in isolation, and does not use nor recommend its use for policy purposes, or for the students to choose the places where they will study; but the results of the other components of the assessment are still to be published, and it is not clear how they will be used. (An obvious use of this assessment would be to select the institutions that receive students which benefit from “University for All”, a government program to provide free higher education in private institutions for poor students). For all these reasons, the publication of ENADE results has been received with much less interest than what used to happen with the publication of the results of ENC.

The Future

The Brazilian National Course exam, in its two incarnations, is a unique and extraordinary experience, which has generated admiration and interest in higher education circles in different parts of the world, and received strong support in the Brazilian public opinion. Its future, however, is uncertain. On hindsight, it is possible to say that the main weakness of the original National Course Exam was its lack of proper institutionalization, and the absence of a clear sense of ownership of the exam within Brazil’s higher education and professional communities. The Exam started as a personal initiative of the Minister of Education, Paulo Renato de Souza (an economist who had been the rector of the University of Campinas and a high-ranking officer of the Inter American Development Bank) who had to start by convincing his own staff of its need, and implementation given to the most flexible and independent branch of the Ministry of Education, the National Institute for Education Research, INEP. In principle, other institutions could have taken this task – the Secretary for Higher Education within the Ministry, the National Council of Education, the National Conference of Rectors – and a new institution could have been created with this purpose, like the National Commission for the Assessment and Accreditation of Universities in Argentina.¹⁵

Had the Minister decided to work through one of these institutions, subject to all kinds of interest groups and administrative hurdles, or to create a new one, he might not have succeeded in moving so rapidly, and achieving so many significant results in such a short time. Acting on the power of his cabinet and thanks to his personal prestige, it was possible to move quickly through the complex legislative process to get the legal authorization, and to place the necessary human and financial resources in the hands of the able head of INEP, Maria Helena Guimarães Castro.

The price, however, was that no institution or segment of the academic community claimed ownership of the Assessment, except a small team within the Ministry of Education. Hundreds of academics were asked to participate in the Commissions and probably did a very important work, but they were there by the Minister’s invitation. The statistical data generated by the exams remained under the Ministry’s control; some qualified researchers and research centers were invited to analyze them, but they were not made publicly available to the academic community of education research specialists. In the effort to keep up with the complex procedures established for the Assessment, most of the energy of INEP’s staff was dedicated to the preparation of technical documents and other materials for the Commissions, the students and the course program coordinators, with little left for the deeper reflection on the general important and significance of the Assessment. To conquer public opinion, the Minister had a competent public relation staff, which kept the press well informed of the achievements of the Assessment, and helped to win the battle of the public opinion against the organized opposition

Without clear ownership in society, established as just one activity within one sector of the Ministry of Education, the National Assessment did not have the strength to resist the impact of a change in administration. The new Evaluation Commission established by the government after 2002 to implement the new assessment system could have been a step in the right direction, if these Commissions could become truly independent and autonomous from vested interests. This did not happen, however, given their membership: representatives of the “organized society” – unions of students, lecturers, and university employees, known to have been the strongest opponents of the National Assessment of Courses as it existed, and political appointees in the Ministry of Education. The new Commission did not have the strength to abolish the national exam of courses altogether, but was effective in reducing their strength as a reliable information for the public and an instrument for quality assurance of Brazilian higher education.

To become a stable and significant feature of Brazilian higher education, the Assessment would have to find a permanent institutional house, which can be neither the Ministry of Education nor the unions and corporations with vested interests against any kind of external assessment of their own work. Between these two extremes, a proper space will have to be found, if the experience of recent years is not to be lost.