TES – From Impressionism to Expressionism

Abstract

A speaker addressing the final session of a conference risks being clumsily repetitive, for two reasons: on the one hand, because of a natural tendency to ride old hobby horses, but more especially, because much of the ground already covered by previous speakers is likely to be repeated. Whilst I make no apology for the first (I have noticed – and perhaps this is more evident in the political life of our country – that the simulacrum of credibility is definable by the regularity with which people uphold their positions), in the case of the second, I can only invite listeners to appreciate the convergence of opinions that has developed over time among many observers of the Italian experience in the area of student-focused Teaching Evaluation Surveys (TES). This convergence appears to be reached, even allowing for the diversity in stylistic expression and depth of feeling, in proportion to the intensity and directness of exposure to the subject matter, and the strength of the documentation and scientific arguments.

1 Foreword: Excusatio Non Petita

A speaker addressing the final session of a conference risks being clumsily repetitive, for two reasons: on the one hand, because of a natural tendency to ride old hobby horses, but more especially, because much of the ground already covered by previous speakers is likely to be repeated. Whilst I make no apology for the first (I have noticed – and perhaps this is more evident in the political life of our country – that the simulacrum of credibility is definable by the regularity with which people uphold their positions), in the case of the second, I can only invite listeners to appreciate the convergence of opinions that has developed over time among many observers of the Italian experience in the area of student-focused Teaching Evaluation Surveys (TES). This convergence appears to be reached, even allowing for the diversity in stylistic expression and depth of feeling, in proportion to the intensity and directness of exposure to the subject matter, and the strength of the documentation and scientific arguments.

Before proceeding, I must make two more short points: firstly, the personal commitments of my recent professional life have been rather institutional, that influences my approach to the subject. My aim is to develop arguments on the cultural and political use of TES, largely ignoring the methodological and scientific methods by which it should be driven; on the other hand, I feel I should explain the title of this talk: those who know me will be familiar with my habit (sometimes paroxysmal or inapt) of exploiting the ample opportunities for expression afforded by the Italian language: likewise on this occasion, at the time of responding to the request of the organizers, I gave in to temptation; I must confess to have dared, although in overcoming the difficulties I had some help from a vocabulary of the Italian language (more exactly: A. Gabrielli, Il grande italiano 2008, Hoepli 2007¹); this gives me access to two definitions which, with a degree of licence, can be seen as consistent with the considerations I intend to present.

Impressionism: “representation of a reality through images that are immediate, detached, selected according to the impression of the moment”. Expressionism: “tendency characterized by a predilection for the more dramatic and intense aspects of the human experience, represented through language that is dramatic, full of conflict and unsettling”.

Now, in the first definition I want to focus attention on the words in italics, which would seem to sum up my critical appraisal of what TES has tended to be thus far: immediate – swiftly formulated and concluded, lacking solid theoretical constructs; detached – occasional, ephemeral, not homogeneous in space and in time; according to the impression of the moment, fragmented, without any process of accumulation and growth. The transition to an expressionist state, by contrast, must be worrying, not only for those who design and conduct the TES – an aspect with which everyone here will be fully and practically familiar – but also for those who may be affected according to their tasks by the results and invited, implicitly or explicitly, to adopt a different approach in their work. And with this end in view, it needs to be dramatic and to highlight the conflicts deriving from the great diversities of the domains in which it operates – thematic and territorial – also from the natural yet stimulating clashes of interpretation still possible (despite being no longer in statu nascendi).

After this lengthy introduction, we need something about the structure of my address, which is in four parts: the first is dedicated to a brief but necessary destruens consideration, keeping in mind the many invaluable contributions in which solid criticisms have been launched to the several factors of TES; secondly I shall then try to define principles and coordinates for a construens hypothesis outlining a more advanced and meaningful approach to the survey; thereafter, there will be an acrobatic attempt to construct a Utopian model intended purely as a goal (a far-off goal …); and finally, returning to the real world and more especially to the principle of progressive refinement, I will suggest a few feasible ideas for improving the current structure of the survey.

2 Pars Destruens: Accusatio Manifesta

As already anticipated, I will offer just a few words on this aspect of the question, obviously so as to minimize any repetition of what has already been said during the two days of the conference. Accordingly, my opinion can be summed up in four adjectives: notwithstanding its 10-year existence, the survey continues to be impertinent, unsuitable, inconsequential, and inefficient.

Whilst in the Italian language, impertinent can also mean “not pertinent”, I use the word here in its everyday sense of impudence and disrespect: the survey has done no more than menace the conformism of tradition, making noise as a naughty boy, a touch rebellious but with no real and firm intention of upsetting the tables and changing the rules; indeed there is even a quirky affection for the survey, though mixed with indifference proportional to the complexity (sometimes abstruseness) of the construction and analysis models that have been employed; there have been no actual refusals or calls for abolition – not institutionally at least, albeit beneath the surface many colleagues might wish it so. For threats to the institutions and the pax academica, and strategic plans of action, one must look elsewhere. The question to pose is, would it be possible, and under what conditions, to apply the other meaning of pertinent: relevant and useful ².

On the accusation of unsuitability I must dwell slightly longer, as this is linked closely to the concept of usefulness; there are three levels of reasoning to express our opinion:

• I. since the mandatory inception of TES, its purposes were not made clear; in effect, there are many objectives that could have been identified, and from these, the setup and content of the survey could have been delineated more clearly and explicitly: quite simply, does one measure satisfaction, or the sense of responsibility displayed by teachers, or the dependability of organizational processes, or the gap between student expectations and effective attainments, or does one assess the overall quality of courses and tuition, or gauge the relationship between value of teaching and other system evaluation components, or indeed all of these? In literature, conventionally, the technique and structure of evaluation are not armed and deployed without first establishing the battleground and the field of conquest; the impression is that the universities have armed themselves with a sabre and a blindfold: the swordsman can choose to strike low and blind, or simply wander around proudly armed but essentially harmless;

• II. the design of the survey was determined locally: the organization, the responsibility, the timing, the nature of the indicators provided (apart from a small set proposed belatedly by CNVSU), and the method of dissemination are established by each local centre, effectively disallowing comparison of the results. Instead, we are convinced there can be no competition – teaching and scientific – when the documentation submitted for comparison and selection is put together on the basis of self-determined rules;

• III. there were no “hot to use results” rules, either at national level or in general even at local level; neither was there any explicit mention of the areas of academic life targeted for change.

These first two considerations already provide grounds on which to justify the inconsequence, which can also be supported by previous research conducted as part of the Dottor Di.Va.Go project,³ as well as by the single experiences of almost all universities; in particular, it will be remembered that the results of the survey:

• I. are not widely disseminated, indeed officially (by resolutions of the Academic Bodies) their circulation is limited (normally to the Deans only, who are invited – though not bound – to use the findings as they wish);

• II. are rarely the subject of comparison and public discussion, whether of the method or of the indications given by the findings;

• III. are not translated into decisions for change in respect of the failings reported.

All these factors have their origin, firstly, in the attitude of most faculties, secondly in that of the universities themselves, and to a lesser extent in the conduct of the ministry, which nonetheless was in favour of the survey and stands to gain from it.

Finally, I believe it can be stated that in view of the various limits mentioned, the survey is also inefficient: in fact it is costly, whatever the basis for the design of the survey adopted in various situations; almost as a direct consequence, there are no cost-benefit analysis studies on its implementation, which could easily lead to abandon the whole survey; it is to some extent a source of distorted information, due principally to its methodological weaknesses (survey conducted only on occasional attendees, missing the opinions of greatest interest, that is to say those who have elected not to attend lessons or lectures; neglect of the “consistency of teaching/verification of learning” equation; etc.); the tired and mechanical repetition of the ritual over time has transformed the survey into a purely bureaucratic exercise, which among other things generates a tendency to become more and more careless or to fill in questionnaires unthinkingly or derisively. To conclude on this aspect, a simple aphorism: the social researcher knows that a survey perceived as statistically harassing (due to its intrusiveness, and especially to its uselessness) will doubtless give results that are statistically (and cognitively) embarrassing.

3 Pars Costruens: Non nova, Sed Nove

3.1 Guiding Principles

Before moving into this dimension of this chapter, which in many ways is based on subjective perceptions and Icarus-like attempts⁴ – on wings of insubstantial feathers attached insecurely to a fragile frame – I ought to mention the principles by which it is inspired. The first principle compasses two related affirmations: (a) whilst accepting the autonomy attributed to the system and to the single universities, as regards the logic and importance of evaluation, there needs to be a uniform approach to implementation of the TES; (b) this derives from the belief that autonomy in the area of evaluation (and perhaps not only in this area) should be the prerogative of the university system overall, rather than the individual university. These two opinions are comparatively strong, and while not original, may not easily be shared. What is more, they originate from the assumption (purely political, ideological, and cultural in nature) that it is the academic system that must share the duty of responding collegially to the expectations of the nation. Consequently it has to ensure it possesses the right tools and the most rigorous procedures, so that it can be judged on the basis of fully and correctly ordered arguments, shunning solipsistic whims, expedients, obfuscations. The reasons underlying this principle are quite trivial:

• I. evaluation is worthwhile only if it produces comparative capacity adopting homogeneous plans of analysis;

• II. homogeneous methods should be a guarantee to give to the survey a wider political significance not easily opposed by objectors, whether secretly or openly;

• III. “true” rules and conditions of comparison generate a greater sense of responsibility both in the managers of the survey and in users of the findings that emerge from it;

• IV. finally, this is the one condition capable of legitimizing national policies that are not necessarily or exclusively punitive. In effect, the teaching evaluation approach would be the one to favour, designed as it is to reward positive trends, changes of direction and effective emulations that could come about over time.

It should not be forgotten that from the outset of the initiative, there was real focus on the idea of uniformity at national level, as witnessed by three documents (the first two from the National Committee and the third from the Body that replaced it, namely CNVSU⁵) in which the authors seek to suggest ideas for a common approach, albeit they are concerned mainly with the content of the questionnaire, as the formulation of the questions and the number of multiple choice answers. Only in the first of these documents there are reflections on the design of the survey and on its functionality, also on the limits expected and, for various good reasons, accepted. To move in the direction indicated, however, common rules would need to be imposed in many aspects of the survey’s design. This, however, is not the place to offer exact solutions, and moreover, it must be remembered that these aspects are prejudicial to the end in view. In effect, univocal answers need to be given for each one of these operational choices:

• I. to state responsibilities/tasks for the design;

• II. to state responsibilities/tasks for organization and administration;

• II. to state times, procedures and operators for the purposes of carrying out the work in the field and administering the questionnaires;

• IV. to state procedures for diffusion of the findings, in order to present results consistently with their nature, responsibility, capacity for interpretation, and expectations of different recipients;

• V. finally, agreeing on clear opportunities and forms for discussion, especially comparisons (local and national).

Just as important, and obvious in our view, is the second principle: TES is just one part – necessary but far from self-sufficient – of a complex evaluation mosaic that must appear more vivid, especially from the outside. In effect, the survey is effective only if integrated with other initiatives systematically and logically connected one with another. Looking only at the dimension of enhancing the teaching function of the universities – and therefore not, in this work, at that of evaluating their research activity and their management-administrative-accounting organization – an organic design should include coordinated surveys on:

• I. freshmen – to assess their educational, cultural and social background, their reasons for going to university, and their expectations. Knowledge of these factors is almost indispensable when seeking subsequently to interpret characteristic differences in their career paths;

• II. careers within study courses – a survey which is increasingly easier by the computer system now present in every higher education organization, capable of responding not only to the need for different analyses between faculties and universities⁶ but also of indicating the nature and timing of circumstances and factors by which the educational process can be stalled;

• III. drop-outs – including consideration of the conditions, timings and reasons, and the possible prospects for social and/or professional advancement that may have forced or prompted the decision to quit;

• IV. undergraduates – to gather general assessments on the life and learning experience during the study period, considering the whole university experience, trying to keep apart single events, in order to indicate/highlight the value and limit of each career phase;

• V. the social and occupational destiny of graduates – information needed, obviously, for a better insight into the effectiveness of the system⁷;

• VI. the reactions from stakeholders – with regard to the quality, pertinence and potential of the training received, with a view to its usefulness that should be more than contingent or quickly obsolescent;

• VII. with a complementary function deriving from the principle that, for completeness, each evaluation must collect the views of all players in a process or a service, as regards the attitudes, expectations and behaviors of the teaching staff, and offer the facility of measuring the distance of opinions and goals assigned by each active component;

• VIII. finally, and with the same objective as the previous point, on the reactions of all the Governing Bodies (University, Faculty, Study Course) to the findings of the survey, with special attention given to the methods and mechanisms of redesigning and upgrading the tools and the content of the didactics.

There is no doubt that for the first five, in particular, of these surveys envisaged as complementary to the TES, it is almost essential that studies should be conducted on an individual basis, allowing linkage of the information relative to each student, since this is the ideal condition for giving consistency and systematicity to an organic evaluation process. We think that, in effect, even studies on aggregate data alone – relative to single survey – can provide particularly interesting elements for analysis. The aim must be to ensure that the various surveys dialogue effectively, on the same subjects and on events and considerations that are different yet integratable.

Before moving on in detail to new proposals, it will be noted that certain of the surveys listed above are already in use at many universities, although one has the impression that they are often implemented in parallel, with no stated aim of aiming them toward a single goal, and that just as often they are entrusted to different administrators, all jealously guarding their respective areas of independence and unwilling to see their labors coordinated – whether on design, concepts, techniques and methods of delivery, or scheduling, or necessary linkages – within the scope of an Overall System Evaluation Plan.

With these two basic principles in mind, accordingly, one can reasonably identify the nature of the product that ought to emerge from the evaluation process, precisely establishing (1) the object/s of the survey; (2) the operating conditions conducive to its implementation; (3) the essential references that can define it, while at the same time stating the inherent advantages and limits. As regards the first element, I feel we have to accept a long-term goal, still too far off to be achieved by all the universities, which stems from the desirability of setting out an integrated and entirely coherent analysis for each study course; a transitional solution might be provided by the temporary need to render the analysis of each faculty more important and pertinent, while able nonetheless to allow for the effects of variability between the study courses offered. As regards the second element, it will be perhaps useful to state some imperative aspects more forcefully; in order to make the analysis of the selected object incisive and meaningful, it is essential:

• I. to adopt the traditional quantitative approach connected with the use of the sources and the surveys conducted, although accomplishing a correct coordination of the single components, which currently are working separately;

• II. to accept and develop the qualitative approach, integrating it with point I. and taking into consideration written materials,⁸ in-depth interviews, consultation of qualified witnesses, reports of debates and comparisons drawn on the findings of the evaluation process;

• III. to make mandatory the production of an annual self-assessment report for the structure in question, the content of which to be prescribed in an explicit and ordered fashion⁹;

• IV. to favour the notion of peer reviews, not least to enable discussion of didactics approaches and the relative operating mechanisms, and the respective results;

• V. to be aware of the need for access to substantial investments – intellectual as well as financial and material;

• VI. to be equally aware of the need to maintain centralized responsibility for the definition of evaluation procedures, and likewise for testing of the final product.

In the case of the third element, it will be as well to move on directly to the next paragraph.

3.2 The Proposal: A First, Almost Utopian Design

Following the stated principles and taking into account the conditions just mentioned, the methodological criteria that should be adopted in an ideal design, careful to the method but also to cognitive/cultural aspects, could in my opinion be these:

• I. to get longitudinal data (linking the information provided by each student census from enrollment to graduation, and possibly post-graduation: this means, firstly, coordinating surveys conducted currently in different ways¹⁰); a design integrating different surveys would, amongst other things, avoid the need to ask repeatedly for the same information and could encourage students to participate more willingly¹¹;

• II. to conduct a statistical survey, either on students or on events characterizing the university experience; to this aim, in our view, a decidedly novel strategy for the TES might be to conduct the survey just once a year, typically during the period of transition from one academic year to the next. In this way it is possible to gather information relating to attendance at all the lessons, participation in final exams, passes or failures, so that the census would gain the perspective of differential approach in its analyses, according to the specific behavior of the single student. On one hand this proposal is cost-reducing for the administrative aspects regarding the questionnaire collection procedure, and, on the other hand it would also guarantee opportunities to examine the differential capacities for judgment shown by students with regard to the courses¹²;

• III. to ensure the involvement of students not only by giving notice of the survey but above all by seeking suitable forms for feedback of the results and for linked comparison between stakeholders;

• IV to keep the survey under a central management structure, thereby ensuring procedural correctness, uniformity of conduct, and comparability of final findings;

• V. to promote “official survey days” with the aim of emphasizing the importance attributed “officially” to TES initiatives;

• VI. to promote and standardize Faculty and University days, open to the public, for presentation and discussion of the analyses and the reports with a notice posted in advance;

• VII. to construct a National Evaluation Report on the basis of material generated at every university, integrating the teaching dimension with other components of academic life, as a way of demonstrating the collective sense of responsibility toward the subject and taking the initiative away from external subjects or agencies, often business or scandal oriented, which in recent years have been more inclined to indulge in épater le bourgeois than to present measured and reliable system evaluation.

Many advantages could come from this overall approach. Looking first just at some political and cultural aspects, it is possible to mention:

• I. the existence of an entity exercising general control over the process of shaping the entire evaluation activity, and in particular, ensuring balanced management and treatment of the TES data procedure;

• II. a declared and direct acceptance of responsibility by the Governing Bodies of the single academic institutions;

• III. affirmation of the principle of transparency, an indispensable element of the evaluation process;

• IV. the objective advantage of comparison based on uniform working conditions;

• V. more generally, the soundness of action policies (local and national) is supported by principles of shared responsibility and uniform methods of observation.

The sharing of all the process of the TES evaluation could provide advantages of operative and cognitive types. A good practical solution consists in just one annual survey including the entire experience¹³ of the academic year just concluded:

• I. less organizational and financial strain of the survey, hence a reduced effort in polling (and harassing) the students¹⁴;

• II gathering opinions from students who have experienced various forms of participation in learning activities: attendance at lessons or not, attendance at other forms of tuition (workshops, practical sessions, etc.) or not, taking exams or not, either passing or failing. Some cognitive value would be gained from the possibility of making up a differential analysis according to different profiles;

• III. with similar cognitive aims in view, the chance to measure the differential capacities to evaluate shown by students, according to their personalities and career choices, and to the type of the teaching activities surveyed¹⁵;

• IV. the guarantee of being able to conduct differential analyses, along time (comparison between years) and space (between universities, faculties, similar study courses);

• V. linkage with further evaluation with the job placement data.

3.3 A Possible Design

We are fully aware that the question is delicate and that structural modifications with excessive strength could even have the effect of stalling the system now in action. We feel it will be useful to outline the minimum or preliminary conditions for a move toward a more organic design (just on the basis of my own personal ideas). Here too, I should begin by stating the principles we see as desirable: (a) we are convinced as to the importance of centralizing responsibility for the definition of rules and procedures determining organization of the eventual national evaluation programme¹⁶; (b) we discard the “all at once” approach in favor of a more relaxed (and practicable) “layer by layer” strategy; (c) we feel it is essential to encourage experimentation, both as regards the procedures, tools and techniques adopted in conducting surveys, and more especially as regards the methods of analysis used on the collected data, which should in time become demonstrably more focused and selective, producing information that will be pertinent and consequential; (d) we see it as essential to develop, on the one hand, the ability for internal and external communication not only of results but also of their effective value, which will always be sharply determined by the singular operating conditions adopted, and on the other, the need to engender osmotic interaction between institutions, so that experiences and the results of innovative trials can be shared. In short, these considerations of mine would lead ideally to acceptance of the principle of steady progress in the process of consolidating the TES, acknowledging its intrinsic weaknesses but welcoming its usefulness as an indicator for the future. To conclude, then, I feel that in the current scenario – not least given the lengthy period during which TES has been adopted and since become supinely mechanical, betraying a slow but clear loss of interest, and giving too few useful results – we are prompted to make certain initial adjustments that should focus on:

• I. standardizing the questionnaires for the different types of survey more fully and widely, and likewise the times and methods of polling, imposing common procedural rules and uniform checks¹⁷;

• II. regularizing the indicators for comparison, adopting standard methods of production;

• III. trialing survey approaches that will also collect information and judgements on the taking of exams;

• IV. widening survey aggregates to include enrollees, drop-outs, undergraduates, teaching staff and outside contacts;

• V. preferring and rewarding initiatives that envisage the integration of surveys;

• VI. illustrating the decisions taken as a result of evaluation, and thereafter, rechecking the effects of any changes introduced.

It will be appreciated that this is an almost minimalist outlook and that the primary concern is to raise the credibility of the evaluation; in effect, the ultimate purpose of TES is to be useful, even if the ways in which it is packaged cannot be regarded as optimal. This, in the absolute conviction (and hope) that the application of TES will continue to unsettle and promote conflict by virtue of the genuinely dramatic role it must play to ensure the world of higher education remains constantly in pursuit of improvement.