Keywords

1 Introduction

Throughout the last few decades, teacher participation in decision-making has emerged as a controversial issue. Critical theorists started questioning perpetuating situations in an attempt to raise teachers’ awareness and change dominating cultures in educational workplaces. Critical language testing, in particular, questioned the roles of different participants in the testing process. This critical exploratory study aims at problematizing the issue of depriving teachers from designing exit tests. It also intends to raise teachers’ awareness about this issue. My initiative to discuss this issue and therefore raise my colleagues’ awareness stems from my personal experience of exclusion. As an English language teacher developing the curriculum and designing daily quizzes and weekly tests, being deprived of being involved in designing exit tests is an issue that should be raised and discussed. To my knowledge, except for one study (Troudi, Coombe, & Al-Hamly, 2009), no published research on teachers’ exclusion from designing exit tests has been conducted in this region. I thought that questioning perpetuated situations and raising others’ awareness about similar experiences can lead to a change in the dominating culture of my workplace. To discuss this topic, the first part of this paper will deal with the theoretical background. It will be followed by sections on methodology, results, and discussion respectively. The conclusion will focus on the limitations of the study and recommendations for future research.

2 Theoretical Background

2.1 Critical Applied Linguistics and the Problematizing Stance

Questioning teachers’ exclusion from assessing students cannot be understood in depth without situating teachers’ exclusion under the umbrella of evaluation in general and Critical Language Testing (henceforth CLT) in particular.

According to Pennycook (2010, p. 16.3), “critical applied linguistics is more than just a critical dimension added on to applied linguistics: It involves a constant scepticism, a restive problematization of the givens of applied linguistics, and presents a way of doing applied linguistics that seek to connect it to questions of gender, class, sexuality, race, ethnicity, culture, identity, politics, ideology, and discourse.” Influenced by the Marxist theory of class struggle, critical applied linguistics tackles the previous concerns to unveil dominant hegemonies, ways of perpetuating existing relations and interests, and seeks to change power relations. Pennycook (2001) raised a number of concerns to be able to reach a better understanding of critical applied linguistics. According to him, applied linguistics, praxis, being critical, micro and macro relationships, critical social inquiry, critical theory, problematizing givens, self-reflexivity, preferred features, and heterosis are the concerns of critical applied linguistics. He looks at applied linguistics in all its “contents as a constant reciprocal relation between theory and practice” (ibid, p. 3). He argues that critical applied linguistics is a “way of thinking and doing” and contends that one of the challenges for critical applied linguistics is to find ways of mapping the micro and macro relationships. It is fundamental to understand the relations between the concepts of society, ideology, education, and classroom conversation discourses and second language acquisition. Therefore, critical applied linguistics should not only highlight the relationship between language contents and social contexts but should also view these relationships as problematic. Critical applied linguistics should go beyond exploring correlational relationships between language and society and raise questions of power, access, desire, and resistance. This is to assume that critical work with a sceptical eye can help to raise such questions as it engages with problems of inequality, injustice, and human rights. The ability to raise questions and be critical cannot be achieved without questioning the givens and asking critically about their assumptions. Pennycook (ibid, p. 7) holds that “a critical component of critical work is always turning a sceptical eye toward assumptions, ideas that have become naturalized, notions that are no longer questioned.” In addition to concerns, Pennycook gave an overview of domains comprising critical applied linguistics. They are critical discourse analysis and critical literacy, critical approaches to translation, language teaching, language planning and language rights, language literacy and workplace settings. Language testing is another domain of critical applied linguistics, especially considering Shohamy’s view that language testing follows a political agenda. She argues that language tests were used as “triggers and vehicles through which bureaucratic agendas could be achieved” (Shohamy, 1997, p. 346).

2.2 Critical Language Testing

Shohamy (2001) discusses the main features of Critical Language Testing. She claims that CLT assumes that the act of language testing is not neutral as it is the product and agent of political, educational, social, and ideological agendas that determine the life and future of the different test stakeholders. She adds that CLT views test takers as political subjects and encourages them to criticize and critique the value inherent in tests as they are embedded in educational, cultural, and political contexts. Moreover, CLT asks about the agendas behind implementing tests. It also asks questions about “what knowledge tests are based on (…) is it something that can be negotiated, challenged, and appropriated?” (ibid, p. 132). Adding to that, CLT challenges the psychometric traditions of language testing and advocates interpretive ones “whereby different meanings and interpretations are considered for test scores” (ibid, p. 132). CLT challenges the reliance on tests as the only and sole instrument of assessment and suggests the use of other assessment tools and procedures to gauge and interpret the knowledge of learners. Furthermore, CLT admits that the knowledge of testers is not comprehensive and that there is a need to rely on other sources to obtain a more solid, valid, and accurate description and interpretation of knowledge. Furthermore, CLT examines the stakeholders of tests and asks about the parties involved in designing and producing tests. It calls for a more democratic process where different stakeholders including policy makers, test writers, students, parents, and teachers are involved in designing tests.

In addition to features, she highlights the powerful uses of tests. She argues that decision makers attribute importance to tests as they “allow those in authority to control and manipulate knowledge” (Shohamy, 2001, p. 38). She emphasizes that, rarely challenged, tests serve the needs of those in power to perpetuate their dominance to enforce policies. In her critical observation, she expresses concerns about “the power of tests and their uses in society” (ibid, p. 5) and draws our attention to the fact that the voices of test takers are silent, that tests have detrimental effects on test takers, and that tests are used as disciplinary tools. She contends that using tests as disciplinary tools is “an extension of the manipulation of tests by those in authority-policy makers, principals and teachers-into effective instruments for policy making.” This issue of using tests as disciplinary tools is also discussed by Foucault (1979, p. 184) who states that “at the heart of procedure of disciplines, it manifests the subjection of those who are perceived as objects and the objectification of those who are subjected.” Parallel to this, McNamara discusses the issue of tests being used as weapons of policy reform and immigration policy, claiming that “language tests have a long history of use as instruments of social and cultural exclusion” (McNamara, 2000, p. 68).

McNamara (ibid, p. 76) states that the principles and practices of testing that have “become established as common sense or common knowledge are actually ideologically loaded to favour those in power.” He adds that critical language testing “is best understood as an intellectual project to expose the role of tests in this exercise of power.” Similarly, Shohamy (2001, p. 131) connects between the use of tests and power and justifies placing the domain of testing within the broad area of critical pedagogy by stating that critical testing “implies the need to develop critical strategies to examine the uses and consequences of tests, to monitor their power, minimize their detrimental force, reveal their misuses and empower the test takers.” She claims that critical testing attempts to criticize the field of testing, monitor, and limit the powerful uses of tests. This criticism includes regarding the act of testing as biased and not neutral, as it shapes the lives of teachers and learners. Critical testing examines the stakeholders of tests, their agendas, testing methods, and the ideology delivered through the test. Shohamy also maintains that one of the issues that critical testing problematizes revolves around the persons included in designing tests, an issue that has been scarcely tackled in the literature but which I will try to illuminate in the subsequent section.

2.3 Teachers’ Exclusion

Problematizing teachers’ exclusion from designing exit tests has scarcely been discussed. The feelings of mistrust, marginalization, and humiliation caused by exclusion have been highlighted by Shohamy (2001) and Rea-Dickins (1997). Shohamy connected exclusion to the democratization of educational systems. For her, it all revolves around power, trust and trustworthiness. She argues that the “selection of the testing body can also provide a good indication of the extent to which the educational system trusts the teachers and is willing to grant them professional authority” (2001, p. 30). She presents the experience of introducing new reading comprehension tests and the way teachers were “humiliated by the system which viewed them as potential cheaters and untrustworthy” (ibid, p. 57) by forcing them out of their classrooms and denying them access to any information about the test. Shohamy highlights on the criticality of the effect of such exclusion on the image of teachers and wonders “about the message conveyed to students when their teachers are not trusted by the system” (ibid, p. 57). Shohamy (2005, p. 106) showed more interest in teachers’ exclusion and their subservient role when they “are viewed as bureaucrats; (…) [and] are being used by those in authority to carry out testing policies and thus become servants of the system.”

Similarly, Rea-Dickins (1997, p. 304), who defines stakeholders as “those who make decisions and those who are affected by those decisions”, relates teachers’ inclusion or exclusion to the issues of power and democratization and highlights the harms of exclusion. She argues that “in terms of obvious power, some stakeholders are more important than others: The more important ones make the decisions and take action while the less important are those affected by those decisions” (ibid, p. 305). Instead of exclusion, Rea-Dickins claims that consulting and involving the different stakeholders “in the process of test development and test use reflects a growing desire among language testers to make their own tests more ethical” (ibid, p. 304). To further discuss the problem, she asked the following question: “How much control do teachers have of the assessment procedures and the tests they administer?” (ibid. p. 307). Instead of exclusion because of incompetence, Rea-Dickins proposes appropriate preparation and empowerment of teachers as potential solutions. She elucidates that teachers’ participation, among other factors, can promote greater fairness in the testing process and advocates “democratization of assessment processes through greater stakeholder involvement” (ibid, p. 3). She argues that teachers can become competent at designing tests if they are heard and given opportunities to develop their understanding of the assessment process.

Hearing teachers’ voices was what Troudi et al. (2009) tried to highlight in their study, which is important for two main reasons. First, it is a recent research study that elicited teachers’ voices and reduced their feelings of marginalization. Second, it was conducted in the Gulf region, where this actual study is taking place. In this study conducted in the UAE and Kuwait, Troudi et al. (2009) investigated issues of assessment design and implementation in these two Gulf countries. The researchers tried to explore teachers’ assessment philosophies and their roles in student assessment. Results of the study showed that the teachers’ role in assessment is minor because of “the top-down managerial approaches to education and a concern for validity and quality assurance in large programmes” (ibid, p. 546). The researchers argue that exclusion was the recurrent theme and that many of such instances were noticed. Results also showed that teachers’ opinions were not solicited and that they were excluded from designing assessment tools because they were “perceived not to have expertise in this area” (ibid, p. 550). What is interesting in the study is the ability of researchers to present reasons of both parties to justify teachers’ exclusion from designing assessment tools. Teachers, for example, expressed how they felt distrusted and disrespected. Those who are in power, on the other hand, argue that assessment should be centralized for reasons of practicality, efficiency, and reliability. Moreover, they expressed a fear that teachers may be inclined to help their students because of these latter’s involvement in teacher evaluation.

Except for the previous study, I was not able to find any study focusing on the issue of teachers’ exclusion from designing exit tests in the Gulf region. The paucity of related research may grant importance to the actual study and may contribute to fill this gap about teachers’ exclusion from assessment decisions. It may illuminate reasons of exclusion and shed light on the justifications of the different stakeholders. Succinctly, this study aims at answering the following research question: “Can problematizing the issue of excluding teachers from designing exit tests help to raise teachers’ awareness?” In order to be able to tackle this issue, the way should be paved by answering the following sub questions: “how do classroom teachers justify their exclusion from designing exit tests?” and “how do decision makers justify teachers’ exclusion?”

3 Method

3.1 Context of the Study

This study was conducted in a technical institute in the UAE. It is a vocational institute that trains students to become future technicians in the oil and gas industry. Students’ age in the foundation program ranges from seventeen to twenty-two. Most of the students are Emirati with a very small number of Omani students.

Students usually spend three terms (levels 1, 2, and 3) in the foundation program and pass the exit level 3 tests to be able to join the technical program. Each term lasts a study period of six months. During the foundation course, English is the medium of instruction. All subjects, Math, Science, Lab, and the four English language skills, are taught in English. Students receive approximately six hours of instruction daily for five days a week. They are tested biweekly in all subjects. In the reading course, for example, level three teachers, design these biweekly tests collaboratively. The biweekly tests, which are administered at the end of every unit, include reading comprehension passages and vocabulary exercises and carry an assessment weight of 70. The remaining 30 are awarded to participation (10), vocabulary journals (10), and short comprehension quizzes (10). Similarly, level three teachers also design these short quizzes in a collaborative way. Teachers correct the daily and weekly tests and give students the opportunity to see and discuss their mistakes. By the end of the term, students should reach the cumulative average of 70 in each subject to be eligible to sit for the final exam. At the end of each term, students must pass an exit test to be promoted to the next level. These exit tests include maths, science, reading, writing, and communication skills. During the final exams, students are tested one subject per day. After exams, students consult teachers to verify some answers or enquire about certain questions.

3.2 Participants

3.2.1 First Participants: Teachers

The choice of the participants was purposeful (Creswell, 2008). I targeted the teachers of the foundation program to which I am affiliated. I started questioning the existing testing policy and examined the situation with sceptical eyes. I decided to design a questionnaire about the issue of designing exit tests in order to raise teachers’ awareness of issues of inclusion and exclusion in designing these tests. The teachers of the foundation program that answered the questionnaire were from different countries: U.S.A, Canada, England, Sudan, Tunisia, Egypt, Jordan, Morocco, Algeria, Kenya, and Iraq. Most of them hold a Masters’ degree and have at least five years of teaching experience. They teach different subjects: Maths, Science, Reading, Writing, Communication, and Lab courses. The identity of the participants in the questionnaire was not revealed as they were promised confidentiality and anonymity.

3.2.2 Second Participant: The Head of the English Department

The second targeted participant consisted of the head of the English department. He is a Masters’ degree holder with more than thirty years of experience in the field of education. His role as head consists of supervising, developing curriculum, hiring new recruits, and deciding on assessment policy. Through the interview, I aimed at questioning teachers’ exclusion from designing exit tests and highlighted his justifications of teachers’ exclusion from assessment practices.

3.3 Methodology

This critical exploratory study is compatible with the critical research paradigm as it aims at questioning an existing situation and raising teachers’ awareness about the issue of designing exit tests. This raised awareness, if achieved, may help teachers to get involved and change future practices. Ontologically, reality in the critical research paradigm is described within a political, cultural, historical, and economic context. Mertens (2008, p. 74–75) states that the “transformative-emancipatory ontology assumption holds that there are diversities of viewpoints with regard to many social realities but that these viewpoints need to be placed within political, cultural, historical, and economic value system to understand the basis for the differences.” Epistemologically, in the critical theory researchers emphasize the importance of the interactive relation between the researcher and the participants and the impact of social and historical factors that influence them. Mertens (Ibid, p. 99) also holds that the “interaction between the researchers and the participants is essential and requires a level of trust and understanding to accurately represent viewpoints of all groups fairly.” Because of the transformative emancipatory assumption and the importance of interaction between researchers and participants, critical methodology is directed to raise the awareness of participants and interrogate accepted injustice and discrimination. Critical theorists are “concerned with action rather than discovery” (Edge & Richards, 1998, p. 341). From this point of view, critical researchers have an agenda of change, to improve the lives and situations of the oppressed. Likewise, raising teachers’ awareness, that may lead them to question previously accepted assumptions about designing exit tests and making their voices heard, was the objective of conducting this research.

3.3.1 Data Collection Tools

Mertens (2008) argues that critical researchers may use qualitative, quantitative or mixed methods but should be aware of the contextual, historical and political factors related to the topic under study. She states that within the assumptions associated with the transformative paradigm, several of these approaches can be combined in the mixed methods design that means the use of qualitative and quantitative methods. Accordingly, critical researchers use the data collection methods that best work and serve their critical enquiry to enable them to critically study situations from cultural, economic, political, and historical perspectives. With this in mind, I used questionnaires and a semi-structured interview to investigate the issue of designing tests and raising teachers’ awareness about this issue. The use of these two research tools enabled me to triangulate data, which is defined by Cohen, Manion, and Morrison (2003, p. 112) “as the use of two or more methods of data collection in the study of some aspect of human behaviour.”

3.3.2 Questionnaires

Brown (2001, p. 6) defines questionnaires as “any written instruments that present respondents with a series of questions or statements to which they are to react either by writing out their answers or selecting from among existing answers.” Dornyei (2003, p. 14) states that the “typical questionnaire is a highly structured data collection instrument, with most items either asking about very specific pieces of information (…) or giving various response options for the respondent to choose from, for example by ticking a box. This makes questionnaire data particularly suited for quantitative, statistical analysis.”

With the above-mentioned advantages of questionnaires in mind, I chose them as my first instrument to collect data (Appendix 2). The choice was purposeful as they enabled me to gather as much data as possible in a very short period (Gillhman, 2000). They also allowed me to gather quantitative data that can be easily classified and analysed. To obtain qualitative data, I added an open-ended question at the end of the questionnaire to allow participants to express their points of views and write about issues that the questionnaire may have overlooked. Dornyei (2003, p. 15) defends this option by stating “that some partially open-ended questions can play an important role in questionnaires.”

The title was clear and the instructions at the beginning were short, informative, and well-pitched. Dornyei (2003) suggests that these may determine respondents’ feelings toward the questionnaires. In the instruction section, I informed respondents about the study and reasons for conducting the questionnaire.

Before designing the questionnaire, I conducted a group discussion during the professional development days in my institute as a part of my preparation. The aim was to brainstorm, elicit ideas, and come out with a short list regarding the issue of designing exit tests. This discussion along with the available literature helped me to identify the critical concepts and provided me with information on the relevant points and issues that I needed to address in the questionnaire.

After modifying the questionnaire, I piloted it with five colleagues from another institution where the testing practices are similar and where teachers are not involved in designing exit tests. The results of the pilot study were encouraging as respondents answered without complaining about ambiguity. Encouraged by the results of the pilot project, I distributed twenty-six questionnaires to my colleagues and gave them ample time to hand them back to me. Finally, only three of the respondents failed to answer and apologized for declining to answer the questionnaires. The content of the questionnaire elicited the following information: Demographic information about the participants, teachers’ roles in evaluation, teachers’ assessment literacy, teachers’ involvement in designing exit tests, and reasons of teachers’ exclusion from designing exit tests.

3.3.3 Semi-structured Interview

In order to triangulate data and create equilibrium between quantitative and qualitative data, I used a semi-structured interview as a second tool of data collection. Punch (2009, p. 144) states that the “interview is the most prominent data collection tool in qualitative research.” Drawing from that, I decided to use a semi-structured interview to gather data from the decision maker, the head of department (Appendix 1). The interview was conducted after analysing the results of the teachers’ questionnaires as some of the questions were based on the questionnaires’ results. It consisted of eight prompts that question the issue of testing and teachers’ exclusion from designing exit tests. The content of the questions consisted of demographic information about the interviewee, the extent of teachers’ involvement in evaluation, reasons of excluding teachers from designing exit tests, and the interviewee’s comments on the reasons of exclusion presented by teachers. I conducted the interview ten days after coding and analysing data in the questionnaire. Interviewing the head of the department was easy and smooth. He already had prior knowledge about the issue as he helped me to obtain the consent of the administration to conduct the study. I could say that this prior knowledge, the timing, and the venue (his office), were appropriate conditions to conduct the interview. After transcribing the interview, I gave the data to two of my colleagues who volunteered to check the transcription. First, I asked my colleagues to listen to the interview, check the transcription and highlight any possible discrepancy in the content. Their feedback confirmed my initial transcription. Later, the final version was given back to the interviewee to check the content. I asked him to read the transcription and make sure that I did not add any information that he had not mentioned in his interview or left out any. He was satisfied with the conformity between what he said and what I transcribed.

3.3.4 Data Analysis and Validation

Lather (1986) contends that the qualities of rigor and care can be achieved by adopting measures of conventional ethnography. She advocates using triangulation, systematized reflexivity, member checks and catalytic validity which “refers to the degree to which the research process re-orients, focuses, and energizes participants (…) [and] knowing reality in order to better transform it” (ibid, p. 67). In this study, I analysed the qualitative data inductively and adopted member check techniques to build credibility. I provided the interviewee with the transcription of the interview and asked him to check the content. Moreover, I used auditing to achieve dependability and confirmability. I gave the interview and its transcription to two independent data coders to check the content. As for catalytic validity, I think that it was achieved as 100 of the participants, who were not previously involved in designing exit tests, expressed their intention to discuss the issue of designing exit tests with their supervisors. Moreover, I used Miles and Huberman’s (1994) techniques to organize data that consists of data reduction, data display, and conclusion drawing and verification. In order to reduce and organize data, I utilized data reduction, which is the process of selecting, focusing, and transforming the gathered information. I coded and classified data into themes and used data display, which includes the use of charts and graphs, to organize information. Moreover, conclusion drawing and verification refers to my efforts to give meaning and interpret data. In addition, I compared the data from the questionnaires and interview for evidence of convergence and divergence.

4 Results

The analysis of questionnaires’ data yielded the following results.

4.1 Teachers’ Qualifications and Roles in Assessment

The twenty-three teachers who answered the questionnaires were mostly M.A or Ph.D. holders (78.3) with a teaching experience of more than twenty years (60.9). They were teachers of English, or Math and Science working in the foundation program. Fifty-two percent were either PET or KET examiners. In their answers to questions about designing daily, weekly, and exit tests, these teachers provided the following data: 34.8 design daily quizzes and 43.5 correct them. As for weekly tests, 65.2 respondents state that they design and 73.9 responded that they correct these tests. Finally, only 26.1 of the respondents designed exit tests whereas 73.9 of teachers were deprived from designing exit tests.

4.2 Teachers’ Assessment Literacy

The second section of the questionnaire was meant to elicit teachers’ awareness of assessment issues, their perception of testing, and the different variables that should or should not be considered in designing tests.

Only 26.1 of the respondents strongly agree that weekly tests can be formative while 60.9 just agree with the statement. Over forty-three percent of respondents strongly agree that weekly tests can have an impact on teaching materials. In a similar way, the same percentage agrees that weekly tests can have an impact on teaching materials. Moreover, 56.5 of teachers strongly agree that weekly tests can help them modify teaching materials while 47.8 strongly agree that weekly tests can have impact on teaching practices.

The second set of questions was meant to elicit teachers’ awareness and conceptions about exit tests. Only 21.7 strongly agree that exit tests can be formative whereas 47.8 simply agree with the statement. Comparatively, only 34.8 agree that exit tests can have an impact on teaching materials, teaching practices, and may help teachers modify teaching materials. As for the reliance on external examiners to design exit tests, 64.7 strongly agree that external designers should be aware of all testing issues of validity, reliability, and tests specs; yet 41.2 strongly disagree that test designers can prepare exit tests without consulting classroom teachers, while 88.2 strongly agree that these examiners should consult classroom teachers before designing tests.

In their responses to statements about testing purposes, 87 of the respondents strongly agree that teachers should be aware of testing purposes. Moreover, respondents were aware of test validity issues as 82.6 strongly agree that teachers should test what they teach, 69.6 strongly agree that teachers should tackle course objectives while designing tests, and 78.3 strongly agree that items should measure the intended point to be tested.

Concerning reliability, respondents showed the same amount of awareness as 69.6 strongly agree that teachers should design reliable tests that enable students to perform regularly if the test is given at different times. A closer high percentage was reached when 65.2 of respondents strongly agreed that teachers should provide learners with multiple opportunities to show what they know and can do. Awareness of reliability issues was less evident in the last statement where only 52.2 strongly agreed that teachers should include more test items to yield more reliable scores.

In their response to a statement about the level of proficiency, 60.9 of respondents agree that teachers should consider the varying levels of learners’ proficiency while designing tests; on the other hand, 56.5 disagree with the idea that teachers should design tests according to the level of low performers against 47.8 who disagree that teachers should design tests according to the level of high performers.

As far as test specs are concerned, most of the respondents strongly agree with most of the statements. For example, 69.6 strongly agree that teachers should be aware of the duration of tests and 73.9 strongly agree that teachers should be aware of the importance of wording in test design. Moreover, 65.2 of respondents strongly agree that words in questions should be familiar to students and that teachers should consider ways of presenting tests. Furthermore, 56.2 of respondents strongly agree that teachers should consider content and cultural differences while designing tests as well as the way students are expected to answer. Similarly, 47.8 of respondents strongly agree that students should be familiar with types of questions and that teachers should expose learners to exam question types. Finally, only 34.8 strongly agree that test questions should be short whereas 52.2 strongly agree that the language for directions should be simple.

4.3 Teachers’ Justification of Exclusion

This section was preceded by a direct question about teachers’ involvement in designing exit tests (question 37). Out of twenty-three responses, seventeen teachers representing 73.9 stated that they were excluded. Only these excluded teachers were asked to complete the subsequent part of the questionnaire. In this section of the questionnaire, I provided them with possible reasons for being excluded from designing exit tests. Sixty-four percent strongly disagree that they were not involved because they are not qualified in testing compared to 58.8 who strongly disagree that their exclusion is related to their lack of knowledge about designing exit tests. Results show that 47.1 strongly deny their need for special training to design exit tests. Teachers’ disagreement with potential proposed reasons for exclusion continued as 35.3 remained neutral and 47.06 disagreed with the statement that their exclusion is because of their heavy teaching load. Moreover, 47.06 disagree that they are excluded because designing tests is time consuming. A high percentage of neutral answers was returned on the question about trust and test leakage. For example, 35.3 remained neutral about the reason that they cannot be trusted while the same percentage disagreed with the statement. On test leakage, 41.18 remained neutral in their reply. They were undecided and only 29.42 agreed that teachers are not involved for fear of test leakage. Finally, only 35.3 agreed that teachers are not involved in designing exit tests because the institute is using a standardized test in the final exam.

With regard to their involvement in designing exit tests, 94.2 of the respondent teachers think that they should be involved since they are aware of the variables discussed in section 2 (reliability, validity, test specs). However, 64.7 strongly agree and 35.3 agree that classroom teachers should be involved in designing exit tests.

Although I provided respondents with many possible reasons for being excluded from designing exit tests, I gave myself the opportunity to gather more qualitative data and the respondents the possibility of expressing themselves by asking them to mention any other possible reasons for exclusion. Teachers’ responses to this question, though sometimes a repetition of the reasons I proposed, were informative and provided some reasons that the questionnaire did not address. Table 1 clarifies the different reasons that teachers presented.

Table 1 Teachers’ justification for exclusion from designing exit tests
Full size table

respondents the possibility of expressing themselves by asking them to mention any other possible reasons for exclusion. Teachers’ responses to this question, though sometimes a repetition of the reasons I proposed, were informative and provided some reasons that the questionnaire did not address. Table 1 clarifies the different reasons that teachers presented.

Apart from lack of trust, security, time constraints, and lack of competence, which were mentioned in the questionnaire, new issues were raised. Four teachers drew my attention to the issue of the policy of the institute. One respondent states that “test institution policy fear of results and their impact in other words, tests may be made easier than the course.” Another respondent writes that it is “the wish of stakeholders (director) to assign the test designing job to a decision maker (coordinator, supervisor) who might not be fully aware of the actual teaching/learning process and not aware of the test designing tools.” Another respondent raised the issue of the impact of designing exit tests on teaching practices. He asserts that “if classroom teachers design exit tests, this might unconsciously direct and influence the choice of the information they focus on in the class.” A respondent attributed teachers’ exclusion from designing exit tests to the “over reliance on habit.” He adds that “someone has always assigned exit tests at rote-learning times and it used to work for the institution. Things did not evolve. Even with the change of curricula that same person will always be in charge of test designing.”

The absence of the culture of communication was another reason presented by one respondent. He notes that the rupture between teachers and “those who are in full control of decision making” is based on the assumption that the teachers’ role is “to teach and someone else designs tests, which is wrongly thought to be beyond teacher’s abilities and knowledge.” Teachers were blamed by one of the respondents who thought they were responsible for their exclusion. This respondent writes that teachers claim that “they know about testing, without really developing their skills beyond the basics they studied in university.” According to this teacher, this development would help them “to find fault with the inaccuracies of any test and communicate them to their direct decision maker (…) absence of action on the part of teachers rendered their role passive and kept old habits of test designing monopoly and teachers’ exclusion lingering this long.”

In their response to the last question (question number 52) which was intended to gauge the impact of the questionnaire on teachers, all teachers who were not involved in designing exit tests expressed their willingness to discuss this issue with their supervisors as a future action.

4.4 Results of the Interview

Results of the interview yielded answers to the second sub question and presented decision makers’ justification of teachers’ exclusion. Overall, the results of the interview confirmed the data collected from teachers. As a decision maker, the interviewee denied total exclusion of teachers as he emphasizes that they are involved but monitored. In response to my question about teachers’ involvement in designing weekly tests, he responds that they “design tests in coordination with unit coordinators and the final product is endorsed by the unit coordinator to make sure that these assessment tools are in line with our performance indicators in our framework.” Contrary to weekly tests, teachers’ exclusion from designing exit tests is total as the interviewee confirmed what respondents in the previous section had declared. He justifies that exit tests “are designed by the assessment and testing specialists in coordination with the Head of academic studies to make sure that students had the linguistic aptitude to be promoted to the next level.” He adds that the “components and layout of exit tests are shared with all instructors so that the assessment procedures will be valid.” He concludes by stating that excluding teachers is a “choice to have minimal teachers’ involvement in order to have valid tests that focus on aptitude rather than achievement.” He justifies exclusion by the necessity to avoid teaching to the test.

Apart from avoiding teaching to the test, the interviewee did not deny incompetence as another reason for exclusion. When I referred to the results of the questionnaire that showed teachers’ awareness of most assessment issues, he changed his position and said that “involving competent teachers in the assessment procedure will bring benefit (…) but the majority of teachers are not competent.” He justifies their incompetence by their lack of training to design tests. He declares that they “may know about basic concepts but they should be trained.” In his response to my question about reconsidering his decision about teachers’ exclusion, he insists that it cannot be reconsidered because of the “nature of the program, the context.” He refers again to validity by claiming that the adoption of this policy stems from considering these exit tests as a type of standardized, external measurement tools to gauge that “what we are doing is valid.”

The impact of raising the issue on the head of the department seems to be minimal. He expressed his willingness to involve some “competent” teachers in designing exit tests in the future but rejects reconsidering the whole policy because of “the nature of the program, the context.” In any case, I think that his approval to involve “competent” teachers is a step toward a minor change in the policy. At least the interview was able to draw his attention to one of the critical issues that are causing controversy in his department.

5 Discussion

5.1 Status Quo: Inclusion and Exclusion

Results of the questionnaires and the interview reflected the status quo of teachers’ exclusion from designing exit tests. Exclusion is a fact and above all a choice imposed by the powerful stakeholders on teachers. Though they are trusted to teach, design, administer, and correct daily and weekly tests, teachers are not trusted to design exit tests. Teachers’ exclusion rate (73.9) is in sharp contrast with their inclusion in designing weekly tests. These findings seem to be in harmony with several previous studies, which highlighted this issue of teachers’ exclusion from designing exit tests. This exclusion has persisted for decades and in different settings (Rea Dickins, 1997; Shohamy, 2001, 2005; Troudi et al., 2009). Despite countless appeals to involve teachers in designing tests to guarantee test fairness (McNamara, 2012), instances of exclusion still persist.

In addition to exclusion, the issue of power was consciously present in the interviewee’s statements as he refers to his dominance as the head of the academic section on the whole process (Shohamy, 2005; Spolsky, 1997). The testing policy is built on a hierarchy that is monitored by him, the decision maker. He states that teachers are involved in weekly tests but their contribution is monitored by their unit coordinators, whose contribution is itself monitored by the head of the section.

5.2 Teachers’ Assessment Literacy

The second section of the questionnaire was meant to discuss teachers’ competence and awareness of the different issues of assessment as it has always been one of the main reasons presented by policy makers to exclude teachers from designing exit tests (Rea-Dickins, 1997; Shohamy, 2001; Troudi et al., 2009). Contrary to the interviewee who states that “the majority of teachers are not competent”, teachers’ responses to the statements about the washback effects on teaching materials and practices reflect their acceptable awareness and competence, a factor that decision makers deny to teachers. A total percentage of 86 agree that weekly tests should be formative and have an impact on teaching materials and practices, a majority of 64.7 agree that external designers should be aware of validity, reliability, and test specs, and 88.2 agree that these examiners should consult classroom teachers before designing exit tests.

Teachers’ responses to statements about reliability and validity, levels of proficiency, and test specs reflected their awareness about these issues. Hence, lack of competence does not seem to be a good reason for exclusion. For example, teachers’ responses to statements about the targeted levels of proficiency reflected their awareness about criterion-referenced and norm-referenced testing as 60.9 of respondents agree that testing should be criterion-referenced and not norm-referenced compared to 47.8 who disagree with designing tests according to the level of high performers. Moreover, teachers showed awareness of testing specs such as test duration, importance of wording, ways of presenting tests, length of questions, and content and cultural differences. Except for test content, this awareness is in harmony with the interviewee’s position who states that teachers’ inclusion in final tests should be restricted to their awareness about the layout and components of tests.

The findings of the second section of the questionnaire were important as I used them in the interview with the head of the section to deny incompetence and lack of awareness as reasons for exclusion. I anticipated the possibility for these reasons to surface during the interview and I wanted to have counter evidence. In general, teachers’ competence and awareness about most of the testing issues seem to be high and therefore not a sufficient and acceptable reason for exclusion. Moreover, in case of lack of competence, policy makers should act as post method educators and recognize teachers’ voices and visions instead of excluding them (Kumaravadivelu, 2001).

In a similar way, Rea-Dickins (1997, p. 312) suggests greater teacher involvement and states that “if teachers are given opportunities, starting through dialogue and working with the materials to develop a greater understanding of assessment processes, then, they will become better skilled at constructing tests.” Parallel to this, Shohamy (2005) argues that the lack of awareness cannot be a reason for exclusion and suggests developing teachers’ skills. She suggests abandoning the culture of viewing teachers as bureaucrats carrying out orders but rather as professionals who take part in testing policies so as to “develop critical strategies to examine the uses and consequences of tests, control their power, minimize their detrimental forces, reveal their misuses, and empower test takers” (ibid, p. 108) in a bid to empower teachers and foster a more democratic and inclusive approach to testing.

The interviewee’s and respondents’ justifications for exclusion may mean two major realities. First, exclusion is a fact, an issue, and a policy. Second, the reasons for exclusion are justified differently. From their perspective, teachers denied incompetence, lack of knowledge, the need for training, and time consumption as being adequate reasons for excluding them from designing exit tests. This denial is in sharp contrast with the interviewee’s position who views teachers as incompetent and in need of training. This same justification of incompetence was presented by decision makers in the study of Troudi et al. (2009). Moreover, both parties denied the lack of trust as a reason for exclusion, but from different perspectives. Teachers’ high percentage of neutrality concerning the issue of trust and test leakage reflects undecided positions. They were neither able to agree nor disagree with the statement. They seem to be torn between their position as teachers who would like to be involved and the alleged accusations of mistrust, dishonesty, and being sources of test leakage. The interviewee was more explicit when he stated that teachers’ inclusion in designing exit tests might lead to teaching to the test, which in principle should be avoided. His position was acknowledged by Shohamy (2001) who reported the impact of tests on educational behaviour of teachers who changed teaching emphasis and whose instruction became test-like.

Similarly, one of the teachers agreed with the interviewee by acknowledging that “if classroom teachers design exit tests, this might unconsciously direct and influence the choice of the information they focus on in the classroom.” The use of the word “unconsciously” is very important as it dispels any alleged accusation of conscious direction of teaching toward testing and clears the teachers of any deliberate dishonesty.

What is noticeable is the interviewee’s use of validity to justify teachers’ exclusion. He considers exit tests as a type of standardized measurement tool that may gauge the validity of what is being done, thus his belief that test design should be done by the testing and assessment specialists only. The justification presented by the interviewee is in harmony with the one presented by decision makers in the study of Troudi et al. (2009). Decision makers in both studies agree that quality assurance and validity are reasons to justify teachers’ exclusion. Such a view ignores the fact that excluding teachers from designing exit tests deprive their students from fair evaluation. Test specialists cannot be aware of the dynamics of classroom, proficiency level of students, and the importance of wording in questions.

The respondents’ and the interviewee’s justifications for exclusion seem to overshadow a deeper conflict of power, in which assessment tools are the arms that every party would like to control. The interviewee, for example, affirms that exit tests are in the hands of testing specialists and that tests are designed hierarchically (Foucault, 1979; McNamara, 2000; Rea-Dickins, 1997; Shohamy, 2001). It is the head who monitors the work of teachers and unit coordinators. However, some teachers seem to be aware of this issue of power in their responses. One of them refers to this as “habit”. Another one defines it as the “wish of stakeholders to assign the test designing job to a decision maker.” Another respondent explains that teachers’ role is to teach and that “someone else [should] design tests.” As for lack of competence, two teachers agreed with the interviewee and justified the exclusion in terms of lack of competence. One of them even criticizes teachers for their refusal to develop beyond the basic skills they studied at university. He claims that this passivity and the absence of “action on the part of teachers rendered their role passive and kept old habits of test designing monopoly and teachers’ exclusion lingering this long.”

As far as the impact of the study is concerned, Cohen et al. (2003) state that catalytic validity embraces the critical theory paradigm. It informs that research will lead to action. It should reveal injustice, dominance and help participants to understand and change situations. The impact on teachers seems to be obvious as 100 of the teachers who were excluded from designing exit tests expressed their intention to raise the issue and discuss it with their supervisors. The questionnaire helped to raise the respondents’ consciousness and consequently realize the injustice of being excluded from designing exit tests. By so doing, these teachers expressed their desire to cease being soldiers and servants of the system (Shohamy, 2005) to become post method teachers who have a say in policy making (Kumaravadivelu, 2001). The impact on the head of the department was not as obvious as that on teachers. The policy maker in this study, though he totally refused to change the testing policy because of the “nature of the program, the context”, nevertheless expressed his acceptance to involve “competent” teachers in the future testing process. Results of this study showed that the given issue of testing was problematized (Pennycook, 2010) and that a process of conscientisation (Freire, 1970) started occurring in my work place. I expect a more overt bargaining of power to take place sooner rather than later, an issue I may explore in future research.

6 Limitations and Recommendations

This small critical exploratory study tried to problematize the issue of depriving classroom teachers from designing exit tests and raising their awareness about this issue. Located within the critical paradigm, this study was guided by a critical agenda. Depriving teachers from designing exit tests was a political more than a pedagogical choice. The results of the questionnaire denied all the alleged accusations of teachers’ incompetence, dishonesty, and untrustworthiness. The head of the department, representing power in this study, acknowledges that the decision to exclude teachers was dictated by the context. Teachers’ marginalization and exclusion is an exclusion of the dominated group that seem to lack the tools to defend its rights. Moreover, the results showed that excluded teachers’ awareness was raised after answering the questionnaires’ questions. Excluded teachers’ intention to discuss the issue of exit tests with their supervisors reflects the success of the study to raise their awareness about one of the injustices in their workplace. This study also shows that teachers need concretization and raising awareness campaigns to be empowered. Being involved in designing exit tests may be interpreted as recognition of their competence and trustworthiness. It is also a step to involve other stakeholders that may lead to fair evaluation.

Limiting the study to my work place diminishes the chances of forming a wider picture of the situation in other educational institutions. Future research should include larger samples from different schools and universities to obtain a wider image. Future research should also seek to highlight instances of exclusion from assessment practices and the necessity to fight for equal opportunities of the different stakeholders to reach fair evaluation.