Introduction

Worldwide, different countries are discussing new methods, ideas and systems in order to face the growing challenges of their educational systems in the twenty-first century with all its regional as well as global issues (Martin 2008; McNamara and O’Hara 2005). Schools nowadays are increasingly performing in a decentralized context which leads to a growing influence of accountability measures in order to guarantee standards in teaching and learning (Altrichter and Maag Merki 2010). Systems of accountability differ considerably in different countries. Whereas in the United States and Latin America, school quality is primarily measured via high-stakes tests and students’ outcomes (Ryan et al. 2013), Europe and the Pacific region rather draw on external evaluations via school inspections (Ehren and Honingh 2011; Ryan et al. 2013). Some European inspection systems have a long tradition regarding these accountability measures (England, Wales, Northern Ireland, and the Netherlands); others have only recently installed these measures (Germany and Sweden). Basically, the range of inspection systems varies from self-evaluation to a combination of both school self-evaluation and inspection or inspection only. Moreover, the standards applied and the whole inspection approaches differ considerably and are situated on a continuum from control to support, which is also true for handling the results (Ehren et al. 2013; Faubert 2009).

In Germany since the end of the 1990s, not only the results of international large scale assessment studies, such as the Programme for International Student Assessment (PISA) (Dobbelstein and Peek 2008), but also a growing awareness regarding the need for school reforms (Maritzen and Altrichter 2015) have led to a widespread debate on the governance of schools and the question of which factors define quality in education. Since then, a number of strategies and measures have been introduced into every German federal state in order to improve the existing school system.

School inspections have only been one of those reforms and they have been implemented into all of the 16 German federal states between 2004 and 2008 (Dedering 2015; Döbert et al. 2008). Compared to international inspection systems which have already been established for a long time, such as in the UK or the Netherlands, school inspections in Germany are still a relatively new phenomenon.

International research shows that school inspections are received and processed differently in the European context (Ball 2008; Bales 2006; Chapman 2001; De Wolf and Janssens 2007; Ehren and Visscher 2006; Perryman 2010; Plowright 2007; Sleeter 2008; Willis 2010) as well as worldwide (Ryan et al. 2013). Since a lot of researchers want to gain knowledge on how inspections are perceived and processed by schools and principals, especially regarding the effects they may have on school leadership, the enhancement of school development and the improvement of teaching and learning, this article treats school inspections as feedback instruments for schools.

Above all, the general aim of this research is to ascertain the effects of school inspections and their feedback on principals in order to understand more about the efficiency, advantages and limitations of the evaluation instrument. In comparison with our former research which explored principals’ attitudes towards a pending school inspection (Bitan et al. 2015), we now explored the principals’ perspectives after school inspection had taken place and feedback had been delivered. Thus, we also focus on understanding the changes of attitudes toward school inspection. Specifically, we aim to investigate school principals’ beliefs, attitudes and interpretations about the feedback of school inspections in order to understand which aspects of external evaluations could be improved to raise their effectiveness.

School inspections: A feedback instrument

Generally, school inspections provide an opportunity for principals to receive external feedback on the quality of their schools aside from their distinctive characteristics. In this context, it is likely that the feedback from school inspections comprises not only positive but also critical and negative information. A number of empirical studies, based on the school context, have shown that the acceptance of feedback is highly complex on general and psychological levels. Table 1 serves as an overview concerning a number of influencing variables in the feedback process and connects those variables with the school context.

Table 1 Influencing variables in the feedback process
Full size table

Table 1 clarifies that critical or negative feedback may lead to its rejection. Furthermore, the acceptance of critical feedback goes along with several influential variables, such as the credibility of the person providing the feedback as well as the perceived quality and the value of the feedback for learning and development processes. Also, relations have been found not only between the degree to which the feedback is estimated as realizable and whether the suggestions for change are implemented but also with the characteristics of the recipient of the feedback, such as the recipient’s self-worth and his or her self-evaluated competency and self-control. Psychological research on self-worth has shown that individuals tend to apply self-worth-enhancing strategies when receiving and processing feedback. With regard to the feedback obtained from school inspections, a factor that determines whether this feedback is accepted is whether the inspection is focused on the control over and reporting on the school or on the support for the development of the school. Feedback given by school inspections, assuming that it is of high quality and therefore valuable, can be understood as an acknowledgement of the performance of a school, its leader and the achievements of the teaching staff. Valuable feedback that positively recognizes the above listed criteria and variables in the feedback process may be helpful and supportive for development and learning. When valuable feedback is not accepted, a possibility of growth and development may end up remaining idle. Concerning the school context, the rejection of valuable feedback could especially be considered dysfunctional because the existing problems will most likely affect many individuals and particularly children and young adults who are in an important stage of their development and need to successfully complete their school education. The epistemological focus of this article was on handling the school inspection as a feedback instrument by principals over time: we first examined the anticipation of feedback (before school inspection) and then examined the processing of this feedback (after school inspection).

International reception of school inspections

A large number of empirical studies have come from countries in which school inspections have already been conducted for a longer duration than in Germany with the aim of developing school systems by applying evidence-based methods, analyzing the schools according to standards and criteria and giving them feedback.

Teachers and principals: Dealing with inspection

Empirical data show that the higher an individual’s position at school, the more likely the feedback via inspection is processed and used afterwards (Brimblecombe et al. 1996). The time before the inspection creates a lot of stress and tension in teachers (Chapman 2000). A positive result leads to higher rates of contentment regarding the inspection (De Wolf and Janssens 2007). Principals may use inspections in order to enforce their own strategies, whereas the key task for teachers regarding inspections is centered on teaching and classroom observations (Ehren et al. 2015).

Students

Empirical evidence showing a relation between school inspections and the performance of students differs. On one hand, slight improvements of students’ performances after inspection could be found by some authors (Klerks 2013; Luginbuhl et al. 2009), on the other hand, studies looking at changes in student performance after Ofsted (Office for Standards in Education, Children’s Services and Skills) in England, consistently found no or a very slight negative effect (Rosenthal 2004; Shaw et al. 2003; Wilcox and Gray 1996).

Side-effects

School inspections have long been criticized for imposing a high workload (Ball 1997) as well as high pressure and threat on schools (Chapman 2000) without offering enough support (Perryman 2010). More explicit side-effects such as gaming, window dressing or teaching-to-the-test may occur (Ehren and Visscher 2006), but also implicit side-effects such as focusing on measures that are asked by inspection which only result in the restriction of schools’ ideas and the paralysis of school development (De Wolf and Janssens 2007). More knowledge and empirical data about the underlying processes of school inspections may help to reduce and prevent any undesired side-effects (Husfeldt 2011).

School improvement

Only 10 % of schools may be able to develop without external inspections, whereas all others are in need of external support (Ehren and Visscher 2006). Current research findings show that the impacts of school inspections on the establishment of internal school development measures are totally different (De Wolf and Janssens 2007; Ehren et al. 2013; Ehren et al. 2015).

Inspections and feedback

Teachers seem to consider inspection feedback as helpful and stimulating for school development (Brimblecombe et al. 1996; Chapman 2001). Valuable and adequate feedback which is given along the existing guidelines for successful feedback (cp. Table 1; Hattie and Timperley 2007) is associated with facilitating school development and improvement (Matthews and Sammons 2004; Ouston et al. 1997). On the other hand, the restrictive inspection systems may lead to a devaluation of the feedback (Chapman 2000). Feedback itself does not necessarily lead to changes in schools, but needs to be combined with several factors such as adequate expectations and target agreements, in order to help motivate change and improvement (Behnke 2015; Ehren and Visscher 2008; Ehren et al. 2015).

Without any doubt the controversial reception and discussion of school inspections (Ball 2008; Bales 2006; Brimblecombe et al. 1996; Chapman 2001; De Wolf and Janssens 2007; Ehren and Visscher 2006; Janssens and van Amelsvoort 2008; Luginbuhl et al. 2009; Perryman 2006, 2007, 2010; Plowright 2007; Ryan et al. 2013; Sleeter 2008; Willis 2010) would not exist if the results of the existing empirical data would be more consistent in showing that feedback from school inspections would by all means be accepted as valuable and furthering school development. Research about school inspections does not always differentiate clearly between all the influencing variables as listed in Table 1. For example, feedback can lead to resistance because of irrational processes within the feedback recipient (e.g. self-worth of the recipient), even if it is valuable. But even when there are no irrational processes taking place regarding the feedback recipient, feedback can be rejected because it is clearly not valuable (e.g. characteristics of feedback). Resistance, therefore, might be realistic and rational in certain cases. The interplay of objective reasons for resistance and the irrational reasons are not easy to disentangle. Characteristics of feedback, of the recipient, the sender, and the context may be rational or irrational, realistic or unrealistic, helpful or destructive.

School inspections in Germany

In Germany, research desiderata on school inspections and external evaluations still exist in many respects. This circumstance may be due to the fact that even though school inspections have been introduced to every federal state in Germany by now, this only has been the case for a couple of years. A number of research projects conducted in both levels of state and republic are currently focusing on school inspections as a research topic and are investigating various research questions (Dedering 2015; Gärtner and Pant 2011).

General structure of German school inspections

Although each of the 16 federal states in Germany decides on its educational system independently due to their cultural sovereignty, some collective characteristics can be recognized when it comes to school inspections. In every federal state, school inspections are based on certain quality criteria, compiled by the state ministry of education. Each school inspection starts with a notification to the school and the obligation of the school to hand in documents or portfolios, with regards to the inspection which is carried out by a team of trained inspectors. The procedures and instruments of all the 16 federal states are uniform for each inspection process. Each school inspection’s main part consists of the inspectors’ visit at school, including classroom observations (Haep et al. 2016) and interviews with different stakeholders, including parents, students and non-teaching staff. After the inspection, the school communities receive oral feedback first and written feedback via an inspection report after a certain number of weeks (Dedering 2012). The function and intention of the inspection’s feedback is to initiate development via information. Since the German approach of school inspections rather focuses on support than on control, no ranking is compiled and no direct consequences arise from the inspection results, neither negative nor positive ones.

Apart from the characteristics mentioned above, which can consistently be found in all German school inspections, some details vary according to the respective federal state. The designation of the inspection instrument differs based on each state; with titles ranging from External Evaluation, School Inspection, and School Visitation to Quality Analysis (QA). Also, the composition of the inspector’s team, the duration of the lesson observations, and whether the inspection report contains a list of recommended actions differ based on the federal state. The last difference is, whether the inspection report is published or not; only three federal states (Berlin, Brandenburg, and Hamburg) have published their inspection reports at this point (Dedering 2015).

The inspection system in question: Quality Analysis North Rhine-Westphalia (QA NRW)

The inspection system in question will be described and interrelated with feedback as an instrument in the following section. Quality Analysis can be understood to be a low stakes form of school inspection since it carries no significant consequences. Schools which receive below-average results are inspected again after a short period of time, for example, after 12 months. Quality Analysis has only recently refined its approach, putting more emphasis on self-evaluation and school participation than at the very beginning. It is based on a Quality Tableau, containing different main and subordinate criteria. One main focus is on teaching and learning and on classroom observations. These are carried out by inspectors during 3–4 visiting days at the inspected schools. At least 50 % of the lessons are observed and rated according to a standardized lesson observation plan. The visiting days are always followed by a direct feedback to the principal first, ensued by a feedback via oral presentation to the whole school community. Feedback regarding classroom observations is given systematically, in such a way that no single teacher but the whole school receives feedback on teaching and learning culture. Approximately 9 weeks after the visiting days, schools receive a written Quality Report containing written and detailed feedback on their results. Quality Analysis sees itself explicitly as an instrument which analyses and hints to facts however it is not involved in counseling the schools on how to use the received feedback afterwards. Data and results, such as the report, are neither published nor used for school ranking, but schools are free to publish their Quality Reports, for example, on their homepages. After receiving and processing the results, school principals are required to set up target agreements in cooperation with their responsible regional supervisor who works independent of Quality Analysis. Although it has been operating since 2008, not all schools in NRW have been inspected yet which is why it is not possible to predict the achievements of Quality Analysis in the long term.

Research findings

In Germany, school inspection was characterized as a non-standardized control by the supervisors of the federal state for a long time. Here, the supervision and administration did not act according to statewide standards. After the disappointing results of the first PISA study and other comparative tests, NRW developed a set of new standards for school improvement. This new standardized school inspection was introduced as “Quality Analysis”. It is no longer processed by individual supervisors, but by trained inspectors who work at a department in the ministry specifically established for school inspection. Since these new and standardized forms of school inspections have only been introduced between 2004 and 2008 throughout Germany, there are only some primitive insights regarding the effectiveness, acceptance and side effects of school inspections. So far, research desiderata exist regarding the German school inspections’ impact on the results of students. Those studies, who have aimed at investigating side effects of German school inspections, conclude that these can only rarely be found (Gärtner and Wurster 2009). More research has shown that school inspections have led to a number of activities in schools (Gärtner et al. 2009; Huber 2008; Kotthoff and Böttcher 2010) and the planning of school development measures (Böttcher et al. 2010: Dedering and Müller 2011; Sommer 2011). Those activities have taken place, although principals as well as teachers have declined the novelty and innovation of the inspection results (Behnke 2015; Böttcher et al. 2010). Research suggests that German principals may have a more positive attitude towards school inspections than teachers (Behnke and Steins 2015; Gärtner and Wurster 2009).

Our research: Principals’ reactions to feedback received by school inspections

Before school inspection: The anticipation of feedback

A first interview study with 50 participating principals from North Rhine-Westphalia (NRW, Germany) showed which attitudes principals held towards the newly installed external school inspection called the Quality Analysis (QA; Bitan et al. 2015). The basic assumption that resulted in the selection of principals as interview partners was that principals as leaders may have a considerable influence on the information and attitudes that are conveyed to their teaching staff. This means that they may also be able to determine the extent to which the feedback from a school inspection such as the QA is accepted by the teaching staff and the entire school community (Price 2012; Shen et al. 2012).

Questions related to the acceptance as well as handling of an external feedback instrument (e.g. QA) are important because low or no acceptance may restrict the use of the results. It may thereby lead to a situation in which an institution does not draw conclusions from valuable results and does not use the results as corrective or for considering where improvement is needed. This, again, would mean that the aim and the potential to support and develop schools in NRW for a relatively costly and elaborate instrument such as the QA would be realized only to a limited degree.

On the other hand, at the first point of data collection (Bitan et al. 2015), it was not only principals with a negative attitude concerning the QA who voiced criticism against it but also those with a positive attitude. This finding hints that there is a general need for the external evaluation to sharpen and develop its policies and procedures. Therefore, using the data from interviews with principals to analyze their reactions to the feedback received by a QA could enable the improvement of the feedback instrument by more intensively considering the specific context of the principals (Mertkan 2011, 2013).

The results of the qualitative study by Bitan et al. (2015) at the beginning of the measurement showed that there were no differences in the acceptance of school inspections before the QA took place with regard to the existing school types. Across all school types, 42 % of the interviewed principals voiced a positive attitude towards the QA and specified that an external evaluation could be seen as a means for school development, 24 % of the principals held a neutral attitude, and 34 % of the principals expressed a negative attitude. As mentioned above, the principals with a negative attitude were not the only group who criticized the QA. However, it was particularly the principals with a negative attitude who made a large number of generalizing negative statements, for example, in reference to the high workload or to the inspectors as the ones providing the feedback. The inspectors in particular were explicitly criticized in the first interviews, even before the school inspections took place and in this way captured the anticipations of the principals. It quickly became obvious that only a few positive statements concerning the inspectors were made in total (Bitan et al. 2015).

After school inspection: Processing the feedback

In the present research we applied a longitudinal research design with the goal of determining whether the attitudes of the principals towards the QA feedback instrument, voiced before the QA occurred, would remain the same or would change after the schools had experienced the external evaluation and its feedback. An event that occurred in the past may be estimated more positively, which might also mean that a person’s attitude towards this event might be remembered more positively, a result that has also been found by Brimblecombe et al. (1995) for the reception of school inspections in Great Britain. Furthermore, the aim was to determine which factors would facilitate, hinder, or prevent the principals’ acceptance of the feedback obtained by the QA.

The value of this longitudinal study, the likes of which has never been conducted in Germany in relation to school inspections before, lies in the opportunity to gather information that would allow us to discern the processes behind the acceptance of or resistance to external evaluations and their feedback more precisely and thus more precisely understand the underlying mechanisms of inspections in general. Therefore, the results could be valuable for researchers, principals, initiators of external evaluations, and inspectors worldwide. The leading research question focused on the reception of feedback: how do principals deal with the feedback that they obtain from a Quality Analysis and how do they process it? The quantitative studies that had previously been carried out by the Ministry of Education in NRW hinted at a relatively high level of satisfaction concerning the attitudes of principals towards the QA (Müller 2008, 2009), but since those studies were carried out without granting the schools anonymity, and hence, linking the questionnaires to schools, there is a possibility that these studies were biased by social desirability.

Method and design

At the first assessment, 50 principals from NRW were interviewed about their attitude towards the QA (Bitan et al. 2015). Every school in NRW, which had not experienced a Quality Analysis before the assessment, was randomly sequenced in a list. An approximate even number of schools from each school type was drawn randomly then contacted and interviewed after their permission. Because the same principals who had already been interviewed before were interviewed at the second time of assessment about the way they processed the inspection’s feedback, the procedure remained the same in order to guarantee the validity and reliability of the data across the entire longitudinal study.

Data collection: The loosely structured focused interview

In order to assess the attitudes of the principals in a relaxed atmosphere without social pressure, a loosely structured focused interview, was chosen as the main method of gathering data (Kvale 1996). By choosing this method over others, we were able to give the principals enough room to provide accurate and detailed descriptions of their attitudes towards the QA and related topics.

Exploratory design

An exploratory design (Creswell 2014) was chosen because of the lack of research on handling the feedback received by principals through an external evaluation. Therefore, using a mixed method approach for this study seemed especially beneficial considering the aspect of additional coverage (Creswell 2014; Morgan 2014). Also, an exploratory approach was preferred over a hypothesis-based approach in order to obtain and aggregate further knowledge in the research area in question (Patton 2002).

The interviewer

The interviews were administered by the first author, a research associate at the University of Duisburg-Essen who has passed the second state examination for the area high school/comprehensive school. Therefore, the interviewer had extensive knowledge of the research area. The interviewer was thoroughly trained before administering the interviews.

The random sample

The original sample consisted of N = 50 principals from schools in NRW. These principals were supposed to participate in the longitudinal study ordered by the Ministry of Education “Attitudes of principals towards QA in NRW—a longitudinal investigation”. But only N = 20 principals could be included in the longitudinal study, because of several dropout reasons. Twenty-six (52 %) of the 50 schools still had not received a date for a QA visit after 3 years, and the time at which the school inspection would occur was unclear. Four (8 %) of the schools that participated at T1 who experienced a QA did not take part at T2 for reasons such as new principals taking over due to retirement and one principal who did not want to take part in the second interview without indicating any reason. The 20 participating schools covered the entire range of existing school types (N = 1 elementary school, N = 3 secondary modern schools, N = 4 middle schools, N = 1 high school, N = 3 comprehensive schools, N = 5 vocational schools, and N = 5 special education schools). The sample that was investigated in the longitudinal study consisted of N = 14 (70 %) male principals and N = 6 (30 %) female principals.

Scheduling the second measurement date (T2)

The schools that participated at T2 were determined by the date on which the QA visited the school. These dates were independent of our study and were set by the Ministry of Education. Since the schools that participated in the study were not known to the Ministry of Education to ensure anonymity, they were integrated into the scheduling of the QA along with every other school. At least every 3 months, the Ministry of Education provided the researchers with up-to-date lists containing the dates of all upcoming inspections. The relevant schools that had already taken part at T1 were then identified and integrated into the longitudinal study. Therefore, the scheduling of T2 was difficult because the schools had to be interviewed after the QA had occurred. Furthermore, the interviews took place at least 6 months after the QA had occurred in order to leave some time for the schools to receive their inspection reports as well as to resolve issues about agreements on objectives and to be able to address these issues in the interviews.

Contacting the schools

Contacting the schools included a coherent structure. After the QA had occurred and the required time span of 6 months had passed, the schools were first contacted in writing. The documents they received consisted of a letter from the researchers explaining the objectives of the study and asking the principals for their participation as well as a legitimizing letter from the Ministry of Education asking for support in order to improve the QA. The written letters were followed by a phone call by the interviewer a few days later to ask the principals whether they were willing to participate in the study for the second time and arranging a date for an interview if the response was positive.

The interview

All interviews took place at the principals’ schools and were conducted with the support of an interview guide. The principals were furthermore shown some material that described the inspection in detail and that had already been shown in the first part of measurement (structure and schedule of the QA, overview of the goals of the QA, and Quality Tableau including the inspection criteria). The length of the interviews varied from 22 min 12 s for the shortest interview to 50 min 19 s for the longest interview.

The overall reception of the interviewer was friendly and welcoming. Because the principals were free to participate in the interviews and frequently emphasized that they viewed the research study as an opportunity to declare their opinions about the QA and therefore contributing to its improvement while remaining completely anonymous, it could be assumed that the interviews were answered openly and honestly to a high degree. Moreover, the interviews took place in familiar surroundings, which is one of the reasons why most of the principals were obviously relaxed and were also able to include a number of documents and other items in the interview.

The introductory question of the interview aimed at identifying the principals’ subjective estimation of their attitude and attitude change regarding the feedback instrument QA (Table 3, subjective estimation). In general, the interview was oriented around the structure and scheduling of an inspection, including the school’s preparation, the visiting days, as well as the post-feedback processing. By using this structure, the attitudes and actions of the principals as well as the entire school community could be captured before, during, and after they received the feedback from the QA. All statements directly related to the attitude towards the QA were categorized within the category of Overall estimation of QA which was the basis for the calculated attitude and the change in the attitude of the principals (Tables 2, 3, Calculation of AC).

Table 2 Attitudes of principals towards the QA: data from the longitudinal study
Full size table
Table 3 Attitude change after the QA
Full size table

The documentation of the interviews

In order to document the interviews, the conversations were recorded with a Smart Pen (Livescribe Smartpen). All interviews were completely transcribed. For the reason of anonymity, all places, names, and individual aspects of recognition were marked with XXX. In order to guarantee greater legibility, the interviews were documented in the form of a literal transcription.

Analysis of the interview material

The goal of a qualitative content analysis is to achieve an interpretation that is intersubjectively understandable on one hand and highly exhaustive on a content level on the other hand (Patton 2002). The aim of this longitudinal study was to obtain a firm indication of the attitudes of the principals towards the QA feedback instrument before it occurred and how their attitudes changed after they had experienced this new type of inspection at their own school. Furthermore, the principals’ way of dealing with and accepting the feedback from the inspection was the subject of this study. The interview questions were oriented towards the scheduling and processing of the QA. The coding of the transcribed interview material was deductively guided by the process of the QA based on the questions in the interview guide, which influenced the coding and the creation of categories. Theoretical approaches from Grounded Theory (Glaser and Strauss 1967) were specifically used for the inductive coding of the subcategories from the interview material. The core category of the content analysis consisted of the principals’ handling of and attitudes toward the QA and the feedback received from it. The first steps of Grounded Theory can be especially useful for the inductive development of subcategories from the data (Schreier 2012). Therefore, the subcategories were created inductively by analyzing the material. “The strategy of inductive designs is to allow the important analysis dimensions to emerge from patterns found in the cases under study without presupposing in advance what the important dimensions will be” (Patton 2002, p. 56).

A mixture of deductive and inductive approaches was chosen to analyze the interview material as it is quite common in empirical research (Patton 2002). In addition, elements from an evaluative qualitative content analysis were integrated into the coding process (Schreier 2012). All interviews were coded with the help of software for qualitative data analysis, MAXQDA. The interviews were first analyzed and coded by the interviewer and then coded by a second person in order to guarantee the validity. The validity was verified via argumentative validation between the first and the second coder. There was no residual category. Whenever a statement comprised two aspects on the content level, it was coded multiple times. The frequencies of all coded statements were entered into a statistics program (SPSS, version 21) so that changes in the attitudes of the principals could be quantified. Therefore, the qualitative study has priority to the quantitative results.

Results

Quantitative data

One of the focal areas of the longitudinal study was reflected by the leading research question which asked about the principals’ attitude towards the external evaluation tool from the Ministry of Education after receiving the feedback from the QA. Table 2 offers an overview of the principals’ attitudes.

The attitudes of the principals were calculated by computing the difference between the negative and positive statements that had been coded into the category of Overall estimation of the QA. On the basis of this calculation, 25 % of the principals reported a negative attitude toward the QA on the first assessment (T1), 40 % had a neutral attitude toward the QA before it took place at their school, and 35 % of the principals voiced more positive than negative statements towards the QA at the first measurement point. The results of T2 showed that 15 % of the principals had a negative attitude toward the feedback instrument QA after experiencing it, 10 % had a neutral attitude, and 75 % had a positive attitude toward the QA.

In order to identify shifts in the principals’ attitudes, their attitude at T2 was additionally assessed in relation to whether it was better, worse, or the same as their attitude at T1. Subsequently, the changes in the principals’ attitudes towards the feedback from the QA are presented in Table 3.

60 % of the principals acquired a more positive attitude towards the QA after receiving the feedback from it, 20 % of the principals maintained their positive attitudes and 20 % of the principals modified their attitudes negatively after receiving the feedback from the QA (Table 3, Calculation of AC).

Some of the principals indicated that they could not remember the attitude that they had presented before the QA at T1 with certainty. The introductory question which identified the subjective estimation of the principals’ attitude and attitude change allowed us to compare both the subjective perspective and the calculated position of the principals. The sum of the percentages of the principals who remained positive and those who became positive was very close to both of the calculated changes in attitude. Interestingly, considerably more principals indicated that they remembered having had a positive attitude towards the QA at T1 (Table 3, Subjective estimation), but this could not be validated in comparison to the calculated attitudes and attitude change.

Qualitative data

Two main categories of the qualitative data analysis including their subcategories will subsequently be presented in order to clarify the reasons for the principals’ perception of the feedback instrument QA and the processing of its feedback: Category I outlines the principals’ attitude and attitude change and Category II illustrates the estimated workload and stress caused by the QA.

Category I: Attitude and attitude change

Category I: attitude and attitude change contains four subcategories with further differentiations. Also, the principals’ five most important reasons for maintaining or changing the attitude are listed in Table 3 (Rationales). In the following, these reasons will be presented in detail and exemplified by quotes and statements of the principals.

Subcategory: Reasons for maintaining a positive attitude towards the QA

Feedback

Receiving objective feedback from an outside perspective was one reason why some of the principals maintained a positive attitude towards the QA. For example, they stated that obtaining feedback from an external perspective “is a positive thing”. And they also stated that the feedback was adequate and not surprising, and therefore, that a QA is a good opportunity “to present one’s school and (…) get an external evaluation” of its quality. Another reason they gave for maintaining a positive attitude towards the QA was that the teaching staff had accepted the feedback. The principals who made a statement that fell into this subcategory explicitly addressed the fact that a criteria-oriented external perspective is very seldom found when working in schools.

Result

Those principals who maintained a positive attitude towards the QA indicated that the attainment of a positive result was another reason for maintaining their attitude. The recognition of their work as reflected by a good result in the external evaluation resulted in the participating schools and principals being “proud and content” as well as “motivated to make further changes and developments”.

Information

Two principals indicated that they were well informed about the QA before the visit. Information was obtained by cooperating and exchanging information with colleagues. One principal described the cooperation between principals concerning the QA as “intensive”. Many principals obtained their knowledge from hearsay and other informal lines of communication which is consistent with our former results (Bitan et al. 2015).

Work of the inspectors

Two principals asserted that the good work of the inspectors led them to maintain their positive attitude towards the QA after the visit. These assertions were related to aspects such as creating a positive atmosphere while the external school inspection was conducted (“nice atmosphere”, the inspectors “behaved positively towards the teaching staff”). Additionally, the principals made positive remarks which indicated that the rumors they had heard about the behaviors of the inspectors were not true. Another reason for maintaining a positive attitude was the inspectors’ accuracy in their perceptions and assessments of the schools as well as their feedback.

Support

Two principals indicated that they maintained a positive attitude towards the QA because the feedback from the QA supported them in further developing their schools.

Further reasons

All of the following reasons for maintaining a positive attitude towards the QA were reported by one principal each. One reason was that the feedback from the QA offered appreciation for the principal’s work. This principal’s work had not been appreciated by anyone from the school community before the QA, but the feedback from the QA explicitly mentioned the positive actions taken by the principal. Another reason was that the QA and especially the classroom observations were perceived as accurate tools for providing an evaluation and feedback. The principal who voiced this aspect of the QA explicitly mentioned previous criticisms of the classroom observations and stated that such criticisms were wrong. Another principal described the QA as a “positive stimulus” that was useful for reflecting on areas of work and assuring oneself of school’s quality. One principal stated the fact that the whole school community voiced satisfaction with the interviews that were conducted by the inspectors as one reason for maintaining a positive attitude towards the QA. Another positive factor was the fact that “the inspection was over” and that “after the visit, there was a relaxed atmosphere and attitude towards it within the whole school community”. One principal stated that he had already experienced a QA as a deputy principal, and therefore, preparing his school for the QA was accompanied by a positive feeling because “I knew what would be important”. Another principal stated that it was important to inspect the schools according to criteria, that the QA serves the function of “providing feedback for schools’ achievements and therefore facilitates the overall quality of schools”, and that the feedback from the QA supports principals in their roles as the leaders of schools.

Subcategory: Reasons for maintaining a neutral attitude towards the QA

Five principals voiced a total of five statements that indicated neutral attitudes towards the QA. We have listed the reasons for the maintenance of a neutral attitude towards the QA below.

Classroom observations

Here, one principal focused on the accuracy of the classroom observations; the other principal criticized the fact that the classroom observations focused on teaching methods while neglecting a high standard concerning the content of the lessons.

Further reasons

One principal stated that the school had not learned anything new from the QA feedback and that the school “has not become much smarter” after the external inspection. A further reason for a neutral position was the criticism of the measurement instruments, which could now be specified. One principal made a neutral statement that the QA has been able to assess the extracurricular activities of schools by studying the portfolio that was handed in by the schools. Another neutral statement mentioned that the QA process was sufficiently transparent.

Subcategory: Reasons for having a more negative attitude towards the QA

Two principals indicated that their attitude towards the QA and their assessment of the QA had changed in a negative way after they experienced the process of being inspected.

Dubiousness of the measurement instruments

An important reason for the negative attitude change was the fact that the measurement instruments were estimated as dubious. Both principals whose attitudes were changed by becoming more negative after the inspection voiced a statement that fell into this subcategory. Two additional principals without an overall negative attitude pointed to the dubiousness of the measurement instruments as well.

Work of the inspectors

Both principals with a negative attitude change addressed the manner and attitude of the inspectors who provided the feedback as one reason for their attitude change. One principal stated that “the attitude of the inspectors in dealing with schools definitely needs to be reconsidered”, without specifying this statement any further. Another principal had the perception that the inspectors had wanted to damage the school and cast a shadow on it since the beginning.

Further reasons

Another reason for the negative attitude change towards the QA was the “disproportionate” burden the inspection imposed. The principal voicing this reason referred to the number of classroom observations, especially in smaller schools, and to the workload for principals in general. Furthermore, one principal stated that a passage in the inspection report concerning the interview with the pupils led to his negative attitude change towards the QA. The principal explained that the pupils had reported to their teachers that they had been interviewed “with a negative aim” and that “they were not correctly understood in their statements and not allowed to rectify themselves as well”. One principal gave five more reasons for her negative attitude shift towards the QA. Reasons were the “limited time slots for the preparation of the QA as well as the insufficient support”. Furthermore, the principal was convinced that the quality of the school was not enhanced by the inspection and the feedback of the QA and that, especially for smaller schools, “costs and benefits” were related to each other in only a disproportionate way. A last reason for the negative attitude shift was the expenditure of time needed to compile the requested documents (i.e. the portfolio), which the principal described as “impossible” (“It took me 4 weeks, all 7 days, 24 h a day”).

Subcategory: Reasons for having a more positive attitude towards the QA

Five principals made statements for the reasons their attitude had changed for the positive after they experienced the QA inspection process. The principals stated that before the QA had taken place, they were “more critical” and “more skeptical” and had “some severe concerns about the QA”.

Result

Three principals referred to the fact that the positive feedback as well as the positive result had a substantial and positive influence on their attitudes towards the QA. Two of the three principals indicated that their attitude would have been more negative if the feedback had been more negative too. [“It (…) definitely is related to the result, how you judge it later”].

Work of the inspectors

Two principals’ emphasized the good work of the inspectors. The inspectors were reported to have shown “impressive knowledge about the inspected school” and to have learned the portfolio accurately.

Teaching staff

Another reason for a positive attitude shift was that the QA had led to “team building” and a higher “sense of community”. Furthermore, fears that the teaching staff would not support the inspection process proved to be gratuitous. According to two principals, being inspected by the QA was a challenge that had been mastered by all participants together.

Legitimation of one’s work

For two of the principals, “the QA and its feedback represented a legitimation” and therefore supported their work.

Further reasons

One principal related to the fact that “the rumors that preceded the QA” were disproved. Another principal stated that the QA had affected the entire school community in a positive and motivating way. The feedback from the QA had been “empowering”, had “led to more self-confidence”, and had improved the “psycho-social mood” of all school members.

Category II: The estimated workload and stress caused by the QA

Category II: The estimated workload and stress caused by the QA contains three subcategories with further differentiations. According to the principals, the main sources of workload and stress are the portfolio, the classroom observations and the objective agreements, the feedback, and the time pressure. In the following, these reasons will be presented in detail and exemplified by quotes and statements of the principals.

The principals in the sample stated their individual estimations of the workload and stress that had come about from the QA. Twenty principals made 100 statements in this category, a number of which showed that the school leaders were preoccupied with the topic. Five principals made a total of seven statements in which they estimated the burden from the QA to be very high; it was for example described as being “outrageous” and “tremendous”. One principal claimed to have anticipated the workload already before the QA took place by indicating that it would be “an exhausting affair” that would cost “a lot of time and work and effort”. Seven principals positioned themselves rather neutrally when it came to the estimation of the QA workload and described it as “appropriate”. The process of preparing for the QA and experiencing the school inspection had “not been too burdensome”. The preparation for the QA had been “very extensive”, but they had tried to regard the whole preparation process in “quite a businesslike fashion”. Two principals stated that the workload and level of stress initiated by the QA had been “manageable and low”.

Main sources of workload and stress

Portfolio

For this subcategory, 15 principals made a total of 35 statements relating to a high workload. The burden and workload were described among others as “enormous”, “a miserably high workload”, “not correct”, “an extremely high amount of work”, “insane”, “inappropriately elaborate”, and “very, very burdensome”. One principal also stated, “Well, I think the QA could substantially reduce the extent of its portfolio and still manage to obtain the same information on a content level”.

Classroom observations

The classroom observations were estimated to be the second highest burden of the QA. Five principals made seven statements relating to a high workload and especially the stress caused by the classroom observations. As one reason for the high level of stress caused by the classroom observations, some principals mentioned the habit of teaching behind closed doors and never sharing one’s lessons with anyone else. Hence, teachers were not very used to dealing with observations and feedback on their teaching abilities.

Objective agreements, feedback, and time pressure

These three aspects were also identified by the principals as discouraging in the subcategory “high level of workload and stress”. Three principals made seven statements that referred to the process of developing and finalizing the objective agreements as a high liability. Two principals referred to the feedback at the end of the inspection process as highly stressful and burdensome. Not being able to deliver an opinion on the feedback from the QA was especially perceived by the teaching staff “as something they needed to get used to”. One principal stated that the feedback from the QA had produced “a large psychological burden and a very bad feeling”. Also, one principal indicated that another burden had been the time pressure that occurred during the preparation for the QA.

Discussion

The leading research question focused on the way the feedback was received: how do principals deal with, evaluate and react to the feedback they obtain from a Quality Analysis? We also wanted to find out the effects of school inspections and the feedback on principals in order to understand more about the efficiency, advantages and limitations of the evaluation instrument.

The factors that essentially led to the attitudes of a large percentage of principals remaining positive or becoming more positive consisted of positive and constructively presented feedback, a positive result, the feeling of being well-prepared and having all the necessary information needed to manage the QA visit at one’s disposal, the professionalism of the inspectors giving feedback, and the experience that the QA offered substantial and valuable feedback and support for the development of schools. Additional factors that were reported by the principals were that the QA provided support for the social cohesion and teamwork of the teaching staff and could offer support in legitimizing the principals’ work, a result also found by Ehren et al. (2015).

The results hinted that one crucial factor that determined the acceptance of the external school inspection was the feedback it provided.

With regard to the estimated workload and stress, the results of the study showed in line with empirical research data by Chapman (2000) that this topic was fundamentally significant and that stress and a high workload occurred mostly before inspection. The highest levels of workload and stress were related to the portfolio and the classroom observations. The portfolio is a task that is primarily completed by the principals, whereas the workload and stress of the classroom observations can mostly be found within the teaching staff. The statements of the principals showed that the workload and stress level which are estimated as inappropriately high may lead to a depreciation of the feedback instrument school inspection.

In general, our results show that 60 % of the principals’ attitudes toward QA became even more positive after receiving the feedback, 20 % of the principals did not change their previous positive attitudes and 20 % of the principals modified their attitudes into negative after receiving the feedback from the QA (cp. Table 3, Calculation of AC). Many principals explicitly reported that receiving feedback was the reason for their positive attitude.

International research shows that inspection systems may have a powerful influence, but it is still hard to declare whether this effect is actually positive or negative. Also, the impacts of school inspections on establishing internal school development measures differ (De Wolf and Janssens 2007; Ehren et al. 2013). As international evidence shows, feedback can be regarded as an essential instrument for initiating any development at school. But feedback itself does not necessarily lead to changes at schools. International researchers suggest that feedback from school inspections needs to be given with adequate expectations in order to make changes and improvements (Behnke 2015; Ehren and Visscher 2008; Ehren et al. 2015). A positive attitude towards the QA may therefore be related to a positive estimation of obtaining feedback by this feedback instrument. Still, a positive attitude towards feedback does not necessarily mean that the feedback is practically administered. Therefore, more research is needed to realize which factors can lead to the implementation of school development measures hence, having persistent improvements at schools.

Overall, the principals voiced a great deal of satisfaction with their inspection results and the inspectors’ evaluations. The principals who explicitly voiced satisfaction described the inspectors as constructive in administrating feedback and that they applied most of the criteria of giving feedback appropriately. The critical principals, on the other hand, distinctly criticized the insufficient professionalism in the attitude of the inspectors who provided feedback. It is very important to conduct further research on the question whether a positive estimation of the feedback is connected to positive results only, meaning that receiving negative feedback would automatically lead to a devaluation of the feedback which has been received by the schools. If so, receiving negative feedback would especially be hindering for schools with a high need for development, because it would not allow them to accept their flaws and improve. As it was seen in the anticipation of the QA (Bitan et al. 2015), the inspectors had a large impact on whether the principals’ assessments of the QA as a feedback instrument were positive or negative.

Other important questions that arise from our research are universal questions regarding the usage of feedback: is feedback always a valuable resource and is it always helpful for schools? Or are there cases in which feedback from school inspection may compromise the development of a school? Are the effects of feedback after school inspection the same in all parts of the world?

Implications

The present study is conducted in one federal state of Germany. Nevertheless, in this study some basic features of school inspections were shown which can be generalized to most school inspection systems.

In the following, implications are drawn from our results along with the classification of feedback interactions which are presented in Table 1. We are sure that these implications are helpful for the improvement of feedback given by school inspections aiming to enhance school development in other parts of the world, because our results are consistent with and add to research conducted in international contexts. In addition, Table 1 reflects basic research results that may be influenced culturally, but are more or less universal for feedback interactions and individuals’ reactions to feedback.

Characteristics of the feedback sender

According to the results of our study, in order to ensure that the feedback from school inspectors and the whole QA process is accepted, it is important for the inspectors to collaborate positively with the entire school community. This finding is in line with international research results suggesting that the effectiveness of school inspections can be improved via cooperation with schools (Whitby 2010).

In order to guarantee a positive collaboration with schools, inspectors should on one hand carefully prepare themselves for the inspection by obtaining information about every single school and they must have a high level of expertise. On the other hand, objective attitudes combined with a friendly and appreciative demeanor are of great significance. This demeanor may also help to reduce the anxiety and reservations of the school staff which are seen in the results of our study as well as in international evidence (OECD 2013). When the inspectors are perceived and evaluated negatively by the principals and the teaching staff, a negative attitude towards the feedback of school inspections may easily be reinforced and then intensified by group processes and cognitive dissonance as well as psychological reactance, which should be avoided at all costs (Brehm and Brehm 1981; Festinger 1957; Forsyth 2010).

Ensuring that the inspectors are informed and prepared and that they take the principals’ concerns seriously seems to play a crucial role in the principals’ acceptance of school inspections. It is also very important for the inspectors to be perceived as partners who may be helpful in the development of schools. This perception may especially be obtained if the inspectors are able to convey a fair and sympathetic attitude during the process of giving feedback. Principals are more likely to accept school inspection when the external evaluation conveys expertise as well as support, a finding also supported by Whitby (2010) and Schildkamp and Visscher (2010). Furthermore, the behavior of the inspectors, their professional and detailed knowledge of the whole school and the portfolio, as well as their ability to make contact on an equal footing play significant roles in how well an external evaluation is accepted and may help to reduce resistance and reactance to its feedback.

Regarding the inspectors’ professionalism, the results show the importance of having adequate qualification and trainings as well as supervision to handle and interact with groups. This result is certainly of high importance to international inspection systems as well, since the role of the inspectors for a positive feedback interaction and inspection process is clearly seen.

Characteristics of the context

One important and highly criticized aspect of school inspections was the stressful work that it brought to the principals. After understanding the stress caused by the paperwork, the problem was discussed at the ministry in NRW and consequently the amount of paperwork which had to be done by the principals before the inspections was reduced. Other inspection systems, also on an international level, should follow suit and reduce the amount of workload for principals if they wish to increase the acceptance of external school inspections and decrease unintended effects.

Besides this factual change on the level of the evaluation instrument itself, psychological implications should be minded: in order to further decrease the stress created by classroom observations, it might be beneficial for principals to focus on the instigation and establishment of an open school culture in which their teaching staff is encouraged to receive and be trained in the open and constructive handling of feedback. Every individual working in a school should be encouraged to develop a constructive attitude towards criteria-oriented and valuable feedback by viewing it as an opportunity to develop and strengthen one’s own as well as the institution’s strengths and abilities (Behnke 2015). This attitude is, on the other hand, only of use when inspections focus on the development of schools rather than on reporting them. A number of research results show that stress, high workload and a negative anticipation of school inspections may lead to unintended side-effects such as window dressing, fraud, gaming, misrepresentation and many more (Chapman 2001; De Wolf and Janssens 2007; Rosenthal 2004). Therefore, in order to reduce the level of stress caused by inspections, it might be useful if organizations in charge were to develop such school inspections which can be anticipated as positive, useful and supportive by principals and can therefore lead to having better schools and higher quality teaching. The anticipation of feedback as real support can also help to increase the acceptance of school inspection among principals.

Characteristics of the feedback and its recipient

As the results of our study show, although school inspections are supposedly planned and conducted by experts, feedback interactions between inspectorates and schools seem to remain a very complex and complicated issue. Although characteristics of good feedback seem to be established by now, applying the theory still remains a challenge (Behnke 2015). One of the key factors that influence the effect of inspections on principals is the quality of feedback. Another factor is the use of support in order to enforce development, because research shows that feedback alone is not strong enough to lead to school development (Elmore and Fuhrman 2001; Matthews and Sammons 2004). Also, feedback from inspections may need to involve all stakeholders, and must be delivered openly with enough space for all participants to share their ideas. It should be individual, relevant and close to the current actions at inspected schools in order to create and motivate each school to consistently work on its own improvement (Gilbert 2012; Hargreaves 2011). Feedback and advice from school inspections needs to be specific and related to the school’s individual context in order to be accepted despite the criticism and negative results that it might have (Behnke 2015; Courtney 2012; Schildkamp and Visscher 2010).

Receiving a positive result from the QA process was indicated by many principals as one of the main reasons for accepting the feedback from the QA. Here, our results are in line with De Wolf and Janssens (2007). Furthermore, the results showed that the principals expressed more motivation to improve the aspects that were criticized if they had received a positive result. These results thus imply that the positive results have a substantial impact on the self-worth as well as self-efficacy of the principals. Consequently, such principals will also feel more competent about initiating and leading the processes of change. Still, it would not be realistic to expect every school to be functioning at a high or a very good level and therefore achieving a good result in the external evaluation. Our findings therefore show the importance of understanding the underlying irrational processes in the acceptance or rejection of feedback. The fact that feedback is accepted does not mean that it is valuable or realistic. Feedback may also be accepted because a school is content about its results and the fact that they could disguise their deficiencies. Therefore, it would be reasonable to raise the question of how the acceptance of school inspections could remain positive even when the result contains a lot of negative criticism and how the self-efficacy and motivation of principals to develop their schools could still be maintained in such cases. Here, basic psychological research indicates the fact that external evaluations and feedback interactions, need to be carefully planned regarding the individuals’ irrational reactions, because these reactions may interfere with the acceptance of feedback (Behnke 2015).

Limitations of the study

Our exploration shares the limits of all studies that are based on interviews. Since only 20 school principals from the seven different school types were interviewed in this study, conclusions and generalizations have to be drawn carefully. Nevertheless, our results are consistent with the results of similar studies, thus speaking to the validity of our results. In interpreting our results, it is relevant to consider when the interviews took place: the principals had already passed the school inspections and most of them had evaluated their results as satisfactory. Another school inspection was not to be expected for at least 5 years, and this distance may have led the principals to express overly positive statements and attitudes towards the QA.

As we exemplified in Table 3, more principals seemed to remember their attitude as being positive compared to the calculation of the principals’ attitude change. This fact hints at interesting processes in the anticipation and processing of feedback which definitely need further research and consideration. Still, the validity of the results is shown by the fact that the overall number of positive principals is nearly the same in the calculated attitude and the subjective estimation.

A last critical point should be mentioned: although we characterized some of the processes in the feedback context as rational or irrational in the introductory part of this paper, we have no measurement to differentiate reliably between the appropriateness of principals’ statements in the present study. To continue to disentangle the different levels of influencing variables in feedback processes the level of rationality may be more closely addressed.

Conclusion

The investigation of the ways in which the principals handled the feedback received from the QA NRW showed that external school inspections are linked to a complex and still developing field in which researchers as well as practitioners have only started to understand its underlying mechanisms and principles, nationally as well as internationally. The school system of NRW is currently characterized by a number of substantial and challenging situations and changes. The development and transformation of the QA into an instrument of support rather than control would help to change and shape the image of the QA and the inspectors even more positively and would therefore ensure that the schools would have even greater acceptance of the feedback obtained from the QA. A dialogical process between the level of educational leadership, quality management, and the principals should be initiated. Therefore, in order to avoid manifesting and reinforcing the hearsay that goes around the schools, it would be beneficial to create more official opportunities for professional exchange.

Another aspect is fundamental to the prospective success of the QA as well as the acceptance and recognition of the QA as a productive instrument for developing schools: more support is needed to help the schools implement the changes initiated by the feedback of school inspections and to ensure that they are not left alone with the results, the feedback, and the objective agreements obtained by school inspections.

This aspect goes along with the question, whether the acceptance of feedback is enough in order to support schools substantiate effective school development. Recent findings by Ehren et al. (2015) show that in order to allow schools to successfully integrate the results of school inspections, something more than feedback is needed. In line with psychological research regarding the use of feedback and results (Behnke and Steins 2015), as well as the results of Ehren et al. (2015), we argue that successful inspectorates may need to put forth high expectancies in combination with low-threshold support in a consistent manner.

Prospects

The questions regarding the extent to which school inspections can lead to the improvement of schools and the extent to which these measures of governance are accepted remain interesting for a number of reasons. On one hand, PISA results from 2003, 2006 and 2009 have consistently led to better results for German students. On the other hand, it may be too early to decide on the question, whether the introduction of new elements of accountability and educational governance have led to a substantial and permanent modernization of the German school system. Therefore, more research focusing on the efficiency of school development measures and school inspections is needed on both the international and national level.

Currently, the school inspection in NRW is being redesigned in order to allow the inspected schools more participation, such as to allow the schools to work with the inspectors to determine the main focus of the inspection. Research questions concentrating on the principals’ needs for support in transferring feedback by school inspection into school development measures as well as on the effects of more participation by schools in accepting and collaborating in the inspection process are part of the actual research projects. Furthermore, it is planned to focus on the role that the teaching staff plays in determining the acceptance of school inspections and their feedback.