1 Introduction

For more than two decades, the use of ICT has been a relevant factor for the subject-specific and interdisciplinary acquisition of competences besides the classical conditional fields of achievement at school such as students’ individual characteristics (among others Hattie 2009; European Commission 2014; Fraillon et al. 2013; Fraillon et al. 2014; OECD 2016; Wittwer and Senkbeil 2008; Voogt et al. 2013). Previous studies on the use of ICT (Information and Communication Technologies) have revealed that the existing findings on subject-specific learning outcomes are ambivalent. In many cases, previous studies on the correlation of the use of ICT and student achievement have emphasized that, apart from variables at the process level of teaching as well as teacher variables such as teachers’ attitudes and a teacher’s age, variables at the school’s process level must be primarily taken into consideration when it comes to describing and explaining under which conditions the use of ICT at school has an effect on the acquisition of competences. Furthermore, an overview of the relevant literature, most of all concerning the subject-specific acquisition of competences, illustrates that often times, previous studies are not able to conclusively tell under which conditions at school level there is a correlation between the use of ICT and students’ successful learning. In so doing, a number of existing studies record framework conditions at school in a well-differentiated way (e.g. Petko et al. 2015). However, they do not usually investigate subject-specific achievement or its change as a result of using ICT by means of achievement tests but only based on students’ self- or mutually assessed competences. Only very few studies make use of tests, however in many cases it remains unclear which competence concepts the applied tests are based on. Furthermore, the current state of international research indicates that there is a more complex relationship between the use of ICT and students’ subject-specific achievement and that so far school level factors are only rarely taken into consideration to explain these differences.

The contribution presented here starts out from these research gaps: It analyzes school-related factors affecting the use of ICT, thus focusing for the first time on school level factors which are considered jointly with the subject-related use of ICT as well as the subject-specific achievement of students, exemplarily by the subject of mathematics. The following analyses are carried out as secondary analyses on the basis of school and student data provided by the PISA 2012 study (Programme for International Student Assessment; OECD 2013a). In the context of an international comparison of five selected educational systems, the approach is based both on sophisticated information regarding school-related process factors and, as a result of the PISA competence tests and on data regarding the mathematical achievement level of students (Grade 9). The subject of mathematics may be taken as an example here as the relevant didactic literature in particular has already pointed out at an early stage that this subject shows a high affinity towards the use of computers for teaching, both concerning subject-related methods and contents (among others Aydin 2005; Barzel et al. 2005).

More specifically, this contribution considers school-level factors which, according to the current state of research, are particularly relevant, such as the IT equipment of schools on the technological level but also variables describing school leadership, school goals and educational strategies and the teachers’ attitudes. In a further step, the research presented in this article for the first time aims to analyze how these factors are connected to the use of ICT for teaching as well as how they are related to the mathematical achievement of secondary school students. As a methodological approach, a multi-level regression model and a multi-level path model are applied in order to identify how these framework conditions affect the use of computers and students’ achievement in the five educational systems (e.g. Hox and Roberts 2011). For the international comparison, which apart from yielding information on the different levels of the educational systems also provides an analysis of the relevance of the considered indicators, we take a closer look at educational systems which have a history of several years of experience when it comes to the implementation of new technologies: At the European level, Germany, Norway and the Netherlands are taken into account. Among the Asian countries, Singapore has been playing a particular pioneering role and has thus been selected as part of the comparison. What is more, Australia has particularly systematically supported the integration of ICT at schools for years. The selected educational systems allow for a comparative analysis of the above-mentioned school level factors and their relevance for teaching, as in the context of PISA 2012 they were identically operationalized in all five countries. Finally, the results of the PISA test are included and serve as an indicator for student achievement, in the context of which they are taken into consideration and modeled both at the individual level, as the mathematics skills of individual students, and at the school level as the mean mathematics achievement of the students of one school. Following a theoretical positioning, the structure and approach of this contribution consist of addressing the state of international research, the derivation and concretization of the research questions including sub-questions, and finally of presenting the results of the authors’ own analyses. As a conclusion, the hence gained results are discussed based on the theoretical and empirical foundation.

2 Theoretical allocation of school level determinants concerning the use of ICT for teaching and the relationship with student achievement as well as the current state of research

2.1 Theoretical background

Many studies on the efficiency of schools are based on models of school effectiveness and school quality which, based on theoretical approaches and empirical findings, provide a coherent model of several previously identified relevant conditions of school effectiveness. Not least, the orientation towards a more output-oriented management of educational systems which is observed all over the world has produced the result that the determinants of school effectiveness are described by means of so-called school effectiveness models, also called CIPO models (context, input, process, output) (Baumert et al. 2004; Creemers and Kyriakides 2008). These models devote a particular interest to the process level of school. Examples of such school level factors are, among others, the role of school leadership, staff development measures as well as structures of inner-school cooperation (among others Scheerens 2000). In view of developing school effectiveness, the main objective of school and class development at the process level is to identify those factors that schools can shape in a way that allows them to optimally support and promote their goals in the interests of (short-term) outputs and (long-term) outcomes with regards to the students’ competences.

Against the background of the transition to an information and knowledge society and the constantly growing possibilities of using new technologies, the use of ICT at schools is discussed as an important predictor of school effectiveness, as mentioned above (see among others Fraillon et al. 2014; Zuzovsky 2013). Whereas research at first focused primarily on the immediate subject-specific and interdisciplinary use of ICT and its possible potential to improve processes of teaching and learning as well as on teacher variables such as teachers’ attitudes (among others Owston 2003; Özgen and Bindak 2012; Petko 2012; Prestridge 2012; Schmidt and Köhler 2013; Tomczyk et al. 2015), current research shows that these aspects cannot be considered separately (e. g. Eickelmann 2011). Factors at school level are considered to be particularly effective for the sustainable and learning-supporting integration of ICT, and it is postulated that these factors are connected to the use of ICT and its effects on competence acquisition. However, due to the above-mentioned limited force of expression of many existing studies, there is no sufficient empirical evidence for these school-related factors. In this context, Eickelmann and Schulz-Zander (2008) developed a model of school effectiveness based on ICT as a comprehensive analysis model for the description and recording of the school effectiveness of ICT. Based on existing models of school effectiveness, this model includes factors at all levels of school, thus taking the specifics of the integration of ICT at school into consideration. Due to its focus on the field of ICT at school, it provides – following general models of school effectiveness – the theoretical framework for this contribution. Apart from input and contextual conditions, the refined version serving as theoretical allocation for this article also includes factors at the process level of teaching (e.g. on the change of the learning culture as a result of ICT) as well as at the process level of school (e.g. aspects of IT management by the school administration; formulating the relevance of the integration of ICT, among others for achieving the educational goals of the respective school). Thus, the theoretical approach deriving from school effectiveness research combines all factors which are currently considered to be relevant for the integration of ICT regarding the promotion of competences.

Based on this understanding, for the first time and by means of an international comparison, the present contribution provides an at least partially empirical assessment of the theoretical relationship between factors at the input, process and output levels of school. The contribution focuses on the already mentioned school level factors, combining them in an analysis model (Fig. 1) that depicts the complex connection structures.

Fig. 1
figure 1

Analytical model regarding the significance of school-related determinants for the teaching-related use of ICT in (mathematics) teaching and subject-specific achievement

Full size image

Based on an extensive review of literature, the analysis model combines the particularly relevant factors (see Section 2.2). They provide the foundation for the analysis presented in this article referring to the connection between conditions at school and student achievement in the subject of mathematics. Apart from the attitudes of school principals, teachers and school administrations, a suitable technological and spatial infrastructure together with a pedagogical concept towards ICT equipment which in the ideal case is aligned with the educational needs of each individual school is considered a necessary, however not sufficient condition for the integration of new technologies into processes of teaching and learning at school (among others Dynarski et al. 2007; Eickelmann 2011; NCTM 2011). Furthermore, the support of teacher cooperation and professional development of teachers, which can be significantly supported by school principals in terms of school management, has to be located at the process level of school (Drossel et al. 2016). The latter underlines why school leadership, also in the sense of taking on a power-promoting function, is relevant for the integration of ICT into everyday teaching (among others Albion et al. 2015; Hatlevik and Arnseth 2012; Hsu and Kuan 2013; Kafyulilo et al. 2015; Razzak 2015; Tondeur et al. 2008). Furthermore, when it comes to the relationship with aspects of organizational development, school strategies regarding the use of ICT play a key role, which in the context of defining school priorities, determining educational goals as well as the value of the use of ICT at school must be considered against the background of an established school culture (among others Pelgrum 2008).

2.2 School level determinants of the teaching-related use of ICT

In the following, relevant research findings will be brought together regarding the school-level determinants taken from the analytical model described above, which serves as the theoretical basis of the research presented in this article.

2.2.1 The role of the IT infrastructure: spatial and technological equipment

The IT equipment of a school is an important precondition for the successful implementation of ICT in processes of teaching and learning. Insufficient, unreliable or outmoded IT equipment, on the other hand, is considered an obstacle to the sustainable use of ICT for teaching (Eickelmann 2011; Owston 2003). Research in recent years has shown that in many countries, the IT equipment of schools has clearly improved (Fraillon et al. 2014; OECD 2015). However, at the international level there are still major differences concerning the availability of computer-based resources (Anderson and Ainley 2010; Fraillon et al. 2014). A current study by the European Commission (2013) was able to assess for Grade 8 that on the EU average, five students share one computer. The Scandinavian countries (Sweden, Norway and Denmark) as well as Spain show the best ratio in comparison to the EU as a wholeFootnote 1 (European Commission 2013). By international comparison, current findings for secondary schools show that more than two thirds of students attend schools whose school administrations report that learning at school is not or hardly affected by a lack or insufficiency of computers used for teaching purposes (66 %) or by a lack or low quality of Internet connectivity (79 %) (OECD 2013b). However, in this respect, occasional, considerable differences between different countries could also be detected. With regard to the countries highlighted in the following research, Germany for example can be located around the OECD average according to the two aspects under consideration, the countries of Singapore and Australia, both of which are also taken into consideration for this contribution, report significantly fewer problems. The situation in Norway, on the other hand, a country which, as has been mentioned, will also be taken into consideration for the following analyses, is slightly more unfavorable compared to the OECD average: Only about two thirds of students in Norway attend a school where the school administration reports no or hardly any problems resulting from insufficient computer equipment or Internet connectivity (ibid.). In the Netherlands, as the fifth country included in the following analysis, an above-average lack of computers for teaching purposes (54 %) is reported. The equipment feature of Internet connectivity, on the other hand, does not deviate significantly from the OECD average, according to the estimations by the school administrations (ibid.). Taking a closer look at the OECD comparison from 2003 to 2012, the review of IT-equipment problems in secondary schools further shows that in many countries, there is an occasionally considerable reduction of teaching restrictions initially caused by a lack or insufficiency of computer equipment.

Thus, both at the scientific level and at the level of school practice, it becomes obvious that problems of IT equipment show a continuous need for discussion, as there are a number of findings revealing that insufficient equipment prevents the use of ICT in the context of teaching and learning processes (European Commission 2013). Despite the significance of IT equipment for schools with respect to the support of computer-based teaching and learning processes, it cannot be assumed that the quantity of ICT resources at school is helpful in itself (Hattie 2009; NCTM 2011; Stoddard 2014). It is much rather the pedagogical alignment of the available IT equipment with the educational goals as well as the qualitative didactical integration into subject-specific learning processes that counts (among others Eickelmann 2011).

2.2.2 The relevance of school agents’ attitudes

The attitudes of teachers towards the use of ICT as well as the self-assessment of their own computer skills are pivotal for the successful integration of ICT at school (Lal 2014; Lawson and Comber 1999; Prestridge 2012). Usually, the integration of new technologies by teachers takes place in already existing, so-called teaching scripts meaning that the approachability of teachers with regards to new ways of learning becomes essential for the integration of new technologies into teaching and in this sense teachers are the key stone species for successful ICT integration (Davis et al. 2013). Teachers pursuing a constructivist approach of teaching and learning are especially willing to make use of ICTs and to integrate these into extended forms of teaching and learning (among others Ertmer 2005). In addition, teachers with a positive attitude seem to be more inclined to recognize the potentials of computer-based teaching (Celik and Yesilyurt 2013).

2.2.3 Findings regarding school leadership

School principals are considered one of the most significant factors when it comes to the successful integration of ICT at school (among others Anderson and Dexter 2000; Dawson and Rakes 2003; Dexter 2008; Eickelmann 2011; Ottestad 2013). Due to their function, they are capable of creating suitable framework conditions which permit or at least facilitate the teaching-related use of ICT by teachers (Tondeur et al. 2008). At that, the school principal’s support of a development of shared visions and goals within the school on how to use ICT for teaching and learning seems to be of particular importance (among others Dexter et al. 1999; Hughes and Zachariah 2001; Pelgrum 2008).

Other studies aside, the findings of the international comparative IEA SITES 2006 study (Second Information Technology Study, see Law et al. 2008), which explicitly refers to the use of ICT in the school subjects of mathematics and science at secondary schools, first and foremost show that school principals in a majority of the participating countries ascribe a high importance to the encouragement of teachers to teach their students how to use the Internet responsibly (Pelgrum 2008). These findings towards the relevance expressed by school principals have been confirmed by IEA ICILS 2013 (International Computer and Information Literacy Study, Fraillon et al. 2014). What is more, school principals in SITES 2006 considered the technology-related vision of lifelong learning to be relevant. In line with other international studies, the previously mentioned study by the European Commission (2013) that almost all Grade 8 students in Europe attend a school whose school administration agrees with the statement that new technologies and the competent use of new technologies are indispensable for students in the twenty-first century (ibid.).

Staff development and the promotion of professional development for teachers may also be described as important working areas for the school leadership. Due to the rapid technological progression, extra-occupational professional development for teachers has gained momentum with regards to the development of teachers’ competence (Abuhmaid 2011; Uslu and Bümen 2012). Contemporary professional development for teachers is considered a crucial determinant for the use of new technologies for teaching worldwide and emphasizes the relevance of school principals’ support (Dynarski et al. 2007; Fraillon et al. 2014; Galanouli et al. 2004). However, it could also be shown that teachers have a tendency to choose technological professional development over didactic training due to a lack of adequate training offers regarding the subject-specific integration of ICT into teaching (Law and Chow 2008). On the whole, however, it must be noted that with regards to staff development on school level, technology-related professional development can only be beneficial for an adequate use of ICT for teaching if other aspects, such as good IT equipment or sufficient technological assistance as well as school principals’ support, are simultaneously available at any given school (Williams et al. 2000). This underlines that the factors summarized by the analysis model of the present contribution cannot be considered independently of each other but must be jointly modeled and analyzed, as has been done in this paper. In doing so, this research article contributes to an important research question in the field of ICT integration into schools.

2.2.4 The relevance of school strategies for the integration of ICT

Another relevant field of school-level determinants for the use of ICT in the context of teaching and learning processes at school is the respective school’s strategies as well as its priorities when it comes to the use of computers. However, in the context of the SITES 2006, it turned out that the focus areas of technology-related priorities are highly divergent by international comparison (Pelgrum 2008). The improvement of the student-computer ratio as part of school strategies, for instance, has top priority at almost two thirds of schools in Chile, Israel and Thailand. In Finland, Singapore and Hong Kong, on the other hand, this is important for less than one quarter of the school administrations. These results show exemplarily how single factors such as school strategies, school leadership and IT equipment concur and that from an international perspective and by means of international comparison, differences can be identified. These and other findings derived from international comparisons show the necessity to adopt an international approach as it is the core issue of the research presented in this article.

2.3 On the relationship between the use of ICT and student achievement in mathematics

In recent years, the effects of the use of ICT on student achievement have already been the topic of a number of studies.Footnote 2 As a preliminary point, it is worth noting that the findings are quite ambivalent. However, the identification of directions of effect and causalities is highly problematic. In this respect, Heo and Kang (2009, p. 189) come to the following conclusion: „However, accomplishments that are convincingly the result of the direct causal impact of ICT use are not always easily identifiable.” Thus in most cases, due to the designs of existing studies, we can only speak of relationships and not of causal effects of ICT use on student achievement.

Concerning the operationalization of student achievement, however, we find both studies which operationalize subjective performance through self-estimations (e.g. E-learning Nordic 2006; Heo and Kang 2009; STEPS: Blamire 2009; Kang et al. 2011) and studies based on achievement data, such as data from final exams or international student assessments. For the present study, the latter studies are of primary relevance and will therefore be subsequently focused on. A study from Korea, for instance, analyzed the impact of ICT use on the achievement of students from different grades (Song and Kang 2012). By means of hierarchical linear modeling, the use of ICT was found to explain a significant share of the variance of achievement in mathematics operationalized by the achieved number of points in the national assessment both at primary and secondary schools. England was the subject of a more extended study dealing with the relationship between ICT use and students’ competences, operationalized by students’ performance in national final examinations. The ImpaCT2 project of the British Educational Communication and Technology Agency was able to generally show for three age groups (11, 14, and 16 years) at 60 schools in Britain in the subjects of English, mathematics and natural sciences that substantial ICT experience strongly correlates with higher rates of achievement in the national final examinations (Harrison et al. 2004). This result proves significant for the group of 11 year-olds in the subjects of English and mathematics as well as for the groups of 14 and 16 year-olds in the natural sciences. Internationally comparative student assessments from the past ten years equally provide an important data basis for the analysis of relationships between ICT use at school and subject-related student achievement but merely focus on the availability of ICT resources. Fuchs and Woessmann (2005) already analyzed the relationship between the availability and use of computers and student achievement based on PISA 2000 data. In doing so, they focused on the competence fields of mathematics and reading and included data from 31 countries into their analyses. Bivariate analyses indicated a positive correlation between the achievement of 15 year-olds and the availability of computers at school and at home. However, when the family background and school features were statistically controlled for, the correlation between a school’s equipment with computers and student achievement in mathematics and reading was no longer significant. Furthermore, the authors were able to identify a reversed U-distribution of computer and Internet use at school: student achievement hence increases in case of a moderate frequency of use, however it declines in case of a too frequent, weekly use.

The findings from the IEA study of TIMSS 2003 (Trends in Mathematics and Science Study) showed that in almost all participating countries the students’ achievement in mathematics and the natural sciences – above all that of 8th graders – was positively connected with the use of computers (Martin et al. 2004; Mullis et al. 2004). According to the results, the achievement in both domains is particularly high if students use computers both at home and at school. A similar picture was painted by the follow-up TIMSS cycle (Martin et al. 2008; Mullis et al. 2008). In 2005, the OECD published analyses based on PISA 2003 data both on the equipment and use as well as on the attitude of students towards ICT and its relationship with student achievement in the international context, using the title “Are pupils ready for a technology-rich world?” It could hence be shown that the achievement of 15 year-olds who frequently use computers at school is not necessarily higher in every country. In Germany, for example, there were no significant differences, however, in the United States, Canada and the Czech Republic, significant differences were found, even when controlling for the socio-economic status of the students’ families (OECD 2005).

A recent study by Skryabin et al. (2015), based on data from TIMSS 2011, PIRLS 2011 and PISA 2012 among others, analyzed by means of multi-level modeling to what extent the national ICT development level and individual ICT use influence the achievement of 4th and 8th graders in reading, mathematics and natural sciences. For secondary school students (Grade 8), the results showed that ICT use at school had a negative impact on students’ academic performance, which means that a more frequent ICT use at school correlates with lower achievement. The current findings by Kadijevich (2015), based on TIMSS 2011 Grade 8 data from 50 educational systems, also indicate a negative correlation between the frequency of computer use and the students’ achievement in mathematics. These findings might partly be explained by the continuously common didactical use of ICT as tool to support low achieving students.

Valuable hints in the context of ICT use at school and student achievement are also provided by the meta-analyses by John Hattie (2009) on computer-specific issues. Hattie was able to identify a mean effect size of d = 0.37 concerning the teachers’ use of computers (Computer Assisted Instruction, CAI) from 81 meta-analyses including 4875 studies. In this context, it is also emphasized that no grade-specific differences concerning the effects that CAI may have a positive influence on attitudes towards school and learning could be detected. CAI – used as a supplementary teaching strategy – shows more effect than CAI as a complete replacement of more or less traditional teaching strategies. Furthermore, the use of computers by teachers is more effective if the latter have been specifically trained and if the use occurs in a student-centered setting and within a cooperative learning environment. While recent studies on CAI such as Bayturan and Kesan (2012) as well as Cheung and Slavin (2013) have found positive effects of computer use on student achievement in mathematics, Tienken and Maher (2008) point out that the integration of ICT into existing curricula constitutes a precondition for sustainable effects on student achievement.

2.4 Research desideratum and guiding research question

Based on the presented state of research, the significance of the factors for the use of ICT selected above is emphasized. Nonetheless, the existing studies also point to clear research gaps: for the time being, there are, for instance, no representative and internationally comparative studies which at the same time consider school determinants for the use of computers in subject-related teaching. A lack of studies that additionally analyze correlations with student achievement based on competence tests is equally worth noting. The present contribution picks up on these research gaps: the goal of the analyses in this contribution is an in-depth investigation which is intended to both provide indicators of whether, and if so, of which features at the school level affect the use of ICT in mathematics lessons, and – on the other hand – determine whether there are correlations with the achievement of secondary students in mathematics. Thus, the present contribution is based on the following overarching research question: What effect do determinants at the school level have on the use of ICT in secondary school mathematics lessons and which relationships can be detected with the students’ mathematics achievement on a level of international comparison?

In this context, two sub-questions can be identified:

  1. 1.

    In which ways do determinants at school level (IT equipment, school leadership, school strategies and teachers’ attitudes) influence the students’ computer use in mathematics by international comparison?

  2. 2.

    How do the previously identified relevant conditions at school level and the computer use in mathematics lessons correlate with students’ mathematics achievement by international comparison?

To answer the first sub-question, a multi-level regression model is calculated on the basis of PISA 2012 data. To answer the second sub-question, a multi-level path model is specified. To allow for an international comparison and to work out similarities and differences between different educational systems, countries from different continents are included, namely Australia, Germany, the Netherlands, Norway and Singapore. The following section will elaborate in more detail on the research approach.

3 Empirical analyses of school-related framework conditions, the use of ICT in mathematics lessons and of mathematics achievement: secondary analyses on PISA 2012

3.1 Methodological design, sample and instruments

The present contribution makes use of the representative student and school data provided by the study of PISA 2012, which, as an international comparative educational achievement study, is carried out every three years as an educational monitoring for the purpose of addressing the achievement of the respective educational system in a context of international comparison (OECD 2013a). In the 2012 cycle, 65 educational systems participated in the assessment (OECD 2013a). Please already note at this point that due to its cross-sectional design, PISA does not yield information on causalities.

Apart from student assessments, which in 2012 focused on mathematics achievement of 15 year-old students, both PISA 2012 and the preceding cycles made use of extensive background questionnaires for students, which provided an important data basis for this contribution. What is more, the school questionnaire also plays a significant role for the research interest of this contribution. In this questionnaire, the school administrations provided important information on their schools’ basic data, but also on school strategies and school leadership (see ibid.).

To answer the two above-mentioned sub-questions, a multi-level regression model (sub-question 1) as well as a multi-level path model (sub-question 2) were specified (e.g. Hox and Roberts 2011) by the help of the Mplus statistics software (Version 7, Muthén and Muthén 2012). Appropriate weighting variables were included in these analyses in order to account for the complex data structure of PISA 2012: For the multi-level modeling by the help of Mplus, a student weight adjusted to the school factor (final student weight divided by school weight) was used on the individual level (within level). At school level (between level), the school weight was applied. The Full Information Maximum Likelihood method (FIML) was applied. Thus, no missing values were imputed, but population parameters and standard errors were estimated on the basis of all observable data (e.g. Enders 2010).

Table 1 provides an overview of the analysis sample (number of students and number of schools) used in this contribution with respect to the five educational systems under consideration. Cases with no valid values for any of the relevant variables were excluded.

Table 1 Analysis sample of the selected educational systems in PISA 2012
Full size table

Numerous indicators were used for the guiding research question. Table 2 summarizes the indicators taken from the school and student questionnaire of PISA 2012 that are of relevance to answer the stated research questions. Following the idea to determine the relevance of the school factors, these determinants deriving from the school questionnaire can be assigned to four fields: (1) a school’s state of computer equipment in relation to both the general extent of computer equipment – without further specification of the respective devices – and possible IT equipment issues, such as a lack or insufficiency of Internet connectivity, (2) school leadership, (3) school strategies, as well as (4) the teachers’ attitudes. However, as a constraint due to using an already existing data set and conducting secondary analyses the statements regarding a school’s equipment in particular do not immediately refer to the availability of digital resources for mathematics teaching but to the general state of equipment and thus-related problems at school level (see OECD 2013b). Moreover, information on the use of a computer by students for certain tasks of mathematics teaching is taken into account and derives from the student questionnaire. Students’ mathematical achievement is assessed using the PISA test results (Table 2) referring to the five Plausible Values of the overall scale of mathematics measured as a latent construct (for more information see OECD 2013a).

Table 2 Overview of indicators used taken from PISA 2012 questionnaires (school and student questionnaire) and PISA 2012 student test
Full size table

The students’ mathematics achievement, which is relevant for answering the second sub-question, has been modeled and included into the analysis model as a latent factor, consisting of the five plausible values of the overall mathematics scale. For the purpose of combining the instrument-related research approach and its theoretical allocation, Fig. 2 shows the fully differentiated analysis model underpinning the presented research.

Fig. 2
figure 2

Fully differentiated analytical model

Full size image

According to the explorative nature of the analysis carried out in the context of this contribution, input features and features at the process level of the respective school will be deemed of equal value in order to assess possible effects on the computer use in mathematics lessons and, at a later stage, to analyze such effects in the context of student achievement.

3.2 Analyses of computer use in mathematics lessons

Before analyzing the effects of school conditions on computer use at school as well as its relationship with student achievement in mathematics in the next section, the characteristics of computer use in mathematics lessons need to be clarified based on the PISA 2012 data set. These analyses were carried out using the IDB-Analyzer (Rutkowski et al. 2010). For the following analyses, an index including the students’ information on computer use in the context of mathematics teaching (e.g. for drawing graphs, see Table 2) has been established in order to operationalize the use of computers in mathematics lessons. By describing students’ computer use with this index, a high value of this index hence implies that students make use of computers for several different tasks. As depicted above (see Table 2), the students’ use of computers with respect to four mathematical tasks ((i) Drawing the graph of a function, (ii) Constructing geometric figures, (iii) Entering data in a spreadsheet, (iv) Finding out how the graph of a function like y = ax2 changes depending on a) has been included in the afore-mentioned index. The highest value (Category 4) means that the computer is used for all four tasks. Table 3 shows the frequency distribution referring to this index.

Table 3 Frequencies of students’ computer use in mathematics lessons based on an index considering four mathematical tasks (in percentages of students)
Full size table

The results show that the students’ computer use for several different tasks in mathematics is overall very low across the selected countries. An intensive use of computers for different purposes in mathematics, for instance, can be identified for a mere 2 % of students in the Netherlands and for approximately 10 % of students in Norway, the latter showing the highest rate compared to all countries taken into consideration for the analyses. Moreover, in Australia, Germany, the Netherlands and Singapore, Category 0, which represents no computer use at all in mathematics lessons regarding the four tasks, shows the highest national values in comparison to the other categories except for Norway where most of the students (36 %) performed one task using a computer. It is further striking that less than a quarter of students in the Netherlands and in Germany used the computer to perform at least one task (17.67 % and 25.2 % respectively), while around a third of students in Singapore (30.4 %) and Australia (35.82 %) have done so. Norway takes front rank with 7 out of 10 students using computers to perform at least one task (70.98 %). This is to say that students in Norway were four times more likely to use a computer to perform at least one mathematical task than students in the Netherlands. Having identified the characteristics of computer use in mathematics lessons, the following section of the paper will now address the school level determinants in order to gain further insight into what factors have an impact on computer use in mathematics teaching.

3.2.1 Analyses of the effects of school level determinants on the use of computers for mathematics teaching

The first sub-question of the research question focuses on the relevance of school-level determinants which affect the students’ use of computers in mathematics with respect to the five educational systems under consideration. Figure 3 shows the specified multi-level regression model, explaining the use of computers in mathematics lessons through determinants at school level. The model’s quality criteria are satisfactory (CFI = 1.00, TLI = 1.00, RMSEA =0.00, SRMRw =0.00, SRMRb =0.00).

Fig. 3
figure 3

Multi-level regression model explaining the use of computers in mathematics lessons through school-level determinants

Full size image

Key: bold type: p ≤ .001, ** p ≤ .01, * p ≤ .05; order of standardized coefficients on the paths: Australia/Germany/Netherlands/Norway/Singapore.

However, only few of the selected school-level factors prove to be relevant in explaining the use of computers for certain mathematical tasks at school. It further becomes obvious that in each of the educational systems under consideration, a different constellation of school-level factors proves significant, thus allowing for the identification of specific conditional structures. In Australia, the promotion of teacher professional development by school principals exerts a negative influence on computer use in mathematics (−.21). The teachers’ attitudes in the sense of open-mindedness towards new ways of teaching, on the other hand, can be identified as a highly significant contributing condition when it comes to the use of computers for mathematics teaching at school (.22). In addition, the results for Germany reveal that school leadership plays an entirely inferior role, if any, regarding the use of computers in teaching. Focusing on school strategies, a significant but negative effect of using the Internet for teaching could be detected with regards to the use of computers in mathematics at school level (−.30), i.e. schools that prescribed a use of the Internet during more than 25 % of class time across all subjects showed an overall lower intensity of computer use. Furthermore, a negative effect on computer availability, i.e. on the student-computer ratio with regards to mathematics lessons (−.51), manifests itself. This means that at German schools with a high degree of computer availability, which finds expression in a favorable student-computer ratio, computers are more frequently used in mathematics lessons. The Netherlands do not show a significant effect for any of the analyzed school-level determinants on the use of computers in mathematics lessons, which is a rather remarkable finding when it comes to school-level factors. In Norway, school goals which are supported and monitored by the school administration turn out to be relevant for the use of computers (.30). Furthermore, insufficient Internet connectivity positively correlates with the use of computers for teaching at school (.25), which might be a hint that schools use computers in mathematical lessons offline or use student-owned computers. In Singapore, school strategies regarding the use of the Internet for teaching prove to be a positive school-level determinant (.25). This means in this case that students who attend schools which have determined that the Internet be used in more than 25 % of class time across all subjects, spend more time using the computer in mathematics lessons for the purpose of performing various tasks.

On the whole, the findings show that for some of the selected educational systems the specified analysis model, taking into account a number of conditions at school, explains a significant proportion of the variance of student computer use in mathematics lessons at school. Showing a rate of 36 % of explained variance at school level, this is primarily the case in Germany. In Australia, the explained variance of computer use at school level amounts to 10 %. In the Netherlands, Norway and Singapore the specified model does not contribute to the explanation of student computer use in mathematics lessons and points to the assumption that in these countries, indicators that do not derive from the school level but e.g. from the individual teacher level play a major role.

3.3 Analysis of school-level determinants influencing the use of computers in mathematics lessons and correlations with mathematics achievement

To answer the second sub-question addressed in this paper, a multi-level path model is specified into which, as a supplement, students’ achievement in mathematics is included as a dependent variable. In doing so, the effects of school-level determinants both on the use of computers and on student achievement in mathematics are modeled. Furthermore, the effect of teaching-related computer use on the students’ mathematical achievement can hence be observed. Only predictors which, in the context of the analyses carried out to answer the first sub-question, proved to be statistically relevant for at least one of the educational systems under consideration were included into the model (Fig. 4). The quality criteria of the model are satisfactory (CFI = 1.00, TLI = 1.00, RMSEA = 0.01, SRMRw = 0.01, SRMRb = 0.01). As expected, the observation of school-level determinants’ effects on the use of computers at school, as depicted by Fig. 4, produces findings which are similar to those of the model answering the first sub-question.

Fig. 4
figure 4

Multi-level path model regarding the effects of school-level determinants on computer use in mathematics lessons and correlations between the students’ mathematics achievement by international comparison

Full size image

Key: bold type: p ≤ .001, **p ≤ .01, *p ≤ .05; order of standardized coefficients on the paths: Australia/Germany/Netherlands/Norway/Singapore; at the individual level the achievement in mathematics was controlled for the student background characteristics such as students’ gender, social background and migration background. The path was further specified to “achievement” instead of “computer use”. Due to the topical focus of the contribution, however, the coefficients are not reported in the model.

Concerning the relevance of school-level determinants, for the students’ average mathematical achievement, it can be observed that school strategies promoting the use of the Internet have a negative effect on the students’ mathematical achievement in Germany only (−.24). With regards to the IT equipment, one statistically significant effect can be identified for Germany, namely computer availability (−.39), and one for Australia: insufficient Internet connectivity (−.14). Concerning school leadership, a negative effect on students’ mean mathematical achievement was detected in the Netherlands: the assessment of school goals by school administrations (−.22). The teachers’ attitudes turn out to be significant in Germany only. Here, a positive correlation between the teachers’ open-mindedness towards new ways of teaching and the students’ average mathematical achievement is striking (.22).

Considering the significance of computer use in mathematics lessons at school for a school’s average achievement in mathematics demonstrates there is not a statistically significant effect in every educational system included in the presented analyses. Therefore, differences between the educational systems can be identified for this sub-question, too. Only in Germany (−.38) and in the Netherlands (−.56) does a negative effect of student computer use on the various tasks in mathematics lessons occur. For Australia, Norway and Singapore, no statistically significant effects of the computer use in mathematics lessons on the students’ mathematical achievement could be identified. The finding – inspite no significant effects have been found – is interesting because it means that at least in these three countries there is no negative correlation between computer use and students’ mathematical achievement.

Finally, the specified model explains a considerable proportion of the variance regarding mathematical achievement at school level primarily in the Netherlands (38 %). In Germany, it explains 23 %. In Australia, Norway and Singapore, on the other hand, the model does not show any explanatory power concerning students’ mathematical achievement at school level.

4 Conclusion, discussion and implications for further research

By means of an international comparison of five educational systems (Australia, Germany, Norway, Singapore and the Netherlands), the present contribution aimed to analyze which conditions at school level could be identified to have an effect on student computer use in mathematics lessons in secondary schools. Furthermore, the contribution aimed to investigate the correlations of computer use with students’ mathematics achievement by modeling the school level factors, computer use and achievement in a holistic multi-level path model. A first inspection of the current state of research initially revealed that although a body of research on individual school-level factors affecting the use of ICT for subject-related teaching at school as well as regarding subject-specific student achievement is available, there is still a lack of systematic, internationally comparative research on school-level determinants affecting the use of computers at school as well as their relationship with student achievement, assessed in the context of competence tests. This research gap is taken up on by the presented research, drawing on PISA 2012 data and focusing exemplarily on the use of ICT for mathematics teaching as well as on student mathematical achievement by international comparison.

Based on a theoretical allocation in the field of school effectiveness including new teaching approaches referring to the use of ICT in school, a differentiated analysis model was derived, by way of which four conditional fields considered can be situated. Among the considered school-level factors are: (1) the technological infrastructure//IT equipment, (2) school leadership, (3) school strategies as well as (4) the teachers’ attitudes which were aggregated at school level. As far as the student use of computers for the various subject-specific tasks in mathematics lessons is concerned, the analyses presented above have revealed significant differences between the five educational systems included in the analyses. In Norway, the strongest use of computers in mathematics lessons can be identified. In the Netherlands, on the other hand, more than 80 % of students make no use of computers for these specific tasks in mathematics at all. The new curriculum in Norway, introduced in 2006, which apart from introducing ICT-literacy as a key competence and extensively providing secondary schools with ICT also deals with the subject-related use of new technologies, may show its effect on school practice here.

When it comes to answering the first of two sub-questions relating to the relevance of school-level factors for computer use in mathematics lessons, the findings suggest that the model has considerable explanatory power particularly in Germany. Broadening the perspective to all five educational systems, it can be concluded that relevant determinants regarding the computer use in mathematics lessons at school level vary between the countries. However, only very few of the selected school-level factors have an effect. Whereas a sufficient availability of computers can be identified as a supporting factor in Germany, school strategies concerning the use of the Internet for instructional purposes have found to be hindering there. In the case of Australia, the teachers’ attitudes in the sense of open-mindedness towards new ways of teaching can be identified as a supporting factor. In Norway, it is mostly the school administrations’ emphasis on school visions that plays a supporting role when it comes to using computers at school, while in Singapore, the implementation of school-wide strategies regarding the use of computers for teaching turn out to be a supporting factor for computer use. Interpreting these findings, it may further be assumed that the long-term strategy of implementing ICT in schools, e.g. by using the eEducation Masterplan, come to fruition. This is in congruence with the debate of mathematical didactics which particularly underline the relevance of incorporating ICT into curricula and educational plans as a supporting factor at the level of the school system as well as the need for preparation and professional development of mathematics teachers. The result for the Netherlands, for which none of the analyzed school factors have proven statistically significant regarding the computer use for various mathematical tasks, might be interpreted as an indication that it is not so much characteristics at school level but rather other factors such as teachers’ competences and teacher education which are relevant for effective and competence-orientated use of computers in mathematics lessons. The results for Germany, showing that a good state of equipment constitutes a supporting factor at school level for the use of computers, underline the relevance of taking school-level determinants into consideration.

In the future, rather the quality and usability than the quantity of new technologies for teaching should equally be analyzed in-depth to follow up on questions regarding ICT equipment appropriate for teaching. In view of mathematics teaching one may consequently add that subject-specific equipment with ICT such as handhelds using subject-specific software or graphics calculators obviates the need for desktop computers in class, in particular if they are not immediately available in the classroom.

To answer the second sub-question, effects on student mathematics achievement were additionally taken into consideration. For the Netherlands, it turns out that the use of computers in mathematics lessons at school level is statistically highly significant and, at −.56, has quite a considerable negative effect on the average achievement of Grade 9 students in mathematics. To a lower degree, however with the same direction of effect, this finding can equally be observed in Germany. These findings are in line with previous studies which – using different analytical designs – were also able to identify negative correlations (among others Kadijevich 2015; Skryabin et al. 2015). In Australia, Norway and Singapore, no statistically significant and therefore also no negative effects of the use of computers for mathematics teaching at school on students’ mean mathematics achievement could be detected. Both on the national and on the international level, the most prominent need for action arises here: In addition to the promotion of interdisciplinary skills, such as ICT-literacy or computer and information literacy as they were measured by the IEA study of ICILS 2013 (see Fraillon et al. 2014), subject-related teaching needs to focus on learning strategies using ICT as the latter are supportive of subject-related learning. From a pedagogical point of view, it must be added that the frequency of computer use is an insufficient indicator of competence acquisition, as from a subject-related point of view the acquisition of mathematics skills depends on how computers are used (among others Aydin 2005; Barzel 2011). For the time being, this information is not provided by PISA, which is a quantitative study with a limited number of items on computer use in mathematics lessons. This constitutes a constraint to the understanding of computer use and a rather specific approach to the conceptualization and measurement of mathematical achievement. In this respect, both at the national and international levels, there is a need for further studies. Both qualitative and mixed-method designs may elaborate on the presented findings and go beyond the examination of quantitative features of computer use in mathematics lessons. Emphasis needs to be put on the fact that the findings of the study presented above are based on data which were collected in the context of a cross-sectional design. Consequently, the assumptions regarding the direction of effect, which are based on preliminary theoretical considerations in the present contribution and which have been assessed by means of hypothesis testing in the context of structural equation analyses, prospectively need to be examined in longitudinal studies or experimental designs. Such approaches may additionally require the inclusion of systematic observations on how computers are used by students during instruction in order to provide more precise information on both the direction of effect and causalities by way of critical-incident based evaluation using actually observed changes. Both the national and the international debate on the didactics of mathematics largely affirms the question of whether computers should be used for the teaching of mathematics (see Rackov 2011; Wiest 2001), although for the time being, there are few empirical studies on successful computer use (see Aydin 2005; Thomas, 2006). It may further be beneficial to prospectively entertain the idea of taking into account the transition to a knowledge and information society by means of differentiated, subject-related instruments in the context of internationally comparative studies such as PISA.

Furthermore, the interpretation of the findings requires a consideration of characteristics specific to the respective educational system, which are also characterized by the educational cultural differences between countries, for instance when it comes to teaching styles and teacher training (among others Tatto et al. 2012). Exemplarily, the comparatively poor didactical teacher training in Norway or the long-lasting tendencies of calculating work ethics in mathematics lessons in Singapore may be mentioned here.

In summary, it can be concluded that the internationally comparative assessment of the significance of school-level factors is by all means possible. However, the limitation consists of the fact that for a first empirical approach, as it has been made in the context of this contribution, the conditional fields of technological infrastructure and teachers’ attitudes as well as school management and leadership and school strategies on the process level of schools were included into the explanatory model with equal value and that additional effects of the input on the process level of school were not modeled.

For future analyses in this field, a multi-step procedure should be taken into consideration, in the context of which, however, the operationalization of the considered features should be refined and additional relevant variables at the student and in particular at the teacher level – such as teachers’ attitudes towards the use of ICT for mathematics teaching or the individual features of a teacher and their technological and educational pedagogical knowledge (among others Koehler and Mishra 2008) – should be included. The successful approach of this contribution, which assessed the effect of input features and school-level process features as of equal value with regards to the use of computers for mathematics teaching, allows for the contemplation that the model may differentiate more rigorously between the different levels than does the reality at schools.

Further limitations of the study can be found primarily in the operationalization of the considered constructs. This is due to the data basis of the study that includes secondary analyses of PISA 2012 data which were not explicitly collected for the research interest of this contribution. Prospective studies in this field may want to address school-level determinants in an even more differentiated way and may also want to draw on other data sources. Due to the design of the study, PISA 2012 namely provides only school data which was used to operationalize school-level determinants. An important extension of the available data for future studies on school-level determinants regarding the use of ICT for teaching would be the inclusion of teacher data instead of having to draw on school principal data.

In relation to the quantitative findings, comparative qualitative case studies in the five educational systems may also be promising in order to gain a further insight with regards to the determinants which have been identified as supporting or hindering and to generate knowledge which, in the sense of an internationally and inter-culturally comparative perspective, may yield valuable indications both for research and the further development of the individual educational systems against the background of the transition to a knowledge and information society including the associated challenges for schools and teaching. In this sense, the presented research might serve as a starting point for national and cross-national studies elaborating on the relevance of schools and school level determinants of effective ICT use in schools.