Research Question

What is an appropriate structure for reporting a study of the development of students’ understanding of covariation when using exploratory data analysis software, TinkerPlots Dynamic Data Exploration, following a pragmatist paradigm, and adopting educational design research methodology?

What Was the Study About?

The first objective of the inquiry was to further understanding of the factors that influence student learning when working with a graphing software package, TinkerPlots Dynamic Data Exploration, through the development of a conceptual framework for learning in exploratory data analysis graphing environments. The second objective was to explore the intersection between the students’ thinking and reasoning about covariation and the influence of TinkerPlots on that process, as students explore data sets to determine the relationship between variables, and identify trends.

What Was the Method?

The inquiry was a teaching experiment that employed a sequential exploratory strategy (Creswell 2003) and an educational design research methodology to facilitate the development of a systematic iterative study (Akker et al. 2006). It adopted an innovative qualitative research approach to capture the complexity of student learning of covariation, which is influenced heavily by the context of the technological learning environment, student prior knowledge, and instructional design (Konold 2007). These influences created a complex research/learning environment that recognised the research process and the learning intervention were intrinsically entwined—one constantly influenced by the other. Embedded within this was the idea that the students’ learning about statistical concepts, learning about data analysis software, and developing exploratory data analysis (EDA) skills (Tukey 1977) were similarly entwined. The research questions explored were:

  1. 1.

    How can the learning behaviours of students as they engage with exploratory data analysis software be characterised through a framework that can then be used to explore and analyse students’ understanding of covariation using TinkerPlots?

  2. 2.

    How do students interact with the exploratory data analysis software, TinkerPlots, to represent data in a variety of forms when exploring questions about relationships within a data set?

  3. 3.

    How do students develop an understanding of covariation in the exploratory data analysis software environment afforded by TinkerPlots and use these understandings to provide informal justification for their conclusions about the relationships identified?

What Were the Theories and Paradigms Employed?

The orientation of this research aligned with a pragmatist paradigm. Pragmatism seeks to link theory and praxis through the exploration of the research problem (Greenwood and Levin 2003; Mackenzie and Knipe 2006). It examines actions and situations to develop an understanding of the meaning of ideas by drawing on qualitative research methods and techniques (Creswell 2003; Johnson and Onwuegbuzie 2004). There is an emphasis on developing an understanding of what works and examining solutions to problems to derive knowledge about the problems (Patton 2002). It follows that the meaning of an idea or a proposition is developed by observing its application in real-world practice (Creswell 2013). Therefore, a pragmatist approach dictates that research methods are matched to the aim and purpose of the research and the specific questions of an inquiry (Boaz and Ashby 2003). Although a pragmatist paradigm is usually applied to scientific investigations (Mackenzie and Knipe 2006), it was appropriate for this inquiry as it allowed for the design of the inquiry to be shaped around investigating directly the use of TinkerPlots by students.

Educational design research was selected for the inquiry because it is iterative and cyclical in nature (Seeto and Herrington 2006; Shavelson et al. 2003). It is underpinned by a fundamental tenet that ongoing evaluation is an essential part of the research process (Kelly 2003; Phillips 2006). Educational design research has been used in many studies that interrogated and informed the design of technological learning environments (Reeves 2006). In the case of Seeto and Herrington (2006), it guided the development of a web site for accessing online learning. The purpose of their study was to create a collaborative research environment where software designers worked with education researchers to develop a set of design principles for the web-based delivery of teacher education courses. Typically, design research studies are longitudinal. The study conducted by Seeto and Herrington demonstrates that design research can also be an effective research approach when studies are short term.

The intention of the TinkerPlots inquiry was to take advantage of the iterative nature of educational design research (Shavelson et al. 2003), which involved using the outcomes from each stage of the inquiry to inform the next stage of the inquiry. Although the inquiry explored the students’ use of the software environment afforded by TinkerPlots (Fitzallen 2013; Watson and Fitzallen 2016) and was interested in the way in which the environment influenced students’ data analysis techniques (Fitzallen 2012, 2016), it was not the intention of the inquiry to make a contribution to the design principles of TinkerPlots.

A general model of educational design research includes four phases: development of the research questions, selection of data and data collection methods, design of artefacts and processes, and analyses and evaluation. It ‘is a research approach that is particularly suited to the exploration of significant education problems and technology-based solutions’ (Seeto and Herrington 2006, p. 742). Seeto and Herrington aligned their research methodology with the integrative learning design (ILD) framework developed by Bannan-Ritland (2003). The phases of the ILD framework are (a) Informed Exploration, (b) Enactment, (c) Evaluation: Local Impact, and (d) Evaluation: Broader Impact. The Informed Exploration phase may include activities such as conducting literature reviews, carrying out needs analyses, and determining the form of teaching intervention to be developed. The Enactment phase is characterised by the development, implementation, and refinement of the intervention over a number of cycles. Refinement of theories and suggestions for redesign arise from the Evaluation: Local Impact phase, while dissemination of data, evaluation of the impact, and consideration of the consequences of the intervention for the long term occur in the Evaluation: Broader Impact phase. The ILD framework is utilised by large projects that are expected to be delivered over a long period of time. The extended research period provides the opportunity for the implementation and evaluation of interventions to be iterative (Bannan-Ritland 2003).

The inquiry utilised the ILD framework developed by Bannan-Ritland (2003) to guide the inquiry design. In recognition that the inquiry was short term, as well as to accommodate the context of the inquiry, the titles of the phases of the ILD framework were modified. Consideration was also given to the educational design based research model used by Seeto and Herrington (2006), who implemented a study that explored the development of a web site for teacher education students. Although their study was relatively short term—less than one year—they applied the phases of educational design research successfully. Table 9.1 details the phases of the ILD framework, the Seeto and Herrington model, and their relationship to the phases developed for the TinkerPlots inquiry. The research followed four phases of inquiry consistent with educational design research methodology through seven stages of inquiry (Fig. 9.1). The figure maps chronologically the four phases of the inquiry process noted in Table 9.1 to each of the seven stages of the inquiry. Alpha-numerical codes are used for each phase of the inquiry to reflect the nature of each phase. For example, L0 includes the literature review for the initial stage of the inquiry, D0 includes the development of the inquiry design, E0 includes the evaluation of the inquiry design and links to E6. The link to E6 reflects the iterative nature of the inquiry. A0 includes the application of inquiry design and its implementation to the structure of the thesis.

Table 9.1 Phases of educational design research in relation to the inquiry
Full size table
Fig. 9.1
figure 1

Stages and phases of the inquiry

Full size image

Structure

The structure of the thesis was based on the seven stages of the educational design research inquiry. It was divided into seven sections with each section of the thesis for the first six stages of the inquiry opening with a literature review, as noted in the first phase of each stage detailed in Fig. 9.1. This was followed by information about the implementation of the other phases for that stage together with information about the methodological considerations relevant to that stage. Providing the detail about each stage of the inquiry as it arose in the thesis addressed the call from Collins et al. (2004) to characterise the elements of an inquiry design and state the reasons for including the elements in the inquiry process.

  • Contents page

    • Stage 0—Inquiry Commencement

    • Stage 1—Development of Model of Learning

    • Stage 2—Evaluation of TinkerPlots

    • Stage 3—Establishment of Prior Learning

    • Stage 4—Sequence of Learning and Outcomes

    • Stage 5—The Findings

    • Stage 6—Inquiry Conclusion

Stage 0—Inquiry Commencement. This stage set the scene for the inquiry and included the broad methodology that underpinned the inquiry and explained how the methodology fit the purpose of the inquiry. Inclusion of Stage 0 acknowledged the research that was undertaken in setting up the research project before its implementation. The aim and objectives, the research questions, the origins of the inquiry, the inquiry design, the sampling design, the significance of the inquiry, and a consideration of the ethical issues associated with the inquiry were also included. Following Stage 0, the next four stages detailed the enactment of the inquiry.

Stage 1—Development of Model of Learning. The purpose of this stage of the inquiry was to develop a theoretical framework, Model of Learning in EDA Graphing Environments (Fitzallen 2006), which exemplified the critical behaviours of working in EDA graphing environments from an interrogation of the literature on student learning about graphing and development of data analysis skills. The model was used repeatedly throughout other stages of the inquiry to inform the research process, evaluate research instruments, design criteria for the evaluation of TinkerPlots, and analyse student interviews. The model of learning was developed in response to Research Question 1 and is revisited in Stage 6 to determine in what ways it contributed to the inquiry meeting its objectives.

Stage 2—Evaluation of TinkerPlots. This stage was used to establish a clear understanding of the features of the software package and the different graph types it produces in order to answer Research Question 2. To do this it was necessary to develop criteria for evaluating TinkerPlots and then apply them to evaluate the potential for TinkerPlots to be used as a learning tool. The Model of Learning in EDA Graphing Environments developed in Stage 1, in conjunction with the literature on evaluating software and technological learning environments, was used to establish the criteria (Fitzallen and Brown 2006). The literature on previous research about TinkerPlots, its application as a learning tool, and its application as a teaching tool were also reviewed in this stage.

Stage 3—Establishment of Prior Learning. The purpose of this stage of the inquiry was to establish the prior learning of students in relation to their understanding of graphs, graph-sense-making, and covariation. In order to do this, an assessment instrument to be used as a student survey was developed. Proven assessment items from previous research that evaluated students’ development of statistical and graphing concepts were used to construct the student survey (Fitzallen 2008). The results from the administration of the student survey informed the design of the sequence of learning experiences developed and implemented in Stage 4—Sequence of Learning (Fitzallen and Watson 2014). The results also informed the selection of the participants for Stage 4.

Stage 4—Sequence of Learning and Outcomes. The purpose of this stage of the inquiry was to develop and implement a sequence of learning experiences that would provide the opportunity for novice learners to use TinkerPlots to develop an understanding of covariation. As part of the implementation of the sequence of learning experiences, the final session was used to administer an interview protocol to gather evidence of the students’ understanding of covariation and determine the way in which they interacted with TinkerPlots to create graphs and interpret data. Student profiles that characterised their statistical thinking and reasoning according to the dimensions of the Model of Learning in EDA Graphing Environments were developed for the students who participated in this stage of the inquiry (Fitzallen 2012, 2013). The student profiles contributed to answering Research Question 1 in Stage 5.

Stage 5—The Findings. In Stage 5 the student profiles built from the results in Stage 4 were analysed and used to answer Research Question 1. The student profiles were then analysed another two times to answer Research Questions 2 and 3, respectively.

Stage 6—Inquiry Conclusion. This stage revisited the research design adopted for the inquiry and the Model of Learning in EDA Graphing Environments developed in Stage 1 as part of an evaluation of the inquiry design using the principles of scientific inquiry developed by the National Research Council (2002). The implications and limitations of the inquiry were developed in this stage of the inquiry as were recommendations for future research.

Commentary

The thesis, Reasoning about Covariation with TinkerPlots, used the structure of the inquiry design to structure the thesis. It was set out to follow the design process through Stages 0–6 as chapters of the thesis. The decision for the structure of the thesis to follow the inquiry design was made so that the thesis reflected the evolving exploratory nature of the inquiry, which is in keeping with design-based research approaches (Akker et al. 2006). Setting out the thesis in this way allowed the thesis to demonstrate clearly how each stage of the inquiry was developed from the literature and how the literature was used to inform each stage. It also allowed the factors that influenced the enactment of each stage to be made explicit at the time they were relevant to the inquiry, which satisfied Collins and his colleagues (2004) call for elements of an inquiry be justified. All of the conventions for a regular thesis such as literature review, methodology and results were included in the thesis; however, they were not presented as individual chapters.

The thesis was organised so that the literature reviews and methodological considerations for each of the stages were presented at the beginning of each stage. Figure 9.2 provides a representation of the relationships among the phases and stages of the inquiry and indicate which research question was the main focus of each stage. The main connections are indicated by the bold lines and arrows. It is important to note that there are many other subtler connections represented by dashed lines. Although not directly responsible for the outcome of a preceding or following stage, information from a particular phase impacted on other stages or phases of the inquiry. For example, evaluation of the students’ prior learning in Stage 3 was used to inform the development of the sequence of learning experiences in Stage 4. The map also places the Model of Learning in EDA Graphing Environments developed in Stage 1 in the centre and indicates its relationship with the other stages of the inquiry.

Fig. 9.2
figure 2

Inquiry map

Full size image