Keywords

1 Introduction

The direct observation of gravitational waves in recent years has generated much excitement. One could say that there has been a sense of exhilaration. Given that gravitational waves were theorised by Einstein a century ago, and indirect observations had been made, what was so special about the direct observations? After all, the exercise was about confirming theory, not unlike school science and undergraduate physics experiments. Or was it? While a lot was known, a lot was experimental in the true sense of the word. The criticality of experimentation is captured by the following quotes.

The principle of science, the definition, almost, is the following: The test of all knowledge is experiment. Experiment is the sole judge of scientific ‘truth’.

Feynman, 1963, Lectures in Physics, 1-1 (Feynman 2019)

I have often had cause to feel that my hands are cleverer than my head. That is a crude way of characterising the dialectics of experimentation. When it is going well, it is like a quiet conversation with nature. One asks a question and gets an answer, then one asks the next question and gets the next answer. An experiment is a device to make nature speak intelligibly. After that, one only has to listen.

George Wald, Nobel Lecture (12 Dec 1967) (Wald 1999)

Can such experiences be recreated in undergraduate laboratory programs? The quest has been on to capture such experiences in experimentation in both school and undergraduate physics experimentation. Various names have been coined, authentic, inquiry, to the practice of physics. In fact, looking back historically, the quest to emulate ‘science as practised by scientists’ and to capture learners’ curiosity by unravelling the mysteries of nature is long standing, see, for example, (Schwab 1960; Driver 1983; Hart et al. 2000). Curricula and pedagogies have been, and continue to, earnestly design ways of engaging students through inquiries, practical work and hands-on investigations. Identifying effective practices and measuring effectiveness is paramount. With changing student demographics, increasing use of technologies and changing nature of education, the need for educators to research the practice of experimentation and hands-on investigations in school and undergraduate curricula and pedagogies continue to be fruitful and essential pursuits.

In this paper, I summarise two decades of work within the Australian context. Most involves partnerships with physics educators within the Australian Institute of Physics, the Australian Science Teachers Association and various science education networks. The work is situated within the Sydney University Physics Education Research (SUPER) group. In particular, I will focus on:

  1. a.

    How do we design ‘effective’ experiments? How do inquiry, authentic and practice surface in the design?

  2. b.

    What does it mean to measure ‘student experiences of experimentation’? How do we measure? How do we interpret results to further improve experimentation?

  3. c.

    Finally, if the experiences are tenuous and ill defined, how do we assess?

But first, I would like to discuss what ‘effective’ means? ‘Effective’ could purposefully have different meanings in different curriculum and pedagogical contexts. For example, in open-ended projects, ‘effective’ could prioritise students demonstrated competence in design, deployment and processes rather than outcomes. In specialised circuits experiments, priority could be given to dexterity with equipment and technical troubleshooting. In other experiments, priority might be given to data collection, analysis and interpretation. Another priority area is comprehension and application of uncertainty. In most cases, a range of skills and knowledge is involved, and some are selectively prioritised to build a jigsaw of learning opportunities for students. Different methodologies are needed, ranging from case studies, evaluation, to quasi-experimental methodologies. Each methodology would predicate instruments, tools, data collection and analysis. Needless to say, ‘what we are striving to teach students is what we need to practice in researching student learning of experimentation’.

2 How Do We Design ‘effective’ Experiments?

Experiments are an integral component of both school science and undergraduate physics education (Sharma et al. 2021,2008; Hofstein and Mamlok-Naaman 2007). Designing ‘effective’ experiments’ is not a trivial task. I will present one pedagogical tool which we have found useful in designing and/or revising school science as well as university undergraduate physics experiments. The approach is based on skills underpinning experimentations which appear in most curricula and science pedagogies, namely (1) questioning/predicting/hypothesising; (2) planning including planning data collection; (3) conducting and recording; (4) processing, analysing and evaluating; (5) reasoning, problem solving and connecting with science; and (6) concluding and communicating. These skills are shown in the first column of Table 1.

Table 1 Advancing science and engineering through laboratory learning (ASELL) schools inquiry slider; from Cornish et al. (2019)
Full size table

As one moves from the left to right of Table 1, there is a progression from directions being provided by the teacher or resources, to the learner being more self-directed. As students take responsibility of their own learning of those skills, they shift from ‘being’ directed’ to ‘self-directed’. The last row shows terms coined for the progression, such as guided inquiry. The pedagogical tool can be found in different formats with variations in the way in which the terms are used. Nevertheless, the conceptual basis is widely accepted. In the form shown in Table 1, it is called the ASELL Schools Inquiry Slider (Cornish et al. 2019; Gordon et al. 2015). The skills shown in Table 1 can readily be shaped, recast and situated within different curricula across all sciences internationally. Not all experiments lend themselves to be open-ended and not all experiments should be. Experiments will not map vertically to each level, not every student will be as adept at each skill. Hence, the Inquiry Slider provides a teaching tool, with some teachers also using it as an assessment tool. It is also a tool for programming school science and for providing variety of inquiries being offered to students.

I present two examples of the use of the inquiry slider which underpins the ASELL Schools project (ASELL Schools Link 2021). First, Science in Your Pocket (Gordon et al. 2019) in which students start off by using an APP on a mobile device to ‘measure light’ followed by generating their own question which they seek to investigate. Second, Vampire Power (Kota et al. 2019) in which students start of by selecting an appliance from those available and recording power readings on a spreadsheet prior to generating their own question which they investigate. Both of these integrate digital technologies and have a level of independence in the second half of the experiment. These are two-part experiment where the first part is to ease students into the ‘space of the experiment’ and to ‘connect the whole class’, and the second part is for the main experiment. The whole class activity on a spreadsheet enables the class to have a shared experience at various stages of the experiment. These experiments are normally completed within 60 min in a school science laboratory. We have run these across Australia in many schools, together with other experiments also based on the inquiry slider, see ASELL Schools project (ASELL Schools Link 2021) for more detail.

For two decades, the School of Physics has been running open-ended projects, often referred to as open inquiry, (Sharma et al. 2014) with large cohort size of around 1000 first-year undergraduate students. The projects integrate into a fairly standard first-year laboratory program. During the first four weeks, students working in teams carry out electricity/circuits experiments with 30 min each week to plan their project guided by tutors. They submit a project proposal and carry out the project in the next 4 weeks. A fleet of formative assessment tasks are deployed to support students culminating in a presentation and written report.

New research is purposefully redesigning the ways by which students engage with experiments by moving theory into appendices so that students explain their findings, rather than confirming the theory presented. We are incorporating strategic instances of ‘stories’ and ‘colour’ focusing on emotional engagement. Perhaps, the most exciting is the ways in which we are considering the teaching and learning of uncertainties and integrating digital technologies.

3 What Does It Mean to Measure or Evaluate ‘Student Experiences of Experimentation’?

The words measure and evaluate are distinct. When measuring, an instrument or tool is specifically designed to measure attributes with a level of validation that the instrument is measuring what it sets out to measure. An evaluation is a broader and more diffuse scoping, often exploratory but guided by questions and a purpose. In some instances, we have evaluated while in others, we have measured.

Here, I start off by discussing the evaluation of the open-ended projects mentioned above through observations and surveys (Sharma et al. 2014). We were scoping if the learning objectives were identifiable by the students when they answered Likert scale items and through open-ended responses. We were also exploring for themes we had not anticipated. The learning objectives which ranged from students being able to undertake independent research to critical interpretation of their results, received predominantly positive ratings on the Likert scale items. The most prominent themes were ‘intrinsic nature of projects’ and ‘their teams, including working with tutors’. These themes attracted substantive numbers of positive responses as well as fewer numbers of negative comments. The aspect which stood out the most was the role of the tutors, their support was welcome but could also be directive and intrusive. The inquiry slider (Cornish et al. 2019; Gordon et al. 2015) provides a pedagogical tool for the tutor to manage their interactions with their teams. The most pleasing aspect was that the phrase ‘critical thinking’ was not in the survey, but students spontaneously self-reported, critical thinking, whatever it means, and however, it was defined in their minds.

A different national Australian project designed a specific instrument, the ASELL Student Learning Experiences (ASLE) survey to measure students ‘learning experiences’ of a particular experiment immediately after students had completed that experiment (Barrie et al. 2015; Yeung et al. 2011). By ‘learning experiences’, we mean tangible pedagogical aspects of the experiments as designed by the academic in charge, as well as notions of ‘experiences’ from the literature on motivation. ASLE was initially trialled with 3153 chemistry students from Australia, US and New Zealand (Barrie et al. 2015), followed by 2691 students from a range of disciplines including physics (Yeung et al. 2019). The two-factor theory of motivation (Herzberg 1968; Bassett-Jones and Lloyd 2005) has been used as an interpretive framework. After checking for assumptions, exploratory factor analysis was used to extract two distinct factors: ‘experiment-based motivators’ and ‘course-level resources’, see Table 2.

Table 2 ASLE items and the factor loadings [from Yeung et al. 2019]
Full size table

So, what is the big deal? When correlating the items with ‘overall learning experience’, we find that all the items in one factor have a similar pattern. However, the pattern for ‘experiment-based motivators’ is distinctively different to ‘course-level resources’, see Fig. 1. Our analysis suggests that the items in the ‘experiment-based motivators’ align with student satisfaction with their experiences. Those in the ‘ course-level resources’ appear to be more subtle; if not done well, these give rise to student dissatisfaction, but once at a certain level do not contribute to further satisfaction. The key message for teachers is to invest in items on the motivators to continue improving student experiences while those in the other factors will not influence student experiences after a certain level.

Fig. 1
figure 1

Distinctly different patterns when correlating items with ‘overall learning experience’ for the two factors; a experiment-based motivators are different to b course-level resources. The data points are from different disciplines represented by different shapes, but do not influence the pattern (Yeung et al. 2019)

Full size image

A final survey, the ASELL Laboratory Program Evaluation (ALPE) focusing on learning experiences of semester-long laboratory programs, has been developed and deployed with 9790 students, in physics and four other disciplines. The essence of most of the items is largely unchanged but have been edited to align with the laboratory program. A few items which are not relevant to the semester-long laboratory program have been removed and replaced with items on ethics and communication. Preliminary analysis suggests that the two factors are still consistent with robust reliabilities and loadings. The conceptual basis of the second factor is now framed around graduate qualities and capabilities. This is an important finding because it is not unusual to find that in curriculum mapping exercises, graduate qualities are mapped onto laboratory programs. Our finding that students self-report on the ALPE their experiences of graduate qualities in the laboratory program is reassuring.

4 How Do We Assess Students?

This is an eternal quest. How do we assess process skills, including those which are often times referred to as soft skills? In a cross-sectional study, we gave first, second and third-year students the same task at the beginning of the year (Richardson et al. 2008). With respectable sample sizes using qualitative coding as well as marking, we found that there is a progression in the ‘levels of sophistications' as students advance through their years doing experiments. In particular, their dexterity with handling equipment and technical troubleshooting improved, as did data collection and analysis skills. There was demonstrated improvement in handling of uncertainty and interpretation of their results. In other words, their experimental skills specific to physics were improving, building their jigsaw of disciplinary expertise.

Currently, we are deliberately aligning each experiment based on constructive alignment (Biggs 2003) and the ASELL Inquiry Slider (Cornish et al. 2019; Gordon et al. 2015). The skills are articulated collectively, giving rise to three assessable tasks:

Conduct and collect data.

Analyse, including uncertainties.

Interpret.

Each experiment has three learning outcomes, see Fig. 2 for an example.

Fig. 2
figure 2

Example of the articulation of learning outcomes, aligning with inquiry slider, for an experiment

Full size image

The experiments contain some well-defined, recipe-type sections while other sections are guided. There is a trajectory of development such that when the students get to the open-ended projects (Sharma et al. 2014) as discussed earlier they should be ready to undertake self-directed experimentation.

Each of the learning outcomes is assessed by an individual, but relatively low stakes, test. The first learning outcome is assessed via a practical test. There is a week when each of the 1000 students book a 40 min slot to undertake a hands-on practical test. Eight practical tests are made available beforehand. The beauty of this task is that students analyse which test will be easiest for them, and during the analysis, they invest a lot of effort and time into learning, which they would not have done otherwise. The second learning outcome is assessed via a mid-semester test which contains questions on lecture material as well as uncertainties and using spreadsheets. Students get to practice uncertainty through weekly online questions which attract a minuscule amount of marks, and all of these questions become available for students to revise prior to the mid-semester test. The third learning outcome is assessed via an individual laboratory report which is uploaded through TURNITIN and checked for plagiarism. While students work in teams of three during their sessions, for the report they need to select a section of one of their experiments for the report. They choose the experiment and particular section with an eye on the data and its analysis. Again, the selection and decision making makes them self-assess their work and see how they have developed through the semester. Each student from a team selects and re-analyses different data sets and mostly from different experiments. They also need to upload an image taken with their phone of the excerpt signed by the tutor of the raw data. The system has been running for two years now and has increased student engagement, reduced complaints and improved students ratings of the laboratories and the courses. We are yet to evaluate and/or measure other parameters. What we are also focusing on is students emotional engagement (Bhansali and Sharma 2019): an untapped avenue in most science education research.

5 Discussion and Conclusion

In conclusion, our measures have provided collective evidence that experimental programs are worthy. In particular, we as practitioners can strive to capture what enthrals experimental scientists in our undergraduate programs (Feynman 2019; Wald 1999; Schwab 1960; Driver 1983). This study shows that we can ‘measure student experiences of laboratory learning’ and use our measurements to iteratively improve student learning experiences (Sharma et al. 2014; Barrie et al. 2015; Richardson et al. 2008). A key challenge is engaging our colleagues in our quest as they often have good intentions but competing demands on their time. Various professional development opportunities have been designed and implemented in the Australian context (Cornish et al. 2019; Yeung et al. 2011). Our efforts are now focused on articulating with university-level key performance indicators, such as graduate attributes, so as to entrench the status and need for experimentation in the sciences. Further erosion of investment and support for experimentation must be halted. We believe that we as practitioners should embark on a campaign to gather and utilise solid evidence aligning with senior management goals to halt the erosion of genuine undergraduate experimentation.