Introduction

Physics learning has great potential to develop students thinking skills. Learning content, from concrete to abstract, allows students to organize knowledge from simple to complex. The learning process-oriented to the inquiry process with a scientific approach practice the ability to think critically (critical thinking skills) and creative thinking (creative thinking skills). In addition, the object of physics observations, a problem-oriented in the surrounding environment, indicates that learning physics exercises problem-solving skills in a structured way (algorithmic thinking). Thus, learning physics is an excellent alternative to practice higher-order thinking skills (HOTs).

Practicing of HOTs in physics learning is based on the context and content of learning, the complexity of the material, the learning process, and the evaluation implementation (Bao and Koenig 2019). In experimental-based physics learning (EBPL), all these components become a single unit that students use continuously where context, content, and physics material are integrated in experimental activities. Hence, optimation of EBPL designing can practice the HOTs easier.

However, the development of learning physics looks relatively slow, particularly in the development of experimental activities to practice HOTs. The availability of laboratory equipment and teacher skills are obstacles to developing EBPL. This condition is complicated by the emergence of new competencies along with technological developments. New competencies are shown in changes to the 4C framework (critical thinking skills, creative thinking skills, collaborative skills, and communication skills) (Kivunja 2019) become a 6C framework (Critical Thinking Skills, Creative Thinking Skills, Communication Skill, Collaboration Skills, Compassion, and Computational Thinking) known as A Global Framework of Reference on Digital Literacy for Indicator 4.4.2 (UNESCO 2018).

Innovative physics learning has a unique characteristic when students apply computational logic to solve the problems. Flexibility and accessibility to technology can support students to develop their potential independently (Lye and Koh 2014). Hence, teachers and practitioners have an opportunity to develop CTs through learning physics. However, this is constrained by the agreement on conceptual and operational definitions of CTs (Kong and Abelson 2019; Román-González et al. 2017; Shute et al. 2017). Not only in course learning, the same thing also happens in physics or science learning which emphasizes exploration through experimental activities.

The framework, so far developed, is designed for highly theoretical learning with a numerical orientation, such as in computational physics (Cross et al. 2016; Dwyer et al. 2013; Moreno-León et al. 2020; Moreno León et al. 2015; Nuraisa et al. 2019; Yağcı, 2019). Hence, the process of measuring students CTs with experimental-based learning (EBL) characteristics, such as in EBPL, does not yet have a specific framework. In addition, this problem needs to be solved, bearing in mind that without proper measurement references and indicators, students' CT will not be mapped properly and accurately.

This article discusses the framework for assessing students' CTs especially to EBL and EBPL. There are two research question which are: (1) What is the measured aspect of CTs in physics education? (2) What criterion for cognitive question for assessing CTs?

Computational thinking skills

Computational thinking skills (CTs) are additional competencies that need to be prepared to support digital literacy. CTs were initially introduced to solve cognitive processes using computer programming rules (Lye and Koh 2014). Currently, with the very rapid development of education, the CTs has been adapted to various problems outside the context of programming (Wing 2008), especially in physics and mathematics.

In physics learning, the ability to think computationally (CT) becomes very important considering the vast role of physics in the development of technology globally (Adeleke and Joshua 2015). In addition, learning physics also practice students to solve theoretical and contextual problems. (Nadapdap et al. 2016). In solving problems, a physicist, including students, must go through systematic stages as characteristic of CT (García-Peñalvo and Mendes 2018).

CTs refer to thinking skills that require students to be able to (1) formulate problems computationally and (2) formulate the best solution through clear procedures (algorithms) or state reasons regarding the factors that cause solutions cannot be found. (Moreno-León et al., 2016). These two main characteristics of CTs have been indirectly taught in physics learning (Shute et al. 2017).

Decomposition

Decomposition is a way of thinking by breaking down complex problems into specific problems (Shute et al. 2017). In learning physics, students can describe the problem more simply. Simplification of the problem can be done by identifying the main problems that can be solved with the available information (Cross et al. 2016). The essential ability to perform decomposition is analyzing and evaluating.

Redefine problems

Identifying problems in learning physics is an early indicator that students can solve problems (Cross et al. 2016; Shute et al. 2017) Students are expected to be able to determine the main problem to be solved by referring to the availability of information presented (Cross et al. 2016). The essential ability to perform redefine problem is analyzing and inference.

Modularity

Modularity is the ability of students to classify information based on the subject matter (Kong and Abelson 2019; Shute et al. 2017). Modularity indicators require students to choose concepts with the information needed to solve problems (Cross et al. 2016). The essential ability to perform modularity is analyzing, evaluating, informing, and explaining.

Data representation

Data representation refers to two forms of ability: understanding data in various representations and presenting data in various representations (Kong and Abelson 2019; Moreno-Leon et al. 2016). The essential ability to perform data representation is interpretating and analyzing.

Abstraction

Students who can think computationally must have the ability of abstraction. Abstraction can generalize solutions from a problem to other problems with similar roots (Moreno León et al. 2015). The essential ability to perform abstraction is analyzing, synthesizing, predicting, and evaluating.

Strategic decision-making

Students who can think computationally must have the ability to make strategic decisions. This indicator requires students to solve complex considerations (Cross et al. 2016). The essential ability to make strategic decisions is analyzing and evaluating.

Computational thinking skill in physics learning

The nature of physics education is reflected in a series of physics learning activities. The implementation of physics learning that needs to be considered is practicing skills according to current global competencies, one of which is CTs. Increasing CTs in physics learning can be practiced by developing supporting skills, i.e. giving questions ranging from simple to complex (Voskoglou and Buckley 2012). The problems presented in the questions are intended to access information related to student performance and guide the course of learning (Indrawati 2010).

CTs in learning physics are seen in learning activities where students are faced with abstract problems. Students cannot solve physics problems without a precise sequence of activities. Complex and abstract problems will usually be broken down into more straightforward problems as a generalization reference. After that, physicists will use a systematic method to find a solution to the problem. Finally, after the solution has been obtained, the physicist will make a generalization of the results to be used as a general reference for solving similar problems. Hence, the activities in learning physics follow the characteristics and indicators of other CTs.

In the EBL, for example, decomposition activities are carried out by students when identifying the objectives of the experimental activities to be carried out. Furthermore, in the problem-solving laboratory (PSL) and higher-order thinking laboratory (HOT Lab) experimental model, students are asked to identify problem to direct experimental activities in formulating solutions (Malik 2015; Malik et al. 2018). Simultaneously, preliminary activities also indirectly practice the redefine problem indicators. Hence, the emphasis on analytical skills is essential in learning physics to practice decomposition and redefine problems indicators.

EBL is presenting many variables related to data analysis. However, not all variable should be considered to solve the problems. CTs emphasize this aspect where students are required to only focus on relevant variables and not be distracted by other variables. The ability to select and focus variables in EBL is called modularity (Kong and Abelson 2019; Shute et al. 2017).

The EBL provide so many information in various representation i.e. numbers, tables, or graphs. Thus, it needs the ability to understand all these representations. Data processing on a computer is capable of extracting data from various representations. On that basis, indicators data representation are essential to be mastered by students studying physics (Gebre 2018; Kohl et al. 2007; Moore et al. 2020). In addition to understanding data from various representations, physics students are also required to make generalizations from their findings. Generalization aims to make general conclusions and will be the initial assumption for future researchers. Generalization or abstraction in physics learning also aims to communicate the findings during experimental activities so that readers will more easily understand the findings.

The last indicator is algorithmic design and strategic decision-making. These two indicators are characteristics emphasized in the CT framework. This statement refers to the pattern of a computer system that is systematic, effective, and efficient. The characteristics of the indicators of algorithmic design and strategic decision-making in physics learning determine the most effective solution to the problems. Physics or science learning, in general, produces many possible solutions to a problem, but using a structured pattern such as in algorithmic design, students will be able to make effective and efficient strategic decisions.

Method

This research is cognitive instrument development to examine the framework research using a psychometric analysis approach. The research stage consists of three main stages. At the initial stage, the researcher developed questions following the defined indicators of CTs. The second stage is instrument testing. The instruments that have been compiled were tested to obtain information about the item and test characteristics. The last stage is the implementation stage as the final data collection. The items in the implementation test are questions that have passed the selection in the previous stage. The data on the implementation test were analyzed using Explanatory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA).

The sample in this study was 354 senior high school students aged 15–17 years, which were selected randomly. It is certain that each student has studied the physics material being tested, in this case, the sound wave material, and is involved in independent EBPL during the pandemic. 108 students participated in the initial instrument test to obtain the characteristics of the instrument items. In the EFA test, 108 students were used as research subjects, and in the CFA test, there were 113 students.

Results and discussion

This article discusses the framework for assessing students computational thinking skills. The framework developed is intended for EBPL. According to the characteristics of physics learning, which depends not only on the description of the material but also the ability to collect data, analyze data, and describe the findings.

The CTs framework developed is different from the previously developed aspects. Yağcı (2019) is developing a framework for assessing CTs that generally apply essential components in the form of problem-solving skills, algorithmic thinking skills, critical thinking, collaborative learning, and creative thinking. Five aspects that serve as a reference for the development of Yağcı (2019) then analyzed in-depth by integrating the opinions of Moreno-Leon et al., (2016) which states that there are four taxonomies of CTs in science learning, namely data practice, modeling, and simulation practices, computational problem-solving practice, and systems thinking practice (for more information see Cross et al. 2016; Moreno-Leon et al. 2016; Wing 2011; and Yağcı, 2019). By considering the learning characteristics, seven indicators representing four aspects of computational thinking skills in physics learning are obtained: decomposition, redefine problems, modularity, data representation, abstraction, algorithmic design, and strategic decision-making.

Item question

The items were developed by referring to the seven assessment indicators developed. 22 item questions of sound wave was developed are a mixture of multiple-choice, complex, and essays. Sample questions for each indicator are listed in Table 1.

Table 1 Sample item question
Full size table

According to the theoretical analysis, this study declare that the characteristics of the items for CTs cognitive test in science or physics on consisted of several rules that are (1) the stimulus has a complex problems; (2) the stimulus is in the form of multiple representations; (3) the problems presented are contextual; (4) emphasizes higher-order thinking skills; (5) the formulation of the solution must be general and applicable, and (6) problem solving which is expected to trigger the ability of analysis, evaluation, synthesis, and creation.

Instrument characteristics

The instrument items that have been developed are then tested to obtain information about the characteristics of the items. This study used the Item Response Theory (IRT) 1PL approach to estimate item characteristics. Thus, only one parameter is estimated, namely the difficulty level of item (b). The 1 PL estimation is based on the Chi-Square score in Parscale (PSL) and Quest software. The Chi-Square value for 1PL estimation is more significant than 2PL, while for 3PL, it cannot be estimated. The results of 1PL estimation using PSL software can be seen in Fig. 1 and Table 2.

Fig. 1
figure 1

Item fit MNSQ

Full size image
Table 2 Summary statistics of test estimates
Full size table

Figure 1 shows the suitability of the items in the estimation of 1 PL which requires the INFIT MNSQ score range to be 0.77 to 1.33. Based on Fig. 1, all items match the 1PL model so that each item is stated to be empirically valid (Boone et al. 2014).

Furthermore, based on Table 2, the student's ability as measured using the questions in the developed framework is 0.02 logit. Conversion of logit scores to a scale of 100 using the equation Score = 50 + 10z resulted in the average score obtained by students being 50.34. With a standard curve, this value indicates that the items have good quality because they can produce estimates of students' abilities in the medium range.

The estimated reliability values in Table 2 indicate that the developed test can measure students' abilities well. This score is indicated by a reliability score of 0.71 with the" Fair" interpretation (Fisher Jr 2007). The result of the subsequent analysis is the parameter estimation for the developed test. These are listed in Table 3 for tests and Fig. 2 for items.

Table 3 Summary statistics of parameter estimates
Full size table
Fig. 2
figure 2

Item parameter information

Full size image

Table 3 shows that the overall level of item difficulty (odd ratio) of the CTs test questions with the developed framework is 0.626. This score has a problematic tendency because it has a score of more than 0.00. However, in the item difficulty range, this score is still quite acceptable with the aim of mapping students' abilities, not for selecting subjects (Chan et al. 2020).

Figure 2 shows the level of item difficulty, discriminating power, and guessing factor for each item. As seen in Fig. 2, the most difficult items are items number 9, 13, and 14 (in figure number 10 to 12), which represent algorithmic design and strategic decision-making indicators. At the same time, the items that show the lowest level of difficulty are numbered items that represent modularity.

Figure 3 shows the correlation between the average score and students' abilities. While this is not the only problem inherent in interpreting test scores, we focus on regression to the mean because it is widely ignored and misunderstood. (Smith and Smith 2005). Overall, the questions developed in the computational thinking ability test could estimate students' abilities in a suitable category.

Fig. 3
figure 3

Bivariate plot

Full size image

Figure 4 is the total test information collected from the information function on each item. Based on Fig. 4, the test items developed could estimate students' ability with an ability range of − 2.00 to 2.75. This value indicates that the questions and the framework developed can estimate students' abilities over a wide range (Boone et al. 2014). A wide range of measurements in the framework of computational thinking skills makes it possible to estimate students' abilities reasonably, considering that computational thinking skills tend to think at higher levels by solving complex problems. (Adams and Wieman 2015; Lye and Koh 2014; Nuraisa et al. 2019; Shute et al. 2017).

Fig. 4
figure 4

Test total information

Full size image

Explanatory factor analysis (EFA)

Table 4 shows that the KMO value is more significant than 0.05. This score indicates that the number of samples used is sufficient to analyze the data. According to Hair (2009), the number of samples is related to determining the loading factor. With a sample size of 100–120 students in the EFA, the minimum loading factor required is 0.50.

Table 4 Summary EFA analysis
Full size table

Based on the criteria set by Hair (2009), The resulting factor loading is declared valid with a significant correlation > 0.5 for a sample of 100–120 people (Table 5).

Table 5 Rotated component matrix
Full size table

Confirmatory factor analysis (CFA)

In order to validate the interpretation of scores by evaluating a stand-alone fit model, it is necessary to use a fit index for which good cutoff criteria, or at least some rules of thumb, are available. Other characteristics of the fit index need to be taken into account, particularly sensitivity to sample size and model specification errors (Sun 2005).

CFA testing is carried out by distributing questions that match the 1PL model and meet the criteria in the EFA. The model's fit based on the CFA is shown in Table 6.

Table 6 Characteristic of CFA goodness of fit
Full size table

Based on Table 6, all of the indicators resulting from the SEM analysis have the goodness of statistics better than the recommended value. This indicates that the current model is statistically adequate. Then, the results of the SEM analysis and the significance of all factor loading (Sun 2005).

The p-value = 0.25, RMSA = 0.037, CFI = 0.94, IFI = 0.96, NFI = 0.74, GFI = 0.95 and AGFI = 0.87 also indicate that the index of conformity from the CFA analysis is at an acceptable level. (Marsh et al. 2020). These findings indicate that the four-factor model obtained as a result of the CFA has a sufficient level of fitness.

As can be understood from the relevant literature, the results of the EFA and CFA analyzes are acceptable. The findings related to the validity and reliability of the scale indicate that the scale can be used to measure the computational thinking ability of students at the secondary school level, especially in learning physics or science. It is believed that this study will significantly contribute to the literature, considering that there are only a handful of valid and reliable measuring tools to measure computational thinking ability. (Korkmaz et al. 2017; Yağcı 2019).

Conclusion

The framework for assessing computational thinking skills in physics learning with characteristics in EBL consists of decomposition, redefining problems, modularity, data representation, abstraction, AQ2 algorithmic design, and strategic decisionmaking. The CTs question has some rules that are (1) the stimulus presented in the form of complex problems; (2) the stimulus AQ3 presented is in the form of multiple representations; (3) the problems presented are contextual; (4) emphasizes AQ4 higherorder thinking skills; (5) the formulation of the solution must be general and applicable, and (6) problem solving which is expected to trigger the ability of analysis, evaluation, synthesis, and creation.