Keywords

1 Theoretical Background

In this paper, we firstly explore some theoretical constructs around the notion of ‘problem-solving’. We particularly invoke Bloom’s revised taxonomy and literature around critical thinking. We then explore the difficulties that students typically experience in learning how to problem solve. Finally, we report on a project during which the correlation between the dimensions of problem-solving and student performance was calculated. It is difficult to determine what knowledge and skills first year programming students possess prior to their programming course. The main objective of computer programming is to implement programs that solve computational problems. In [3, 10] we can find that problem solving ability is an indicator of programming performance. Critical thinking, also referred to as problem solving, reasoning or higher order thinking skills, can be defined as “disciplined, self-directed thinking which exemplifies the perfections of thinking appropriate to a particular mode or domain of thought” [18] and also as “a process of gathering and evaluating data to make decisions and solve problems” [15]. Taxonomies of learning have been implemented worldwide to describe learning outcomes and assessment standards reflecting what learning stage a student is at. The original learning taxonomy developed by Bloom and several of his colleagues in 1956 identifies six levels of thought [2]:

  • Knowledge: rote memorization, recognition, or recall of facts;

  • Comprehension: understanding what the facts mean;

  • Application: correct use of the facts, rules, or ideas;

  • Analysis: breaking down information into component parts;

  • Synthesis: combination of facts, ideas, or information to make a new whole;

  • Evaluation: judging or forming an opinion about the information or situation.

These levels of thought start from the lowest order process to the highest order process with higher levels building on lower levels [2]. Once a student reaches the highest level they can be said to have grasped a subject matter. Bloom’s Taxonomy was revised in 2011 to address the differences between comprehension and application and to better define the term evaluation. The changes made to the revised taxonomy are as follows [1]:

  • Remember (previously ‘knowledge’);

  • Understand (previously ‘comprehension’);

  • Apply (previously ‘application’);

  • Analyse (previously ‘analysis’);

  • Evaluate (previously ‘evaluation’);

  • Create (previously ‘synthesis’).

According to [9, 12], students learn to write complete computer programs in their first year of a programming module which falls within the top two levels of Bloom’s Revised Taxonomy of teaching and learning [1]. These two levels however, depend on the first four levels before a student is said to be able to grasp computer programming. For example, in computer programming, ‘learning syntax’ is the lowest order process [11] and efficiently utilising syntax in order to ‘produce effective computer programs’ is the highest order process [4]. Accordingly, lecturers expect students to be able to write programs within the first few weeks of their programming module [6]. These programs may be basic and get more difficult as the module progresses, however, many students may be left behind whilst still struggling to find solutions to basic problems. This means that the difficulty level at which the programming module starts, is already at too high a level for a novice programmer, which can lead to a lack of motivation and ultimately a student failing the module. Although novice students may have little experience programming, they do have experience solving problems in everyday life [19]. Problem solving is a mental process of analysing a given problem, developing a solution to the problem and presenting the solution [13]. When students solve problems either independently or in collaboration with other students they are learning by doing. While learning by doing is synonymous with problem solving [13] computer programming as a discipline is also synonymous with problem solving. However, according to [7], students struggle to solve problems for the following reasons:

  • Students do not fully understand the problem either because they have not interpreted the problem statement correctly or they just want to start writing code.

  • Students fail to transfer the knowledge that they have already acquired from past problems over to new problems.

  • Students who take too long to find a solution just give up trying and wait for the solution to be given to them.

  • Many students do not have enough mathematical and logical knowledge.

  • Students lack specific programming expertise and struggle to detect simple syntactical and logical programming errors.

According to [3], more attention should be paid to novice programming students’ problem-solving abilities by encouraging them to practice problem solving, as learning to solve problems algorithmically contributes to learning to program. Students need to think about the processes they go through in solving everyday life problems and look at how to use the same processes to develop algorithms, for example: “they need to identify things that are familiar to them, divide the problem into smaller problems and use existing solutions” [5] — the very same things that [7] identifies as what students struggle with. The purpose of this paper is to establish what correlational relationship exists between students’ problem solving ability and their academic performance in first-year level programming courses.

2 Method

The participants of the study were a group of 186 first year students enrolled for the National Diploma in Business Information Technology (NDBIT) at the University of Johannesburg (UJ), and 193 first year students enrolled for the National Diploma in Information Technology (NDIT) at the Tshwane University of Technology (TUT).Footnote 1 The research process was preceded by a thorough literature review.

2.1 Instrumentation

Four programming aptitude tests (which measured a student’s ability to problem solve) were used with permission from the University of Kent Careers and Employability Service Department. All tests were completed in a test-like setting with hired venues and appointed invigilators.Footnote 2 Ten items from each test were used to reduce the load on students and due to time constraints.

Fig. 1.
figure 1

Logical reasoning test questions

Full size image

Logical Reasoning Test. The first programming aptitude test was the logical reasoning test which involved letter sequences and tested the students’ ability to think logically and analytically. The test involved looking at a specific sequence of letters and working out the next letter of the sequence: see the example in Fig. 1. The average score for the logical reasoning test was 6.7/10.

Non-verbal Reasoning Test. The second programming aptitude test was the non-verbal reasoning test which determined a student’s ability to understand and analyse visual information and solve problems using visual reasoning — for example: identifying relationships, similarities and differences between shapes and patterns, recognizing visual sequences and relationships between objects, and remembering these. The non-verbal reasoning test enabled students to analyse and solve complex problems without relying upon or being limited by language skills. The test involved looking at a specific sequence and working out the next member of the sequence from the pictures given: see the example in Fig. 2. The average score for the non-verbal reasoning test was 4.65/10.

Fig. 2.
figure 2

Non-verbal reasoning test questions: In the first example question the top row of four boxes make up a series from left to right. You have to decide which of the 5 boxes underneath, marked A to E, will be the next in the sequence. For example in the first example, the top four boxes have 1, 2, 3, and 4 dots respectively. Obviously, the next box in the sequence will have 5 dots, which is box D. Answer 1\(\,=\,\)D

Full size image
Fig. 3.
figure 3

Numerical reasoning test questions

Full size image
Fig. 4.
figure 4

Verbal logic test questions

Full size image

Numerical Reasoning Test. The third programming aptitude test was the numerical reasoning test which included mathematical questions: see the example in Fig. 3. The average score for the numerical reasoning test was 3.24/10.

Verbal Logic Test. The fourth programming aptitude test was the verbal logic test which included verbal logic puzzles, some of which had a numerical element. This test, tested the students’ ability to think logically, analytically and numerically, and also to extract meaning from complex information: see the example in Fig. 4. The average score for the verbal logic test was 2.79/10.

Programming Examination Results. Data were also collected from the examination results of the students programming module, Development Software 1: UJ — Development Software 1A (DSW01A1) and Development Software 1B (DSW01B1) and; TUT — Development Software 1A (DS0171AT) and Development Software 1B (DS0171BT). Student numbers were used as the key field to link the data sets. The Development Software 1 (DS1) results were used as the dependent variable throughout the study.

3 Data Analysis

The four programming aptitude tests for logical reasoning, non-verbal reasoning, numerical reasoning and verbal logic were correlated with the DS1 final mark of the students. The results of the computation are presented in Table 1.

Table 1. Correlation of programming aptitude tests and DS1 mark
Full size table

3.1 Logical Reasoning

A Pearson product-moment correlation coefficient was computed to assess the relationship between the logical reasoning test mark variable and the students’ performance in DS1 variable. Logical reasoning refers to a student’s ability to think logically and analytically. There was a small, positive correlation between the two variables, \(r = .199\), \(n = 341\), \(p = .000\). Overall, there was a small, positive correlation between the non-verbal reasoning test mark and performance in DS1.

3.2 Non-verbal Reasoning Test Mark

A Pearson product-moment correlation coefficient was computed to assess the relationship between the non-verbal reasoning test mark variable and the students’ performance in DS1 variable. Non-verbal reasoning refers to a student’s ability to understand and analyse visual information and solve problems using visual reasoning. There was no correlation between the two variables, \(r = .095\), \(n = 347\), \(p = .078\). The results for this group show an insignificant correlation between students’ non-verbal reasoning ability and performance in DS1.

3.3 Numerical Reasoning Test Mark

A Pearson product-moment correlation coefficient was computed to assess the relationship between the numerical reasoning test mark variable and the students’ performance in DS1 variable. The numerical reasoning test included mathematical questions. There was a small, positive correlation between the two variables, \(r = .257\), \(n = 348\), \(p = .000\). Overall, there was a small, positive correlation between the non-verbal reasoning test mark and performance in DS1.

3.4 Verbal Logic Test Mark

A Pearson product-moment correlation coefficient was computed to assess the relationship between the verbal logic test mark variable and the students’ performance in DS1 variable. The verbal logic test included logical, analytical and numerical questions. There was a small, positive correlation between the two variables, \(r = .143\), \(n = 341\), \(p = .008\). Overall, there was a small, positive correlation between the verbal logic test mark and performance in DS1.

4 Conclusion

The University of Kent’s Careers and Employability Service Department assesses student’s computer programming aptitude with tests measuring competencies such as numerical reasoning, logical reasoning, verbal reasoning and non-verbal reasoning which are required in computer programming jobs. These tests were adapted for this study. The findings show that there is a correlation between a student’s logical reasoning (\(r = .199\), \(p = .000\)), numerical reasoning (\(r = .257\), \(p = .000\)) and verbal logic (\(r = .143\), \(p = .008\)) and performance in computer programming modules. This supports findings of earlier studies [8, 14, 16] telling us that problem solving ability is a major predictor of performance in programming courses. The correlation between students’ non-verbal reasoning and performance in computer programming modules was, however, not significant. This could be because the ability to use pictures in thinking is to a large degree a matter of practice, not aptitude [17]. A similar finding was reported in [14]; in this study Computer Science 1 (CS1) students were identified as having experienced difficulties with: decomposing problems, developing sufficient solutions, and re-using previously seen solutions (even for elementary problems). To this end they introduced the course Development of Algorithmic Problem-Solving Skills (DAPSS) to be taken in parallel to studying CS1. The main focus of DAPSS was to set aside the details of the programming language and concentrate on reflective processes, awareness to problem-solving behaviour and development of cognitive skills. Results showed that the DAPSS course had a positive effect on students’ problem-solving skills which in turn improved their programming skills [14]. It is thus recommended that the teaching of problem-solving skills at the University of Johannesburg (UJ) and the Tshwane University of Technology (TUT) be introduced as part of the programming module, to provide opportunities to enhance students’ programming performance and thinking processes.