Microworlds, games, animations, mobile apps, puzzle editors and more: What is important for an introductory programming environment?

Abstract

Teaching and learning programming constitutes a challenge. Although several teaching approaches and programming tools have been proposed, it seems that they have limited impact on classroom practice. This article investigates students’ perceptions on five educational programming environments that are widely used and the features that any introductory programming environment should have. The environments investigated are: BlueJ; objectKarel; Scratch; Alice; and MIT App inventor. These environments were studied and used by experienced undergraduate students of Informatics in the context of a fourth year course. The main features of the environments and the way of presenting them to students, as well as the assignments in the context of the course are presented, in order to help the reader realize what experience was gained by the students that evaluated the environments. Based on a questionnaire filled in by students interesting conclusions were drawn. Students identified the main features of the environments and evaluated them positively, although problems were identified. An introductory programming environment should engage students through the development of programs connected to their interests, such as games and mobile apps. Moreover, an ideal introductory programming environment should provide a simple and user-friendly Graphical User Interface (GUI) that supports visualization of objects and classes, includes a puzzle-like editor for program development, reports simple and understandable error messages in natural language, and finally the ability to execute the program in a step by step manner. Although no single environment fulfils all these features, it seems that the most successful environment is Scratch.

1 Introduction

Teaching and learning programming constitutes a challenge both for instructors and students respectively. Extended research has been carried out the last 4 decades on the teaching of introductory programming and several teaching approaches have been proposed and specially designed tools have been devised. However, this research seems to have limited impact on classroom practice, as Pears et al. (2007) state in their highly referenced paper “A survey of literature on the teaching of introductory programming”. This survey covers four aspects of teaching introductory programming: curricula, pedagogy, language choice and tools for teaching. Tools for teaching are grouped in three categories: visualization tools; automated assessment tools; programming environments. Programming environments - that are of interest in the study presented in this paper - are grouped in two subcategories: programming support tools that support students in developing programs using a rather standard language and execution environment and microworlds that are based on physical metaphors and concrete actors (objects) that are depicted during program execution.

It must be stressed that Pears et al. (2007, p. 209) did not intend to provide a survey of the aforementioned tools, but as they state their aim was to “give an outline of the field” and “provide examples of work in several sub-fields of tools research that have had demonstrated success or that we consider potentially important in the future”. Of course, other papers surveyed programming tools used for novices. These surveys, which categorized programming tools in different categories, are briefly reviewed in the next paragraphs in order to give a clear picture of the field.

One of the first surveys and critical analysis of tools devised for the teaching and learning of problem solving and programming was carried out by Deek and McHugh (1998). The tools were classified in four categories: visually-based and text-based programming environments; debugging aids; intelligent tutoring systems; and intelligent programming environments. A critical analysis of a big number of tools falling in the aforementioned categories showed that they did not manage to fulfill their goals.

In a subsequent study, Mark Guzdial (2004) investigated the answers to the question “What makes programming hard?” that are provided by programming environments for novices that aim at making the learning of programming easier. In his study, three families of environments that were considered to have influenced the development of programming environments at that time were studied: the Logo family of environments having its roots on Lisp; the rule-based family of environments that originated from Logo, Smalltalk-72 and Prolog; and the traditional programming language family of environments. This study revealed some trends regarding novice programming environments, namely: using more traditional language syntax; giving the chance to work on meaningful computational artifacts; provide immediate feedback on students’ work. Concluding, “the progress in the field is toward making programming more interesting, more relevant, and more powerful for students” (Guzdial 2004, p. 152).

However, evaluating such features of programming languages and environments is not an easy task. Furthermore, there is no universally accepted framework for evaluating programming languages and environments. McIver (2002) has proposed one of the most referenced methods for evaluating programming languages, evaluating programming environments in isolation, as well as comparing and evaluating Integrated Development Environments (IDEs), especially for novices. Although, McIver (2002) does not propose specific criteria the following categories of features are proposed: GUI, Editor, Debugger, Error messages, Visualization, Individual method execution and object construction and Students' evaluation.

Georgantaki and Retalis (2007) carried out a comparative analysis of educational tools used for teaching object-oriented design and programming, which was based on the approach proposed by McIver (2002). Georgantaki and Retalis (2007) reviewed: programming microworlds; and educational programming environments. Moreover, a reference is made to: tools for enhancing the capabilities of programming environments, such as Expresso for enhancing error messages; software tools that support an “objects-first” teaching approach, including environments such as BlueJ and Karel microworlds; the Robocode gaming environment; and educational programming languages.

All the aforementioned studies surveyed, compared and/or critically analyzed various programming environments for novices. As mentioned these environments were categorized differently. The two main categories are:

• Programming microworlds (Georgantaki and Retalis 2007; Guzdial 2004; Pears et al. 2007). Guzdial (2004) refers to this type of environments using two subcategories instead of the term microworlds: the logo family; and rule-based family of environments.

• Educational programming environments (Deek and McHugh 1998; Georgantaki and Retalis 2007; Guzdial 2004; Pears et al. 2007). Various terms are used to refer to this kind of environments: visually-based and text-based programming environments (Deek and McHugh 1998); traditional programming language family of environments (Guzdial 2004); and programming support tools (Pears et al. 2007).

and additional categories are:

• Visualization tools (Pears et al. 2007)

• Automated assessment tools (Pears et al. 2007)

• Debugging aids (Deek and McHugh 1998)

• Intelligent tutoring systems and programming environments (Deek and McHugh 1998)

A detailed taxonomy of both programming languages and environments for supporting novices - that was devised based on the pedagogical goals of each tool and not just its technical features - is provided by Kelleher and Pausch (2005).

Both this taxonomy and the aforementioned surveys have made clear that a great number of tools have been developed and are still being developed. David Valentine (2004) in a meta-analysis of SIGCSE Technical Symposium Proceedings for the period 1984–2003 investigated the kind of CS Educational Research carried out at the CS1/CS2 level. Valentine found that 22 % (99 out of 444) of the papers described tools for CS1/CS2. It is clear that not all of these tools have been used in classroom and as a consequence have not been evaluated. Even for the environments that assessment has taken place it is not always easy to replicate and validate the conclusions drawn (Gross and Powers 2005). Of course, assessing an environment is not an easy task. Common shortcomings with assessment of introductory programming environments are the following (Gross and Powers 2005):

• The research questions posed for the environment are too vague and/or open to interpretation.

• The impact of an environment on students’ learning process and understanding is not usually studied.

• The assessment takes place under opportunistic and not realistic conditions. For example the grades achieved in a course selected by the students themselves are analyzed.

• The assessment is carried out by the developers of the environment that have expertise and deep knowledge of the pedagogy behind the environment and consequently achieve the best possible outcomes.

In this paper, we present an evaluation of five educational programming environments by a group of undergraduate students attending a course on “Educational Programming Environments and Techniques”. In the context of this course students used, studied relevant literature and carried out an assignment in each one of the following environments: the best known “objects-first” IDE of BlueJ; the microworld and learning environment of objectKarel; the game-oriented environment of Scratch; the 3D animation and story-telling environment of Alice; and the mobile app oriented environment of MIT App Inventor. These particular educational programming environments were chosen from a wide variety of environments based on three main criteria. Specifically, our aim was to use introductory programming environments that: are representative examples of the main categories of environments presented in the previous paragraphs; incorporate a wide range of features that aim to support novices in different ways; and have an impact on classroom practice. After the utilization of the aforementioned environments throughout a whole semester students evaluated the pros and cons of each environment; the pedagogical contexts that each environment is most suitable for; and finally the features that every programming environment targeted to novices should have in order to assist them in learning programming. The novelty of this work lies in the fact that five representative educational programming environments that fall into different categories were evaluated under the same conditions by the same group of last-year students having prior experience in programming languages and environments, as well as the difficulties of learning programming.

The rest of the paper is organized as follows. In section 2 a description of the usage of the environments in the context of the course is provided, including a review of the main features of the five educational programming environments and the way that these features were presented and utilized by students. The methodology of the evaluation carried out in the context of the course is presented in section 3, while in section 4 the results are analyzed. Next, in section 5 a discussion of the results and the main conclusions of this study are presented.

2 Usage of the environments in the context of the course

The course “Educational Programming Environments and Techniques” is offered as an elective course at the 7th semester of an 8 semester bachelor degree on Applied Informatics. The aim of the course is to support the teaching and learning of programming: (i) technologically through the appropriate educational programming environments; (ii) by studying the difficulties, errors and misconceptions of students recorded in the literature. The course includes 12 lectures and the environments that are studied are the following: BlueJ, objectKarel, Scratch, Alice and MIT AppInventor. In the following paragraphs we present the content of each lecture, the literature studied and the assignment for each one of the aforementioned environments. The aim of this section is to present: what the main features and didactical rationale of each one of the environments is; what experience the group of students that evaluated the environments was. Actually, emphasis is given on the context of using and assessing the environments and their features and not the course per se.

2.1 Lectures 1 & 2: The “objects-first” approach - BlueJ

The goal of the first two lectures was firstly to introduce students to a programming environment that supports the teaching and learning of the object-first approach, i.e. incorporating functionalities such as the interactive creation of objects and invocation of methods; the inspection of objects’ situation; and the visualization of classes and objects. Secondly, the presentation and study of papers which discuss (a) the features that an educational programming environment should include and (b) the difficulties and misconceptions that students face when learning Object-Oriented Programming (OOP) using BlueJ.

BlueJ (Kölling et al. 2003) is “an integrated Java development environment specifically designed for introductory teaching” and is widely used in OOP courses. The main window of BlueJ presents students with a simplified UML class diagram of the currently open project. Students can create objects by invoking a constructor through interaction with the classes of the UML diagram, and then interact with the depicted objects by calling methods through each object’s popup menu. Furthermore, students can inspect an object’s state during method execution. Visualization of the object – oriented entities, such as objects and classes, is an important feature that aims at the comprehensive presentation of their differences. BlueJ incorporates a simple editor that uses different background colours for highlighting blocks of code, reports explanatory error messages and provides a java interpreter (code pad), as well as a debugging ability (Kölling et al. 2003). Overall, BlueJ is characterized by the simplicity of using all the aforementioned features. Finally, a large number of plug-ins has been developed for providing important features missing from BlueJ, such as a class card for presenting the structure of a class.

“The BlueJ system and its pedagogy” (Kölling et al. 2003) and the established “guidelines for teaching object orientation with Java” (Kölling and Rosenberg 2001) proposed by the team of BlueJ were presented to students using the corresponding papers in the context of the 1st lecture. Besides the activities proposed for the pedagogical use of BlueJ in the aforementioned papers, activities proposed in the accompanying textbook (Barnes and Kölling 2004) for the environment were also utilized.

In the 2nd lecture the difficulties, errors and misconceptions of students when introduced to OOP were studied using the widely known paper by Ragonis and Ben-Ari (2005), as well as a paper on a teaching approach for alleviating the difficulties of learning OOP that takes into account the relevant literature and the special features of BlueJ (Xinogalos 2012).

2.1.1 Assignment

Students were asked to devise a project for teaching the basic concepts of OOP following the guidelines proposed by Kölling and Rosenberg (2001) and the sequence of activities proposed in (Kölling and Rosenberg 2001; Kölling et al. 2003). This type of assignment was selected because students had been taught OOP and were familiar with relevant problems.

The aim of the assignment was: (i) to give students the role of the teacher and have them use BlueJ for teaching the main concepts of OOP; (ii) to investigate in what degree can students take into account the findings of the literature regarding a more effective teaching of the main OOP concepts applying the “objects-first” approach and an environment that has the necessary features for supporting this approach.

2.2 Lectures 3 & 4: The microworlds & mini-languages approach - objectKarel

The goal of these two lectures was to present educational microworlds and mini-languages for programming, as well as structure editors as an alternative teaching approach of OOP, or programming in general. The typical approach to teaching programming is based on a conventional programming language and environment and a set of number and symbol processing problems. On the contrast, programming microworlds: have a limited instruction set, which however covers all basic constructs; is usually based on a physical metaphor that includes an actor controlled by students through their programs; program execution takes place in a step by step manner and is depicted – the actor(s) execute the statements; a set of problems based on every-day activities are solved. In some cases, students are further supported by incorporating a structure editor in the microworld, so as to eliminate the need to memorize syntax details, avoid syntax errors and help students focus on understanding programming concepts and acquiring problem solving capabilities.

The approach of microworlds and mini-languages as “a way to learn programming principles” was presented using the work by Brusilovsky et al. (1997) and also a “case study in supporting levels of cognitive development in the computer science curriculum” based on the microworld JKarelRobot (Buck and Stucki 2001). The technology of structure editors was studied through the “evolution of novice programming environments developed at Carnegie Mellon University based on structure editors” (Miller et al. 1994).

The microworld objectKarel was used for studying and applying all the aforementioned technologies. objectKarel (Xinogalos et al. 2006a) is based on Karel++ (Bergin et al. 1997) which uses the well-known metaphor of robots carrying out various tasks on a restricted world. objectKarel incorporates a series of e-lessons and hands-on activities for motivating students through the use of concepts before they are asked to implement them. Programs are developed with a structure editor, which has the form of a menu and aims at avoiding syntax errors and focusing on concepts. Highly informative error messages are reported for the few errors that can occur, explaining the source of the error in natural language. Students can run, trace with a predefined speed or execute programs step by step (only forward). Also, objectKarel incorporates the technology of explanatory visualization, which means that students are presented with explanatory messages for the semantics of the statement being executed.

2.2.1 Assignment

Students were assigned to solve four problems that required using selection and repetition structures and the following OOP concepts: objects-classes, inheritance, polymorphism and overriding. The aim of this assignment was for students to get familiarized with solving a different type of problems from the ones they had already confronted (processing symbols and numbers or developing applications with GUI), as well as with using a structure editor and a programming environment based on a microworld and a mini-language.

2.3 Lectures 5 & 6: The game-oriented approach - Scratch

The goal of these lectures was to present a visual programming language and environment where students do not face syntax errors and where they are able to develop a different type of programs like games, animations, and simulations that can be shared with a universal community on the web. The environment used for this reason was Scratch.

Scratch (Maloney et al. 2010) is a visual, block-based programming language and environment for children. Scratch projects are built of graphical characters (called sprites) on a stage, and each sprite’s behavior is managed by one or more stack of visual instruction blocks. Scratch scripts are built by putting together blocks illustrating statements, expressions, and control structures. The shapes of the blocks indicate how they fit together, and a drag-and-drop technique allows for the blocks to be connected in meaningful ways. A Scratch program does not need to be compiled or edited/run. The programmer can click on a stack of blocks or a command at any time to see what it does. Similarly, Scratch has no error messages. Blocks fit together only in meaningful ways and thus syntax errors are avoided. The Scratch GUI makes navigation easy. It uses a single window, multi-pane design to ensure that key components are always visible.

With Scratch, children program and create their own stories, games, animations, simulations, interactive art and share their creations with others in an online community.

In order to present the rationale and the special features of Scratch students first watched the talk “Let’s teach Kids to code” by Mitchel Resnick on TED (http://www.ted.com/talks/mitch_resnick_let_s_teach_kids_to_code). “The Scratch programming language and environment” were presented using the homonym paper by (Maloney et al. 2010). Students used Scratch, while various tutorials and games were presented.

2.3.1 Assignment

Students had to develop a game with the following requirements: the player has at least three trials (lives); the player has a predefined time for playing; there are at least 3 opponents; the game records the best score for each player; the game gradually becomes more difficult as the player continues playing.

The aim of this assignment was for students to familiarize themselves with the capabilities of Scratch and use it in a creative way according to its design.

2.4 Lectures 7 & 8: The story-telling & 3D animation approach - Alice 2.0

The goal of these two lectures was the same as this of lectures 3 and 4.

Alice (Cooper et al. 2003) is a 3D interactive, animation, programming environment that aims to teach computer programming to students by visualizing objects and their behaviours. Specifically, students can use/modify 3D objects in order to create their own virtual worlds and write scripts for controlling the objects' appearance and behaviour. Alice utilizes a drag-and-drop smart editor for dragging objects into the editor and drop-down menus for selecting messages to send to an object. User-defined methods and functions are automatically added to the dropdown menus. Students can see immediately how their animation programs run – each action is animated smoothly - enabling them to easily understand the relationship between the code and its result.

Alice 2.0 was presented using the tutorials and educational material that is available in the web site of Alice (http://www.cs.duke.edu/csed/alice/aliceInSchools/).

2.4.1 Assignment

The assignment aimed at familiarizing students with Alice, its features and the projects that can be developed with it. This assignment was carried out in the lab in contrast with the rest that were assigned as homework. Students were given the presentations from various workshops carried out at Duke University for developing projects (http://www.cs.duke.edu/csed/web/alice09/tutorials.php). Most of the above mentioned projects are mainly related to mathematical notions and we wanted to familiarize students with different types of problems that could be solved using a programming language and environment.

2.5 Lectures 9 & 10: The app-oriented approach - MIT App Inventor

The goal of these two lectures was to study a visual programming environment that is similar to Scratch and Alice, which additionally focuses on developing apps for mobiles. In addition, we wanted to show that it is possible to learn the fundamental notions of programming when the incentive is to develop interesting apps for everyday devices, deploy them on the web, and share them with one’s friends and family.

MIT App Inventor is a blocks-based cloud programming tool that enables everyone to develop fully functional apps for Android devices.

In App Inventor the programmer develops the program as in Scratch in an interactive way using the drag and drop technique. He patches together blocks with specific meaning like loops, if-then-else statements or variables. The pieces fit in each other like in a jigsaw puzzle.

App Inventor consists of three parts:

• The Designer: where the programmer selects components from a palette (object of a certain class) and drags them to the Viewer (screen layout) to easily build the GUI of his app. Then, the static attributes and properties of the selected component can be changed.

• The Block Editor: provides blocks of code (“build-in drawers”) with specific behavior. The programmer drags & drops them to the blocks viewer and he puts them together so as to define his app’s dynamic behavior. The app can respond to various events, such as user events (touching the screen) or system events (a new location).

• App Inventor provides databases, geolocation, audio and video contents and other features.

• Testing app: the user can test an app by using a pre-configured emulator or an android phone while building it without compiling a program.

• App Inventor was presented through the tutorials and the projects that are available in the site of the environment (http://www.appinventor.org/). App Inventor 2 was just released.

2.5.1 Assignment

This assignment was carried out by groups of two students. Students were asked to study the material regarding “Module 5: Shared and Web Data An introduction to apps with shared data” from the site https://sites.google.com/site/appinventorcourse/communications-app-creat and implement an app similar to the ones proposed in this web page.

2.6 Lectures 11 & 12: Literature study

Students were assigned to select an article from the proposed literature regarding the presented educational programming environments, study it, prepare and present a presentation. Students’ presentations were made in class, followed by discussion with the instructor and classmates.

3 Methodology of the study

3.1 Participants

The study presented in this paper has taken place in the context of the 7th semester elective course “Educational Programming Environments and Techniques” at a Department of Applied Informatics. The course that was described in Section 2 is being taught for three years now by the second author of this article. During the academic year 2013–14 thirty seven students selected and attended the course. The majority of students was in the 4th year of their 4 years (8 semesters) bachelor studies (Table 1).

Table 1 Participants

All the students had been introduced to programming in the context of the 1st semester compulsory course “Procedural Programming” that uses C as the programming language and environments like wxDev-C++ and Code::Blocks. Students were also taught OOP in the 3rd semester compulsory course “Object-Oriented Programming” with Java and Eclipse. In a relevant question included in the final questionnaire one student stated that s/he is an expert in OOP and Java, while the rest stated in exactly the same percentages that have either basic or advanced knowledge (Table 2).

Table 2 Students’ knowledge of OOP and Java

3.2 Research questions

The main research questions that we aimed to investigate with this study are the following:

• Research Question 1 (RQ1): What do students think of the pedagogical usage of the five educational programming environments?

All the participants of the study had attended, as mentioned above, compulsory programming courses. As is usually the case, several students had faced various difficulties and some of them had not managed to acquire a deep knowledge of some programming concepts, especially OOP concepts that are more advanced. One of the aims of this study was to investigate whether the educational programming environments used would help such students to comprehend or deepen on concepts they had not mastered. Also, the study aimed to investigate which environment students consider the most effective for an introduction to programming no matter the age of the learners, and moreover in which context they consider each environment could be used successfully.

• Research Question 2 (RQ2) : What are the advantages and disadvantages of the five educational programming environments?

RQ2 aimed to record students’ perceptions on the pros and cons of each one of the environments that were studied and used by students throughout the semester. The data collected from this question was expected to help evaluate the five environments under the same conditions, but most important to draw conclusions on the effectiveness of the wide variety of features incorporated in them. This was considered to assist in answering the main question of the study stated in the title.

• Research Question 3 (RQ3): What features are considered important for an introductory programming environment in terms of its GUI, visualization capabilities, editor, error messages and execution?

RQ3 aimed to address directly the main question of the study regarding the features that are important for an introductory programming environment. Students were asked to define important features for each one of the widely accepted and utilized categories proposed by McIver (2002) for evaluating programming environments for novices.

3.3 Data collection and analysis

In order to investigate the aforementioned research questions, students filled in a questionnaire after the end of the lessons. The questions are provided in the Appendix and fall into three categories that correspond to the aforementioned research questions:

• Pedagogical usage of the studied environments (Q1, Q2, Q3)

• Evaluation of the studied environments (Q4)

• Investigation of main features of an introductory programming environment (Q5)

Q2 was a Likert type question and Q3 provided some choices but students could also provide freely a response. The rest of the questions were open type questions.

The open type questions were analyzed by the three authors in consecutive sessions in order to group their answers in categories and validate them. The results of students’ replies in the questionnaire are presented and analyzed in the following section.

4 Analysis of the results

4.1 Pedagogical usage of educational programming environments

4.1.1 Enhanced comprehension of programming concepts

• Question 1 (Q1): Did any of the environments help you comprehend or deepen on concepts (i.e. object, class, inheritance) that you had been taught in other courses and you had not fully comprehended?

In this question 54 % of the students (20 students) stated that they already had good knowledge of OOP. The rest 46 % of the students (17 students) stated that the educational programming environments helped them clarify some OOP concepts. Specifically, these students stated that the educational programming environments used in the context of the course helped them better comprehend both basic (objects, classes, methods) and advanced OOP concepts. Students’ replies are presented in Fig. 1. The reply “basic concepts” is quite subjective and could refer to different concepts based on the environment. For example, in BlueJ and objectKarel most probably it refers at least to objects and classes, while in Scratch to control structures.

Fig. 1

Support provided in comprehending programming concepts (N = 17)

The majority of students (10 out of 17 students - 59 %) were benefited by BlueJ that helped them comprehend the most concepts in comparison with the rest of the environments: basic concepts; objects; classes; methods/messages; and inheritance. The concepts of class and inheritance are visualized in the simplified UML class diagram presented in the main window of BlueJ, while objects are also depicted in the main window. Moreover, objects are created interactively using direct manipulation and each object’s methods are invoked in the same way. No one from the students benefited by BlueJ made a reference to polymorphism and overriding. This might be due to the fact that these concepts are not directly supported by the visualization and interactivity features of BlueJ and are mostly connected to the source code. A reference both to basic concepts and the more abstract concepts of inheritance, polymorphism and overriding was made for objectKarel, where students can watch simultaneously the line of code being executed each time and the way the object responds. For example, when a polymorphic method call is made for a robot the flow of control is transferred to the method declared in the class that the robot belongs to and the robot responds to this polymorphic method call with the way dictated by its type/class. Having robots from different classes in an inheritance hierarchy execute a polymorphic/overriden method helps students clearly comprehend the underlying concepts. It seems that step by step execution with visualization of the results seems to make abstract concepts more easily comprehended, as recorded in a previous study with objectKarel (Xinogalos et al. 2006a). Scratch was the second more referenced environment after BlueJ that helped 35 % (6 out of 17 students) of the students to comprehend OOP concepts, although its support for OOP concepts is limited (duplicate objects, method calling). Students mentioned that Scratch helped them comprehend basic concepts, as well as the OOP concepts supported by the environment: objects and methods/messages. Both MIT App Inventor and Alice helped students comprehend basic concepts and the former also methods/messages.

4.1.2 The most effective environment for teaching introductory programming

• Question 2 (Q2): What is the most effective environment for an introduction to programming independently of the educational level?

Students’ replies in this question are summarized in Fig. 2, while more statistical data are presented in Table 3.

Fig. 2

Mean value - effectiveness for an introduction to programming (scale 1 to 5)

Table 3 The most effective environment for an introduction to programming (N = 37)

As expected, Scratch is considered by students the most effective environment for an introduction to programming for all educational levels (T3.1: Mean = 4.49, Std. Dev. = 0.77). The next more effective environment with a considerable “distance” is considered to be MIT App Inventor (T3.2: Mean = 3.59, Std. Dev. = 1.07), followed with a small difference by BlueJ (T3.3: Mean = 3.38, Std. Dev. = 1.06) and objectKarel (T3.4: Mean = 3.22, Std. Dev. = 1.11), and finally with a big difference by Alice (T3.5: Mean = 2.76, Std. Dev. = 1.09).

The placement of MIT App Inventor in the second place and Alice in the last was quite unexpected, since MIT App Inventor is oriented to mobile app development and not to fundamental programming concepts/constructs. This might be due to the great interest for mobile apps nowadays and consequently the motivation provided to students. However, students’ replies in question 3 that asked students to think separately for each environment what aim(s) it fulfils gave different results.

4.1.3 Usage of educational programming environments in different contexts

• Question 3 (Q3): For what aim(s) is each one of the environments appropriate? You can choose one or more from the following or define other(s) aim(s):

  • Introduction to basic algorithmic structures & concepts

  • Introduction to programming

  • Introduction to OOP

  • Introduction to fundamental OO concepts (e.g. object, class, inheritance)

36 students answered this question for BlueJ and all the students for the other four environments. Students’ replies in this question are presented in Fig. 3 and can be summarized as follows:

Fig. 3

Usage of the environments in different contexts

• Scratch and objectKarel are considered to be the most appropriate environments for an “introduction to basic algorithmic structures and concepts” by 54 and 49 % of the students respectively (F3.1).

• Scratch and objectKarel are also considered appropriate for an “introduction to programming” by the majority of students (F3.2: 70 and 65 % respectively).

• BlueJ seems to be the most popular environment both for an “introduction to fundamental OO concepts” (F3.4: 83 %) and an “introduction to OOP” (F3.3: 67 %). Scratch and objectKarel are also considered appropriate for an “introduction to OOP” by the majority of students (F3.3: 57 and 51 % respectively).

• Alice is considered appropriate for the articulated aims by one-third (and less) of the students, while MIT App Inventor is considered appropriate for less than one-fourth of the students.

4.2 Advantages and disadvantages of educational programming environments

• Question 4 (Q4): What are the main advantages and disadvantages of each environment?

4.2.1 BlueJ

Students’ replies in Q4 regarding BlueJ are presented in Table 4. The two most referenced advantages of BlueJ are the environment’s simplicity and usability (T4.1: 36 %) and the fact that it generally supports the comprehension of concepts (T4.2: 31 %). Some students mentioned specific features of BlueJ, namely: the presentation of a UML class diagram for a project in the main window (T4.3: 14 %); the visualization of concepts (T4.4: 11 %); and the interactive creation of objects and calling of methods (T4.5: 6 %). Consequently, students identified and evaluated positively all the main features and design decisions made by its developers (Kölling et al. 2003).

Table 4 The advantages and disadvantages of BlueJ (N = 36)

Moreover, 2 students consider important the fact that the student is “writing code in Java” (T4.6: 6 %) and not some educational language and another student made a reference to the “error messages” reported by BlueJ (T4.7: 3 %).

When we examine the disadvantages of BlueJ we see that more than one out of five students states that the environment is “good for beginners, but not for more experienced students” (T4.8: 22 %). However, BlueJ was designed mainly for educational purposes and this cannot be actually considered as a disadvantage. Another disadvantage refers to the “poor graphics” of BlueJ’s GUI (T4.9: 11 %). Probably this response was due to the GUI graphics and the graphical output of the problems solved in the rest of the environments used in the context of the course. The most important of the disadvantages mentioned, in our opinion, lies in the fact that BlueJ does not support program development (T4.10: 11 %). More specifically, two out of the 36 students mentioned as a disadvantage that the student “writes code” and another 2 students that BlueJ “does not protect from syntax errors”. In these students we can add three more students which mentioned that BlueJ “requires prior knowledge” (T4.11: 8 %), in the sense that there is no support in program development as in the other environments (e.g. structure/block editor). Of course we have to take into account that BlueJ supports Java and not some kind of a mini-language.

4.2.2 objectKarel

Students’ replies in Q4 regarding objectKarel are presented in Table 5. The most referenced advantages of objectKarel based on students’ replies are that the environment is “good for learning programming concepts” (T5.1: 38 %) and it is characterized by simplicity and usability (T5.2: 30 %). Another recorded advantage refers to the incorporation of the structure editor in objectKarel that according to 13 % of the students supports program development (T5.3). More specifically, 3 students stated that there is “no need for memorizing syntax” and 2 students stated that the structure editor “protects users from syntax errors”. Other advantages recorded are the following: “entertaining environment” (T5.4: 11 %); “organization of activities and theory” (T5.6: 5 %); “good examples are incorporated in the environment” (T5.7: 3 %). An advantage that we anticipated that would be identified by more students refers to the execution of programs in objectKarel that utilizes both program animation and explanatory visualization. In Q4 just one student mentioned “step by step execution” and another one that “objectKarel shows how statements work” (T5.5: 5 %). However, in Q5 that is analyzed in section 4.3 more than one third of the students made a reference in the execution system of objectKarel.

Table 5 The advantages and disadvantages of objectKarel (N = 37)

The most obvious disadvantage according to one third of students is program development in objectKarel (T5.8: 30 %). More specifically, 7 (out of 37) students commented as a problem “program development”, 3 students “correcting the program” and 1 student the “lack of copy-paste” within the structure editor. To these students we can add 3 more students which mentioned that the environment is “difficult to use at the beginning” (T5.11: 8 %) and 2 students that consider the environment is “not user-friendly” (T5.13: 5 %). Consequently, although the structure editor is one of the main features of objectKarel that it is supposed to support novices in learning programming, a significant number of students consider it as a disadvantage. However, this result is not surprising. Structure editors restrict users in specific actions and make editing more difficult since they do not allow violating syntactic rules, which is sometimes the easiest way for correcting a piece of code that does not give the desired results. Novices overlook these disadvantages and appreciate the support provided by a structure editor: no need to memorize syntax; avoidance of syntax errors that are difficult for novices; guidance through program development. More experienced students that are accustomed to using typical editors for writing code, as is the case in this study, seem to prefer the flexibility of such editors.

Other disadvantages of objectKarel are the following: it “does not offer many programming capabilities” (T5.9: 27 %); it is “using the same scenario (robots)” for teaching various concepts and this fact “does not make it very attractive” (T5.10: 8 %).

4.2.3 Scratch

The advantages and disadvantages of Scratch as recorded in students’ replies are presented in Table 6.

Table 6 The advantages and disadvantages of Scratch (N = 37)

Several advantages, but disadvantages as well, were recorded in students’ replies for Scratch. Nearly one third of the students made a reference to the usability and simplicity of Scratch (T6.1: 32 %). On the other hand, students mentioned various relevant problems: Scratch has “many functionalities that distract” (T6.12: 11 %), while “in complex programs it is confusing” (T6.12: 3 %); it is “difficult using it at the beginning” (T6.13: 11 %); “there are some bugs” (T6.15: 5 %); “working in one screen” (T6.18: 3 %); “no warning for potential bugs” (T6.20: 3 %).

Several students were fond of the “nice graphics” (T6.2: 27 %) and the fact that the environment offers “many capabilities” (T6.3: 24 %), such as “easy creation of games” (T6.9: 5 %). Another advantage of Scratch is the support provided in program development (T6.4: 19 %) and more specifically the “drag and drop code, simple statements, categorization of statements” (14 %) and the fact that it “protects from syntax errors” (5 %). A relevant problem that was referenced by just one student is that the environment “does not explain why a block does not fit” (T6.17: 3 %). Also, 3 students consider as a disadvantage that there is actually “no code writing”, that the “code is not realistic” and that there is a “restricted number of statements” (T6.14: 9 %).

Finally, the following advantages that refer to important features of Scratch were recorded, but in low percentages: it is “good for teaching introductory programming” (T6.5: 14 %); it is “entertaining” (T6.6: 14 %); it “promotes critical thinking and creativity” (T6.8: 8 %); it is characterized by a high degree of “interactivity” (T6.10: 5 %); and has a large “community” (T6.11: 5 %).

4.2.4 Alice

The advantages and disadvantages of Alice, as recorded in the replies of 35 out of the 37 students, are presented in Table 7. The most referenced advantage is the many capabilities offered by Alice and the impressive results, while the most important disadvantage recorded in the majority of students’ replies is the complexity of the environment.

Table 7 The advantages and disadvantages of Alice (N = 35)

Forty percent of the students commented on the various capabilities offered by Alice and the very nice results (T7.1: 40 %): the fact that Alice offers “many capabilities (even modeling, math,…)” was mentioned by 10 students; the “nice, even impressive results” was mentioned by 3 students; and the “easy creation of games” by 1 student. These impressive results are mainly due to the use of “3D graphics” (T7.2: 17 %). Another advantage is the support provided for program development (T7.3: 11 %) through the use of the “block editor” (2 students) that helps “avoid memorization of syntax” and “syntax errors” (1 student). Alice is considered “good for an introduction to programming or code” (T7.5: 9 %) and it provides an “entertaining, different teaching approach resembling a game” (T7.6: 9 %). 2 students mentioned “visual representation of program execution” and that a “project is visible while developed and can be easily improved” (T7.7: 6 %). “Interactivity” and “promotion of creative thinking” were mentioned by just 1 student each (T7.9, T7.10: 3 %).

The main disadvantage of Alice refers to its complexity (T7.11: 57 %). Specifically, students made the following comments:

• “complex environment” (34 %)

• “not user-friendly” (14 %)

• “the initial screen should be simpler” (3 %)

• “having different functionalities in various places of the main screen confuses” (3 %)

• “could not decide what and how to do it - more complex than the other environments” (3 %)

Moreover, some students brought up two issues that they think are discouraging for novices:

• Program development is difficult (for novices) (T7.12: 17 %): Alice “requires prior knowledge of programming” according to 3 students; the “statements are not simple for novices” according to 2 students; and “implementation is difficult” according to 1 student.

• The environment is discouraging for novices (T7.13: 14 %): “enough time is required for get familiarized” as stated by 2 students; “the volume of the information provided can confuse novices” as stated by another 2 students; or it generally “discourages novices” as stated by 1 student.

4.2.5 MIT App Inventor

Students’ replies in Q4 regarding MIT App Inventor are presented in Table 8. Students reported several advantages for MIT App Inventor. As noted by students “mobile apps are interesting nowadays” (T8.5: 19 %) and MIT App Inventor gives the chance of “designing and implementing easily” (T8.3: 22 %) “good quality apps” (T8.4: 19 %). More than one fourth of the students made a reference to the usability and simplicity (T8.1: 28 %) of the environment that constitutes it “good for an introduction to Android” (T8.2: 28 %). MIT App Inventor is good for novices, since it supports them in the whole process:

Table 8 The advantages and disadvantages of MIT App Inventor (N = 36)

• “good tutorials” (T8.9: 8 %)

• program development (T8.6: 17 %): “drag and drop code” was mentioned by 3 students; “avoidance of syntax errors” was mentioned by 2 students; and “very good organization of statements” was mentioned by 1 student.

• easy testing (T8.7: 14 %): “watching the project while being developed gives the chance for improving it” (2 students); “you can download the app in your mobile device and test it” (1 student); “functionality and communication with the emulator was very good and useful” (1 student); “way of presentation” (1 student).

• “using complex tools through the GUI” (T8.10: 3 %)

As expected, some students also mentioned the “special features/capabilities” (T8.8: 14 %) of MIT App Inventor, such as the fact that: “no installation is required”; “functionalities not supported by other environments (such as web viewers)” are offered; it constitutes “the most complete environment”.

Besides all the aforementioned advantages, 19 % of the students considered that the environment is not so user-friendly (T8.13: 19 %). These students mentioned that the environment “is not so user-friendly/complete” or “that is tiring” (5 students), “the user must devote some time in order to comprehend the environment” (1 student), and that is “difficult for a novice to understand and use its capabilities” (1 student). The same percentage of students faced problems with MIT App Inventor (T8.14: 19 %), namely: “slow emulator with limited capabilities” (2 students); “bugs (i.e. source code is unexpectedly deleted)” (2 students); “hidden functionalities” (1 student); “security issues with Java – problems with Java 7” (2 students). The limitations of MIT App Inventor are the following (T8.15: 11 %): “limited size of apps” (2 students); “not for professional use – limited capabilities” (1 student); “no ability to duplicate code” (1 student).

4.2.6 The main advantages and disadvantages of the environments

The main advantages and disadvantages of the five environments, presented analytically in Tables 4, 5, 6, 7 and 8, are summarized in Fig. 4 and Fig. 5 respectively. The advantages/disadvantages that were reported by at least 10 % of the students are included in Fig. 4 and Fig. 5.

Fig. 4

The main advantages of the environments

Fig. 5

The main disadvantages of the environments

Regarding the advantages, students made references to features that could be grouped into the following categories:

• GUI: simplicity/usability (F4.1: BlueJ-36 %, objectKarel-30 %, Scratch-32 %, MIT App Inventor-28 %); nice graphics (F4.2: Scratch-27 %); 3D graphics (F4.3: Alice-17 %).

• Pedagogical support: supports comprehension of concepts (F4.4: BlueJ-31 %); good for learning programming concepts (F4.5: objectKarel-30 %); teaching introductory programming (F4.5: Scratch-14 %); good for an introduction to Android (F4.6: App Inventor-28 %).

• Capabilities/types of programs: offers many capabilities (modeling, math, games,…) and very nice results (F4.7: Alice-40 %); many capabilities (F4.8: Scratch-24 %); mobile apps are interesting nowadays (F4.9: App Inventor-19 %); entertaining (F4.10: objectKarel-11 %, Scratch-14 %).

• Visualization: UML class diagram (F4.11: BlueJ-14 %); visualization (F4.12: BlueJ-11 %).

• Program development (editor): supports program development (F4.13: objectKarel-13 %, Scratch-19 %, Alice-11 %, App Inventor-17 %); designing/implementing apps easily (F4.14: App Inventor-22 %); implementing good quality apps (F4.15: App Inventor-19 %).

• Testing: easy testing (F4.16: App Inventor-14 %).

Regarding the disadvantages, students made references to features that could be grouped into three main categories:

• GUI: complex/not user-friendly (F5.1: Alice-57 %); discouraging for novices (F5.2: Alice-14 %); not so user-friendly/complete (F5.3: App Inventor-19 %); faced problems (F5.4: App Inventor-19 %); distracting/confusing environment (F5.5: Scratch-14 %); difficult using it at the beginning (F5.6: Scratch-11 %); poor graphics (F5.7: BlueJ-11 %).

• Program development (editor): program development/editing is difficult (F5.8: objectKarel-30 %, Alice-17 %); does not support program development (F5.9: BlueJ-11 %).

• Capabilities/types of programs: not many programming capabilities (F5.10: objectKarel-27 %); limited capabilities (F5.11: App Inventor-11 %).

We must stress that some of the disadvantages that refer to the GUI and the editor might be attributed to the fact that the students were not novices. As already mentioned, some students that have experience in programming seem to have difficulty in implementing programs that are not based on text. Moreover, some students may simply be non-visual learners. Finally, mapping the semantics of shapes to already known statements might cause a cognitive overload to some students (Green and Petre 1996), with no added value in the case they already have experience in programming.

4.3 Important features for an introductory programming environment

In section 4.2 students’ replies on an open-ended question (Q4) regarding the advantages and disadvantages of each one of the five environments were analyzed. Students made references to features regarding the GUI of the environment, the pedagogical support provided, the capabilities or else the types of programs developed in the environment, its visualization capabilities, and finally the way program development is accomplished in the incorporated editor. In this section, students’ replies regarding the main characteristics that any introductory programming environment should have under specific categories are analyzed. Some of these categories are the same as the ones proposed by students themselves, namely: GUI, visualization, and editor. Moreover, question 5 (Q5) made a reference to compiler/error messages and execution. The question posed was the following:

• Question 5 (Q5): What are the main characteristics that an introductory programming environment should incorporate for each one of the following modules: GUI, Visualization, Editor, Compiler/Error messages, Execution? You can make reference to the environments you used.

Student’s references to the five environments when describing the features they consider important for an introductory programming environment are summarized in Fig. 6, while the features themselves are presented in Fig. 7. Students were asked to define the main features for the categories proposed in McIver (2002) with the following exceptions:

Fig. 6

References to the environments regarding fundamental features

Fig. 7

Important features for an introductory programming environment

• “Debugger”: students were not asked to provide features for this category, since: this ability was not presented to students due to time restrictions; this capability is offered only by BlueJ; some environments use some kind of debugger as the standard execution system, such as objectKarel.

• “Individual method execution and object construction”: this category was substituted by the more general category “Execution”.

4.3.1 Graphical User Interface (GUI)

Nearly one third of the students identified as the most important feature for the GUI of an educational programming environment simplicity and user-friendliness (F7.1:31 %). Usability and simplicity was identified in Q4 regarding the advantages and disadvantages of the educational programming environments used as one of the main advantages for 4 out of the five environments with nearly the same percentages: BlueJ (T4.1: 36 %); objectKarel (T5.2: 30 %); Scratch (T6.1: 32 %); MIT App Inventor (T8.1: 28 %). The only exception was Alice that was characterized as “(relatively) user-friendly” by just 9 % (T7.4) of the students, while the majority (T7.11: 57 %) of them identified as the most prevalent disadvantage that the environment is “complex and not user-friendly”. For App Inventor also 19 % of the students stated that they find the environment not so user user-friendly (T8.13).

Other features identified by students regarding GUI, but with lower or even minimal percentages are graphics (F7.3: 8 %) and feedback/interactivity (F7.6: 3 %). Two features referenced by one student each that are not related to the GUI in general but to programming more specifically is the use of characters (F7.4: 3 %), such as the ones used in microworlds, and adoption of visual programming (F7.5: 3 %).

The environment that seems to better fulfill collectively the aforementioned features is Scratch that was referenced by 42 % (F6.3) of the students in Q5, followed by MIT App Inventor (F6.5: 25 %).

4.3.2 Visualization

Visualization of classes and objects seems to be important for more than one third of the students. 29 % of them mentioned the visualization of both classes and objects (F7.7), while an additional 9 % (F7.9) of them considered important the usage of characters/actors as objects in the programs (like robots and so on). Some students considered important even the visualization of statements as puzzle pieces (F7.8: 14 %).

BlueJ was referenced by 40 % of the students (F6.1) and Scratch by 34 % (F6.3) in their replies regarding visualization (in the context of Q5). In Q4 students identified the following advantages for BlueJ that are relevant to visualization: UML class diagram (T4.3: 14 %); visualization (T4.4: 11 %); interaction with classes and objects (T4.5: 6 %). Moreover, the “support provided for comprehending concepts” in BlueJ (T4.2: 31 %) could be largely attributed to the visualization and direct manipulation features of the environment.

4.3.3 Editor

Puzzle-like statements seem to be the most popular feature regarding the editor of an educational programming environment. Nearly one third of the students identified a puzzle-like editor (F7.10: 31 %) and just 6 % (F7.11) of the students identified a structure editor as an important feature. Adding the 6 % of the students which think that the editor must prevent errors (F7.12) we see that 43 % of the students consider that novices should be supported in developing programs and should not write code.

As expected, most students referenced Scratch (F6.3: 40 %) and MIT App Inventor (F6.5: 31 %) when describing the features of the editor they consider important.

4.3.4 Compiler/error messages

Error messages are a great source of difficulties not only for novices, but for more experienced students as well. According to students a good compiler must report “simple and understandable messages” (F7.15: 21 %), as well as “the exact line of the error” (F7.16: 18 %). The messages reported should “propose ways of correcting the error” (F7.17: 12 %) and use “natural language” (F7.18: 12 %). A small percentage of students mentioned that a “program should compile during development” (F7.19: 6 %) and “pop-up warnings” should be reported (F7.20: 6 %), features that students knew from their prior experience with Eclipse. Finally, one student mentioned that errors should be prevented in the first place, which of course requires some kind of structure editor.

4.3.5 Execution

The most referenced feature not only for the category of execution, but for all categories included in Q5, was step by step execution (F7.22: 42 %). “Inspecting elements” (F7.23: 3 %), “watching variables” (F7.24: 3 %) and “executing during development” were mentioned by 1 student each.

The execution system that seems to suit better to novices’ needs is objectKarel that was referenced by 36 % of the students (F6.2), followed by Scratch that was referenced by 21 % of the students (F6.3).

5 Discussion and conclusions

The study presented in this paper investigated the perceptions of students regarding 5 educational programming environments that are widely used and the features that an ideal introductory programming environment should incorporate. The study was based on a questionnaire filled in by 4th year undergraduate students of an Informatics Department attending a course on “Educational Programming Environments and Techniques”. The same group of students having prior experience in programming languages and environments, as well as the accompanying difficulties of learning programming, used the five environments in the context of the course and evaluated them. The main conclusions that are drawn are presented in the following paragraphs.

Regarding the pedagogical usage of the educational programming environments (RQ1) examined the most important results can be summarized as follows:

Based on the replies in Q1, 54 % of the students had already had a good knowledge of OOP programming. All the environments helped more or less the rest of the students comprehend or deepen on various programming concepts: basic concepts (all the environments); objects (BlueJ, Scratch); classes (BlueJ); methods/messages (BlueJ, Scratch, App Inventor); inheritance (BlueJ, objectKarel); polymorphism/overriding (objectKarel). The term “basic concepts” of course is quite subjective and can refer to different concepts based on the environment: objects and classes in BlueJ and objectKarel, control structures in Scratch and so on. The environment that benefited the majority of students (59 %) to comprehend or deepen on all the recorded concepts, with the exception of polymorphism and overriding, was BlueJ. Consequently, this study confirms that an educational programming environment cannot be used only for an introduction to programming, but also for helping students in dealing with difficulties from a prior introduction to programming with a conventional programming language and environment (Xinogalos et al. 2006a).

Regarding the environment that is considered by students to be the most effective for an introduction to programming for all educational levels (Q2), Scratch was a clear winner. In the second position but with a considerable “distance” students placed MIT App Inventor, followed in the third position with a small difference by BlueJ and objectKarel, and finally with a bigger difference by Alice. Both Scratch and App Inventor engage more students most probably due to the fact that they give them the chance to write programs connected to their interests, such as games, simulations, stories (Utting et al. 2010) and mobile apps (Papadakis et al. 2014). Alice also gives the chance to develop programs with impressive results connected to students’ interests, however several students in this study considered it complex and not user-friendly (Q4) and this is the most prominent reason for considering it the least effective from the examined environments for an introduction to programming. So, an environment used for an introduction to programming should engage students through the development of programs connected to students’ interests using a simple and user-friendly interface.

However, an environment that engages students and is simple is not always the best choice for an introduction to either basic or object-oriented programming (concepts). In Q5 that asked students to specify for which of the following aims

• introduction to basic algorithmic structures and concepts

• introduction to programming

• introduction to fundamental OO concepts

• introduction to fundamental OOP

each environment is appropriate, quite different results were recorded in comparison with the open-type question Q4. Specifically, Scratch was still among the first choices, but App Inventor was considered appropriate for less than one-fourth of the students. Scratch and objectKarel were considered to be the most appropriate environments both for an “introduction to basic algorithmic structures and concepts” (by 54 and 49 % of the students respectively) and an “introduction to programming” (by 70 and 65 % respectively). Scratch and objectKarel were also considered appropriate for an “introduction to OOP” by the majority of students (57 and 51 % respectively). BlueJ was the most popular environment both for an “introduction to fundamental OO concepts” (83 %) and an “introduction to OOP” (67 %), while Alice was considered appropriate for the articulated aims by one-third (or less) of the students.

Taking into account the results regarding the pedagogical use, the advantages and disadvantages recorded for the five environments (Q4) in combination with students’ replies regarding the features that any introductory environment should have (Q5) the following conclusions can be drawn regarding students’ perceptions for the five environments (RQ2):

The most important advantages of BlueJ according to students are its simplicity/usability (F4.1), the incorporated visualization features (F4.11 & F4.12) and the support provided for comprehending concepts (F4.4). Students distinguished BlueJ amongst the other environments for its visualization features (Fig. 6) and especially the visualization of classes and objects (F7.7). Also, the majority of students consider BlueJ as the most effective environment for an introduction both to fundamental OO concepts (F3.3) and programming (F3.4). However, they do not consider it as good for more experienced students (F5.12). The disadvantages outlined by a small percentage of students (F5.7 & F5.9: 11 %) are the poor graphics used in the interface of BlueJ and the lack of support for program development.

Scratch is clearly students’ first choice for an introduction to programming irrespectively of the educational level of the learners (Table 3). Specifically, Scratch was considered by the majority of students appropriate for an introduction to basic algorithmic structures and concepts, programming, even OOP (Fig. 3). The advantages of Scratch are several: it has nice graphics (F4.2); it offers many capabilities (F4.8); it is entertaining (F4.10); and it supports program development (F4.13). Students distinguished Scratch for its GUI as a whole (Fig. 6) and its editor with the puzzle-like statements (Fig. 6, F7.14). Simplicity and usability were referenced by 32 % of the students as one of the advantages for Scratch (F4.1), however several students mentioned the following relevant drawbacks: the environment is distracting/confusing (F5.5:14 %); and difficult using it at the beginning (F5.6: 11 %).

The most important advantages of objectKarel according to students are its simplicity/usability (F4.1) and the entertaining nature of the environment/problems (F4.10), while they consider the environment good for learning programming concepts (F4.5). Specifically, objectKarel was considered by nearly the same percentages with Scratch appropriate for an introduction to basic algorithmic structures and concepts, programming and OOP (Fig. 3). Students distinguished objectKarel amongst the other environments for its execution system (Fig. 6) that supports step by step execution (F7.22) using highlighting of the current statement with simultaneous depiction of its result on the robots’ world and explanation in natural language of the relevant semantics (explanatory visualization). The main disadvantage of objectKarel lies in the fact that it does not support many programming capabilities (F5.10). However, it was a main design decision of objectKarel to support the most fundamental OOP concepts in a straightforward manner. Of course objectKarel was designed 15 years ago and students’ interests and needs have changed towards programming games, simulations and mobile apps. Also, 30 % of the students reported difficulties with the structure editor (F5.8), in contrast with 13 % of the students that considered the editor to support program development (F4.13). As recorded in a previous study with two groups of students with different experience, a combination of a structure editor targeted to novices and a text editor for more experienced students would be ideal (Xinogalos et al. 2006b).

Alice is distinguished for the many capabilities it offers, such as modeling, math and games, and the very nice or even impressive results (F4.7) and its 3D graphics (F4.3). Although 11 % of the students state that Alice supports program development (F4.13), more students (as was the case for objectKarel as well) consider that program development is difficult in the environment (F5.8: 17 %). However, the main disadvantage of Alice is not this. The majority of students consider Alice to be complex and not user-friendly (F5.1), while some students (14 %) consider it even discouraging for novices, as recorded in their comments. Alice is considered appropriate for an introduction to algorithmic concepts (F3.1: 35 %), programming (F3.2: 27 %), OO concepts (F3.4: 30 %) or OOP (F3.3: 35 %) by approximately one-third (more or less) of the students.

MIT App Inventor differentiates from the other environments for the type of applications it supports. Mobile apps are interesting nowadays (F4.9) and App Inventor supports students in designing and implementing good quality apps (F4.15) easily (F4.14), testing is easy (F4.16) and it offers special features and capabilities (no installation, web viewers) (F4.17). So, App Inventor is considered good for an introduction to Android (F4.6). App Inventor was considered the second more effective environment – after Scratch – for an introduction to programming irrespectively of the educational level (Q2), probably due to the motivation provided by the development of mobile apps. However, when students were asked (Q3) to define explicitly if they consider each one of the examined environments appropriate for an introduction to basic algorithmic concepts and/or programming and/or OO concepts and/or OOP, App Inventor was considered appropriate for less than one fourth of the students (27 % for OOP). Finally, although 28 % of the students mentioned simplicity and usability (F4.1) as an advantage, 19 % of the students mentioned that the environment is not so user-friendly (F5.3) and 19 % that faced problems (F5.4). Frequent problems in usage are also recorded in (Papadakis et al. 2014).

Generally, students identified and evaluated positively the main features of the examined environments with few exceptions.

Regarding the main features that an introductory programming environment should have (RQ3), the main results that were recorded are the following:

• GUI: the most important feature of the GUI is to be simple and user-friendly (F7.1). A good example of such an environment is Scratch (Fig. 6).

• Visualization: visualization of classes and objects is the most important feature in this category (F7.7) and an exemplary environment that has successfully implemented this feature is BlueJ (Fig. 6).

• Editor: the support for puzzle-like statements is clearly the most important feature for the editor (F7.10) and a good example of such an editor is Scratch (Fig. 6).

• Compiler/error messages: it is important to report simple and understandable error messages (F7.15) with the exact line of the error (F7.16), and moreover to use natural language (F7.18) and propose ways of correcting the error (F7.17).

• Execution: step by step execution (F7.22) was the most referenced feature among all the features referenced by students as important. A good example of an execution system is offered by objectKarel (Fig. 6).

Consequently, an ideal introductory programming environment should provide a simple and user-friendly GUI that supports visualization of objects and classes, includes a puzzle-like editor for program development, reports simple and understandable error messages in natural language that provide help for correcting them, and finally the ability to execute the program in a step by step manner. Although students find these features in the examined environments, there is no single environment that fulfils all these features. However, the most successful environment seems to be Scratch. Interested researchers could consider the aforementioned features as recommendations for designing new introductory programming environments.

Concluding, it must be highlighted that this study attempted to investigate the wide variety of features that have been incorporated in introductory programming environments and find out what is actually important for students. Relevant studies tried to evaluate the features and didactical rationale of a specific environment each time (Buck and Stucki 2001; Cooper et al. 2003; Xinogalos et al. 2006a) or reviewed various environments based on some sort of criteria (Deek and Mchugh 1998; Guzdial 2004; Kelleher and Pausch 2005). On the contrary, in this study five representative educational programming environments that fall into different categories were evaluated under the same conditions by the same group of last-year students having experienced the difficulties of learning programming using various programming environments during their studies. This helped us draw conclusions not only on the pedagogical value and pros and cons of each one of the five specific environments, but also on the most important features that any introductory programming environment should have. This is considered to be the main contribution of this work. The conclusions drawn could be utilized by anyone interested in the teaching and learning of programming, as well as in devising the next generations of programming environments. Future work could include developing a programming environment that encompasses all these features, as well as repeating the study with even more environments that are also used in practice.

Appendix - Questionnaire

Question 1 (Q1): Did any of the environments help you comprehend or deepen on concepts (i.e. object, class, inheritance) that you had been taught in other courses and you had not fully comprehended?

Question 2 (Q2): What is the most effective environment for an introduction to programming independently of the educational level?

For each environment give one answer in the scale of 1 to 5 where 1 = least effective and 5 = most effective.

	1	2	3	4	5
BlueJ
objectKarel
Scratch
Alice
MIT App Inventor

Question 3 (Q3): For what aim(s) is each one of the environments appropriate? You can choose one or more from the following or define other aim(s):

• Introduction to basic algorithmic structures & concepts

• Introduction to programming

• Introduction to OOP

• Introduction to fundamental OO concepts (e.g. object, class, inheritance)

Question 4 (Q4): What are the main advantages and disadvantages of each environment?

Question 5 (Q5): What are the main characteristics that an introductory programming environment should incorporate for each one of the following modules: GUI, Visualization, Editor, Compiler/Error messages, Execution? You can make reference to the environments you used.