Introduction

Traditional pedagogical approaches relying on lectures and passive learning however, are being replaced or augmented with educational environments that facilitate active learning and engagement to foster self-learning and critical thinking, within a domain or area of specialization (Limniou and Smith 2014). These changes are not occurring in a vacuum, but are occurring in universities embedded specific institutional, social, political and cultural contexts. Our research was undertaken in a Chinese university, currently undergoing a rapid transformation from traditional lecture-based pedagogies to pedagogies grounded in experiential learning, related to overall societal change from an agricultural-industrial economy to a knowledge-based globally connected economy (Criscuolo and Martin 2004). In one sense, as educators we are merely catching up with practices that are commonplace in the West, but new to China. At the same time, these practices are evolving within a range of domains globally, including engineering. When adopting this learner-oriented pedagogy, we were confronted with the problem of ascertaining how students subjectively understood the assignments in relation to their own goals.

The problem, however, is how to shift to a learner-oriented assessment process given the inherent power dynamics of a traditional classroom setting. References (Tao et al. 2015, 2016) describe two similar questionnaires administered according assessment schedule, a pre-test conducted during the initial class and a post-test on the last day of last class measuring student perceptions with a set of Likert-scale questions and open-ended questions. Student perceptions of a Massive Open Online Course (MOOC) teaching technique deploying multi-task assessment throughout academic year were investigated, questionnaires with Likert scale items gauged student perceptions and attitudes toward the pedagogy, an Analysis Of Variance (ANOVA) showed an improvement in student perceptions of resources and assessments and over time, higher overall satisfaction with the course (Liu et al. 2017). These studies however, deployed questions with fixed response categories reflecting the power relationship in a classroom.

The objective of the research reported here was to inductively collect, explore, analyze and interpret student perceptions of an active learning assignment in a scientific writing and methodology class for software engineering students in Geo-informatics in Wuhan University, framed in terms of action research, treating students as “original thinkers in the making”. (Denzin and Lincoln 2003) Hence, the process of assessing the course and educational experiences is fully integrated into the pedagogy, with the overall goal of producing critical, socially engaged researchers through learning situations geared to solving real-world problems related to national development objectives.

We argue that the techniques for evaluating instruction and student performance must also evolve and change, cognizant of both the power relationships at the global scale and within the classroom. Assessment must go beyond simple testing to achieve a subjective understanding of student perceptions and motivations, re-centering engineering education on the learner rather than the instructor; the learner becoming a co-creator of knowledge (Freire 1970). Assessments must include student feedback on the relevance of specific assignments to student goals and interests to evaluate the worth of educational experiences (Gweon et al. 2017). By scrutinizing educational experiences, collaborations, and attitudes, quantitatively and qualitatively (Al Bahadly 2006), student feedback acts as an effective means of communication between teachers and students. Likert scale questions are often deployed to understand student perceptions of new pedagogies; however, Likert-scale questions might not capture student perceptions, but instead reflect the interests and concerns of the instructors (Hartley 2014). In our pedagogy student feedback points out problematic areas, providing direction for more effective teaching in terms of student expectations and needs, thereby encouraging innovation in teaching methodology, and prompting modifications in assignments and tasks (Limniou and Smith 2014).

The rest of the paper is organized as follows. “The pedagogy” section details the pedagogy and the sequential exploratory field study design is presented in “Sequential exploratory field study design” section. Qualitative data collection, open coding, and connecting approaches are elaborated in “Analysis and discussion” section, and “Conclusion” section concludes this paper and includes a presentation and discussion of spatial analysis, integration, and interpretative visualization results using cartographic methods.

The pedagogy

To bypass this limitation of fixed category answers to questions when evaluating pedagogies, this paper presents both pedagogy and an integral mixed methods assessment approach to assess that pedagogy incorporating a sequential exploratory field study design. This qualitative field study integrates written essay responses to two open-ended survey questions, using open coding techniques and Multi-Dimensional Scaling (MDS) and cartographic visualizations, to expose underlying semantic themes evident in student assessments of one particular assignment in the pedagogy. Student feedback essays were submitted anonymously, collected in an unmarked envelope. Students could opt out by not submitting essay answers. The essays were returned to the students after coding, 2 weeks later. The essays were returned to students along with other assignments; making it impossible to discern who wrote what, or if they wrote anything at all. Hence, there was no impact on grades.

In China, students pursuing Doctoral degrees must be published in peer-reviewed SCI indexed journals as a condition for advancing to candidacy. As many of the leading scientific journals are English language publications, non-native speakers need to master English language reading and writing skills so they can participate in international scientific discourses (Hess and Ghawi 1997). To be recognized internationally, Chinese researchers regardless of national origins, must become “transnational researcher-writers” (Singh and Fu 2008) with the capacity to fully engage with the disciplines globally. To that end, Chinese universities offer high-level English writing and speaking courses.

Given the current international world order, English is a privileged language, linguistically reinforcing power inequalities existing at the global scale. Scientific articles published in double blind peer reviewed journals, international conferences, and meetings conducted in English are the principal means for sharing and ultimately validating truth claims; therefore, proficiency in formal English writing and speaking is essential for success in scientific and engineering disciplines. In American universities, writing skills have been recognized as a problem area at all levels, prompting writing across the disciplines initiatives at many universities (Conrad 2017). Hence, our research deals with the problem of localization of best practices, and at same time, exploring inductive techniques to improve subjective understanding of student interests and needs. New pedagogical practices in the context of this paper are active learning, project-based approaches to either transmit the core ideas in a domain or guide students toward developing skills related to discovery, and use of information or existing knowledge to generate and communicate new knowledge. Ultimately, writing up research results involves twisting out narratives that comply with the ontological and epistemological standpoints integral to a scientific community based principally in the West.

Scientific Writing and Research Methodology (SWRM) is a course offered by Wuhan University that combines English composition and exploration of scientific publishing to train our PhD students in writing techniques and methods appropriate for scientific papers. Writing promotes synthetic thinking and deeper understanding of one’s own field, and permits the communication of ideas, thus students need to learn to frame their work in relation to questions and theory within a specific discipline. (Weaver et al. 2016; Chang 2013; Tomasek 2009). It is taught in the first or second year of a graduate program leading to a PhD and is required course organized around four tasks and a final paper. Grades are essentially irrelevant as short of plagiarism or failure to submit, all participants will earn a score of at least 80/100 points. They can resubmit assignments until they are satisfied with the grade, mimicking the peer-review process.

Each task is a puzzle to be solved, involving locating, synthesizing and communicating information. These tasks include an analysis of journals and journal standards, an annotated bibliography, a paper outline with figures and tables, and a one page extended abstract of the final paper. For non-native English users, learning ways to circumvent language issues and differences in writing conventions requires critical thinking and problem solving skills (Gweon et al. 2017; Torregosa et al. 2014). Three key issues are recognized as particularly challenging to second language writers, English proficiency issues, the language-related components of academic writing and most critically, the research topic selection process (Bitchener and Basturkmen 2006). Throughout the course, topic selection and formation of an appropriate argument is a recurring theme, as that is one of the most difficult aspects of academic writing (Tajino 1997). Lectures, talks, and in-class assignments included small group discussions and sharing, presentations on library resources, brainstorming, and concept mapping techniques support work on the four assignments.

Non-native speakers of English, in particular Chinese students learn academic English, but this training rarely addresses the writing style used to present research (Commandeur and Heiser 1993). Hence, the course in SWRM is designed to help Chinese writers navigate the process of converting their research into concise, well-organized research papers in English by empowering participants with critical thinking and problem solving tools to navigate around language proficiency issues in the writing process. A condensed version of the syllabus is seen in Table 1, showing topics, tasks, and products.

Table 1 Showing topics, tasks, and products
Full size table

Table 1 shows the structure of the class, and how the various kind of activities, related to research and writing, are linked to lecture themes. The class is comprised of sixteen one and a half hour sessions for 8 weeks. The starred items in the product column are assignments turned in for grading based on a set of clearly defined rubrics. The two double starred items, the outline and graphs, images, or tables are integrated into a single one-page assignment for peer review and grading. Assignments can be resubmitted as often as students chose, until they are satisfied with the score or feedback. Each week, there is at least one in-class writing assignment and one activity such as group brainstorming exercises or individual presentations.

In-class activities are designed to facilitate interaction across domains, language groups, and work teams, thus encouraging the development of shared vocabularies and higher-level academic communication skills. This in turn will stimulate critical thinking and creative problem solving capabilities, as measured by graded assignments. However, in active learning environments, students and instructors are co-learners, and assignments must be evaluated in terms of relevance to student interests and needs rather than in terms of a fixed set of instructor aims (Young and Miller 2004; Kuebel et al. 2018). The stage of research and basic command of English communications skills were uneven. Some students were finishing research and in the process of writing results up for publication, while others have not even identified a research question or topical area.

Student perceptions must be actively solicited. This is not entirely straightforward, given the power dynamics of a classroom setting. Rather than using a structured questionnaire or interview technique that presupposes a range of responses, a sequential exploratory field study design was developed and implemented to elicit student perceptions from a state of evenly suspended judgment for inductive analysis and interpretation to guide continual improvement and modification of the pedagogy. The primary ethical commitment of the instructors was to the students. In China, there are no human subjects review boards; oversight is exercised by the student affairs office. Foreign researchers must work collaboratively with Chinese colleagues as stipulated in contract, work permit, and visa agreements. Thus, this research was collaborative with the involvement of students, staff, and faculty in addition to the instructor. The objective of this research was to find out what students thought about the pedagogy by analyzing their perceptions of one key lab, despite the power relationship between teacher and student. The research question can be framed more abstractly as the problem of other minds; asking how can people communicate given the limitations of language.

Sequential exploratory field study design

The intent of this research was really to explore student perceptions and needs subjectively from the standpoint of qualitative research, adopting an attitude of evenly suspended disbelief to arrive at a thick description grounded by the axial codes derived from textual analysis. Effectiveness in this instance was understood as whether or not the course content matched student interests and backgrounds. The power differential between teacher and student created a dilemma as direct approaches to ascertain student needs and interests would necessarily reflect the biases of the instructor. The qualitative, mixed methods exploratory approach, combined with member checking and visualization was an attempt to generate insights in the same way as anthropological field studies.

The protocol was an exploratory mixed method field study and not an experimental research design incorporating control groups and pre- and post-test measurements. Students answered two open-ended questions in a one-paragraph format, suitable for qualitative textual analysis. The aim of qualitative research is not generalized explanations, but rather grounded interpretation that generates empirically testable propositions. As a part of a reflexive pedagogical practice, these techniques were deployed to modify the course content and materials. The research question can be framed more abstractly as the problem of other minds; how can people communicate given the limitations of language when language is all we have?

A sequential mixed method approach is a set of procedures that combines qualitative and quantitative techniques in a single study to get a deeper understanding of a research problem (Dal 2013). An exploratory research design devised using inductive field techniques for data collection and analysis, and quantitative data mining employed to reveal deep themes inherent in the data. The connection between quantitative and qualitative data collection and analytical methods however is fragile (Creswell 1994). Quantitative visualization techniques incorporating cartographic methods act as a bridge to create maps to aid interpretation and review. Following reference (Hess and Ghawi 1997; Coxon 2005; Ivankova et al. 2006; Kruskal 1964) a graphical representation of the mixed-methods procedures was used to visualize the sequence and priorities.

Figure 1 shows each of the six phases of this exploratory study, in three columns by sequence, with the procedures and products. Essentially this was a qualitative study incorporating quantitative techniques for visualization designed to prompt classroom discussion and exploration of semantic links between themes.

Fig. 1
figure 1

Visual model of the exploratory field study of student assignment perceptions

Full size image

Participants and qualitative data collection techniques

The participants included 58 Chinese, Algerian and Spanish first year, first term PhD Geo-informatics and Software Engineering students with CET-6 level or equivalent English competency, who completed the first of four assignments. This assignment asked them to collect, assemble, analyze, synthesize, and present graphically, key information about five key journals in their research area. Student research domains spanned the spectrum of the geosciences, but with little commonalities between their respective domains in Fig. 2.

Fig. 2
figure 2

Student research interests across geo-informatics disciplinary domains. All of the students from non-English speaking countries while one third of the class from other countries

Full size image

The largest group studied Geographic Information Science, 21%, followed by Remote Sensing, 17% trailed by other sub-domains and allied disciplines. Students from different domains might use different vocabularies and methodologies, even within a language and cultural group. At the initial phase of the study, it was not apparent that domain specific technical terms might prevent communications. Thus, this exploratory study focused on themes emergent within the entire group with no distinction between nationality, gender, or domain in the collection or evaluation of the data. Further, this approach added more assurances of anonymity when collecting data. In this setting, visualization is indispensable as classroom discussions between students occurred in English, Chinese, Arabic, Berber, French, and Spanish.

The classroom setting was a traditional lecture hall, not conducive to the seminar type environments most suitable for writing and composition courses. The class size was not ideal. Thus, the course focuses on active experiential learning from assignments. Given the centrality of imagery and cartography in the Geosciences, graphical presentations of data are an essential part of the communications repertoire. The objectives of the assignment used in this study were; to learn the basics of using academic indexes, understand journal aims, scopes, standards and rankings, and to analyze, synthesize, and present the results in an easy to understand, one-page graphic. They were allowed 2 weeks to complete the task. During that period, lectures focused on academic search tools, journals, and the peer review process. An example of a final assignment is shown in Fig. 3.

Fig. 3
figure 3

One solution to the assignment

Full size image

After the assignment was complete, in the first qualitative phase (Fig. 1) students were asked to write a three to five paragraph essay in English on “what they thought about the assignment” and “what they learned”. The data were these short essay answers to the two open ended questions, N = 58. Students were asked not to include any identifying information, and the essays were collected anonymously. Attendance is not taken, and attendees might or might not be officially registered for the course. The actual, list of formally enrolled students is not made available until the end of the entire course. Typically, about one third to a half the participating students are not officially enrolled. Additional supporting data included instructor’s ethnographic field notes and memos collected routinely throughout each semester. The unit of analysis was not individual students, but themes or codes emergent in the essays and how those themes were semantically clustered across all 58 sets of essays.

Open coding and connecting approaches

In the second qualitative phase, the essays were analyzed using open coding techniques to extract expressions to determine axial codes across all the essays. There were twelve axial codes, Table 1; these in turn were transposed and multiplied to reduce dimensions in the third phase of the exploratory study (Fig. 1), connecting the qualitative phases to the two quantitative analysis and visualization phases.

Table 2 is key expressions used as primary codes by frequency of mentions across all 58 essays. In the 2 by 2 matrix, each mention in an essay was a link between codes.

Table 2 Key expressions used as primary codes by frequency of mentions across all 58 essays
Full size table

The frequency distribution of expressions (codes) in Table 2 gives an idea of student perceptions, but not the connectivity between expressions (codes) or an indication of underlying core themes. At this point, social network analysis was applied to explore the relationships between codes and participants, and visualized as two and single mode networks, directing attention to potential clusters, prior to the fourth phase.

In the fourth phase (Fig. 1) actually the first quantitative phase, to explore those connections, a 2 by 2 matrix of the raw correspondences between the 12 codes (distributed in Fig. 4) was converted into a Euclidean proximity matrix, and visualized using Multi-Dimensional Scaling (MDS) in the Statistical Package for the Social Sciences (SPSS). MDS is a family of exploratory data mining techniques used to understand and visualize patterns in data. The MDS visualization executed in SPSS illustrates the relationships between codes spatially as a map in two dimensions. In this instance, codes closer together are more alike than those farther away from each other, and points located at the center of the map illustrate codes that occur more frequently and are more connected to other codes than points representing codes at the edges or periphery of the map.

Fig. 4
figure 4

Four combined themes: communication (red), organization (green), creative (yellow) and technical (violet). (Color figure online)

Full size image

This preliminary mapping of codes can prompt meaningful and productive discussions concerning strategies and tactics in marketing campaigns as well as help researchers define more structured approaches to understand a market segment or develop market segmentation models. MDS illustrates the pattern of similarities in a dataset from a set of vectors in p-dimensional space; the matrix of Euclidean distances among the vectors ideally corresponds to a function of the input matrix according to a fitness function or stress (Singh and Fu 2008; Shepard 1962a, b). The fitness function or stress suggests how closely the MDS map represents the proximities found in the 2 by 2 matrix, and is not a test of some quantitative hypothesis about student perceptions or attitudes.

For the data set in this exploratory study, the stress value = .15910 in an ASCAL analysis with an RSQ of .90081 for two dimensions. The stress values for two dimensions fell within the range considered acceptable after six iterations. The visualization however was difficult to interpret. A second analysis using the PROXSCAL method (Commandeur and Heiser 1993) yielded a less satisfying goodness of fit value but a more readable graphic result, The normalized raw stress = .06615, and Stress-I = .25719.

Spatial analysis, integration, and interpretation

The fifth phase, the second phase of the two quantitative data analysis and visualization phases as shown in Fig. 1, used spatial analysis tools in Arc Geographic Information System (GIS) software to explore the data cartographically further, using the proximity matrix and x, y coordinates. Values from the proximity matrix were joined with the values for the proximity table and mapped, using Inverse Distance Weighting (IWD) to convert the proximity values for each code into continuous uneven surfaces. IWD is a deterministic algorithm and one of many interpolation techniques used in spatial analysis to translate point level values into continuous surfaces. In IDW interpolation, the assigned weight decreases as distance from one point to other points increases. In this instance, the proximity values for each of the 12 codes to every other code served as the weighting value. Interpolation generalized the relationships between points representing each of the codes. The results were grouped into four themes using the average proximity value, based on a qualitative analysis of these visualizations on two continuums: visualization and techniques, and communication and organization.

These maps guided interpretation of the unstructured student essays in an iterative process, uncovering underlying themes in the semiotics of the textual data. The proximity values of the three most common codes were used to produce maps. Each of the 12 codes were mapped and combined to create two Continuous surfaces along the x and y-axes. The process of making these layers was an active reflexive engagement with the data, collaboration, and dialogue; eventually yielding themes that linked together the twelve codes into related four themes. All four themes along the x and y-axes were combined, as shown in Fig. 4. This map proposition captures commonalities as well as ambiguity and dynamic tensions in student perceptions of this first class assignment.

Analysis and discussion

The results as illustrated in Fig. 4 are an artifact of the two open-ended questions the essays answered, “What they thought about the assignment” and “what they learned”. Three broad regions appear when looking at this visualization of transparent colored overlays illustrating the four general themes, communication, techniques, organization, on both the x and y-axis. On the bottom right, the largest region in green are themes related to organization and techniques, while the upper left region in yellow includes themes linked to creativity and communication.

The upper right hand region is more ambiguous, incorporating themes dealing with communication and techniques, in red and purple. These results are an artifact of the diverse composition of the class and the unique interests of each student (Fig. 2). Starting with 12 codes, we intuitively reduced them to four themes along two axes, and then, in turn further generalized them into three regions, with the two questions traced along the z-axis, as clustered codes at the center and dispersed codes along the edges of the map. This mapping illustrates one potential set of student perceptions of the assignment.

Qualitative analysis of key themes emerging in those short writing assignments (Fig. 4) reveals two continuums along the x and y-axes, techniques and organization and communications and visualization, with little overlap. The third dimensional z-axis, representing core codes as seen in Fig. 4 can now be related to themes, also along a continuum; what students individually learned and shared thinking about the assignment.

Figure 4 is a small audience mapping, thus it is unintelligible and inaccessible to an audience unaware of the context; but at the same time is well suited to generate subjective understandings of student perceptions in an exploratory, collaborative, interpretive process. This is the final phase of the mixed methods approach as seen in Fig. 1.

The themes can be further generalized in terms of domain and course related distinctions, as summarized in Table 3, a schematic chart of the key ideas.

Table 3 Schematic chart of key themes and interpretive generalizations from the mapping in Fig. 4
Full size table

As shown in Table 3, class participants approached this assignment with different focuses on aspects of visualization and communications related to gathering and synthesizing information. Other participants were more focused on technical issues and organization of materials. These were very individualistic, and dispersed expressions, probably related to the heterogeneity of the field and unique qualities of research. At the same time, participants had shared perceptions of the assignment and its intended goals. As a challenging exercise, the assignment achieved its purpose of confronting and solving a problem within constraints.

The results were presented in the class to stimulate classroom discussion in English, Chinese, and French incorporating participant feedback (summarized in English) to validate the trustworthiness of the co-produced knowledge. The discussion revolved around natural language processing, visualization, and data mining techniques as well as mixed methods protocols, and the differences and similarities between inductive qualitative methods and quantitative approaches to research. Based on these results, more techniques to increase information exchanges, and collaborative work were implemented within the confines of the classroom structure and in accordance with student expectations as suggested from field notes of observations, discussions with other instructors, and group discussions with the class about these results. This study was limited and exploratory aimed at description and interpretation rather than explanation. Based on these findings, a more structured survey instrument will be designed for a more rigorous evaluation of classroom pedagogies.

Conclusion

As part of the innovation in Wuhan University, SWRM course deals not only with English but also with exploration of scientific publishing and the philosophy of science. Task oriented problem-solving experiences are central in a pedagogy to promote critical thinking in this required course. Student needs and interests are the foremost concern but students come from an array of software engineering and geoinfomatics domains and do not share common disciplinary vocabularies or languages. Compounding this difficulty, participating students are from multiple countries, whose command of spoken and written English is not a given. In this context, it is paramount to understand student motivations and needs in an unbiased way. Unstructured questions rather than structured surveys are one means to expose student perceptions. Given the heterogeneity of responses, a qualitative approach for evaluating and assessing the results is necessary. Open coding techniques are a way to extract themes, but these themes are in themselves, difficult to interpret. Multi-Dimensional Scaling (MDS), enhanced with cartographic methods, was used to explore student perceptions of a class assignment in the SWRM course; the resulting graphical visualization provided a basis for qualitative interpretation. What could be inferred from the visualization is that while the participating students generally perceived the exercise as challenging, nevertheless the exercise also helped them understand journal requirements and surveying information. Furthermore, the results suggested that this consensus breaks down in terms of each participant’s own goals, domain, and research interests. Student responses fell along two continuums that could be described as techniques and organization (x-axis) and communications and visualization (y-axis); respectively, with little overlap. What we gained from this process was a stronger subjective understanding of the motivations driving these students, unmediated by the forced response categories of a traditional questionnaire with Likert type questions. As an exploratory study, the insights informed subsequent iterations of the course. Unstructured questionnaires, open coding techniques, and MDS visualization might also prove to be helpful in the process of devising and assessing other student centered pedagogies in which teaching goals related to a fixed body of knowledge are secondary to imparting critical thinking and problem solving skills, independent of the content itself.