Introduction

The mathematical sciences are essential in undergraduate education and the ability of students to succeed is fundamentally dependent on their understanding of basic mathematical tools and concepts. In conjunction with this there is a need for better integrating the mathematical sciences into other disciplines and improving instruction in the mathematics through incorporation of other disciplinary perspectives. Unfortunately, typical institutional separation of courses often makes it difficult for students to grasp the intimate connections that exist between mathematics and its applications in other disciplines. To help make these connections clearer a library of interactive, web-based learning modules linking important mathematical topics with contemporary applications in various fields has been developed. This library covers the mathematical areas normally studied by undergraduate students, including calculus, linear systems, and probability and statistics. The same modules are also used in many science courses at all levels, from the freshman to senior year. Linking subjects in this way has provided critical connections across disciplines and also provided longitudinal subject reinforcement as students progress in their studies.

The 40+ modules developed to date are part of Project Links, an NSF supported undertaking based at Rensselaer Polytechnic Institute, with collaboration from the University of Delaware, Hudson Valley Community College, Siena College, and Virginia Polytechnic Institute. Included in this effort has been an innovative and comprehensive evaluation program to assess the viability and effectiveness of the modules. For this we collaborated with The Evaluation Consortium from the University of Albany School of Education.

The paper begins with an overview of the project, including a discussion of its objectives and organization. After this the process used to develop and test the modules will be described. This will include the evolution of the learning outcomes in a module as well as the technical requirements needed to construct an effective technological learning tool. After this the evaluation outcomes will be presented along with conclusions on what transpired.

Overview of Project

Project Links was conceived to tie crucial topics in mathematics with one or more contemporary applications in science and engineering. The four main objectives of Project Links are:

  1. 1.

    To stimulate greater cooperation in educational development across traditional disciplinary boundaries.

  2. 2.

    To encourage interactive teaching and learning strategies and to produce instructional materials for use in workshop or studio-type courses.

  3. 3.

    To create a library of interactive learning materials that link topics in mathematics with applications in science and engineering.

  4. 4.

    To continue our efforts in the application of contemporary technology for educational purposes and to encourage the widespread distribution of the results.

The central component of this effort was the production of instructional web-based modules that exploit the Internet and its attendant technologies, including the Java programming language. For example, in regard to item (2), the modules were designed to be used in a studio classroom, with an instructor present, with significant student-to-student interaction, and with many open-ended challenges included (a listing of the modules is given in Figure 1. At the same time the modules were not developed to replace textbooks, professors, or entire courses. Rather, they were designed to give the instructor the flexibility to emphasize certain well-contained topics that are a one- to three-day part of a regular course.

Fig. 1.
figure 1

Listing of modules, from the Project Links webpage, by general mathematics topic. There is also a listing by general application topic on the web-site.

Full size image

The library of modules, and how they are used, reinforce many current ideas on how to successfully integrate technology into the learning environment. In particular, technology can be used to establish an effective learning environment by using real world problems, by providing scaffolding support, by increased feedback, by building communities of learners, and by expanding opportunities for teacher learning (Hilton, 2002). As will be seen later, all of these are central components of the project. Similarly, the project closely reflects the criterion measures identified in the National Science Education Standards (National Committee on Science Education Standards and Assessment, 1996). For example, Science Education Program Standard C states, “The science program should be coordinated with the mathematics program to enhance student use and understanding of mathematics in the study of science and to improve student understanding of mathematics.” This is one of the core objectives of the project. A detailed comparison of how Project Links aligns with the standards set by this group, the National Science Education Standards for Teaching, the NCTM Content and Process Standards, and the ISTE Standards for Students can be found at the project web site (Project Links, 2005).

To successfully integrate faculty from multiple disciplines to produce effective technologically enhanced learning tools required careful organization and planning. Accordingly, the development and testing of each module involved the close interaction of three teams: the faculty team (who were responsible for the content), the technical support team (who were responsible for template development and interface design), and the evaluation team (who were responsible for usability tests and external reviews).

One reason for separating the faculty from the technical component of module development is that the creation of educationally oriented content presents new challenges and opportunities. The use of interactive learning materials, and the resources available electronically, involves the development and analysis of new learning techniques and strategies. Simply converting textbook material to a web format is insufficient and the role of multimedia objects makes the potential for student interactivity much greater.

An issue that arose immediately between the faculty and technical teams was what functionally was necessary and what was actually possible. An early example of this was the expectation, and strong desire, by the faculty to be able to manipulate three-dimensional objects in the Java applets. The fact was, however, that at the time this was not possible in Java although it was expected it would be in the near future. Accounting for the developing nature of the technology therefore began integrated into the design of the modules. Another issue that arose concerned disciplinary differences related to emphasis and presentation of the content. A example here was the desire by the physicists to use scalar notation while the mechanical and electrical engineers wanted vectors. For these reasons, we developed a framework for presenting technical course content useful to multiple disciplines. This framework includes learning modules plus an index to shared key concepts. We also built customizable tools that promote a high level of student interactivity, plus authoring tools enabling content creation and maintenance in the quickly evolving technological environment.

In order to assist in the creation of inherently interactive documents, a complete editing and publishing system was developed by Project Links. This included the design of a consistent interface where the common elements include graphical components, a navigation tool, as well as a general look and feel that remains throughout all the materials. This allowed content developers to focus on the production of materials and how to effectively use them in the classroom. Moreover, our development environment encourages the re-use of pre-built interactive objects, hence minimizing the content developers need for technical expertise. At the same time, our environment is optimized for automated maintenance and upgradeability of materials. The result is a single integrated system. The development of this software environment will be discussed below and a focus on the use of this environment in the classroom will also be presented.

Module Design and Objectives

The modules are designed for small-group student interactions with an instructor nearby. They are a self-contained conceptual unit, intended for use over 1–3 days in the normal course of the term. They rely heavily on hypertext construction, animations, and interactive Java applets. However, a complete module also includes an Instructor’s Manual, which presents the design intent of the module, recommendations for use in the classroom, and handouts designed to accompany the interactive module. We have found that the handouts, which are mostly worksheets based on the module subject matter, are a critical component of the learning process in the classroom. There are several reasons for this. One is the multi-modality achieved when combining writing with the interactive and collaborative nature of the module (Kress et al., 2001). A second reason involves the properties of a collaborating group, which includes the topic discussed, the insights that emerge, and the collaboratively determined framing of the module’s outcomes (Cobb, 1995; Sawyer, 2004). Together these are opportunistically emerging processes and the worksheets provide a scaffolding to help the group access and then master the material.

Many questions and examples in each module are purposely left open-ended to encourage communication and self-discovery. They are also designed to encourage students to think creatively in how they approach problem solving and how the concepts developed are transferable to related situations. This is done, in part, by providing multiple contexts for learning the underlying concepts, and having the examples that are used in the modules based on real-world situations. To achieve this, the developers incorporate actual experimental results, demonstrations, or design problems. The modules use videos, real-time experiments run over the web, animations of experimental results, and data-reduction.

A typical example illustrating the above ideas can be found in the Spring Mass Module where students study the dynamic behavior of a weight at the end of a spring. This module is used by multiple departments, including math (in sophomore differential equations), physics (in mechanics), and mechanical engineering (in mechatronics). A schematic of the system is shown in Figure 2 along with the experimental apparatus used to generate data for this problem. The module is capable of acquiring and using real time data and this is typically done by the students in the science and engineering courses. In math it is usually the instructor who carries out the experiment although we also have pre-recorded videos and data sets built into the module for those who prefer to concentrate on the analytical aspects of the module. After analyzing the mathematical and physical problems for this system, students are then given the images and questions shown in Figure 3. Each picture is a link to a video showing common objects exhibiting the dynamic behavior of a spring mass system. Based on what they have learned in the module they are asked the following:

  • Which of these examples are forced spring mass systems?

  • Which show significant damping characteristics?

  • When modeling these systems, what assumptions must be made to simplify while maintaining essential characteristics in the model?

These questions are open ended so they can serve as collaboration and classroom discussion points. More importantly, however, is that they are an integral component of the learning process as they are reconsidered at multiple places within the module. The reason is that is essential that the students be able to transfer the fundamental math and physical concepts developed in the module to non-classroom situations and these questions are intended to address this point.

Fig. 2.
figure 2

Problem investigated in spring mass module. On the left is the idealized lumped parameter model of the spring mass system shown on the right.

Full size image
Fig. 3.
figure 3

Common objects demonstrating spring mass behavior.

Full size image

Module Layout

A page from the drag forces modules is shown in Figure 4 to illustrate the content and functional navigational schemes. In the upper left hand corner is the PRIOR/NEXT arrow buttons. This is the path through the module the authors recommend in normal use. The pages used in this path are explained in the materials made available to the instructor.

Fig. 4.
figure 4

Typical module page illustrating the layout and navigational systems used.

Full size image

Along the left side of the browser window is the content navigation bar (Objectives, ..., Hints). This is a clickable list of the main module topics and sub-topics. Each of these may also be reached when using the PRIOR/NEXT arrows, but this list allows the student to jump around, as one would skip through parts of a textbook. A triangular icon appears next to the current topic shown in the frame.

Along the top of the browser window in Figure 4 is the functional navigation bar, used to branch off of the main topic areas. There are five choices available to the developer during design of the module. They are Concepts, Discover, Applications, Collaboration, and Practice. Any or all may be used from any one page. Again, a small triangular icon appears next to the current branch. Concepts are main topics, usually those that appear on the side navigation bar. Discover pages lead to questions or exercises that allow the student to explore a new area with information acquired from the Concepts pages. Applications are current uses of the topic in real-world situations. Collaboration supplies challenges that must be solved with a partner or by discussion between groups, perhaps with instructor guidance. Practice contains problems that the student must answer to allow the instructor to assess the learning that has taken place. These can have the form of pencil-and-paper worksheet problems, applets, or online submissions.

Towards the bottom left corner of the browser window in Figure 4 is a listing of reference or help pages. Four pages are available to the student at all times from anywhere in the module:

  • Crib-sheet—a summary of important concepts and formulas used in the module

  • Library—a list of the multimedia elements and major concept pages included in the module and a link to the site-wide glossary

  • Map—a conceptual map of the module material and/or a site map (we employ the Hyperbolic Tree (2001), which allows for dynamic links to anywhere in the module)

  • Help—help files for students, instructors, installation and technical tips, and known issues and incompatibilities

Furthermore, the front page of any module can always be reached by clicking Module Home at the bottom of the side navigational bar.

As stated earlier, module development is the responsibility of three teams, which work closely together: the content experts, the technical group, and the external evaluators. The content experts are two to four faculty, at least one from math and one from outside math. The technical group, which includes HTML and Java programmers as well as an interface design expert, is responsible for implementing the content and maintaining the web site. The evaluators are responsible for carrying out the various external assessment tests that are described later in this paper. There is also a small select committee consisting of members from all three groups that oversee what modules have in common, which includes the interface design, navigation capability and the functionality available in the modules.

Content Experts

Content experts, also known as faculty, provided exceptional ideas for subject matter but required time to learn how to communicate effectively with the other teams. One of the primary communication issues with the technical group centered on the construction and use of storyboards in developing multimedia modules. When working with the evaluation team, the faculty needed to learn the reasons and benefits of formative assessment, and the value of an observer visiting the class when the module is used. To help with all of this we had several workshops, where the module development process was outlined and where they learned how to develop collaborative teams involving faculty outside their own departments. Interestingly, it was relatively easy for the faculty to find common ground on which they could develop materials for their respective courses. This required an interest in educational innovation, an ability to think a bit differently on what is or is not important in the discipline, and the willingness to discuss this in a group setting.

To supplement this effort we also provided written materials detailing the steps and outcomes for good module development. Nearly all faculty stated at least once that the best way to learn how to develop a multimedia educational module was through an example or model. The items we provided them for this are listed below and all are available on the project web site (Project Links, 2005). They were required reading for any and all who participated in the project.

  • Design Questionnaire: This consists of a sequence of questions concerning the scope of the content of the proposed module. They were asked to describe the multidisciplinary nature of their proposed module, their constructive use of interactivity and multimedia, and examples of real-world applications. They were also required to provide a draft storyboard for the module. The latter was used to help the content developers work with the technical team.

  • Module and Template Overview: This provides an outline of the conceptual units that make up a module, the standard template for the various web-pages within a module, and the functional navigational paths through a module.

  • Development Process: This includes explanations and resources to help develop a Links module, including the milestones expected during the development process. For example, it explains when and where assessment is used, how the technical teams will contribute to the effort, and the role of the storyboard in the development process.

  • Developer Resources: Information about the resources available to developers, including programmers, media requirements, Java applets, and video tools and requirements.

  • Copyright Policy: Steps to take in protecting our material and that of others.

  • Navigation Scheme: This provides schematics of the basic functional and content navigation schemes and explains the differences between them.

  • Storyboard Examples: A workshop was held so the faculty could learn how to create, and then use, storyboards. These were then used as our examples that we made available to faculty who were new to the project.

  • Applet Storyboard Example: We provided examples of storyboards for the various interactive applets that would be used in the module.

Assessment

Each module passed through a rigorous assessment program, involving both internal and external evaluations. To identify the progress of a module through this process, a version number is assigned to each module and these are shown in Figure 5, and in Table I the various scales are described. Alpha and beta tests are conducted by the Interface Designer, who is a member of the technical group. The test descriptions are:

Fig. 5.
figure 5

Version numbers, and indicator system, used to identify progress of modules through the development process. This is on the same web-page as the listing in Figure 1 and is linked to the information in Table 1.

Full size image
Table I. Module version numbers and their meanings for the indicator in Figure 5
Full size table

Alpha Testing

The interface designer checks modules on all platforms and browsers for broad editing, aesthetic and usability issues. Review includes checking for:

  • Broad Issues—consistency with the Links format, logical use of the content and functionality navigation, general usability, bugs/technical problems with the animations, videos, applets, etc.

  • Editing Issues—titles accurate, links broken, pages missing or incorrectly named, clarity of the text, clarity of graphics/charts, etc.

  • Organizational Issues—how is the material presented? Does the module make sense? Can there be more interactivity? Is the medium used effectively?

  • Aesthetic Factors—does the module look good? Can it look better?

Beta Testing

The interface designer provides a fine-toothed copyedit for grammar, punctuation, misspellings, broken links, broken applets, etc. Beta testing also ensures that the design and usability look good and work smoothly.

The Evaluation Consortium conducted the formal external assessment. This group standardized the process and implemented the evaluation plan. Four types of testing were designed and/or conducted, and these are the Content Review, Usability Testing, Educational Technology Review, and Pilot Testing/Classroom Observation.

  1. 1.

    Usability Testing: A small number of students are asked to use the module and provide information pertaining to usability from a student’s point of view. An observational checklist and interview protocol is used which includes videotaping students while using the module.

  2. 2.

    Educational Technology and User Interface Review: This is a standard review that looks at the module from a technological and instructional point of view and provides validation of the module’s appropriate use of current learning theory. A written review is provided based on a checklist of instruction design concepts.

  3. 3.

    Pilot Testing and Classroom Observation: This involves classroom observations and semi-structured interviews with students. The objective is to determine how well the module accomplishes its intended tasks. To date over 1000 students have been observed and interviewed. Data has also been collected from multiple institutions, as well as multiple math and non-math courses (approximately 50).

  4. 4.

    Content Review: Once the module has been developed to the satisfaction of the authors and the technical manager, a qualified expert in the subject matter is found outside of the developing institution, and provided with a checklist. In general, content reviewers are asked to work through the module, validating the module content and the accuracy of the materials presented. They are asked to complete a short review of the module delineating content viability.

Evaluation Findings

Findings from the evaluation program obtained from student perceptions of module usage are presented below. For comparison the results for Year 1 and Year 2 are given. This is done to indicate the affects of improvements made in the modules based on the first year assessment. The data reported here are from all modules used during the indicated years and includes both math and non-math courses.

Relevance of Module

One issue addressed was the perceived appropriateness and relevance of the module content. This information was gathered using in-class observations, paper–pencil surveys and semi-structured interviews participating in classroom activities. The results are given in Table II. From the data it is seen that there is a marked increase in the second year in the percent of students who perceived the modules as relevant to the coursework and relevant to the academic area. What is significant is that this is true whether or not it was a math course. There were also marked increases in all other surveyed responses in this category. This is strong evidence that the students consider the modules to successfully bridge multiple disciplines and break down the disciplinary barriers commonly found in such courses.

Table II. Student perceptions of module relevance. Values are % of those who agree with the given statement
Full size table

Perceived Cognitive Outcomes

A second assessment issue was the students’ perceived cognitive outcomes from using the modules. This information was gathered using paper–pencil surveys and semi-structured interviews with students. The results are given in Table III. From the responses it is clear the students have a strong perception that the modules facilitated learning of the content in a variety of ways and there was also a marked increase in all categories in the second year. Two items of particular interest are that 92% agree using the modules helps them to apply the course content to new problems and 85% agreed that they help to transfer knowledge to problems outside the course. The ability to transfer concepts and methods to new situations is critical in a student’s educational development and it is significant that they consider the modules successful helping them with this.

Table III. Student perceptions of cognitive outcomes. Values are % of those who agree that the modules assisted them in the tasks listed below
Full size table

When the students were queried as to the benefits of using the modules, they reported a variety of direct cognitive benefits, including enhanced learning and problem solving skills, as a result of module use. More specifically, students reported that the modules provided practice with the use of collaborative skills, hands-on/real world applications, and different types of problem solving methods. Faculty reported multiple positive cognitive outcomes resulting from module use, including greater understanding of course concepts, making connections between mathematics and science concepts, and developing problem solving skills.

Perceived Affective Outcomes

A third assessment issue was the perceived effective outcomes of the modules by the students. This information was gathered using paper-pencil surveys and semi-structured interviews with students. The results are given in Table 4. As with the other two categories, the responses in Year 2 are quite positive and, in this case, are approximately twice that was observed the year before.

Table IV. Student perceptions of affective outcomes. Values are % of those who agree with the given statement
Full size table

Conclusions

The assessment results from Year 2 provide strong evidence that from the students point of view the modules are accomplishing what they were designed to do. In particular, the modules successfully bridge multiple disciplines and break down the disciplinary barriers commonly found in their math and non-math courses. Moreover, the modules help apply the course content to new problems and they help transfer knowledge to problems outside the course.

It is also evident that significant improvements were made between the first and second year. This observation generates the question of what exactly was done to achieve this improvement. A partial list of the changes made in the program is given below.

  • Instructor workshop—we held a workshop for the instructors at the end of Year 1 to discuss the evaluation results. This began with a round-table like discussion of what did and did not work, and what changes were needed in each module so they were effective learning tools.

  • Content improvement—all faculty received assessment reports of the modules and they then spent the summer between Year 1 and Year 2 making improvements based on this information and what was discussed during the Instructor Workshop.

  • Technology upgrade—the modules push the technological envelope and the laptops used in the second year were significantly better than those used the first year.

  • Worksheets—even with interactive computer materials involving collaborative projects, the addition of pencil-and-paper worksheets that the students completed as they worked through the module appear to have helped them to more actively engage with the subject matter.

All four of the above listed changes were potentially important factors for achieving the improved student responses in the second year.