Keywords

The Empirical Nature of Design and Development

Design models often parallel the scientific problem solving processes. Thus, the practice of design and development is to a great extent empirical by nature. Therefore, it would reasonable to assume that design and development processes have robust empirical support. Yet historically there has been a scarcity of research on our models, products and tools. While there has been increased empirical work on design and development recently, we have been writing about and advocating this type of research for the past 15 years (Klein, 1997; Richey, 1997; Richey & Klein, 2005, 2007, 2008; Richey, Klein, & Nelson, 2004; Richey & Nelson, 1996).

This chapter is a continuation of our work. It examines design and development research (DDR) by providing an overview of its definition and scope. The major part of the chapter focuses on representatives of recent design and development research. Finally, we summarize this new work with special emphasis on the problems it addresses, the settings and participants examined, the research methodology used, and the role evaluation plays in these studies.

Design and Development Research: Rationale, Definition, and Scope

Opinions on the role of research on design and development often depend on one’s own view of what it actually is. We take the position that design and development is a science, even though it is highly influenced by the creativity of the designer. We approach design and development (and in turn research on it) with the assumption that science and empiricism provide a more effective and reliable route to disciplinary integrity than depending on artistic tactics and craft-based solutions. As a science, design and development should be bound by understandings built upon replicated empirical research. Our models and procedures should be validated. The solutions to our problems should be supported by data. We believe that our field has not sufficiently employed empirical methods to facilitate our understanding of design and development processes. The need for research is especially critical with respect to the models and processes employed by designers and developers. Few models, design strategies, and tools employed in practice have been empirically tested and validated. This is the gap that design and development research seeks to address.

Design and development research is a type of inquiry unique to the Instructional Design and Technology (IDT) field that is dedicated to the creation of new knowledge and the validation of existing practice. We define DDR as “the systematic study of design, development and evaluation processes with the aim of establishing an empirical basis for the creation of instructional and non-instructional products and tools and new or enhanced models that govern their development” (Richey & Klein, 2008, p. 748). This definition aligns with recent suggestions that professionals in the IDT field facilitate learning and improve performance by creating, using and managing appropriate instructional and non-instructional interventions (Definition and Terminology Committee of the Association for Educational Communications & Technology, 2007; Reiser, 2012; Richey, Klein, & Tracey, 2011).

Design and development research covers a wide spectrum of activities and interests. It includes the study of the design and development process as a whole, of particular components of the process, or the impact of specific design and development efforts. Such research can involve a situation in which someone is studying the design and development work of others. It can also involve a situation in which someone is performing design and development activities and studying the process at the same time. In either case, there is a distinction between doing design and development and studying the processes.

Design and development research is an umbrella term for a wide range of studies that employ an assortment of traditional quantitative and qualitative research methods and strategies. Most design and development research, however, tends to rely more on qualitative strategies and deals with real-life projects, rather than with simulated or contrived projects. Many studies can be viewed as multi-method research.

Understanding the nature of this research is a matter of understanding the range of problems to which it can be applied. It is also a process of recognizing those research interests and endeavors that are not a part of this orientation. DDR does not encompass the following: instructional psychology or learning science studies; media delivery system comparisons or impact studies; message design studies; and research on the profession. While results from research in these areas impact design and development, the study of variables embedded in such topics does not constitute DDR.

Design and development research, as with all research endeavors, leads to knowledge production, a more complete understanding of the field, and the ability to make predictions. DDR reaches these goals through two main categories of research projects: (1) research on products and tools and (2) research on design and development models. We previously referred to these two categories of design and development research as Type 1 and Type 2 developmental studies (Richey, Klein, & Nelson, 2004). Others have referred to instructional product development studies as design-based research (Wang & Hannafin, 2005), systems-based evaluation (Driscoll, 1984), and formative research (Reigeluth & Frick, 1999; van den Akker, 1999).

Below we describe design and development research in detail and briefly examine 11 studies conducted since 2007 in this line of inquiry. In addition to being quite recent, these studies were selected to exemplify the major categories of DDR, the types of methodologies commonly employed, and the range of research settings examined. They also highlight studies conducted in a variety of locales around the world. We begin by discussing research on product and tools, followed by an examination of research on models.

Research on Products and Tools

The most straightforward type of DDR falls into the first category—research conducted during the design and development of a product or tool. Often, the entire design and development process (analysis–design–development–implementation–evaluation) is documented. In some cases, researchers concentrate only on one facet of design and development (e.g., needs assessment). Many recent studies focus on the design and development of technology-based products and tools.

Below, we discuss three classes of product and tool research. These include studies of (1) comprehensive design and development projects, (2) specific ID project phases, and (3) tool development and use. We review recent representative product and tool research in each of these categories.

Recent Comprehensive Design and Development Research

Studies of comprehensive design and development projects usually demonstrate the range of design principles ­available to practitioners. Frequently, the entire design and development process is studied and documented. The design processes used in a particular situation is described, analyzed, and a final product is evaluated. Consistent with ­predominant practice in the field, the procedures employed usually follow the tenets of instructional systems design (ISD), encompassing analysis through evaluation. This is the case in a research study by Visser, Plomp, Armiault, and Kuiper (2002) who describe the design and development of a product which addresses learner motivation in distance education programs. This work includes an initial pilot study, as well as a year-long try-out and evaluation of the product. A study by Sullivan, Ice, and Niedermeyer (2000) is also an example of comprehensive DDR that focuses on the impact of an instructional program. These researchers use field evaluation to test a comprehensive K-12 energy education curriculum that was the product of a design, development and implementation project on-going for 20 years. While few researchers have the opportunity to study an instructional program for such a long period of time, recent research continues to examine comprehensive design and development projects. Below we discuss two such projects.

Developing a Web 2.0 System for Community and Teacher Use

Research by Cifuentes, Sharp, Bulu, Benz, and Stough (2010) provides an example of a comprehensive product design study. The purpose of this 2-year study was to investigate “the design, development, implementation, and evaluation of an informational and instructional Web site in order to generate guidelines for instructional designers of read/write Web environments” (p. 378). The researchers implemented and documented the entire ISD process. Needs analysis was conducted on a practical problem—individuals with disabilities and their families have difficulty gaining access to information about support services. Findings pointed to the development of an online directory of resources using the capabilities of Web 2.0 technologies. Design decisions were based on theory including social constructivism, distributed cognition, and rapid prototyping. Formative evaluation occurred throughout product development. Participants included the design team, a variety of intended users in multiple locations, college students contributed resources to the Web site, as well as internal and external evaluators. The researchers provide context-specific findings related to ­problems and issues encountered, resources required, and ­product impact and use. They also give generalized recommendations for others designers of Web 2.0 solutions.

The Cifuentes et al. (2010) study is a good example of comprehensive product design research. The researchers identified and analyzed a real-world problem, used theory and formative evaluation to inform design and development decisions, meticulously documented these decisions, employed multiple research methods, considered issues such as researcher bias and instrument reliability, and collected data from several sources.

A Task-Centered, Peer-Interactive Course Redesign

A descriptive case study by Francom, Bybee, Wolfersberger, and Merrill (2009) provides another example of comprehensive design and development research focusing on a product. This study addresses the real-world problem of converting a passive, face-to-face college biology course to an online course that includes peer-interaction and task-centered instruction. The authors describe how the instructor selected content topics and “complex, authentic tasks that would require students to gain a sufficient knowledge of the subject area in order to complete the task” (p. 37). They also explain how the First Principles of Instruction (see Merrill, 2002) were used to redesign instructional activities and assessments. A formative evaluation was conducted during the first semester the course was offered; data included observations of the instructor, classroom activities, and online discussions, as well as a student survey measuring perceptions of the course and their learning. The authors offer a discussion of how these data were used to revise and improve the course.

While the work by Francom et al. (2009) does not provide the same level of detail as the Cifuentes et al. (2010) study, both are examples of comprehensive design and development research. They report on projects in which a researcher studies design and development while comprehensive ISD processes are used to produce a specific product.

Recent ID Phase Research

Not all DDR pertains to a comprehensive project. Instead, some researchers examine specific phases of an ID effort. These studies typically relate to data gathering phases of the ISD process (e.g., needs assessment, formative evaluation). For example, Klein, Martin, Tutty, and Su (2005) identify the optimal research competencies of graduate students by conducting a content review of course syllabi from several leading instructional design and technology programs and by administering a survey to faculty and students. In addition, Fischer, Savenye, and Sullivan (2002) conduct a formative evaluation of computer-based training on an online financial and purchasing system to verify the program’s effectiveness and identify necessary revisions. Below, we discuss another design and development study that is representative of very recent research on a component of ID.

Formative Evaluation of a Learning Game

A recent study by Sahrir (2012) is an example of DDR on a specific phase of ID, namely, formative evaluation. This research investigates the development of an online vocabulary game for beginning Arabic language learners at a Malaysian university. The researcher employed a mixed-method approach to collect data on prototypes of the online learning game. Data sources included instructors, subject-matter experts, evaluators, and learners who participated in one-to-one, small group and field test phases of formative evaluation. Characteristic of most product design studies, the researcher provides context specific findings (i.e., the online game improved student enjoyment, immersion and knowledge of Arabic). In addition, issues, problems and lessons learned are discussed to inform other designers of similar products. For example, the researcher suggests “there should be sessions of cooperative work and research activities between language teachers … instructional designers and computer experts to design and develop … effective games” (p. 366). This study is particularly notable because it provides an empirical test of how the phases of formative ­evaluation suggested by ISD scholars (e.g., Dick, Carey, & Carey, 2009; Tessmer, 1993) can be used in actual practice.

Recent Tool Development and Use Research

Some researchers concentrate on studying the development and use of tools, rather than on the design of products. These tools may support design and development or teaching and learning processes. Many of these studies focus on computer-based tools and some of this research is directed toward automating design and development. For example, Nieveen and van den Akker (1999) focus on a computer system that serves as a performance support tool for designers during the formative evaluation phase of an ID project. In addition, Mooij (2002) conducted a tool study examining the development and use of an instructional management system for early education. Below we describe a two other studies that are representative of recent research on tool development and use.

Development of Performance Support Tool for Teachers

A recent comprehensive study by Hung, Smith, Harris, and Lockard (2010) illustrates research on a tool to support the teaching/learning process, specifically the design and development of a performance support system (PSS) for classroom behavior management. These researchers “adopted design and development research methodology … to systematically investigate the process of applying instructional design principles, human–computer interaction, and software engineering to a performance support system” (p. 61). The study was conducted in six phases that mirrors an ISD approach. Qualitative and quantitative techniques were used to collect data from several sources. For example, a Delphi technique was used with subject matter experts to enhance the design of the PSS. In addition, elementary and junior-high school teachers completed a survey about system requirements, tested the usability of two prototypes of the PSS, kept activity logs during implementation, and participated in post-implementation interviews. The researchers report contextually based findings about their tool (e.g., navigation, functionality, efficiency, and ease of learning). They also discuss how design and development research served as a “conceptual guide to not only maintain a systematic approach to the development process but also to broaden the perspective of the system’s instructional implications to a holistic approach that addressed system, user, and development process as a whole” (Hung et al., 2010, p. 78).

Design of a Computer Support System for Multimedia Curriculum Development

A study by Wang, Nieveen, and van den Akker (2007) focuses on the design of an electronic performance support system (EPSS) to help teacher-designers in China develop scenarios for multimedia instruction. The main purpose of the study was “to produce a practical computer support system for multimedia curriculum development by following an evolutionary prototyping approach” (p. 277). The researchers wanted to create an EPSS that was valid, practical and effective. They created four prototypes of the tool and collected data from experts and end users who completed questionnaires and participated in focus groups. Summative evaluation of the tool was also conducted. During this phase of the study, teacher-designers were observed using the tool to create scenarios for multimedia instruction; they also provided suggestions for improving it. Results indicate that participants found the tool to be usable and practical. An unintended outcome was that the tool helped “teacher-designers become acquainted with a systematic approach to multimedia instructional design” (p. 289).

The Wang et al. study is particularly noteworthy because it includes summative evaluation. This type of evaluation is often not included in DDR and is infrequently used in practice.

Research on Models

The second type of design and development research pertains to studies of the development, validation and use of models. These studies focus on the design and development models and processes themselves, rather than their demonstration. While it is possible to conduct model research in conjunction with the development of a product or program, many model studies concentrate on previously developed instruction, and consequently are not project-specific. Model research may address the validity or effectiveness of an ­existing or newly constructed development model, process or technique. In addition, these studies often seek to identify and describe the conditions that facilitate successful design and development. Since model research studies are oriented toward a broad analysis of design and development processes, they tend to be more generalizable than product and tool studies.

Model research tends to address three major related phases—model development, model validation, and model use. Here we review very recent representative model research in each category.

Recent Model Development Research

Model development research may result in new, enhanced, or updated models that guide the ID process or a part of the process. Such research has produced a rapid prototyping ID model (Jones & Richey, 2000), components of a model of ID competencies (Vallachia, Marker, & Taylor, 2010), and a Web-based knowledge management system model that provides for its continuing development (Plass & Salisbusry, 2002). Model research encompasses a wide range of settings and participants and it employs a variety of research methodologies (see Richey & Klein, 2007).

We examine two recent model development studies that address very different design problems using research methods that are totally different from each other. However, both contribute to the advancement of design and development models.

A Model for the Design of Visual Information

Message design is a specialized task of those who select and develop instructional materials, and it is an area informed by a broad knowledge base. Consequently, Voss (2008) conducted an extensive literature review resulting in the development of a model to guide designers as they work with one particular type of message—two-dimensional visual images that will transfer information to the learner/viewer. This study explores the research literature of message design, cognitive psychology, neurology, and information theory to identify those principles that govern visual communication, mental imagery, and visual memory. The literature and the resulting model suggest that visuals have their own set of rules that are based upon the nature of perception rather than the view of communication as being controlled by language.

While reviewing the literature is an important early step in conducting any research, a literature review is not typically used as a research methodology in the IDT field. However, large-scale reviews such as Voss’s provide an opportunity to build an empirically based model that covers many variables. For instance, the Voss model addresses pre-attentive and attentive brain functions, the mental processes of selecting visual images, the varying functions of pictures, symbols and signs, as well as a range of distortions that can occur during message transmission.

Reviews of large bodies of research and theory are likely to cover many settings, be they instructional, transfer, or design and development. Therefore, this technique facilitates the identification of factors that are not context-specific or learner-specific. This is the case with the Voss model.

Identifying the Components of a Transfer Model

Like many researchers and practitioners interested in improving workplace performance, Hillsman and Kuptritz (2010) seek to identify empirically based predictors of transfer. Specifically, they focus on elements in the physical work environment. Their work is an example of a study that can be viewed as content-specific research, but then ID researchers can also interpret their findings in terms of design and development. More specifically, the research can be viewed as model development since it provides justification for including an entire class of variables into a design model directed towards transfer of training.

The Hillsman and Kuptritz study was a multi-methods project (both qualitative and quantitative) that collected data from 50 supervisors who had participated in 4 hours of interpersonal communication training and then applied their new skills working with their employees on the job for 6 months. The research involved conducting 6 hours of field observations, surveys, and structured personal interviews. In addition, there was an archival review of work records.

The results show that workplace design did “contribute to transfer outcomes. Supportive as well as unsupportive workplace design features were elicited as most often facilitating or impeding transfer” (Hillsman & Kuptritz, 2010, p. 23). While this research clearly has implications for workplace design, it also adds to the body of literature that seeks to model those factors that impact transfer of training. Thus, it also informs training designers of those aspects of context that are critical to the success of their interventions. These researchers do not fully develop a design-related model, but instead they identify the building blocks required to construct a comprehensive model.

Recent Model Validation Research

While the ID literature is rife with models of the design process, far less attention is paid to the validation of these models. Such validation is an empirical process that demonstrates the effectiveness of a model’s use in a real-world setting (i.e., external validation) or provides support for the various components of a model (i.e., internal validation) (Richey, 2005). In some validation research, experienced design practitioners are used as subject matter experts to authenticate a design model or specific design phases. For example, Cowell (2001) interviews current designers to substantiate the regular use of needs assessment techniques (even though other terms are often used for the process). In other research, learner data confirms the model. (See Roszkowski & Soven, 2010, for their research which validates an updated Kirkpatrick ­evaluation model.) We summarize two other recent DDR studies which highlight the characteristics of model validation research.

Updating and Validating Gilbert’s Behavioral Engineering Model

Thomas Gilbert’s Behavioral Engineering Model (BEM) has profoundly influenced designers who work in employee training environments, and it has been credited with the origination of cause analysis. (See Gilbert, 1978, for a full discussion of this model.) Crossman’s (2010) research examines BEM’s relevance in the contemporary workplace.

The participants in Crossman’s study are 600 fire fighters and the specific area of interest is safety culture, the motivation to follow safety rules. The fire fighters completed a survey whose items reflected the environmental elements of the BEM—information (i.e., communication), resources, and incentives. Data were analyzed using correlations and path analyses. Crossman found that the combined effects of the three variable categories did influence safety motivation. Furthermore, she found that incentives directly impacted safety motivation while absorbing the indirect effects of communication of information and resources. The environmental facet of Gilbert’s long standing model was validated in this setting.

Crossman’s study exemplifies an internal model validation asking whether the parts of the model are justified. It is an empirical study that relates to an actual work environment. It is statistically sound and based in both theory and practice. However, there are other ways to approach model validation.

Testing the Impact of the Multiple Intelligence Design Model

Tracey (2009) uses very different tactics to validate her ID model which blends multiple intelligence (MI) theory with traditional instructional systems design. The study has two parts—a designer usability test and an examination of product impact. As such, it provides both internal and external validation of the MI Design Model.

Designer usability was tested by randomly assigning two Masters-trained designers to a 2    hours team building workshop project using the MI Design Model; two similar designers were assigned to the same project using the Dick and Carey ISD Model. (See Dick et al., 2009 for a full discussion of this model.) Work conditions were the same for each design team. Following completion of the workshop materials, the MI Model designers filled out a model usability survey detailing their reactions to the model. Product impact, on the other hand, was tested by using the two design team’s products. Five sessions with eight to ten learners each were conducted using the MI-oriented workshop, and another five similar sessions were conducted using the ISD product. Posttest and attitude-toward-training data were collected. While both groups felt confident in their new skills, ­participants who were trained with the MI materials scored slightly (but significantly) higher on the posttest and learning seemed to be stimulated by the use of the MI instructional strategies.

Tracey’s research supports the use of the MI Design model. Like other design and development studies, it exemplifies comprehensive model validation techniques performed under real-world design conditions.

Recent Model Use Research

While it is not unusual for model validation research, such as Tracey’s (2009) study, to address usability issues, there is another genre of design and development research that emphasizes how models are used. Many of these studies focus on the conditions that impact model use; these show the interplay between varying design and development contexts and model effectiveness. For example, Roytek (2000) conducted a comprehensive case study which focuses on two design projects using rapid prototyping procedures; this study is designed to determine which contextual factors, strategies, and events facilitate or impede project success. Other research focuses on the designers themselves to understand exactly how the design and development process is actually used. Visscher-Voerman and Gustafson (2004) conducted interviews with designers working in diverse settings and reviewed related project materials to determine the procedures designers used and their rationales for these approaches. Recent model usability research continues in a similar vein with much of the current work concentrating on the role of technology.

The Rapid Implementation of e-Learning

Many academic programs are faced with the prospect of changing quickly from face-to-face delivery of their courses to on-line learning. Coetzee and Smart (2012) present a case study describing the process of developing a module and placing it on learning management system (LMS). In doing so, they demonstrate the merger of two models—the traditional ADDIE (analysis, design, development, implementation, evaluation) design model and The Technology Process model used in technology development projects. The situation in this study was realistic in that the university instructor was working essentially alone with only one other person giving advice. Resources were limited. There was little lead time and thus the modules were used as developed. The course thus moved from a face-to-face delivery to a blended delivery. Subsequent units were modified based upon student feedback. This case study demonstrates how two recognized models can be modified to meet the demands of a given instructional situation. The models can be scaled up or scaled down to meet the varying needs of a particular intervention.

The Coetzee and Smart study is particularly useful because model use is not examined in a technology-rich environment. This demonstration takes place in university located in an underdeveloped country. Nonetheless, the two models are successfully adapted to the conditions present in their specific context.

Teacher Technology Integration

There are many models directed towards classroom teachers as they integrate technology into their lessons. The existing research, however, provides little data supporting teacher use of either classic ID principles or the consistent use of technology in their classes. Hart (2008) uses a “think aloud” protocol to study how middle school language arts and social studies teachers actually integrate technology into their lesson plans While these teachers were not applying specifically designated design or technology integration models, Hart explores which model components are used by identifying the design decisions made and the rationale for these decisions. Eight teachers (four of whom had graduate education in instructional design) completed a background survey, then a technology design task using the think-aloud techniques, and a post-design interview. Hart found that in general these teachers demonstrated a reliance on mental planning rather than use of design principles. Technology was not incorporated into all teachers’ lessons in a meaningful way (even though that was the assigned task). Moreover, when it was used, the technology did not tend to support student higher level thinking. Contextual factors (e.g., accessibility) had the most impact on technology use.

Model researchers typically look forward to positive results that confirm their model’s utility. Hart’s (2008) research, on the other hand, produces less satisfying results. The data, however, provide an empirical basis for changing models to accommodate the real-world conditions.

Summary of Key Characteristics

We have described 11 studies published since 2007 which are representative of the most recent design and development research. These studies encompass the various types of both product and tool research and model research. They were conducted in a variety of work and geographical settings and address diverse problems currently being faced by the field. The researchers also use a wide assortment of approaches and methodologies to study design and development problems. Table 12.1 below summarizes this recent research.

Table 12.1 An overview of recent representative design and development research
Full size table

Source of Design and Development Problems

Design and development research typically stems from problems encountered in the workplace (Richey & Klein, 2007). Of the 11 studies reviewed in this chapter, seven of them are directly rooted in real-life problems. For example, this research was used to answer questions such as: What facilitates transfer of training to the job? How can we help teachers to take on the role of designers? How can we get vital information to individuals and families?

In keeping with the increased use of technology in education and training, it is not unusual for design and development studies to have a technology focus. Over half of the studies we highlighted had a technology emphasis, and five of the seven studies with a workplace focus also examined problems related to technology. These studies concentrated on online learning, Web site design, technology integration, electronic performance support systems, and gaming. All of these topics reflect emerging technologies as well as current trends in the IDT field.

Over half of this body of recent research also reflects theoretical problems and issues. While some of these studies (such as Voss’s, 2008 exploration of factors affecting the ­perception of visual messages) emanate only from an interest in theory, others (such as Francom et al.’s, 2009 study of online course design using Merrill’s First Principles) combine a theoretical orientation with practical concerns.

Most DDR addresses problems which have multiple sources. In our sample of 11 recent studies only the model research with a theoretical focus seemed to have a more singular focus. This conclusion, however, may only be a peculiarity of the particular sample of studies we reviewed.

Research Settings and Participants

Design and development research problems (like ID itself) are typically contained in a specific context which includes distinct participants. ID is now used extensively in business and industrial settings, healthcare organizations, community and government agencies, as well as schools and universities. The 11 studies described in this chapter reflect this diversity for the most part. Four of the five product and tool studies are situated in educational settings—two at the P-12 level and two in higher education. On the other hand, half of the model research reviewed pertains to employee training.

All but one of the recent design and development studies reviewed in this chapter were conducted in a setting that included adults as participants, although in some cases the participants were adult learners rather than instructional designers alone. For example, product and tool research was done in the context of a learning community that included the parents of children with disabilities and county extension agents (Cifuentes et al., 2010). Model research was conducted with managers in a training setting (Hillsman & Kuptritz, 2010) and with fire fighters employed by a local government (Crossman, 2010).

Even recent design and development research conducted in P-12 school and higher education settings focuses primarily on adults. Our review identified three studies that concentrated on school teachers. The product and tool studies by Hung et al. (2010) and Wang et al. (2007) examined the design and use performance support tools for teachers while the model use study by Hart (2008) focused on how teachers integrate technology into their lesson plans. Furthermore, 3 of the 11 studies we reviewed in this chapter were conducted in a higher education setting (Coetzee & Smart, 2012; Francom et al., 2009; Sahrir, 2012). In all three cases, participants included faculty who were responsible for designing instruction for their students.

Research Methodology

The majority of design and development studies use multi-method approaches typically blending both qualitative and quantitative methods (Richey & Klein, 2007). This may be a reflection of the complexities of most projects and the multiple sources of the problems address in such research. However, qualitative methods were dominant. Nine of the 11 studies reviewed in this chapter employed qualitative techniques. We believe that this is a typical phenomenon. The qualitative methods, however, vary widely. They include the use of case studies, participant interviews, focus groups, field observations, activity logs, archival reviews, and think-aloud techniques.

Many studies also employ quantitative methods and may at times use experimental designs. Not surprisingly, evaluation phases of design and development research often rely upon assessment measures. Probably the most common quantitative method involved the use of surveys and questionnaires. For example, Crossman (2010) used survey data collected from fire fighters to validate the Gilbert model and Hung et al. (2010) surveyed classroom teachers to identify the requirements of their performance support system. Standard statistical techniques such as correlations and path analyses were then employed.

Finally, there is a critical methodological issue somewhat unique to design and development research. In many of these studies, the researcher also serves as the designer/developer. This situation is a common and often unavoidable by-product of the practical constraints of studying real-life design projects. These conditions occur in all of the recent product and tool studies summarized in this chapter and in one of the model studies (i.e., Coetzee & Smart, 2012). In these cases, data validity can be an issue, but when special attention is given to instrument design, data collection and triangulating multiple sources of data, the concerns have been addressed. The position of the designer/researcher is comparable to the role of participant observer in qualitative research, and similar data collection tactics are employed.

The Role of Evaluation

Evaluation is a major part of the design and development process and correspondingly plays a role in DDR although it is far more prominent in product and tool research than in model research. Since designers who follow a systems approach typically evaluate the intervention during ­development, researchers who study the design and development of a product or tool often collect similar evaluation data to determine its impact on learning. As expected, the comprehensive research projects such as those conducted by Cifuentes et al. (2010), Hung et al. (2010) and Francom et al. (2009) include formative evaluation tasks in their studies. However, all of the product and tool studies and one of the model studies we summarized in this chapter included some form of formative evaluation. Sahrir (2012) placed a major emphasis on this process when he empirically tested how the various phases of formative evaluation can be used by university instructors. Typically these data include learner assessments, but it often includes designer reactions as well.

Of special consideration is the study by Wang et al. (2007) which also includes a summative evaluation to investigate the impact of a performance support tool on teachers who develop curriculum. The inclusion of both formative and summative evaluation data is an encouraging trend in the IDT literature and we hope it continues.

Researchers who study design models are less likely to concentrate on evaluation data unless they are studying evaluation models. However, some researchers such as Coetzee and Smart (2012) may include formative evaluation in their studies of model use. Additionally, others develop and implement an intervention to test the efficacy of the model. In this process learner assessment data is often used.

Conclusions

In the past, instructional design strategies were supported primarily by research on the learning process. While that is still a valuable source of information, ID is now substantiated by a much broader array of research. One trend in the field is the use of design and development research. It establishes practical and theoretically sound solutions to the many problems faced in the IDT field. While this type of research is not yet commonplace, it is growing. The studies reviewed in this chapter reflect this phenomenon. Design and development research is being conducted in many parts of the world. It is being applied to many new topics and areas of concern. These researchers are providing the field not only with innovative examples of how such studies are conducted, but with new knowledge about how to design and develop interventions which address critical problems in education, training, and organizational improvement.