Keywords

1 Case Study Motivation

The goal of this research is to investigate the evolution in the requirements document of novice designers. In studying how requirements evolve from the start to the completion of a project, engineering educators can identify the types of changes that occur throughout the design project. This knowledge can then be used for developing tools and methods to support the novice engineers for both engineering design process and project management aspects.

In order to investigate the evolution of requirements, a case study is conducted with senior design students at Clemson University. Pattern matching is often used to study the analyzed data in case study research [1, 2] while counter-patterns are used to improve the qualitative objectivity [3]. Case study research has been used in the past to study novice engineers through capstone design projects at Clemson University [4–7].

The anticipated pattern for this study is that the requirements document of novice designers (students) will evolve in multiple ways; that there is no dominant mechanism for requirements change within a capstone project. This is based on the assumption that the students get feedback from the advisory committee throughout the project, thereby necessitating changes to the requirements. Moreover, these changes essentially illustrate the growth in problem understanding as the problem and solution co-evolve. Alternately, the counter pattern is that the requirements document will not change. This is based on the assumption that although the students receive weekly feedback, it is not specifically focused on the requirements document.

Capstone design class at Clemson University is a three credit, 15 week long class that falls at the end of the undergraduate curriculum and essentially serves as an exit exam for the mechanical engineering students. Student teams are created based on a balancing of expertise and work experience. Four teams of students are assigned to the same industrial sponsored project to work in parallel. The sponsored projects are funded to support prototyping, while serving as an incentive for the customer to be responsive to the team. The teams meet weekly with an advisory committee, consisting of graduate and faculty advisors. These meetings are design reviews where the teams get feedback on the project, but without direct management by the advisors. This course has been thus structured for roughly three decades, with many research studies derived from this course (e.g., [8]).

The case study to investigate the evolution of requirements in the requirements document of novice designers was conducted on one of the capstone projects from Spring 2011. The project was sponsored by Parker Hannifin and the problem given was:

Design and build a system to automatically splice seals.

For this project four teams of four students were given a presentation explaining the problem and sponsor requirements. Each team worked independently to further define and elicit requirements while developing solutions. The project is representative of a typical capstone design project at Clemson University, considering factors such as the complexity of the problem, the formation of the teams, the composition of the advisory committee, and the execution of the process.

As the goal of this study is to investigate the evolution of the requirements, the data collection entailed collecting weekly requirement documents from all four teams. In order to facilitate the data collection, a requirements update sheet was created as shown in Table 1.

Table 1 Snap shot of requirements update sheet
Full size table

This sheet captures details such as requirement, whether it was a constraint or criteria, the source of the requirement, justification for having the requirement, the target value, verification method, and any updates. All four teams were asked to update the requirements and submit it every week. This requirements documentation tool is similar to that which is taught in the preceding course and is common in design textbooks [9].

However, submitting the weekly documents was not a course requirement, thus most teams did not submit the requirements every week. The frequency of submission varied between the teams and, thus, the comparisons presented in this paper are made between initial weeks and final weeks. As with best practice in case study research, the context of study was intentionally not altered. It may be noted that initial week may refer to either week one, two or three. While the self-reporting of the requirements documents were not sufficient to ensure weekly change evolution, the study presented here is sufficient to provide preliminary evidence of the types of changes that occur within capstone projects. To this point, no other studies have been found to have explored this aspect of requirements in engineering design education.

2 Requirements Tracing Protocol

Three evolution aspects are investigated: (1) the number requirement additions, deletions, modifications, and no changes, (2) the changes in the requirement elements (system, necessity, behavior, object and condition), and (3) the types of change per the taxonomy of [10]. Protocols were established to study each of these change types. It may be noted that since this is a preliminary study investigating change types in requirements of novice designers, the goal here is to identify whether or not the requirements documents of novice designers change. Therefore, the robustness of these protocols is not tested for this preliminary study. In order to study this evolution, a requirement tracking sheet was created for each team and each requirement was traced as it evolved.

Figure 1 illustrates a snap-shot of requirements tracking sheet for Team A. The first row represents the requirement code and includes two elements, the requirement number and the state (I = initial weeks; F = final week). The mapping of the requirements from the initial document to the final document was done manually based on semantic content of the requirement. Some requirements were deleted and new requirements added. Thus the change type for these requirements was identified as ‘addition’ or ‘deletion’. For example, in Fig. 1, there was no 6(I) requirement to correspond to the 6(F) requirement as this was only found in the final document. This requirement is designated as “addition” for change type. Similarly, if a requirement in the initial requirement document is not found in the final, then the change type is “deletion”. If there was a change to the requirement, then the aspects of change are recorded. These change types are based on the work of [10].

Fig. 1
figure 1

Example requirements tracking sheet for team A

Full size image

The next step is to identify specific changes in requirement in terms of the elemental changes. The requirements are parsed into the elements: system (subject), necessity (modal), behavior/characteristic (main verb), and object and condition (complement or adjunct) [7, 11]. The elements that are different between the requirements are highlighted in Fig. 1. For instance, consider requirement 7 for Team A:

  • 7(I): Cool the spliced ring.

  • 7(F): The design must cool the spliced ring in less than or equal to 3 min.

In 7(F), the subject “design”, modal “must”, and adjuncts “in less than or equal to 3 min” are added to 7(I). These changes are highlighted (system, modality, adjunct). Identifying the requirement elements is based on the protocol of [11]. After identifying the change in the requirement elements, the next step is to identify the change types based on the taxonomy of change types established in [10]. The definitions and examples of each change type provided in the taxonomy are used to identify the change type for each requirement pair. Table 2 provides explanation of change types identified by examining the requirements for this study based on the types proposed by [12–14].

Table 2 Description of change types, adapted from [10]
Full size table

A requirement pair can have multiple change types, as seen in the changes for requirement 7(I)–7(F) with changes in the introduction of a new system, the definition of an importance for the requirement, and a change in the specificity of the requirement (Fig. 1). The change types for all requirement pairs for all teams are identified in this manner and a summary table created by counting the number of each change type for each team.

3 Findings from Requirements Tracing

3.1 Number of Additions, Deletions, Change and No Change

The first evolution type investigated in this case study is the high level modification of the requirement document. This includes changes such as addition of, deletion of, change to a requirement (Fig. 2). It may be noted here that, this is a preliminary study to investigate the change types. While only four teams were investigated, the goal was to identify whether or not any change types can be identified in the requirements documents of novice designers. Once, these patterns are identified, extensive studies with multiple teams and different projects will be conducted to further support the findings. However, these four teams are representative of the other senior design teams in terms of the complexity of project, execution of the design process and final deliverables. So the findings can be extended and the recommendations are valid for other senior design teams.

Fig. 2
figure 2

Types of change for project teams

Full size image

From Fig. 2, it is clear that each team had different types of changes to the requirement document, with the most common change being a detail change to an individual requirement. Team D had eight requirements in initial week and 24 requirements in final week resulting to the most number of additions. Team C, on the other hand, had 20 requirements in initial week and 14 requirements in final week resulting to the most number of deletions. Of the four teams, only team B has requirements with “no change”. These findings suggest that requirement document of novice designers’ change significantly from initial weeks to final weeks. Further, these changes are not just limited to addition of new requirements as the understanding of the problem grows but also include deletions and modifications of existing requirements.

3.2 Change in Natural Language Requirement Element

The second evolution type investigated in this case study is tracing the change in natural language requirement elements [10]. This accounts for changes in the system, necessity, behavior/characteristics, object or condition of a requirement. Figure 3 illustrates the overview of the changes in the natural language requirements elements.

Fig. 3
figure 3

Changes in requirement elements

Full size image

From Fig. 3, it can be observed that all four teams had multiple changes in the natural language requirement elements. Most number of changes was observed in the ‘system’ element. The least number of changes were found in the ‘object’ element of the requirement sentences.

Syntactically, ‘system’ is represented by the subject within a sentence. Thus, the system changes also represent the change in the subject of the requirement sentence. There were several types of system changes observed across the four teams such as introduction of new system, splitting, with drawl of a system and consistency.

The element of ‘necessity’ within a requirement statement is syntactically represented by a modal and it shows the importance of the requirement [10]. Necessity element of requirement changed for all teams except team D. Investigating the requirements tracking sheet for all four teams, it was found that for most teams, the change in ‘necessity’ stemmed from addition of a modal such as ‘must’ or ‘should’ in the final week requirements document.

Next, the element of ‘behavior’ within a requirement represents the function of the system or component being designed and is syntactically represented by a verb [10]. Most changes in the ‘behavior’ stemmed from either change in the function of the system (change type—application—for example—must “remove” excess adhesive and must “minimize” excess adhesive) or inconsistency in the vocabulary used to describe the function of the system (change type-consistency—for example—must “provide” economic advantage and must “present” economic advantage). Apart from these two, the behavior changes also resulted from merging of two requirements, splitting of a requirement into multiple requirements and change in the scope or measurability of requirement.

The next element is ‘object’ and it represents what the system is affecting [10]. The changes in the ‘object’ of a requirement stemmed from change in the vocabulary within requirements. So for instance, one of the requirement for team D changed from “the system must count completed spliced parts” to “The final design solution must count spliced O-rings”. Here the object changed from “spliced parts” to “spliced O-rings”, which is essentially inconsistency in the vocabulary as the completed spliced parts are the O-rings.

After object, the next natural language element of a requirement is ‘condition’. The changes in the ‘condition’ stemmed from change in the vocabulary or change in the specific details of the requirement. For example, condition for requirement-2 for team B changed from “ranging from 2 inch to 2 feet in ring diameter” in initial week to “ranging from 2 inch to 24 inch in ring diameter” in final week. Here the unit changed from 2 feet to 24 in. Other type of condition changes resulted from change in measurability, application, change in associated user, change of scope or splitting of a requirement.

Thus, this section describes the changes in the natural language elements of a requirement. Each of these elements is critical within a requirement and any changes must be documented with the justification for the change. Next, Sect. 3.3 describes the changes in the requirements document as per the change taxonomy.

3.3 Change Type as Per Change Taxonomy

The next evolution type investigated within the requirement documents was to trace the ‘type’ of changes as per the change taxonomy established in [10]. It may be noted that while the taxonomy captured all the change types occurring in student requirement document, not all the change types described in the taxonomy were found in the requirement documents. Example of some of these change types include replacement of system, updating, incorrect raw data interpretation and correcting among others. Figure 4 illustrates the summary of change types for all four teams.

Fig. 4
figure 4

Change type occurrences

Full size image

It can be observed from Fig. 4 that all teams had multiple types of changes in the requirements, with the introduction of a new system being the most frequent change type. This change was observed in teams A and C as a result of addition of the subject in the requirements for final week. Other change types that are more frequently observed across all four teams include consistency, importance and specificity. These changes stemmed from change in vocabulary or units, addition or deletion of a modal and change in the specific details of requirement respectively.

The change types that are less frequent include application, measurability/testing and withdrawal of system, while the change types associated user, splitting, scope change and merging only occurred once. Further, the changes that are less frequent were mostly accompanied by other change types that are more frequent. So for instance, splitting and merging occurred once, but in each case they were accompanied by a new system.

Some of these changes can be more critical than others and if not properly document can lead to information loss. For instance the change types such as splitting or merging involve either separating a compound requirement or uniting two requirements into one. This may not always lead to major change in information as it is essentially re-writing requirements in different form. For example the requirements “(the system) Must handle organic and silicone adhesives” and “(the system must) apply adhesive to ends of extrusion” were merged into one requirement “The design must have the capability to apply either organic or silicone adhesive to an extrusion end” which is essentially conveying the same information.

On the other hand the change types such as introduction of new system, importance, specificity and consistency which can alter the purpose of a requirement can prove to be more critical making the documentation of these changes more important. For example for team C, the requirement in initial document changed from “Apply controlled amount of organic or silicone adhesive to extrusion ends” to “The system must apply 1 drop per square inch of organic or silicone adhesive to extrusion ends” in the final document. Here the team added specific detail about the “controlled amount” and thus this change is specificity change. Adding information about how much adhesive enhanced the purpose of this requirement making this change more critical compared to just merging or splitting a requirement. It is interesting to note that the changes which are more critical are also more frequently observed in requirement documents of the teams.

4 Conclusion and Recommendations

It was observed that requirements documents have multiple additions, deletions, and changes from initial weeks to final week, though the novice designers rarely states the justifications for the changes. It is interesting to note that other documents such as weekly summary, presentations, mid-term and final reports that the students are required to submit as a part of project deliverable do not mandate providing updates on requirements. None of these documents require the students to provide justification for the changes in requirements document. If there is no justification for these changes, valuable information could be lost. Further, the novice designers are not able to trace if a requirement was accidently deleted or added if they do have the documented justification for additions and deletions of requirements. Next, investigating the evolution in the requirement elements, it is shown that each type of element can change throughout the project, yet again the students rarely documented the justification for the change. A survey of design textbooks was conducted to investigate the tools pertaining to requirements [9] and none of the textbooks describe tools for managing the requirement changes. With the lack of tools or methods to track requirement changes, the novice designer cannot track down if a change was accidently introduced in a requirement. Finally, the evolution in the requirements document in terms of changes as per change taxonomy was investigated. Again, if the students are not taught to identify these change types, the criticalities associated with the change types and appropriate tools or methods to document and track these changes; it can lead to potential loss of valuable information and jeopardize the successful completion of the projects.

Several recommendations are made based on the study presented above (Table 3). These recommendations may serve as guidelines for faculty helping novice designers manage changes in the requirements documents.

Table 3 Recommendations for faculty
Full size table