1 Introduction

The role of creativity in software development is undeniable. Some authors have investigated creativity’s importance to the entire software development process (e.g., [2]). Others have focused their attention on its importance to requirements engineering [3–6]. However, the potential of techniques to foster creativity in requirements engineering are still under-appreciated [7] and under-investigated. The most popular creativity fostering technique used for requirements elicitation is brainstorming [8],Footnote 1 a classical technique which dates back to 1935 Footnote 2 [9]. More recently, some have applied role-playing-based scenarios in an attempt to bring more creativity to requirements elicitation [6] and to Joint Application Development (JAD) [10]. Still others have set up workshops that integrate creativity provocation with use-case and system-context modeling [7, 11]. A common characteristic of many of these techniques is that each of them tries to address the problem of identifying and enhancing the viewpoints of all the stakeholders, albeit in a different way.

In this paper, we propose an innovative creativity fostering technique, called EPMcreate, based on a model of the pragmatics of communication, the Elementary Pragmatic Model (EPM), developed more than 30 years ago by Alberto Silvestri, a physicist and computer scientist, and Piero De Giacomo, a psychiatrist Footnote 3 [12]. The model was developed to describe the relational patterns [13] of interacting subjects and to predict their evolution [12]. Since the end of the 1970s, the model has been applied successfully to solve relationship problems in families [14–16] and in company teams [17]. It has been applied also in economic psychology [17], to model electronic communication in the area of Computer Supported Cooperative Work (CSCW) [18], and to model e-learning web-site users [19]. More recently, the EPM has been used to develop a technique for fostering mental flexibility [20], which is useful for promoting creativity.

The EPMcreate (EPM Creative Requirements Engineering TEchnique) technique consists of 16 steps. In each step, a problem is analyzed according to one of the elementary behaviors identified by the EPM. To apply EPM in requirements elicitation, we regard each step of the process as suggesting a way for an analyst to look at the problem from a different combination of stakeholders’ viewpoints. EPMcreate has the analyst look at the problem in 16 entirely different ways, each corresponding to one of the 16 basic Boolean functions of two variables [21, 22]. To evaluate the feasibility and applicability of EPMcreate, we conducted experiments on two projects with very different characteristics. In each experiment, we compared the performances of two analysis teams, one of which used EPMcreate and the other of which used brainstorming. The results of the experiments were analyzed both

  1. 1.

    quantitatively, counting the numbers of ideas and requirements generated, and

  2. 2.

    qualitatively, comparing the feasibility and the novelty of the ideas and requirements. In each experiment, EPMcreate proved to be more effective than brainstorming.

This paper is structured as follows: Sect. 2 describes the twofold and paradoxical role of creativity in requirements engineering. Section 3 describes the role of communication and interaction, and hence of pragmatics, in creativity. Section 4 introduces the EPMcreate technique, starting from a short description of the EPM on which it is based. Section 5 describes when and how to carry out EPMcreate in practice. Section 6 describes the experiments we designed and carried out to investigate the feasibility and effectiveness of EPMcreate. The experiments were conducted on the developments of a web-based tool for distance learning and of a software system (SWS) for public administration. Section 7 discusses lessons learned from the conduct of the experiments, the subjects’ comments, and some additional analyses of the data. Section 8 discusses issues not tested by the experiment, which should therefore be covered in future experiments. Finally, Section 9 describes some proposals for future research.

2 The role of creativity in requirements engineering

Many authors have investigated the subject of creativity [23–27]. Some have investigated creativity experimentally [28] and in the context of the development of information systems [29–33]. Still others have applied creativity fostering to requirements elicitation for air-traffic control [7, 11]. Finally, there is a recognition that in spite of all the tools [34] and technology available to help requirements elicitation, the requirement engineer’s problem-solving skills are key to defining good system requirements and that creativity and imagination is an essential component in successful problem solving [8, 35, 36].

There are many definitions of creativity. However, fortunately there appears to be convergence among researchers on a concept of creativity related to problem solving (e.g., [37, 38]), encompassing also problem finding and solution thinking. In particular, creativity is understood as the generation of innovative, unexpected solutions to complex, non-trivial problems, or to ill-formed, wicked problems.

A wicked problem [39] is one whose very definition is part of the problem itself. In other words, a wicked problem is not structured enough to allow a straightforward rational, scientific, or engineering approach. A wicked problem, in all probability, will require a healthy dose of creativity in its solution. Also, any problem that is interdisciplinary, that has multiple stakeholders, that has a highly dynamic context, that has uncertainty in its parameters, and that has many possible tradeoffs calls for creativity in its solution. We summarize all these kinds of problems requiring creativity for solution as tough problems.

Besides those that recognize creativity as a fundamental factor in software development [2, 28, 32, 41–43], there are those that see creativity as a threat, as something to monitor and control carefully in order to prevent it from compromising their projects [44]. That is, new requirement ideas discovered after implementation has started can be very expensive to accommodate and are often not appreciated by the implementers. Indeed, at REFSQ’04, Neil Maiden was heard to exclaim, “Creativity is a dangerous thing!”

These two positions capture the nature of creativity itself. For any project dealing with a tough problem, creativity needs to be encouraged, but in a structured and controlled way. Techniques that foster creativity can be used for this purpose.

It is worth noting that requirements engineering [8] has all the characteristics of tough problems [40] that call for creativity, in particular when groups of stakeholders are involved. Moreover, creativity gives the hope of attacking what some consider to be the most difficult problem in requirements engineering, that of discovering missing requirements [45–47]. The source of many a disaster is a real-world situation that the involved SWS was not prepared to handle because no one had thought of the situation, what Don Gause has called “Nature’s Last Laugh”. Creativity gives the hope that more of these otherwise overlooked situations will be identified before the SWS is built.

Some requirements for a creativity fostering technique for requirements elicitation are that it be

  1. 1.

    domain independent, applicable to SWSs of any kind,

  2. 2.

    easy to learn, by analysts with varying amounts of experience, including none, and

  3. 3.

    applicable by individuals as well as by groups, so that it can be used in small organizations or in situations in which a requirements engineer has to work alone.

Brainstorming meets all these requirements, even the third [48]. The rest of this paper is the beginning of studies aimed at proving that also EPMcreate meets these requirements.

3 The role of communication and interaction in creativity for requirements engineering

It is not surprising that the role of communication and interaction is central in many of the creativity fostering techniques. Footnote 4 Many techniques prescribe group sessions and these groups sessions imply that the participants are communicating and interacting.

It is worth noting that the complexity of these techniques varies considerably. At the simple explanation end, the Creative Pause Technique (CPT) [49] advises simply to “Interrupt your routine in order to pay deliberate attention to some particular issue.” At the other end are the highly structured methods, such as Creative Problem Solving (CPS) [50]. While the CPT is simple to describe, it may not be easy to carry out successfully. Thus, the more structured techniques may be preferred. Footnote 5

For each of a majority of techniques, a process is given. A technique can be characterized by the way its steps differ from those of classical problem solving:

  1. 1.

    analysis of the problem,

  2. 2.

    design of a solution,

  3. 3.

    implementation of the chosen solution, and

  4. 4.

    evaluation of the chosen solution.

For example, the step of CPS are:

  1. 1.

    objective finding,

  2. 2.

    fact finding,

  3. 3.

    problem finding,

  4. 4.

    idea finding,

  5. 5.

    solution finding, and

  6. 6.

    acceptance finding.

The main difference between CPS and classical problem solving is that in Steps 1 and 6, CPS seeks shared objectives and acceptances.

Moreover, each of many of these techniques combines the process with a toolkit of creativity fostering techniques to be used in one or more of the steps. At the theoretical level, EPMcreate satisfies the need to foster creativity in a controlled way. The method is more structured than any of a majority of the other techniques, and its instructions force the analyst to focus on different stakeholders’ viewpoints. In this respect, EPMcreate differs from, for example, the well-known Six Thinking Hats technique [26], which suggests different types of thinking corresponding to six thinking roles for the analyst, associated with hats of six different colors. Two examples are the (1) White Hat thinking role, which is neutral and objective and concerned with facts and figures, and the (2) Red Hat role, which adopts an emotional view.

4 EPMcreate

EPMcreate is a creativity fostering technique that is based on the EPM. EPMcreate itself is the result of a bit of creativity on the part of the first author arising from

  • her familiarity with EPM as model of both human and computer-mediated relations [18, 51],

  • her familiarity with De Giacomo’s description of an EPM-based creativity process [20, 21], which as stated is not usable by a requirements engineer, and

  • her own experiences in requirements elicitation.

It seemed to her that the logical structure of a process based on EPM would be more suitable than brainstorming and other techniques, particularly for mathematically, technically, computationally inclined analysts. In thinking about EPMcreate, she has formulated a conceptual framework, based on the mental flexibility scheme exploited in De Giacomo’s explanation of his EPM-based creativity process [21] that explains why EPMcreate works. However, space and audience considerations preclude discussing the framework here. Besides which, the proof that the idea works is in the actual doing. Consequently, the second author agreed to conduct experiments to validate that the idea works. The focus of this paper is on explaining EPMcreate to requirements engineers who wish to use it and on examining the results of these experiments.

The explanation of EPMcreate in Section 4.2 below is directed at the reader’s being able to apply it. For this explanation of EPMcreate, it is useful to explain the EPM first, in Section 4.1.

4.1 The EPM

The EPM was devised as an analytical tool to help analyze relational patterns of interaction among groups of people. The basic model works on pairs of people. The tool may be applied to groups of more than two by considering all possible pairings within the group. However, there remains the problems of combining the findings of the pairs in a meaningful way and, if the number of people involved is large, the combinatorial explosion of pairs to consider.

As a first approximation, if we focus on the pragmatic contents of messages, a single interaction between two persons, P1 and P2 consists of four steps.

  1. 1.

    Person P1 proposes an action A1.

  2. 2.

    Person P2 proposes an action A2.

  3. 3.

    Person P1 proposes a possibly different revised action A1′ based on A1, A2, and his perceived relationship with P2.

  4. 4.

    Person P2 proposes a possibly different revised action A2′ based on A1, A2, and his perceived relationship with P1.

Possible examples for Step 3 include P1 insisting on his own proposal, P1 changing his proposal to match P2’s, P1 changing his proposal to one contrary to P2’s, etc. In Step 4, P2’s response could be equally varied.

If the proposal were a binary one, that is, to do or not to do something, then each of Steps 3 and 4 can be regarded as modeled by a truth table on two variables.

There are 16 possible patterns of responses, corresponding to the 16 Boolean functions on two variables. These functions are named fi for 0 ≤ i ≤ 15; fi names the function for which i is the decimal numeral corresponding to the four-digit binary numeral, R 1 R 2 R 3 R 4, obtained from the response column of the table for the function. Some representative function names and their corresponding tables, abbreviating “Pn’s Proposal” as “Pn” and “P1’s or P2’s Response” as “R”, are:

Each function can be considered as representing one pattern of response. The analytical use of EPM ascribes to each function a mode of behavior based on factors such as the responder’s self-esteem and his perception of his relationship with the other in an interaction. For example, assuming that the responder is P1:

f0:

represents a person who always says “No” to any proposal regardless of his and the other’s original proposal (e.g., a totally pessimistic or depressed person).

f3:

represents a person who always insists on his own proposal, regardless of the other’s original proposal (e.g., a very confident or very stubborn person).

f5:

represents a person who always says the same as what the other says, regardless of his own original proposal (e.g., a so-called yes man).

f10:

represents a person who always says the opposite of what the other says, regardless of his own original proposal (e.g., a so-called contrarian).

f15:

represents a person who always says “Yes” to any proposal regardless of his and the other’s original proposal (e.g., a totally optimistic or enthusiastic person).

In any situation requiring therapeutic intervention, e.g., a dysfunctional married couple, family, work group, etc., the therapist as analyst, observes the interaction of each pair of participants and tries to identify a pattern in the form of one of f0, ..., f15 for each person’s responses in that pair. Once a pattern is identified, the therapist tentatively ascribes the identified basis as dictating the person’s interaction in the pair. While the EPM is excellent in terms of exhaustively covering all possible interactions, classifying an interaction on the model’s basis is fuzzy at best. First, any given interaction between a pair may not be typical of that pair’s interaction. Second, it is the rare relationship that consistently follows only one interaction pattern. Therefore, the classification must be made empirically on the basis of many observations, and the conclusions would be subject to the same statistical uncertainties that any other experiment’s conclusions are. Moreover, unless the history of observations for a pair includes an instance of every truth table line, the conclusion can be based on insufficient data. Finally, there may be possibly temporary reasons other than those that have been catalogued for a particular history of interaction. For example, on board a Star Fleet vessel, a very logical Vulcan science officer may adopt what appears to be a yes-man response towards the proposals of a very competent chief engineer, not because the Vulcan is a yes man, but because the Vulcan knows that the chief engineer just happens to be right all the time.

EPM has been employed, as mentioned in Sect. 1, in family therapies in which the therapist searches for a family’s problems. In a similar way, EPMcreate could be viewed as a requirements elicitation [4] technique. For more information about therapeutic, analytic uses of EPM, please consult some of the references cited in the second paragraph of Sect. 1.

4.2 The EPMcreate technique

EPMcreate, as a creativity provoking technique, is based on sort of an inverse use of the EPM. Rather than trying to deduce an individual’s behavioral pattern by observation of his interactions with others, EPMcreate uses the EPM as a means to help a requirements elicitor (RE) to generate all possible reactions to two stakeholders’ positions. Thus, each of the 16 Boolean functions represents one method of combining two stakeholders’ viewpoints to generate yet another viewpoint from which creative ideas can flow. If there are more than two stakeholders, the technique is applied several times, for different pairs of stakeholders, up to \(\left( {\begin{array}{*{20}c} n \\ 2 \\ \end{array} } \right)\) times for n stakeholders. Because of the potential combinatorial explosion of the number of pairs, clearly not all such pairs can be considered. The RE must choose pairs that yield the maximum new information. Perhaps she should try to choose pairs of contentious stakeholders, i.e., stakeholders whose views are likely to yield conflicting requirements, to insure the discovery of as many ideas as possible. In any case, each stakeholder group should appear in at least one of the interviewed pairs to ensure that everyone has some voice. Moreover, as with the EPM, when there are findings for more than one pair to consider, there is the basic problem of how to combine the findings (See first paragraph, Sect. 4.1). Fortunately, for EPMcreate, the findings are lists of ideas, and in any case, the items of these lists have to be examined carefully later for quality. Simply concatenating lists does not make this examination more difficult.

The description of technique refers to the Venn diagram in Fig. 1. In this diagram, the two ellipses represent two different stakeholders’ viewpoints. Thus, for example, the intersection region represents the stakeholders’ shared viewpoints.

Fig. 1
figure 1

Venn diagram of two stakeholders’ viewpoints

Full size image

A creative session supported by EPMcreate starts with the identification of two stakeholders, or classes thereof, that are relevant for the SWS to be developed. For example, for an e-learning application, two possible stakeholders are students and lecturers. For an information system that supports a company’s B2B activities, two possible stakeholders are employees of the selling and of the buying companies.

Then a multi-step process is started in which the RE has to assume different attitudes towards the stakeholders’ viewpoints. The steps actually follow the Boolean functions f0, ..., f15 in order of increasing index.

In the first step, corresponding to f0, the RE must blank her mind. This step resembles the CPT and allows the RE to increase her concentration for the subsequent steps.

In the second step, the first operative step, f1 suggests that the RE focus on elements common to the stakeholders’ viewpoints in order to seek solutions of the problem that can be shared by the identified stakeholders.

Then, f2 asks the RE to focus on elements that are in only the first stakeholder’s viewpoint, in order to seek solutions favorable to only the first stakeholder, to the exclusion of the second.

Step f3 allows the RE to concentrate on all elements that are in the first stakeholder’s viewpoint, in order to seek solutions that are favorable to the first stakeholder, even those that are favorable to the second.

Step f4 is symmetric to f2, having the RE switch the focal stakeholder.

Steps f5, ..., f15 are defined in a similar way, using the function name to chose the way to combine the stakeholders’ viewpoints in order to look for new solutions.

An important change in the process occurs in Step f8, which represents a kind of watershed. From this step on, the RE can draw new ideas from elements that are outside the viewpoints of the identified stakeholders. In particular, for Step f8, the RE has to empty her mind, as for Step f0, and then follow up on the first solution that comes to her mind.

The last step, f15, is a catch-all step; any solution whatsoever can be accepted independently of either stakeholder’s viewpoint.

5 EPMcreate in practice

EPMcreate can be applied in any situation in which ideas need to be generated, e.g., at any time one might apply brainstorming. EPMcreate is by no means the only technique for identifying requirements; it is but one of many that can be used. Its place is as one technique in any RE’s bag of techniques to apply when ideas for requirements are needed.

When an RE determines that EPMcreate is an appropriate technique during requirements engineering for a system under consideration, she first chooses two kinds of stakeholders, SH1 and SH2, usually users of the system, as those whose viewpoints will be used to drive the application of EPMcreate. She may ask the system’s analysts for assistance in this choice. She then convenes a group of these analysts.

The RE tells all convened,

“Today, we are going to generate requirement ideas in 16 mini brainstorming sessions. In each mini session, all of you will pretend to think from the viewpoint of two stakeholders, SH1 and SH2, and for each viewpoint.

  • In Step 0, you will blank out your minds.

  • In Step 1, you will try to come up with ideas for problem solutions that are needed by both SH1 and SH2.

  • In Step 2, you will try to come up with ideas for problem solutions that are needed by SH1 but not by SH2.

  • In Step 3, you will try to come up with ideas for problem solutions that are needed by SH1 without concern as to whether they are needed by SH2.

  • ...

  • In Step 15, you will try to come up with ideas for problem solutions without concern as to whether they are needed by either SH1 or SH2.”

In the event that the RE believes that more than two stakeholders’ viewpoints should be considered, she will convene more EPMcreate sessions, one for each pair of stakeholder viewpoints she believes to be useful. Her experience tells her how to identify stakeholders and stakeholder pairings that will yield the most new ideas for the fewest pairs. Note that not every pairing of considered stakeholders needs to be the subject of an EPMcreate session.

The list below shows examples of one or two ideas generated for each step from one of the experiments described in Sect. 6. This list is meant to give the reader a concrete feeling of the kinds of ideas generated in each step. The system under consideration is to manage online courses at a university. SH1 is the student registered for online courses, and SH2 is the lecturer teaching online courses. It may be necessary to have read Sect. 6, particularly Subsects. 6.1 and 6.2 for a full understanding of the ideas. However, the domain should be clear enough to most readers. Do note that a given participant may not interpret the instructions for a step completely correctly. Nevertheless, the ideas generated during a misinterpreted step may still be useful.”

Each item header shows a step name and shows the logical expression named by the step. The logical expression uses S for the student’s viewpoint and L for the lecturer’s viewpoint. In these logical expressions, “⊕” means exclusive or, i.e., one or the other, but not both.

f1— (S ∧ L) :

  • Put the link to personal data about students and lecturers in a more visible place.

  • The menu item now labeled “check students” should be better labeled as “visualize registered students”.

f2— \((S \wedge \neg L):\)

  • Automatically remove a bulletin board notice after its expiration date.

  • In a course’s bulletin board, give also information about temporary variations in the lecturer’s office hours.

f3— S :

  • Inside the information about the lecturer, add a link to the lecturer’s publications.

  • Videoconferencing does not function properly (so that there is no lecture today!).

f4— \((\neg S \wedge L):\)

  • Propagate all changes to any item in the Corsi Online Web site to all copies of that item in the University Web site (propagation is done manually now).

  • Allow only one path for a lecturer to add material to a course’s Web page (there are currently several path).

f5— L :

  • Allow a lecturer to see a day divided into time slots.

  • Allow a lecturer to see the teaching schedules of all other lecturers.

f6— \((S \oplus L):\)

  • Add to a student’s section of the Web site links to the online courses in which the student is registered.

  • Allow a student to see administrative office information needed to check tax payments and examinations that are not registered yet.

f7— \((S \vee L):\)

  • Allow both students and lecturers to follow links to the minutes of the meetings of the faculty board.

  • Request timely reminders of appointments and time-critical news that is noted in a user’s diary.

f8— \((\neg S \wedge \neg L):\)

  • The system should send an SMS message to a student’s mobile phone if the student’s section of the Web site has been updated.

f9—(S ≡ L) or \((\neg S \wedge \neg L) \vee (S \wedge L):\)

  • Add a photograph of each lecturer or student to his or her section of the Web site.

  • Report the percentages of students who passed and who failed a course.

f10— \((\neg L):\)

  • Request printing of a student’s timetable according to the courses chosen by the student.

  • Allow a student to vote for student representatives online.

f11— \((S \vee \neg L):\)

  • Publish the students’ evaluations of the lecturers and courses.

  • Make all lectures available by videoconferencing.

f12— \((\neg S):\)

  • Classify each lecturer by whether or not his or her courses’ materials are at the Web site.

f13— \((\neg S \vee L):\)

  • Display lecturers’ office hours online.

  • Display a map of the city.

f14— \((\neg S \vee \neg L)\;\hbox{or}\;\neg (S \wedge L):\)

  • Allow a student to have peer-to-peer access to his or her lecturer’s computer, to avoid overloading the network.

f15— true or anything:

  • Give students and lecturers more powerful computers for use in studying and teaching the courses.

6 Experiments

6.1 Design and realization of the experiments

The goal of the experiments was to demonstrate the effectiveness of EPMcreate as a technique for fostering creativity for requirements elicitation, in order to convince practitioners to use the technique when they need to elicit requirements. This effectiveness would be measured by the number of creative ideas generated and the amount of resources needed to generate them, with the hope to maximize the number and to minimize the amount. An experiment involving only EPMcreate would give a number and an amount. However, no one would know what that number and amount means. Thus, it is necessary to compare the effectiveness of EPMcreate with that of another technique for fostering creativity for requirements elicitation.

We chose to compare EPMcreate with brainstorming, rather than another more structured technique, because brainstorming is the most well-known and widely-used creativity fostering technique and is probably the only such technique used in most software developing companies [52]. A manager in one of these companies would probably be unconvinced of the effectiveness of EPMcreate if we had compared it with a technique other than brainstorming, with which he or she was familiar. The same manager would be even more unconvinced if we determined effectiveness by a non-comparison method. Probably for this very reason, it has become traditional to compare new creativity fostering techniques with brainstorming [52]. Finally, brainstorming already meets our requirements, listed in Sect. 2, for a creativity fostering technique for requirements elicitation.

Thus, to demonstrate the feasibility of the EPMcreate technique in requirements elicitation, we conducted experiments on each of two projects with very different characteristics. In each experiment, we compared the numbers of ideas generated by two analysis teams, one of which used EPMcreate and the other of which used brainstorming. Such experiments yield specific numbers of creative ideas generated. We were careful to have all competing pairs of teams use the same number of people for the same amount of time so that the resources expended by competing teams are the same and thus, only the number of creative ideas generated needs to be compared in order to compare the effectiveness of the two techniques.

The experiments reported herein were exploratory, first to see if the EPM could even be applied to foster creativity and second to learn what the steps of EPMcreate actually are. Because of the small number of subjects involved, we knew that it would be highly unlikely that the results would be statistically significant. The best we hoped for were data that argued that EPMcreate is promising and is worth additional study. Our goal was a clear understanding of how a requirements engineer might use EPMcreate to provoke creativity for requirements elicitation.

6.1.1 The two experiments

The first experiment, called “Corsi Online”, concerned a web-based system for distant learning. The second, called “Civilia”, concerned a SWS for public administration.

The subjects for the first experiment were chosen from among students. From 40 students asked by e-mail to participate, 6 women and 2 men accepted. Their ages ranged from 23 through 28. Seven were Economics students and 1 was a Sociology student. The subjects for the second experiment were chosen from among professional analysts working for the company that developed Civilia. The Civilia project manager selected 8 analysts from the company, 7 men and 1 woman, to be the subject REs.

The student subjects in the first experiment were novices, compared to the professional subjects of the second experiment. However, they had taken some computer science courses, in particular, on information systems and database design, and had the skills that companies, including that doing the Civilia system, demanded for junior analysts. The subjects for the second experiment were, as indicated, professional analysts.

Two groups, each with 4 people, were created for each experiment. For the first experiment, the 8 subjects were assigned randomly to the 2 groups. For the second experiment, the subjects’ experiences in system analysis and knowledge of the SWS used for the experiment were taken into account in order to create homogeneous groups with equivalent spreads of experiences and knowledge. The results were that 1 group had an average age of 36 and 10 years of analysis experience and the other group had an average age of 38 and 8 years of analysis experience. Each of three members of each group had about 1 year’s experience in the Civilia project and 1 member of each group had no experience with Civilia at all.

In each experiment, after assigning the subjects to the 2 groups, we drew lots to determine which group was going to use EPMcreate.

In order to be able to interpret more correctly the results of the experiments, the subjects were given two tests. One test was a creativity test adapted from one developed by Williams [53, 54]. The second was the SISCI test [14], a test of a person’s relational patterns as described by the EPM model. The goal of these tests was to detect the presence of significant differences in personal creativity and relational patterns among the members of the groups, differences that could affect and explain the outcome of the requirements elicitation sessions. In particular, if the effectiveness of the groups did prove to be different, we hoped that the creativity test results could be used to exclude that the observed differences were caused by differences in the subjects’ natural creativity.

The results of the creativity test confirmed that the second experiment’s groups were balanced also in personal creativity. However, for the first experiment, the members of the group that used brainstorming showed more personal creativity than the members of the group that used EPMcreate [22]. This particular imbalance did not affect the final results, since the group that used EPMcreate proved to be more creative, in spite of its members’ lower personal creativity; in essence, the final result is strengthened.

For the Corsi Online experiment, the creativity and SISCI tests were administered on a day preceding that of the experiment. In each experiment, the two groups met separately, except for explanations valid for both groups. The steps, in order, of each experiment and their approximate times were:

  1. 1.

    20 min on each of the creativity and SISCI tests,

  2. 2.

    20 min hearing a brief introduction to the experiment and the technique to be used, each group separately,

  3. 3.

    15 min, in only the Corsi Online experiment, hearing a description of domain of the problem to be solved in the requirements elicitation session, and

  4. 4.

    120 min, maximum, for the requirements elicitation session using the group’s creativity fostering technique, each group separately.

In the brief introduction to the experiment, we emphasized the main goal of the requirements elicitation session, to produce as many requirements as possible. We recommended not evaluating any idea, in particular not to inhibit anyone from participating. We requested that all ideas be entered immediately into a logging file. We explained that each team had not more than 2 h for requirements, but that it could finish its session earlier if its members agreed that no other aspect or idea could be discussed. As explained later, each group, except the Civilia brainstorming group, used the full 2 h for requirements elicitation. The Civilia brainstorming group finished 20 min early, claiming that they could not generate any new ideas.

Each group in both experiments was given a short training session about the technique it was to use. However, because nearly all the participants said that they could not give us much time, the training was reduced to the minimum possible. This minimal training could be considered a threat. On the other hand, it simulates what happens in the real world, in which processes are often done with minimal training.

In both experiments, none of the subjects had ever participated in an EPMcreate session (for no other reason than that the technique was too new). In the Corsi Online experiment, none of the subjects had ever participated in a brainstorming session. Thus, for the Corsi Online experiment, the comparison would be between equally unfamiliar techniques. In the Civilia experiment, each of the subjects had participated in brainstorming in the past.

6.1.2 Corsi online

The development of the Corsi Online system of the first experiment started in 2001 in the Faculty of Economics at the University of Trento. The system was a Web application to help manage on-line courses for the faculty, providing services mainly to students and lecturers participating in the courses. The system provides functions for use also by the course managers.

Since the owners and requirements analysts of the Corsi Online system are in the same institution as the experimenter, the original requirements analysts were available for consultation during the experiment. These analysts helped the experimenter evaluate the ideas generated in the experiment.

The Web-based system has three main sections:

  • Lecturer area.

  • Student area.

  • Management area.

In the experiment, we asked the subject REs to use the lecturers and students as the stakeholders on which to apply the EPMcreate steps. Thus, the viewpoints considered by the REs were those of lecturers and students involved in the on-line courses.

To facilitate the requirements elicitation, we gave each group a list of 26 existing functions, 8 in only the lecturer area, 8 in only the student area, and 10 in both areas. Some examples of these functionalities are: “adding material to syllabus for today’s lesson”, in only the lecturer area; “selecting a course”, in only the student area; and “videoconferencing”, in both the lecturer and student areas.

6.1.3 Civilia

The second experiment was conducted at the software company that had developed Civilia to support community services for citizens. Civilia is a modular system, made of a number of different subsystems, one for each kind of target user. In the experiment, we asked the subject REs to focus on the people and territory subsystems as the sources of stakeholders and their viewpoints. The people subsystem deals mainly with registration of people, and the territory subsystem deals mainly with registration of land.

For this second experiment, the Civilia project manager helped the experimenter evaluate the ideas generated in the experiment.

6.2 Results of the experiments

The lists of ideas and requirements produced by the four groups, working with EPMcreate and brainstorming, were analyzed both

  1. 1.

    quantitatively, counting the numbers of ideas and requirements generated, and

  2. 2.

    qualitatively, comparing the feasibility and the novelty of the ideas and requirements generated.

The comparison of the results of the two subject RE teams of each experiment confirmed the higher effectiveness of the EPMcreate technique. In particular, the EPMcreate groups produced 71 ideas in the Corsi Online experiment and 98 ideas in the Civilia experiment, while, the brainstorming groups produced only 22 ideas in the Corsi Online experiment and only 43 ideas in the Civilia experiment.

For the Corsi Online experiment, we classified each requirement idea as to whether it was a functional requirement or a nonfunctional requirement. For the brainstorming group, 15 of its 22 ideas were functional requirements while 8 were nonfunctional requirements. For the EPMcreate group, 58 of its 71 ideas were functional requirements while 13 were nonfunctional requirements. Thus for each group, a clear majority of its ideas were functional requirements. For the Civilia experiment, almost all the requirements generated were functional. These results are interesting because some believe that nonfunctional requirements do not require creativity to be discovered [11].

To measure the quality of the ideas, we classified each idea into one of four categories:

  • New and realizable.

  • New but not realizable.

  • Already known but not realizable.

  • Already known and realizable.

For the Corsi Online experiment, an example of a nonrealizable idea is “a competition with awards to encourage the use of the tool”, and an example of a realizable idea is “an online schedule of the instructors’ office hours”. For the Civilia experiment, an example of a nonrealizable idea is “adding information to permit management of cycle paths”, and an example of a realizable idea is “adding a list of natural disasters in the geographical area”. Basically, each nonrealizable idea has problems with security or with control or maintenance of contents or was considered out of scope.

Note that prior to the experiment, each project had done a requirements process that had found the requirements for the systems that had been and were being implemented. The subjects of the experiments were not told the requirements that had already been identified. Thus, the experiments would test the effectiveness of EPMcreate and brainstorming in finding really new, creative requirements ideas. An idea was to be classified as new, realizable or not, if the idea had not been identified earlier by the projects’ requirements processes. An idea was to be judged as not realizable only if it was too expensive to be implemented in its project’s current budget or if it would be included in another module of its project. Note also that it would be best to have the evaluations of the ideas done by persons very familiar with the requirements of the projects.

The fact that the subjects were finding requirements for systems for which requirements were already known does not create a limitation in the experiment, because the subjects were not told the existing requirements and initially had no more knowledge about the systems under consideration than the average person on the street. However, having the subjects find requirements for systems with known requirements was necessary to allow evaluation of the quality of the generated ideas.

For the Corsi Online experiment, the original project manager and the original analysts assisted in the classification. The EPMcreate group generated about three times as many new ideas, both realizable and not realizable, as did the brainstorming group. Specifically, The EPMcreate group identified 62 new requirements, some for functions considered very innovative by the project manager, while the brainstorming group identified only 19 new requirements. The EPMcreate group outperformed the brainstorming group, even though the members of the brainstorming group scored higher in personal creativity than the members of the EPMcreate group.

For the Civilia experiment, the project manager of the SWSs department of the developing company classified the ideas. In this experiment, the number of new ideas, both realizable and not realizable, generated by the EPMcreate group were about ten times higher than those generated by the brainstorming group. Specifically, the EPMcreate group generated 37 new requirements, and the brainstorming group identified 4 new requirements.

Figure 2 shows the numbers by category, method, and experiment.

Fig. 2
figure 2

Graphs of the numbers of ideas

Full size image

Observe that in the Corsi Online experiment, ranking of the categories was essentially the opposite of that in the Civilia experiment; i.e., the largest category in each group of the Corsi Online experiment was the new-and-realizable category while the largest category in each group of the Civilia experiment was the already-known-and-realizable category. Initially, we thought that this difference in ranking of categories could be explained by the difference in levels of development of the two systems. Civilia is an iteratively prototyped system that has undergone several revisions. Thus, it seemed only natural that the subject REs, who had not been involved in Civilia’s development, would find requirements that had already been discovered by the Civilia development team. This explanation was initially confirmed by the project manager who had assisted in the classification of the ideas. However, after further discussions with this manager, we now believe that the difference arises also and more from the greater degree of self-control of the Civilia subject REs due to their greater experience both in general and with the Civilia system. Even so, the EPMcreate group found 22 new ideas while the brainstorming group found only 4 new ideas, and the Civilia project manager was very satisfied with these new ideas.

As mentioned, each group in the experiment, except the Civilia brainstorming group, used the full 120 min for requirements elicitation according to its assigned technique. The Civilia brainstorming group stopped after 100 min, claiming that they had identified all the requirements that they were going to. We offered them the opportunity to continue for another 20 min, but they declined the offer. If we accept at face value the Civilia brainstorming group’s explanation that they would not find any more requirements in the additional 20 min they had coming to them, we can regard each group as having expended the same resources, i.e., 8 person hours, to achieve its list of requirement ideas. Thus, we are justified in using the number of requirements ideas generated by a group using a technique as the measure of the effectiveness of the technique.

Altogether, these results are extremely encouraging and point to the higher effectiveness of the EPMcreate technique. The results, though remarkable, are not statistically significant; there are too few data points. Besides the small number of data points, another possible threat to validity is the use of students instead of professional analysts. However, as mentioned, these students had had some experience in SWSs analysis. Moreover, comparing the student and professional subjects would give us an opportunity to study the learnability of the technique as a function of professional students. Finally, the main results were the same with both kinds of subjects.

The strongly positive results do say that EPMcreate has merit and do call for additional experimentation aimed at obtaining statistically significant and generalizable results.

Please consult the second author’s degree thesis in Italian [22] or a derived paper also in Italian [55] for more details about the experiments. In particular, the thesis contains a set of experimental materials to assist in replication, including instructions to the subjects, the statistical measures, etc. It has also a discussion of threats. Other positive results mentioned in these other documents include the ease of learning of the technique as well as the high satisfaction level of the subjects and assisting analysts.

For the EPMcreate sessions, we had kept data that allowed us to allocate each idea to the EPMcreate step that generated it. We noticed in both experiments that the odd numbered steps tended to be more productive than the even numbered steps. For each experiment, form a list, each of whose element is a step number paired with the number of ideas generated in the step; then sort this list by increasing numbers of ideas. The sorted lists are shown in Table 1. For the Corsi Online experiment, the 5 most productive steps were odd numbered steps and the 4 least productive steps were even numbered steps. For the Civilia experiment, the effect is not as pronounced. However, among the 9 most productive steps, 5 are odd numbered, and among the 7 least productive steps, 5 are even numbered. The odd numbered steps represent functions that focus on elements shared by the considered viewpoints. In both cases, the least productive of the odd numbered steps is f15, which serves as a catch-all step that gives no particular advice to the participants.

Table 1 Rankings of EPMcreate steps by numbers of ideas
Full size table

These observations suggest that additional experiments should be carried out to try to identify a subset of the 16 steps as an optimized creativity fostering technique to reduce the length of requirements elicitation sessions. Other experiments can be done to investigate the effectiveness of alternative orderings of the steps. For example, an examination of other creativity fostering techniques finds many techniques starting off with a step corresponding to EPMcreate’s Step f8. Perhaps, it would be better to start with Step f8, which forces focusing on elements outside both viewpoints, possibly even on absurd viewpoints.

7 Subjects’ observations and new hypotheses

In the course of talking with the subjects during and after the experiments, we gathered a number of observations about EPMcreate from a user’s perspective to allow us to check whether the technique is satisfying to its users. An often ignored, tacit requirement for a software engineering method is that the method be pleasant and satisfying to its users, that they be happy applying the method.

Please note that these observations are the subjects’ opinions, and they do not necessarily reflect reality. However, the fact that subjects have volunteered these observations is telling.

  • More than one professional subject user of EPMcreate volunteered that EPMcreate was easier to apply than other creativity facilitating techniques they had used in the past.

  • More than one of both kinds of users of EPMcreate volunteered that they felt satisfaction that they were successful in generating new and useful requirements ideas.

  • More than one of the student subject users of EPMcreate volunteered that they felt satisfied with their outputs even though they were tired from their efforts.

  • More than one of the professional subject users of EPMcreate volunteered that they felt relaxed as well as satisfied with their outputs.

  • More than one of the professional subject users of EPMcreate volunteered that, during the process, they started foreseeing the next steps as they began to figure out the EPM basis of EPMcreate, and they felt much satisfaction from this feeling.

  • More than one of the professional subject users of EPMcreate volunteered that they were driven to greater productivity by the large number of new ideas that they were finding with the new technique, with a clear implication that brainstorming was the old technique to which they were comparing EPMcreate.

Based on these observations by the subjects, it is our opinion that EPMcreate appeals to CS-type (used to algorithmic thinking) people more than do other creativity enhancement methods, probably because of EPMcreate’s systematic recipe-book nature. We believe that less psychological expertise is needed to facilitate EPMcreate than is needed to facilitate role-playing techniques such as Six Thinking Hats. We even believe that for EPMcreate, no human facilitator is needed. A well trained CS-type person can begin to internalize and apply EPMcreate on his or her own from reading a written description or under the direction of an automated EPMcreate assistant. Future experiments should try to test these opinions.

8 Issues not considered in the experiments

It is clear in retrospect that there were some particulars of the experiment that could have affected the results. These include the scale of the problems considered, the domains of the problems considered, and the viewpoints chosen for the EPMcreate sessions.

Scale

The systems involved in the experiment were of a medium to large scale and were certainly real-life systems. However, the scaling issue in this method is not the size of the system per se, but rather the number of different types of users. The number of user type pairs grows as the square of the number of user types, and it is the number of user type pairs that determines the number of EPMcreate sessions that need to be run. Since it will not be feasible to run more than a few of these sessions, the user type pairs will have to be selected carefully. The results could depend on how well this selection is done. In any case, the problems of dealing with large numbers of users was not considered in the two experiments and will have to be considered in future experiments.

Domain

The two applications in the experiments were chosen not for perceived applicability of the method, but because their organizations were willing to participate in the experiments. The two systems were quite different; their domains were different; their code sizes were different; their implementation languages were different; their hardware platforms were different; their integration with other systems were different. About the only thing they had in common was their being Web based. From our admittedly limited experience, the EPMcreate technique does not seem to be restricted to any particular domain and does not seem to exclude any domains. However, future experiments will need to vary the domain so as to be able to show whether there is any impact of the domain on the effectiveness of the technique.

Choice of viewpoints

In each experiment, two particular user types were chosen to be those whose viewpoints would drive the EPMcreate session of the experiment. The user types were chosen carefully by the experimenter for their representativeness and expected high yield of relevant ideas. Would other user types have worked as well? Would more user types have added enough new ideas to warrant the costs of additional EPMcreate sessions? These questions need to be answered in future experiments.

9 Conclusion

This paper has described two experiments in which EPMcreate technique was used for the first time to try to foster creativity in requirements engineering. The results of the experiments strongly suggest the effectiveness of the technique. The history of the experiments suggest also that EPMcreate is easy to manage; and the subjects, particularly the senior analysts in the second experiment, found the process very intuitive and intriguing. This last conclusion is based on the second experiment subjects’ direct feedback after their session and on the fact that the brainstorming group did not use all the time available for the elicitation, having run out of steam and ideas. As a result, it seems that the structured process suggested by the EPMcreate technique plays a positive role. It can be argued that EPMcreate tries to guarantee the presence of ever-new stimuli for the search for new ideas. In particular, each of the Steps f8 through f15 asks the RE to focus on elements that are outside the problem and the application, for outside-of-the-box thinking.

Given that these were the first applications of the EPMcreate technique to requirements elicitation, it is necessary to repeat these experiments with more groups of subjects with the aim of obtaining statistically significant results. The history of the experiments brings to light a number of questions, such as those mentioned at the end of Sect. 7, that can be the subject of future experiments. It is necessary to compare EPMcreate also with creativity fostering techniques that are more structured than classical brainstorming, such as the Six Thinking Hats family of techniques [26] or P.a.p.s.a. [56]. Such studies should also investigate whether EPMcreate is easier to learn than the other techniques.

Future research could also investigate the application of EPMcreate in the JAD approach [10]. A properly run JAD session has the presence of all stakeholders needed to provide the viewpoints that are needed to carry out EPMcreate.

At the cognitive psychology level, it would be worth investigating the relation between an individual’s creativity and his relational patterns as measured by the SISCI test. It would be necessary to administer a more articulated creativity test and then to analyze the results for the individual’s creativity and his relational patterns. We have done some preliminary investigation using the data obtained from the subjects of the experiments described in this paper, who were tested with an automatic version of the SISCI. The results of these preliminary comparisons are consistent with the literature about creative personalities [22].

As mentioned, our main goal was to demonstrate the feasibility of EPMcreate. Even though we do not yet have generalizable data, the data are very encouraging. At least in the two specific requirements elicitation problems to which EPMcreate and brainstorming were applied, EPMcreate proved to be very effective in finding requirements that had not been known to the managers of the projects involved. Moreover, EPMcreate proved to be greatly more effective than the familiar brainstorming. Our conversations with these managers indicate that they consider EPMcreate to be very feasible and that they will use it in the future.

Recall the requirements for a creativity fostering technique for requirements engineering. EPMcreate appears to satisfy the first two, i.e., that it be domain independent and that it be easy to learn. EPMcreate was applied to two different domains. EPMcreate proved to be greatly more effective than brainstorming even with only less than 20 min training.

We have not tested EPMcreate for applicability by an individual, even though we see no problem doing so. EPM is applicable by an individual therapist. To test individual applicability properly, we would have to compare individual applications of EPMcreate with individual applications of brainstorming and with team applications of EPMcreate to position individual application of EPMcreate with other individual methods and with group applications of EPMcreate. It is interesting in this respect that Aybüke Aurum and Errol Martin [48] have observed that individual brainstorming may be more effective than group brainstorming, calling to question the synergistic effects of groups [57].