Keywords

1 Introduction

Walls et al. [1] developed a design theory and motivated their readers by writing that “.. design is central to such varied fields as engineering, architecture, and art. It is also clearly an important topic within the Information Systems (IS) discipline.” We, IS researchers must build and evaluate new information systems or information technology (IT) artifacts because researchers in other disciplines cannot do them. Lee [2] emphasized this as follows “.. research in the information systems field examines more than just the technological system, or just the social system, or even the two side by side; in addition, it investigates the phenomena that emerge when the two interact”.

Iivari [3] advocates Design Science Research (DSR) by paying attention to conceptual mess and stating that “The idea of DSR in IS is still in its formative stage. As new members join the DSR research community, each of them may bring in his or her own interpretation of what DSR is. While the plurality of ideas is definitely beneficial, especially at this early stage, it is also good for people to understand what they are talking about.” To this end, it is important to know a variety of ways how the design research process has been conceptualized.

We recognize some essential phases in the design research process: its beginning (problem definition), actual research project, its end (various outcomes, e.g., knowledge and instantiations) and its evaluation. Concerning the first phase of the design research process we like to ask: Question A: What is a research problem of design research and how it can be stated?

In analysis of outcomes of design research we found many kinds of knowledge and instantiations and hence we like to ask: Question B: Which kind of knowledge does design research produce? Question C: Which kind of innovations design research will produce?

Design research seems to differ from traditional research that it emphasizes utility of innovation, not truth of theory describing a certain phenomenon. But are the traditional evaluation principles still valid for design research or not? To this end we ask: Question D: How can we specify a goodness of design research?

2 Methodology

In order to study the questions above we like to apply phenomenography [4] as a research method. According to Tesch [26] research interest in phenomenography is in the discovery of regularities discerning of patterns in conceptualization. Phenomeno-graphy [4] is intended to describe, analyze and understand conceptions: the aim is to describe the qualitative different ways in which various aspects of reality are seen and conceptualized and to search for logical relations between the categories of description arrived at. A conception in phenomenographic terms is a very starting point from which a person views some aspect of reality. The aim with the phenomenographic research is to show the qualitative variation in which persons understand a certain phenomenon. Researchers using phenomenography are thus not so interested in why persons think as they do. The idea of phenomenography is to describe the variation of how persons view something, not to explain the reasons for the variation. Kaapu [27] performed a literature review of phenomenography in IS and 5 specific phenomenographic studies in her doctoral dissertation.

To give a more concrete view on activities of the phenomenographic study we take an example [28]. A Järvinen asks interviewees: Please, define a conception of health by using your own words. The researcher will receive free-form answers that she classifies into a few classes. The researcher must not criticize the truthfulness of answers. She first differentiates the definitions described with a set of properties of health from the definitions where some relationships of health with other concepts were used (cf. Bunge’s class and relation concepts [29]). She then tries to find similar definitions, she groups such ones together and finally forms five categories as in Grounded Theory [30]. Thereafter she puts the categories into order from the simple to more multi-faceted ones: (1) no illness, organism in good condition, (2) a physical and psychical balance or steady state, (3) interaction between mind and organism in good condition, (4) interaction between mind, organism and environment is without dysfunctions and (5) an individual is functioning, active and target-oriented. - From the short example above we can find the following phases of phenomenographic study: (I) Collect raw data, (II) formulate a definition per each person, (III) formulate categories from similar definitions (conceptual cluster analysis) and (IV) try to find a certain structure of categories (if such one exists).

Phenomenographic studies usually use interview or questionnaire to collect raw data. Instead of that we are using scientific articles as raw data in phenomenographic study (cf. literature review). A single author is a ‘person’ in phenomenographic sense and a group of authors of a particular article, too. In order to collect our raw data, some articles, we partially used our set of articles read in our doctoral seminar during years 1991–2015, on an average 30 new articles per year. (The titles of articles read during years 1991–2009 are in [31] and more recent ones under title IS Reviews 201n on web pages of our school). A major part of the articles read is method articles, because every doctoral candidate needs to select an appropriate research method. Readings were collected into a method book [32]. Concerning this study we selected highly ranked journal articles that have played a leading role in a progress of design (science) research (Phase I). In order to give a critical reader a chance to check our results we use direct citations from the articles to describe the author(s’) definitions (Phase II).

In addition of phenomenograhic analysis of scientific articles we shall bring our own comments and frameworks to amend a discussion when it is possible. We shall structure our paper according to the four questions above and in each question we present the material found in the chronological order of publications.

3 Question A: What Is a Research Problem of Design Research and How It Can Be Described?

In this section we shall first try to find how various authors defined a building problem in design research. Our purpose is to pick up the problem formulation as it is or try to interpret and write it into a form of question, if it does not be written as such a form. The formulations will be collected into Table A. We shall comment the results thereafter and present how the authors have described a context of design problem. We shall also propose a framework of a problem-solving situation.

Nunamaker et al. [5] guide a systems development: “Formulate first a concept (i.e., a framework) that is found useful in organization of ideas and suggesting actions.” Walls et al. [1] “take a position that design commences immediately after problem identification and terminates when the customer signs off on the system.” According to March and Smith [6] “.. we build an artifact to perform a specific task.” (They mean IT artifact.) Iivari [7] likes to “.. emphasize more the nature of Information Systems as an applied, engineering-like discipline that develops various ‘meta-artifacts’ to support the development of IS artifacts.” According to Hevner et al. [8] “The objective of research in information systems is to acquire knowledge and understanding that enable the development and implementation of technology-based solutions to heretofore unsolved and important business problems. … Design science approaches this goal through the construction of innovative artifacts aimed at changing the phenomena that occur.” Van Aken [9] is a researcher who in Management Science emphasizes the role of design research, and he differentiates Organization Theory research from Management Theory research. The latter “.. uses the perspective of a player and uses in prevision intervention-outcome logic: what intervention should a player use in the given context to realize the desired outcome.” We have collected formulations into Table 1.

Table 1. Qualitative variations how the building problem could be formulated
Full size table

In our phenomenographic analysis we pay attention on various differences (object, problem owner, idea behind of solution) in the descriptions of research problem. From Table A we can find that the object in building project is not always the same, but Nunamaker et al. [5] and Walls et al. [1] are building an information system, March and Smith [6] and Hevner et al. [8] an IT artifact, Iivari [7] a meta-artifact and van Aken [9] as a management scientist does not tell the object of intervention. The publication sequence of the articles selected shows that the perspective on research object is in the course of time narrowing from the whole information system to its part, IT artifact.

Walls et al. [1] clearly said that the customer is a problem owner, other researchers directly or indirectly said that the researcher is a problem owner. In the Walls et al.’s [1] case it may in the worst case lead to the situation where the customer stops the project before any concrete outcomes.

Nunamaker et al. [5] are only ones who inform their idea, concept, behind of solution. We think that other researchers except van Aken [9] will use the most advanced IT in their solutions, although they do not mention it. Van Aken [9] will utilize some relationship A → B of a certain theory in his intervention. In our field Davison et al. [13] used a similar relationship of a particular theory in their action research.

We can conclude that our phenomenographic analysis shows that there are a large variety of problem definitions in design research. It is possible to at least give three different classifications of problem definitions (categories). These classifications can be based on either research object or problem owner or an idea behind of solution. The phenomenographic analysis can be continued by searching combinations of two or three factors (object, owner, concept) but it does not shed much new light on research problem of design research.

Next we shall move outside of phenomeographic analysis and comment on the problem-solving paradigm by considering it graphically and developing a framework of a problem-solving situation in Fig. 1.

Fig. 1.
figure 1

A framework of a problem-solving situation in design research

Full size image

For design research we define: An organization stays now at a problematic state, called an initial state, and this organization likes to transform itself into a desired state by building a particular artifact. The described situation resembles March and Smith’s [6] description: “Models represent situations as problem and solution statements. Methods are often used to translate from one model or representation to another in the course of solving a problem.”

To our mind, transition from the initial state to the desired state is unique and hopefully irreversible and this transition can be described. This description can be considered as a descriptive method (‘we performed a building process in this way’). The Gregor’s [10] taxonomy does not, however, have any type category for descriptive methods; it has type V category for prescriptive methods [11]. But a certain descriptive method can act as a starting point for developing a prescriptive method.

In Fig. 1 there is also a final state by which we like to demonstrate that in design project it will sometimes happen that designers will not exactly achieve the desired state but another state, called a final state. The latter can differ from the desired one because of some extra functionality or because of larger or less utility.

We emphasize that Fig. 1 can also be interpreted in such a way that at least the initial state is unknown and a researcher has some novel ideas to build a new artifact that did not exist beforehand. This can be based on new technology, new social or new information resources. We then have an opportunity problem: Which kind of a new artifact could we build by using either new technical, social or informational resources or their combination? Our idea is not totally novel because Iivari [3] has noted that “.. design science is also about potentiality. A new idea or artifact may provide totally new opportunities to improve practice long before practitioners recognize any problem.”

We recommend that a problematic initial state and a desired state will be described by the tasks to be performed and resources needed in performing the tasks. A researcher can then consider all the resources, technical, social and informational resources and their new opportunities. The change method to be followed shows a road from the initial state to the desired state either by obeying consecutive phases or by stepwise moving from one state to the next state towards to the desired state.

4 Question B: Which Kind of Knowledge Does Design Research Produce?

In this section we shall collect different types of knowledge produced by design research. The main results are prescriptive knowledge but also conceptual and descriptive knowledge [14] as a side results are taken into account. We do not collect different knowledge types into table form because of much repetition created. We shall present our important addition at the end of this section. Design research produces instantiations too but we shall consider them in the next section.

According to Iivari [15], “Conceptual development as a category of constructive research methods refers to the development of various models and frameworks which do not describe any existing reality but rather help to create a new one, and which do not necessarily have any ‘physical’ realization (e.g., IS development methodologies).”

Walls et al. [1] state that design theory has two aspects. “The first aspect of a design theory deals with the design product and consists of meta-requirements, meta-design, kernel theories and testable design product hypotheses. The second aspect of a design theory deals with the design process consisting of design method, kernel theories and testable design process hypotheses. Information system development life cycle (SDLC) is a widely accepted informal information system design theory.”

March and Smith [6] state that “.. design science products are of four types, constructs, models, methods, and instantiations. As in natural science, there is a need for a basic language of concepts (i.e., constructs) with which to characterize phenomena. These can be combined in higher order constructions, often termed models, used to describe tasks, situations, or artifacts. Design scientists also develop methods, ways of performing goal-directed activities.” March and Smith continue that “.. it is important to determine why and how the artifact worked or did not work within its environment. The interaction of the artifact with its environment may lead to theorizing about the internal workings of the artifact itself or about the environment.”

Hargadon and Sutton [16] describe various ways to store knowledge as follows: “It was evident that much of the knowledge of potential solutions resides in the minds of the individual designers as products they have seen or used before. Designers augment their individual memories and written materials by collecting, looking at, and talking about products or parts of products, which act as records of existing technologies. Designers stockpile old products and parts in their offices and hallways or hang them from the ceiling.”

Lee [17] defines “.. the instrumental model of practice as including the following elements. A researcher formulates, tests, and validates a theory that specifies independent variables, dependent variables, and the relationships among them. In doing this, the researcher is careful to make sure that, first, the dependent variables represent the outcomes that the practitioner is interested in achieving and, second, the independent variables represent factors that not only indeed influence the outcomes but also can be manipulated or changed by the practitioner. A practitioner could then apply the theory by manipulating the independent variables in order to achieve the desired levels in the dependent variables.”

Hevner et al. [8] repeat four types of products (constructs, models, methods, and instantiations) presented by [6]. Hevner et al. consider that “.. effective design-science research must provide clear contributions in the areas of design construction knowledge (i.e., foundations, system development methodologies, modeling formalisms, ontologies, problem and solution representations, design algorithms), and/or design evaluation knowledge (i.e., methodologies, new evaluation metrics).”

Van Aken [9] states that “.. a professional will make three designs: an object-design, the design of the intervention or of the artifact; a realization-design, i.e. the plan for the implementation of the intervention or for the actual building of the artifact; and a process-design, i.e. the professional’s own plan for the problem solving cycle, or, put differently, the method to be used to design the solution to the problem”. He continues that “Design-repertoires contain three types of design knowledge, according to the three types of designs discussed above. Within each of the three types of design knowledge, prescriptions are an important category. The prescription can be used as a design exemplar. A design exemplar is a general prescription which has to be translated to the specific problem at hand; in solving that problem, one has to design a specific variant of that design exemplar. The typical research product is the prescription discussed above, or the technological rule that is tested and grounded. According to van Aken [9] both successful and especially the less than successful applications, should be reported.

Iivari [3] uses [1, 17] to identify prescriptive design science knowledge both for design product (for the artifact: idea, concept, style; functionality, behavior; architecture, structure, and for design process: technological rules [12] and technical norms [18]). (In order to achieve A do {act1, act2, …, actn}; If you want A and you believe that you are in a situation B, then you should do X; it is rational for you to do X; it is profitable for you to do X). Iivari [3] also specifically wish “.. to point out that it is not necessary for a kernel to be from some reference discipline external to IS. A kernel theory can be a theory specific to IS and generally that at the conceptual level the outcomes are new concepts and frameworks, at the descriptive level new theories and models.”

In this section the citations above can be used as raw data for phenomenographic analysis and they show a chronologically increasing variety of research outcomes especially at the prescriptive level. There are, however, two large classes based on design product and design process (Walls et al. [1], Iivari [3] and van Aken [9]), and the latter even differentiates results into two classes (successful and less than successful).

In addition of phenomenographic analysis we found that Iivari [3] criticizes when Walls et al. [1] suggest that the information systems development life-cycle is a design theory. Iivari [3] is not aware of any kernel theory on which it is based. When the information systems development life-cycle consists of the following phases: requirements determination, design, construction, implementation, and operation, those phases are assumed to be consecutive. To our mind, also an evolutionary approach, sometimes called a state-transition approach, is possible and often used to improve an existing problematic system.

5 Question C: Which Kind of Innovations Will Design Research Produce?

In the previous section we analyzed different (abstract) knowledge types as outcomes of design research. Here we concentrate on different (concrete) innovations as outcomes of design research. They are considered as important results.

According to Nunamaker et al. [5] “The pivotal role of systems development in a framework of research is the result of the fact that the developed system serves both as a proof-of-concept for the fundamental research and provides an artifact that becomes the focus of expanded and continuing research.”

Iivari [15] states that “.. technical development produces as its outputs ‘physical’ artifacts, the adjective ‘physical’ being interpreted here broadly to include executable software (e.g. CASE environments)”.

According to Walls et al. [1] “A hypothesis that a certain method will result in an artifact which meets its goals can be verified by using that method to build the artifact and testing the artifact to see whether it satisfies its goals. Clearly, then, prototype construction is a major aspect of design theory research.”

March and Smith [6] state that “.. constructs, models and methods can be instantiated in specific products, physical implementations. Progress is achieved when a technology is replaced by more effective one.”

Iivari [7] likes to “.. emphasize more the nature of Information Systems as an applied, engineering-like discipline that develops various ‘meta-artifacts’ to support the development of IS artifacts.”

According to Hevner et al. [8] “The result of design-science research in IS is, by definition, a purposeful IT artifact created to address an important organizational problem. Furthermore, artifacts constructed in design science research are rarely full-grown information systems that are used in practice. The instantiations produced may be in the form of intellectual or software tools aimed at improving the process of information system development. System development methodologies, design tools, and prototype systems (e.g., GDSS, expert systems) are examples of such artifacts.”

Iivari [14] states that in his “.. view the primary interest of Information Systems lies in IT applications. I propose a typology for IT applications which provide an alternative categorization of services to that in [19]. The typology distinguishes seven archetypes of IT applications based on the function/role the application serves.” The roles/functions are to automate, augment, mediate, informate, entertain, artisticize and accompany. In Iivari [3] there is the eighth archetype with role/function to fantasize.

Lee et al. [20] like to re-conceptualize artifact in IS design science “.. from just the ‘IT artifact - to what we are calling the ‘IS artifact’. We ‘unpack’ what has been called the ‘IT artifact’ into a separate ‘information artifact’ and ‘technology artifact’ that, together with a ‘social artifact’, interact to form the ‘IS artifact’. An IS artifact is itself a system, in which the whole (the IS artifact) is greater than the sum of its parts (the IT artifact, the social artifact, and the information artifact), where the constituent parts are not separate, but interactive, as are any subsystems that from which a larger system emerges. Hence, our [Lee et al.] naming of it as an information system artifact or IS artifact”.

In this section too the citations above can be used as raw data for phenomenographic analysis and we find two similar groups as in connection with Question A: March and Smith [6], Hevner et al. [8] and Iivari [3, 7, 14, 15 restricted to IT artifacts or meta-artifacts only. Nunamaker et al. [5], Walls et al. [1] and recently Lee et al. [20] emphasize an information system as a whole.

To our mind, it is interesting that Lee et al. [20] defined an IS artifact consisting of technology, information and social artifacts, and it nicely corresponds to our view that an information system is built of the three types of resources (technical, social and informational).

6 Question D: How Can We Specify a Goodness of Design Research?

In this section we shall first collect various ways to evaluate design research. Thereafter we shall present one correction.

Nunamaker et al. [5] state that “.. system development could be thought as a ‘proof-by-demonstration’. The integrated research efforts can be identified by the stages through which they grow (concept – development – impact). Systems must be developed in order to test and measure the underlying concepts.”

Walls et al. [1] emphasize both a design product and a design process as follows: “The design process is analogous to the scientific method in that a design, like a theory, is a set of hypotheses and ultimately can be proven only by construction of the artifact it describes. If it is to be a good theory, a design theory must subject to empirical refutation. An assertion that possession of a particular set of attributes will enable an artifact to meet its goals can be verified by building and testing the artifact. A hypothesis that a certain method will result in an artifact which meets its goals can be verified by using that method to build the artifact and testing the artifact to see whether it satisfies its goals.”

March and Smith [6] state that “.. it [design science] is technology-oriented. Its products are assessed against criteria of value or utility – does it work? is it improvement? Design science consists of two basic activities, build and evaluate. Evaluation is the process of determining how well an artifact performs. Instantiations demonstrate the feasibility and effectiveness of the models and methods they contain. Evaluation refers to the development of criteria and the assessment of artifact performance against those criteria. Evaluation requires the development of metrics and the measurement of artifacts according to those metrics. Building the first of virtually any set of constructs, model, method, or instantiation is deemed to be research, provided the artifact has utility for an important task. The research contribution lies in the novelty of the artifact and in the persuasiveness of the claims that it is effective. Actual performance evaluation is not required at this stage. The significance of research that builds subsequent constructs, models, methods, and instantiations addressing the same task is judged based on ‘significant improvement’, e.g., more comprehensive, better performance.” For evaluation of constructs, models, methods and instantiations March and Smith [6] present some universal criteria, respectively.

In the connection with discussion about the main research domain of the IS discipline Benbasat and Zmud [21] propose that “.. our focus should be on how to best design IT artifacts and IS systems to increase their compatibility, usefulness, and ease of use or on how to best manage and support IT or IT-enabled business initiatives.”

Hevner et al. [8] propose that “.. prescriptive theories must be evaluated with respect to the utility provided for the class of problems addressed.” According to their Guideline 3, “The utility, quality, and efficacy of a design artifact must be rigorously demonstrated via well-executed evaluation methods.” They continue that “.. IT artifacts can be evaluated in terms of functionality, accuracy, performance, reliability, usability, fit with the organization, and other relevant quality attributes.” They propose five evaluation methods (observational, analytical, experimental, testing and descriptive). They stress on that “.. design-science research holds the potential for three types of research contributions based on the novelty, generality, and significance of the designed artifact.” Hevner et al. [8] also state that “.. design-science research often simplifies a problem by explicitly representing only a subset of the relevant means, ends, and laws. Ends are represented using a utility function.” To our mind [32], the utility function at the problematic initial state can be measured. The same utility function can be used for measuring the utility of the new instantiation at the desired state. We shall receive two values of the utility function before and after an innovation. If the latter is better than the former we have achieved progress.

According to van Aken [9] technological rules must be tested within the context of its intended use, and they must be grounded on scientific knowledge. “The utility of technological rules must be examined to the extent to which they fulfill the five key user-needs of practitioners (descriptive relevance, goal relevance, operational validity, non-obviousness and timeliness)”.

Peffers et al. [22] developed the methodology of six steps: problem identification and motivation, definition of the objectives for a solution, design and development, demonstration, evaluation, and communication. They underline such criteria like the artifact’s functionality with the solution objectives, budgets, results of satisfaction surveys, client feedback, or simulations etc.

According to Weber [23] “A theory has emergent attributes – attributes of the theory as a whole rather than attributes of its parts. Many such attributes exist, and researchers often differ in their views on the significance they ascribe to each of them. Nonetheless, some emergent attributes (importance, novelty, parsimony, level and falsifiability) have widespread acceptance among researchers as being significant when assessing the quality of a theory.”

In this section the citations above can be used as raw data for phenomenographic analysis. We many times find a special emphasis of the novelty the outcomes of design research, and here design research resembles traditional research. But in design studies there are rarely new innovations, for example, Gregor and Hevner [24] did not find any new invention but many improvement innovations. The five typical criteria (productivity, profitability, performance, efficiency and effectiveness) were in many citations used in evaluation of design products for business applications (Tangen [25]). In addition, some criteria for design process too were proposed.

Among the articles read we found Iivari’s [3] claim: “Evaluation as a DSR activity lies at the descriptive level. It studies how effective and efficient the artifacts are compared with existing artifacts.” To our mind, Iivari might mean that measurement activities are similar as in descriptive studies. But we prefer such a view that when goals are directing build activities at the prescriptive level then also evaluation activities measuring satisfaction of goals must be performed at the prescriptive level.

7 Discussion

In this study based on some important articles on design research we demonstrated a large variety in definitions of research problem (Problem A), descriptions of knowledge produced (Problem B) and instantiations built (Problem C), and ways to evaluate results (Problem D). Phenomenography as a research method helped to pay attention to differences and encouraged to find some structures, often classifications. Especially in Question A it gave three different set of classifications for research problem in design research. The classifications based on either research object or problem owner or an idea behind of solution. Our framework of a problem-solving situation (Fig. 1) by which we can explain a common problem-solving situation can be in the future used to reduce a conceptual confusion in defining a research problem in design research, and also an opportunity alternative can be then taken into account for complementing a definition of a research problem.

Concerning knowledge produced (Problem B) and instantiations built (Problem C) a variety seems to be natural and it will increase in the course of advances in technology, social and informational innovations, and innovations can concern both design product and design process. Concerning Problem D improvement studies seem to play a central role in design research and hence some measurements are needed to test that improvement has achieved. The authors, however, did not ask from whose point of view improvement is considered. There can be different interested parties with differing utility or goal functions. For some cases there is Analytic Hierarchy Process (AHP) method [34] to take care those differing interests.

We also showed that the information systems development life-cycle with consecutive phases is not an only method for design research but also an evolutionary approach is possible and used. Our tentative guess is that our proposal to record a descriptive method in connection with a construction project of an information system or an IT artifact, and this descriptive method can act as a starting point when researchers want to develop a prescriptive construction method in design research. We like also to emphasize that evaluation of a certain information system or IT artifact will take place at the prescriptive level, not at the descriptive one.

A particular referee expressed a wish to compare phenomenography here and literature review as research methods. They both use secondary data as their starting point, but phenomenography differs from literature review in formulating categories (more abstract concepts) from original conceptions, and their relationships, possibly shown as structures.

We tried to collect all the best articles in design research from the high-ranked journals. But we must admit that our literature review is not exhaustive and there can be few excellent article to supplement our set of raw data. Hence, we recommend our colleagues to fulfill the results achieved here. Another limitation is based on the fact that the differences found in Problem A could show up in other problems (B, C and D) because conceptions on research problem, knowledge, innovation and evaluation of results are not totally independent. Fortunately, we considered phenomenographic results in the latter at the higher, more abstract levels than in connection with Problem A.