Keywords

1 Introduction

There have been decades of work dedicated to making significant contributions to, and expanding the field of, conceptual modeling. Contributions have come from many computing fields such as database design, software development, business process articulation, and ontology. However, it is still difficult to describe the breadth and depth of the field in a short phrase or a single definition.

Previously, we synthesized many definitions of conceptual modeling from the literature and our colleagues to define a framework for Characterizing Conceptual Modeling (CCM) [5]. However, CCM focuses on conceptual modeling (as done by modelers) as opposed to conceptual modeling research with its rich and varied contributions.

The objective of this paper is to take steps toward the characterization of conceptual modeling research to enable researchers to easily articulate their contributions to conceptual modeling (CM). This may facilitate discussion and debate among researchers and eventually promote more effective search of the literature. It might also acknowledge the various disciplines that contributed to CM and contributed to the recognition of CM within the computing disciplines.

This paper provides a brief description of CCM in Sect. 2 followed by a description of the Characterizing Conceptual Modeling Research (CCMR) framework in Sect. 3. We describe our evaluation of CCMR in Sect. 4 and provide discussion and related work in Sect. 5. Conclusions and future work are given in Sect. 6.

2 Characterizing Conceptual Modeling Framework

Our initial framework focused on conceptual modeling. We first compiled a broad set of definitions for conceptual modeling; this unearthed a number of purposes for CM as shown in Fig. 1. These purposes allude to the various models, intents, and people involved in conceptual modeling. Color indicates similar purposes: blue—understanding/communicating a domain, phenomenon, or (future) system; purple—setting forth the meaning of terms/concepts; green—supporting system building; red—eliciting and documenting conceptual models; and black—supporting formalization and reasoning. The ideas for Fig. 1 are abundantly supported in the literature (e.g. [8, 13, 26, 28, 41]).

Fig. 1.
figure 1

The purpose of a conceptual model (drawn from the literature) (Color figure online)

Full size image

Our framework to characterize conceptual modeling, CCM [5], described what was being modeled: information, events, processes, and/or interactions and the activity of conceptual modeling, in addition to the purpose of conceptual modeling. CCM focused on when the activity of conceptual modeling was done (Timing) and whether there was automated assistance (Automation) and whether the conceptual models (and conceptual modeling languages) were generic or domain-specific. However, when we applied the CCM framework to characterize CM papers, it was obvious that researchers engage in activities that lead to contributions that are not reflected in CCM. (E.g. does the paper contribute a new or extend an existing conceptual modeling language? does it contribute a conceptual model? or does it contribute a method or tool intended to improve the practice of conceptual modeling?).

3 The Characterizing Conceptual Modeling Research Framework

We used the following methodology as we developed the CCMR. We first tried to classify several papers from the CM literature using the CCM framework but we were able to characterize only the CMs that appeared in those papers, e.g. in examples. Based on our collective experience in the field of CM over the decades, we discussed the types of contributions made by papers in CM. This led us to the CCMR framework which we then applied in a series of experiments, as detailed in Sect. 4 below.

The CCMR framework consists of three parts (see Fig. 2). The first part provides the context for the conceptual modeling in a given project. The second and third parts expose the various types of research contributions that can be made in the field of conceptual modeling in two aspects: the use of conceptual models and/or conceptual modeling languages (CMLs), and the methods or tools provided.

The first part includes the intended purpose and intended domain (of the conceptual modeling activity considered in the paper), the domain used in any case studies or examples in the paper, and the intended users of the ideas/tools/ models etc. presented in the paper. Each field is optional. For example, a paper presenting a conceptual modeling language may be generic and thus would not have any particular intended application domain. But a paper might apply conceptual modeling to a new purpose (with existing conceptual modeling languages and processes) or identify a possible new class of intended users.

Fig. 2.
figure 2

Characterizing conceptual modeling research framework

Full size image

The second part of the framework accommodates significant new or extended conceptual models, conceptual model patterns, conceptual modeling languages; the evaluation of one or more modeling languages/models (often through comparison with other modeling languages/models); formalization of modeling languages, concepts, or perhaps even models; or the philosophical grounding that underlies modeling languages, concepts, or models.Footnote 1 The framework includes a level of modeling between a conceptual modeling language and a conceptual model for a conceptual modeling pattern or ideal model that could be adjusted, adapted, or extended in an actual conceptual model.

The third part of the framework captures the various activities involved in conceptual modeling through methodological support and automated tool support. As with part 2, a paper or research effort might contribute or extend, evaluate or compare, or even formalize methods or tools. Note that the portion on automated tools includes the use of mappings (from one conceptual modeling language to another conceptual or implementation-oriented language) and reverse engineering (i.e. developing a conceptual model from another, lower-level model). The philosophical perspective is not in this third part at this time because we suspect that the consideration of philosophical grounding would likely have influenced the conceptual modeling languages and models used as opposed to the methods, processes, or tools.

4 Evaluating the CCMR Framework

Over 15 months, we evaluated the CCMR framework with the results summarized in Table 1. In our experiments, we set out to evaluate the ease of use/feasibility/utility of the framework, correctness and consistency when using the framework, and the coverage of the framework for the field of CM. In our evaluation activities, we used various methods for selecting papers to characterize; this work could be viewed as an initial, small-scale step towards a systematic classification of CM.

We used the first version of CCMR (described in Sect. 3 above and referred to as CCM 2.0 in [6]) for the first three evaluation activities. Based on the results, we modified the framework slightly to include a fourth row in the second part (for Metamodels of Conceptual Modeling Languages), visually eliminated the vertical spectrum labeled Generic ... Domain-specific and eliminated the adjective business in front of business process, listed as one of the items that can be modeled. Thus, part 2 of the CCMR framework is not specific to a domain. Finally, we added a field in part 1 for Intended Level of Abstraction to delimit the focus of the conceptual modeling effort. Figure 3 shows the modified CCMR framework.

Table 1. Evaluation activities for the CCMR framework
Full size table
Fig. 3.
figure 3

The complete CCMR framework after adjustments

Full size image

4.1 A Range of Papers Characterized in CCMR

We selected three papers from the CM literature and we independently characterized them. There was much agreement about the main contribution(s) of each paper. Our efforts differed in the amount of detail. At one extreme, two of us adopted a minimalist approach where we described the context of the conceptual modeling research, as appropriate, and then focused on the (typically one or possibly two) key, significant contribution(s) of the paper. The other two tended to provide more detail: adding a number of secondary contributions on the one hand and filling in nearly every square (to provide a detailed overview of the paper) on the other hand. The difference was trying to identify the main/significant contribution of the paper vs. considering the CCMR framework as a “checklist” that needed to be filled in.

We selected 6 additional papers on diverse topics to characterize; this selection intentionally included papers authored by one of us because we wanted to characterize papers that at least one of us was highly familiar with. Note that these were characterized by one of the four of us and then discussed with the others. We adopted a minimalist approach because a description of the main contribution is likely to be more helpful for discussion and literature search.

The first paper is Chen’s classic ER paper [3] as Fig. 4 shows. (Note that the third part of the CCMR framework is omitted because the paper did not contribute tools or methodological guidance). Figures 5 and 6 provide summaries of the characterizations of the contributions of the nine papers (\(3+6\)) by showing an iconified version of the framework and then describing contributions according to its indicated aspects. Primary contributions are marked by filling in the corresponding cells in dark gray; secondary contributions are marked by filling in cells using a cross-hatch pattern.

Fig. 4.
figure 4

Characterization of Chen’s ER Paper [3] Using CCMR

Full size image
Fig. 5.
figure 5

Summary characterizations of Chen [3], Maté et al. [25], and Halpin [15]

Full size image
Fig. 6.
figure 6

Summary characterizations of Pastor et al. [29], Ruiz et al. [34], Pastor et al. [30], Terwilliger et al. [40], Embley et al. [7], and Storey [37]

Full size image

We were able to characterize all nine papers, demonstrating coverage of the framework. All fields in the first column of part 2 of the framework were used. Every column of parts 2 and 3 was used at least once, although for some fields it was only through secondary contributions, and every row in parts 2 and 3 was used at least once. Overall, this exercise provided an initial indication of usability, a very preliminary indication that it can be used consistently for the first three papers (albeit with differences between the minimalist/maximalist approach), an initial indication of coverage, and an indication of the utility of the fields in the framework.

4.2 Experience of Others Using the CCMR Framework

We organized two interactive workshops where CM experts tried our framework.

The first was at the ER2018 Conference [4], where 14 conference attendees selected a paper they had (co)authored and presented their characterization to the group. We offer the following anecdotal observations. All participants were able to complete the task and present to the group. No one gave up, as far as we were able to observe. Initially, some of the participants used the “maximalist” approach. But, with some prompting, all were able to identify the contribution they considered the most significant.

Papers evaluated by ER2018 workshop participants fit the patterns shown in Fig. 7 (some of which were repeated multiple times by different participants or for different papers). Figure 7 only gives the thumbnail of the actual characterization. Each field was filled in with textual descriptions of contributions.

Fig. 7.
figure 7

Variety of patterns from the first workshop participants

Full size image

At another research seminar, we repeated the exercise with approximately 24 experts and received 19 different characterizations of research contributions. Figure 8 shows a representative sample of the resulting evaluation patterns. The pattern in the upper-left corner represents the characterization of an entire book. The use of a large number of cells seems appropriate given the range of material in the book. Two others of the 19 evaluations used most of the cells in a “maximalist” approach.

Fig. 8.
figure 8

Sample of patterns from research seminar participants

Full size image

Through discussions and feedback from our colleagues, we were encouraged to: provide clear definitions for terms in the framework, introduce an additional level in part 2 for metamodels (of conceptual modeling languages), provide an MDA-oriented perspective where different levels of abstraction (e.g. CIM/PIM/PSM) are considered, and provide methodological guidance for using the framework. With the exception of metamodels, the framework provided reasonable coverage. The utility of the fields was demonstrated with significant use of most rows and columns in the framework. The Philosophical Grounding column tended to be used the least, but it was used repeatedly.

4.3 Assessing the Consistency of CCMR and Attempting to Scale Characterization

The final row of Table 1 summarizes our last assessment of CCMR. We selected 10 papers at random from the ER2018 Conference [12, 14, 18, 21,22,23, 27, 35, 36, 43]. We wanted to find out whether the four of us—with diverse backgrounds in conceptual modeling—could characterize these papers and do so consistently. For this evaluation, we used the CCMR framework shown in Fig. 3 and used only the paper titles, abstracts, and conclusions.Footnote 2

The first part of the CCMR framework was problematic. The intended purpose was filled in a diverse manner. One of us described the intended purpose of the paper. Some of us attempted to describe the (apparent) purpose of the conceptual modeling that was being considered in the paper. We used the “Other” choice (for Intended Purpose) an average of 3.2 times per paper (out of 4 reviews). Sometimes, we used basically the same intended purpose but at different levels of detail. For example, one or two of us chose “Business process modeling” while one or two chose “Process extraction,” which could be considered a subfield of business process modeling. The Intended Domain field was filled in a diverse manner for half of the papers. For the Intended Level of Abstraction field, there was a suggested controlled vocabulary: CIM (computation-independent model), PIM (platform-independent model), and PSM (platform-specific model) from the model-driven architecture (MDA) standard. One of us did not use that vocabulary; three of us selected values in diverse ways. Only one paper was described the same way by the three of us who used this vocabulary.

In parts 2 and 3, we exhibited strong consistency in the use of the columns. For approximately half of the papers, one of us (alone) indicated something in the Evaluate or Compare column: perhaps attributable to listing a secondary contribution (the “maximalist” approach). Our use of the rows in these two parts exhibited several interesting patterns; each occurred in approximately half of the papers characterized. A method/tool split occurred when some of us described a paper as contributing a method and the others, an automated tool. This is likely because a new tool is used in a method; a new method may have one or more tools. Also, using only the abstract and conclusions to characterize a paper made it difficult to discern if the paper was primarily contributing a method, a tool, or both. A similar pattern was the representation/tool-or-method split, where some of us described the paper as contributing a new representation (in the form of a new CML or CM pattern) whereas others described the paper as contributing a new method or tool (using the representation).

The following are some examples of these differences. When evaluating the paper of Ishikawa et al. [18], the excerpt in Fig. 9 from our combined reviews shows R2/R3 vs. R4 exhibiting a method/tool split and R1 vs. R2/R3/R4 exhibiting a representation/method-or-tool split.

Fig. 9.
figure 9

Excerpt of the authors’ combined reviews of Ishikawa et al. [18]

Full size image

As another example, the excerpt in Fig. 10 of our combined reviews of the paper by Leoni et al. [21] shows a representation/method-or-tool split, with Ra/Rb showing a new representation and Rb/Rc/Rd showing a new tool. There is also consistent use of the columns with every reviewer using the Contribute or Extend and Formalize columns. Three of four reviewers used the Evaluate or Compare column.

Fig. 10.
figure 10

Excerpt of the authors’ combined reviews of de Leoni et al. [21]

Full size image

As a final example, when reviewing Santos et al. [36], there was perfect consistency among the four reviewers as Fig. 11 shows. (There were no entries in the Formalize or Philosophical Grounding columns).

Fig. 11.
figure 11

Excerpt of the authors’ combined reviews of Santos et al. [36]

Full size image

The first part of the CCMR framework, the Conceptual Modeling Context definitely requires some modification for consistent use. The Intended Purpose field was inspired by our early review of conceptual modeling definitions, as Fig. 1 shows. Thus, it is not particularly appropriate for describing conceptual modeling research. The Intended Level of Abstraction field would likely benefit from defining additional controlled vocabularies for various contributing fields to conceptual modeling (e.g. MDA uses CIM/PIM/PSM; ontology could use foundational/reference, upper, and domain; database could use conceptual, logical, and physical). The Domain of Case Study/Examples field is unlikely to be filled in based only on the title, abstract, and conclusions of a paper.

For CCMR parts 2 and 3, we are cautiously optimistic. The various activities involved in research (as reflected in the major contributions), namely Contribute or Extend, Evaluate or Compare, Formalize, and (Provide) Philosophical Grounding were used in a consistent manner and offered major coverage of the field. For the rows in part 2, the CM:CMPattern:CML:Metamodel for CML spectrum is familiar to most researchers. It is often quite easy to describe a contribution as being in one of these areas. However, there are uses of these terms that blend across the levels of this spectrum. Still, this spectrum provides value. In the third part of CCMR, the description of methods or tools is very important; many papers were described as contributing one or both. The method/tool split in our evaluation of papers from ER may derive primarily from our use of (only) the title, abstract, and conclusions which, in general, may not be an effective way to characterize papers. This is the problem of classifying any scholarly work: how familiar does a librarian or other professional need to be with the work in order to characterize it correctly? However, this approach does give an effective starting point.

5 Discussion and Related Work

Our work seeks to define a knowledge structure that can be used to characterize and search the conceptual modeling literature. It may also provide a basis to compare existing bodies of work or help to articulate the contribution of various disciplines to the field.

CCMR comprises a set of metadata fieldsFootnote 3 (part 1) followed by a two-dimensional knowledge structure where each field can be filled with text (parts 2 and 3). One or more controlled vocabularies or taxonomies (from which to choose values) could be useful for the fields in part 1. Such a vocabulary may exist or be synthesized from the literature (e.g. lists of relevant topics from CM conference calls for papers). The fields in CCMR parts 2 and 3 are not intended to be mutually exclusive; a paper may make several significant contributions. Reasonable independence of the columns (and rows) would be helpful because characterizations are likely to be more precise and thus more useful for characterizing papers and searching the literature. The representation/tool-or-method and tool/method splits do not necessarily reflect a misunderstanding of the row titles. Conceptual modeling experts undoubtedly understand the difference between a new representation scheme, a new method, or a new automated tool. Our analysis primarily indicates that it may be difficult to pinpoint the specific, significant contribution based only on the title, abstract, and conclusion of a paper—specifically for a topic unfamiliar to the reviewer. Overall, our assessment based on the various evaluations is that the CCMR framework is easy to use and exhibits very good coverage of the field.

In terms of related work, other efforts have tried to characterize CM, and this work has been ongoing for nearly four decades. For example, issues related to conceptual modeling from three reference disciplines is found in Brodie et al. [2]. Teorey et al. distinguish conceptual-versus logical-models [39]. Storey et al. provide a brief history and attempt to extract emerging research themes [38]. Thalheim argues that even the distinction between “model” and “conceptual model” is not clear [42]. These and other similar efforts are valuable and cover aspects that are significant in the conceptual modeling domain, but this coverage is partial and it neither allows delimiting in a general way the kind of CM research a paper involves nor comparing different papers in a precise way. In contrast, our proposed CCMR framework consists of a knowledge structure that provides a holistic perspective intended to succinctly characterize research contributions and facilitate literature searches.

More broadly, there is a significant amount of related work with respect to providing frameworks to characterize fields of study in computer science, information systems, and related disciplines. In some cases, broad frameworks explain a field of study and give guidance to researchers on how to go about the various research activities. Examples include movements toward design science [16, 24, 45] and evidence-based software engineering [19, 20, 32, 33]. In other cases, frameworks are intended to provide a taxonomy for classifying work done in a field (e.g. [46, 47]). Our approach fits somewhere between these two points, and is customized to the field of conceptual modeling.

The most distinctive aspect of our approach comes from the particularities of our working domain, namely, conceptual modeling research. We, therefore, focus on research that is strongly related to the process of conceptualizing and its associated results. For example, the analogous approach of March and Smith [24] states a similar problem, but for a different domain. They propose a research framework in information technology (IT). In their case, research in IT must address the design tasks faced by practitioners. IT is technology used to acquire and process information in support of human purposes.

In contrast, we are interested in research in conceptual modeling, so we must address how conceptualizations are faced by modelers, using CMLs to create CMs. This is why, even if the main goals are shared, the proposed dimensions are different, because they are adapted to the respective working domains. Considering the topic of a relevant research paper (in terms of conceptual modeling context, languages, tools, and processes), March and Smith [24] categorize the research outputs (constructs, model, method, instantiation). They identify the research activities (build, evaluate, theorize, justify) that are similar to the horizontal dimensions that we propose in Figs. 2 and 3.

These dimensions also have analogies in the framework of Wieringa et al. who propose classification and evaluation criteria for requirements engineering papers [46]. Under the claim that in requirements engineering much of what is called research is really design, they emphasize that while design produces an artifact, research produces new knowledge. Their classification scheme uses three top level aspects: research activities, design, and others. With a similar goal but in a different context (conceptual modeling research), our work proposes specific dimensions that focus on how to characterize the context for the conceptual modeling that is under consideration in a given paper, and the various types of research contributions that can be made in this field.

Wieringa extends that framework for design science research in terms of research goals [45], where design, mathematical analysis and empirical investigation correspond somewhat to our “contribute or extend,” “formalize” and “evaluate or compare” dimensions. Our “philosophical grounding” dimension is additional evidence of the particularity of the CM domain: conceptualizing is strongly connected to the work on foundational ontologies; therefore this aspect must be explicit when characterizing research in conceptual modeling.

At a narrower scope, some frameworks describe a particular area succinctly and clearly in a way that helps other researchers position their work. An example is Flynn’s description of approaches to computer architecture that included so-called SISD, SIMD, and MIMD types [9]. Research in computer organization and architecture since that point has generally accepted and used these labels.

Survey research also contributes considerably to the understanding of a field of interest by comparing specific primary research projects within the field. Well-known exemplars in conceptual modeling include surveys by Hull and King [17] and Peckham and Maryanski [31]. Batini et al. summarize approaches to view and schema integration [1]. A survey naturally requires the creation of a framework for comparison, but the scope of the framework is generally tailored to the immediate needs of the survey, and so is narrower than the kind of framework we propose. However, the CCMR framework should be useful for future researchers performing surveys within conceptual modeling.

6 Conclusion

This paper has presented and applied a framework for characterizing conceptual modeling research, with the results showing that it is, indeed, possible to provide a common way for researchers to characterize their work.

Future work will continue to evolve the CCMR framework by providing a glossary of (multiple versions of) standardized terminology, conducting more in-depth evaluations of the framework, and elaborating the labels for all parts of the framework with respect to their scope and intent. We also intend to develop guidelines for using the framework and to continue our evaluation and improvement of CCMR.

If the framework shows evidence of being a relatively complete, consistent, and reasonably stable way to characterize the field, we (or others) could consider providing tools to support the characterization of papers, search for similar papers, and so forth, which would be especially useful for scholars submitting their work for peer review. Finally, if the framework is deemed to be a relatively complete characterization of the field, we (or others) could provide a systematic review of (portions of) the literature using the CCMR framework as a guide.