Keywords

1 Introduction

We want information systemsFootnote 1 to be successful, but we do see projects fail in many cases [1]. So, the implementation of information system requirements is not that straightforward. Architecture principles play, in theory, a key role in guiding the design and the implementation of information system (IS) requirements [2,3,4]. But the question is: are architecture principles effective in practice, i.e. do they have a – hopefully positive – contribution to the implementation of IS requirements?

To answer this overarching question of our research, we consider in our research the use of architecture principles during the implementation of IS requirements. We measure both the architecture principles and the success of the implementation of the information system requirements, in order to determine some contribution. In this paper we focus on the first part of our research: the measurement of architecture principles.

This paper is the result of creating an architecture principle measurement instrument out of theory, challenged in a case study. To do so, we first identified and described architecture principles based on theory. From these results, we created a measurement instrument and tested it in a real-life case. We start this paper with the research methodology in section two. In the next section we describe a definition as well as a framework for identifying and describing architecture principles. Section four provides an architecture principle measurement instrument. The validation results of the measurement instrument, based on a case study, are described in section five. We finish this article with limitations, conclusions and further research.

2 Research Methodology

2.1 Research Questions

Based on the problem analysis described in the introduction, we answer the following main research question:

Main Question: “How to identify and measure architecture principles?”

We answer this research question by splitting the question into two sub research questions. To identify and measure architecture principles we first need to define and describeFootnote 2 them. So, the first sub research question will be:

Sub Question 1: “How are, according to literature, architecture principles comprehensively and consistently defined and described?”

In this first sub research question we address two elements. First is the focus on the identification of architecture principles: what kinds of statements are in scope as architecture principles? Secondly, we need a model to describe an architecture principle in a manneer as exhaustive as possible.

After answering sub question 1, we have to challenge the outcome in practice. Are the definition and description of the principles useful in a real-life situation and can we used it for measuring architecture principles? Therefore, we phrased the second sub question as:

Sub Question 2: “How to measure architecture principles in practice?”

To answer this question, we need an measurement instrument. This instrument should be able to identify architecture principles in the first place. Secondly, we have to determine the possible values of the characteristics and attributes of the principles. In determining the characteristics and attributes, it is important that the description is coherent and complete. Though, a measurement instrument in itself is not enough. We also need a method to collect and to analyze the information.

To answer these research questions, we used the approach described below.

2.2 Research Approach

Obviously, we have chosen a literature study for answering Sub question 1. For Sub question 2 we used the case study approach. There are, in general, two reasons for using the case study approach:

  1. 1.

    Early phases of research: case studies are useful “in early phases of research where there may be no prior hypotheses or previous work of guidance” according to [5], but also stated by [6].

  2. 2.

    Context-related: if a phenomenon is strongly related to its context, case study research “is used to investigate a specific phenomenon through an in-depth limited-scope study” [6]. Yin states that case studies are necessary “when the boundaries between phenomenon and context are not clearly evident” [7].

Architecture principles have been studied since the early 1990’s. This would imply that the first reason does not apply to our research. But in [2] we already identified that there is a lack of research about the practical use of architecture principles. And although there is a scientific basis laid out by this theory, we would like to challenge its adequacy, “because they have little empirical substantiation” as Eisenhardt [6] would call it. And, although some of the past publications introduced new definitions and descriptions, they all confirmed the conclusions of previous publications [3]. It is time to juxtaposition theory and practice.

For answering our research question the second reason to choose for the case study approach, is valid as well. In our literature review we cited several authors that “the context in which the architecture principles are used, is important as well, in particular for the effect of a principle.” [3]. So, architecture principles are conceptual instruments used by people in the context of the design and implementation process.

2.3 Research Process

For the literature review we used a six-step approach, based on the literature review method of Webster and Watson [8]. This method was also used by Stelzer [9] and Haki [10] and with that, we have equivocality of the research process in this area of research. The six steps were:

  1. 1.

    Defining the boundaries of the literature review;

  2. 2.

    Compiling two models (one for each research question) needed for analyzing the results of the literature review;

  3. 3.

    Identifying and selecting relevant literature;

  4. 4.

    Reviewing the results of the literature review using our models;

  5. 5.

    Answering the sub research questions based on the results of the analysis;

  6. 6.

    Addressing the limitations, discussing the results and presenting implications for further research.

Models Used Supporting Literature Analysis.

In step two we defined two models to help us analyzing the results that we found in the selected publications. Our aim is to have an instrument helping to decompose the definitions found. This decomposition helps to identify all elements relevant for describing the essence of the term ‘architecture principle’ and also to determine which additional elements are not distinctive.

To experience the essence of a subject, the interrogative WH-questions [11] are helpful. We used the 7 most-used interrogative WH questions (what, why, how, with which, who, when, and where) to decompose the definitions. Each element of a definition is attached to one of the questions. After the decomposition of all definitions, we analysed per question the similarities and differences in phrasing. Based on the analysis we formulated a new phrasing for a strengthened definition (see Table 3 in the Appendix) (Fig. 1).

Fig. 1.
figure 1

Decomposing architecture principle definitions with WH-questions [3].

Full size image

For describing the architecture principle, we used a framework to model the architecture principle. In this framework (see Fig. 2) we distinguish the relevant entities in our research, like architecture principle, design, and strategy. Most of them are artefacts of the development process.

Fig. 2.
figure 2

Framework for modelling the characteristics of an architecture principle [3].

Full size image

An entity has characteristics, defined as “a feature belonging typically to an architecture principle and serving to identify it”. A characteristic, in its turn, has one or more attributes, as “an inherent part of the characteristic”. We also address the relationships between the entities and between the entity and its characteristics. In designing the model we used the UML in accordance with [12, 13].

During the literature review we listed each new characteristic or entity we found in a table, including the definition, attributes and relationship. When we did find a synonym, we added that one to the one found earlier. In the end this resulted in a list of unique characteristics, entities and relationships to each other. As a final step, we designed the model describing the architecture principle using the framework.

Search & Selection Criteria.

In step three we used the various well-known databases, journals, and search mechanisms (for instance, EBSCO, Google Scholar, AISeL and Research Gate), to find relevant literature. In selecting the right publications, we used the following inclusion and exclusion criteria. First criterion is the selection of English publications only. Secondly, the title or abstract has to contain at least the combination of terms <architecture principle> AND (<Enterprise> OR <IS> OR <IT>). Based on those criteria we obtained a list of publications (see Table 1 in the Appendix).

We analyzed the abstracts of those publications and selected those ones addressing the definition or description of architecture principles in general. We excluded all other literature covering the application of architecture principles. Each of the selected publications was read and analyzed extensively and all relevant information in the publication was structured for analysis.

In some publications we did find citations to prior literature as well. When those publications were addressing specific elements of architecture principles, we added them to our list of publications. In most cases those publications did not satisfy all selection criteria, because they were addressing another (related) subjects.

Fig. 3.
figure 3

Case study approach [4].

Full size image

Case Study Approach.

For the case study approach we looked at the steps defined by [6]. She based her steps on literature from authors like [7, 14,15,16] and experience from authors conducting case studies like [17,18,19]. We grouped the steps of Eisenhardt in the following three phases (Fig. 3):

  1. 1.

    Preparing the Case Study: in this phase we defined the research question, as described above. Secondly, we selected a case. Because our overall research program is focusing on Dutch government organizations, we chose a case from the Dutch Tax Agency. One reason to choose this case study was, of course, that the project did use architecture principles in the first place. Besides, the chosen project had to be finished: we could see the use of principles during the entire implementation life cycle. We crafted the instruments and protocols to be used during the case study research. We defined our research team with subject matter experts, defined our survey, and built our measurement instrument and method.

  2. 2.

    Doing the Research: this was the iterative phase of data collection, analysis and theory building. We collected the data from documents, surveys, interviews, and a site visit. Based on the results of our desk research, we aimed at specific subjects during our interviews. We added all collected data to the spreadsheets that contained our measurement instrument. During this phase we sharpened our measurement instrument, adding new characteristics and attributes. After all data were collected, the research team evaluated both the measurement instrument and method and made suggestions for improvement.

  3. 3.

    Closing the Research: the last phase compared the results with existing literature and to end the case study because there were no new data sources to investigate. We ended the case study after evaluating all possible data sources in our spreadsheet and reporting the results of the case study back to the Dutch Tax Agency. We closed the research by answering the research question.

Because a research approach has to be reliable and valid [20], we used these steps to guarantee the objectivity of the fact finding in this case study approach.

2.4 Boundaries of Research

Our research is focusing on architecture principles to be used for implementing information system requirements. In the initiation of this literature review, however, we found out that there is a lack of architecture principle literature related to the scope of IS specifically. There is, however, literature on enterprise architecture principles and software architecture principles. IS architecture is part of enterprise architecture [21], and therefore it is possible to confine the literature review to the enterprise, which we did. Therefore, all conclusions related to principles used in enterprise architecture can be applied to principles for the IS architecture as well. As a consequence, we scoped our literature search on architecture principles related to Enterprise, Information System, or IT.

Secondly, as mentioned in Sect. 2.2, the context in which architecture principles are used, is relevant. With narrowing the scope to Dutch government organizations, we are scaling down the research scope, resulting in more reliable research results.

3 Defining and Describing Architecture Principles

By describing the results of our literature study, we answer sub question 1. We start with the general research results, and in the two following sub-sections we provide the definition and description of the architecture principles respectively.

3.1 General Literature Study Results

After the search and selection of publications we found 28 publications we rated as most relevant in defining and describing architecture principles (see Table 1 in the Appendix). Those publications are covering a time span between 1990 and 2017. Some of those publications introduce new definitions and descriptions, while others strengthen existing ones. Many publications confirm and use the conclusions of previous publications.

All authors confirm the importance of architecture principles for the architecture and design of systems. There was only limited practical research about architecture principles. In that research, it turned out that architects state that architecture principles have added value, according to different surveys [22,23,24].

Of all publications found, most of them are describing architecture principles in general, calling it Enterprise Architecture (EA) principles. Only a few are related to a specific layer of the architecture, such as business, information system, application, or technical infrastructure [25,26,27]. Specific publications related to architecture principles and information systems are difficult to find. Therefore, as discussed in Sect. 2, we used the more general yet still applicable literature on EA principles instead.

Our research shows, generally speaking, consensus about architecture principle definitions and its characteristics over the previous 27 years. In 2013 Haki already mentioned the increasing consensus on what he was calling “the nature and definition of EA principles” [24]. Since Haki’s paper, there were only a few new publications with similar ideas. Nevertheless, between the 28 publications we did find some inaccuracy or incompleteness, which we will elaborate in the next two sections to strengthen both the definition and description of an architecture principle.

3.2 Definition of an Architecture Principle

For defining architecture principles, we first listed all found definitions in literature in a table (see Table 2 in the Appendix). In this overview it is interesting to see that the definitions in later publications are a consolidation of previous definitions and are evolving to more comprehensive ones. In [12, 24] the elaboration of the definitions is quite detailed, which would make it in our terms more a description than a definition. It is noteworthy to see there were no really deviating definition or remarks on prior publications whatsoever.

To give insight in the similarities and differences between those definitions, we decomposed the definition with the WH-questions. During the analysis we were focusing on the essence of an architecture principle, while the definition should be comprehensive and consistent as well. Here we will address the similarities and differences per WH-question as also summarized in Table 3 of the Appendix.

In describing the determining elements of an architecture principle most authors do agree that an architecture principle is a statement, as a type of design principle. In accordance with Haki et al. [24], and Fischer et al. [12] we state that the architecture principle should be “based on business and IT strategy”, because with architecture we want to focus on the essential requirements. Although many authors do agree that an architecture principle is a type of design principle, we omit this because we address the design-element later in the definition.

Although defined in many different ways, the purpose of the architecture principle can be summarized as describing restriction to the design. This is consistently formulated by Greefhorst and Proper [26] with “normatively describes a property of the design of an artefact”. In our case the artefact is the information system.

In many definitions the objective of the architecture principle is described as “should be met by the architecture” or “justification for decision making throughout an EA”. But an architecture in itself has the objective that a system meets its essential requirements. And because architecture is focusing on the ‘essential’ requirements, we would like to address this in the definition. With that, it is the distinguishing element between design and architecture principles.

Looking for additional elements in describing the essence of an architecture principle, we do not see real distinctive parts. In our analysis of the remaining WH-questions, we only identify elements, which we can link to elements in our definition. E.g., “a rationale is formulated” can be linked to the elements “is based on business and IT strategy” and “its essential requirements”. That does not mean that those elements are irrelevant: those elements have to be part of the description of the architecture principle, as we already indicated above. This hypothesis is strengthened by the fact that only four authors are addressing one or more remaining WH-questions in their definition.

By combining all these findings, we define an architecture principle for information systems as:

“An architecture principle is a declarative statement, based on, at least, business and IT strategy. It normatively describes a property of the design of an information system, which is necessary to ensure that the information system meets its essential requirements.”

With this analysis we conclude that there is consensus in literature about the definition of an architecture principle. The differences in the definitions found are related to the use of undefined terms or the use of synonyms. Furthermore, we found incomplete or copious definitions, without catching the essence of an architecture principle.

3.3 Description of Architecture Principles

Although the architecture principle now has been defined, it still has to be described as well. As a consequence, we answer the question with which characteristics architecture principles can be described and how architecture principles are related to its environment. In answering this question, we started investigating the different types of principles.

For many years, there were, in general, two types of architecture principle: design principles and representation principles [9, 10, 23]. The latter type refers to the way architectures should be represented, while the first directs the design of a system itself. In literature they were described as having different characteristics and serving different objectives.

Recently Lumor et al. [28] introduced a third type of principle, namely architecture management principles. Those architecture management principles are reflecting the process nature of EA. The idea behind this third type of principles is the fact that in general architecture and its principles might be a product, process, result, etc. [26, 28,29,30], and an architecture principle should address the process view as well.

In this literature review we take the view that these different types are different perspectives on the same kind of architecture principles. This is in accordance with Lindström [31], who distinguishes syntactic and semantic characteristics. Syntactic characteristics are describing the elements and their interrelationships of a principle. Semantic characteristics describe the quality elements of the principle. Haki posed in [10] that this differentiation is the same kind of subdivision as the differentiation of design and representation principles. So, depending on perspective, the architecture principle has more or less specific characteristics.

We do understand that, with this choice, we will collect all kinds of characteristics, which also might be related to each other. We encountered this consequence already in our previous case study research [32].

In describing architecture principles, we distinguished the characteristics of the architecture principle itself on the one hand and the relationship with entities in its environment on the other. This breakdown is comparable with the definition of Richardson and Aier et al. [12, 23, 33] in a core definition and basic extensions, and helpful to get more transparency in the description of the architecture principle. In literature we found all kinds of characteristics and entities described. Using our framework as described in section two, we listed all these characteristics (see Fig. 4 and Table 4 in the Appendix).

Fig. 4.
figure 4

Model for describing the architecture principle and set [3].

Full size image

Characteristics.

We start with the ‘specification’ characteristic. There is consensus on the specification of an architecture principle by the attributes ‘statement’, ‘rationale’, and ‘implications’. All authors naming these three attributes as an inherent part of an architecture principle. We group these three attributes together in one characteristic, because together they specify the architecture principle.

The second characteristic of an architecture principle is called ‘measure’. This characteristic describes the level of fulfilment of the principle. To some authors [12, 24, 31], this is a typical characteristic of an architecture principle, because an architecture principle should to some extent be respected.

Hoogervorst [27, 34], endorsed by [13, 23], and Greefhorst and Proper [26], introduced the characteristic ‘Key action’ as guidelines for implementing the principle. Recently Marosin [35] added the characteristic ‘Precondition’, which has to be fulfilled by key actions before a principle can be applied. Because both elements are strongly related to each other, we consider them as attributes of one characteristic called ‘Prerequisites’ of which the principles depend on.

We also introduce the ‘meta data’ characteristic. This characteristic typifies the architecture principle so it can be managed. Attributes like name, assurance, visualization and generic information are in scope of this characteristic. Many of such attributes are defined by Greefhorst and Proper [26] and till now there is no exhaustive overview of this kind of attributes.

Finally, there are all kinds of quality, or semantic, attributes defined in literature, which the architecture principle should meet. TOGAF [25], Van Bommel [36], Lindström [31], Marosin [35], and Greefhorst [26] all have their own list of quality attributes. A more detailed comparison shows that they only use different terms for the same type of attributes or use a slightly different definition of the quality attribute. Therefore we choose the quality attributes used by Van Bommel [36] and Greefhorst [26]: ‘Specific’, ‘Measurable’, ‘Achievable’, ‘Relevant’ and ‘Time-framed’ (SMART). The reason to choose this list of quality attributes, is the fact that they are defined quite detailed in [26] and that they are easy to remember because of the re-use of the SMART criteria for objectives.

Entities in Its Environment.

Next to their characteristics listed above, the context in which the architecture principles are used, is important as well [9, 12, 13, 26, 37], in particular for the effect of a principle. The key context of an architecture principle, according to literature, consists of the ‘design’, ‘requirements’, ‘the architecture’, ‘the strategy’, and ‘the architecture principle set’. We describe those relationships one by one (see also Fig. 4).

The most direct relationship an architecture principle has, is with the ‘design’. Architecture and therefore also architecture principles restricts the design freedom of a system, according to [26, 27, 34, 38]. As we already have seen in the definition of the architecture principle, that restriction is necessary “to ensure the information system meeting its essential requirements”. So, via the design the architecture principle should ensure the information system satisfies the ‘requirements’.

In the architecture principle definition, we also have described the statement “…based on business and IT strategy”. In our literature review we did not see a clear clarification of the relationship between ‘business and IT strategy’ and an architecture principle. We consider architecture principles as part of the ‘architecture’; the ‘strategy’ guides this architecture. Because an architecture principle is part of the architecture, it is based on the strategy as well.

And, most of the time, an architecture principle is part of a set of principles. Although in most literature the focus is on individual architecture principles, a principle is only effective if it is part of a set [9, 26, 31, 35, 39]. Because we are interested in the contribution of architecture principles, we have to describe “the architecture principle set” as well. We define an architecture principle set as:

“a group of architecture principles defined and presented as a collection”.

Because a set of principles is an entity in itself, it has characteristics and attributes as well. Based on Greefhorst and Proper [26] we define three types of characteristic: ‘classification’, ‘meta data’ and ‘quality of the set’ (see Table 5 in the Appendix).

First, architecture principles are grouped together based on a ‘classification’. This ‘classification’ is based on the type and scope of the architecture principles. The type is related to the architecture layers of an architecture model. There are many definitions of architecture layers in use like TOGAF [25], Zachmann [40], and IAF [41]. In our scope we consider the ‘information system’ layer and within this layer the subdivisions ‘application’ and ‘infrastructure’. Architecture principles can also be classified based on the (organizational) level of use: for a ‘specific solution’, a ‘division’, for ‘an entire organization’, etc.

Secondly, an architecture principle set can be typified by ‘meta data’ to manage the principle set, such as ‘name’, ‘release number’, ‘amount of architecture principles in the set’, etc. Some authors do address the point that the amount of principles in the set should be as small as possible [31, 36, 42]. Many attributes may be added to the characteristic ‘meta data’, and for now there is no complete list available.

And lastly, similar to the individual architecture principle, an architecture principle set meets quality standards: ‘quality of the set’. In the case of a set of principles we distinguish the attributes ‘representative’, ‘accessible’ and ‘consistent’.

Literature Review Results.

Based on the literature review we built a model to describe the characteristics of the architecture principle including his environment (see Fig. 4). As discussed in our analysis above, this model is diverging slightly from the meta-model of Aier et al. [12, 13].

In our literature review we determined consensus on the characteristics. Although we are of the opinion that the different types of architecture principles are perspectives on the same kind of principles, we did not find any contradictions. We added and reorganized some characteristic and attributes. As already addressed in a previous case study research [32], some characteristics were defined relatively subjective in literature, using terms like “significant”, “easy”, or “obvious”. We have strengthened the definitions where possible (see Tables 4 and 5 in the Appendix), while we are aware of the fact that architecture principles are semi structured, informal and written in natural language [21, 35, 39, 43, 44]. Furthermore, we have described the architecture principle set with characteristics and attributes as well, because the contribution of architecture principle is only effective in a set.

4 Measuring Architecture Principles

After the definition and description of architecture principles in theory, we need both a measurement instrument and a corresponding method to be able to measure principles in practice. In section five we will challenge this measurement instrument in a real-life situation.

4.1 Measurement Instrument

To measure architecture principles and related architecture principle sets in practice we need a well-defined measurement instrument. This measurement instrument should be able to:

  1. 1.

    Identify the architecture principles and the architecture principle sets related to them;

  2. 2.

    Describe architecture principle and the architecture principle sets with their characteristics and attributes.

Identification.

Based on the literature review as described in section three, we formulated the definition of an architecture principle and set respectively:

“An architecture principle is a declarative statement, based on, at least, business and IT strategy. It normatively describes a property of the design of an information system, which is necessary to ensure that the information system meets its essential requirements.”

and

“a group of architecture principles defined and presented as a collection”

We use these definitions to identify the principles and sets, by checking whether or not an architecture principle (set) fulfills all elements of the definition. So, in the measurement instrument we designed a definition check for the elements ‘declarative statement’, ‘based on business and IT strategy’, ‘normatively’, ‘describes a property of the design’, and ‘necessary meeting its essential requirements’. If it is an architecture principle, each of these elements has to be present. And for each element there are explanatory facts as well. For an architecture principle set we do see every group of architecture principles described together as a set, as in the definition has been set.

Description.

For describing both the individual architecture principles and the sets of architecture principles we used the model of characteristics and related attributes as described in section three (see Fig. 4). Using this model, we list in our measurement instrument all characteristics and attributes, including their definitions as defined in Table 4 of the Appendix.

Per architecture principle we collect all data related to its characteristics and attributes. In a spreadsheet we record per principle all relevant data found, categorized by source. We use the same approach for describing architecture principle sets.

4.2 Measurement Method

To be able to use the measurement instrument, we also need a reliable and valid measurement method to measure the architecture principles. The measurement method helps with collecting and analyzing the right data and finally with measuring the architecture principles. The measurement method is an iterative three-step approach (see Fig. 5).

Fig. 5.
figure 5

The measurement method [4].

Full size image

Collect Data.

The objective of this step is to collect data about architecture principles and the architecture principle set. Therefore, data about the related artefacts are also relevant: IS architectures, IS designs, requirements and business & IT strategies. We use different methods to collect data: desk research, surveys, interviews and site visits.

For the desk research all kind of documents might be useful, like architecture descriptions, requirements specifications, test reports, and so on. All these address elements of the architecture principles and/or the related artefacts. The survey is used to collect data before the interview sessions, to be able to focus on specific items during the interview. The survey consists of open questions based on the characteristics in the framework. The interviews take place with at least two members of the research team. All interviews are recorded, and the minutes of the interview are sent to the interviewee for feedback. Interviewees are architects, software engineers, the test manager, the project leader, and the system owner. Site visits are useful to see the information system running in the daily operation and to consider to what extent the essential requirements are implemented.

We record all relevant data per architecture principle and per source, in order to have different facts about the same characteristic or attribute available. This is useful for the analysis of data, when differences or even conflicts among the data about a specific characteristic or attribute occur.

Analyze Data.

In this step we analyze the data to check the precision and accuracy of the data and to find exceptions and trends. For all data collected we check for inconsistencies between sources. If so, we have to go back for data collection to find the right or new data. If, afterwards, data conflicts remain, we have to explain the differences or decide not to use the data.

Secondly, we check whether or not so-called architecture principles are in accordance with the architecture principle definition. We determine if the principle satisfies each element of the definition and write down that reasoning in a spreadsheet. If not, the so-called architecture principle is declared be out of scope.

We then analyze the qualitative data on exceptions. The analysis has to be done per principle, but also between different principles. We look at remarkable differences between attributes or characteristics of a principle or between principles. For instance, the key action cannot be related to the prerequisite of the principle. Or, one architecture principle is fulfilled completely, while another one is not, although those two are strongly related.

The final action is to quantify the data and find specific trends from the quantified data. We simplify the analysis between principles out from different cases. We quantify the data of each principle and set as follows.

  • We review the different sources per attribute and set the numerical score;

  • The score reflects the level of fulfilment of the definition of the attribute: ‘0’ is no fulfilment, ‘1’ is partly fulfilment and, ‘2’ is complete fulfilment. We call this our code scheme [45]. The reasoning for the score is described in the spreadsheet;

  • The score of the characteristic is the average of the score of its attributes;

  • Only for the “classification” characteristic of the principle set we use an alternative score: the scores used refer to the specific values the attribute can have. See the appendix for all attribute values of principle sets;

  • Finally, we calculate the average score of each characteristic over all principles and sets.

We now are able to make cross-section analyses, to create graphics and to search for trends.

Measure the Principle.

This final step is to evaluate the exceptions and trends. We describe the architecture principles and the architecture principle sets, including the overall conclusions about their state. Based on the qualitative and quantified analysis we evaluate the exceptions and trends. We explain those exceptions and trends and draw our conclusions as subject-matter experts. Of course, we add evidence supporting those conclusions.

We end up with describing the architecture principles and the architecture principle sets by describing their characteristics and attributes. In this description we add the qualitative and quantified analysis, including the conclusions.

5 Case Study

We started challenging our measurement instrument with one case study only. Before we use the instruments for many cases simultaneously, we first would like to test to what extent the instrument is useful in practice. Depending on the outcome of the first case study, we can decide how to continue. If there is a big misfit with the instrument itself, we will focus on improving the instrument. If it is working in practice quite well, we can start directly with the research itself and optimize the measurement instrument where necessary.

We used the ‘Teruggaaf Dividendbelasting’ (TDi), in English: ‘Return of Dividend Tax’, of the Dutch Tax Agency as case in our research. TDi is an information system supporting the return of tax on dividend payed to legal entities. TDi was rebuilt in an agile project, which we investigated until system release in December 2017. For this case study we reviewed nineteen documents, conducted five interviews with six different stakeholders and examined the TDi system itself during a site visit. These activities were done by our research team consisting of three researchers.

5.1 Architecture Principles of the TDi Case

In the rebuilding of the TDi system 55 potential architecture principles were used. According to our definition (see section four), only 36 architecture principles could be identified as such. In Fig. 6 we see the level of completeness of all those 36 architecture principles together, while the individual scores may differ between the principles.

Fig. 6.
figure 6

Level of completeness of the 36 architecture principles [4].

Full size image

Looking at the specification characteristic of the architecture principles we did recognize that none of the principles included a rationale, while the statement and implications were worked out well. A reference to the rationale, as they were described in other architecture documents, was missing as well.

80% of the principles has been fulfilled partly (36%) or fully (44%). Only one principle was not followed (“from object based to subject based working”) and in 17% we could not determine whether or not the principles had been fulfilled because of missing resource data (see Fig. 7). Interestingly, developers didn’t respect some of the principles as meant to be and implemented parts of the system in other directions.

Fig. 7.
figure 7

Level of fulfilment of the principles [4].

Full size image

With respect to the prerequisites, for seven principles specific preconditions were defined, while for all, some overall preconditions were defined. Not all of the preconditions have been fulfilled (completely) at the start or during the project, like “B/CAO building blocks available”. Therefore, not all principles could be fulfilled, as we did see. Surprisingly, there were no ‘key actions’ identified, to get the preconditions fulfilled.

All meta data was in place, so that managing the architecture principles was no issue. Information about author, status, version, users and much more was easy to find.

The architecture principles were meeting the quality attributes as well. The main reason why the overall score of the quality is not 100 percent, is that the rationale was missing. Therefore, we were not able to determine the principle’s intention and relevance as described in the ‘specific’ and ‘relevant’ attribute. All architecture principles, even when they were imposed from outside the project, were translated to the TDi context.

The architecture principles originate from two documents: a High Level Design (HLD) and a Project Start Architecture (PSA). The HLD describes the process, application and technical infrastructure of the TDI system, while the PSA focuses on the application and technical infrastructure only. In the HLD, twelve architecture principles are defined explicitly but most principles are only addressed by referring to other architecture documents. In the PSA, nine ICT principles are described, including directives for using in the TDi system implementation. In both documents many meta data attributes can be found, like authors, administrator, status, target audience, etc.

Given the many architecture principles mentioned in both sets we conclude that the sets are not representative for meeting the essential requirements. There are too many architecture principles adopted from the overall architectures, resulting in overlap. Those principles are not translated to a single principle specific for the TDi system. Although there are many overlapping architecture principles, there are no contradictory principles in the sets. The accessibility of the sets themselves is good, because they were managed by the architects of the TDi project. Because the sets refer to other documents, the accessibility of the original sets of principles is less evident.

5.2 Evaluation of the Measurement Instrument

For evaluating the measurement instrument, we have to test for both the identification and the description of the architecture principles, whether they are complete and coherent. To start with the identification of the architecture principles, the instrument was helpful in determining which of the so-called architecture principles are fulfilling the elements of the definition. As a result, nineteen of the 55 so-called architecture principles did not pass verification. On the other hand, we did not find any other statements, not explicitly called architecture principles, that did fulfil the elements of the definition.

The definition of the architecture principle set was not differentiating enough. There are many ways to present a group of principles together. In our case we analyzed the different kinds of sets to understand the interrelationships between those sets. We did see in this case study that the presentation of the architecture principles was related to other architecture documents, which were already in place. So, the way of presenting the principles is not necessarily related to the system itself but influenced by external factors. Therefore, our case study resulted in a changed definition of the architecture principle set: “a group of architecture principles defined and presented as a collection based on a similar type or scope of the architecture principles”.

Although we might state that - in this case - the identification of all individual principles was done, we also learned that the identification is also related to the essential requirements. In our case study the essential requirements were defined at a high level, e.g. “the system has to be future-proof”, so it was quite easy to link architecture principles to the essential requirements. So, in next cases more in-depth research to the essential requirements is necessary.

The coherence of the architecture principle definition was already theoretically explained in our literature review with the WH-questions approach [3]. During the case study we did not find any inconsistencies between the elements, which might suggest that the elements of the definitions are incorrect.

The second part of the measurement instrument describes the architecture principles. Looking at the completeness of the instrument we found in the case all but two of the attributes defined in the model. The ‘rationale’ attribute was defined in other documents and, although no ‘key actions’ were defined, in some interviews necessary key actions to take were brought forward. So, none of the attributes are irrelevant.

In our case study we detected some omissions in the model. The attribute ‘degree of acceptance’ has to be added to the ‘measure’ characteristic, because in our case this element was addressed by several sources. It describes an aspect relevant to the fulfilment of the principle and will be defined as: “level of acceptance of the principle by all of its users”. The attribute ‘preconditions fulfilled’, related to the ‘prerequisites’ characteristic, is also relevant to add. We explicitly saw in the case that, when preconditions were set, it was also relevant to know whether the preconditions were fulfilled. The definition of this attribute can be described as “the level of fulfilment of the preconditions defined”.

For the architecture principle set we will add an extra characteristic: ‘prerequisites’. We discovered in the case study that some prerequisites were not related to a specific principle, but to a group of principles. Besides the ‘precondition’ attribute, ‘basic assumptions’ were described for some sets as well. Basic assumptions are “relevant criteria for successful use of the principle”.

In this case we did not find any inconsistencies in the coherence of the description model. Some of the relationships as described in the model, e.g. ‘depends on’ or ‘level of fulfilment’, were described explicitly in the documents or way mentioned during the interviews. The amount of information, though, is insufficient to make fact-based statements about the consistency of the coherence. In this case it was clear that there are interrelationships between attributes, e.g. the missing of the rationale and therefore a lower score at quality, as we already described in [32]. More research data is necessary to make clear statements about the coherence.

5.3 Evaluation of the Measurement Method

We evaluate the measurement method by discussing the reliability and validity of the case study results. To challenge the reliability of the results, we want to know to what extent the results would be consistent when doing the case study research again. In the measurement method are different protocols defined to assure the reliability of the outcome: using different kinds of data collections, working with a research team, minutes including feedback, etc. All these mechanisms are important, because this case demonstrates the subjectivity of facts collected. Two architects, for example, were working closely together during the project, but had different opinions about the fulfilment of some of the architecture principles. The research team could, based on all different sources, make an expert judgement about the fulfilment.

Although we used different ways of collection data, in this case study we were lacking some in-depth information about the essential requirements. As a result, it was difficult to see to what extent architecture principles were adding value in meeting the essential requirements. Additional sources related to the essential requirements, e.g. interviewing extra business owners, would help to bridge this gap.

In evaluating the validity of the measurement method, we concluded that the description of the architecture principles reflects the real situation of TDi. Although we found some contradictory data, especially in the interviews, we were able to explain the differences in the data. In this specific case we were not always able to go into details of specific architecture principles. Because the case used quite some principles, 36 in total, it was difficult to address all individual principles. So, in following cases we need mechanisms to get more in-depth information about the individual principles.

6 Limitations, Conclusions and Further Research

6.1 Limitations

Literature Study Limitations.

Related to the literature study there are two threats to the validity of our results. The first limitation we have identified is the interpretation of the words that have been used in the definitions and the characteristics. Although semantics of natural language is always an issue in literature study, we encountered in several papers descriptions that were rather vague, and therefore subject to (personal) interpretation and possible wrong conclusions. Because many publications confirm and use the conclusions of previous publications, we judge the risk of misinterpretation low.

The other limitation is the rather broad scope of the literature search by considering architecture principles in the enterprise domain instead of architecture principles in the IS domain. In certain cases, we translated architecture principle characteristics to specific IS ones without knowing whether that would be valid in practice. This is a topic for further research. Because architecture principles can affect multiple architecture domains [26], we consider this as low risk.

To discuss the results of our literature review we also sent our draft paper to a small group of senior experts in this research area. We used their response to eliminate indistinctness in the draft paper. Besides that, the experts addressed some specific remarks related to the paper.

Several experts mentioned that formulating architecture principles is important, but that the use of architecture principle is the real issue in practice. The description of the architecture principle is a precondition, necessary to determine the use and the effectiveness of architecture principles.

The second remark is related to the ordering of the characteristics. As addressed in Sect. 3 we have chosen not to order the characteristics in this part of our research, although there are different perspectives on architecture principles. Two experts suggest to order the characteristics using some framework, for example the dimensions framework defined in Greeforst and Proper [26]. Such frameworks will help in evaluating the relationships between the different characteristics as well.

The last remark was related to the definition of the architecture principle. The original definition suggests that architecture principles are based on business and IT strategy only. We do agree with the experts that business and IT strategy are just two, be it important, sources to formulate architecture principles. So, we reframed the definition by adding ‘at least’ in the phrasing.

Case Study Limitations.

Although the arguments for using the case study approach are still valid, there are some limitations important to address in this case study. We are aware that one case cannot prove the completeness of the measurement instrument. As discussed in section four, the objective of this research is to test to what extent the instrument is useful in practice in the first place. For an extended test on completeness and coherence of the measurement instrument, we need more test cases.

A second limitation might be that the researchers, although all kind of protocols are defined, are biased in searching for characteristics and attributes. The moment we are introducing a model as a description of our research object, we see architecture principles through this model. We tried to avoid this prejudice by avoiding naming of attributes during the survey and interviews. The fact that we identified new attributes and characteristics, shows that we were open for new elements as well.

Finally, we can also note that there is currently no other instrument that measures architectural principles, so we cannot compare this instrument with other instruments. Studies have been done on the added value of architecture in general and we could apply that approach to the same case to assess whether comparable results will be achieved.

6.2 Conclusions

Conclusions for Defining and Describing Principles.

In most relevant publications we found 15 different definitions of architecture principles and a large set of characteristics. In a period of 27 years the definition of an architecture principle has been consolidated. Besides, there is consensus about many characteristics. Nevertheless, we did find some inaccuracy or incompleteness in definitions and descriptions, as we addressed in this paper.

First, we found all kinds of definitions in literature, with most of them incomplete or just not striking the essentials of an architecture principle. By decomposing and rephrasing the elements of those definitions we ended up with a more comprehensive and consistent definition.

In describing the architecture principles with characteristics, we defined all types of principles as different perspectives on the same kind of principles. We grouped together the rationale, statement and implication into one characteristic ‘specification’. Besides we distinguished ‘key action’ and ‘preconditions’ as two separate attributes, combined in the characteristic ‘Prerequisites’. We also considered ‘Meta data’ and ‘Quality’ as explicit characteristics of an architecture principle, which has not been done in all past literature. Where possible we tried to define the characteristics more objectively – despite the fact that in natural language, interpretations of words are always possible. Altogether, there were no contradictions found in past literature, but we have extended the description of the Architecture principle with new characteristics.

Analyzing the contribution of individual architecture principles without looking at the set of principles is of no use. An architecture principle is only effective in combination with other architecture principles. We therefore also described the characteristics of the architecture principle set: ‘Classification’, ‘Quality’ and ‘Meta data’. We described the characteristics of both an architecture principle and the architecture principle set into a framework also related to entities in its environment.

Conclusions for Measuring Principles.

To answer Sub question 2, we had to describe a measurement instrument and test it in a real-life case study. Based on that experiences from practice we can conclude that the measurement instrument is fit for purpose. Although the instrument can be improved with extra characteristics and attributes. These extensions are ‘degree of acceptance’ and ‘preconditions fulfilled’ for describing the individual principle. Besides the characteristic ‘prerequisites’, including the attributes ‘precondition’, and ‘basic assumption’ can be added for the architecture principle set. Although we know that we tested the instrument with one case only, we are beyond doubt that with these add-ons the model has added value in measuring architecture principles and in measuring related architecture principle sets.

Secondly, we had to investigate how to use this measurement instrument in practice. We defined a three-step method to collect, analyze and measure the architecture principle (set). We used this measurement method in our case study, with a description of the architecture principles and principles sets for the TDi case as a result. We have the opinion that the method yields reliable and valid results, although we discovered in this case that more information on the requirements and the individual principles, would strengthen the results of the case study.

6.3 Further Research

In summary, the conclusions and limitations combined confirm that it is feasible to enlarge the number of tests of architecture principle measurements with more cases. With more cases we are able to test the ability to compare architecture principles and principle sets between case studies.

When carrying out new cases, we would prefer the number of cases to be as large as possible. However, substantial scaling up the number of cases requires automatic processing of the data. In our opinion, this is only to a limited extent possible, as compliance with architecture principles always requires some sort of human interpretation.

Secondly, we can use the new case studies to test the completeness and coherence of the measurement instrument and method. We need to investigate whether or not the vision of using architecture principle is a characteristic in itself and we also have to see how we can elaborate on the essential requirements. So, new case studies will give new insights in the use of the instrument and method and help in optimizing them both.