1 Introduction

Software architecture provides abstract and top-level models of a system as the software production line. It also plays a fundamental role in overcoming problems arising from the complexity and size of software projects. As technology, growth definitely increases the size and complexity of the software, the role of the architecture becomes more important in this process [17]. The most challenging problems in designing software architecture are related to the cost and time spent in the design and analysis. In addition, the architect’s experience is also an important consideration in this regard. The use of design methods and architectural styles were introduced as a means to help software developers overcome these problems by reducing risk, cost and providing guidelines for less experienced architects. Although using architectural styles in designing the software system guarantees that some quality features of the existing style and architectural style may not satisfy the architectural requirements of some complex systems. To circumvent this issue, a combination of two or more different styles, referred to as heterogeneous style, can be used [35].

Since architecture is the first element in the software production cycle associated with the analysis of the quality attribute, providing a suitable architecture has a significant impact on meeting these kinds of requirements and the ultimate success of the system. On the other hand, the distribution and development of software systems, as well as the complexity of the full range of dependent requirements, make designing an efficient software architecture important, so much that the success or failure of software projects are largely determined by the design of efficient software architecture. Hence, the evaluation and verification of software architecture resulting from software architecture evaluation are basic requirements [5].

Due to the high costs of changes, the software architecture evaluation starts simultaneously with the design process and continues until its end. Predicting system quality, meeting its quality attributes and identifying potential risks in the design, are the goals of software architecture evaluation. Numerous methods have been proposed for software architecture evaluation, which differ in terms of the business objectives, desired quality attributes, techniques, activities and the relationship between stakeholders [10]. For example, from a measurement-theory point of view, having quantitative methods can add amounts of computational overhead and evaluation inaccuracies. Most of these inaccuracies are due to using a logic different from that used by the domain expert, which may cause many structural and semantic problems. Therefore, methods based on fuzzy logic could be used as an appropriate solution to improve the performance and precision of architecture evaluation. Moreover, in order to prevent making decisions based on incomplete criteria, taking advantage of information such as working platforms, application types, available quantitative records and many other effective parameters as complementary criteria can improve the decision-making process. The fact that software producers must consider many often overlapping and conflicting aspects and criteria turn the architecture selection process into complicated and multi-criteria enterprises. However, this complexity and vagueness arise much more in heterogeneous architectural styles, making the problem of their selection and evaluation more complicated.

In this study, we seek to propose an innovated method to evaluate different architectural styles based on their specific quality properties. To do so, we use a flexible multi-criteria decision-making approach, namely the hierarchical analysis process, as well as fuzzy logic in order to remove the vagueness and increase the precision in the comparison and evaluation of candidate architectural styles. The most important advantages of this method are accurate evaluation and the ability to extend the evaluation to aid in choosing among complicated heterogeneous architectural styles in a rational way. First, we introduce the required quality attributes and then we describe fuzzy logic. Next, the fuzzy hierarchical analysis process technique is described and after that, the proposed method is explained in detail. Finally, we present our conclusions in the last section.

2 Related work

In this section, we first discuss architecture evaluation and its various types, and then we shall deal with related works in the field of multi-criteria problems for architecture style selection.

There are different methods and structures to evaluate architectures, each adaptable to particular purposes. Current evaluation methods could be classified into two categories: quantitative and qualitative. To use special software architecture, research typically provides us with some guidelines by which qualitative attributes could be inferred [23]. Quantitative description is a means to compare the level of satisfying different quality attributes in different domain spaces. In other words, quantitative evaluations represent the significance level of described benefits and capabilities. The qualitative evaluation of architecture styles is, in fact, the combination of each architectural style with their inherent special quality attributes [23, 44]. A combination of the mentioned methods can be used. In the following, qualitative and quantitative methods are discussed in more detail.

In qualitative evaluation, questioning techniques are the most common methods used to evaluate architecture with every quality. Measuring techniques can be used to answer special questions and address special software qualities (e.g., performance or scalability); they are not as extensive as questioning techniques. There exist three types of questioning techniques, namely scenarios, questionnaires and checklists. There are some differences among these in terms of their applicability, but all have the same goal, namely to enhance our understanding of the degree of consistency between architecture and required quality attributes. In the scenario technique, quality attributes (e.g., maintainability, security, performance and reliability) become meaningful in the context. Scenarios are descriptive tools used to evaluate quality attributes in a context. In the questionnaire technique, a list of general and relatively open questions is provided which are applicable to all architectures [1]. Some questions are about architecture creation and documentation and some of them focus on descriptive details of architecture. In the checklist technique, a set of very detailed questions are used which have been obtained from several experiences about some similar domains. These questions focus on special quality attributes of a system. A checklist is helpful to keep a balanced focus on all areas of the system [29].

In quantitative evaluation, measuring techniques lead to quantitative results. These techniques answer the questions of the measurement team concerning the quality attributes of architecture instead of providing us with ways to design questions. These techniques are more complete and mature than questioning techniques. Metrics are quantitative interpretations based on particular observable measurements on the architecture, such as fan-in/fan-out of components [42]. In measuring techniques, the evaluation process should focus not only on the results of the metric but also on the assumptions underlying the use of the technique [1]. Constructing a prototype or performing a simulation of a system may help to establish and clarify architecture, but it is often costly. In other words, they are often a part of the development process.

Several other methods are used to evaluate software architecture. Scenario-based methods like SAAM (Scenario-based Architecture Analysis Method) [21, 38] were introduced in 1993 to provide a better understanding of the architecture and to prove that architecture deals with functional, in addition to qualitative requirements [1, 25]. This method was developed to ensure the compatibility of the architectural assumptions with the system’s desirable attributes, understand risks associated with the architecture and potential conflicts among different quality attributes. Other methods like ATAM (Architecture Tradeoff Analysis Method) [22, 38] deal with qualitative attributes and the balance between the attributes. Such methods evaluate the architecture to show to what extent specific quality objectives are met. They also focus on the contrast between qualitative attributes and their effects on each other. ATAM is based on SAAM.

There are other cost-based methods such as CBAM (Cost-Benefit Analysis Method) [38, 37], CBAM, which contrary to the other two methods, has created a bridge between software development and economic issues. In this method, cost (especially budget) is considered as a qualitative attribute. With this strategy, the interaction between economic issues and other qualitative architectural concepts is measured. ALMA (Architecture-Level Analysis Method) [6, 25] uses maintenance cost estimation and risk assessment to analyze the variability of the system. FAAM (Family-Architecture Analysis Method) [47] is a method to measure the interoperability and expandability of a family of information systems.

Another category of methods focuses on formal methods. Formal methods include different types of architectural language descriptions, which are symbolic languages for expressing and describing the architecture of software systems [31]. In recent years, many studies have been conducted on formal architectural languages descriptions and the result is the creation of different architectural languages like Aesop [15], C2SADL [30], Darwin [27], MetaH [7], Rapide [26], UniCon [43] and ACME [16].

Now, we shall review the work related to multi-criteria problems for architecture style selection. (Tan et al. 2010) discusses the Choquet integral, which is a fuzzy operator enabling the selection of suitable architectural styles. The integral is a general tool in multi-criteria decision-making processes. Moaven et al. [33, 34] introduce a Decision Support System (DSS) which makes use of a fuzzy concept to represent concepts of quality attributes more precisely and efficiently. The system takes advantage of fuzzy inference support decisions of software architects. Moaven et al. [35] introduces a framework and a technique to provide an environment for evaluating heterogeneous architecture styles of a software system. The framework exploits the quantity attributes obtained during their use in order to increase the evaluation validity. Babu et al. [4] explain the ANP approach which is used to represent the various criteria and concepts of quality attributes more precisely and efficiently. One of the goals of this approach is to optimize software architecture design. Galster et al. [14] have introduced SYSAS as a method for systematic and traceable selection of architectural styles. This method is based on the characteristics of basic architectural elements relevant to the developer, and also those of the target system that are visible to the end-user.‏ Moaven et al. [33, 34] explain that it is possible to use fuzzy inference to support the decisions made by software architects. The main objective of the study is to get the most out of the properties of styles. An integrated approach of SPL with architecture style selection and component-based design is presented in Zaki et al. [50]. This approach seeks to select the best architectural style using the fuzzy analytic hierarchy process. Dwivedi and Rath [13] present a library of styles to select an appropriate style for software systems. This study uses formal modeling language alloy for reusing and extensible modeling of a complex and highly distributed components. Tahmasebipour and Babamir [45] have evaluated the interaction between architectural tactics and architectural styles. By taking advantage of a new ranking scheme for the architectural styles, the best architectural style may be obtained for every individual quality attribute or their combinations. An intuition-based fuzzy Choquet integral is proposed in Tan and Chen [46] for multiple-criteria decision making. In fact, this integral takes account of interactions among the decision-making criteria. A method for the selection of the best software architecture styles, called SSAS, was suggested in Babu et al. [3] for improved selection methodology, reflecting interdependencies among the evaluation criteria using the analytic network process within a zero–one goal programming model. A meta-model called ADUAK is introduced in Dhaya and Zayaraz [12] for fuzzy-based quantitative architectural evaluation. The main objective of this study is to improve the efficiency of the architectural design process through architectural knowledge management (AKM) support. Saaty [39] explains that many decision problems cannot be structured hierarchically because they involve interaction and dependence of higher-level elements on a lower-level element. The AHP and ANP are used to solve the problem of independence on alternatives or criteria and the problem of dependence among alternatives or criteria, respectively [18].

In the context of reviewing the evaluation methods, Abrahão and Insfran [2] are among the most insightful studies. In this work, the quality-driven architecture derivation and improvement (QUADAI) method is compared against the well-known architecture tradeoff analysis method (ATAM), showing that QUADAI obtains better results than ATAM in terms of effectiveness, perceived usefulness and intention to use.

Some research is done in Mahdavi-Hezavehi et al. [28] to demonstrate which architecture handles qualitative attributes to help the researchers know how to score and weight these attributes in self-adaptable systems. Some other studies have been conducted in the context of qualitative attributes of software product lines that may be generalizable in the architecture area [36]. This paper presents a systematic literature review with the objective of identifying and interpreting all studies from 1996 to 2010 that present quality attributes and measures for SPL. These attributes and measures have been classified using a set of criteria that includes the life cycle phase in which the measures are applied, the corresponding quality characteristics and their support for specific SPL characteristics. Moaven et al. [32] and Dasanayake et al. [11] provide other approaches that are useful for decision making about choosing architecture and its styles. In addition, Moaven et al. [33, 34] focuses on the decision-making procedure and using knowledge that could be useful in this selection.

In all mentioned methods, there are important challenges that remain open, like the overlapping of quality attributes or the contradiction between them. This is especially relevant when there is a need for selecting a suitable architectural style for the reconstruction of a legacy system. In selecting styles, due to multi-criteria selection and uncertainty (which depends on the experts and the system architecture), special attention must be paid for integrating system architecture experience and involving mechanisms able to learn. Using numerical concepts and binary thinking cannot reflect the real attributes of a style. Problems like these cannot be solved completely and always there are mechanisms for improving accuracy and effectiveness of choosing a suitable style. In this paper, we propose a method for covering the existing challenges.

3 Quality attributes

Quality attributes are a part of the requirements of application programs which express the concerns of users and other stakeholders to access the functional requirements of the systems [17]. Several efforts have been made to provide models (software quality models) for the description and classification of quality attributes. The ISO/IEC quality model is one of these software quality models which has had more coverage and covers more aspects of software quality. This model was presented in 1991 by the International Organization for Standardization (ISO) to meet the need of the software industry to develop a standardized software evaluation. It was modified again in 2001 [20]. In the ISO/IEC model, as shown in Fig. 1, software quality is described based on six main quality attributes, each of which have several sub-quality attributes. These attributes are as follows:

Fig. 1
figure 1

ISO/IEC quality model

Full size image

  • Functionality: The ability of the software product to provide functional requirements in a specific situation is called capability. Sub-features of this feature are suitability, accuracy, interoperability and security.

  • Reliability: The ability of the software product to remain at a certain level of performance when used in certain circumstances is called reliability and its sub-features are maturity, fault tolerance and recoverability.

  • Usability: This feature describes how much a software product can be learned, attractive and practical in specific usage conditions. Understandability, learnability and operability are sub-features of this characteristic.

  • Efficiency: Efficiency demonstrates the capability of the software product to provide appropriate performance, related to the amount of resources used in certain circumstances [20]. In other words, efficiency is the required system response time for a certain number of events in a given time interval. It shows how well the interactions between system components are conducted. It seeks to reduce time waste and response time in the system. The sub-features of efficiency are resource utilization and time behavior.

  • Maintainability: The ability of the software product to apply changes and modifications. These changes are analysis capability, variability, stability and testability.

  • Portability: This attribute expresses the ability to run the system under various computing environments, which include hardware, software and organizational environments. In order to evaluate portability adaption, installation and co-existence, sub-features should be examined.

4 Fuzzy logic

Fuzzy sets and fuzzy logic are powerful mathematical-based tools for modeling uncertain systems which also facilitate hypothetical reasoning in decision making where there is partial and inaccurate information. The main characteristic of a fuzzy set is its ability to represent vague information [49]. Given that triangular fuzzy numbers demonstrated in Fig. 2 lead to a simple process and simple computations in displaying data in a fuzzy environment, employing these numbers is suitable in most applications. The function of a triangular fuzzy number with the M = (l, m, u) triple is defined with respect to membership function \( \mu_{A} (x) \) as follows:

Fig. 2
figure 2

Triangular numbers M1 and M2

Full size image
$$ \mu_{A} \left( x \right) = \left\{ {\begin{array}{*{20}l} 0 \hfill & {x < l;} \hfill \\ {\frac{x - l}{m - l}} \hfill & {l \le x \le m;} \hfill \\ {\frac{m - x}{u - m}} \hfill & {m \le x \le u;} \hfill \\ 0 \hfill & {x > u;} \hfill \\ \end{array} } \right. $$
(1)

Consider two triangular numbers \( M_{1} = (l_{1} ,m_{1} ,u_{1} ) \) and  \( M_{2} = (l_{2} ,m_{2} ,u_{2} ) \) which are plotted. Different operations are defined on triangular fuzzy numbers; the most important of them are as follows:

$$ M_{1} + M_{2} = \left( {l_{1} + l_{2} ,m_{1} + m_{2} ,u_{1} + u_{2} } \right) $$
(2)
$$ M_{1} *M_{2} = \left( {l_{1} *l_{2} ,m_{1} *m_{2} ,u_{1} *u_{2} } \right) $$
(3)
$$ M_{1}^{ - 1} = \left( {\frac{1}{{u_{1} }},\frac{1}{{m_{1} }},\frac{1}{{l_{1} }}} \right).M_{2}^{ - 1} = \left( {\frac{1}{{u_{2} }},\frac{1}{{m_{2} }},\frac{1}{{l_{2} }}} \right) $$
(4)

5 The fuzzy analytic hierarchy process (FAHP)

The analytic hierarchy process (AHP) is a multi-criteria decision-making method which selects the best option from all candidates [41] based on several criteria, which may be different or even in conflict with each other. In this method, after decomposing the problem by hand and constructing the hierarchical tree, completing a pairwise comparison between the available criteria in the same level, a certain number is assigned to each option to rank the options and allow the selection of the best one [40]. Based on AHP, the essential steps to solve complex and multi-criteria problems are:

  1. 1.

    Defining decision-making criteria in the form of a hierarchical structure so that the goal is placed in the topmost level, criteria and sub-criteria are in the middle levels, and the candidate solutions and options are in the lowest level.

  2. 2.

    Defining computing criteria, sub-criteria and replacements weights based on the relative importance of each member from its higher level.

  3. 3.

    Ranking the options.

In spite of AHP efficiency in the management of quantitative and qualitative criteria, the lack of specific standards for measuring and representing the numerical values of the qualitative criteria and ambiguity and uncertainty in relative comparisons leads to uncertainty in the ranking of options [9]. As a solution to this problem, a combination of analytic hierarchy process and fuzzy logic was first introduced in Van Laarhoven and Pedrycz [24] in order to adapt better the process to reality, to consider uncertainty and inaccuracy issues into account, and finally to increase precision in ranking options [24]. In this method, triangular fuzzy numbers and logarithmic least-squares were used to compute proportional weight. In addition, an extension to this method is FAHP, which was proposed by Chang et al. [9] to reduce computational overhead compared to the method. The development stages of FAHP based on the extended analysis method are as follows [8]:

  1. 1.

    Creation of the Fuzzy Pairwise Comparison Matrix. In this stage, a pairwise comparison matrix is determined based on triangular fuzzy numbers, and by decision-makers. In this case, if mij is the real value obtained from the comparison of two criteria i and j, the upper and lower bounds of the triangular fuzzy number, based on the fuzzification degree (δ), are determined by \( (m_{ij} - \delta ,m_{ij} ,m_{ij} + \delta ) \). So, the element mij of the matrix can be specified by the triangular fuzzy number, which the subscripts i and j refer to the row and column, respectively. Table 1 in Sect. 6 is based on this pairwise comparison. From Eq. 5 to 7, let \( G = \{ g_{1} , g_{2} , \ldots , g_{n} \} \) be a set of criteria, where n is the number of criteria.

    Table 1 Allocated fuzzy values to any conversational values
    Full size table
  2. 2.

    Calculation of the Fuzzy Composite Extension Si (the value of fuzzy synthetic extent with respect to the ith object). In fact for estimating of the weight values for each criterion and for each alternative with reference to a given criteria, Si is calculated for each row of the pairwise comparison matrix according to the following equations:

    $$ S_{i} = \mathop \sum \limits_{j = 1}^{m} M_{{g_{i} }}^{j} .\left[ {\mathop \sum \limits_{i = 1}^{n} \mathop \sum \limits_{j = 1}^{m} M_{{g_{i} }}^{j} } \right]^{ - 1} $$
    (5)
    $$ \mathop \sum \limits_{j = 1}^{m} M_{{g_{i} }}^{j} = \left[ {\mathop \sum \limits_{j = 1}^{m} l_{ij} ,\mathop \sum \limits_{j = 1}^{m} m_{ij} ,\mathop \sum \limits_{j = 1}^{m} u_{ij} } \right] $$
    (6)
    $$ \mathop \sum \limits_{i = 1}^{n} \mathop \sum \limits_{j = 1}^{m} M_{{g_{i} }}^{j} = \left[ {\mathop \sum \limits_{i = 1}^{n} l_{ij} ,\mathop \sum \limits_{i = 1}^{n} m_{ij} ,\mathop \sum \limits_{i = 1}^{n} u_{ij} } \right] $$
    (7)

    Related parameters in these equations \( (l, m, u) \) are positive numbers and demonstrated in Fig. 2. Besides G is the goal set.

  3. 3.

    Calculation of the Priority Degree (Feasibility) of Si over Sk. The degree of possibility for convex fuzzy number M to be greater than the number of k fuzzy numbers Mi (\( i = 1, 2, \ldots , k \)) is given by the use of the operations max and min. Here the degree of importance of each Si is calculated with respect to each other. In general, if M1 and M2 are triangular fuzzy numbers, in order to find the degree of importance of M1 over M2 or compare M1 and M2, where (M1 ≥ M2) or \( (M_{2} \ge M_{1} ) \), the values of V are needed and are calculated by the following equation:

    $$ V\left( {M_{2} \ge M_{1} } \right) = \left\{ {\begin{array}{*{20}l} 1 \hfill & {{\text{if}}\quad m_{2} \ge m_{1} ;} \hfill \\ 0 \hfill & {{\text{if}}\quad l_{1} \ge u_{2} ;} \hfill \\ {\frac{{l_{1} - u_{2} }}{{\left( {m_{2} - u_{2} } \right) - (m_{1} - l_{1} )}}, } \hfill & {\text{otherwise}} \hfill \\ \end{array} } \right. $$
    (8)

    As it is clear, the degree of importance of a certain triangular fuzzy number compared to the other triangular fuzzy numbers is calculated by:

    $$ V\left( {M \ge M_{1} ,M_{2} , \ldots ,M_{k} } \right) = V\left( {M \ge M_{1} ),V\left( {M \ge M_{2} } \right), \ldots ,V(M \ge M_{k} } \right) = \hbox{min} V(M \ge M_{i} ),i = 1,2, \ldots ,k $$
    (9)
  4. 4.

    Calculation weights of the criteria. The criteria’s weight in the pairwise comparison matrix is calculated using the following equation:

    $$ W^{{\prime }} \left( {x_{i} } \right) = {\text{Min}}\left\{ {V\left( {S_{i} \ge S_{k} } \right)} \right\},k = 1,2, \ldots , n\quad {\text{and}}\quad k \ne i $$
    (10)

    Therefore, the criteria weight vector is:

    $$ W^{{\prime }} \left( x \right) = [W^{{\prime }} \left( {x_{1} } \right),W^{{\prime }} \left( {x_{2} } \right), \ldots ,W^{{\prime }} \left( {x_{n} } \right)]^{T} $$
    (11)
  5. 5.

    Weight Normalization. Results of the previous step are normalized as follows:

    $$ W_{i} = \frac{{W_{1}^{{\prime }} }}{{\mathop \sum \nolimits W_{i}^{{\prime }} }} $$
    (12)

6 Proposed method

In Moaven et al. [33, 34], architecture styles selection was introduced as a multiple-criteria problem and resolved it using a fuzzy integral. In this paper, the method that proposed helps software architects and developers to evaluate candidate architectures or architectural styles based on the system goals and stakeholder viewpoints and select the most useful architectural style with respect to their particular and required goals. To this aim, first, the required hierarchical structure is defined based on ISO/IEC software quality model; next, the fuzzy variables are used to explain the hypothetical reasoning about the relative importance of different criteria and options. Finally, candidate architectural styles are ranked by employing the FAHP method in order to select the best option. These steps are at length described and analyzed in the remainder of this section. Furthermore, the process of attributes weighting and method evaluation are discussed in the following.

6.1 Identifying criteria and sub-criteria

In order to choose an architectural style, we encounter different quality attributes that are viewed as decision-making criteria. Software hierarchical quality models, such as ISO/IEC, can be used to classify the structure of the quality attributes (criteria) and sub-criteria of the architectural style’s selection problem. The hierarchical structure of this problem would then contain 6 criteria and 21 sub-criteria based on the ISO/IEC software quality model.

6.2 Constructing the hierarchical structure

Here, the architectural style selection problem is decomposed and described in a simple format according to the human mind. Also, based on a hierarchical structure, we see that the problem, comprising four levels:

  • Goal: evaluating architectural style.

  • Criterion: software quality attributes.

  • Sub-criterion: secondary quality attributes corresponding to each quality attribute.

  • Options: architectural styles.

To create a hierarchical structure, the decision-making goal, which is an evaluation of architectural style, is placed in the root of the tree. In the next level, the required quality attributes are appended to the tree as the branches of the root and the impact of each attribute in fulfilling the goal is considered as the weight of the edge. At the third level, the secondary attributes affecting each quality attribute are considered as the children of each attribute, with the impact of each attribute constituting the weight of the edge connecting them. Figure 3 illustrates the hierarchical structure for architectural style evaluation based on the ISO/IEC quality model.

Fig. 3
figure 3

Hierarchical structure for architectural style evaluation

Full size image

6.3 Creating fuzzy decision-making matrices

After constructing the hierarchical structure for the problem, a pairwise comparison for the criteria, sub-criteria and options is carried out, and finally, the fuzzy decision-making matrices are developed. The collection of viewpoints of decision-makers and experts are required to define the pairwise comparisons among the members at any level of this hierarchical structure. Employing questionnaires is one of the ways of gathering information. In the process of selecting the most appropriate architectural style, a number of questionnaires should be prepared to be used by decision-makers and experts. These questionnaires compare the criteria, sub-criteria and options for each two of them and according to the existing upper-level members pairwisely and also explain their relative importance with respect to one another using items (such as the same importance, a bit more important, more important, very important and the most important). After this step, the conversational results of these comparisons are replaced with fuzzy numbers based on the prevalent pattern. For example, as shown in Table 1, in this method, the conversational values are replaced with triangular fuzzy values.

After collecting experts’ viewpoints, the decision fuzzy matrices are created. Each member of the fuzzy decision-making matrix is indicated by a triple vector defining the comparison triangular fuzzy number between i and j. We suppose that is a pairwise comparison between i and j from the viewpoint of decision-maker k, denoted by Dijk, as shown in Fig. 2. Then, the triangular fuzzy numbers for any pairwise comparisons are defined based on Eqs. (13) to (15) as follows.

$$ l_{ij} = \min_{k}\,(D_{ijk} ) $$
(13)
$$ u_{ij} = \max_{k} \left( {D_{ijk} } \right) $$
(14)
$$ m_{ij} = \sqrt[k]{{\mathop \prod \limits_{k = 1}^{k} D_{ijk} }} $$
(15)

These equations are used for solving equations such as (6) and (7).

6.4 Identifying criteria weight

The relative importance of each quality attribute depends on the conditions of the system and the type of the system architecture to be designed or evaluated. In other words, criteria weights are different for each system and depend on the problem domain and the importance of the quality attributes in the system. For example, portability may be the criterion with the highest degree of importance in one system, whereas it has the lowest degree of importance in another. Therefore, to explain the relative importance and make a pairwise comparison among the criteria, the evaluation process must be focused on a particular area, and the system or the proportional weights of different criteria must also be considered as parameters.

In this study, to evaluate the proposed method, a special domain is supposed for the problem at hand. Thus, the decision-making team fills out the questionnaires based on the selected domain and uses this method to select the most appropriate style among candidate architectural styles. Suppose that we need to choose the most appropriate architectural style for a system with the following characteristics. The instance of this domain is the embedded system as a broker gateway between the banking system and different type of customer systems. Because of the platform, banking security is not the issue of our case study in this paper and it handled in another layer of general systems. The important things of this especial system are as below:

  • The system must have high performance.

  • System flexibility is the second priority in terms of importance ranking.

  • Availability and fault tolerance are important factors in the system.

  • The data transfer paths must be determined.

  • The cost of planning, designing and other activities are also important.

Based on these assumptions, the fuzzy decision-making matrix used for pairwise comparison among the criteria is defined according to Table 2.

Table 2 Fuzzy decision-making matrix for comparing pair criteria
Full size table

The relative weights of criteria are calculated according to Table 2 and the FAHP method as follows:

$$ \begin{aligned} & S_{c1} = (8.50,11,13.50) \otimes (1/50.83,1/41.30,1/33.17) = (0.17,0.27,0.14) \\ & S_{c2} = (5.90,7.50,9.17) \otimes (1/50.83,1/41.30,1/33.17) = (0.12,0.18,0.28) \\ & S_{c3} = (6.83,8.40,10) \otimes (1/50.83,1/41.30,1/33.17) = (0.13,0.2,0.3) \\ & S_{c4} = (4.40,5.33,6.67) \otimes (1/50.83,1/41.30,1/33.17) = (0.09,0.13,0.2) \\ & S_{c5} = (4.30,5.17,6.33) \otimes (1/50.83,1/41.30,1/33.17) = (0.08,0.13,0.19) \\ & S_{c6} = (3.23,3.90,5.17) \otimes (1/50.83,1/41.30,1/33.17) = (0.06,0.09,0.16) \\ \end{aligned} $$

The degree of importance of the values was calculated in the previous step is defined with respect to other values.

$$ \begin{aligned} & V(S_{c1} \ge S_{c2} ) = 1,\quad V(S_{c1} \ge S_{c3} ) = 1,\quad V(S_{c1} \ge S_{c4} ) = 1, \\ & V(S_{c1} \ge S_{c5} ) = 1,\quad V(S_{c1} \ge S_{c6} ) = 1,\quad V(S_{c2} \ge S_{c1} ) = 0.56, \\ & V(S_{c2} \ge S_{c3} ) = 0.87,\quad V(S_{c2} \ge S_{c4} ) = 1,\quad V(S_{c2} \ge S_{c5} ) = 1, \\ & V(S_{c2} \ge S_{c6} ) = 1,\quad V(S_{c3} \ge S_{c1} ) = 0.68,\quad V(S_{c3} \ge S_{c2} ) = 1, \\ & V(S_{c3} \ge S_{c4} ) = 1,\quad V(S_{c3} \ge S_{c5} ) = 1,\quad V(S_{c3} \ge S_{c6} ) = 1, \\ & V(S_{c4} \ge S_{c1} ) = 0.2,\quad V(S_{c4} \ge S_{c2} ) = 0.62,\quad V(S_{c4} \ge S_{c3} ) = 0.47, \\ & V(S_{c4} \ge S_{c5} ) = 1,\quad V(S_{c4} \ge S_{c6} ) = 1,\quad V(S_{c5} \ge S_{c1} ) = 0.14, \\ & V(S_{c5} \ge S_{c2} ) = 0.57,\quad V(S_{c5} \ge S_{c3} ) = 0.42,\quad V(S_{c5} \ge S_{c4} ) = 1, \\ & V(S_{c5} \ge S_{c6} ) = 1,\quad V(S_{c6} \ge S_{c1} ) = 0.04,\quad V(S_{c6} \ge S_{c2} ) = 0.31, \\ & V(S_{c6} \ge S_{c3} ) = 0.16,\quad V(S_{c6} \ge S_{c4} ) = 0.67,\quad V(S_{c6} \ge S_{c5} ) = 0.70 \\ \end{aligned} $$
$$ \begin{aligned} W^{{\prime }} \left( {C_{1} } \right) = {\text{Min}}\left\{ {1, 1, 1, 1, 1} \right\} = 1 \hfill \\ W^{{\prime }} \left( {C_{2} } \right) = {\text{Min}}\left\{ {0.56, 0.87, 1, 1, 1} \right\} = 0.56 \hfill \\ W^{{\prime }} \left( {C_{3} } \right) = {\text{Min}}\left\{ {0.68, 11, 1, 1} \right\} = 0.68 \hfill \\ W^{{\prime }} \left( {C_{4} } \right) = {\text{Min}}\left\{ {0.2, 0.62, 0.47, 1, 1} \right\} = 0.2 \hfill \\ W^{{\prime }} \left( {C_{5} } \right) = {\text{Min}}\left\{ {0.14, 0.57, 0.42, 1, 1} \right\} = 0.14 \hfill \\ W^{{\prime }} \left( {C_{6} } \right) = {\text{Min}}\left\{ {0.04, 0.31, 0.16, 0.67, 0.7} \right\} = 0.04 \hfill \\ \end{aligned} $$

Based on the values obtained in the previous step, the following criteria weight vector is defined:

$$ W^{{\prime }} = (1,0.56,0.68,0.2,0.14,0.04) $$

Finally, the criteria weights are normalized as follows to create the structure tree of architectural style evaluation in Fig. 4.

Fig. 4
figure 4

Structure tree and weigh of primary criteria in architectural styles evaluation

Full size image
$$ W = (0.38,0.21,0.26,0.08,0.05,0.02). $$

6.5 Calculating sub-criteria and style weight or proportional priority

In this step, by conducting a pairwise comparison between the sub-criteria of each quality property and performing computations similar to what we observed in the previous step the weight of all the sub-criteria are calculated according to their corresponding criterion as shown in Fig. 5. After calculating the proportional weight of each sub-criterion, the proportional weight of any architectural style can be calculated based on any sub-criterion just as in the previous method.

Fig. 5
figure 5

Tree structural and proportional weight of sub-criteria in architectural styles evaluation problem

Full size image

6.6 Style scoring

In the final step, the architectural styles are scored based on both final weights of sub-criteria and relative weights of architectural styles. In more detail, the weight of any criteria must be multiplied by the relative weight of its parent (the weight of the main criterion related to the sub-criterion). Figure 5 illustrates the final weights of the criteria and sub-criteria. Finally, the weight or score of any architectural style is calculated using the following equation, which is an aggregation of the relative weights of all the styles in this example multiplied by the final weight of the corresponding sub-criterion:

$$ OP\left( {Q_{i} } \right) = Y_{i} $$
(16)
$$ OP\left( {P_{j} } \right) = \mathop \sum \limits_{i}^{m} X_{ij} \times OP\left( {Q_{i} } \right) = \mathop \sum \limits_{i}^{m} X_{ij} \times Y_{i} $$
(17)
$$ OP\left( {S_{k} } \right) = \mathop \sum \limits_{j}^{n} Z_{jk} \times OP\left( {P_{j} } \right) = \mathop \sum \limits_{j}^{n} Z_{jk} \times Y_{j} $$
(18)

where n is the number of sub-criteria, Zjk is the relative weight of style Sk compared to the sub-criterion Pj, and Yj is the final weight of sub-criterion Pj. According to Table 7, Zjk is calculated. The knowledge and experience of an expert in the questionnaires determine the behaviors of architectural styles in comparison with pairwise comparison method in every quality attribute and the computations will be done accordingly.

Table 3 shows the calculated score of all styles in this example. Therefore, based on this ranking, the most appropriate architectural styles in this special problem are Pipe and Filter, Client–Server, Layered, C2, Object Oriented, Blackboard and Storage styles, respectively.

Table 3 Privileging architectural styles
Full size table

Table 3 is calculated by the 18th formula which is shown below:

$$ OP\left( {C_{2} } \right) = 0.162 \times OP(Portability) + 0.1893 \times OP(Usability) + 0.162 \times OP(Reliability) + 0.1982 \times OP(Efficiency) + 0.0173 \times OP(Maintainability) + 0.271 \times OP(Functionality) $$

As in the previous step, the Yj is calculated for each quality attribute in the problem, Fig. 6 in this subsection helps clarify this step, which is taken from Fig. 3.

Fig. 6
figure 6

The instance of tree structural and proportional weight

Full size image

The value of other architectural styles is calculated in Table 3.

6.7 Weight and scoring

In the proposed method, we need to rank and compare styles and qualitative attributes and their sub-attributes. There are two general methods that could be used for this purpose: by using background and expert knowledge. The latter method utilizes questionnaires and methods to facilitate these issues and help to perform the work more accurate. We have used questionnaire forms and saved the results in a hierarchical format [32], which will be used as an example of this work.

In the following, Sect. 6.7.1 as mentioned before, we present a few examples of the forms (not all of them) which the required comparisons are derived from them. In fact, we provide some questionnaires (like the papers below) for all the quality attributes, its characteristics and for the comparison of both architectural styles.

6.7.1 Example of quality attribute sub-criteria questionnaires

The proposed method could be used in different models with respect to various quality attributes which leads us to give several samples (examples).

In Tables 4, 5, and 6, qualitative attributes like correctness, usability and flexibility are evaluated at the level of sub-attributes.

Table 4 Correctness and its sub-attributes
Full size table
Table 5 Usability and its sub-attributes
Full size table
Table 6 Flexibility and its sub-attributes
Full size table

6.7.2 Example of questionnaires’ style comparisons

In this subsection, a sample of common questionnaires of styles has been presented. In fact, in Table 7, the questionnaires have provided an implicit comparison between architectural styles based on architects’ opinions and those of the system experts and in Table 8, the three categories of 30 experts (answered the questionnaires) are shown.

Table 7 The questionnaires for taking expert’s opinions
Full size table
Table 8 The questionnaires for taking expert’s opinions
Full size table

The results obtained by this method can help us find the required coefficients in various formulas and approaches. In addition, they can be used to extract reliable results for evaluating our output with regard to a specific problem. For example, for Broker, C2, Object Oriented, Layered, Blackboard, Repository and Pipe & Filter styles, qualitative attributes like Correctness, Usability and Flexibility in the scope of information systems (backup and restore) the following results (Table 9) were obtained after averaging the comments provided by 30 experts.

Table 9 Coefficients of each qualitative attribute
Full size table

For using FAHP/AHP methods and also weighting each quality attribute related to the architectural styles, the different existing references could be helpful [1, 2, 4, 7, 9, 12, 19, 29, 44, 50]. The proposed methods divide the problem into small part for comparing and weighting but the user should be familiar with the architectural styles and their specification.

7 Evaluation

Regarding the evaluation of the proposed methods in the previous step, we chose a real case (as a case study) and in this regard, the proposed method was described which its results can be considered as a reference to evaluate the proposed methods. Also, based on the existing methods in this context (mentioned in the related work), we examine the analysis of the proposed method.

Actually, to evaluate our method, we have examined several studies. As our method is based on an expert’s opinion and questionnaires, and our goal is to analyze their influences. In fact, our method is evaluated in different ways such as evaluation by experts, the Fuzzy-based AHP method and analytical evaluation based on previous studies.

7.1 Advantages of the proposed method

We discuss the advantages of the proposed method based on KPI in Table 10 and it is compared with the other methods from the literature. This comparison is based on eight quality attributes: Reusability, Scalability, Understandability, Usability, Completeness, Cost, Accuracy, Accountability. Tables 10 and 11 evaluate the proposed method from the perspective of each quality attribute, as presented in Hämäläinen [19], Wohlin et al. [48].

Table 10 Evaluation of the KPI’s
Full size table
Table 11 Comparison of the proposed method with other methods
Full size table

According to the related work section, we select four categories from architectural evaluation methods, which we review and compare with the proposed method. Table 11 presents the results of this comparison. “R” indicates that this method reduces the quality attribute and has a lower performance compared to other methods. “-” means the better method cannot be identified. “I” means that the proposed method improves the quality attribute.

As seen in Table 11, we survey the different viewpoint of the architectural evaluation method. In some of them there may be certain and customized cases which are difficult to compare against our method. In the comparison shown in Table 11, our method exhibits higher reusability compared to the other methods except for scenario-based methods; this is due to the fact that such methods, the scenario designer can plan it to be reusable, and this decision solely rests with the architect. As shown in the evaluation description table, for Scalability, our method provides sequential evaluation because of the quantitative capabilities and measurement methods based on a simulation that is easier to scale than our method because of already-existing simulation platforms and tools. The proposed method provides greater insight into the evaluation process and is better than both cost-based and formal methods. However, we definitively cannot compare our method to the scenario-based and measuring ones. According to Table 11, there is no need to know about simulators and formal hard methods or even determine the cost for different components. Therefore, our method is better than others. Understandability and usability in general, in this viewpoint, and by breaking the problem into small parts, the major advantages of our method is simplicity and understandability in which we do not use any tools, special language or specific knowledge, making it more useful than other procedures. Concerning completeness, since our approach fully depends on the designer and developer of the evaluation method, we cannot offer a definitive judgement about it when compared to scenario-based and measuring methods. However, in relation to the formal and Cost-based methods, we can take advantage of a wide range of issues. The cost associated with the proposed method for achieving acceptable results is lower than that of all the other. In fact, it does not include even the cost associated with the definition of specific scenarios and the expert merely makes a pairwise comparison. In terms of accuracy, it is clear that ours has lower accuracy than formal and measuring methods. Nevertheless, in regards to the use of hierarchical structure in the evaluation, in its category and compared to the other methods, the defects are identified and corrected at each step. As a result, this is the best method in its category. Accountability is one of the important features of it because it provides capabilities such as modularity, step-by-step process and input and output separation and checking. In spite of this, it is clear that our method has a lower performance than formal methods. As for the measurement-based methods, since we do not consider readability and understandability here, no judgment can be made.

7.2 Threats of the proposed method

In general, any perspective in the context of software architecture evaluation comes with a certain set of pros and cons which should be considered. Our proposed approach, for weighting in the various domains has some risks such as increasing costs or decreasing accuracy (because of the involvement of human factor). In such situations, we should utilize some complementary methods to prevent these risks or reduce the probability of their occurrence and negative impacts.

The existing evaluation approaches related to the AHP questionnaires can be beneficial in preventing the inconsistencies in the answers. However, the lack of sufficient knowledge about weighting and the values of each quality attribute or architecture style can be considered as a potential risk. As we mentioned in the future works (refer to 8. Conclusion), in order to reduce the risks we can make use of a knowledge management system. It is worth mentioning that extracting and recording knowledge from architects would require more attention in order to prohibit misleading people and subjective judgments.

8 Conclusion

Selecting an appropriate software architecture is one of the key steps in any software development process, and satisfying quality requirements is one of the major challenges in this area. In addition, the relationships among these quality attributes must be studied when selecting the appropriate architecture from among a set of architectural styles or from among general candidate architectures. Hence, adding more information that must be analyzed and evaluated in the style selection process transforms the style selection issue into a multi-criteria problem. Prioritization uncertainty and the absence of quality metric evaluation standards are other problems in this context. To deal with these problems, a multi-criteria decision-making method based on fuzzy logic can be a useful solution for evaluating architectures and selecting the most suitable candidate. Therefore, this study proposes a fuzzy hierarchical analysis process (FHAP) in order to evaluate and compare candidate architectures. This process is a multi-criteria decision-making technique that resolves the aforementioned problems arising from the uncertainty of information using the fuzzy set theory. The proposed method ranks all options as well as providing a verifiable mathematical justification for this ranking.

Further work is needed to facilitate and improve the task of analyzing and evaluating software architectures, such as the design of Machine Learning mechanisms based on software reengineering activities. The proposed method could make this approach feasible by being integrated into decision support systems and prioritizing solutions based on information extracted and learned from previous cases. Therefore, by automating learning from domain information (i.e., a knowledge repository) instead of expert weighting and questionnaires, the method could generate better practical results to architects. In fact, the integration of the proposed method into the development tools and its use on the operational activities are also suitable future works.