Keywords

1 Introduction

Software acquiring organizations generally have an IT management team that plan/establish technologies to be adopted or standardized to support their applications (i.e., software products and services). In this context, an IT architecture is a list of technologies to be used as standard within an organization [14], often classified according to the technology categories used by the organization (e.g., database, programming language etc.). In addition, an IT architecture contributes to meet organizational business demands through a set of technical decisions [1, 5]. Architecture modifications may involve the adoption of a new technology, removing part of the architecture, or replacing technologies that an organization already uses. Changes in such architecture are not trivial as they affect the development or acquisition of new applications that must be adherent to IT architecture, e.g., new software development projects should use an existing technological standard according to the IT architecture.

In addition, they may reflect on development team’s training needs regarding the new technology, aversion to changes and switching licensing costs, or even applications that depend on discontinued technology [16]. Because of rapid technological evolution, organizations frequently need to update/reevaluate their IT architecture. Evaluating the technology in relation to pre-established, manageable, and well-structured criteria provides greater transparency to the process, as IT managers and architects should be able to check/audit the adopted criteria. In [8], one of the most successful actions pointed out by companies is to use a well-defined procedure for IT acquisition. Part of the definition of such procedure is to establish evaluation criteria for technology selection [16]. Revisiting an IT architecture is necessary to maintain the technological platform. Moreover, it is a challenge considering that organizations are relating themselves as a software ecosystem (SECO). SECO involves elements out of the organizational scope, e.g., applications, technologies, internal and external developers, suppliers, and users. As such, there are architecture attributes related to the maintenance of an IT architecture, from organizational or technical nature, not identified or used together in the SECO context [9]. For public companies, this issue has even more restrictions, such as adherence to governmental norms and standards, current legislation, electronic procurement process with less control over technology selection processes, and budget. Private organizations usually have more freedom to choose technologies and applications. However, both types face the lack of indications to guide technologies’ modification to maintain IT architecture (and how to collect them) [12].

This research aims to identify architecture attributes that affect a SECO and its platform and technologies from the literature. With the intention of comparing this research to a standard, ISO/IEC 25000 characteristics were also analyzed against architecture attributes. Finally, we have evaluated such attributes with experts from industry and academia based on a survey research. This paper is organized as follows: Sect. 2 presents the background; Sect. 3 brings the mapping study; Sect. 4 presents the survey research; Sect. 5 brings a discussion and Sect. 6 concludes the paper.

2 Background

2.1 Software Ecosystems

SECO is described as a set of actors interacting with technical artifacts such as software products and services based on a common technological platform [4]. As an organization stops developing its independent products, i.e., limited to its internal resources and actors, it creates relationships with companies, suppliers and products that change organizational business [3]. Thus, organizations are more dependent on external partners, suppliers and tools – none of that is controlled by them. Therefore, it is imperative to study not only the platform itself, but also the network of actors and artifacts as a SECO [16].

Actors are people inside and outside the organization who interact in several ways, e.g., developers, users, suppliers, competitors, and external players. Artifacts include software products, components, requirements, documentation, services, among others. According to a recent systematic literature review on the topic [15], health is on the top 25 paper keywords on Business and Management in SECO. A healthy SECO can maintain and increase the number of actors and artifacts, also creating opportunities for its actors. As such, a healthy SECO should be aware of technology management.

2.2 Platform Maintenance

Following SECO platform and its guidelines, it is possible to standardize processes and technologies for the application development. Modifications in platform involve the adoption of a new technology, removal of part of the architecture, or technology replacement. As such, technology change within the platform development must be carefully performed since this action affects the organization’s standards. For example, choosing a technology that fails to support a legacy system can be costly, or that generates training costs without sufficient benefits to justify them. There are other organizational constraints that may affect adoption or discontinuation of technologies [16], such as:

  • Organization policies and standards, e.g., encourage open source software or national suppliers, not accepting certain types of proprietary license;

  • Legislation, e.g., especially in cases of public companies, the country’s legislation may affect candidate technologies (taxation and embargoes);

  • Economic issues, e.g., budget for the period of modification and country’s economic situation; and

  • Organizational culture, e.g., aversion to paradigm shifts, rejection of technologies that reuse external components, and rejection of certain vendors.

3 Mapping Study

Systematic reviews aim at incorporating evidence and providing a synthesis of the area, while mapping studies are mostly involved in exploring a research area. In addition, there are specific guidelines to expose the result for a systematic study. The type of literature assessment used in mapping studies mainly focuses on structuring a research area and its topics, gathering frequency and definitions. Hence, it offers a general idea of the research area scope. Besides, it also aids the determination of research gaps and tendencies [17]. This study is presented as a Mapping study because it is an exploratory approach for gathering information on the main architectural characteristics of SECO and painting a picture of the literature context.

This work serves as an initial basis to aid IT managers and architects to understand how their choices regarding technology acquisition can affect a SECO platform, as well as provide some actions to diminish harmful effects. With the intention of comparing this research to a well-accepted standard, ISO/IEC 25000 [6] characteristics were also analyzed against the critical factors resulted from the mapping study and validated by the survey, as described in Sect. 4.

The authors of this paper participated in a previous mapping study that primarily investigated how scientific literature studies software architecture in the SECO context, e.g., key characteristics, research needs, and reference architectures. The search string covered title- abstract-keyword with the terms (“software architecture” OR “software architectures”) AND (“software ecosystem” OR “software ecosystems”). For each search engine, the search string was adapted according to the syntax rules but keeping terms and logical operators. The search string was run on the Scopus, Springer, IEEE, ACM, and Science Direct search engines. This first mapping study grounded the study presented in this section because, by participating in it, it brought better understanding of the architectural facet of SECO, the most researched topics, and gaps. In addition, its accepted papers and search strings were reused as a starting point for the mapping study to serve as a corpus for the extraction of architecture-related quality attributes for technology selection in SECO. It was not found in the literature a study that concise SECO attributes specific to quality and architecture context. This mapping study complements the literature by offering the list of attributes scattered in literature papers from the main search machines.

3.1 Planning

We defined the following research questions (RQ):

RQ1. What are the architecture-related quality attributes that describe or qualify software ecosystems and their platform regarding the architecture perspective?

RQ2. How do the architecture-related quality attributes relate to each other?

The activities planned for this study were executed in five steps. The set of studies was obtained after executing the mapping study (Step 1). Then, the full reading allowed us to extract the attributes and track the source papers (Step 2), as well as identify relations among attributes described in the selected papers and other possible associations (Step 3). From such relations, it was possible to group the attributes based on similarities, level of abstraction, or interactions reported at the papers (Step 4). Finally, we analyzed the possible effects those attributes could have on a SECO platform (Step 5).

This mapping study followed the same procedure and search string of the previous study. It was conducted by a Master student and supervised by two PhD students and a senior professor. There was not a specific term to be searched for, i.e., papers were scavenged for any term that characterized a SECO as well as its architecture or platform, considering that all the included papers have discussed SECO/architecture. As inclusion criteria, the studies must meet the following requirements: (1) the studies must present a discussion about SECO, its elements and architecture, regardless of which element of the SECO they focused on; and (2) the studies must be written in English and available online.

3.2 Execution

The execution was performed so that we reached as many studies published as possible along with those studies brought from the previous mapping study. In [11], a systematic literature study captured the main keywords related to ‘software ecosystem’. The third popular term was “architecture”(s)/“architectural” and they were accompanied by “open”, “parallel”, “service oriented”, and “software” in the papers keyword fields. Since there were no keywords for “technology architecture” or “IT architecture”, the search string was generalized for the expression “software architecture” since it represented a very common expression for SECO context according to [11]. The search engines used were ACM, IEEE, ScienceDirect, Springer, and Scopus. In Scopus, some studies were rejected because they already appeared at other search engines. Accepted studies from the previous mapping (34) were also accepted in our mapping. Additionally, 10 new studies were included.

3.3 Results

The final literature base is composed by papers published from 2009. After reading the title, abstract and keywords, few papers were excluded because they fell out of the scope/context of this work by not focusing in any quality related SECO subject or they referenced SECO but did not ground the work on its concepts or research scope. Some papers were not reachable (i.e., full text was not available online, although we requested some of them to the authors) and thus removed from the literature base. From the 44 accepted studies, 16 (36.36%) did not present architecture-related quality attributes concerning SECO or its architecture or platform. Many papers mention the same attribute, e.g., 11 papers cite “integration”, even appearing in different SECO contexts. Quality attributes were mentioned as attributes, for example, “openness”.

The extraction was manually executed while reading the full text of selected papers and considered attributes seen as technology evaluation criteria. The criteria for identifying an attribute was being explicitly mentioned in the papers as SECO quality attributes, or key factors, properties or challenges. In addition, some papers report on SECO requirements regarding the platform architecture. Other attributes are nouns and adjectives used to describe a specific SECO; in this case, studies or more generic models in the context of architecture or platform. Only 6.2% (4 attributes) has more than five citations. Perhaps the great number of attributes with only one citation (42.2%) since specific SECO contexts are explored in the studies. Although the set is general, it also reaches many contexts. Table 1 shows the classification of attributes according to the papers and how attributes can comprise others as critical factors. The last three critical factors are health measures according to [7]. Critical factors are aggregations based on relationships indicated in the selected papers. Their definition and the references to the papers that mention them are presented in detail in [10]. The mapping does not bring new attributes but adds to the literature in identifying them and gathering its uses.

Table 1. Results organized as critical factors and attributes
Full size table

3.4 Analysis

The extraction resulted in 64 architecture-related quality attributes. The more generic attributes (bold font in the row above quality attributes in Table 1) are critical factors. They help technology assessment as they represent categories of criteria for comparing candidate technologies. Attributes associated with critical factors (subsequent rows in Table 1) can be perceived as different perspectives to assess a factor. Associations were directly extracted from the papers or assigned by the researcher according to critical factors and attributes’ definition in the papers. An association happens in cases when an attribute definition includes another one, then the attribute becomes a critical factor related to the attribute contained in the definition, even if both are not explicitly linked as key factors, challenges, or another similar relationship. It might not be necessary to use the whole list of attributes, since a specific organizational context might differ from others. Thus, an organization should decide what information is available or relevant. Attributes cited once might be too specific, new or less relevant. Since it is a long list, practitioners may want to start assessing technologies after using a subset of attributes, e.g., the most popular ones.

The study can also minimize decision bias (commonly based only on manager experience) and better justify technology selection rather than an ad hoc process. For each attribute, an interpretative scale might be associated, e.g., cost: range from feasible to not feasible. IT management team then should choose a value within the range and at the end its members will have a comprehensive comparison of technology candidates. When looking at the literature on software product evaluation based on quality attributes, there are proposed quality models [1, 3, 4]. Assessing quality from standards that compose the ISO/IEC 25000 series, also known as SQuaRE (System and Software Quality Requirements and Evaluation), can help IT management teams in acquisition rounds [6]. However, those guidelines reflect traditional paradigms that leave SECO out of scope. ISO/IEC 25000 defines 8 characteristics and 31 sub-characteristics to assess product quality. They use a similar structure to the one presented in this research, i.e., critical factors/quality attributes would match characteristics/sub-characteristics from ISO/IEC 25000 SQuaRE. Nevertheless, there is a high resemblance in their use and definition. Critical factors and ISO/IEC 25000’s characteristics present many similarities (Table 2). Columns show ISO/IEC 25000’s characteristics and rows represent SECO’s critical factors. If applicable, each cell contains a critical factor or attribute that is related to an ISO/IEC 25000’s characteristic, also considering its sub-characteristics. For example, the critical factor “extension” (third row) has an attribute “modifiability” that is similar to “portability” (“adaptability” sub-characteristic).

Table 2. SECO’s critical factors versus ISO/IEC 25000’s characteristics and sub-characteristics
Full size table

ISO/IEC 25000 models lack characteristics to address SECO concerns related to the external player activities (development of extensions or applications). Those matters are illustrated by quality attributes that had no correspondence, e.g., “extensions’ deliver”, “extensibility” and “quality of extensions”. All ISO/IEC 25000’s characteristics are considered by at least one critical factor. On the other hand, “stability” and “niche creation” (SECO’s critical factors) are not similar to any ISO/IEC 25000’s characteristic (according to the sub-characteristics’ definition). “Stability” encompasses “framework stability”, “interface stability”, “rate of change”, and “parallel development”.

3.5 Threats to Validity

We can point out some threats to validity as follows: the literature base used for the mapping study could have overlooked some studies with architecture-related quality attributes; attributes might have been incorrectly identified; and attributes might not have been identified from the accepted studies; and the attributes extraction was performed manually and had not followed a formal method. The attributes’ extraction was performed manually and did not consider differences among SECO contexts, e.g., mobile sector, health sector or agriculture section. This set of attributes is an initial list and must be broken into subsets according to the suitability for each platform (IT management team’s members should select a subset to work with). Surveys participants did not have access to the glossary while the study was conducted. In addition, the survey could benefit from analysis focusing in particular SECO platforms or specific SECOs contexts, e.g., Mobile and Cloud.

4 Survey Research

The survey research used an electronic questionnaire written in English to be filled in 20–30 min. It was sent to the invitees’ e-mails who are experts in SECO, technology selection and IT management/architecture. The complete questionnaire is divided as: (1) Research Summary; (2) Term of Consent; (3) Characterization Form; (4) Critical Factors’ Relevance; and (5) Critical Factors’ Attributes Relevance. Considering the objective of capturing participants’ experience, a five-point scale similar to previous surveys run by SECO researchers was used [1, 10]. This objective was to investigate if critical factors (Sect. 3) are relevant for technologies selection in a SECO platform. As a result, experts’ opinions on the critical factors and their attributes were collected. From the experts’ opinions, critical factors and their attributes were evaluated. In addition, we analyzed if the attributes represent relevant perspectives on the related critical factors. In Table 3, the goal of this survey research is described following the GQM (Goal – Question – Metric) model [2]. Applying GQM approach helps to clarify the study strategy and purpose by specifying a group of targets and how to interpret it. GQM uses Goals representing the Conceptual Level that measures Processes, Products, and Resources. Question means the Operational Level specifying how it is going to be measured. Metric represents the Quantitative Level identifying the data to be collected. The GQM Model structures the questions and goals considering a particular context and point of view. 144 participants were invited from 22 countries. Invitations were sent by e-mail and participants were chosen from websites of events related to the topics, including: ICSOBFootnote 1; WDESFootnote 2; IWSECOFootnote 3; and WEAFootnote 4.

Table 3. GQM for the survey goals
Full size table

4.1 Execution

From those 144 researchers invited to participate in the survey research, 28 invitees responded the questionnaire (19.4%). A rate of response of 20% is adequate when the sample size exceeds [13], thus this survey response rate is acceptable considering the samples size. Participants had no obligation to answer all the relevance questions from parts (4) Critical Factors’ Relevance and (5) Critical Factors’ Attributes Relevance.

4.2 Results

Characterization.

Participant’s experience on the related topics was relatively high, as shown in Fig. 1. Participants’ characterization information shows that 86% are Postdoctoral/PhD and 14% of Master and PhD students. It shows their experience as researchers on the related topics and likely strengthens their contribution to this survey. Moreover, participants can be considered experts in the related topics with experience on research (61%) and industry (7%) and both (32%).

Fig. 1.
figure 1

Participant’s responses regarding experience

Full size image

Critical Factors.

All critical factors were assessed as ‘Some relevance’ and ‘Highly relevant’. As shown in Fig. 2, ‘No Relevance’, ‘Little Relevance’ and ‘Limited Relevance’ answers all together did not reach 50%. It means that experts find those critical factors relevant and therefore applicable for technology selection in a SECO platform. Few features were suggested, some of them already presented as attributes of critical factors.

Fig. 2.
figure 2

Critical factors relevance assessment

Full size image

Participants had not seen the attribute list before the question regarding suggestions of critical factors, so it is positive that they might recommend some features that are attributes already proposed by this research. The critical factors from Fig. 2 are: CF1 Configurability; CF2 Cost; CF3 Extension; CF4 Openness; CF5 Quality; CF6 Reuse; CF7 Scalability; CF8 Stability; CF9 Support; CF10 User experience; CF11 Version compatibility; CF12 Niche creation; CF13 Robustness; and CF14 Productivity. Some participants identified critical factors that might be interrelated, although this study did not consider such relationships. From 28 participants, 15 left general comments about the critical factors.

Critical Factors’ Attributes Assessment.

For each critical factor, participants were asked for assessing how relevant its attributes were, based on a five-point scale. For CF1, the majority of participants found both attributes to be ‘highly relevant’ or with ‘some relevance’. CF2 attributes are balanced when comparing the sum of ‘highly relevant’ and ‘some relevance’. Those terms are not strange to researchers and are related. For example, a close platform (low openness) might be private software and its licensing may have some cost. CF3 has only one of its six attributes that has been voted as ‘no relevance’, in fact ‘little relevance’ is 3.7% in average among its attributes. Those are low rates in comparison to other critical factors reaching over 60% of the top level of relevance in the scale.

CF4 refers to openness and was the third most cited quality attribute in the mapping study presented in Sect. 4. The best-evaluated attributes are flexibility and security, but availability is the only one that had no vote for ‘no relevance’. CF5 was the top relevant critical factor with 60.7% of ‘highly relevant’ votes. Testability had no vote for the lower two levels of relevance, and it was the attribute with more ‘highly relevant’ votes. On the other hand, certification was not as well evaluated. Testability refers to the capability to be tested by anyone and certification implies a third party attesting for the quality.

Reuse (CF6) is a critical factor that is hard to find properly in several organizations. The most ‘highly relevant’ attributes are extensibility, integration, and modularization. Those are the most technical attributes. Attributes such as transparency, understandability, and cost are usual among practitioners, but they were not assessed by the experts with upper level of relevance.

A usual concern when scaling up a platform is that its performance would not keep up with more users or greater data flow. In CF7, performance was not considered the most ‘highly relevant’. Participants said they did not miss any attribute, not even for cost, since it was listed in this research. CF8’s most relevant attributes refer to the stability of the platform components (frameworks and interface). Parallel development may interfere with matters of time, but it was not considered very relevant since it is not a dominant practice in organizations. Rate of change might depend on the framework and interface stability, since their rate of change can influence the platform’s demands for changes.

CF9’s attributes assessment is similar. Documentation is essential for supporting developers in understanding the functionalities and differences between releases. The shared information is not necessary from external parties, e.g., forums and FAQs, but also among the developers and architects working in the platform that also refer to non-technical problems, such as lack of communication. User experience (CF10) is essential for a SECO that deals with end users, as they can stop using the platform if user experience fails. User experience is not restricted to the interface they interact with, but also to how easy and simple it is to find and use the platform’s functionalities.

Version compatibility (CF11) considers the compatibility of functionalities among the platform versions. In the perspective of developers using the platform for their own development project independently from an organization, it is very harmful to keep changing the stable platform version based on all releases. All attributes are equally ‘highly relevant’.

Backwards compatibility and maintainability influence the problem a developer has to face during the development process. In a bigger change (e.g., replacing the platform), the project might suffer with specific native functionalities and it should be necessary two separate projects, e.g., developing for different app’s versions (Android/iOS).

Niche creation (CF12) is a health indicator for SECO. The more and diverse opportunities the SECO provides, the better its niche creation is. Innovation is the most relevant attribute and directly relates to niche creation. When a SECO produces and promotes innovation, new opportunities and niches are created.

Robustness (CF13) is defined as the capability of a SECO to resist disturbances. Availability is assessed as the most ‘highly relevant’ attribute. It makes sense since comparing SECO availability before and after a disturbance may be an indication of how much a SECO is robust. For that, other attributes influence a SECO regarding the platform and technologies, e.g., how resilient and stable are the technologies that support the platform. The offline capability was not considered very relevant, perhaps because it is a specific requirement.

Productivity (CF14) reflects how many projects the SECO produces. The relevance of each attribute does not differ much. All attributes refer to the existence of external parties developing on that platform.

Deployability affects how fast a developer is able to deploy and then publish his/her projects, affecting the individual productivity that composes the overall productivity. Extension’s delivery may stop the projects’ developments if the platform extension is not updated (or a new one is not delivered).

Learnability can prevent new projects to begin. If understanding is too difficult, the rate of developers giving up their projects may increase; thus, the overall productivity falls. The majority of participants voted for at least ‘some relevance’ for all attributes, and then no attribute was removed at first. Collected suggestions were confronted to the assessments to decide whether or not they should be adopted. Putting together votes for the two highest relevance points in the scale (‘some relevance’ and ‘highly relevant’). Only four participants left comments; they mainly expressed concerns about lack of definitions and variations on relevance according to different SECO contexts.

4.3 Threats to Validity

Although most of the terms are common at the literature and participants are experts in the related topics as their characterization profile, some participants might have slightly different conceptions of the same term used for critical factors and attributes. Participants were free to not assess critical factors and attributes, as some questions missed zero, one or two answers. The question with the most absent values had two missing answers compared with the total of 28 participants, then they are not threatening to the study significance. The survey was executed as an electronic questionnaire in order to reach the international community. Interviews might help to collect more data outside the questionnaire (informal), although the survey had questions for comments. Finally, it could not be assured that the sample size was optimal and that they had a high representativeness of the population. Likert scale might not assure that participants used the same criteria for each relevance level.

5 Discussion

The survey shows positive relevance on the use of SECO’s critical factors and attributes. Table 4 presents the percentages of the two highest grades in the response scale (‘some relevance’ and ‘highly relevant’) for each critical factor. No critical factor was dismissed since no participant asked for removal of any in the questionnaire. Thus, no critical factor was excluded from the list. 57% of the participants said they did not miss any critical factor. Some of them suggested few properties as critical factors: Institutional policies; Vendor trustworthiness; Continuity; Market speed; Flexibility, portability, trustworthiness, sustainability, interoperability; Security, integrity, portability; Innovation; Flexibility and communication; Buildability and learning curve (i.e., how easy it is to learn it); Architecture; Usability; and Supplier reputation.

Table 4. Critical Factors’ evaluations in percentage related to number of respondents for each question. (SR = Some Relevance and HR = Highly Relevant)
Full size table

The set of criteria used in this research (list of critical factors) relate to SECO platform and its architecture/ technologies. Thus, some properties such as vendor trustworthiness, continuity, market speed, sustainability, communication, and supplier reputation were not considered as critical factors.

From all 64 attributes (some of them are repeated from different critical factors), only five had less than 50% when putting together ‘some relevance’ and ‘highly relevant’. In order to decide if they should be removed, we looked into participants’ suggestions and comments to find out if anyone expressed an intention of dropping attributes out of the list. As a result, “synchronization” was eliminated from CF4 - Openness and “parallel development” was moved to CF14 - Productivity. Moreover, 68% of participants said they did not think any attribute was misplaced.

After analyzing participants’ suggestions as well as consulting their respective proposed relevance levels, the set of critical factors and their attributes was updated after removing, including, copying or moving some attributes, as explained in this section. In addition, some critical factors that appeared as attributes were removed. The final list is presented in Table 5.

Table 5. Final set of critical factor and attributes
Full size table

6 Final Considerations

SECO’s platform is broadly supporting the use and development of software artifacts. In this context, the adopted technologies that support a SECO platform affect what actors enter and keep playing within a SECO. In this paper, we reported on the identification of architecture attributes that affect SECO from the literature. In order to compare results with ISO/IEC 25000 characteristics. Next, we evaluated results with experts from industry and academia based on a survey research. Then, 64 attributes were identified and grouped by 11 critical factors.

In this context, it is necessary to compare the candidate technologies and integrate information with SECO data. The information used to compare needs to cover not only technical properties, but also socio-technical elements, i.e., communities (users, developers and organization), feasibility, quality, cost, and support. This is because the SECO perspective brings information external to the organization/platform and its relationships. Managing platform technologies has many benefits to an organization, e.g., technology standardization, saving money, avoiding unnecessary acquisitions, and supporting a controlled number of technologies. Fast market changes of technologies, deployment of new versions or discontinuation of support require frequent modification and assessment of the SECO platform’s reference technologies. In addition, there are effects on finances, users, politics, training, and other perspectives that need to be considered when an organization is changing platform technologies.

As a contribution the research and practice community, we identified and validated a set of architecture attributes to aid IT managers and architects to understand how those choices can affect the SECO platform and technologies and take actions to mitigate the negative effect. Practitioners can use this list as guide for a criterion when comparing technologies in a SECO, since it is necessary to evaluate the technology itself and its relationships to other software artifacts. As future work, we are developing a tool to help IT manager and architects to perform semi-automatic analyses of critical factors and architecture attributes based on the SECO.