Keywords

1 Introduction

Modern enterprises rely on Information Systems (IS) specifically designed to manage the increasing complexity of their operation. In the last decades, several alternatives to the traditional development of software from scratch, have emerged in order to construct such IS. In the usual case, they are built as hybrid systems which integrate several software components of different nature and origins e.g., legacy systems, web services, commercial components (typically referred as COTS) and, Free and/or Open Source Software (FOSS). Because of the nature of hybrid systems, proper selection evaluation of software components plays a prominent role. This critical task is usually conducted with the support of Software Quality Models (QM) [1], artifacts specifically engineered to support the specification, prioritization and evaluation of requirements, the description of software components, and the identification of gaps among their features and requirements.

Several proposals for QM and QM standards have been proposed [1,2,3,4], particularly in the case of FOSS several proposals exist e.g. QSOS [5], OpenBRR [6], QualOSS [7], EFFORT [8], most of them as extensions of the ISO/IEC 9126 [9] standard. However, these approaches focus on the evaluation of technical functional and non-functional (internal and external) requirements, forsaking other non-technical aspects which can be just as relevant (e.g. economic, legal and quality of the provider), particularly when acquiring software from third parties e.g. communities and commercial suppliers.

In [12, 13] we found an extension to the ISO/IEC 9126 quality standard, which deals with technical and non-technical quality requirements in a homogeneous and holistically way. 3 characteristics and 15 non-technical characteristics were added to the original ISO/IEC 9126 catalog (which can also extend its successor, the ISO/IEC 25010 standard) for this purpose. This extension is called the extended ISO/IEC 9126 catalog. However, due to its nature, and after conducting an extensive systematic literature review (SLR) to identify critical success factors, potential benefits and risks in the adoption of OSS, we identified that this extension is not enough to deal with non-technical quality requirements in the case of FOSS. In this paper, we propose a new extension to the extended ISO/IEC 9126 quality standard required to support the evaluation and selection of OSS.

In addition to this section, this paper is structured as follows: Sect. 2 introduces previous and related work; Sect. 3 presents the SLR in which this work is based and its main findings; Sect. 4 introduces the new proposal for the extended ISO/IEC 9126 catalog; Sect. 5 evaluates its relevance in relation to four FOSS industrial adoption processes in which we have actively participated; finally, Sect. 6 presents the conclusions and lines of future work.

2 Previous and Related Work

2.1 The ISO/IEC 9126 Quality

The ISO/IEC 9126 software quality standard is one of the most widespread quality standard available in the software engineering community. It proposes quality models as the artifacts that keep track of the quality factors that are of interest in a particular context, i.e. for a software domain of interest. From the 4 parts that compose it, we focus here on the 9216-1 part of the standard [9], which presents a catalogue of quality factors intended for the evaluation of software components.

The ISO/IEC 9126-1 standard fixes 6 top-level characteristics: functionality, reliability, usability, efficiency, maintainability, and portability. It also fixes their further refinement into 27 sub-characteristics but does not elaborate the quality model below this level, making thus the model flexible. Sub-characteristics are in turn decomposed into attributes, which represent the properties that the software products belonging to the domain of interest exhibit. Intermediate hierarchies of sub-characteristics and attributes may appear making thus the model highly structured.

When the domain of interest is complex, building ISO/IEC 9126-based quality models may be tough. We apply the methodology described in [6] for ISO/IEC 9126-based Quality Model Construction.

2.2 The Extended ISO/IEC 9126

[12, 13] contain an extension of ISO/IEC 9126 technical characteristics and sub-characteristics with additional non/technical ones arranged in an ISO/IEC 9126-1 tree-like structure, thus the resulting catalogue includes high-level quality characteristics and sub-characteristics, and also lower-level quality attributes. To build the hierarchy we followed the 6 step method presented in [10, 11].

The top-level of the hierarchy has been structured with 3 non/technical characteristics: Supplier, Costs, and Product. These three characteristics group non-technical quality features required to measure the supplier capacity to address and support the project, the implementation costs and the out-of-the-box quality and effort required to get the component running. These non-technical quality characteristics have been further decomposed into 15 non-technical sub-characteristics (see Table 1). Some of them have been decomposed into other sub-characteristics, whenever they were required for structuring or leveling purposes. Following the construction approach, sub-characteristics have been further decomposed into over 160 non-technical quality attributes.

Table 1. Non-technical quality features upper-level hierarchy.
Full size table

Other approaches (please refer to [7, 8] for detailed description) also address the need of non-technical attributes to be considered in evaluation and selection, however, their proposals can be considered non-structured a subset of our approach. We claim that technical and non-technical aspects shall be dealt similarly using a common framework instead of divided assets. To achieve this goal, we propose to extend the ISO/IEC 9126-1 catalog of quality factors with non-technical factors following the same layout than in this standard. In particular, as done in the standard, we fix just the two higher levels of the hierarchy, avoiding excessive prescription of the proposal. We call this catalog the non-technical extension of the ISO/IEC catalog (NT-ISO/IEC catalog for short) to distinguish it from the previous one (Fig. 1):

Fig. 1.
figure 1

The different catalogues found.

Full size image

2.3 OSS Adoption Strategies

The concept of strategy comes from the Greek ‘strategos’ to denote ‘leadership’. An OSS Adoption Strategy is the way in which an organization incorporates OSS as part of its customer offering. In [14], authors describe six different OSS adoption strategies which in the following paragraphs, are presented in ascendant order according to the strength of the relationship between the adopter and its FOSS Developer Community (FOSS-DC). For instance, in Release strategy, there is not relationship with FOSS-DC. In the following bullets, the benefits, and requirements are presented for each strategy.

  • Release: It implies that the organization releases personalized software as FOSS but does not care whether an OSS community takes it up or forms around it. Additionally, no FOSS-DC is involved. The purpose of this strategy is to develop FOSS (with technical quality) for own use, and offer it to organizations with similar requirements in order to: (a) standardize its subjacent processes; or (b) spread good practices, or (c) reduce cost and risk by avoiding the development. The first two may be related to improving the market, and the third may be focused on strengthening the corporate group companies. In this sense, an example of this strategy can be observed in the public sector, where public entities make their software available to others, releasing it under an OSS license. This strategy demands a minimum involvement with FOSS-DC and the organization does not care for FOSS evolution or maintenance.

  • Acquisition: This strategy refers to the use of existing FOSS code without contributing to its OSS project/community. Because the adopter organization does not necessarily has, an interest in new releases of the FOSS component, the involvement with the community is minimum after obtains the software and its documentation. In some cases, other companies or freelancers can provide the FOSS component. The specific benefits are that it requires a minimum involvement with FOSS-DC, and does not care about FOSS evolution or maintenance.

  • Integration: It involves the active participation of an organization in an FOSS community (to share and co-create FOSS) but not necessarily leading or influencing it. The organization establishes a relationship with a new external stakeholder: The Support Forum, from which obtains bug fixes that not necessarily are present in the next version that the FOSS Community releases for the component.

  • Fork: It means to create an own independent version of the software that is available from an existing FOSS project or community. The FOSS-DC is forked too. This strategy is applied by an organization that decides to continue the (generally critical) FOSS component and FOSS-DC evolution for its own account, to achieve specific requirements.

  • Takeover: In this strategy, the organization attracts an existent FOSS community to support its business activity. The creation of its own FOSS-DC pursues to ‘take the control’ of critical software development.

  • Initiative: It is oriented to initiate an FOSS project and to establish a community around it. The organization creates its own FOSS-DC, because requires to ‘take the control’ of critical software development.

3 Searching for Relevant NT-Quality Attributes for OSS

Although FOSS has been increasingly used, particularly in the last decade, critical success and failure factors, risks, benefits, and barriers associated with this paradigm have not been deeply analyzed. The lack of knowledge in these relevant issues hampers FOSS adoption both in private and public industries. In 2018, we performed a systematic literature review (SLR) to gain understanding of how these factors affect FOSS implementation. 11 previous SLR where identified, none of them addressing these issues, which endorses our claim that more study is required in the field.

The SRL was conducted following Kitchenham’s [15] methodological guidelines. To be rigorous, a protocol was established to search and collect data from the existing FOSS literature. Research question included in the protocol where:

  • Q1: Which success and failure factors have been documented in the adoption of FOSS in the industry?

  • Q2: Which are the main risks identified in the adoption of FOSS?

  • Q3: Which are the perceived benefits in the adoption of FOSS?

  • Q4: In which software domains has the adoption of FOSS been successful?

  • Q5: Which are the main factors that prevents organizations from adopting FOSS?

The generic search string was constructed using the words Open Source Software, OSS, FOSS, FLOSS and Freeware as synonyms; to complete the string we added the words risk, failure, success, barriers, advantages, benefits and application domain. The search string was built using the “OR” and “AND” logical operators. Search string was constructed to address titles, abstracts and related words: Title: (Open Source Software OR OSS OR FOSS OR FLOSS OR Freeware) AND Abstract: (Open Source Software OR OSS OR FOSS OR FLOSS OR Freeware) AND (risk OR failure OR success OR barriers OR Advantages OR Opinion OR Impediments OR FLOSS). The string was properly adapted to the characteristics of the repositories where searches where conducted: IEEE Xplore, ACM, Springer Link, and WOS.

A total of 782 documents were retrieved from the repositories; titles and abstracts where stored in an excel sheet. In a first refinement, only academic articles written in Spanish or English, with methodological basis (e.g. experiment, case study, Systematic Reviews, Systematic Mappings), with at least 5 references in google scholar where included in the study.

Duplicated articles were excluded and we limited the study to written after 1998 of the following types: Journals, Conferences, Magazines, Technical Reports and Books. In a second refinement, all abstracts were reviewed in full, by at least two researchers of the team, to select which were relevant and sound for the study. Documents marked as “relevant” or “maybe relevant” for at least one of the researchers were selected to be full text redden. The documents were extracted, to read the summary, to see if the article could contribute with the investigation, once read, the researchers granted ratings of: “Yes”, “No” and “Maybe”. Only articles written in Spanish and English where included in the study. At the end, only 155 documents were selected to be full text reviewed. From those, only 92 papers answered, at least, one of the research questions. In addition, backward snowballing techniques allowed for the identification of 33 additional references; only 17 resulted relevant for the study. A table was specifically designed for data extraction and implemented using a spreadsheet editor. Table was refined/extended through the study, to include all information extracted from selected papers, relevant to answer research questions.

The most relevant success factors identified in the study are: (a) Management: software evolution, licensing, project management; (b) Community: community activities and communication; (c) Support: internal and external; (d) training: staff training; (e) Policies: IT enterprise policies.

Regarding failure factors, the most relevant are: (a) Human Resources: IT provider, end users, technical users, development and community; (b) Competitiveness: Market characteristics; (c) Policy: OSS and IT adoption policies absence.

Several risks were also identified, the most relevant are: (a) Laws: assume the responsibility; (b) Software: complexity, processes, tests, support, technology, integration, and architecture; (c) Human resources: developers, community, experts, management, lack of guarantee.

Several benefits have also been identified in the study: (a) Flexibility: source code availability; (b) human resources: community, providers and contributions.

The application domains identified in the study are: (a) system software: Operating systems, server software; (b) application software: mobile apps, office, medical, commercial software, educational, browsers, databases managers, automation and control.

Finally, several barriers for the adoption of FOSS have also been identified: (a) human resources: training, developers, experience, uncertainty, motivation, lack of trained professionals; (b) software: quality, migration, product, version, repositories.

Table 2 presents a relation among the factors and the percentage of papers relating them with the main issues considered in the SLR, namely success and failure factors, risks, benefits, and barriers associated with the FOSS paradigm.

Table 2. Percentages of documents relating factors with categories in the SLR.
Full size table

4 The Catalogue of NT-Quality Attributes for OSS

Similarly, to the catalogue in [12, 13], the Extended FOSS NT-ISO/IEC 9126-1 quality catalog (see Table 3), has been structured with a three-levels hierarchical decomposition. At the first level, we have included four main Categories: External Context Actors Relationships, Organizational Issues, Government Policies, Staff Capacities Development, and FOSS Support. We have also proposed a refinement of the Support sub-characteristic, categorized under the Supplier Characteristic, in the original Extended NT-ISO/IEC 9126-1 quality catalogue, to make it more suitable for the FOSS evaluation context. This sub-characteristic has been renamed FOSS Support when used for FOSS evaluation purposes.

Table 3. Extended NT-ISO/IEC 9126-1 quality catalogue.
Full size table

Each characteristic has been further decomposed with two additional hierarchical levels, the first one including sub-characteristics and the second one including attributes, which are observable quality features associated to one or more measures, relevant for the statement of organizational quality requirements and the assessment of FOSS quality, during evaluation.

We have also incorporated additional columns to shape the catalog structure. The first one “Meaning to Adopter”, allows for traceability with the main category of factors in the SLR from which the attribute has been identified, namely, Benefit, Entry Barrier, Risk, Success Factor, Fail Factor, or Support. In addition, we have also considered the six OSS adoption strategies proposed in [14] (see Sect. 2.3). These strategies model the strength of the relationship between the adopter and its more representative external context actor: the FOSS Developer Community (hereafter called FOSS-DC). Each attribute has a different relevance (or even no relevance) depending on which adoption strategy is selected by the organization. For instance: if the organization opts for the Acquisition Strategy, the attributes A.2.1 FOSS-DC feedback and A.2.2 Lack of FOSS-DC Support are not applicable, because, in the practice, the FOSS component is used as-is, without modification, and in consequence, FOSS-DC feedback or support is not required. On the other hand, if the adopter opts for the Fork Strategy, the active role of FOSS-DC in terms of feedback and support is critical.

The attributes A.2.6 Version Control, and A.2.7 Version Frequency already appear in the extended NT-ISO/IEC 9126-1 quality catalog (under the product/stability sub-characteristic), but our study has shown that it may have greater impact in FOSS adoption processes, in terms of risks introduced to the project. Version release frequency is usually higher and more flexible in FOSS, since it responds to the dynamics of the FOSS-DC. The increase in version release frequency hampers ability of technical staff to incorporate (parametrize, adapt, modify, test, integrate, and deploy) new versions of FOSS, with significant impact maintainability of IS.

Attributes B.1.3 Cost Management and B.5.1 Cost of Training are included in the original extended NT-ISO/IEC 9126-1 quality catalogue, but in this work we present a refinement with measures specific for FOOS. In the same way, attributes categorized under Quality Policies (B.2.1 Security strengths, B.2.2 Security weakness, and B.2.3 Reliability) are part of original extended Technical ISO/IEC quality catalogue, but here they are oriented to FOSS adoption support.

In addition, the original extended NT-ISO/IEC catalogue includes a significant amount of attributes intended for the evaluation of licensing schemas. In our context, however, we need to be specific in relation to aspects such as the access, manipulation and redistribution of open source code. Specific measures for each attribute have been defined (not included in this paper due to space limitations, see Table 4 for an excerpt).

Table 4. Except of measures for extended quality catalogue
Full size table

5 Validating the Catalogue in Practice

In 2008, the Government of “Country Blinded” issued a specific order to enforce the FOSS adoption in the Public Administration. During the first years of the ruling, the efforts for the adoption were isolated and chaotic, causing some disappointment and a kind of misgiving about the regulation.

To address this issue, a group of engineers working in the public sector, conducted a project to develop a set of artifacts to guide the adoption of open technologies in the country. The results of this project include a catalog of templates and guidelines specifically designed for this purpose. With the support of these deliverables, since 2016 the number of projects to migrating to FOSS alternatives in the country has increased both, in the public and private sectors.

In the last two years we got involved in four of such projects, see Table 5 for some details. We use the knowledge gained in these projects to validate the relevance of the attributes included in the catalogue introduced in Sect. 4.

Table 5. Validation cases.
Full size table

In Table 6, for each verification case, we have assessed the impact of each of the non-technical quality attributes described in the catalogue. Rows include the non-technical quality attributes in the catalogue whilst columns represent} the verification cases. We have used Likert’s five levels scale to assess the relevance of each attribute in the projects, (one for the lowest and five for the higher impact).

Table 6. Impact of NT-quality attributes in validation cases.
Full size table

Last column presents the total impact for each attribute, calculated as the sum of impacts in the four projects. 24 of the 44 non-technical attributes included in the catalogue (54%), were graded with a relevance higher tan of 15 over 20 possible points; 7 (16%) additional attributes, scored between 10 and 15 points, which makes evident the relevance of the attributes in the catalogue for these kind of projects.

It is important to mention that some attributes in the A category were not evaluated. This is due to the fact that, these organizations had already adopted some FOSS and the focus of the project was on upgrading and giving it a better maintenance to the software instead of adopting new software. On the other hand, it is important to consider that, in some cases, public institutions are forced by ministerial decrees to implement specific FOSS products, without chance to evaluate alternatives.

Organizational/Quality Policies attributes have been valuated with the highest scores. It can be argued that OSS products could be more vulnerable due to their open code nature. Consequently, security policies are required to minimize related threats. In contrast, some factors e.g., the independence of the provider, are perceived as means to increase protection of personal data, since code can be explored to prevent the existence of “backdoors”, used for some manufacturers to introduce remote control codes into the software. In addition, open standards, that grant the freedom of users to exchange information with a FOSS community, are perceived as an important security factor when considering FOSS adoption.

Licensing is another interesting factor to consider. In the context of “Country Blinded”, most Chief Technical Officers (CTO) still perceive Open Source as synonym of free software. Therefore, it is evident the lack of accurate economic feasibility and resource allocation studies in the projects.

In three of the four cases presented in Table 5, some consultants tried to sell expensive licensing schemas with the promise of “easiness” in FOSS product’s troubleshooting. In fact, lack of technical skills and knowledge has been identified (in the study presented in Sect. 3), as one of the main barriers and risks in the adoption of FOSS. At the end, this fear factor causes some enterprises to purchase expensive licensing for premium functionality.

Despite the fact that attributes in OSS-DC SUPPORT category have the highest impact, it is important to consider attributes in the STAFF CAPACITIES DEVELOPMENT category. Some anthropologic aspects e.g., soft skills, risk acceptance, openness and flexibility ae included in this category. These attributes are highly important in the context of FOSS adoption projects, since successful adoption largely relies in technicians’ attitude. Cultural change of technical and end users needs to be carefully addressed in this kind of project.

6 Conclusions and Future Work

The traditional approach to evaluate software components rely on technical requirements widely described in diverse quality models, but does not consider the high relevance of non-technical factors. In this research work, we have proposed a Non-Technical attributes catalogue, which can be used join with Technical ones, to evaluate FOSS Software Components considering the way in which they were adopted by the organization. These requirements are an extension of preexisting software quality models, and were obtained from SLR as a basis to identification success and failure factors, risks, benefits, main software domains, and entry barriers related to FOSS adoption. These requirements were structured in a three level hierarchical catalogue of non-technical attributes, covering from strategic to operative issues in FOSS evaluation context. Finally, the relevance of non-technical attributes was validated in relation to four industrial FOSS adoption cases taken from the experience of our team. Even when some of the attributes couldn’t be evaluated, we confirm the suitability of the catalogue when evaluating and selecting a FOSS product. As future work, we will focus on refinement and validation of the measures set for each non-technical attribute in the catalogue, in order to make feasible an objective and systematic evaluation of FOSS component adoption.