Keywords

1 Introduction

Digitalization is increasingly leading to the development of numerous electronic teaching and learning innovations. In addition, educational content is more frequently being provided cooperatively [1]. Open Educational Resources (OER) have gained increasing attention as they transcend demographic, economic, and geographical educational boundaries and promote lifelong, personalized learning [2]. The OER movement has successfully promoted the idea that knowledge is a public good and has expanded the aspirations of organizations and individuals to promote OER [3]. Such rapid growth of OER offers new opportunities for teaching and learning. However, the potential of OER to transform teaching in higher education institutions (HEIs) has not yet been realized [2, 3]. An overview of OER research in recent years shows the challenges related to OER no longer lie in the availability or accessibility of those resources, but rather the challenges lie in the area of use. The necessary framework conditions for the use of OER at HEIs often remain unresolved. The current situation can thus be characterized as follows: Although OER are high on the agenda of social and inclusion policies and are supported by many actors in education, their use in higher education (HE) has not yet reached a critical threshold [3].

The conceptualization of a maturity model for HEIs in the context of OER addresses this problem and can serve as a tool for HEIs. By determining a maturity level that indicates how far an HEI’s current circumstances have matured for the inclusion and steady use of OER, universities could position themselves to identify problem areas and develop concrete solutions. Maturity in this context refers to “an evolutionary improvement towards a target state or to a natural end state” [4]. Maturity can also refer more specifically to competencies, skills, business processes, or products [5]. The maturity development to a higher, more advanced stage takes place in steps and is described by maturity levels [6].

To develop such a maturity model, it is necessary to know process models for conceptualization. De Bruin et al. created a generic framework describing the different developmental phases of a maturity model, which serves as a basis here [5]. In this context, maturity models were identified and developed specifically for the field of HEI. Many existing maturity models in the education domain can be found. There are two different approaches to these maturity models. On the one hand, specialized models focus on a subsystem of education, while, on the other hand, more comprehensive models represent the educational institution as a whole [7]. In this context, no maturity model for OER in HEI could be found. Based on the problem description, an OER maturity model should be developed. Due to the lack of empirical research in this area and no comparable established maturity model for OER in HE, a subsequent evaluation of the developed maturity model with different higher education stakeholders from diverse higher education institutions seems reasonable and necessary. The target group includes HEI management, support institutions such as platform operators, media competence centers, and didactics, as well as lecturers, i.e. professors, research assistants, and tutors. By developing a model in the first step, it is now possible for the respondents to express specific suggestions for improvement and criticism and not to come up with their own model in a purely hypothetical and very abstract way. For this reason, an online survey was conducted to evaluate the developed model. The results provide information on how the model can be adapted and optimized in a target-oriented manner. Accordingly, the research subject is guided by the following two research questions (RQs):

RQ 1: What might a first draft of an OER maturity model for HEI look like?

RQ 2: How do higher education stakeholders evaluate the developed OER maturity model draft and which necessary adaptations emerge from it?

To answer the research questions in a targeted manner, this paper is structured as follows. First, the theoretical basis for OER and maturity models is presented in the following section Theoretical Background. The third section deals with the development of a maturity model and the framework used for this purpose. The developed model is then shown and explained in more detail. This is followed by the evaluation, which is divided into methodology and results. In the end, the results are summarized, interpreted and specific adaption measures are derived.

2 Theoretical Background

2.1 Open Educational Resources

Open Educational Resources (OER) have been part of the educational landscape since 2001, through the announcement of MIT’s OpenCourseWare project, and even longer if the Learning Objects movement is considered as a precursor to OER [8]. The term OER was first introduced at a conference organized by UNESCO in 2000 and was promoted in the context of free access to educational resources on a global scale. There are now several definitions of OER, but they largely overlap. The William and Flora Hewlett Foundation, which funded the MIT project, defines OER as resources that include full courses, course materials, modules, textbooks, streaming videos, tests, software, and any other tools, materials, or techniques that support access to knowledge [9]. However, the most recent definition of OER from UNESCO is “Open Educational Resources (OER) are learning, teaching and research materials in any format and medium that reside in the public domain or are under copyright that have been released under an open license, that permit no-cost access, re-use, re-purpose, adaptation and redistribution by others” [10].

OER help to reduce access barriers to educational materials and support the opening of HEIs, as well as open education in all educational sectors with the active participation of all interested parties [11]. Therefore, OER have gained increasing attention in recent years because of their potential and promise to reduce such barriers as demographic, economic, and geographic boundaries in education and to promote lifelong and personalized learning [2]. The potential benefits of OER are widely advocated [12] and include improving access to higher education, lowering its cost, promote the culture of sharing and improving the quality of materials that result from collaboration and peer review [13, 14]. In general, OER have come to be seen as an invaluable educational resource for institutions and faculty in every region [15]. According to the OER Atlas 2017, the number of OER-based events has increased fivefold compared to 2015, while the number of OER projects has almost tripled, and the supply has increased by 70% [16].

However, previous studies of OER use have suggested that while educators are beginning to embrace open educational practices, understanding of the breadth of teaching and learning practices that OER enables is still limited. This problem may be exacerbated by the tendency of education-related change efforts to focus on educational content or resources, such as OER [3, 17]. Studies have found that limited teacher adoption of OER is influenced by factors at both the individual level [18,19,20] and the organizational/institutional level [21,22,23]. OER have not yet been able to sustainably establish themselves in the mainstream discourse of digitalization [24]. The reasons for the still rather low use of OER by teachers and learners are complex and multi-layered (pedagogical, technical, and organizational) [25]. It could therefore be important for an HEI to question its own framework conditions for the use and creation of OER and to be able to classify them in a maturity model.

2.2 Maturity Models

“A maturity model consists of a sequence of maturity levels for a class of objects” [6]. These models show which characteristics need to be assessed to determine the maturity of an object. Those characteristics are then collected and evaluated to determine the corresponding maturity level specific to the organization. Therefore, predetermined procedures, like questionnaires, can be used for application. The highest maturity level represents the total maturity of an object that can be achieved by an organization [6]. With the maturity principle, a distinction can be made between a stage model and a continuous maturity model. The former is characterized by the fact that the next higher level is only reached when all elements of the previous level have been fulfilled [26]. In the latter, however, the dimensions can be at different levels [27].

Many various maturity models can be found in the literature. The most popular maturity model is the Capability Maturity Model (CMM) from 1986, which has achieved global acceptance [5]. The development of CMM occurred at the Software Engineering Institute (SEI) of Carnegie Mellon University in Pittsburgh, USA and was commissioned by the US Defense Department. CMM was intended to be used to assess the US Defense Department’s software suppliers. It distinguishes a total of five maturity levels: Initial, Repeatable, Defined, Managed, and Optimizing. Each level contains process goals that must be achieved to increasingly stabilize the software development process. Processes in the Initial level are ad hoc and only defined to a small extent. In this step, project planning is lacking, so success depends on individuals and is not systematically repeatable. The second level, Repeatable, represents a process discipline, which is characterized by project planning, so that previous successes of similar projects can be repeated. A documented and standardized development process is found in the third level, Defined. Within the fourth level, Managed, quantitative quality targets for products and processes as well as measurement parameters are defined. The Optimizing level focuses on continuous improvement of the development process [28]. Over time, CMM has been applied in other fields, such as system and product development, so that the model was further advanced to Capability Maturity Model Integration (CMMI) in the 1990s. The levels of CMMI correspond to CMM, except for a change in the designation: Initial, Managed, Defined, Quantitatively Managed, and Optimizing. In contrast to CMM, maturity levels in CMMI refer to the entire process areas [29]. Both the CMM and later CMMI provided the basis for many other maturity models [6].

Published literature reviews have identified and analysed existing maturity models in the higher education field [7, 30], which provides a basis for this study. The analyses have shown that most educational maturity models have CMM or CMMI as their basis, despite different emphases like e-learning, online courses, or information and communication technologies. Maturity models explicitly related to OER have not been identified in previous papers [7, 30]. Different institutions have been involved in the development of the maturity model in the education sector. These have included educational companies, research organizations, as well as academic experts [7]. In the development process, a distinction is made between specialized models for a particular education sub-system, such as senior management training or project management, and broader models, which represent the education institution as a whole. Maturity models for education are at an early stage of development and need further improvement [7].

3 Development of a Maturity Model for OER in HEI

3.1 Development Framework

In the past, many maturity models have been developed. In most cases, it is not possible to retrace how the development and evaluation of the maturity model took place [6]. To address this, de Bruin et al. created a generic framework that divides the development of maturity models into generic phases [5]. The present study is based on this framework. The phases are divided into Scope, Design, Populate, Test, Deploy and Maintain and must be followed in this sequence [5]. In response to the research questions, the approach is followed within the first three phases Scope, Design and Populate.

In the first phase, the Scope of application is defined, which significantly influences the subsequent phases. A key decision that needs to be made in this phase relates to the focus of the model and development stakeholders. Regarding the focus of the model, it must be decided whether it should be a domain-specific or general model [5]. The aim of the contribution is to develop a maturity model specifically for OER at HEIs. Thus, the focus of the model is domain-specific, as the framework conditions for OER at HEIs are to be classified. The question about stakeholders is about who is involved in the development of the maturity model. The authors of this paper undertake the development of the maturity model as academics with practical experience in the use and creation of OER.

Design is the content of the second phase, in which the structure of the model is determined. The goal of this phase is to capture the complexity of reality in a simplified maturity model. Therefore, it must be decided which target group the maturity model is intended for, whether it is to be used internally or externally, and who are the respondents in the maturity survey [5]. The target group of this maturity model being developed is the management of several HEIs to get an assessment of the maturity of OER. The application is internal, and all stakeholders of the institution can be involved. In addition, existing maturity models for HEIs were considered from published literature reviews and examined for the most thematic overlap with OER. The Online Course Quality Maturity Model (OCQMM) was used as a basis due to the interfaces between OER and online courses. As already mentioned, within most existing models, the levels of CMM or CMMI are the basis, and so it is also in OCQMM. For this reason, the present maturity model, like OCQMM, is based on the five levels of the CMM: Initial, Repeatable, Defined, Managed, and Optimizing, and is designed as a continuous model.

In the next phase, Populate, the structure of the maturity model is filled. Characteristics are identified that indicate the level of maturity. Methods such as literature research or exploratory survey methods are suitable for this purpose [5]. Since published literature reviews have revealed that there are no existing maturity models on the topic, a pure literature-based creation of the maturity model is not sufficient, and therefore a survey was chosen as the method. However, an exploratory survey with higher education stakeholders would have been very abstract and not very promising in terms of results. With the help of a model as a basis, the respondents were presented with something tangible to consider and form their opinions on. To form this base for the survey, the contents of the model were independently formulated in the initial step. The fundamental structure of the model was provided by the dimensions from the selected OCQMM, which were adopted to the content of OER. For each dimension and each level, the characteristics were defined, which the authors derived from experience values and literature and discussed in depth. The developed maturity model for OER in HEIs is presented in Subsect. 3.2 and provides a basis for the survey by evaluating the content and gradations of the model by a range of stakeholders from several HEIs. The evaluation of the developed model is shown in section four.

During the Test, the developed model is examined for its relevance, validity, reliability, and generalizability. In the Deploy phase, the model must be provided for the application. The last phase, Maintain, includes the further development and update of the model. In order to maintain the relevance of the model, framework conditions must be continuously monitored, and adjustments must be made [5]. The last two phases go beyond the research questions and are for this reason not considered further in the approach of this paper.

3.2 Developed Maturity Model

The represented model in Table 1 includes six different dimensions derived from a literature review of existing maturity models in the field of higher education. The dimensions Learning Resources, Teaching Process, Teaching Platform, Faculty Conditions, Monitoring & Evaluation, and Management were taken from the OCQMM and were applied to the OER context. Each dimension contains different characteristics and criteria, which have been assigned in the table. These criteria are classified according to an ascending numbering of levels to distinguish between the five different levels of maturity. Accordingly, the maturity level of the respective dimension is lowest in level 1 and highest in level 5.

Table 1. Initial draft of a maturity model for OER in HEI
Full size table

The five maturity levels, Initial, Repeatable, Defined, Managed, and Optimizing, were taken from CMM. Each level contains process goals that must be achieved to increasingly stabilize the development process. In the first level, Initial, the processes are defined ad hoc and only to a limited extent. The next level, Repeatable, represents a process discipline characterized by project planning so that previous successes of similar projects can be repeated. In the third stage, Defined, a documented and standardized development process takes place. Quantitative quality targets for products and processes as well as metrics are defined in the fourth level, Managed. The final stage, Optimized, focuses on continuous improvement of the development process [28].

In the following, the content of the dimensions in the respective levels will be discussed in more detail. The first dimension, Learning Resources, reflects OER as learning content. Here, the criteria for creating OER as a learning resource, the quality assurance, the specification of metadata, and licensing surrounding OER are all considered. For example, in level one, no OER are used as learning resources in teaching, while in level five, OER are permanently created and checked by a standardized quality assurance process, and well-maintained metadata (e.g. author information, learning objectives, description text, etc.), and free licensing are specified. The Teaching Process represents the second dimension and describes the degree to which OER is integrated into the teaching process. The extent to which OER is used in teaching can vary greatly, so the degree of integration is an important criterion for determining an HEI’s OER maturity. In level three, the lecturers in this context already have experience with the creation and use of OER and integrate these as additional offerings in their own teaching. In this level, however, OER use by students is voluntary, so the OER is not an integral part of the teaching process. In stage five, on the other hand, complete courses are based on OER content, so the OER is fully integrated into teaching. The third dimension, Teaching Platform, refers to the technical requirements necessary to enable the use and dissemination of OER. Central characteristics are whether an HEI has its own OER platform, the provision of external OER offerings (e.g. via links), and the availability of internal OER on external platforms so that the content can be used outside the HEI. In stage three, for example, this means that an OER platform is available at one’s own HEI where lecturers can upload their OER and students at this HEI can access it. In terms of Faculty Conditions, the focus is on faculty and their use and creation of OER. The lecturers’ criteria knowledge about OER, the readiness to create or use OER and the readiness to share the created OER across the HEI are addressed. For example, universities are in level four if the use of external OER is established among the lecturers but the teaching staff also create and use OER themselves. The Monitoring & Evaluation dimension includes the collection of usage data (e.g. downloads, retrieval numbers, dwell time) and the collection of feedback from users and learners. Accordingly, the criteria of internal feedback by students, external feedback by the OER community, and the collection of usage data are considered. Consequently, in level one, neither feedback nor usage data are collected. In level five, feedback is regularly collected from students, and the usage data is collected and evaluated. Regular exchanges within the OER community and peer support are also promoted in this context. The last dimension, Management, focuses on the leadership and strategy of the HEI. Special attention is paid to the criteria of OER awareness at different levels (department, faculty, HEI management), the provision of resources for OER, performance recognition of OER offerings, and the HEI’s OER strategy. In level three, for example, OER are already established at the department level and are used regularly. For this purpose, internal departmental resources are made available, and the students themselves must check and inquire whether performance recognition is provided by the respective department.

4 Evaluation

4.1 Method

As already mentioned in Subsect. 3.1, an acknowledged maturity model development process is followed according to de Bruin et al. [5]. Regarding the third phase, Populate, a survey was chosen as the method for filling in the designed framework of the maturity model for OER in HEIs. However, to provide a basis for the survey, a first potential draft of such a maturity model was developed (see Subsect. 3.2). Using the model, respondents were given something specific to think about and be questioned on. To analyse the developed maturity model for OER and to answer the RQ2, an online survey was chosen. The survey was addressed to staff of HEIs throughout Germany where OER are used and/or created, which were identified by research. In the first step, a search on seven major German OER repositories (e.g. HOOU, OpenRUB, Twillo) helped to identify HEIs and the persons who provide OER. Based on this, online research was conducted to determine additional contacts at the identified HEIs. Independently, an additional general online search was undertaken to identify additional HEIs and the associated contacts related to OER. Since our study was aimed at participants from German HEIs, the survey was also conducted in German. Before the online survey was launched, a pretest was done with eight researchers to check the survey for comprehensibility and feasibility.

An embedded mixed-method approach was used to create the survey using the online survey tool LimeSurvey (www.limesurvey.org). With this approach, quantitative and qualitative data can be collected simultaneously or sequentially [31, 32]. The survey consists of qualitative and quantitative elements, so these data were collected simultaneously. A welcome text appeared at the beginning of the online survey explaining the purpose of the study and defining OER and maturity model to help develop a common understanding among participants. This was followed by the survey questionsFootnote 1, which were divided into three question groups. The quantitative research design dominated, which was supplemented by qualitative data. The first set of questions addressed each dimension of the maturity model by presenting each dimension individually with its content and levels. Questions were asked about each dimension separately and were answered by indicating “Yes” or “No” (quantitative data). Only when assigning a level to a dimension there were the answer options “Yes”, “No” and “Partly”. To get additional information from the respondents on their assessments of the dimensions, they had to justify their responses in the comment field (qualitative data). After the questions on each of the six dimensions were answered, questions concerning the maturity model in its entirety followed. In this question group, too, the answers were given via a “Yes”/“No” selection (quantitative data) with a justification in the comment field (qualitative data). Questions with a drop-down menu and one five-point scale (quantitative data) about the respondents’ demographic data and knowledge about OER follow at the end. The survey results were exported and evaluated to analyse the quantitative survey data. The response options were counted, which can be seen in Subsect. 4.2. In addition, the qualitative data were evaluated by using the qualitative content analysis (QCA) according to Mayring [33]. First, the text from the comment fields were sorted and then paraphrased. This was followed by a generalization of the paraphrases, and, in a third step, the first reduction was carried out by shortening semantically identical paraphrases. In a second reduction, identical paraphrases were grouped, which were then formed into categories. This is in accordance with the inductive approach, since the categories were not determined in advance but derived directly from the analyzed material [33].

In total, the survey was online over a period of four weeks between October and November 2021. The link to the survey was sent to the respondents by e-mail. A sample of 51 fully completed questionnaires were included in the evaluation. The average time to complete the survey was 30 min. The sample is composed of 22 females, 27 males, and two diverse participants. The 40–49 age group represents the largest respondent group with 19 participants, followed by the 30–39 age group with 16 participants. Eight of the respondents were between 50–59 years old and five of the respondents were between 20–29 years old. The smallest group surveyed was the 60–69-year-olds with only three participants. All respondents were from German HEIs from eight states in total. With the largest share of 49%, the respondents were lecturers (25 persons). This includes professors, research associates, and visiting lecturers. Employees at other higher education establishments like libraries or examination offices made up 27,5% of the respondents (14 persons). Eight out of 51 respondents (15,7%) were academic project staff. One respondent was from IT support of an HEI, and three surveyed persons (5,9%) stated other. All respondents were asked to rate their own knowledge of handling OER on a scale of 1 to 5, with 1 representing no knowledge and 5 representing the regular use and creation of OER. The arithmetic mean of value was 3.55. Most respondents (21) rated themselves a 4 on the scale.

4.2 Results

This subsection presents the results of the online survey on the maturity model for higher education institutions in the context of OER. In the survey, initially the same questions were asked for each dimension (1st question group), so that the results are described below for each dimension as well. The questions on the respective dimensions are divided into “Yes”/“No” selection options and free text fields. The “Yes”/“No” selection options provide a direct overview of clear agreement or disagreement on the part of the respondents, so that a first impression becomes apparent. The results of the quantitative questions can be seen in Table 2. After the presentation of the findings for the individual dimensions, a summary of the results is provided for the second question group, which asked questions about the maturity model in general. The third group of questions includes the demographic data, which was already examined in more detail in the previous section.

A total of 51 questionnaires were collected from various higher education stakeholders across Germany (N = 51). Table 2 shows the count of “Yes”/“No” responses based on the sample size 51. The table highlights the five questions, with the corresponding question number, that were asked in the context of each dimension. The count is given in percentages, and the particularly high values (> 70%) are shown in bold in the table. To be able to better classify these values, the answers from the comment fields are of particular interest. The most important findings from the free-text fields are discussed below.

Table 2. Quantitative results (based on N = 51)
Full size table

With regard to Learning Resources, there is strong support for this dimension, with 41 respondents rating the dimension as relevant for determining OER maturity at HEIs. It is also positive to note that over 80% of respondents would exclude none of the criteria presented, which emphasizes the high relevance of the criteria. One of the few statements on the exclusion of certain criteria simply states that “The degree of openness of the licenses does not say much about the degree of maturity. The choice of license is quite context dependent after all, e.g., the question whether commercial use is excluded, CC BY NC.” In addition, some suggestions for improvement were made in order to make the dimension and the associated criteria even more understandable and comprehensible. With regard to the comprehensibility of the levels, 58,8% of the respondents stated that there was a need for optimization. For example, the gradations of the quality criterion were criticized, so that the terms should be better distinguished from each other and explained in a more comprehensible way. The license criterion was also misunderstood by many respondents, as most argued that an OER without a license is not an OER and therefore there can only be a yes or no gradation. For metadata, examples should be given to clarify what the important metadata are, and level 1 should not mention usage but rather creation like the other levels. In this context, it was also stated that usage is a crucial criterion in dealing with OER and should be included as an additional criterion. Almost half of the respondents (49%) agreed to add criteria. File formats and aspects such as OER policy and the didactic preparation of resources were mentioned as further additions.

The levels of the Teaching Process dimension were rated as comprehensible by around 64,7% of respondents. Those for whom the levels were not intelligible gave wording and lack of information as their reasons. The survey participants would like to see a more precise definition of the term “teaching process”, i.e. what is meant by it. In addition, it was questioned how OER can represent a learning outcome or a complete course. The term “mandatory” used in level four of the dimension caused particular irritation among the respondents. One respondent commented that an obligation to use OER would be contrary to the basic idea of OER. One related comment was: “Here I see a restriction on the freedom of research and teaching”. Among 72,5% of the respondents, the criteria of the dimension were perceived as complete. Only 12 of all respondents would exclude criteria from the dimension. Here, primarily the obligation to use OER was mentioned, which was already questioned by the respondents in the previous questions. In addition, it was doubted whether an entire OER course could be regarded as the highest maturity or whether this criterion should also be excluded or adapted: “I don’t really know whether I would regard it as a sign of high maturity whether OER represents an entire course”. With a majority of 70,6%, the dimension was considered important by the respondents. However, there were a few hesitations about the extent to which this dimension could be adequately determined. The teaching process is seen as very heterogeneous, in that it can neither be centrally categorized nor uniformly determined for the whole HEI. This is also reflected in the analysis, where a slight majority of respondents (49%) considered it difficult to assign their own HEI to a level, while 29,4% would only be able to do it partially. In general, it is assumed that there is heterogeneity in each HEI regarding the dimension. It is pointed out here that a survey would have to be carried out within the respective HEI to be able to create a holistic picture.

The relevance of the Teaching Platform dimension for the determination of OER maturity was again confirmed by the survey participants. Respondents also overwhelmingly (70%) acknowledged the logical comprehensibility of the levels. However, it became quite evident, through many mentions, that the criterion of internal platform did not seem to make sense, although 76,5% of respondents indicated that the exclusion of criteria is not necessary. Many of the respondents expressed their opposition to the inclusion of this criterion because it was not a suitable indicator of OER maturity. Furthermore, the cooperative idea of OER was affirmed, so it seems to make more sense to rely on already existing infrastructures of, for example, portals at the state level, such as Twillo for Lower Saxony. In addition to the exclusion of the criterion of internal platform, recommendations were given for content-related extensions. Overall, 41,2% of respondents voted in favor of adding criteria, including interoperability (import and export of certain media and formats), accessibility (for which can OER be used), and the addition of certified OER platforms. In general, reference was often made to external platforms, which should play an overriding role in the context of the Teaching Platform dimension. Interactive potentials should also be included and further promoted in this dimension by including integrated modules in learning platforms for the development and provision of OER rather than focusing on pure file provision.

The Faculty Conditions dimension was rated by the respondents as very relevant for the determination of the OER maturity at higher education institutions and achieved the highest agreement, with almost 90% compared to the other dimensions. Furthermore, all selected criteria seemed to be important, as 78,4% of the respondents stated that no criterion should be excluded. In addition, only a few recommendations for supplements to criteria were made, as the majority, with 60,8%, of the respondents considered a supplement of further criteria unnecessary. In this context, criteria such as digital competencies or knowledge of OER platforms were mentioned as potential additions. The greatest challenge for the Faculty Conditions dimension was the gradation of the criteria for the participants. Level 1 should be adjusted accordingly, as it was argued that if there is no knowledge, the question about willingness to use or create OER is invalid. In addition, the order of use and creation seems to be very individual, as it was stated that some teachers either first used external OER and then created some themselves or not, or that teachers directly created OER themselves because they did not find suitable OER or did not trust the quality of external resources. The difficulty of grading these criteria also made a clear classification harder.

Regarding Monitoring & Evaluation, the majority (64,7%) confirmed that the levels were logically comprehensible. However, the wording in this dimension also led to difficulties in understanding. Potential for adaptation was seen in the areas of “OER community”, “feedback”, and “usage data”. Here, the respondents expressed a need for more information. This also applies to the other questions on the dimension as well. When asked about supplementary criteria, which the respondents saw as necessary, the specification of the OER community was mentioned to ensure that all important actors were considered. The expert community was mentioned, which was considered important for internal feedback. In addition, in terms of quality assurance, it was suggested: “If feedback is available, there should also be an adjustment loop for quality improvement”. Further, 90,2% saw no need to exclude criteria in this dimension. Only one respondent questioned the extent to which usage data was relevant for determining the maturity level. It is interesting to note that two respondents suggested a division of the dimension into monitoring and evaluation in order to make the levels even more comprehensible. A clear majority (86,3%) considered the dimension important, but an alignment change “much more from quality assurance point of view” is recommended. The respondents considered a classification into a level only division-specific (19,6%) or not at all (51%) realistic.

Within the dimension Management, the participants saw heterogeneous organizational differences as a special challenge in the dimension. Of all respondents, 58,8% considered the levels as comprehensible. In this context, the respondents criticized that no gradation was made within the OER strategy and suggested a differentiation. Better wording was called for regarding recognition of students’ achievements of OER and resources for OER. In the latter case, there was a particular lack of information on what was specifically meant by resources. One respondent asked: “What kind of resources should we be talking about? Time? Money? Technology? Personnel?” In addition, an overlap regarding the OER awareness criterion was seen, because “awareness has already been addressed with teachers”. Therefore, it is considered a duplication of the criterion knowledge of OER within the Faculty Conditions dimension. Additionally, 70,6% of the respondents saw no need to add more criteria to the dimension. However, additions arose from among the remaining respondents. Incentives for teachers, OER policy, and legal frameworks were mentioned several times as additions. The majority of participants (76,5%) also saw no need to exclude criteria. Among the minority, on the other hand, the exclusion of OER awareness was recommended, due to the overlap already described above. In addition, the recognition of students’ achievements of OER was also mentioned. One of the reasons given was: “Recognition of students’ achievements of OER is not so much linked to the use of OER as to appropriate exams. I am not sure if this is a reflection criterion for the university to OER”. Overall, the dimension was given high importance as confirmed by 84,3% of the respondents.

Regarding assigning their own HEI to a level (QN 5), there was a tendency across all dimensions with the exception of Teaching Platform. With respect to Teaching Platform, the assignment was often perceived as easier by the respondents, since it is relatively simple to find out whether the HEI uses its own or external platforms and which ones. With regard to the other dimensions, the majority of respondents were able to assign their own HEI either partially or not at all to a level. There are always two justifications for this. Either there is a lack of knowledge among respondents to make a holistic assessment due to the heterogeneity within each institution, or there are different classifications within the criteria of a dimension, which makes it difficult for participants to make an overall classification. Regarding the first reason, it is noted here that a survey would need to be conducted across the HEI to make a classification.

After the first group of questions on the six dimensions, five further questions were asked about the maturity model in general (question group 2). As a result, the most important stakeholder groups (QN 6, Yes 50,9%, No 49,1%) and dimensions (QN 9, Yes 72,5%, No 27,5%) were covered in the model. Only support units such as the libraries and administration, as well as ministries and the student group, were named as additions for other relevant stakeholder groups. Among the few suggestions for additions to the dimensions (QN 10, Yes 29,4%, No 70,6%), support activities, legal frameworks, and continuing education offerings were listed most prominently. The general comprehensibility was rated as very high with 43 agreements (QN 7, Yes 84,3%, No 15,7%). Participants who found the model incomprehensible stated that the gradations of the individual criteria needed to be improved as a reason. Furthermore, it was emphasized that the level division was good, “especially for an orientation about where one stands. However, in some cases a clear assignment to the levels is not possible or difficult”. Here, there is a need to improve the operationalizability and thus the applicability of the model. Still, it should be emphasized that the number of levels was mainly considered appropriate and sufficient (QN 8, Yes 76,5%, No 23,5%).

All of these findings culminate in an adapted OER maturity modelFootnote 2 that provides the basis for further research.

5 Conclusion

Maturity models optimize processes and provide an aid to better assess the current state of an organization [34]. In the education sector, in particular, such a maturity model for OER in HEI was lacking. In this context, two research questions were posed related to the draft and evaluation of a maturity model. The first research question asks what a first draft of an OER maturity model might look like. For this purpose, a model based on the development framework of de Bruin et al. was developed [5]. In addition, both the levels and the dimensions are based on different models that have been transferred and adapted to the OER context. To assess the developed model, an evaluation was conducted with 51 higher education stakeholders intended to answer the second research question. The most important findings are summarized below.

The survey confirms the high relevance of such a maturity model for HEI. The general comprehensibility of the overall model was also considered being given by almost 85% of the respondents in the survey. The respondents rated the selected number of levels, criteria, and dimensions as applicable and comprehensible. Among the exceptions that did not agree with the number of dimensions, only a split of the Monitoring & Evaluation dimension into two separate dimensions and support services (technical, legal, didactical) as additional dimensions were suggested. In this context, however, it was also stated that these structural support services represent an important framework condition for OER to find its way into HEI in the first place. Another criticism from the respondents relates to the selection of stakeholder groups. It was emphasized that students and support units, such as administration and IT support, should be included. Despite the confirmation of a high relevance of the maturity model for HEIs, some criticisms were made by the respondents in the evaluation with regard to the criteria and their gradations. In this context, several recommendations and suggestions for improvement were made, which are reflected in all dimensions and criteria. Among them were often difficulties in understanding, showing the need for more information and examples, e.g. the meaning of metadata or specific formulations should be adapted. Furthermore, the gradation of some criteria was criticized, and the differentiation of individual criteria should be made more precise in some cases. For some criteria, the assignment was also described as very individual, so that some kind of prioritization is required to be able to cover all circumstances. Only a few criteria were listed that should be excluded from the maturity model. Instead, there were some ideas regarding additions to various criteria that could be subsequently included. The assignment to a specific level was found to be particularly difficult. The reason frequently given was that there may be different variations in the criteria, making it impossible to assign them unambiguously. Furthermore, a lack of information and department-specific knowledge were cited as constraints to assigning the entire HEI to a particular level.

The results have been incorporated into numerous adjustments, so that the adapted OER maturity model is no longer a first draft model, but an empirically based model. However, the applicability and operationalizability of the model must be facilitated by a survey instrument for the necessary data. This tool needs to be developed in follow-up research. According to de Bruin et al., this still takes place in the Populate phase, which is then followed by the Test, Deploy, and Maintain phases [5]. In general, however, it should be noted that the numerous suggestions for adjustments that have emerged in the evaluation have led to a highly modified model. For this reason, the evaluation of the conceptual design during the Populate phase is considered only partially successfully completed. Rather, a further evaluation of the adapted model is desired to re-examine all adaptations. The goal is to use a subsequent Delphi study to successfully conclude the conception phase with expert knowledge and, at the same time, to provide an instrument that is ready for use. Figure 1 illustrates the steps that have already been carried out and those that are still necessary in the Populate phase. Thus, the first three steps towards an adapted OER maturity model for HEIs have already been completed (highlighted in gray) and the remaining steps (highlighted in white) will be considered in a further contribution of the follow-up research.

Fig. 1.
figure 1

Procedure of the populate phase

Full size image

Nevertheless, the existing evaluation confirms the first draft of an OER maturity model for HEI in its basic structure with six dimensions and five levels. The evaluation was important and useful, as many good optimization approaches were identified. In this way, a big step was made towards a final maturity model, which lays the foundation for further research.