Lean and Agile Assessment Workflows

Abstract

The chapter presents and discusses a structured approach to assessment planning and organization. The approach is inspired by Kanban-style notations as well as by the SEMAT approach for agile software engineering processes. It is based on a breakdown of assessments into their essential elements and phases. It is hence more suitable for agile assessment planning than traditional workflow and process models. It allows to define assessment workflows very easily, so that teachers, item authors and staff can focus better on their individual duties. The chapter is organized in three main sections: The first section introduces the breakdown of assessments into their essential elements and phases. The second section demonstrates how to use these elements in a Kanban-style notation to formulate assessment scenarios. It also discusses examples from practical experience in different scenarios. The third section very briefly elaborates on tool support.

1 Introduction

Preparing and conducting educational assessments is not an easy thing. First of all, the contents have to be created carefully to make sure that the assessment considers right what is right and wrong what is wrong. Second, test pedagogy and psychometrics have to be considered to make sure that the assessment really measures what it is supposed to measure. Finally, a lot of organizational aspects concerning the when and where have to be considered. Depending on the setting of the assessment, this may involve communication with assessment authorities in case of formal assessments, set-up and management of electronic assessment tools in case of computer-assisted assessments, but also simple communication with participants that has to happen even in informal low-tech assessment scenarios. Consequently, educators will follow some kind of workflow even in very simple cases to make sure that the right things happen in the right order (Reynolds, Livingston, & Willson, 2009; Johnson & Johnson, 2002; Banta & Palomba, 2015; Dick, Carey, & Carey, 2014). One can even dig deeper into the formal definition of workflows and processes and notice that training and instruction are sometimes understood as a system which follows not only a process definition, but also has input and output (Laird, Holton, & Naquin, 2003). Since these are classical terms from development and production, one can also think of adapting more terms from that domain, such as lean and agile processes.

In the particular context of technology-enhanced assessment, there was a tendency in recent years to create models of assessment processes that where very detailed and formal. An example for this is presented in Danson, Dawson, and Baseley (2001), which is related to a university-wide process, but limited to a particular existing tool. We can identify roles like ‘Exam Office’ or ‘Students’ in conjunction with their activities in these models. The FREMA (Framework Reference Model for Assessment) project collected a more complete set of elements that might occur in process models. One of the project’s outputs is a concept map for e-learning assessment processes (Millard et al., 2006; Wills et al., 2007). This map lists activities that turned out to be relevant based on interviews within the assessment community. It covers several didactic aspects such as authoring of assessment items, checking solutions for plagiarism or creating feedback to students, but also organizational issues such as checking the availability of candidates and staff or preparing digital and physical environments. This concept map for e-learning assessment processes is a valuable source of didactic and organizational activities related to assessments, but does not provide an actual technique for modelling actual workflows.

Anyway, restricting educational assessments to strict process definitions following some formal guidelines does not match the daily experience and requirements of educators. There is not the single ideal workflow for educational assessments and there is also not the single ideal workflow for a particular assessment scenario. Instead, educators have to amend and adapt their workflows based on the resources available, the time frame they have to prepare the assessment, and also the number of participants expected take part in the assessment. Educators also usually do not want to care for a large process description in case of small assessments, but are satisfied with lean descriptions covering the essential elements of the assessment workflow. These essential building blocks may be customized for a particular assessment situation or tailored in the way they are used. These requirements towards an appropriate workflow description can be summarized in a small list of principles

• Workflow elements should be concise and represent reusable steps. This is a value known from lean product development.

• Workflow descriptions should only contain the necessary elements while unnecessary steps are eliminated. This is a value known from lean production.

• Workflow descriptions must be changeable and adaptable. This is a value known from agile development.

There is currently no approach documented in literature that covers all these requirements. However, these requirements represent an understanding of lean and agile workflows that is not specific to educational assessments. Hence, techniques and standards can be reused in this domain that have proven their value already in other domains in which workflows play a major role.

The ESSENCE standard (Essence—Kernel and Language for Software Engineering Methods, 2015) is a modelling standard created by the ‘Software Engineering Methods and Theory’ (SEMAT) initiative and issued by the ‘Object Management Group’ (OMG). It tackles the very same topic of lean workflow descriptions for the domain of software engineering. It defines a modelling language for software engineering process descriptions and a so-called kernel of key elements (named ‘alphas’ and ‘activity spaces’). These are supposed to be relevant in any software engineering project. Each alpha defines a set of states with checklists that allow to track project progress. It is possible to create simple process descriptions by grouping states across alphas and thus defining phases or milestones. It is possible to add more details by assigning ‘work products’ to alpha states or ‘activities’ to activity spaces. This allows a very lean and agile style of defining and using workflows with exactly as much detail and formalisms as necessary (Jacobson, Spence, & Ng, 2013). As a means of graphical representation, the ESSENCE standard introduces the notion of alpha state cards. They are concise representations of an alpha state and its checklist items that can actually be used in the form of small physical cards. These allow enacting many agile practices in planning and monitoring of workflows in a very smooth and natural manner.

While the standard explicitly talks about software engineering, there is no reason to refrain from using the concepts of kernel and alphas in other domains as well. Thus, this chapter uses the ideas of ESSENCE to create both a kernel of key concepts related to educational assessments and some sample workflow descriptions based on this kernel. These descriptions are intended to serve as a blueprint for different workflow descriptions covering several kinds of educational assessments, such as traditional written exams, modern electronic assessments, and oral assessments as well as less formal assessments. A short section on electronic tool support is also included towards the end of the chapter. Hence, the reader may get two main contributions from this chapter: First, it creates a general and unified process model in the domain of educational assessment that covers both traditional and modern forms of assessment. Second, it demonstrates how to define lean and agile workflows on top of this kernel. It thus provides a starting point for modelling one’s own workflows.

2 Kernel for Educational Assessment

The ESSENCE Kernel for Educational Assessment presented in this section is intended to form a common base for all kinds of educational assessment processes. It is neither limited to a particular didactic purpose of the assessment (e.g. diagnostic, formative or summative) nor to a particular form of assessment (e.g. written assessment, oral assessment or electronic assessment). In order to achieve full but flexible coverage of all kinds of processes in educational assessment, the kernel consists of eight alphas from which two are optional in most cases. The alphas can be grouped into three so-called ‘areas of concern’ similar to the original ESSENCE standard. An overview of the kernel alphas and some of their relationships is shown in Fig. 1.

Fig. 1

Overview of the eight alphas in the Kernel for Educational Assessment and some of their relationships. Dashed borders indicate optional alphas

2.1 Area of Concern ‘Content’

Probably the most important parts of an assessment are its functional contents. They reflect its professional or scientific domain and are represented by this area of concern. Typically, experts in the particular domain create and maintain these contents and assure that they are right. Failure in reaching the desired quality of content in an assessment most likely causes useless assessment results. This area of concern consists of three alphas, representing the different bits an assessment and its results are composed of.

2.1.1 Alpha ‘Test Items’

A test item is the smallest consistent unit within an assessment that allows candidates to demonstrate their competencies. For the purpose of this kernel, it is assumed that a test item contains some kind of task description and that the candidates are expected to respond to it in some way, e.g. by ticking answer options, drawing a diagram or answering orally.

The alpha ‘Test Items’ covers all items potentially used in the assessment and does not ask how an actual test is composed. The test items may form a general item pool or several distinct item pools from which a certain number of items is used in the actual assessment. However, it is assumed that all test items that are potentially used need to be prepared in the same way.

The proposed kernel defines an alpha with four states to represent all essential aspects of test items (see Fig. 2). The names of the first three states are ‘Scoped’, ‘Designed’ and ‘Verified’. They are concerned with the different stages of preparation for test items. The alpha particularly reflects the observation that test items have some formal properties (such as an item type, language and intended difficulty) which are defined in the first state, while their functional properties (such as a task description and a sample solution) are defined in the second state. As legal regulations may explicitly require a second author to do a review of all proposed test items, the third state handles verification and double-checking. The name of the fourth state is ‘Outcome reviewed’. It reflects the didactic practice to review the outcomes of a test with respect to test item performance in order to identify test items with unexpected results (e.g. ones that were often answered wrong by good candidates or ones that were answered right by anybody).

Fig. 2

Alpha states and checklists for the four states of alpha ‘Test Items’

2.1.2 Alpha ‘Test’

A test is the actual collection of test items that is delivered to the candidates of the assessment in some way, e.g. by handing out papers, displaying on a screen or asking questions orally. The alpha refers to the test as an abstract construct and hence does not ask whether a candidate actually sees the whole test at once or only can see and answer the test items within the test one after another. There is also no assumption made on whether the test is a static composition of test items or generated adaptively like in computer adaptive testing. Consequently, a test may be the same for all candidates or may be composed individually from one or more item pools.

The proposed kernel defines an alpha with five states to represent all essential aspects of tests (see Fig. 3). The names of the first and second state are ‘Goals clarified’ and ‘Designed’. They correspond to the first two states of the alpha for test items, as also the whole test needs both a definition of its formal and functional properties. The name of the third state is ‘Generated’. It is fulfilled when an actual instance of the test is created for each candidate. As already mentioned above, this may be a physical representation such as some pieces of paper, but it may also be the specific sequence of questions asked to one particular candidate in an oral exam. The name of the fourth state is ‘Conducted’. It is fulfilled when all candidates have completed their tests. Notably, in a written exam this state may be reached days or even weeks after ‘Generated’ depending on how long before the day of the test the exam sheets are printed. Different to that it may be reached minutes or even seconds after the last question is posed in an oral exam. The fifth state with name ‘Evaluated’ represents the fact that a test needs to be evaluated. This also includes the retrospective analysis of test item performance as above.

Fig. 3

Alpha states and checklists for the five states of alpha ‘Test’

2.1.3 Alpha ‘Grades and Feedback’

As the outcome of test evaluation can be very different depending on the didactic purpose and context of an assessment, it is worth modelling grades and feedback as a separate alpha. Each response to a test item contributes to the actual test result, which may consist of marks, credit points, texts, or anything else that is used to inform the candidates about their performance. Results can be assigned both to single test items and to the whole test (or arbitrary parts of it). The alpha covers all these different kinds of feedback and makes no assumptions about whether candidates have access to results during the assessment or only afterwards.

The proposed kernel defines an alpha with four states to represent all essential aspects of grades and feedback (see Fig. 4). Again, the first two are concerned with preparations: State ‘Granularity decided’ reflects that fact that there are many ways of how to give feedback and that the didactic purpose of the assessment determines the choice. State ‘Prepared’ refers to the creation of appropriate marking schemes or alike as well as organizational set-up of grading sessions. The name of the third state is ‘Generated’. It is fulfilled if all grades and feedback are created. The final state is fulfilled when grades and feedback are available to the candidates and is thus named ‘Published’. Notably, in a written exam it may take some time after the submission to reach state ‘Generated’ and it may also take some more time to reach ‘Published’. Different to that, feedback in an oral exam is often generated right after a candidate answered a question and is also published immediately by responding to the candidate’s answer. However, as the alpha refers to grades and feedback in general, state ‘Published’ may nevertheless be fulfilled later, as grades are typically not mentioned after every answer, but only at the end of an exam or even at some later point in time.

Fig. 4

Alpha states and checklists for the four states of alpha ‘Grades and Feedback’

2.2 Area of Concern ‘People’

Although we already mentioned domain experts as the authors of assessment content, they are not the people in the focus of an assessment for two reasons: First, assessments can be conducted by using predefined tests or test items, keeping the authors completely out of the process. Second, the steps performed by test item or test authors may be domain-specific and are thus out of the scope of a generic kernel for educational assessments.

Hence, this area of concern focuses on people who are more directly concerned with an assessment. It represents each group with a separate alpha: The organizers running the assessment (who may also author test items as part of their duties while preparing the assessment), the candidates taking part in the assessment and optionally the authorities concerned with the legal aspects of the assessment. If either of these parties fails to fulfil their role within the assessment process, there is no guarantee that it will produce the desired outcome.

2.2.1 Alpha ‘Organizers’

For each assessment, there is at least one person responsible for organizing it and thus managing the assessment process. For larger assessments, it can be assumed that more people are involved in setting up and conducting the assessment, including test item authors, assessors and technical staff. Each of them pick up parts of the responsibility for conducting the assessment and are thus responsible for some part of the assessment process.

The proposed kernel defines an alpha with four states to represent all essential aspects of the organizers’ duties (see Fig. 5). The name of the first state is ‘Identified’. It represents the fact that it may require some work to find out who needs to be involved into the assessment for which tasks. The name of the second state is ‘Working’. It is fulfilled when all responsible persons have picked up their duties. Once they have done everything that is required to start the actual assessment, state ‘Satisfied for Start’ is reached. Similarly, the final state ‘Satisfied for Closing’ is reached when all evaluation and post-processing is done and the organizers have no more open duties.

Fig. 5

Alpha states and checklists for the four states of alpha ‘Organizers’

2.2.2 Alpha ‘Candidates’

The largest group of people concerned with an assessment are usually the candidates, which are the persons who take part in the assessment by solving a test. Although they are involved personally in the assessment process for a relatively short period of time, the proposed kernel includes an alpha with seven states to represent all essential aspects related to candidates (see Fig. 6). The names of the first two states are ‘Scoped’ and ‘Selected’. They refer to the part of the process in which it is first defined who is allowed to take part in the assessment and second the actual persons are identified. The third state ‘Invited’ is fulfilled when candidates know how to prepare themselves for the assessment. The following two states ‘Present’ and ‘Dismissed’ refer to the physical presence of the candidate at the location where the assessment takes place. Notably, that does not mean that all candidates will be at the same place at the same point in time. They are also considered ‘Present’ if they are in different locations. It is also possible that some candidates are already dismissed, before the last one is present, as it is usual in oral exams. The names of the sixth and seventh state are ‘Informed’ and ‘Satisfied’. They reflect the fact that candidates need explicitly to be informed about their results, which corresponds to state ‘Published’ for grades and feedback. In addition, they also often have some time frame to place complaints before the grades formally count as accepted.

Fig. 6

Alpha states and checklists for the seven states of alpha ‘Candidates’

2.2.3 Optional Alpha ‘Authorities’

Depending on the didactic and formal setting of the assessment, some official party may be formally responsible for any legal issues related to conducting the assessment. As this may introduce additional process steps or dependencies between states, authorities are introduced as an additional optional alpha in the kernel. This alpha is only relevant for formal assessments. States and checkpoints for alpha ‘Authorities’ are shown in Fig. 7. The name of the first state is ‘Identified’. It covers the same aspects as the corresponding state of alpha ‘Organizers’. The name of the second state is ‘Involved’. It is fulfilled when all assessment information relevant to the authorities have been provided. The naming of the state is different from the second state of alpha ‘Organizers’, as authorities are supposed to play a less active role in the assessment process. Hence they may be involved in terms of providing information or verifying documents, but do not necessarily work in terms of creating contents or making design decisions. The names of the third and fourth state are ‘Satisfied for Start’ and ‘Satisfied for Closing’. This is again similar to the states of alpha ‘Organizers’. They are reached when there are no more legal obstacles to start the assessment or the legal files for the assessment are ready to be closed, respectively.

Fig. 7

Alpha states and checklists for the four states of alpha ‘Authorities’

2.3 Area of Concern ‘Logistics’

Besides contents and people, there is also a demand for physical or technical facilities to conduct an assessment. In any case, there are one or many physical locations where candidates are located while taking the assessment. Optionally, they are also using some technical system in case of a computer-aided assessment. This area of concern thus consists of two alphas for the physical and technical aspects of assessment organization. One can imagine adding a third optional alpha for materials or equipment needed during the assessment in case the candidates have to perform physical experiments in natural sciences, artistic or musical exercises using instruments or requisites, or similar. However, the states and checkpoints necessary for this kind of alpha are very likely to be domain-specific. Thus, they are out of scope for a generic kernel. Instead, they can be added as domain-specific extensions to the kernel, similar to the domain-specific extensions that are defined in the ESSENCE standard.

2.3.1 Alpha ‘Location’

It is assumed that each assessment needs some physical location where candidates will be located while taking part in the assessment. Depending on the kind and size of assessment, this may be a single room for all candidates (at the same time or one single candidate or group after another) or a set of distributed locations.

The proposed kernel defines an alpha with six states to represent all essential aspects related to the assessment location (see Fig. 8). Quite similar to the states for candidates, the names of the first two states for the location are ‘Defined’ and ‘Selected’. They refer to the fact that first some abstract requirements are formulated towards the properties of the assessment location and then an actual room or set of rooms is selected. As rooms are physical resources, they may cause conflicts with other assessments happening at the same time. Hence, state ‘Reserved’ is explicitly introduced to cover the necessary communication as well as the calculation of set-up time. If all set-up is done, the location is considered ‘Prepared’, which is the fourth state (corresponding to ‘Satisfied for Start’ for the organizers). The names of the final two states are ‘In Use’ and ‘Left’. They correspond to some extent to ‘Present’ and ‘Dismissed’ for the candidates but also cover the fact that the location needs to be restored after the assessment.

Fig. 8

Alpha states and checklists for the six states of alpha ‘Location’

2.3.2 Optional Alpha ‘System’

In case a computer-aided assessment system or similar electronic system is used to conduct the assessment, it can be represented by an additional optional alpha. The alpha covers all possible duties of this system such as administering the tests, accepting submissions, associating grades and feedback to submissions and performing grade and feedback generation automatically. This alpha is only relevant for electronic assessments. States and checkpoints for alpha ‘System’ are shown in Fig. 9.

Fig. 9

Alpha states and checklists for the six states of alpha ‘System’

Similar to the previous alpha, the names of the first two states are ‘Defined’ and ‘Selected’. This again reflects the fact that (at least in an ideal scenario) one would first define some abstract requirements towards the assessment system and then select and actual system fulfilling these requirements. In reality, organizers sometimes have no choice, as they must use the system provided by their institution. In this case, these two states are fulfilled by default and the features of the available system may restrict organizers in the selection of test item formats they can use. Since the ESSENCE notation does not require to define dependencies between states from different alphas explicitly, processes for both orders can be defined and monitored using this kernel. The name of the third state is ‘Available’. It refers to the fact that the selected system also needs to be accessible to continue preparation. This in turn will lead to the fourth state, which is named ‘Ready for Start’. The name of the fifth state is ‘In Use’. It depicts the period of time in which candidates interact with the system. This is also the period of time in which it performs tasks like automated grading on its own. The name of the final state is ‘Ready for Closing’. This state makes no assumptions on whether the whole system will actually be closed or whether it is just the assessment that is closed and archived. However, it is assumed that any remaining steps of the process will not require any more interaction with the assessment system.

3 Sample Workflow Definitions

To demonstrate how to define processes based on the Kernel for Educational Assessment we consider a summative e-assessment such as an electronic exam. This scenario is based on practical experience of the author with exams held several times a year. In this scenario, candidates come to the exam hall that is equipped with computers and an appropriate e-assessment system. There is no need to provide direct feedback to the candidates while they are present in the exam hall so that solutions can be graded asynchronously. In fact, this scenario requires a quite large and complex workflow. However, it can be described in very lean and concise way, as the next sections will demonstrate.

The technique used to model the process is to group states from several alphas into a phase and define the process as a linear sequence of phases. One phase can cover more than one state of a single alpha, but there may also be alphas that do not contribute one of their states for a particular phase. There are other ways to model processes as well, e.g. linking alpha states via activities, but as neither activities nor activity spaces have been discussed in this chapter, this way of modelling is also ignored here. The process model thus comes close to what is suggested as ‘big picture of assessment’ as suggested in Banta and Palomba (2015).

Our scenario of a summative e-assessment can be described using five phases:

• 1. The preparation phase contains all states that are considered while planning the assessment. While scope, shape or the number of people involved in the assessment are not clear at the beginning of this phase, most of these bounds and circumstances should be made clear during this phase. However, states dealing with details that are considered of minor importance can be deferred to later phases. On the other hand, any state bearing major decisions about cancelling the assessment should be included into this phase, as cancelling later will result in wasting significant amounts of work.

• 2. The construction phase contains all states that relate in some way to the production of resources and artefacts needed during the assessment. It can be assumed that a significant amount of time will be spent on tasks arising from this phase. Any state that must be completed before that assessment starts should be placed in this phase at the latest.

• 3. The conduction phase represents the time frame in which the actual assessment is conducted. Thus, all states related to delivering tests, collecting submissions and monitoring the assessment should be grouped in this phase. In particular, this is most likely the only phase in which the candidates have direct contact with the assessment.

• 4. The evaluation phase bundles states related to assessing submissions or answers from the candidates and generating feedback. From the didactic point of view, this is one of the most important phases, as this phase produces the actual outcome of the assessment and thus contributes much to its overall value. Depending on the domain of the assessment, the test item types used and the mechanisms applied for grading, this phase can consume a lot of time in the whole assessment process. As we assumed asynchronous grading for our sample process, this phase is clearly distinct from the conduction phase.

• 5. The review phase is considered to be the final phase in the assessment process. It should cover both legal and organizational post-processing and also tasks on documenting how well the assessment process actually worked. It is likely that some people who have been involved in the assessment process so far have no duties in this phase and thus can leave the process early. Consequently, some alphas may have reached their final state already in an earlier phase and do thus not contribute to the review phase.

The resulting process description in terms of alpha states assigned to phases is depicted in Fig. 10. All alphas including the optional ones are used, as we employ an electronic system and have to involve the exam authorities. Notably, we can skip the alpha ‘System’ from the process and retain a process that represents a traditional written exam that is graded manually after conduction.

Fig. 10

Overview on the assessment process for a summative e-assessment using five phases. The process assumes the application of asynchronous grading, so evaluation happens in a separate phase after conduction

Although this is a concise representation of a complex process, the process itself is not very lean. However, the kernel and the phase model can also be used to represent much more lightweight processes by skipping not only optional alphas, but also some more alphas and also particular states of alphas. To illustrate this, we consider a second scenario in which an assessor interacts spontaneously with some candidates just where they are. This is what many educators do when running lab exercises or alike. In contrast to the scenario used before, we can expect to see a very lean workflow here. Consequently, it is quite unlikely that a process description for this scenario will be used to guide the assessor in this process, but it can be used descriptively to explain what is going on.

The process differs in several points from the one discussed so far: First, we can exclude alphas ‘Authorities’, ‘System’ and also ‘Location’, as the assessment is informal, includes no e-assessment system and can happen anywhere. Second, we can exclude several states of some of the involved alphas: As the assessor interacts with the candidates who are just present, we can exclude the first two stages of alpha ‘Candidates’. Thus ‘Present’ is the first state for candidates to be considered in this process. With similar arguments, we can also exclude state ‘Identified’ for alpha ‘Organizers’. Third, the scenario poses less strict requirements with respect to verification and review of assessment contents. Hence, we can exclude the last two states for alpha ‘Test Items’ as well as the final state for ‘Organizers’. The resulting process description in terms of alpha states assigned to phases is depicted in Fig. 11.

Fig. 11

Overview on the assessment process for a lightweight ad hoc assessment using just two phases. The very informal setting allows to skip the alphas ‘Authorities’ and ‘Location’ from the process description. Also ‘System’ can be skipped as this assessment is not considered to be an e-assessment

The remaining states of the five alphas can then be grouped into just two phases. The construction phase consequently contains the first two states for ‘Test Items’, ‘Test’, ‘Grades & Feedback’ and ‘Organizers’. It thus describes the time frame in which the organizer thinks about doing the assessment and plans what to ask. As we assume this scenario to be a spontaneous assessment, no preparations have happened before. Candidates are not involved in this phase. The other phase is the conduction phase in which only ‘Test’, ‘Grades & Feedback’ and ‘Candidates’ are involved in terms of changing states. This phase is rather similar to the one seen above besides the fact that state ‘Satisfied’ for alpha ‘Candidates’ is also included here. The idea is that in an ad hoc assessment, any appeals are handled directly and thus no formal review phase is needed. As already discussed above, the organizer is also not interested in detailed verification and review. Thus, the respective states from the review phases in the other case studies are simply skipped here.

One could think of making the process description even smaller by skipping state ‘Dismissed’ for alpha ‘Candidates’. This would stress the point that the assessment can happen anywhere and candidates are not required to come to a certain location (and consequently leave it later). On the other hand, one can understand the state ‘Dismissed’ also in a less literal way and consider a candidate dismissed once the organizer stopped asking questions to this candidate. Notably, the ESSENCE standard allows to make customizations to states in terms of adding or removing checklist items. Consequently, one could define an even more fine-grained adoption of the kernel for this particular scenario by changing the checklists but keeping the overall idea of each of the alpha states included in the process description.

4 Tool Support

The ESSENCE standard defines the notion of cards for each alpha state. This idea was also used throughout this chapter to present the different alphas. Hence, a very native way of tool support is to print out small cards and use them on a pin board or desk to arrange them into phases and tick off checklist items. However, this is hardly practical for educators who have to prepare and run several different assessments in parallel. Electronic tool support can be considered much more practical in this case. On the other hand, tool support in terms of strict workflow engines is less appropriate when agile processes are to be used.

A simple web application that provides an overview on a process but deliberately provides no automated enactment of processes is shown in Fig. 12. It is based on an industry tool for software engineering process management (Brandt, Striewe, Beck, & Goedicke, 2017) that can be used with different kernel and process descriptions based on the ESSENCE standard.

Fig. 12

Web application showing an overview on the sample process from Fig. 10. Phases run from left to right here instead of top to bottom. Users can click on the small boxes to get a detail view for each alpha state and to tick off checklist items

There are two ways how educators can use this tool to handle their workflows: First, they can use it to provide a concise description of their assessment process. This can be helpful when discussing processes with colleagues or comparing different assessment workflows. At the same time, they can use it to customize their workflows by moving states to different phases, hiding alphas or adding specific checklist items. As the tool does not provide any formal workflow engine as other tools do, it also does not make any constraints on how educators can change a process description.

Second, educators can use the tool to enact and monitor their workflow by ticking off checklist items and thus tracking progress. The tool indicates fulfilled checklist items and states with different colouring, so educators always can see what is already done and what has to be done next. Again, the tool does not force them to perform a particular activity at a particular point in time, but allows an agile enactment in which the educators set their own goals. The tool also allows working collaboratively and thus sharing the responsibility for a particular assessment process. Practical experience with staff responsible for managing assessment processes in universities revealed that they prefer using a tool like this with customized workflows over using general workflow management tools. They also preferred using a tool like this over managing assessment processes by hand or with standard office tools.

5 Summary and Discussion

This chapter introduced the Kernel for Educational Assessment and demonstrated how to model lean and agile assessment workflows. The kernel contains a universal set of elements that can be used as building blocks for individual workflow descriptions. Each element is small and has a concise representation. Practical experience as expressed in the two sample processes shows that there are enough elements for complex workflows. At the same time, the set can also be stripped down to a very small number of elements used in very lightweight assessment processes. The notion of states and checkpoints can be used both for describing an assessment process and for monitoring the workflow while enacting it.

Notably, the process of stripping down a complex workflow by removing alpha states looks very mechanic. This seems to be an interesting antithesis to the ideas of agile processes on the first glance. However, one has to watch the different meta-levels: The way of describing processes is somewhat mechanic. The processes themselves can be as complex or lean as needed and can be changed in an agile way whenever needed.