Managing Uncertainty Through Experiment-Based Validation

Abstract

Two types of mistakes can often be observed in strategy design processes. The first is, executives believing that they know their customers better than customers do know themselves. This leads to offerings being developed that nobody wants, or nobody is willing to pay for. The second big mistake often observed, on the opposite end of the scale, is decision takers only being willing to decide if they are 100% convinced that change will be successful. A key feature of design thinking for strategy to address these mistakes is experiment based validation with real customers. During designing the detailed business model, choices are made based on sound assumptions. Although strategy designers believe in the assumptions they make, that does not necessarily mean that these assumptions are true. Assumptions must be validated. To do so validation experiments are designed. They primarily focus on trying to invalidate the made assumptions rather than confirm the already known, in line with the credo fail fast, to succeed faster. Experiments are prioritized based on their probability to fail and significance for the validity of the prototyped business models. The outcome of the validation phase is one or more business model prototypes that are desirable, feasible, and economically viable.

Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day—Jeff Bezos

Two types of mistakes can often be observed in strategy design processes. The first is, executives believing that they know their customers better than customers do know themselves. This leads to offerings being developed that nobody wants, or nobody is willing to pay for. Typical examples were the DAT¹ offering music in a digital format on an analog cassette medium, or the personal digital assistant Newton,² Sony’s Betamax video tape format, Nissan’s Murano CrossCabriolet, McDonalds McWings, New Coke, Microsoft Zune, or 3D television sets, just to name a few. CB Insights (2018) identified that 42% start-ups fail because there is no market need for their products and services. The second big mistake often observed, on the opposite end of the scale, is decision takers only being willing to decide if they are 100% convinced that change will be successful. They ask for validation after validation in an attempt to remove any business risk from their decision making. This can often be traced back to significant above average risk aversion, a trait that is at odds with successfully leading organizations into the future.

The challenge any strategy designer is facing is finding a compromise between not moving ahead fast enough and taking too much risk. The validating step of the business model layer aims at supporting decision takers by providing enough evidence to convince them that it would be very hard to come up with additional evidence that would make them change their mind. The focus of the validation step is on reducing risk down to an acceptable level. That does not and should not mean completely eliminating risk . Validating assumptions is forward-looking and should not be confused with hypothesis testing , as known from statistics theory, which, by nature, is related to extrapolating the past into the future.

1 Validating Objectives

While designing the detailed business model using process D (Chap. 10), choices are made based on sound assumptions. Although strategy designers believe in the assumptions they make, that does not necessarily mean that these are true. Assumptions should be validated earlier rather than later during the strategy design process. Validating assumptions early avoids possible costly mistakes later on.

There exist two possible approaches to validate formulated assumptions:

• (1) The assumption is translated into a quantifiable hypothesis that can be tested using statistical methods and algorithm (Kuehl 2000). This is the typical approach used in academic research. Statistical hypothesis testing relies on historical data and is inherently backward-looking, making it inappropriate for achieving the forward-looking goal of strategy design.

• (2) The assumption is related to a design decision in the detailed business model, either directly to a specific element, to a relationship between elements, or to the environment. Rather than relying on historical data to validate the assumption, judgmental insights are gathered up to the point where the marginal added knowledge from any additional insight on the validity of the assumption becomes nearly zero. Judgmental validations take a forward-looking stance and aim at getting first-hand insights. Judgmental insights go beyond a simple yes or no answer. Getting to an 80% certain positive answer with sound explanations is preferred over a 90% certain answer without such explanations.

In design thinking, judgmental validation comes to application. Validation is used as a decision support tool rather than a truth finding mechanism. The focus is on managing the uncertainty related to strategy decisions rather than getting them exactly right. Strategy includes, by definition, a certain degree of uncertainty. Statistical hypothesis testing can be used in a simplified form, if the strategic goal is to extrapolate the past into the future. This may be the case for incremental or fast-follower strategies.

2 The Validating Process

The assumption validating process V is by its nature a forward-looking confirmatory process. Even though it is not trivial and requires significant experience to formulate sound testing experiments, validation is systematic and straightforward. This sometimes leads to the fatal mistake being made, believing that validation can be performed by junior staff or outsourced to external consultants. As the primary goal of validation is to support strategic decision making, decision takers involved in the decision-making process should also be involved in the validation process. If executives have heard first hand from a customer that a given idea, the assumption, is valid or invalid, they will be much more confident into the associated decision than if that information would have been relayed to them by a third party.

To ensure success of validations, it is necessary to educate and coach the decision takers to be involved in validating assumptions, especially, if they did not participate in the designing step of the strategy design process. Having decision takers perform mock-up experiments jointly with strategy designers provides the necessary confidence for both parties that the validation outcome can be relied upon. A final caveat to note is that validation activities are not and must not be considered sales activities. The goal is getting objective feedback that helps deciding and not convincing someone that a given idea or a new offering is great. This is hard but must be ensured at all cost. Therefore, owners of an idea should not play a leading role in its validation.

Process V—Validating

• V.1 Formulating assumptions

• V.2 Classifying assumptions based on their impact on desirability, viability, and feasibility and prioritizing them relative to their relationship to the strategic focus, their design impact, their validation costs, and their strategy risk

• V.3 Designing experiments to validate/invalidate the assumptions made

• V.4 Performing the designed experiments

• V.5 Deriving consequences from the experiments’ outcome on the designed detailed business model, its elements, and connections

• V.6 Testing the desirability, viability, and feasibility of the detailed business model as a whole using a top-down perspective

Although described as a separate process, validation is an integral part of designing the detailed business model. Each time an assumption, whose validity is key for the next design decision, is made, it should be validated as soon as sound. Validations should not solely be scheduled at the end of the business model layer. Sometimes it is sound to prefer an early validation of an idea using a simplified experiment over a full-fledged experiment at a later stage. This is especially the case when the validity of the assumption significantly impacts subsequent design decisions. The full-fledged validation of such assumptions may be combined with related validations later on. Determining the timing and effort required for each validation activity is a key skill a strategy designer must exhibit. It is a trade-off between

• the impact of the assumption on subsequent design decisions in the detailed business model,

• the cost, with respect to time and money, of performing an experiment to validate the assumption, and

• the permitted strategy risk or uncertainty underlying the resulting detailed business model,

as illustrated in Fig. 11.1.

Fig. 11.1

Trade-offs relevant for prioritizing assumption validations

If the design impact is minor and/or validation costs are excessive, it may sometimes be sound to accept the associated strategy risk and not validate a given assumption, only validate it later in the strategy design process, or combine its validation with one or more related assumptions at a later stage.

3 Formulating Assumptions

The first step in reducing the uncertainty behind the designed detailed business model or elements of it, is to formulate assumptions . An assumption is a belief related to the future that may or may not be true. There exist three kinds of assumptions to consider:

• (1) Element-based assumptions . Are the descriptions of specific elements of the detailed business model valid?

• (2) Relationship-based assumptions . Are the descriptions of the relationships between elements of the detailed business model valid?

• (3) Externality-based assumptions . Are the assumed causalities between externalities and the descriptions of the specific elements of the business model valid?

Each detailed business model is based on many assumptions. To avoid unnecessary analysis, only those assumptions

• that have a material impact on the validity of the detailed business model, and

• for which the confidence is insufficient to accept the consequences of an incorrect decision

should be validated. A typical detailed business model contains between 10 and 30 assumptions to be validated.

Example Consider a suburban retail bank having chosen an offerings-based strategic focus by becoming a pure digital bank. The strategic focus aims at making the live of customers easier through solely relying on technology, such as mobile apps, to deliver the offerings. Figure 11.2 illustrates excerpts of a prototyped detailed business model.

A typical element-based assumption to validate is “There are sufficient home owner families requiring mortgage financing in the suburbs covered by the bank (CS and CJ elements).” The assumption can even be extended to whether that customer segment, in addition to being large enough, is growing and currently under-serviced. A relationship-based assumption underlying the detailed business model is “The targeted customers (CS element) are willing to do all their payments (CJ element) via their mobile phone (OVP and OPS elements).” The assumption “Gas stations (externality) are willing to function as human serviced ATMs (OVP element) for a fee (FR element)” represents a typical externality-based assumption. The assumptions whether there is a market for mortgages, given a sufficiently large home owner customer segment, is a typical assumption that needs no validation, unless the neighborhood is very rich (externality) and its residents do not finance their home ownership through mortgages. Relationship-based assumptions are the most common ones to validate, followed by externality-based, and element-based ones.

Fig. 11.2

Excerpts from a prototype of a detailed business model describing a suburban retail bank developing a customer-centric purely digital strategy

4 Classifying and Prioritizing Assumptions

Not all assumptions are created equal. The relevance of an assumption depends on

• whether or not it supports the firm’s strategic focus, and

• whether or not it is relevant for the firm’s detailed business model to be desirable, viable, or feasible.

Once the assumptions have been formulated, they are classified in categories using the two-dimensional framework illustrated in Fig. 11.3, focusing on the primary impact on the success of the detailed business model with respect to desirability, viability, and feasibility, on the x-axis and on whether or not the assumption relates to the strategic focus on the y-axis. The classification helps distinguish between critical and non-critical assumptions. Non-critical assumptions that are found invalid can, in general, be fixed rater easily. Therefore, they do not require high priority attention, especially if they are hard or costly to validate. In addition, assumptions without any expected material impact on the detailed business model can often be ignored.

Fig. 11.3

Classification of assumptions in categories based on their strategic relevance and impact on success

To ensure a cost efficient and effective validation, relevant assumptions classified in the same category should be prioritized based on two criteria, that is,

• what effort, in terms of time and money, is required to test the assumption, and

• how significant would the impact of a failed assumption test be on the validity of the detailed business model as a whole.

Figure 11.4 illustrates a framework describing the resulting order in which assumptions belonging to the same category should be validated.

Fig. 11.4

Matrix for prioritizing assumptions falling into the same category

5 Designing and Conducting Experiments

Once assumptions have been formulated, classified, and prioritized, the creative work around validation starts. For each assumption or cluster of assumptions, an experiment must be designed. Similar assumptions may be clustered together and validated using a single experiment. Sometimes, multiple complementary experiments may be necessary to validate a single assumption.

An experiment to validate an assumption consist of five parts:

• (1) A closed-end formulation of the assumption that allows for a yes or no answer, avoiding as much as possible a maybe answer.

• (2) An experiment to be performed for finding out if the answer to the assumption is yes or no.

• (3) A representative and reasonably sized target informant population on which to perform the experiment.

• (4) A measurement criterion that translates the outcome of the experiment into a yes or no answer related to the assumption.

• (5) A threshold on the measurement criterion that allows accepting or discarding the assumption.

Example Table 11.1 illustrates the description of a typical experiment used for validating assumptions regarding the digital retail banking example from Fig. 11.2. The measurement criteria can be defined in an incremental way, rather than as an absolute figure. Having at least 80% of positive or negative responses, is considered conclusive. In a first stage, 25 informants are questioned. If the outcome is not decisive, the threshold is slightly reduced, for example, to 75%, and the number of additional informants in the target population increased by 10, and so forth. Such an approach allows validating assumptions with minimal effort, as an increased effort is only needed when a heightened uncertainty exists.

Table 11.1 Description of a sample experiment to validate the assumption that homebuyers are willing to contract their first-time mortgage via a mobile app

Experiments should adhere to the 5 × 5 × 5 rule (Schrage 2014), that is, require no more than 5 weeks to be performed, cost no more than $5000 (or equivalent in local currency), and require no more than 5 strategy design team members and decision takers participation. For low effort and/or low impact assumptions, the rule may be simplified to 5 × 5 × 2, that is, no more than 5 days, $500, and 2 strategy designers and decision takers involved.

Designing experiments is a forward-looking and creative process. Academic insights into experiment development is often of lesser relevance due to its backward-looking nature. Using external support for designing, but not performing, experiments often proves to be of value as it allows for a fresh view and avoids potential confirmatory biases in the designed experiments. The primary goal of any experiment design should be on attempting hard to invalidate the to be tested assumption, rather than confirm its validity.

5.1 Typical Experiments

Although the space for designing experiments is limitless, typical experiments fall into one of the four categories, in decreasing order of their relevance, that is,

• feedback around mock-ups or prototypes,

• confirmatory interviews,

• split tests, or

• traditional surveys.

All experiments have in common that their outcome is only as good as their design. Enough time must be allocated for their development. Any experiment should be tested on a mock-up population before being administered to informants in the target population.

5.1.1 Mock-up or Prototype Feedback

Mock-up or prototype experiments present the informant a mock-up or a prototype of the assumption to be validated. Prototypes, whether physical or mental, are used instead of questions. The informant should be able to play around with the prototype and give feedback on its validity.

Mock-up based experiments are especially useful to validate offering features and distribution channels. They are regularly preferred over interviews to validate user experiences, as they avoid potential biases introduced by questioning.

Example Going back to the digital bank example illustrated in Fig. 11.2, testing if customers would be willing to buy their mortgage on-line, a key assumption behind the designed detailed business model, a mock-up-based experiment could be used. A sequence of screen masks would be presented to the informant to navigate through the mortgage application process to find out if such an approach would appeal.

5.1.2 Confirmatory Interviews

Confirmatory interviews , in contrast to explanatory or ethnographic interviews, focus on getting answers to close-end questions , rephrasing the assumptions to be validated. They focus as much on validating assumptions as on understanding the answers.

As with ethnographic interviews, confirmatory interviews start with putting the informant at ease to ensure that the answers are comprehensive and trustworthy.

Typical confirmatory interviews include questions along five dimensions:

• (1) Dou you agree or disagree with the assumption?

• (2) Why do you come to your conclusion? Which insights impact your decision most? Which insights did you discard or consider irrelevant?

• (3) What would make you change your mind?

• (4) What missing information could solidify your opinion?

• (5) What attributes underlying the assumption were irrelevant to your decision making and could subsequently be ignored?

Answers to these questions allow not only testing the assumptions, but also understanding how the detailed business mode could be updated to better meet the formulated assumptions, if validated, or adjust it to address identified issues.

Example Consider the assumption that customers want to be able to retrieve cash at any time from their bank account, as suggested in the example in Fig. 11.2. Assume that the informant does not agree with the assumption. He may comment his answer by indicating that what is important to him is the possibility to get cash early in the morning in order to be able to pay for a coffee on his way to work (his job-to-be-done) or get cash late in the evening to pay for the home-delivered pizza (his job-to-be-done). These insights may be used to rephrase the “at any time cash availability” assumption, by a 6am to midnight alternative or even add to the gas station cash withdrawal option, a pizza boy-based cash home delivery service. In addition, the closeness to the location to get cash may be described as more important than the nature of the location, that is, gas station. A grocery store with extended opening hours may be an acceptable alternative. Note that in contrast with the designing step, validating is not about coming op with alternative payment methods, but validating is about how and when customers want to retrieve cash.

5.1.3 Split Testing

Split testing experiments, either through simple A/B testing (Siroker and Koomen 2015) or more sophisticated multi-variate testing (Izenman 2008), are used when the assumptions lead to validating possible alternatives, rather than answering pure yes or no question. Split tests are commonly used to test assumptions around

• offering features,

• packaging and combination of characteristics, and

• pricing models.

Example A split test may be used to find-out whether customers are willing to pay up-front, prefer payment in installments, or pay only after the product or services has been fully delivered.

Split test experiments are easy to design and allow going beyond a simple yes-no answer. They can be included in interviews or administered through surveys. On the downside, split-tests often lack by design the insights that can be gained from confirmatory interviews. Sometimes split tests may be complemented with confirmatory interviews, especially if initial results are inconclusive.

5.1.4 Surveys

Surveys are the most common and easiest to administer type of validation experiments. A large informant population can be reached with minimal effort. Even more important than in other experiment approaches, is the quality of the formulated question used to test the assumption. As the informant filling out the survey usually cannot be observed and ask understanding questions, the surveyed questions must be structured in a way that ensures honest and complete answering. Biases must be avoided. Confirmatory questions, rephrasing previous questions, should be included to test for consistency of the answers.

Example Figure 11.5 illustrates a subset of a questionnaire administered via an online tool to validate the assumptions behind the detailed business model of the digital bank example in Fig. 11.2. Administering a validation survey for a digital bank strategy via an online platform introduces an informant selection bias that must be addressed, for example, by administering the survey to a random subset of informants through phone or via paper forms or by conducting interviews.

Fig. 11.5

Excerpt from a survey used to validate assumptions around the digital bank detailed business model example illustrated in Fig. 11.2

6 Validating Desirability, Viability, and Feasibility

Up to now, validation has focused in a bottom-up way, on individual assumptions behind elements of the detailed business model, relationships among them, and interdependencies with the environment. To ensure consistency of the designed detailed business model in a holistic way, the validation process V concludes with a set of top-down tests focusing on ensuring desirability, viability, and feasibility.

To validate desirability, viability, and feasibility, distinct experiments must be performed as these three test areas are complementary and only provide minimal overlaps. Significantly different approaches to experimenting are required. Validating that a give feature is desirable requires getting objective feedback from end-users, whereas validating viability of the same features is related to finding out if the purchase decision maker, rather than the end-user, is willing to pay for the feature. It is a mistake to assume that if a business model is desirable, it is automatically viable and feasible, and vice versa.

6.1 Validating Desirability

The desirability requirement of a detailed business model, that is, the offerings satisfy customer needs and support one or more of their jobs-to-be-done, can be validated by testing three high-level categories of assumptions:

• (1) There exist enough customers in the targeted customer segments. The customer segments are expected to grow over time, or at least, not shrink.

• (2) The firm can build a relationship with the targeted customers in a way that the firm’s offering falls into the customer’s consideration set.

• (3) The value proposition offered by the firm covers enough attributes of the target customers’ jobs-to-be-done to trigger a buying decision.

Validating these desirability assumptions can be done in a comparable way to testing other assumptions. Figure 11.6 illustrates the relationship between elements of a generic detailed business model and the assumptions. Competitive aspects of the desirability, for example, why a customer should favor the firm’s offering over that of competitors, are addressed in Chap. 12.

Fig. 11.6

Nine typical assumptions underlying desirability, viability, and feasibility, of a typical detailed business model

6.2 Validating Viability

Not all desirable detailed business models are also viable. A business model is considered viable if customers are willing to pay for the offered value proposition a price that exceeds the costs of producing and delivering the offering, including costs of capital. Key assumptions to validate the viability are:

• (4) Customers are willing to pay a given price for the offering to satisfy their jobs-to-be-done. This price allows the customer to perceive receiving sufficient value from the offered value proposition to trigger a buying decision.

• (5) The expected revenues exceed the incurred costs, that is, the sales price is appropriate from the firm’s perspective.

• (6) A sufficiently large number of customers are willing to buy the offering and pay for it such that the investment made as well as fixed expenses are covered.

6.3 Validating Feasibility

To be successful, a firm must be able to deliver upon the promises made to their customers with the value proposition. It must be feasible for the firm to produce the offerings at the quality level expected by the customers. Unless the detailed business model is based on untested inventions, the firm aims at disrupting existing offerings, or the firm is completely inexperienced in the target industry, feasibility is often the least hard trait to ensure. The three most important assumptions to validate the firm’s business model feasibility are:

• (7) The firm is able to identify and perform the activities required to deliver upon the promises made by the value proposition, that is, address the jobs-to-be-done of the targeted customers.

• (8) Sufficient resources are available at reasonable costs allowing the production and delivery of the offerings in the quality expected by the customers.

• (9) The firm is able to use key assets and resources in an efficient and effective way, minimizing the risk of failing, to produce the offering in a desirable and viable way.

It is important to ensure that the assumptions behind desirability, viability, and feasibility remain valid not only at a given point in time, but throughout the lifetime of the detailed business model and associated strategy.

7 Risks to Avoid

At the end of the validation step, insights gained from experimenting should be fed back into the detailed business model layer of the strategy design process, mainly the design stage. Before doing so, three key traps must be avoided, that is,

• the false positive bias,

• the false negative bias, and

• the wrong data trap.

In statistics theory, the first two traps are called type I and type II errors respectively. Particular care must be taken when selecting an unbiased sample of informants to avoid all three traps in forward-looking business model validation experiments. One way of doing so, is having the experiment set-up and related data reviewed by an outside expert in experiment validation. This is especially important for those assumptions that make or break the detailed business model designed.