Innovation and New Products Research: A State-of-the-Art Review, Models for Managerial Decision Making, and Future Research Directions

Abstract

This chapter has a three-fold purpose. First, we provide a literature review of major papers in the field of new products research. We organize our review into four tables, one for each of the four stages of the new product development process, and then by topic within each stage. We provide a short summary of each paper in the tables. Second, we highlight specific models within each stage of the new product development process. These models are useful for marketing researchers and managers tackling challenges in the new products domain. Third, after reviewing the literature, we suggest numerous general research directions as well as specific research questions to guide future investigations in this area.

3.1 Introduction

Innovation is the lifeblood of economies. It spawns new firms, revitalizes established organizations, enriches entrepreneurs, builds professional reputations, and raises living standards throughout the world. Over the years, innovation has received considerable attention in the marketing literature, which is not surprising given its importance to both companies and consumers.

Our purpose in writing this chapter is threefold. First, we provide a literature review of major papers in the field of new products research. We organize our review into four tables, one for each of the four stages of the new product development process, and then by topic within each of these stages. Moreover, we provide a short summary of each paper in the tables. These tables and short summaries provide an overview of research and findings in the field, plus direct readers to papers particularly suited to their interests. Second, we highlight specific models within each stage of the new product development process. These models are useful for marketing researchers and managers tackling challenges in the new products domain. Third, after reviewing the literature, we suggest numerous general research directions as well as specific research questions to guide future investigations in this area. We believe this will be particularly useful to those new to the field of new products research , especially those interested in applying quantitative models (e.g., business school Ph.D. students, consultants, practitioners).

Several researchers have previously documented the innovation subfield of the marketing discipline.¹ In addition to this chapter, we refer interested readers to chapters by Lehmann and Golder (2014) and Rao (2014), prior journal articles by Hauser et al. (2006) and Peres et al. (2010), and the entire Handbook of Research on New Product Development (Golder and Mitra 2017). Next, we briefly summarize these writings, and then provide the motivation and positioning of our chapter.

Lehmann and Golder (2014) wrote their chapter on new products research for the History of Marketing Science. In it, they organize important new products research beginning in the 1960s going through the 2010s. They find that research is limited on the topics of opportunity identification and concept generation. However, they find that researchers have paid greater attention to design and development (particularly conjoint analysis ), sales forecasting , and some aspects of strategy (e.g., competitive response, order of entry).

Rao (2014) provides a focused, comprehensive review of conjoint analysis, in contrast with other authors who discuss conjoint analysis as part of the broader new products literature. Rao’s (2014) chapter updates earlier reviews by Wittink and Cattin (1989), Green and Srinivasan (1990), and Wittink et al. (1994).

Hauser et al. (2006) organize research on innovation into five fields of inquiry: (i) consumer response to innovation (e.g., consumer innovativeness, new product growth models, network externalities), (ii) organizational impact (e.g., contextual drivers, adoption of new methods), (iii) market entry strategies (e.g., technological evolution, portfolio management), (iv) prescriptive techniques for product development (e.g., fuzzy front end, design tools), and (v) outcomes from innovation (e.g., market rewards, defending against new entrants, internal rewards for innovation).

Peres et al. (2010) review research on innovation diffusion. Building on prior reviews (e.g., Mahajan et al. 1995), they incorporate more recent research on interpersonal influence, network externalities, and social signals. Their review covers within-market effects, cross-market effects, and cross-country effects for all social influences, not just word of mouth.

The Handbook of Research on New Product Development (Golder and Mitra 2017) contains 19 chapters that provide depth on specific aspects of new product development and innovation. These chapters describe the frontiers of new products research as well as offer numerous insights for extending these frontiers of our knowledge base.

Empirical findings across a range of new products research studies can be found in Empirical Generalizations about Marketing Impact (Hanssens 2015). Topics in this book of interest to new products researchers include customer satisfaction and product reviews, objective and perceived quality, order of entry, sales diffusion and social influence, product innovation , and competitive reaction.

Our chapter for this handbook provides a unique perspective on innovation and new products research. In contrast with earlier works, we organize our review by the major stages of the new product development process: (i) opportunity identification , (ii) product design and development , (iii) sales forecasting , and (iv) commercialization . Our review is more comprehensive than Lehmann and Golder (2014) since their purpose was to provide exemplars of research themes illustrating the historical evolution of the field. One aspect of innovation covered in Lehmann and Golder (2014) that we do not cover is how to value innovations. Most importantly, we focus on models useful for managerial decision-making in various innovation and product development contexts. Therefore, we provide details of several models so that managers and researchers can better understand how these models could be used and the insights that can be generated for informing their decision making.

Overall, our chapter has three overriding objectives: (i) to organize the vast marketing literature on innovation and new products according to the four stages of the new product development process, (ii) to provide detailed discussions of decision-making models useful for each stage of the new product development process, and (iii) to identify the most promising opportunities for future research in each of these stages. Next, we discuss research categorized into the four stages of the new product development process. Within each of these stages, we elaborate on key models. We conclude with our agenda for future research.

3.2 Organizing Research on Innovation and New Products

The new product development process can be broken down into four stages: (i) opportunity identification , (ii) product design and development , (iii) sales forecasting , and (iv) commercialization . The first stage subsumes idea generation and idea screening, which are sometimes treated as separate stages. The final stage, commercialization, includes some research on post-commercialization activities but does not attempt to cover the vast literature on the product management of mature products.

3.2.1 Opportunity Identification

Current research on opportunity identification focuses on three areas: (i) opportunities identified by investigating lead users, (ii) opportunities identified by using online platforms, and (iii) opportunities identified through innovation templates (see Table 3.1). Research on lead users began in the 1980s and is largely associated with the work of Eric von Hippel. Lead users are consumers who have needs ahead of the general market and also create make-shift solutions for those needs. When firms can identify lead users, they benefit in two ways. First, they become aware of unmet needs that exist in the marketplace. These needs are important enough that consumers actively generate their own solutions. Second, potential new product ideas are generated through lead users’ solutions. While often these rudimentary solutions are not firms’ final solutions, they can be very useful in contributing to firms’ attempts to develop products for a broad market. Research has identified many successful market applications of the lead user concept.

Table 3.1 Opportunity identification

Another area of research on opportunity identification to emerge was the development and application of innovation templates. This research stream offered a structured approach to what might otherwise be an undefined process with random outcomes. For example, by identifying five templates of innovation that most successful products belong to, Goldenberg et al. (2001) show how future new products can be identified and refined more easily.

A third and more recent area of research on opportunity identification is the use of online platforms such as Dell’s Idea Storm (Bayus 2013). This research tends to investigate how the structure and process of online communities contribute to or detract from opportunity identification and idea generation. Unlike the previous two areas of research, we need better documentation of the marketplace success of new products generated through online platforms.

3.2.1.1 Modeling Opportunity Identification

The most common goal in this research area is to predict the success of new product ideas. Here, researchers have usefully employed logit models (e.g., Bayus 2013; Goldenberg et al. 2001). For example, Bayus (2013) used an individual-effects logit model where the dependent variable $(y_{it} )$ is a binary variable that indicates whether ideator i proposes an idea that is eventually implemented (1 for yes and 0 otherwise).

$$\Pr \left( {y_{it} = 1} \right) =\Lambda (\alpha_{i} + \beta x_{it} + \gamma z_{i} ),$$

(3.1)

where $\Lambda$ is a logit model $\Lambda \left( w \right) = e^{w} /(1 + e^{w} )$. The independent variables ($x_{it}$ and $z_{i}$) include the past success of this ideator in generating implemented ideas, the diversity of this ideator’s past comments, and control variables such as the ideator’s demographic information.

In addition to predicting the success of an idea, researchers are also interested in the number of ideas generated by an ideator. One model used in this context is the Poisson model. For example, Bayus (2013) uses an individual-effect Poisson model where the dependent variable $(y_{it} )$ is the number of ideas generated by ideator i. This count variable takes on positive integer values, which are assumed to follow the Poisson distribution. The model is specified as follows.

$$\Pr \left( {y_{it} } \right) = \lambda^{{y_{it} }} e^{ - \lambda } /y_{it} !$$

(3.2)

where $\lambda$ is an empirically-estimated parameter.

3.2.2 Product Design and Development

Once firms have identified potential opportunities, generated new product ideas, and screened those ideas for the most promising alternatives, firms need to move these ideas into product design and development. Unlike opportunity identification , product design and development has a rich and reasonably large literature in marketing (see Table 3.2 for an overview organized by topic). Much of this literature is due to one model-based area of research: conjoint analysis . Over several decades, many researchers have made important contributions to this literature, which in turn has had a substantial impact on management practice. Conjoint analysis may be the most impactful method developed in marketing research and it almost certainly is such within the area of innovation and new products research. Rao (2014) provides a focused, comprehensive review of conjoint analysis, which updates earlier reviews by Wittink and Cattin (1989), Green and Srinivasan (1990), Wittink et al. (1994), and Rao (2008).

Table 3.2 Product design and development

One of the key challenges in conducting conjoint analysis research is that so many product variants exist in multi-attribute product space. As a result, much of the research in conjoint analysis has dealt with methodological and practical approaches by asking consumers about a subset of alternatives while still being able to estimate their preferences across the multi-attribute space.

Besides conjoint analysis , researchers have proposed other approaches for incorporating customers’ preferences into product design and development . These approaches include quality function deployment (QFD) (Hauser and Clausing 1988) and voice of the customer (VOC) (Griffin and Hauser 1993). More recently, the idea of heterogeneous design or product morphing has been proposed as a way to take advantage of flexible production in order to better satisfy consumers’ specific preferences.

3.2.2.1 Models for Product Design

Homogeneous Product Design. Traditional models of product design often use McFadden’s (1974) random utility model of consumer choice where all consumers have the same value for attributes. Specifically, the utility $u_{sj}$ for alternative j in choice set s is

$$u_{sj} = x_{sj}^{'} \beta + \varepsilon_{sj} ,$$

(3.3)

where $x_{sj}$ is a vector of the attribute levels and $\beta$ captures the corresponding weights (or part-worths) consumers have for each attribute level. $\beta$ s are homogeneous across consumers and therefore assume that the product design is the same for all consumers. $\varepsilon_{sj}$ is an error term following an extreme value distribution. The maximization of consumer utility generates the probability that alternative j is chosen from choice set s:

$$p_{sj} = \frac{{{ \exp }(x_{sj}^{'} \beta )}}{{\mathop \sum \nolimits_{j = 1}^{J} { \exp }(x_{sj}^{'} \beta )}}$$

(3.4)

Recent studies incorporate consumer heterogeneity where consumers have different values for the attributes. Here, a common model is the mixed logit, where $\beta$ s contain random effects. For example, Sandor and Wedel (2002, 2005) assume that $\beta$ is multivariate normally distributed with mean $\mu_{B}$ and variance $\Sigma$. If $\Sigma$ is assumed to be a diagonal matrix, $\beta$ can be written as $\beta = \mu_{B} + V\sigma_{B}$ with vector $\sigma_{B} = \left( {\sigma_{1} , \ldots , \sigma_{m} } \right)^{'}$, and V is an $m\,*\,m$ diagonal matrix. Assuming utility maximization, the probability that product i is chosen from choice set s is

$$\varphi_{sj} = \mathop \smallint \nolimits p_{sj} \left( v \right)f\left( v \right)dv,\,{\text{where}}\,p_{sj} \left( v \right) = \frac{{{ \exp }\{ x_{sj}^{'} \left( {\mu_{B} + V\sigma_{B} } \right)\} }}{{\mathop \sum \nolimits_{j = 1}^{J} { \exp }\{ x_{sj}^{'} \left( {\mu_{B} + V\sigma_{B} } \right)\} }},$$

(3.5)

where $v$ is the vector containing the m diagonal elements of matrix V.

Determining the optimal product design requires evaluating the information matrix of this mixed logit model. A widely used measure of the information matrix is the inverse of the determinant of the Fisher information matrix, which is called the D-error (Sandor and Wedel 2002).

$$D{ - }error = { \det }[I(\mu_{B} , \sigma_{B} |X]^{ - 1/2K} ,$$

(3.6)

where $I$ is the Fisher information matrix and K is the number of attribute level combinations. Minimizing this D-error generates the optimal product design.

One limitation of the mixed logit model is the computation infeasibility when the number of possible designs is too large. To address this limitation, Liu and Tang (2015) propose a particular conjoint choice context, which can be specified as $\left\{ {\left( {C_{k} , w_{k} } \right)} \right\}, k = 1, \ldots , K$, where $C_{k}$ is a choice set and $w_{k}$ is the continuous weight for the choice set with the constraint such that $0 \le w_{k} \le 1$ and $\sum\nolimits_{k = 1}^{K} {w_{k} = 1}$. Instead of searching for the globally optimal product design, this approach finds a globally optimal continuous design by minimizing the D-errors over the entire space of continuous designs. It generates completely heterogeneous designs for each individual respondent in the choice experiment and more importantly, it is computationally feasible.

Conjoint Analysis . Traditional choice-based conjoint analysis relies on consumers’ evaluations of product attributes, resulting in the desirability of attribute levels and the importance of each attribute. However, there are difficulties when consumers need to evaluate a large number of product attributes at one time. Recent research extends the traditional stated preference methods and solves this problem by adaptively choosing a subset of attribute comparisons and interpolating the importance of other attributes (Netzer and Srinivasan 2011).

In particular, Netzer and Srinivasan (2011) ask respondents to allocate 100 points across attributes to reflect the importance of each attribute. To alleviate the problem of too many product attributes, they break down the constant-sum allocation across the full set of attributes into a subset of constant-sum allocations between two attributes at a time. In order to reduce the number of paired comparison questions, they propose an adaptive approach where respondents are asked to compare only a subset of all possible paired comparison questions. Then researchers interpolate the importance of the attributes not included in the subset of comparison questions. Meanwhile, researchers can select the attributes for the next comparison to minimize the interpolation errors. Then the importance ratios between two attributes ($r_{{j_{1} j_{2} }} = W_{{j_{1} }} /W_{{j_{2} }}$, where $W_{{j_{k} }}$ is the importance of attribute $j_{k}$) can be calculated from the paired comparisons. With the set of attribute importance ratios, a log-linear multiple regression is used to estimate relative attribute importance. The essence of their approach is to improve the estimation of individual-level attribute importances ($W_{j}$) in a way that avoids respondent overload.

3.2.3 Sales Forecasting

Forecasting can be done from the time an idea has been generated until after it has been commercialized. However, most research has focused on forecasting sales at the end of the product design and development stage.

Research on the sales forecasting of new products has been conducted in marketing for several decades. The most well known sales forecasting model in marketing is the Bass model which spawned a huge literature on diffusion models . Conceptually, these models are based on the sociological literature on diffusions of innovation , largely driven through a process of communications (e.g., Rogers 2003). We elaborate on this model in discussing one of its extensions later in this section.

The forecasting literature for non-durables primarily uses stochastic models applied to both first-time purchases and repeat sales. Another class of forecasting models, flow models, incorporate test market results to project initial sales to full-market sales. Regression models have also been used to predict new product sales. Yet another approach combines elements of flow models, regression models, and stochastic approaches to forecast the sales of new consumer nondurables. The well known ASSESSOR model has improved forecasting accuracy because researchers and managers can project laboratory results to market results. The most recent research on sales forecasting has either incorporated online-enabled approaches or focused on forecasting approaches for online environments (e.g., online word-of-mouth, online reviews, virtual markets, sentiment analysis, and blogs; e.g., Chevalier and Mayzlin 2006) (Table 3.3).

Table 3.3 Sales forecasting

3.2.3.1 Sales Forecasting Models

In this section, we provide details on diffusion model extensions and customer lifetime analysis.

Extensions of Diffusion Models. The standard Bass (1969) diffusion model incorporates a hazard rate whereby consumers who have not yet adopted a new product do so at time t:

$$h\left( t \right) = p + qF\left( t \right),$$

(3.7)

where F(t) is the proportion of consumers who have adopted this new product at time t; p captures the intrinsic tendency to adopt (coefficient of innovation ), which can be influenced by consumer characteristics, innovation appeal, etc.; and q measures the effect of social contagion (coefficient of imitation). The proportion of new product adoption at time t can be written as

$$f\left( t \right) = \frac{dF\left( t \right)}{dt} = h\left( t \right)\left[ {1 - F\left( t \right)} \right] = \left[ {p + qF\left( t \right)} \right][1 - F\left( t \right)].$$

(3.8)

The solution of this equation can be used to predict the cumulative penetration of a new product, in other words, the proportion of consumers who have adopted a new product at time t. Specifically,

$$F\left( t \right) = {{\left[ {1 - { \exp }\left( { - g - \frac{p + q}{t}} \right)} \right]} \mathord{\left/ {\vphantom {{\left[ {1 - { \exp }\left( { - g - \frac{p + q}{t}} \right)} \right]} {\left[ {1 + \left( {\frac{q}{p}} \right)\exp \left( { - g - \frac{p + q}{t}} \right)} \right]}}} \right. \kern-0pt} {\left[ {1 + \left( {\frac{q}{p}} \right)\exp \left( { - g - \frac{p + q}{t}} \right)} \right]}},$$

(3.9)

where g is a location parameter.

One limitation of the original Bass model is that consumers are homogeneous and influenced by the same factors. One extension is the asymmetric influence model (AIM) where two consumer segments, i.e., influentials and imitators differ in the factors that drive their adoption behavior (Van den Bulte and Joshi 2007). The hazard functions for these two segments are

$${\text{For}}\,{\text{influentials}}\,\left( {{\text{denoted}}\,{\text{as}}\, 1} \right),h_{1} \left( t \right) = p_{1} + q_{1} F_{1} \left( t \right);$$

(3.10)

$${\text{For}}\,{\text{imitators}}\,\left( {{\text{denoted}}\,{\text{as}}\, 2} \right),h_{2} \left( t \right) = p_{2} + q_{2} \left[ {wF_{1} \left( t \right) + \left( {1 - w} \right)F_{2} \left( t \right)} \right],$$

(3.11)

where w denotes the relative importance that imitators attach to influentials’ versus other imitators’ adoption behavior $(0 \le w \le 1)$. Assuming the proportion of influentials is $\theta$ and the proportion of imitators is $1 - \theta$, the overall cumulative market penetration is:

$$F_{m} \left( t \right) = \theta F_{1} \left( t \right) + (1 - \theta )F_{2} \left( t \right).$$

(3.12)

And the fraction of population adopting at time t is:

$$f_{m} \left( t \right) = \theta f_{1} \left( t \right) + (1 - \theta )f_{2} \left( t \right).$$

(3.13)

From Eqs. 3.6 and 3.7, the population hazard function can be derived as

$$h_{m} \left( t \right) = f_{m} (t)/[1 - F_{m} \left( t \right)] = [ \theta f_{1} \left( t \right) + (1 - \theta )f_{2} \left( t \right)]/(1 - F_{m} \left( t \right)).$$

(3.14)

As individual consumer level social interaction data become available, researchers have extended the Bass model by incorporating data such as social ties and new product recommendations among consumers (Toubia et al. 2014). Assume that consumer i receives $r_{it}$ recommendations in period t. $r_{it}$ is assumed to follow a binomial distribution specified as:

$$\begin{aligned} r_{it} & \sim Bin\left( {ties_{i} ,aF_{t - 1} } \right) \\ & \Rightarrow P(r_{it} | ties_{i} ) = \left( {\begin{array}{*{20}c} {ties_{i} } \\ {r_{it} } \\ \end{array} } \right)(aF_{t - 1} )^{{r_{it} }} (1 - aF_{t - 1} )^{{ties_{i} - r_{it} }} \\ \end{aligned}$$

(3.15)

where $ties_{i}$ is the number of consumer i’s social ties, $F_{t - 1}$ is the cumulative penetration in period t − 1, and a is the probability that a given tie would recommend the product to consumer i conditional on the tied consumer having adopted.

The hazard rate is influenced by the social recommendation $r_{it}$ and is specified as

$$h\left( {r_{it} } \right) = 1 - \left( {1 - p} \right)\left( {1 - q} \right)^{{r_{it} }}$$

(3.16)

where p and q have similar interpretation as in the Bass model.

Toubia et al. (2014) show that the aggregate diffusion process can be derived from Eqs. 3.9 and 3.10. Let $P(ties)$ be the probability mass function of the number of social ties, and let $f_{t}^{ties}$ and $F_{t}^{ties}$ be the marginal and cumulative aggregate penetration in period t among consumers with ties. The marginal penetration $f_{t}^{ties}$ is equal to the proportion of non-adopters,$1 - F_{t}^{ties}$, multiplied by the expected value of the hazard rate in period t among these consumers:

$$\begin{aligned} f_{t}^{ties} & = \left( {1 - F_{t - 1}^{ties} } \right)E_{{r_{t} }} \left[ {h\left( {r_{t} } \right) |ties} \right] \\ & = \left( {1 - F_{t - 1}^{ties} } \right)\mathop \sum \limits_{{r_{t} = 0}}^{ties} h\left( {r_{t} } \right)P(r_{t} |ties) \\ \end{aligned}$$

(3.17)

The Bass model can be further extended by incorporating the number of consumer recommendations (Toubia et al. 2014).

Extensions of Customer Lifetime Models. Another common model used to predict sales is the customer lifetime model , where statistical models predict how long a customer will stay with a company and how much he will buy (i.e., purchase rate). For more information about customer lifetime models and related issues, we refer to Chap. 10 of this Handbook: “Marketing Models for the Customer-Centric Firm” by Ascarza, Fader, and Hardie. The most influential model in this stream is the Pareto/NBD model (Schmittlein et al. 1987). Here, the time at which a customer becomes “dead” (i.e., no longer buy from a company) is denoted $\tau$. For any time T > 0, if the customer is still alive at T (so $\tau > T$), the number of purchases (x) in $(0, T]$ is assumed to follow the Poisson distribution:

$$P\left[ {X = x |\lambda ,\tau > T} \right] = e^{ - \lambda T} \frac{{\left( {\lambda T} \right)^{x} }}{x!};\quad x = 0,1,2, \ldots ,$$

(3.18)

where the purchase rate $\lambda$ is assumed to follow a gamma distribution:

$${\text{g}}\left( {\uplambda | {\text{r}},\upalpha} \right) = \frac{{\upalpha^{\text{r}} }}{{\Gamma \left( {\text{r}} \right)}}\uplambda^{{{\text{r}} - 1}} {\text{e}}^{{ -\upalpha \uplambda }} ;\quad\uplambda > 0;\quad r,\upalpha > 0.$$

(3.19)

The time $\tau$ until becoming “dead” is assumed to follow an exponential distribution:

$$f\left( {\tau |\mu } \right) = \mu e^{ - \mu \tau } ;\quad \tau > 0,$$

(3.20)

where the death rate $\mu$ is also assumed to follow a gamma distribution:

$$h\left( {\mu |s,\beta } \right) = \frac{{\beta^{s} }}{\varGamma \left( s \right)}\mu^{s - 1} e^{ - \beta \mu } ;\quad \mu > 0;\,\,s,\beta > 0.$$

(3.21)

With these equations, the purchases made while a customer is “alive” follow the NBD model and have the distribution:

$$P\left[ {X = x |r,\alpha ,\tau > T} \right] = \left( {\begin{array}{*{20}c} {x + r - 1} \\ x \\ \end{array} } \right)\left( {\frac{\alpha }{\alpha + T}} \right)^{r} \left( {\frac{T}{\alpha + T}} \right)^{x} ;\quad x = 0,1,2, \ldots$$

(3.22)

“Deaths” for a sample of customers follow the Pareto distribution:

$$f\left( {\tau |s,\beta } \right) = \frac{s}{\beta }(\frac{\beta }{\beta + \tau })^{s + 1} ,\quad r > 0$$

(3.23)

Overall, this combined purchase event/duration model is called the Pareto/NBD. One challenge with this model is its complicated likelihood function and numerous evaluations of the Gaussian hypergeometric function. To address this limitation, Fader et al. (2005) develop a beta-geometric/NBD (BG/NBD) model, a variation of the Pareto/NBD, which is easier to implement. Similar to the Pareto/NBD model, the BG/NBD model assumes that the number of purchases follows the Poisson distribution and the purchase rate follows the gamma distribution. The difference between the Pareto/NBD and BG/NBD is how/when customers become inactive. The Pareto/NBD assumes that customers can “die” at any point in time, independent of the occurrence of actual purchases, whereas the BG/NBD assumes that customers “die” immediately after a purchase. Specifically, the BG/NBD assumes that after any purchase, a customer becomes “dead” with a probability p which follows a beta distribution with p.d.f:

$$f\left( {p |a,b} \right) = \frac{{p^{a - 1} (1 - p)^{b - 1} }}{B(a,b)},\quad 0 \le p \le 1,$$

(3.24)

where B(a, b) is the beta function, which can be written as $B\left( {a,b} \right) =\Gamma (a)\Gamma (b)/\Gamma ({\text{a}},{\text{b}})$.

The point at which the customer “dies” is distributed across purchases according to a (shifted) geometric distribution with p.m.f

$$\begin{aligned} & P\left( {{\text{inactive}}\,{\text{immediately}}\,{\text{after}}\,j{\text{th}}\,{\text{transaction}}} \right) \\ & \quad = p(1 - p)^{j - 1} ,\quad j = 1,2,3, \ldots \\ \end{aligned}$$

(3.25)

Fader et al. (2005) show that the BG/NBD model can be estimated easily in Microsoft Excel and therefore is usable in most business applications.

Researchers also extend the Pareto/NBD model by making it more flexible and more powerful for sales prediction. For example, Bemmaor and Glady (2012) develop the gamma/Gompertz/NBD (G/G/NBD) model. Similar to the Pareto/NBD model, the G/G/NBD model assumes that the number of purchases follows the Poisson distribution and the purchase rate follows the gamma distribution. The difference between the Pareto/NBD and G/G/NBD is that the probability that a customer dies before time $\tau$ is assumed to follow a Gompertz distribution:

$$F\left( {\tau |\eta } \right) = 1 - \exp \left( { - \eta \left( {e^{b\tau } - 1} \right)} \right),\quad \eta ,b > 0, \tau > 0$$

(3.26)

Compared with the Pareto/NBD model, the G/G/NBD model is more flexible because the p.d.f. of the Gompertz distribution can be skewed left or right and it can exhibit a mode at zero or an interior mode. Bemmaor and Glady (2012) also show that the G/G/NBD predicts sales better than the Pareto/NBD model.

3.2.4 Commercialization

The final stage of the new product development process is commercialization. This stage has received much attention in marketing because it lies at the nexus of firm strategy and consumer response to new products (e.g., Boulding and Christen 2003; Golder and Tellis 1993; Min et al. 2006; Robinson and Fornell 1985). Research on commercialization can be categorized into several areas.

First, there is a large and rich literature on entry timing strategies. Initial research in this area found that market pioneers or first movers enjoyed substantial advantages over later entrants. However, this research suffered from several limitations including survivor bias and misclassifying successful firms. Correcting for these limitations resulted in the finding that market pioneers tend to have much higher failure rates, lower market shares, and lower rates of market leadership than previously believed.

Second, another stream of research has attempted to identify the factors associated with new product success. These include differentiation (both meaningful and seemingly meaningless) (Carpenter et al. 1994), introducing innovative product attributes (Shankar et al. 1998), and market characteristics like network effects (Wang et al. 2010). A related stream of research looks specifically at the marketing mix variables associated with new product success (e.g., Bruce et al. 2012; Kopalle and Lehmann 2006; Spann et al. 2015).

Finally, another stream of research examines how incumbents defend against new entrants. Much of this work is analytical (theoretical) in nature (e.g., Hauser and Shugan 1983). Some of these papers make strong prescriptive recommendations of strategies for firms to follow. Surprisingly, some empirical research shows that firms actually do very little to respond to competitors’ innovations (Table 3.4).

Table 3.4 Commercialization

3.2.4.1 Modeling Commercialization

In this section, we provide details on models of channel acceptance and customer lifetime value.

Channel acceptance of new products. When a new product is introduced to a market, there are two major challenges: how to position it so that consumers will choose it over alternative products and how to convince retailers to accept it. Luo et al. (2007) develop an approach to positioning and pricing a new product that directly incorporates the consumers’ preferences and retailer’s acceptance criteria.

The authors propose two frameworks of market estimation before and after a new product’s entry (see Figs. 3.1 and 3.2). In these two frameworks, consumers’ preferences are first estimated with a random utility choice model for a conjoint choice experiment with N individual consumers evaluating K choice sets with G alternative product designs. The utility of consumer i for product design g in choice k is defined as:

$$U_{i} \left( {x_{gk} ,p_{gk} } \right) = \left( {x^{'}_{gk} \beta_{ix} + p_{gk} \beta_{ip} } \right) + \varepsilon_{igk,}$$

(3.27)

where $x_{gk}$ is a vector of product attributes of design g, $p_{gk}$ is the price, and $\varepsilon_{igk,}$ is the random component of the utility. The probability of consumer i choosing design g can be derived from Eq. (3.1). Specifically, it is expressed using the logit expression.

$${ \Pr }_{igk} = \frac{{{ \exp }(x^{'}_{gk} \beta_{ix} + p_{gk} \beta_{ip} )}}{{\mathop \sum \nolimits_{{g^{'} = 1}}^{G} [{ \exp }(x^{'}_{g'k} \beta_{ix} + p_{g'k} \beta_{ip} )] + { \exp }(a_{i} )}},$$

(3.28)

where $a_{i}$ is the utility of no-choice option for consumer i.

Fig. 3.1

Source Luo et al. (2007)

Estimation of market specifics—before entry.

Fig. 3.2

Source Luo et al. (2007)

Market scenario development—after entry of design alternative.

The next step is to estimate wholesale prices and marginal costs of incumbent products. First, the wholesale prices can be determined by maximizing the retailer’s profits. In particular, before a new product is introduced to the market, the retailer’s profit maximization is specified as:

$$\mathop {\hbox{max} }\nolimits_{{p_{1} , p_{2} , \ldots ,p_{J} }} \pi^{r} = \left\{ {\sum\nolimits_{j = 1}^{J} {\left[ {m_{j} \,*\,\left( {p_{j} - w_{j} } \right)\,*\,S} \right]} } \right\} - sc\,*\,J,$$

(3.29)

where $\pi^{r}$ is the retailer’s profit, $m_{j}$ is product j’s market share, $w_{j}$ is its wholesale price, S is market size, and sc is the marginal shelf cost.

Alternatively, the marginal costs of incumbent products can be estimated by maximizing manufacturer’s profits. That is,

$$\mathop {\hbox{max} }\nolimits_{{w_{j} }} \pi_{j}^{m} = \left( {w_{j} - c_{j} } \right)\,*\,m_{j} \,*\,S - F_{j} \quad j - 1, \ldots ,J,$$

(3.30)

where $c_{j}$ is product j’s marginal cost, and $F_{j}$ is its fixed cost.

The manufacturer’s goal is to select a product design and a wholesale price so that the product will be accepted by retailers, and be more profitable than other designs. As shown in Fig. 3.2, we need to estimate the new market scenario after the entry of the new product by estimating new wholesale and retail prices. The procedure includes solving two optimization problems iteratively: the retail profit maximization problem (second block in Fig. 3.2) and the manufacturer profit maximization problem (third block in Fig. 3.2). The maximization equations are similar to Eqs. 3.29 and 3.30 but with new wholesale and retail prices.

An Application of Customer Lifetime Value (CLV). When a new product is introduced to the market, managers can use customer lifetime value (CLV) to identify and target the most profitable customers. Customer lifetime value (CLV) is the present value of all future profits obtained from a customer over his life of relationship with a firm. It is specified as (Gupta et al. 2004; Reinartz and Kumar 2003):

$$CLV = \sum\nolimits_{t = 0}^{T} {\frac{{\left( {p_{t} - c_{t} } \right)r_{t} }}{{(1 + i)^{t} }} - AC}$$

(3.31)

where $p_{t}$ is the price paid by the customer at time t, $c_{t}$ is the direct cost of serving the customer at t, i is the discount rate for the firm, $r_{t}$ is the probability of a customer being “alive” at time t, AC is the acquisition cost, and T is the time horizon for estimating CLV. If the margin $(p_{t} - c_{t} )$ and retention rate are constant over time and the time horizon is assumed to be infinite, CLV can be simplified to (Gupta and Lehmann 2003):

$$CLV = \sum\limits_{t = 0}^{\infty } {\frac{{\left( {p - c} \right)r^{t} }}{{(1 + i)^{t} }} = m\frac{r}{(1 + i - r)}}$$

(3.32)

Haenlein and Libai (2013) use CLV to identify profitable customers (“revenue leaders”). They argue that targeting “revenue leaders” can accelerate these customers’ new product adoption and therefore create an earlier and larger cash flow. More important, these customers can create higher-than-average social value. This effect is due to “network assortativity”—a phenomenon whereby people tend to be connected with others who are like them.

In order to assess the value of “revenue leaders,” Haenlein and Libai (2013) use stochastic network-based cellular automata, an ABM (Agent-Based Model) technique, to simulate new product adoption based on local interactions among individual customers. The basic idea of the ABM technique is to start with a social network where no customer has yet adopted the product. Then the utility of buying the product is randomly generated and assigned to each customer and customers whose utility is larger than the product price will adopt the new product. Customers who have adopted the new product influence other customers by word-of-mouth and thus more customers adopt the new product in their social network. Details of the ABM technique are given in Goldenberg et al. (2002). One benefit of using the ABM technique is that it allows researchers to explore the effectiveness of various seeding programs and helps mangers target the most profitable customers. In Haenlein and Libai (2013), the value created by a seeding program is:

Total value = Direct value (i.e., new product adoptions by customers who are seeded) + Social value (i.e., new product adoptions by customers who are connected with the seeded ones) − Cost of the seeding program. Haenlein and Libai (2013) demonstrate that targeting “revenue leaders” is more profitable than targeting “opinion leaders.”

3.3 Future Research Opportunities

While prior research has contributed much to our understanding of innovation and new products, many opportunities remain to fulfill the potential in this important area of research. We begin by discussing key areas of emphasis that transcend particular stages of the product development process.

One key topic of future research should be to focus more on metrics than models. Prior research has already done much to develop useful models. However, going forward, developing appropriate metrics for firms to use systematically over time offers great potential benefits. With these metrics, researchers would be able to enhance our understanding of new product development, show firms how they should reduce the inherent inefficiencies, and help them deliver successful innovations on a more regular basis.

A second key topic of future research is business model innovation. Nearly all marketing research use the new product as the level of analysis. However, the product is only one part of the overall offering delivered to customers. Firms can grow through other elements of the marketing mix (i.e., pricing, communication, channels) and other aspects of their business model (e.g., financing, sourcing, partnering), and are influenced by actions that lie outside the firms’ control.

A third key topic of future research is to more thoroughly document the process of generating and commercializing the most innovative new products. Currently, we mainly know anecdotes and selected pieces of complete innovation success stories. The first step in repeating these successes is to at least thoroughly understand how they occurred in the past and to see what differentiates them from failures.

Next, we outline some more specific questions for future research. We organize these using the same four stages of new products research that we used to organize prior research: (i) opportunity identification , (ii) product design and development , (iii) sales forecasting , and (iv) commercialization .

3.3.1 Opportunity Identification

The following research questions are most important to address in the area of opportunity identification:

• 1. How should firms identify the most relevant lead users?

• 2. Which lead users are predictive of the future preferences of the general consumer market?

• 3. How can online platforms (e.g., user groups, Facebook, snapchat, and Instagram) be used to identify potential opportunity areas, generate new product ideas, and screen those ideas?

• 4. What are the best approaches for generating or moderating business-to-consumer communications and consumer-to-consumer communications?

• 5. When and how are new technologies incorporated into new products? Do these new technologies lead or lag firms’ efforts to identify new opportunities?

3.3.2 Product Design and Development

Research questions important to address in the area of product design and development include:

• 1. How do firms document and learn from failures during the new product development process? How should they do this?

• 2. How should firms make use of online platforms to design and develop their new products? What are the best ways to involve consumers at various points during design and development? What are the downsides of doing so?

• 3. What are the similarities and differences in designing and developing new products versus new services versus integrated products and services?

• 4. When is it appropriate for firms to rely on product champions versus cross-functional teams?

• 5. How should intrapreneurship be encouraged within organizations? How should firms fund and reward innovators? When and how should firms use think tanks or skunk works?

• 6. How and when should firms pursue joint development projects with other firms or with potential customers (especially business customers)?

• 7. When and how should firms abandon new products during design and development?

• 8. When and how should firms incorporate new technologies into new products?

• 9. When and what should new products borrow from past products, e.g., to be compatible with behavior or expectations?

• 10. Are portfolio approaches useful for managing risks in new product development projects?

3.3.3 Sales Forecasting

The following research questions are important to address in the area of sales forecasting:

• 1. How should firms use online platforms and social media (e.g., Facebook, Instagram, snapchat, etc.) to forecast sales?

• 2. How should the use of these platforms vary for business-to-business versus business-to-consumer products?

• 3. How can sales forecasting techniques be more diagnostic by decomposing the overall sales forecasts into the various elements of each new product or service offering?

• 4. What testing techniques provide better information about ultimate market acceptance earlier in the product design and development process?

• 5. How can firms generate better estimates of cannibalization across their product lines?

• 6. How do forecasts themselves impact strategy and success?

3.3.4 Commercialization

While this stage is typically ignored by academics, it is often the most critical one. Important questions to address in this area include:

• 1. How should firms document, learn from, and apply lessons learned from failed new product launches?

• 2. How should firms use social media to promote new product launches? How should these efforts differ between business and consumer markets?

• 3. What is the role of opinion leaders in markets with high social media activity?

• 4. Who are the opinion leaders in markets with high social media activity and how do they differ from opinion leaders in markets with low social media activity?

• 5. When and how should firms kill new products after commercialization?

• 6. What does the concept of relative product advantage really mean? How is it measured? How much does it contribute to a new product’s success?

• 7. How do new categories obtain their names? Should firms be more proactive about promoting new category names?

• 8. How important are informational cascades in driving new product adoption?

• 9. What are the contextual factors that determine when being fast to market is more or less important? What are the differences between incremental innovations and radical innovations?

• 10. What is the appropriate scale of entry for new products? What are the factors that determine when it should be large or small?

• 11. How should firms manage consumer disadoption and disposal?

3.4 Conclusion

This review has highlighted some key marketing research on innovation and new products. Where we provided less extensive coverage, we refer readers to other useful references. For each of the four stages of the new product development process (opportunity identification , product design and development , sales forecasting , and commercialization ), we organize literature by sub-topics within each of these stages. This hopefully will give readers a good sense of the state-of-the-art in each of these research areas. We also provide thoughts on some important research to conduct going forward. Overall, much has been learned already. Nonetheless, given the importance of new product innovation to firms, to individuals, and to societies, we hope that tomorrow’s researchers continue to generate newer and richer insights in this vitally important field of investigation.