1 Introduction

The dramatic slowdown of economic growth in Japan during the 1990s drew much attention from economists. Among the several hypotheses that have emerged, the most controversial one has been the finding by Hayashi and Prescott (2002) that the lost decade was explained by a fall in exogenous TFP, as it casted doubt on the popularly held belief that a credit crunch arising from financial sector problems slowed down growth by reducing investment. More recently, Peek and Rosengren (2005) show that misallocation of credit, caused by Japanese banks’ incentive to extend lines of credit to financially troubled, “zombie” firms, contributed to the prolonged recovery from recession. Footnote 1 This paper revisits these issues: it investigates whether the reduction in productivity growth in Japan was due to a drop in within-firm productivity, or due to the misallocation of resources across firms. More specifically, we conduct productivity decomposition exercises to examine if the cleansing effect of recessions (i.e. downsizing/exits of less productive firms) was in place during the first half of the 1990s, and also how much of the reduction in total productivity can be explained by the fall in within-firm productivity, the component that is not affected by changes in shares across firms.

The 1990s marked the first decade of sluggish economic growth for the Japanese economy since the end of the Second World War. The deterioration of Japan’s economic performance, which persisted over a decade, has been a unique macroeconomic event, yet not enough evidence has been unmasked to generate a consensus regarding the factors that have contributed to the lengthy recovery. The early stage of the preceding discussion centered around policy failures in the area of demand management, notably highlighted by a “liquidity trap” hypothesis or “credit crunch” problem. However, formal evidence in support of this hypothesis is yet to be found.

The proponents of the “liquidity trap” hypothesis claim that the monetary authority’s inability to stimulate investment by lowering interest rates, or consumer spending by creating inflationary expectations, unnecessarily prolonged the recovery phase. On the other hand, the “credit crunch” hypothesis speculates that the poor financial condition among many Japanese banks was leading to the banks’ reduced lending to profitable projects, thereby contributing to lower investment. However, Motonishi and Yoshikawa (1999), using the Bank of Japan diffusion indices of “real profitability” and “banks’ willingness to lend,” find that except for 1997 when the government finally allowed some big banks to fail, drops in investment were unrelated to banks’ willingness to lend and were mainly driven by a fall in real profitability. Footnote 2

Using aggregate growth accounting, Hayashi and Prescott (2002) argue that the economic stagnation during the 1990s in Japan is largely explained by a fall in exogenous TFP growth. As in Motonishi and Yoshikawa, they also demonstrate that firms were able to finance their investment from alternative sources even during a period of constrained bank lending (i.e., by the BIS capital ratio imposed on banks). Controlling for labor quality and capacity utilization, Fukao et al. (2003) conduct similar growth accounting exercises by industries. Their results also produce a fall in TFP growth during the 1990s, but its magnitude is much smaller in comparison to Hayashi and Prescott’s results, as they isolate the impact of the slower rate of labor quality growth during the 1990s in their estimates of TFP growth rates.

More recent literature identifies the reallocation issue as the primary problem. For example, Peek and Rosengren (2005) find evidence of misallocation of credit by Japanese banks as they engaged in “ever-greening” loans. Namely, they claim that financially troubled firms were more likely to obtain further loans from banks than their healthier counterparts during the 1990s, as banks sought to make financially troubled firms look artificially solvent on their balance sheets. Likewise, using stock returns, Hamao et al. (2003) suggest that there was a lack of resource reallocation in Japan during the 1990s. In particular, when a firm’s idiosyncratic risk is measured as the deviation of its stock return from the average response to the market rate, they show that the role of idiosyncratic risk in explaining the total time-series volatility of firm stock returns decreased during the 1990s. Consequently, they point out that this apparent increase in homogeneity of corporate performance may have hindered the ability of investors and managers to distinguish high quality firms from low quality firms, and discouraged capital formation.

In fact, a considerable amount of research relates reallocation to economic performance and growth over the business cycle. The theoretical aspects of the literature often focus on Schumpeter’s idea of “creative destruction.” Aghion and Howitt (1992), for instance, construct an endogenous growth model in which old technology is immediately destroyed with the emergence of new technology, thereby constituting the underlying engine of economic growth through the introduction of a competitive research sector that generates vertical innovations. In a similar spirit, Caballero and Hammour (1994, 1996) created a model in which only entering firms have access to the latest vintage of capital, and therefore the destruction of firms with old vintages facilitates the flow of new entries and is productivity enhancing. Footnote 3

We conduct productivity decomposition exercises using the Nikkei NEEDS dataset to examine whether or not the cleansing effect of recessions was taking place via downsizing and exits of inefficient businesses, and how much of the reduction in productivity growth is accounted for by within-firm productivity growth. The main advantage of these decomposition exercises is that it allows us to isolate the component of aggregate productivity growth related to changes in shares that arise from the reshuffling of resources across firms, from within-firm productivity growth. Accordingly, one could argue that within-firm productivity growth is a better measure of “exogenous productivity shock” than a measure constructed using usual growth accounting, since it focuses on the component of productivity growth that is not affected by the changes in weight assigned to firms over time.

Foster et al. (2001) show that, the reallocation of inputs and outputs from less to more efficient establishment accounts for about half of the productivity growth between 1977 and 1987 in the US manufacturing sector, while the other half is explained by within-establishment productivity growth. Moreover, they find that roughly 30% of the total productivity growth is attributed to entry/exit dynamics. A similar study of retail sector by Foster et al. (2002) shows that the entry/exit explains virtually all productivity growth in this sector between 1987 and 1997.

Unlike the longitudinal datasets used by Foster et al., the NEEDS dataset has several disadvantages. These disadvantages will be discussed in detail later. Footnote 4 The main disadvantage of the dataset is that the dataset does not permit us to accurately assess the contribution of the net-entry dynamics (i.e., replacement of inefficient firms by more efficient firms) to overall productivity growth. Thus, the paper primarily focuses on resource reallocation among continuing firms rather than reallocation at the entry/exit margin. Moreover, the available data only allows us to examine the reallocation dynamics of large manufacturing firms. Accordingly, we are unable to examine the effect of credit crunch among small firms. In addition, the productivity gains associated with reshuffling of resources across establishments within a firm will be captured as within-firm growth. Despite these disadvantages, however, the dataset provides some important insights into the changes in the productivity dynamics in Japan over time since 1969.

The key results are the following. First, while the overall TFP growth rates of the firms examined remained more or less the same between the two periods covering 1979–1988 and 1988–1996, within-firm TFP growth fell roughly by half between these two periods. The unusually large drop in within-firm TFP is in line with the result obtained by Hayashi and Prescott that the fall in exogenous TFP during the 1990s plays a large role in explaining the slower growth of the 90s. Second, although we find that misallocation of resources among continuing firms represents a substantial drag to overall TFP growth throughout the sample period, the negative contribution of misallocation was least visible during the 1990s. Third, while the data limitation prevents us from appropriately capturing the effect of misallocation at the entry/exit margin, it seems reasonable to speculate that the lack of exits by inefficient firms before 1997 may have reduced productivity growth during the 1990s. The surprisingly low level of exits despite the dramatic slowdown of economic activity is in line with the conclusion by Peek and Rosengren (2005) that banks deliberately helped financially troubled firms to stay in business. Footnote 5 Finally, when the sample was split into two by employment size, we find that misallocation of resources among continuing firms essentially applies to larger firms.

2 Description of the dataset

The main dataset used in this paper is the Nikkei NEEDS dataset, which covers relatively large nonfinancial firms since 1964. The primary advantage of the dataset is that it allows us to examine changes in productivity dynamics over time. Footnote 6 The dataset is an unbalanced panel, in which the majority is based on annual reports while the remainder is based on semi-annual reports. We use only manufacturing firms for the decomposition exercises as the sectoral deflators provided by the Bank of Japan (CGPI) are available only for manufacturing industries. Firms included are those that are listed on the Tokyo Stock Exchange, JASDAQ and other regional stock markets, leading unlisted companies submitting financial reports to the Ministry of Finance, and other leading unlisted companies that are not included in the above mentioned categories but submit reports to their shareholders. The dataset has financial as well as employment data, with some corporate information.

One of the main concerns with this dataset involves the treatment of entries. Unlike a longitudinal dataset such as the LRD, most entries into the NEEDS dataset begin after the firms’ initial public offering, and not when the firm first started its business. Thus, the entering firms in this exercise are not necessarily new players in their industries, but they are new players in the financial market. As a result, this is not a standard decomposition for the purpose of analyzing the role of entrants in Schumpeter’s creative-destruction sense. Footnote 7

Furthermore, dropping out from the dataset could either mean mergers or bankruptcies. Fortunately, the NEEDS dataset has an index of mergers so that we were able to sort out these two and differentiate them in the productivity decomposition. Footnote 8 Although we also calculate the productivity contribution of mergers, we consider only bankruptcies as exits that are relevant for the study of reallocation.

Another issue regarding the examination of reallocation at the entry/exit margin is that the number of entries into the dataset is significantly larger than the number of exits from the database. In particular, the number of exits are very small throughout the sample period, while the number of entries into the dataset is particularly high during the 1970s due to the incorporation of regional stock markets. Footnote 9 For these reasons, the impact of net entry needs to be interpreted with caution, and we mainly focus on the reallocation dynamics of continuing firms.

The last concern is the sample selection problem: the companies which are used for this analysis are relatively large leading manufacturing companies. Accordingly, their overall performance and reallocation activities may not be entirely representative of the extent of the deterioration of the aggregate economic performance. Footnote 10 However, we believe that the results we observe from these companies with respect to their productivity dynamics have significant implications for the economy wide productivity growth, as most of these companies have access to leading technology. In addition, many of these companies are likely to have strong ties with large Japanese banks, and therefore it should be relevant to the investigation of the impact of the posited misallocation of resources.

Table 1 provides descriptive statistics of firm level employment for manufacturing firms in the NEEDS dataset used for the decomposition exercises. Footnote 11 Note that the statistics in the table correspond to the average of the annual statistics over each time interval. Footnote 12 The top part of the table gives descriptive statistics of all manufacturing firms used for the decomposition. As we can see, the average firm size in terms of employment falls while the total number of firms increases over time, most likely reflecting the incorporation of smaller size firms. The middle part of the table gives the descriptive statistics of firm level employment for entering firms only, and the bottom part of the table identifies statistics for firms that dropped from the dataset. The average size of dropped firms, in terms of employment, is smaller than the overall average, but not as small as the average size of entering firms.

Table 1 Descriptive statistics of firm level employment for manufacturing firms in the NEEDS dataset for 1969–1996, for all firms, entering firms and dropped firms

3 Productivity decomposition

Using plant level data from the Census of Manufactures, Foster et al. (2001) show that reallocation of outputs and inputs across establishments as well as reallocation through entry and exit play an important role in explaining aggregate productivity growth. In order to explain productivity dynamics among relatively large Japanese manufacturing firms, two types of decomposition exercises, following Foster et al., are conducted using the NEEDS dataset. Denoting ΔP jt as the productivity growth of industry j between t − 1 (beginning period) and t (ending period), the first decomposition is given by the following equation:

$$ \begin{aligned} \Updelta P_{jt}&=\sum_{i\in C}s_{it-1}\Updelta p_{it}+\sum_{i\in C}(p_{it-1}-P_{jt-1})\Updelta s_{it}+\sum_{i\in C}\Updelta s_{it}\Updelta p_{it}\\ & \quad +\sum_{i\in N}s_{it}(p_{it}-P_{jt-1})+\sum_{i\in X}s_{it-1}(P_{jt-1}-p_{it-1}), \end{aligned} $$
(1)

where s i is the share of firm i in industry j, p i and P j are the productivity indices for firm and industry respectively, and C, N and X indicate the set of continuing firms, entering firms and dropped firms respectively. The second decomposition is given by:

$$ \begin{aligned} \Updelta P_{jt}&=\sum_{i\in C}\bar{s}_{it}\Updelta p_{it}+\sum_{i\in C}(\bar{p}_{it}-\bar{P}_{jt})\Updelta s_{it}\\ & \quad +\sum_{i\in N}s_{it}(p_{it}-\bar{P}_{jt})+\sum_{i\in X}s_{it-1}(\bar{P}_{jt}-p_{it-1}), \end{aligned} $$
(2)

where a bar over a variable indicates the value averaged over t − 1 and t.

The first term in both Eqs. 1 and 2 shows contribution of the within-firm productivity growth to aggregate productivity growth. On the other hand, the second term in Eqs. 1 and 2 shows the between-firm effect, which captures the contribution arising from the reshuffling of inputs or outputs across continuing firms. Here, the changes in shares are weighted in both cases by the deviation of firm productivity from the corresponding industry productivity index. The index in the first decomposition uses beginning period industry productivity, P jt−1, while the second decomposition uses industry productivity averaged over t − 1 and t. The last two terms in Eqs. 1 and 2 represent the contribution made by entrants and dropped firms respectively. Note that a firm’s entry into the dataset raises aggregate productivity when its productivity is above the industry productivity index. Likewise, a firm’s dropping from the dataset raises aggregate productivity when its productivity is below the industry productivity index. As mentioned previously, dropped firms are divided into exits and mergers.

For the second decomposition given by Eq. 2, the average values over t − 1 and t are used for the within-firm component’s share indices and the between-firm component’s productivity indices. Therefore, the interaction effect between changes in share and changes in productivity (i.e. covariance effect) is incorporated in the first two terms, while the first decomposition given by Eq. 1 explicitly controls for this effect with the third cross-firm term. While the first method provides a more accurate decomposition, it is more sensitive to measurement errors as discussed in Foster et al. (2001), and thus, the results using both decomposition methods will be presented. Footnote 13

Two types of productivity measures, labor productivity and total factor productivity (TFP), are constructed for the decomposition exercises. Since the NEEDS dataset has information on employment but not on hours, the measure of manhours was constructed by multiplying firm employment by sectoral work hours taken from Monthly Labor Statistics. Footnote 14 The labor productivity measure used here is the log difference of real gross output and manhours. Note that the real gross output figures were summed over each year when firms submit reports more than once a year, while the average employment figures are used for these firms.

The index of TFP is measured simply as follows:

$$ \ln TFP_{it}=\ln Y_{it}-\alpha _{M}\ln M_{it}-\alpha _{L}\ln L_{it}-(1-\alpha _{M}-\alpha _{L})\ln K_{it} $$
(3)

where Y it is real gross output for firm i at year t, M it is real materials, L it is labor input in terms of manhours, K it is the real capital stock, α M is material’s share of total cost, and α L is labor’s share of total cost. Footnote 15 Detailed explanations of the construction of real gross output, real materials, labor input and real capital stock using the NEEDS dataset are provided in the Appendix.

Note that the notations for the material cost share α M and the labor cost share α L are simplified here, as the shares actually used vary across three-digit NEEDS industry classifications, although not over time. Footnote 16 The material and labor cost shares are calculated at the industry level by aggregating firm-level cost shares using firm level total cost as a weight. When aggregated across all firms in the dataset, the material cost share is 67%, while the labor cost share is about 13% and the capital cost share is about 20%.

The time horizon over which we investigate productivity growth is set between 8 and 10 years. This time horizon indicates the distance between the subscript t and the subscript t − 1 in Eqs. 1 and 2. Thus, the analysis decomposes productivity growth dynamics over the long-run. Ideally, the starting period and the ending period should encompass the full business cycle. This allows us to compare the results across different time periods while avoiding short-run business cycle effects on productivity. Hence, we divided the entire productivity series into three sub-periods based on the following business cycle considerations: (1) a high growth period (from the peak of 1969 to the peak of 1979), (2) the bubble economy period (from the peak of 1979 to the peak of 1988) and (3) the sluggish growth period (from the peak of 1988 to the peak of 1996).

4 Results

In this section, we present the productivity decomposition results of the entire sample as well as the results broken down by size. Table 2 shows the number of firms used in productivity decompositions. Footnote 17 As mentioned previously, the number of entries is high during the 1969–1979 period since some regional stock markets were incorporated during this period. The number of exits is small in all periods as there were a relatively small number of bankruptcies among listed manufacturing firms before 1997, but the exit number is surprisingly small during the 1988–1996 period. Footnote 18 Some firms are excluded from the exit component as we could not confirm their bankruptcy status. The list of exiting firms is provided in the Appendix.

Table 2 Number of firms used in productivity decomposition

Table 3 shows the results of productivity decompositions using labor productivity and TFP for the entire sample. The measure of the share (s it ) used for labor productivity is employment, while that used for TFP is real gross output. The upper panel of the table shows the results using the first decomposition method and the lower panel of the table shows the results using the second decomposition method. Note that TFP is a preferred measure of productivity since growth in labor productivity can imply both rise in capital intensity and rise in TFP. Footnote 19 Accordingly, reallocation of workers towards high labor productivity firms may arise from a shift towards more capital intensive firms, rather than reshuffling of resources towards high TFP firms.

Table 3 Productivity decomposition results for the manufacturing sector using labor productivity and TFP for 1969–1996

First, the TFP decomposition results show that while the overall TFP growth rate remained more or less constant between the two periods covering 1979–1988 and 1988–1996 for the group of firms examined, the within-firm TFP growth for these firms dropped substantially, roughly by half in both decompositions. This result seems to support Hayashi and Prescott’s main result that there has been a slowdown in the exogenous productivity growth rate during this period. Footnote 20

Second, the results indicate that, by and large, relatively more productive firms contracted in output with an accompanying reduction in TFP and vice versa, since the between-firm component for the first TFP decomposition is negative throughout the sample period while the cross-firm component is mostly positive (except during the bubble period). Footnote 21 Because the negative between-firm component far exceeds the positive cross-firm component, the total combined contribution of reallocation among incumbent firms is negative, and it represents a sizable drag to overall TFP growth. However, the combined negative contribution is relatively small between 1988 and 1996, since the negative between-firm component is the smallest and the positive cross-firm component is the biggest. Accordingly, the misallocation among continuing firms was the weakest during the 1988–1996 period at least for large manufacturing firms. The results for the second decomposition are similar.

One possible explanation why reallocation was less of a drag during the 1988–1996 period is that very strong performance of within-firm TFP growth especially during the 1960s and 1970s had allowed Japanese businesses to adopt practices that are, in the short-run, counter-productive in terms of reallocation efficiency. The practice of life-time employment, for instance, was a standard business model, and policymakers and managers often believed that the long-term gains associated with having workers with firm specific skills outweighed the costs of sustaining workers during short-term downturns. Moreover, Japanese banks often provided funding based on traditional ties and long-run prospects rather than short periods of performance over a business cycle. Although Japanese businesses attempted to respond to various shocks more flexibly by increasing flexibility within firms (through the practice called syukko), it was probably only during the 1990s that it became clear to many businesses that reshuffling within firms would not be sufficient to survive, and many businesses started a large scale downsizing as a part of restructuring.

As for labor productivity, the within-firm productivity growth dropped by more than half in both decompositions. Unlike TFP decomposition, the results indicate that, by and large, firms with initially high labor productivity expanded in employment with an accompanying reduction in labor productivity and vice versa, since the between-firm term in the first decomposition is mostly positive (except during the bubble economy period) while the cross-firm term is negative. The combined effect of reallocation among continuing firms is negative in all cases, but unlike TFP decomposition, it accounts for a very small fraction of overall growth. Footnote 22 Again, the combined negative effect is the smallest between 1988 and 1996, although the improvement is less dramatic compared to the TFP decomposition. The negative but small role of reallocation among continuing firms is also observed for the second decomposition except between 1969 and 1979.

Despite the several issues regarding entries and exits as discussed previously, it is worth mentioning that the exit component turns negative during the 1988–1996 period indicating that those firms which went out of business, on average, had higher productivity than the industry average. This may be a result of the limitation in the sampling of exiting firms in our dataset. However, the small number of exiting firms in spite of a significant slowing down in within-firm productivity growth may be partly a result of the banking practice which prevented bankruptcy of inefficient firms as documented in Peek and Rosengren (2005) and others.

As mentioned previously, examination of the entry component requires more caution, because the entry into the database does not imply a market entry and also because the large number of entries during the 1969–1979 period is due to incorporation of regional stock markets. Finally, for the time period being considered, those manufacturing firms which dropped out from the dataset due to mergers were not necessarily low productivity firms, especially in terms of labor productivity.

We also conducted similar exercises using annual productivity growth rates, in which the resulting annual contributions of each component are averaged over the three sub-periods. The main findings remained robust for the annual productivity decomposition. In these exercises, the within-firm TFP component did not fall during the bubble-economy period. As a result, the drop in within-firm TFP is a prominent feature of the post-bubble period. These results are shown in the Appendix.

Next, Table 4 shows the results of the TFP decomposition broken down by size. Here, firms are categorized into large and small firms based on the average employment and capital stocks across the time they appear in the dataset. The threshold level of each measure is simply given by the median firm of each measure. Footnote 23

Table 4 TFP decomposition results for the manufacturing sector by size

We first begin with the examination of the decomposition results by employment size. The key finding is that, unlike the previous results, reallocation among smaller continuing firms plays a positive role in boosting TFP growth except during the bubble period. In particular, the combined contribution of reallocation (i.e., between-firm and cross-firm components) explains large proportion of overall TFP growth during the 1988–1996 period. The between-firm component is all positive for smaller firms, suggesting that firm’s initial TFP level relative to the industry average is a good indicator of firm’s output growth over the next decade. On the other hand, the results for larger firms are similar to the results for all firms. Accordingly, the malfunctioning of reallocation among continuing firms essentially applies to larger firms.

When the sample was divided between large and small firms in terms of capital stock, the results for larger firms are again similar to the results for all firms. Unlike firms with smaller employment, the combined contribution of reallocation among continuing firms is largely negative for the firms with smaller capital stock. Since average capital-labor intensity is higher for the firms with smaller employment compared to the firms with smaller capital stock, the difference seems to suggest that the reallocation process among capital intensive firms tends to be productivity enhancing whereas the reallocation process among labor intensive firms tends to be counter-productive. Footnote 24

Despite the negative between-firm component for the firms with smaller capital stock, the combined effect is less negative in comparison with larger firms and it turns positive during the 1988–1996 period. Hence, the results suggest once again that misallocation took place more often among larger firms. Finally, for smaller firms, within-firm TFP growth fell even prior to the sluggish growth period. This finding implies that less capital intensive firms were already starting to experience a sharp drop in TFP growth immediately following the high growth period. Footnote 25

5 Conclusion

The main result shows a significant drop in the within-firm productivity growth rates during the 1988–1996 period for large manufacturing firms. This result is in line with Hayashi and Prescott (2002) that the fall in exogenous productivity is important. Moreover, while the data limitation prevents us from appropriately capturing the role of reallocation at the entry/exit margin, we speculate that the lack of exits by inefficient firms before 1997 may have reduced productivity growth during the 1990s. The surprising low level of exits during the 1988–1996 period is in line with the findings of Peek and Rosengren (2005) that financially troubled and heavily indebted companies were able to survive as Japanese banks extended them lines of credits.

One possible explanation for the reduction in within-firm TFP is a considerable labor hoarding due to high labor adjustment costs. While this may partly explain the reduction in TFP, it is unlikely that labor hoarding alone can explain a decline of this magnitude. Footnote 26 Other explanations include slowing down in the pace of technological innovations, R&D, and technological adoption. Understanding these components may provide a key to fully account for the slowing down of productivity growth.

Finally, even though we attempted to isolate the impact misallocation from within-firm productivity growth, we do not intend to suggest that the reduction in within-firm TFP growth occurred independently from the reallocation process. To illustrate, the reallocation process may interact with the growth of within-firm TFP through a higher level of competition. Footnote 27 While the high level of aggregate productivity growth despite substantial misallocation prior to the 1990s does not point to the importance of reallocation, one key change may be that the Japanese economy has matured. Accordingly, innovation, rather than imitation, may have become increasingly critical for its economic performance.