Keywords

For almost two decades, a wellspring of initiatives have been undertaken to improve American’s financial literacy, with financial education as the tonic. The energy and resources devoted to improving financial literacy through financial education cannot be understated. In the federal sector alone, an estimated $68 million dollars was spent on financial literacy activities in 2010, not accounting for $137 million spent on housing counseling, which often includes a financial education component (GAO, 2012a). A lively discourse has emerged about the value and efficacy of financial education efforts (Willis, 2011), but it has been characterized as polarized and a disservice to financial education (Baumann & Hall, 2012). The growing body of literature demonstrates the value of financial education; but it is equally clear that financial education in not the only contributing factor to the financial security and wellbeing of consumers (Fernandes, Lynch, & Netemeyer, 2014; Sherraden, 2013).

Discussion around financial literacy education is typically motivated by the increased complexity in financial products, the burden of shoring up one’s own financial security for retirement, and the recent financial crisis and its association with a lack of financial literacy (Hastings, Madrian, & Skimmyhorn, 2012; Willis, 2011). A report from the President’s Advisory Council on Financial Literacy (2008) stated: “while the crisis has many causes, it is undeniable that financial illiteracy is one of the root causes” (p. 1). The aforementioned debate is largely fed by the lack of evidence demonstrating a relationship between financial education, financial literacy, and financial behaviors (Hastings et al., 2012; Hung, Parker, & Yoong, 2009). However, empirically, conceptually, and theoretically advances are being made in the field of financial literacy education and evaluation and several of these advances are highlighted in this chapter.

The chapter highlights some of the key challenges facing providers of financial education programs as they evaluate program effectiveness. We work from Jacob’s (2003) operational definition of evaluation as “a set of systematically planned and executed activities designed to determine the merit of a program, intervention, or policy or to describe aspects of its operation” (p. 63). The chapter defines the scope of financial education interventions, describes the breadth of current efforts, summarizes the evidence behind establishing the need for financial education, reviews the evidence of linkages between education, knowledge, and behavior, and finally describes a general framework for evaluation that can be applied to programs with few resources and/or more fully developed programs.

Financial Literacy and Financial Education Defined

Financial literacy denotes one’s understanding and knowledge of financial concepts and is crucial to effective consumer financial decision making that can potentially lead to improved consumer financial security and wellbeing. We use a broad definition of financial literacy, but nationally and internationally, a rich dialogue has been advanced on defining, conceptualizing, and measuring financial literacy (Remund, 2010; World Bank, 2013). The use of a comparable, consistent, and similar definition among scholars and policy makers is needed to advance the field, yet a variety of meanings exist; and to date, widespread adoption of a singular definition is not evident (Hastings et al., 2012; Hung et al., 2009). A clearly defined and single definition that is used consistently helps educators and organizations to develop financial education programs that are meaningful and effective for consumers (Remund, 2010).

Financial literacy, thought to be a narrow concept, has been used interchangeably with the concept of financial capability. The Financial Literacy and Education Commission (FLEC, 2011) defines financial literacy as “the ability to use knowledge and skills to manage financial resources effectively for a lifetime of financial well-being” and financial capability as “an individual’s capacity, based on knowledge, skills, and access, to manage financial resources effectively” (GAO, 2011, p. 3). There is general agreement among scholars that financial literacy is a necessary component to a consumer’s financial capability (Sherraden, 2013, Xiao, Chen, & Sun, 2015).

Financial education can include any program that addresses the knowledge, attitudes, and/or behavior of an individual toward financial topics and concepts. Current trends promoting financial literacy have moved to a more comprehensive approach to promoting financial capability (Xiao et al., 2015). When assessing the effectiveness of financial education, comprehensive measurement that captures financial literacy, financial behavior, and perceived financial capability need to account for relevant knowledge, attitudes, behaviors, and outcomes (Xiao et al., 2015; Xiao, Chen, & Chen, 2014). Most scholars agree that financial education does not occur in isolation and is just one contributor to a consumer’s financial wellbeing. The Organization for Economic Co-operation and Development (OECD, 2014) includes financial education, financial inclusion, and financial consumer protection as the three legged approach to achieve financial empowerment and wellbeing, whereas the PACFC (2013) suggests financial education, regulation and consumer protection, and design of options (choice architecture) as integral to a framework for financial capability.

Financial education programs operate on the assumption that it will increase a consumer’s understanding of financial topics (information and knowledge aka financial literacy) which then leads to an improvement in financial decision making and behavior (aka financial capability) (Hilgert, Hogarth, & Beverly, 2003; Tisdell, 2014). Many scholars agree that it isn’t enough to measure the link between financial education and an increase in financial literacy , typically measured as a knowledge score. Instead, demonstrating the causal links between financial education, financial literacy, financial capability, and financial outcomes has been advocated, with financial inclusion most recently being added to the causal chain (Sherraden, 2013). Individual characteristics (demographic traits, personality traits, emotional status around money), family context, sociocultural context (financial markets, access to financial products, legal and regulatory), and societal factors (capitalist economy, social structures) (Van Campenhout, 2015; Way, 2014) have also been studied within this framework.

Efforts in Financial Education and Evaluation

Financial literacy education activities are now so pervasive and widespread that it is difficult to accurately inventory the program initiatives engaged in by public and private entities. Local, state, and federal government agencies, community organizations, employers, financial services and banking institutions, faith-based organizations, secondary and post-secondary schools, and the US Military offer financial education programs (Vitt et al., 2000; Vitt, Reichbach, Kent, & Siegenthaler,2010). Many of the challenges associated with advancing the field are a result of this proliferation of financial literacy education. Some organizations and agencies deliver financial education with good intentions, but lack the skills and/or desire to contribute to a wider dialogue about how their efforts relate to pedagogy or effectiveness. Organized efforts around program design, development, delivery, and evaluation are likely stunted by the duplication of efforts.

Activities at the federal level include two statutory mandates that have the goal to develop a national strategy for financial literacy (GAO, 2012b). First, to unify early efforts addressing financial literacy and education, the Financial Literacy and Education Improvement Act, passed under Title V of the Fair and Accurate Credit Transactions Act (FACT) of 2003 created the Financial Literacy and Education Commission (FLEC) . FLEC’s (2011) financial literacy strategy focuses on: (1) “the need for increased financial literacy and effective financial decision making and (2) the educational efforts required to achieve those worthy objectives” (p.2). The commission is composed of 22 federal agencies and chaired by the Treasury Department (GAO, 2014). With mandated reporting, over a dozen GAO reports and testimonies have been published since 2003 with recommendations for the development and improvement of the nation’s financial literacy educational efforts.

An overview of federal activities reported 13 programs delivering financial literacy as a primary teaching objective, additionally three housing counseling programs had financial education as a component (GAO, 2014). Due to consolidation and elimination of redundancies, the number of programs, operated by federal agencies has dropped. The most recent GAO (2014) report did not report on evaluation data or activities of these programs, however, they “encourage research of the various financial literacy initiatives to evaluate the relative effectiveness of different approaches” (GAO, 2012b, p.i).

In terms of resources, along with a hotline (1-888-My Money), FLEC developed mymoney.gov, a website serving as a point of entry to all financial literacy education resources and tools that have been produced by the 22 federal agencies. Additionally, FLEC is developing a national clearinghouse to inventory evidence-based research and evaluation studies (GAO, 2012b, p. 9). A national repository with an inventory of financial education evaluations would allow researchers and practitioners to access the technical details around program delivery and effectiveness. A tool of this nature would eliminate the isolation of current efforts, potentially moving the field forward.

Subsequent to the development of FLEC, the Dodd-Frank Act mandated the creation of the Consumer Financial Protection Bureau (CFPB) , a new agency member of FLEC whose Director serves as Vice Chair (GAO, 2012b). Like FLEC, the CFPB has a central role in promoting a national financial literacy strategy and has “developed a strategy and a broad range of initiatives to help consumers take control of their financial lives” (GAO, 2012b, p.7). The CFPB’s Division of Consumer Education and Engagement houses the Office of Financial Education. The CFPB advocates for evidence-based research to inform policies and programs. The CFPB has developed its own research program focusing on “(1) determining how to measure financial wellbeing and identifying the knowledge, skills, and habits associated with financially capable consumers, (2) evaluating the effectiveness of existing approaches to improving financial decision making and outcomes, and (3) developing and evaluating new and innovative approaches to helping consumers make financial decisions” (CFPB, 2014, p. 28). Recently, the CFPB commissioned the development of financial literacy metrics and outcomes with the Corporation for Enterprise Development. Additionally, the CFPB contracted with the Urban Institute to explore the use of Randomized Control Trials (RCTs) with two financial capability programs (GAO, 2014).

Financial literacy education varies by the setting, audience, and subject matter (Vitt et al., 2000). Efforts can also be organized into categories based on themes or topics in personal finance. First, there are programs directed at improving financial literacy by broadly addressing personal finance topics, such as budgeting, saving, and credit management. Second, there are programs that give specific training in retirement and savings and are generally offered by employers. The third major category of programs addresses home buying and home ownership.

In the first category, several wide-ranging initiatives are aimed at school-age students. The Jump$tart Coalition for Personal Financial Literacy is a public–private partnership that pushed financial literacy into the spotlight two decades ago when they released a biennial survey of high school seniors that showed a failing grade on financial literacy scores. Jump$tart is composed of more than 80 educator, corporate, and government organizations with the mission to advance personal finance education in schools, particularly through promoting the use of standards for grades K–12 (Jump$tart Coalition for Personal Financial Literacy, 2015). Jump$tart coalitions work locally in states to advance the national mission and these efforts have been met with success. In 2014, 43 states required personal finance content in their K–12 standards, up from 21 states in 1998. Currently, 35 states require the implementation of personal finance content into its standards, up from 14 in 1998. In 1998 no state required a personal finance course to be taken by high school students; today 19 states have the requirement (Council for Economic Education, 2014).

There has been a rise in the number of post-secondary schools that provide financial education programming, despite limited support in funding and staff resources. Out of 200 colleges and universities surveyed, 65 % currently offer a financial education program and 43 % anticipated developing one in the next 12 months (Inceptia, 2012). Another survey found that 90 % of financial aid administrators from 36 states reported delivering financial education either in-person or through a webinar, most frequently on the topic of loan repayment, enhanced exit interviews, budgeting, credit, and enhanced entrance counseling (Hackett, 2015).

General financial education initiatives also target broader audiences. For example, the Federal Deposit Insurance Corporation’s (FDIC) Money Smart curriculum targets adults with a 10-module curriculum covering basic financial topics such as budgeting, saving, and credit management. The Money Smart Alliance Program invites partners to become members and adopt the curriculum. Financial institutions are actively involved in financial literacy efforts. A survey of 576 credit unions found 61 % conducted an in-person financial education workshop and more than 150,000 adults were reached through 8000 credit union seminars (National Credit Union Foundation, 2011). Similarly, 97 % of retail banks surveyed reported sponsoring or supporting through a partnership a financial literacy program (Consumer Bankers Association, 2002).

The second category of financial education programs consists of employer-sponsored programs offering training in retirement planning and savings . Almost 6 in 10 US employers offered financial education to their employees and 21 % planned to offer it in the next 12 months (SHRM, 2014). Retirement planning was the most common topic offered to employees (79 % of employers), followed by employee assistance programs (75 %) and investment planning (56 %) (SHRM, 2014). Workplace financial education activities include counseling, seminars, e-learning, workshops, benefit fairs, or newsletters (SHRM, 2014; Todd, 2002). For a more extensive discussion, see Chap. 20 Workplace Financial Education in this book (Kim, in press).

The third category of financial education programming is anchored in home buying and home ownership programs, which have the longest history among financial education initiatives and typically extend into training relevant to other financial goals , such as improving savings rates and decreasing debt (Todd, 2002). Just over a decade ago, over 1000 organizations received funding from foundations to programs (Todd, 2002). As evidenced, there is no shortage of initiatives, campaigns, and partnerships undertaking financial literacy education. With this fervor for financial education delivery, the important question and impending challenge are discerning the effectiveness of these efforts.

Evidence of Consumer Financial Illiteracy

Advances in organizing the study of financial literacy education could eventually yield promise, as seen in the expansion of global and national initiatives and data collection efforts. In 2009 and 2012, the National Financial Capability Survey (NFCS) , a State-by-State Survey, and a Military Survey were administered, the first to focus on financial education and capability of US adults. NFCS data collection will continue, with plans to link it to a more detailed, longitudinal dataset, the American Life Panel (personal communication with David Rogofsy, March 13, 2014). Earlier datasets that targeted financial literacy, such as the Jump$tart Coalition Survey and Survey of Consumer Finances (SCF), lacked detailed questions on financial education and financial decision making (Hung et al., 2009), and none follow consumers over time; the NFCS will improve on these limitations. The Surveys of Consumers financial literacy measure is a 28-item true–false knowledge quiz on financial management topics (Hilgert et al., 2003). In 2004, the National Institute on Aging’s Health and Retirement Survey (HRS) added three financial knowledge indicators to allow researchers to demonstrate ties between financial knowledge and financial outcomes over time (Lusardi & Mitchell, 2007a), however, HRS only samples adults over 50. In 2012, the OECD added financial literacy questions to the Programme for International Student Assessment (PISA) data collection that tests 15-year-olds worldwide (OECD, 2014). The assessment will be re-administered in 2015. PISA is not panel data, but could reveal changes in performance over time and “provide further evidence on the design and implementation of policies to enhance financial literacy” (OECD, 2014, p. 13).

Based on these datasets, and other studies, the evidence is well established that consumers consistently score poorly on financial literacy tests. The 2012 NFCS found that Americans answered 2.88 questions correctly, on average, on a five question financial literacy test (FINRA, 2013). Inceptia’s (2013) National Financial Capability Study of undergraduate students found 67 % surveyed scored either a “D” or “F” on a 50 question knowledge test and not one student scored in the “A” range. In 2012 the inaugural PISA financial literacy assessment tested 29,000 youth in 13 countries and found that 10 % of students can handle the most difficult financial literacy tasks—for example, figuring out transaction costs and income-tax brackets, 15 % of students scored below the performance baseline (OECD, 2014). In the USA, youth between 15 and 18 were given a financial literacy test that covered national financial literacy standards; just 4.7 % scored at 90 % or higher, with 62 % scoring below 69.9 %, and 21.9 % at or above 70 % (National Financial Educators Council, 2014).

Below average financial literacy scores have been associated with low income and less educated individuals (Lusardi & Mitchell, 2011), women (Fonseca, Mullen, Zamarro, & Zissimopoulos, 2010), Hispanics (Hogarth, Beverly, & Hilgert, 2003), African Americans (Hogarth et al., 2003), younger adults (in their 20s) (Lusardi & Mitchell, 2011), and older adults (retirees and near retirees) (Agarwal, Driscoll, Gabaix, & Laibson, 2009). The evidence of failing financial literacy has resulted in a general call for financial education over the past decade and more.

Evidence of the Relationship Between Financial Literacy and Financial Capability

The association between formal knowledge and financial behaviors is becoming well established (see Lusardi & Mitchell, 2007b for a review). Financial literacy studies suggest that financially literate individuals are better at budgeting and controlling spending (Perry & Morris, 2005); following recommended financial practices (Hilgert et al., 2003); handling mortgage and other debt payments (Stango & Zinman, 2009); saving money (Perry & Morris, 2005); maintaining a checking account and emergency fund (Hilgert et al., 2003); avoiding costly credit card revolving behavior (Lusardi, 2011); avoiding high-cost mortgages (Gerardi, Goette, & Meier, 2010); avoiding the use of the high-cost alternative financial sector (Lusardi, 2011); participating in the stock market (Hilgert et al., 2003; Lusardi, 2011); and planning for retirement (Lusardi, 2011; Lusardi & Mitchell, 2007a, 2007b). Individuals with low levels of financial literacy have an increased likelihood of late mortgage payments (FINRA Investor Education Foundation, 2013). Financially literate individuals also accumulate greater wealth (Lusardi, Mitchell, & Curto, 2013), supporting the link between financial capability and financial wellbeing. The question remains about the relationship between financial education and financial capability, the discussion of the next section.

Evidence on the Impact of Financial Education on Financial Literacy and Capability

The number of studies examining financial literacy education has begun to catch up with the proliferation of programming initiatives. To make sense of the information and to assess whether financial education effectively improves a consumer’s financial literacy, several review articles have been published over the past decade (Collins & O’Rourke, 2012; Fox, Bartholomae, & Lee, 2005, Hastings et al., 2012). Conclusions drawn from these reviews characterize the impact of financial literacy education on financial outcomes as mixed, inconclusive, negligible, ambiguous, inconsistent, and suggestive (Gale, Harris, & Levine, 2012; Hastings et al., 2012; Vitt et al.,2010), yet some were more optimistic (Collins & O’Rourke, 2012; Hogarth, 2006). The contradictory summary findings reported in recent review articles are likely explained by the lack of overlap in the articles included in each review. Miller, Reichelstein, Salas, and Zia (2014) present a correlation table based on the studies included in nine review studies published since 2007. These nine studies cover over 500 publications, yet in only two instances is the correlation coefficient above 0.2. This implies that recent reviews are summarizing different sets of studies, leading reviewers to different conclusions on the relative effectiveness of financial education in the financial capability building process.

Even with the lack of overlap in review studies, there is value in making observations and drawing conclusions based on a synthesis of evidence, particularly when a field is early in its development and when there are mixed claims, as witnessed in reviews of the efficacy of financial literacy education (Deeks, Higgins, & Altman, 2011). Once a field has progressed, a more promising pursuit is to assess existing empirical work systematically and quantitatively with a rigorous meta-analysis. In a meta-analysis comparable studies are combined statistically, providing the benefit of increasing the number of observations and the statistical power, and improving the estimates of the effect size (Deeks et al., 2011). Two meta-analyses were recently published to help summarize the results of multiple financial literacy education studies and document the extent of their effectiveness (Fernandes et al., 2014; Miller et al., 2014).

Fernandes et al. (2014) conduct a meta-analysis of 168 papers (covering 201 studies) to map how effective a financial literacy intervention is on “downstream’ financial behavior ,” adjusting for psychological factors. The analysis showed that consumers with higher levels of financial literacy demonstrated better financial behaviors, but once psychological traits were accounted for the direction of causality came into question. Overall, the impact of financial education helped explain “only 0.1 % of the variance in financial behaviors, with weaker effects in low-income samples” (p.1861). The efficacy of financial education does not appear to be long-lasting, Fernandes et al. (2014) describe a “decay” in the effects, with consumers forgetting what they learned within 20 months. The meta-analysis confirms the idea of “just in” time financial education, and the efficacy of delivering information when a consumer is preparing to make a specific financial decision (e.g., purchasing a car).

Another systematic meta-analysis of the effect of financial education on financial behaviors included 188 studies (140 from the USA and all 168 papers in the Fernandes et al. study); 43 % of these examined financial education that provided instruction on a variety of financial topics, 30 % on savings and retirement, and 5 % on mortgages (Miller et al., 2014). Based primarily on 19 of the 188 studies, a financial education intervention improved consumer savings, did not improve retirement savings or loan default rates and was inconclusive about whether there was a positive impact on record keeping (Miller et al., 2014). Intensity, the number of hours exposed to the financial education intervention was not generally associated with financial behaviors (Miller et al., 2014). The meta-analyses enable us to assess the general impact of financial literacy, while demonstrating the challenges faced by program evaluators.

The lack of conclusive evidence supporting a causal link can be attributed to a number of factors, including inadequate research design, data limitations, and the inadequate measure of these concepts in existing data (Amromin, Ben-David, Agarwal, Chomsisengphet, & Evanoff, 2010; Hung et al., 2009). The study’s omission of a consumer’s “biases, heuristics, and other non-rational influences on financial decisions” is another factor (Willis, 2011, p. 429).

Until we can substantiate the effective impact of financial education, some scholars suggest these interventions come at too great a cost (Fernandes et al., 2014; Willis, 2011). Posing one of the stronger arguments, Willis (2011) believes financial education is too costly because the baseline of consumers’ financial literacy is so low and financial literacy so complex (e.g., requiring numeracy skills). Consumption-oriented messaging and advertising starts in early childhood (Cross, 2002), consequently others question the capacity of financial education to cut through well-funded campaigns that work against the consumers’ interest. These points are valid, but human and monetary resources will continue to be invested in financial education; consequently, it is important for program developers and educators to consider evaluation strategies, as offered in the next section.

Evaluating Financial Literacy Education

Financial literacy education programs are abundant, yet too few are evaluated in the rigorous standard required to be published in peer-reviewed journals. The causal relationship between financial education, financial literacy, and financial outcomes has yet to be demonstrated due to either a lack of any real and measurable effect, or the lack of adequate efforts in evaluation design and measure of key concepts in existing data (Hung et al., 2009; PACFC, 2013). Fernandes et al. (2014) suggest setting standards for reporting evaluations, with consistent inclusion of the program characteristics (e.g., program length, period of measurement, curriculum, instructor, and participant characteristics). Even when promising results are reported, there is little opportunity for replication if details on the overall program are not shared. Moreover, comparisons are not possible between widely different interventions (e.g., comparing a one-time workplace workshop to a recurring course for college students with a financial counseling component). Fernandes et al. (2014) are not alone in suggesting improvements in the financial education evaluation process. Atkinson and colleagues (2007) review financial education evaluation efforts in the United Kingdom and highlight the need for: clear objectives, quality data, careful consideration of the sample size, well designed benchmarked measures for outcomes and literacy, a control group, and consideration of the time period necessary to identify change.

Responding to the calls for better evaluation, educators and organizations have provided guidance and insights on the design of financial education programs (Collins & Holden, 2014; Hogarth, 2006). The National Endowment for Financial Education has shared an online toolkit (http://toolkit.nefe.org/) with an evaluation manual and templates for data collection suitable for multiple program formats. There is general agreement that randomized controlled trial (RCT) is the gold standard for showing program impact. An RCT measures program impact through random assignment to a program group and a control group. The CFPB advocates for RCT level evaluation processes but also highlight the prohibitive expense (both time and money) in conducting RCTs. Other barriers to conducting a randomized study include: generating a large enough sample, identifying effects that are likely quite small, denial of services at random, self-selection, education consistency within a given program, and lack of collaboration between evaluators and program providers. While the need for experimental (or even quality quasi-experimental) evidence is clear when addressing wider policy decisions, other less rigorous approaches to program evaluation are likely more feasible and may be equally valid to assess a program.

Jacobs’ (1988) five-tiered approach to evaluation is presented as a basic guide for organizations delivering financial education programs. The advantage of this framework is that it encourages evaluation at each stage of programming, from conception to implementation to conclusion and follow-up. The assumptions underlying this framework are that evaluation (1) should be collected and analyzed in a systematic manner, (2) is an essential component to every program, (3) serves several functions, (4) has many audiences, and (5) should not detract from delivering a program (Weiss, 1988). The five-tiered approach is comprehensive in scope; it entails both formative and summative evaluation.

The elements of a comprehensive program evaluation, as outlined by Jacobs (2003), can be summarized in five tiers: (1) needs assessment, (2) monitoring and accountability, (3) quality review and program clarification, (4) achieving outcomes, and (5) establishing impact. The components of the model build upon one another, with each level requiring “greater efforts at data collection and tabulation, increased precision in program definition, and a greater commitment to the evaluation process ” (Jacobs, 1988, p. 50). The five-tiered approach should be used stepwise, particularly at first because later tiers require information collected from earlier tiers (Jacobs, 2003). However, evaluators can engage in several tiers at once, and previous levels will likely need to be revisited because evaluation is an iterative process (Jacobs, 1988; 2003). Immediately evident is the fact that evaluation is a graduated process , where identification of program impact comes only in the final stages of an involved, often costly, and comprehensive process. Table 4.1 outlines key stages with an application to financial education.

Table 4.1 Jacobs (1988) five-tiered approach to program evaluation
Full size table

Tier 1, the needs assessment, occurs during the initial stages of development when an organization is establishing the need for the program. Community indicators are collected and analyzed to show evidence of the problem (Jacobs, 2003). The need for financial literacy programs has been demonstrated with bankruptcy rates, consumer debt levels, and savings rates, that may be the result of financial illiteracy. The Jump$tart Coalition studies are examples of establishing a national need for youth financial education through an ongoing literacy test (Mandell, 2006). These data assist in determining the targeted goals and for planning effective program strategies. Only 22 % of 90 financial education programs conducted any formal needs assessment and in many instances program organizers assume the need for financial education so great that no further evidence was required (Vitt et al., 2000).

The monitoring and accountability tier of the evaluation framework collects information on four program elements: basic participant information, the education and services provided, personnel, and program costs (Jacobs, 2003). The goal is to document who has been reached by a program, and whether the program is being delivered as intended. It is important to provide program data to funders, participants, and the community, with a larger goal of sharing program data to draw broader attention to the issue of financial literacy (Jacobs, 1988). Frequently, monitoring and accountability in financial education programs is measured by collecting information during registration, an exit survey, or some other indication of participation. An example of monitoring and accountability data is the Consumer Federation of America’s America Saves program. Based on a program survey an estimated 10,000 Cleveland residents were persuaded to save more and 1500 savers were officially enrolled for accounts, counseling, and/or workshops (Cleveland Saves, 2002). This tier provides a description of the “status quo” of a program; the next tier evaluates program quality and consistency (Jacobs, 2003).

Relative to other tiers, quality review and program clarification contains more formative information for program organizers by assessing the program’s strengths and weaknesses, and goals and objectives, in an overall effort to improve the services provided (Jacobs, 1988). An evaluation would assess whether the program operates as it is meant to function. For example, if an organization delivers NEFE’s High School Financial Planning Program does it meet the performance standards set by the program developers? Second, program evaluators assess whether program-generated standards are being met, for example, if financial coaching is the intervention, an evaluation would critically describe the elements of the client–coach relationship . Additionally, information drawn from observations by program staff and participants is utilized to improve the program (Jacobs, 1988). For classroom-delivered material, information used for the third tier is commonly derived from an exit survey of teacher ratings, overall satisfaction with the class, and increases in knowledge. In early stages of a program, open-ended comments of participants often guide program changes.

Information collected during the achieving outcomes tier measures the effect of the program on the individual, whereas the monitoring and accountability tier described earlier simply highlights program utilization (Jacobs, 1988). In most cases short-term outcomes are measured, and research designs are less rigorous (Jacobs, 2003). A method of providing evidence for the achieving outcomes tier would be the use of a pre- and posttest. An evaluation of NEFE’s High School Financial Planning Program effectively uses this pre- and posttest approach to measure increases in financial knowledge, confidence, or intended improvements in financial behavior following the delivery of financial education (Danes & Haberman, 2004).

The most common approach to gathering information for tier four, achieving outcomes, is through follow-up contact to identify actions being taken that are congruent with program goals. In the workplace, whether the employee increases retirement contributions or enrolls in a retirement program is evident. In a high school financial literacy program the outcome goals are typically more wide-ranging, participants are more difficult to track, and measuring the fourth tier (achieving outcomes) becomes a significant challenge. The differential effects of programs are examined during this tier, for example, whether a financial education program has a greater impact on males versus females. This type of information assists in the improvement of programs. An external evaluator is often contracted to conduct this tier, particularly when new program-specific measures need to be developed (Jacobs, 1988). Programs planning to replicate and/or broaden their support (e.g., funders and stakeholders) require evidence from this stage to show effectiveness (Jacobs, 1988).

The fifth and final tier, establishing impact, builds on the fourth tier (achieving outcomes) and entails the measurement of more long-term impacts of a program (Jacobs, 1988, 2003). RCT occurs in this stage of the evaluation. Program impact evaluation again reflects the goals and objectives of a program, making it difficult to compare programs that do not have the same focus, and nearly impossible to identify the impact of programs with vaguely defined goals. At this stage, measurable levels of differences in treated and non-treated populations are reported. Thus, formal experimental or quasi-experimental approaches are required, where those receiving some form of financial education are contrasted with a similar sample not participating in the program (Jacobs, 1988). Only through such an experimental approach can the independent impact of the program be identified.

As the field of financial education develops, more evaluations have reached the fifth stage of evaluation. For example, in the Miller et al. (2014) meta-analysis only 14 % of the evaluation studies published before 2008 conducted an RCT, since 2008 43 % of evaluations were carried through to this fifth stage . While the studies in the meta-analyses draw on a wide range of samples, the approach to evaluation of programming efforts is decidedly more focused and straightforward.

Selection of a control group from the same population targeted in the needs assessment provides the necessary baseline for comparison. If the control group cannot be drawn from an identical population, then control variables measuring known determinants of the desired outcomes must be collected for both the treatment group and the control group. For example, if the desired outcome is increased personal savings, then information on income, wealth, household status, education, age, and employment status should be collected and controlled for the program impact (quasi-experimental) analysis. In this final stage the impact of a financial education program is identified. At this point, there are still too few examples of financial education evaluation research that have reached this fifth conclusive tier, largely because large scale, long-term, well-funded programs are required. Because of this simple fact, definitive statements on the impact of financial education remain premature.

Summary

Whether an educational intervention is offered in the workplace, school, or community, the current literature on the effectiveness financial literacy education remains mixed (Fernandes et al., 2014; Miller et al., 2014). The collective response by public and private organizations has been the delivery of financial education. Such investments come with the expectation of demonstrated and significant benefits to program participants. Without reliable, valid, and relevant information collected from well-designed program evaluations, financial educators jeopardize their ability to provide effective recommendations for the direction of education policy.

Currently, financial education programs often omit evaluation from program design. We described and outlined a comprehensive evaluation framework to aid programs in the evaluation process (see Table 4.1). Jacobs’ five-tiered approach to program evaluation is easy to understand and has the advantage of offering flexibility in its application. It is designed to address the needs of all financial education programs—programs in the design and development stage and/or programs that are well-established and ready to measure effectiveness through an RCT or quasi-experimental approach.

The evaluation of financial education programs should be an integrative part of the program development and delivery process, not an independent procedure used only to identify the benefits of undertaking the process. The assumptions underlying Jacob’s framework are a strength, and data should be collected and analyzed in a systematic manner and as an essential component to every program (Jacobs, 1988). Through replication of this process within all types of financial education programs, we stand to significantly increase our understanding of the independent effect of financial education on desired financial outcomes.

Continued study is needed to advance the field of financial education research and evaluation. Much remains to be understood about the effectiveness of program characteristics. Evidence of program development and delivery issues are much needed. As noted earlier, there is a lack of rigorous evaluation on the programs being delivered, and when appropriate, the use of RCTs will help us advance any conclusions that can be drawn about effectiveness. Research is needed to advance our understanding on issues of measurement (e.g., the use of psychometrically sound instruments that assess knowledge, financial capability, financial literacy, attitudes, and behaviors). When the researchers work from common definitions and measures, progress can be made. Additional research is needed to better understand programmatic effectiveness for varying audiences and populations. Improvements continue to be made, but the field will benefit from high quality studies on financial education programs and their associated outcomes.

About the Authors

Suzanne Bartholomae is an adjunct Associate Professor and Extension State Specialist in the Department of Human Development and Family Studies at Iowa State University. Her research interests include financial stress and coping, the financial socialization process, and the efficacy of financial education programs. Her work has been published in several scholarly journals including Family Relations, Journal of Family Issues, Journal of Family and Economics Issues, Journal of Adolescent Health, and Journal of Consumer Affairs. She earned her PhD in Family Science from The Ohio State University.

Jonathan Fox is the Ruth Whipp Sherwin Endowed Professor in the Department of Human Development and Family Studies; Director of the Financial Counseling Clinic; and Program Leader in financial counseling and planning at Iowa State University. His research focuses on financial education and financial socialization. He has served as PI for several financial education evaluations and his publications appear in journals such as Financial Services Review, Financial Counseling and Planning, Journal of Family Issues, and Journal of Consumer Affairs. From 2013 to 2014, Jonathan served as President of the American Council on Consumer Interests. He teaches courses in personal and family finance and received his PhD in Consumer Economics from the University of Maryland.