LEAP Evaluation (Printer-Friendly Format)

	http://www.researchforum.org/project_general_11.html

LEAP Evaluation

General Information

View a brief abstract of this project.

View a complete, printer-friendly profile of this project.

Evaluator(s)	MDRC
Investigator(s)	Not Reported
Sponsor(s)	Ohio Department of Human Services
Funder(s)	Ohio Department of Human Services Ford Foundation BP America Treu-Mart Fund George Gund Foundation Procter and Gamble Fund US Department of Health and Human Services, Administration for Children and Families US Department of Health and Human Services
Subcontractor(s)	Not applicable

Domain	Income Security/TANF
Status	Completed (final report released)
Duration	Jul 1987 - Jun 1997
Type	Research and/or Program Evaluation
Goal	To evaluate the components of Ohio's Learning, Earning and Parenting Program (LEAP).
Program/Policy Description	LEAP is a statewide welfare initiative that uses financial incentives and penalties, combined with case management and support services, to promote school attendance by pregnant and parenting teenagers on welfare. Participation is mandatory for all pregnant women and custodial parents under 20 years old who are receiving Aid to Families with Dependent Children (AFDC) and do not have a high school diploma or GED certificate.
Notes	No notes reported.

Last Updated	09/08/97
Type of Summary	Reviewed
External Reviewer(s)	Veronica Fellerath (MDRC)
Contact(s)	Veronica Fellerath (not reported) MDRC 16 East 34th Street 19th Floor (T) (212)-532-3200 (F) (212)-684-0832
Publications Department	MDRC Publications (publications@mdrc.org) MDRC 16 East 34th Street 19th Floor (T) (212) 532-3200 (F) (212) 684-0832

Populations Studied

Target Population	Recipients/participants/clients Pregnant/parenting teens Applicants
Subgroups Analyzed	None
Sample Size and Unit	Almost 7,017 pregnant or parenting teenage welfare applicants/recipients, selected from all teens eligible for LEAP in the 12 study counties from the time the program began operation in July 1989, through September 1991 (99% female). Random sample of 5,611 recipients in program group (80%); 1,406 recipients in control group (20%).

Sites Studied

Cuyahoga county (Cleveland), Ohio
Franklin county (Columbus), Ohio
Hamilton county (Cincinnati), Ohio
Lawrence county (Ironton), Ohio
Lucas county (Toledo), Ohio
Muskingum county (Zanesville), Ohio
Stark county (Canton), Ohio
Jefferson county (Steubenville), Ohio
Lorain county (Lorain), Ohio
Montgomery county (Dayton), Ohio
Summit county (Akron), Ohio
Trumbull county (Warren), Ohio

Program Components, Policies, and Activities Evaluated

Financial incentives

Increase in income benefits for program enrollment
Increase in income benefits for program participation
Financial Incentives - misc.

Financial disincentives/Sanctions

Reduced benefits for non-compliance

Program requirements

School attendance

Social/Support services

Child care
Transportation
Case management

Administration/Implementation

Administration/Implementation - misc.

Variation in program components across sites?

Yes

Notes on program components

Financial disincentives/ sanctions: $62 deducted per month for noncompliance. Financial incentives: $62 added for school enrollment. $62 added monthly for regular school attendance. Program operations: Areas examined include identifications of eligible teens, development of an internal structure, and linkages between school and human service agencies.

Program requirements: Participants required to enroll in school and maintain specified attendance record. Social/Support services: Subsidies for child care and transportation to school are provided. Case manager assigned to each participant.

Outcomes Assessed

Education

High school graduation/GED receipt
School attendance

Employment

Job attainment

Family and relationship outcomes

Births/pregnancies
Family formation and stability/Living arrangements

Income security

Child support payments
Earnings
Food stamps receipt
Welfare receipt

Service utilization

Service utilization - misc.

Program implementation

Program Implementation - misc.

Financial costs and benefits/cost-effectiveness

Financial costs and benefits/cost-effectiveness - misc.

Types of Studies

Type	Cost-Benefit Study
Aim	To assess whether the benefits attributable to LEAP exceed the program's costs, from the perspectives of eligible teens, government budgets and society as a whole.

Type	Implementation/Process Study
Aim	To analyze the institutional structure of the county LEAP programs, operational issues confronted by LEAP staff and the way eligible teens experience LEAP.

Type	Impact Study (Controlled Experiment)
Aim	To assess whether LEAP has improved eligible teens' school enrollment and attendance and, in the longer term, whether the program produces higher rates of school completion and other longer term impacts.

Data Sources

Source	Survey
Title	In-person and telephone baseline survey
Sample Characteristics/Data Collection	7,017 welfare recipients. Random sample of 5,611 (80% of total) teen mothers in program group, 1,406 (20% of total) in comparison group. Collected at baseline.
Sites	Cuyahoga county (Cleveland), Ohio Franklin county (Columbus), Ohio Hamilton county (Cincinnati), Ohio Lawrence county (Ironton), Ohio Lucas county (Toledo), Ohio Muskingum county (Zanesville), Ohio Stark county (Canton), Ohio Jefferson county (Steubenville), Ohio Lorain county(Lorain), Ohio Montgomery county (Dayton), Ohio Summit county (Akron), Ohio Trumbull county (Warren), Ohio
Response Rate/Attrition Notes	Response rate not reported.
Additional Execution Notes	No notes reported.

Source	Survey
Title	In-person and telephone first survey
Sample Characteristics/Data Collection	2,089 welfare recipients. Random sample of 1,051 teen parents in program group; 1,038 in control group. Collected 4 - 21 months after random assignment.
Sites	Cuyahoga county (Cleveland), Ohio Franklin county (Columbus), Ohio Hamilton county (Cincinnati), Ohio Lawrence county (Ironton), Ohio Lucas county (Toledo), Ohio Muskingum county (Zanesville), Ohio Stark county (Canton), Ohio
Response Rate/Attrition Notes	Fielded sample: 2,808 welfare recipient teens (random sample of 25% program group, 100% control group). Reported response rate: 74.4%
Additional Execution Notes	Respondents with 12 months of follow-up data: 1,188 total (605 program group, 583 control group).

Source	Administrative data
Title	High school and ABE/GED enrollment and attendance records
Sample Characteristics/Data Collection	Records for 889 welfare recipients. Collected 1989-90 and 1990-91 academic years.
Sites	Cuyahoga county (Cleveland), Ohio Franklin county (Columbus), Ohio Hamilton county (Cincinnati), Ohio Lawrence county (Ironton), Ohio Lucas county (Toledo), Ohio Muskingum county (Zanesville), Ohio Stark county (Canton), Ohio
Response Rate/Attrition Notes	N/A
Additional Execution Notes	No notes reported.

Source	Administrative data
Title	High school/ GED completion records
Sample Characteristics/Data Collection	Records for 4,325 welfare recipients. Sample of all program and control group members (3,471 program group; 854 control group). Type of sampling method not reported. Collected 1989-1992.
Sites	Cleveland, Ohio East Cleveland, Ohio Columbus, Ohio Cincinnati, Ohio Toledo, Ohio
Response Rate/Attrition Notes	N/A
Additional Execution Notes	No notes reported.

Source	Administrative data
Title	Case Files
Sample Characteristics/Data Collection	263 case files of welfare recipients. Random sample of LEAP program members and AFDC control members. Collected 18 months after random assignment.
Sites	Cuyahoga County (Cleveland), Ohio Franklin County (Columbus), Ohio Hamilton County (Cincinnati), Ohio
Response Rate/Attrition Notes	Reported response rate: 74%
Additional Execution Notes	No notes reported.

Source	Focus Group
Sample Characteristics/Data Collection	55 members of case file sample who were active in program in 1992 and additional teens who had not complied with teen rules. Data collection schedule not reported.
Sites	Cuyahoga County (Cleveland), Ohio Franklin County (Columbus), Ohio Hamilton County (Cincinnati), Ohio
Response Rate/Attrition Notes	250 teens fielded.
Additional Execution Notes	No notes reported.

Source	Survey
Title	In-person three-year survey
Sample Characteristics/Data Collection	913 welfare recipients. Non-random probability sample of 446 program group members and 467 control group members. Collected 2/94-7/94.
Sites	Cuyahoga county (Cleveland), Ohio Franklin county (Columbus), Ohio Hamilton county (Cincinnati), Ohio Lawrence county (Ironton), Ohio Lucas county (Toledo), Ohio Muskingum county (Zanesville), Ohio Stark county (Canton), Ohio
Response Rate/Attrition Notes	1,178 teens fielded. Random sample of 25% program members and 100% control members. Reported response rate: 77.5%
Additional Execution Notes	No notes reported.

Source	Field Research
Title	Key informant interviews, observations, and site visits
Sample Characteristics/Data Collection	LEAP supervisors and staff and Ohio DHS staff (number in sample not reported). Sampling method not reported. Collected prior to late 1992.
Sites	Cuyahoga county (Cleveland), Ohio Franklin county (Columbus), Ohio Hamilton county (Cincinnati), Ohio Lawrence county (Ironton), Ohio Lucas county (Toledo), Ohio Muskingum county (Zanesville), Ohio Stark county (Canton), Ohio
Response Rate/Attrition Notes	Fielded sample not reported.
Additional Execution Notes	No notes reported.

Source	Administrative data
Title	Unemployment Insurance (UI) and AFDC payment records
Sample Characteristics/Data Collection	4,151 welfare recipients (excluding older teens). Sample of all program and control group members (3,479 program, 672 control). UI records collected 14 calendar quarters after random assignment. AFDC records collected 8 calendar quarters after random assignment.
Sites	All sites.
Response Rate/Attrition Notes	N/A
Additional Execution Notes	No notes reported.

Findings Available

Interim Implementation Findings
Final Implementation Findings
Interim Impact Findings
Final Impact Findings
Final Cost-benefit Findings

Findings

08/01/97: LEAP Evaluation: Final Report of Ohio's Welfare Initiative to Improve School Attendance Among Teenage Parents

Final Implementation Findings: "The state and county welfare departments successfully implemented LEAP statewide, forging a link between welfare departments and schools and putting the incentive system in place"(ES-7). "Almost all eligible teens (93%) were touched by LEAP’s incentives, with 75% earning at least one bonus and 56% qualifying for at least one sanction"(ES-7). "The majority of teens with multiple sanctions reported diminished spending on essentials for their families, especially clothing and food. Most teens with multiple bonus payments reported spending a large share if the additional money on their children"(ES-7). Final Impact Findings: "LEAP’s immediate goals- and the behaviors directly targeted by its financial incentives- were to improve the teens’ enrollment and attendance in school or in a GED program. After one year of follow-up, LEAP achieved a substantial increase in these outcomes, for both initially enrolled and not initially enrolled teens. The program also produced modest, but statistically significant, increases in completion of the 9th, 10th, and 11th grades"(ES-8). "LEAP’s effect on school completion- its longer-term education goal- was found to be limited when measured three years after teens entered the program. The program increased GED receipt among teens who were enrolled in school when they entered LEAP. There were no impacts on high school graduation or GED receipt among teens who were not enrolled in school when they entered LEAP"(ES-8). "Two-thirds of the teens in the sample did not receive a high school diploma or a GED certificate within a three-year follow-up period. However, approximately one-sixth of the teens were still in school at the end of that period and could have graduated or received a GED subsequently"(ES-9). "Teens in both the program group and the control group experienced substantial growth in their employment rates and earnings during the four-year follow-up period. By the end of the follow-up, four out of five teens worked in a job covered by Unemployment Insurance. Overall quarterly employment rates for such employment increased from 17% in the third quarter or year 1 to 40% in the last quarter of year 4. If employment not covered by Unemployment Insurance could have been taken into account, the employment rates and earnings presented in this report would have been higher"(ES-10). "The LEAP program increased employment for initially enrolled (in school) teens, but not for teens who were not initially enrolled. These impacts were strongest early in the follow-up period"(ES-10). "LEAP increased initially enrolled teens’ earnings the first two years of follow-up, but the impacts became smaller in later years"(ES-10). "Rates of AFDC receipt remained high throughout the four-year follow-up period, but were declining over time. More than 60% of all teens were on welfare for at least one of the last three months of follow-up, and close to 30% received AFDC continuously during the last two years"(ES-13). "During the final two years of follow-up, LEAP reduced welfare receipt by increasing the number of teens who were not receiving any AFDC and reducing the number who were receiving AFDC in every single month"(ES-13). "LEAP reduced both the number of teens receiving AFDC and the amount of AFDC they received. Reductions were somewhat smaller for those not initially enrolled than they were for teens who were enrolled in school when they entered LEAP"(ES-13). "During the last two years of follow-up, LEAP caused reductions in the amount of AFDC received that exceeded the program’s positive effects on earnings"(ES-16). Final Cost-benefit Findings: "The LEAP program was relatively inexpensive, with an upfront investment by the Ohio Department of Human Services of $1,388 per teen, or $747 for 12 months. Costs were higher for initially enrolled teens than for those not initially enrolled"(ES-16). "Because of LEAP-induced savings on AFDC, Food Stamps, and related Medicaid expenditures, ODHS recovered its investment in the program over the four-year follow-up period"(ES-17). "LEAP teens experience a net loss of $1,110 over the four years"(ES-17).

03/01/96: LEAP Evaluation: Three-year Impacts of Ohio's Welfare Initiative to Improve Attendance Among Teenage Parents

Interim Implementation Findings: "[For subsample of teens in 7 of the 12 counties] County LEAP programs experienced difficulties in implementing LEAP’s incentive structure during the program’s first two to three years of operation, particularly in urban counties, but have efficiently carried out bonuses and sanctions since then" (ES-5). Interim Impact Findings: "Almost all eligible teens (93%) were touched by LEAP’s incentives, with 75% earning at least one bonus and 56% qualifying for at least one sanction" (ES-5). "Bonus and sanction rates were strikingly different for teens who were enrolled in school when they became eligible for LEAP and teens who were not" (ES-5). "The majority of teens with multiple sanctions reported diminished spending on essentials for their families, especially clothing and food. Most teens with multiple bonus payments reported spending a large share of the additional money on their children" (ES-6). "For the full sample of teens, LEAP substantially increased high school enrollment, attendance, and progress through the eleventh grade, but did not have a significant impact on high school graduation" (ES-6). "Two-thirds of the teens did not receive a high school diploma or GED certificate within the three-year follow-up period" (ES-8). "LEAP’s school impacts varied substantially across geographic areas. The program significantly increased high school graduations in Cleveland, produced no significant effects in Cincinnati, Columbus, and Toledo, and significantly increased GED receipt outside these large urban areas" (ES-10). "Overall, LEAP increased the likelihood that teens would be working three years after they became eligible for the program and reduced the likelihood that they would be receiving AFDC benefits" (ES-12).

05/01/93: LEAP Evaluation: Interim Findings on a Welfare Initiative to Improve School Attendance Among Teenage Parents

Interim Implementation Findings: "LEAP has operated relatively smoothly during the study period, considering that the program is complex to administer. Not surprisingly, all counties experienced operational problems, particularly during the start-up period" (xxi). "LEAP has reached a large and diverse population of teens. However, identifying eligible teens - particularly those who do not head welfare cases - was quite difficult, in large part because of the limitations of the welfare computer system" (xxi). "Most of the difficulties were in processing needed grant adjustments. The severity of this problem varied depending on the organizational structure of LEAP in each county" (xxii). "Several counties have gone beyond the requirements of the LEAP model to create a relatively expansive definition of the case management function" (xxii). "School districts have generally cooperated with LEAP. Still, it often has been difficult for LEAP staff to establish reliable attendance reporting systems" (xxiii). "Because it provides no education services, LEAP is dependent on the local school environment, which varies across counties and school districts. One particularly important factor in the operation of LEAP is the availability and accessibility of alternatives to traditional high school" (xxiii). "LEAP has largely avoided the legal challenges that initially hindered the Learnfare program in Wisconsin, which includes only grant reductions. It is likely that this has resulted in part from specific aspects of LEAP’s design" (xxiv). "The vast majority of teens have been scheduled for at least one bonus or sanction during the time that they have been eligible for LEAP. More teens earned bonuses than sanctions" (xxiv). "Although almost all teens earned at least one grant adjustment, many were not scheduled for large numbers of adjustments during their time in LEAP" (xxv). "There are important differences across counties, both in the rates of sanction and bonus requests and in the proportion of requested adjustments that actually occurred" (xxv). "Some teens were scheduled for many sanctions and never cooperated with LEAP" (xxvii). "There have been important changes in the operation of the financial incentive system over time" (xxvii). "Largely because of the program’s unusual design, LEAP’s sanctioning rate was much higher than the rates measured in previous evaluations of welfare-to-work programs for adults" (xxix). "Overall, in a survey, about half of eligible teens characterized LEAP as "fair" and a third called it "unfair". Not surprisingly, teens who had been sanctioned had much more negative views of the program than did other teens" (xxix). "Many teens report that child care arrangements are critically important to their decisions about school attendance. However, most LEAP teens did not use child care assistance offered by the program" (xxx). Interim Impact Findings: "LEAP had two important and statistically significant effects on teens’ enrollment in high schools and adult education programs: It increased school retention among in-school teens and induced many dropouts to return to a school or program. Most teens who were recent dropouts and resumed their schooling because of LEAP returned to high school. Longer-term dropouts who returned always entered adult education programs" (xxx). "LEAP’s impact on enrollment was significant for most subgroups of eligible teens. However, LEAP appears to be less effective for older teens and those who had two or more children when they were identified as eligible for the program" (xxxii). "LEAP had a consistent overall impact on enrollment across counties. However, the composition of this impact - i.e., the proportion of the effect that is attributable to high school versus adult education program enrollment - varied substantially, partly owing to school district policies" (xxxv). "The impact estimates probably understate the effects that would be found in an ongoing LEAP program because the groups of teens for whom LEAP had slight impacts were larger during the period covered by this study than they will be later in the program’s operation" (xxxvi). "In addition to promoting enrollment and retention, LEAP improved the daily attendance of teens enrolled in high school. Among teens who enrolled in adult education programs, LEAP teens attended less than controls. However, there was still an overall small positive impact on total attendance in these programs" (xxxvi). "Data on the performance of in-school teens indicate that LEAP’s impacts on high school enrollment have already translated into more graduations, although it is too early to tell how large this increase will eventually be. LEAP has also led to an increase in the proportion of teens who have passed the GED test" (xxxviii). "Many teens have negative views of their high school experiences. These perceptions appear to have affected their decisions about enrollment and whether to attend high school or adult education programs" (xxxvii).

Recommendations

LEAP Evaluation: Final Report of Ohio's Welfare Initiative to Improve School Attendance Among Teenage Parents (08/01/97)

"The LEAP results suggest that it is sensible to include a LEAP-like approach as one element of a state’s welfare strategy"(ES-20). "The LEAP experience highlights the importance of meeting the implementation challenges inherent in such programs, particularly the needs for ongoing case management and well-designed management information systems"(ES-20). "The importance of initial school enrollment status as a predictor of program success underscores the need to prevent teen parents from dropping out of school. Recent program changes address this problem by limiting LEAP’s pregnancy and age-of-child exemptions, which could have accounted for some teens’ dropping out after becoming pregnant"(ES-21). "LEAP’s impact on school attendance was encouraging, and would be strengthened if more teens responded to the program’s incentive structure. This outcome might be achieved by changing the incentive structure and its implementation, and by directly addressing teens’ reservations about going to school"(ES-21). "In addition to stimulating school enrollment and attendance, programs like LEAP should provide incentives that specifically reward academic progress and school completion. Special school-based services may be needed to improve school outcomes for teen parents who respond to incentives and return to school"(ES-22). "Special attention is needed for teen parents who had dropped out but who returned to school because of LEAP, and for older teens who were far behind their schooling when they dropped out"(ES-22). "The 1996 federal welfare law poses important challenges for policy makers who develop and administer teen parents programs like LEAP. In addition to getting teen parents to attend school, program must address work requirements and time limits on receipt of federally provided welfare assistance that will affect young parents after they graduate or age out of programs like LEAP. Therefore, these programs may have to combine their education message and incentives with more employment-oriented activities"(ES-23). "The problems facing teen parents on welfare are substantial and complex. LEAP successfully addressed one problem area by increasing these teens’ school attendance. However, more needs to be done to improve the self-sufficiency of these teens and to reduce the persistent poverty among them. The LEAP evaluation shows that there are no easy answers. More experimentation is necessary, both within and outside the context of the LEAP program"(ES-23).

LEAP Evaluation: Three-year Impacts of Ohio's Welfare Initiative to Improve Attendance Among Teenage Parents (03/01/96)

"The findings also point to the limits on what incentives alone can do to increase high school graduation. LEAP gets more young people to the schoolhouse door, but too many subsequently walk back before getting a diploma. The greater success in Cleveland suggests some strategies for improving on these results. But more far-reaching changes in teen’s school experience will likely be needed if LEAP is to realize its full potential" (ES-13).

Existing Publications

08/01/97	LEAP Evaluation: Final Report of Ohio's Welfare Initiative to Improve School Attendance Among Teenage Parents	MDRC
03/01/96	LEAP Evaluation: Three-year Impacts of Ohio's Welfare Initiative to Improve Attendance Among Teenage Parents	MDRC
05/01/93	LEAP Evaluation: Interim Findings on a Welfare Initiative to Improve School Attendance Among Teenage Parents	MDRC
06/01/98	LEAP Evaluation: Teenage Parent Programs: A Synthesis	MDRC
09/01/98	LEAP Evaluation: Moving Teenage Parents in Self-Sufficiency: Lessons from Recent Demonstrations	Mathematica