Data help us look at mechanisms and a variety of outcomes, tell a diversity of stories, and ensure that we serve the most vulnerable individuals.
In describing his partnership with the Department of Youth & Community Development, Judd Kessler says that this research agenda is in large part a story about data. In episode three of J-PAL Voices: The Impact and Promise of Summer Jobs in the United States, we look at the role that data play in summer jobs programs. Chekemma Fulmore-Townsend and Julia Breitman discuss how data on program participation have helped them meet racial equity targets in program delivery. Alicia Sasser-Modestino illustrates the variety of data sources used in an impact evaluation of Boston’s summer jobs program. Data help us look at mechanisms and a variety of outcomes, tell a diversity of stories, and ensure that we serve the most vulnerable individuals.
We would love to hear your comments and feedback at podcasts@povertyactionlab.org. J-PAL Voices is brought to you by J-PAL North America (https://www.povertyactionlab.org/na). Stay in touch via Facebook (https://www.facebook.com/JPAL.NorthAmerica/) and Twitter (https://twitter.com/JPAL_NA).
Please take the short, one-minute survey at https://j-p.al/voicessurvey.
Links:
Narrator
In a 2009 TED Talk, the author Chimamanda Ngozi Adichie cautions against what she calls “the danger of a single story”. “The single story,” she says, “creates stereotypes, and the problem with stereotypes is not that they are untrue, but that they are incomplete.”
Narrator
Throughout this series, we bring you the story of summer jobs programs by highlighting a multitude of perspectives. At J-PAL North America, we try to avoid telling just a single story by using data to get a snapshot of different experiences. In combination with individual narratives, quantitative data can help us look at the outcomes and issues we care most about. Rigorous evaluation and data analysis play a critical role in the work done by people like Julia Breitman, Senior Director of Youth Employment Programs at the New York City Department of Youth and Community Development.
Julia Breitman
We want to know if we're doing the right thing, and if we need to pivot, and if we need to improve something. And we want to know that we're making an impact.
Narrator (preceded by the short musical interlude that we use after every episode’s cold open)
From J-PAL North America, this is J-PAL Voices. I’m your host, Rohit Naimpally. On this season of J-PAL Voices, we take a close look at the impact and promise presented by summer jobs programs in the United States. We will bring you the stories behind the impact, as told by the people who create, participate in, and sustain these programs. Episode Three: Transform Data Use.
Julia Breitman
When you're very busy, you don't have time for that kind of introspective work. And so, when you're going through an evaluation, it gives you that moment to step back, and look at your processes, and not just your data, but also your implementation. I always appreciate that opportunity to really step back.
Narrator
That’s Julia Breitman again, talking about the benefits to doing rigorous evaluation. New York City’s Department of Youth and Community Development, or DYCD, has worked with researchers like the University of Pennsylvania’s Judd Kessler to measure the impact of its Summer Youth Employment Program.
Judd Kessler
In the literature about the New York City Summer Youth Employment Program, they said that it accomplishes three main goals: providing support to low income youth and their families, giving youth sort of a step up the ladder of their careers or to sort of help them with future earnings or future educational attainment, and third, that it would keep youth out of trouble.
Narrator
Judd Kessler and his team had to figure out how they were going to get at whether the Summer Youth Employment Program, or SYEP, was meeting each of those three goals.
Judd Kessler
We took each of those things and we asked: what kind of data could we get that would evaluate the program on those dimensions that it was designed to achieve?
Narrator
On the third dimension, of keeping youth out of trouble, Judd and his co-authors found that SYEP was saving 20 lives a year, on average. I asked Judd if he could speculate on what could be behind these remarkable results on reducing mortality.
Judd Kessler
I can do more than speculate in part because I spent what can only be described as dark and depressing days, weeks I should say, in a basement room of the Department of Health and Mental Hygiene in New York, where they keep the New York City death records.
Judd Kessler
What we found in that data was that the deaths that were being prevented by the program were deaths from external causes, and primarily that's homicide. So, the way that the story fits together for us was that here's this program, it's keeping youth out of trouble, and the ways that we observed that are they're less likely to be incarcerated in state prison and they're more likely to be alive.
Narrator
In addition to looking at the New York City death records, Judd and his team also looked at publicly available data on incarceration in New York state prisons. Combining the trends that they saw in both datasets, they better understood the story behind how Summer Youth Employment Programs save lives.
Judd Kessler
The sort of narrative that played out is that it's getting you out of unsafe activities that you might do if you are unoccupied during the summer and into safer activities both in that summer when you’re working, but also for the longer term.
Narrator
This narrative resonates with what Sara Heller found in her study of the One Summer Chicago summer jobs program. She found that the program reduced violent crime arrests by 42% over the first year after the program.
Sara Heller
One thing you might've thought is that this is just a story about keeping youth busy. That when you're giving them a job over the summer, they're literally busier and so they can't at the same time be standing on the corner getting into trouble. But it turns out that's not the only reason. So, you can in fact throw out the summer months of the program from the data entirely, look only after the program is over, and you still see a decline in violence. And so, the youth are taking something with them that changes their future behavior. It’s not just a question of keeping them mechanically busier over the summer.
Narrator
Researchers like Sara and Judd are able to read and measure these stories in large part because they can access high quality administrative data. These are datasets that are primarily collected for operational purposes. When studied with the right research questions in mind, they can yield valuable insights.
Judd Kessler
This research agenda, at least for me, is a story about the ability of organizations to keep track of data and the availability of the administrative data.
Narrator
Judd and his co-authors use a variety of administrative datasets, from publicly available data from New York state prisons to more restricted tax data from the Treasury’s Office of Tax Analysis. Kenny Hofmeister is a Senior Research Analyst at the University of Chicago Poverty Lab. Through his work with Sara Heller on the summer jobs evaluations in Chicago, he sees a number of advantages to using administrative data.
Kenny Hofmeister
With administrative data, we can also observe a lot more information about the students who are participating in the program, for example. So, you know, have access to things like test scores, GPA, in some cases attendance records, things that are measured directly and hopefully with fidelity, in a way that you don't have to worry about people reporting their attendance. You don’t have to worry about people reporting their test scores.
Narrator
In addition to measurement of important outcomes, administrative data allow us to look at a program’s long-term impacts. When data are available over a number of years, we can study how a program’s impacts manifest over time at a relatively low cost.
Sara Heller
This is the benefit of using administrative data, is that we don't have to raise a bunch of money to go find them wherever they are and survey them. But we can sort of track them in administrative data. So, we're hoping to do that for longer term.
Narrator
For all the advantages that they bring, administrative data are ultimately only as good as the outcomes that they capture. Kenny Hofmeister is cognizant of these limitations to what can be asked by relying solely on administrative data.
Kenny Hofmeister
In some ways we're also limited by what's in that administrative data. So, if we think that there are things that are important, about selection into applying to One Summer Chicago for example, or related to some of the outcomes we're looking at – and they're not in the administrative data and they're not sort of proxied by anything in the administrative data, or we can’t be sure – we're kind of stuck, you know, we don't have a way to measure things that we don't think are in the data. So, if we were doing surveys, we would be able to maybe ask questions that we thought were important to measure.
Alicia Sasser-Modestino
A lot of this research is very interdisciplinary and combines big data analysis using administrative datasets with survey data analysis, to be able to understand the mechanisms behind what we're finding in the administrative data.
Narrator
That’s Alicia Sasser-Modestino, an economist at Northeastern University, describing her research on Boston’s Summer Jobs Program. When Alicia started studying the program, the city already ran a survey of program participants. In partnership with the city, Alicia expanded the survey to include questions designed to help us understand how the program effects change.
Alicia Sasser-Modestino
We measure different aspects of youth development in these three different areas. So, one of them is looking at community engagement and social skills. So, we measure things like how to manage your emotions, how to resolve conflict with a peer, how to ask for help from an adult when it's needed. How much you feel engaged with your community? How much do you feel like you have something to contribute to the groups that you belong to? We ask about your role models and your mentors. In a second area, we ask about your academic aspirations after high school to attend either a four-year college or a two-year college, or engage in some kind of workforce development training. And then we also ask about your job readiness skills. So, there's a part of the program that's aimed at teaching them how to write a resume, how to draft a cover letter, how to practice interviewing with an adult, explore different career opportunities. So, we ask about all of those job readiness skills as well. And then what we do is we look to see which one of these buckets of changes that we see over the summer are correlated with the subsequent reduction in crime over the next 18 months. And what's really interesting is with each of these different types of outcomes, there's different mechanisms that are going on.
Narrator
In her research, Alicia finds that the summer jobs program reduced criminal arraignments for violent crime by 35%, an impact very similar to the one measured by Sara Heller in Chicago. Alicia’s analysis of the survey data in Boston finds that the reduction in crime is strongly correlated with improvements in community engagement, social skills, and conflict resolution.
Alicia Sasser-Modestino
We've been deliberately testing these different mechanisms over time so that we can inform policymakers about which parts of the program seem to be impactful, where they can increase their investments, or how can you translate some of the lessons learned from the summer job program to other year-round activities that you do.
Narrator
Policymakers act on the lessons from evaluations like Alicia’s to refine their programs further. Julia Breitman in New York City told me how the Department of Youth and Community Development reacted to the findings from their research with Judd Kessler and his team.
Julia Breitman
We realized that it has this largest impact on young people who are considered vulnerable. And so, we increased funding for those initiatives, we increased recruitment of young people who would be considered at-risk because the data was incredibly clear.
Narrator
The data also show that following their time in New York City’s Summer Youth Employment Program, youth tend to gravitate to the jobs that they worked in during those summers.
Julia Breitman
Another piece of the puzzle that Judd’s research showed us was how important it is, the industry where young people are working, that it does have an impact on their long-term earning.
Narrator
Julia’s team at the Department of Youth and Community Development embodies policymaking that is responsive to evidence and data. Even as they have doubled down on the program’s most successful aspects, they have adapted to enhance the program’s effectiveness further.
Julia Breitman
Young people who worked in social services, tend to stay in social services, and it's a wonderful career choice as I can attest, but we want them to know other options. And so, we started growing our private sector, just to ensure that even young people who are coming from low income backgrounds have as many opportunities as young people coming from any other background, right? And that's really what it all goes back to, is we want to ensure that we are opening doors for all of our young people.
Narrator
High quality data infrastructure can help inform summer jobs programming beyond the results from an impact evaluation. Commonpoint Queens is a community-based organization that implements New York’s Summer Youth Employment Program across the borough of Queens. CEO Danielle Ellman discusses how Commonpoint uses data to get a holistic perspective on the families of the youth whom they work with.
Danielle Ellman
We invested significant resources in a really sophisticated database client management software that would help better link families for us, so that we could be really much more intentional in our service strategy. And we would also uncover the places where, because we're operating in 50 locations in the borough of Queens, where we might already be servicing families and not really know and potentially allow us to do that work even better.
Narrator
Like Danielle, Chekemma Fulmore-Townsend, the CEO and President of the Philadelphia Youth Network, sees great value in getting a comprehensive picture across locations and services. The Philadelphia Youth Network, or PYN, manages Philadelphia’s WorkReady summer jobs program and is well positioned to coordinate across 80 non-profit agencies and private companies.
Chekemma Fulmore-Townsend
One of the things that we've found that works well is, if you think about this system in scale, right, we could have a system where 80 different organizations would have to report to multiple entities, because we are collecting resources from the workforce system, from child welfare system. And so, you could just be spending hours of duplication in terms of the reporting if we had all of these singular systems not working together. The beauty in the coordination is you have one entity collecting all of the information from all 80 organizations and is, you know, quite literally in an up-to-the minute kind of ways, can give you a comprehensive picture of what's happening with the whole, as well as disaggregate that comprehensive picture to tell you about what you're specifically interested in. So, if you want to know what's happening in this zip code of the city, I can tell you how many programs I have there, how many kids are participating, how much money did they earn. But I can also tell you that for the whole system. And I think what that does is shift the dynamic so that the service providers who are delivering the services can really focus on building the relationships, making sure that the experience is quality, building those partnerships and PYN will focus on reflecting the sum of the whole to all of the parts. And it just really allows everybody to operate from their position of strength and of focus.
Narrator
For coordinating entities like PYN and organizations like Commonpoint, high quality data can foster a cycle of continuous improvement. Data also play a crucial role in helping advance their mission of giving voice to underserved youth.
Chekemma Fulmore-Townsend
Certainly, it has influenced our approach to equity and access. So, we were able to ask very clear questions about who's actually turning their application into participation, and who’s consistently falling out of that group. And that led to some different strategies. When we work specifically with the juvenile justice system over the last eight to ten years, I would say would say over the last eight years, we've constantly revisited that question to say, “What in our practice is really working to ensure that those young people get connected?” And what kinds of partnerships and relationships do we need in order to ensure that there's greater, not just access to the opportunity, but participation or uptake of the opportunity? And so, we constantly use data to refine that. We use data also to articulate who we're serving and where they're coming from.
Narrator
In New York City, a similar focus on racial equity and equitable service provision proved especially valuable when the COVID-19 pandemic struck.
Julia Breitman
There was a racial inclusion taskforce that really looked at neighborhoods that were the hardest hit by the COVID-19 pandemic, which as you know, across the country, but in New York City specifically, it really hit communities of color and low-income communities the hardest. And so, this taskforce identified neighborhoods that should be targeted for services. And while our program operated partially on a lottery, partially by selection to targeted neighborhoods, 91% of the participants came from those neighborhoods. So, we know the services are really going to young people who need them the most
Narrator
When viewed in the abstract, topics like data access and linkage can seem far removed from the heart and soul of summer jobs programs. In many ways though, the story of summer jobs programs could not be told without the rich administrative and survey datasets that so much investment has been poured into. These data help us act on crucial questions of equity and opportunity. They may not paint the full picture of summer jobs programs. But we could not tell the story of the impact and promise of these programs, without the deep insights that these data produce.
Narrator
J-PAL voices is produced by Dave Lishansky and written and narrated by Rohit Naimpally. Elizabeth Bond designed our logo. Special thanks to Yijin Yang for her inputs and support. Transcription assistance was provided by Caroline Garau and Yiping Li. For this week’s interviews, we thank Julia Breitman, Judd Kessler, Sara Heller, Kenny Hofmeister, Misuzu Schexneider, Alicia Sasser-Modestino, Danielle Ellman, and Chekemma Fulmore-Townsend. Our email address is podcasts@povertyactionlab.org; we would love to receive your comments and feedback.
Narrator
On the next episode of J-PAL Voices...
Rashad Cope
Young people are more engaged by way of them participating in summer jobs. They're more engaged because they develop a sense of who they are by participating in summer jobs. They develop increased competency in certain areas, they know how to deal with conflict resolution, and they just know how to engage better with adults and peers