Improving outcomes in health, behavioral health & social services through applied social research
It’s Behavioral Health week on the American Evaluation Association’s AEA365 blog, and I had the opportunity to contribute today’s post on lessons learned in evaluating cross-systems programs. I’m grateful for the opportunity to share some of our experiences. It was an interesting exercise to think about how our work has taken a systems approach over the years.
How is program evaluation like tennis?
I’ve been playing tennis on and off for as long as I can remember. I took a long break while my son was younger and my free time was a lot more limited. Now that I’m an empty nester and working on some fitness goals, it seemed like a good time to pick up my racquet again. But once I got on the court, I discovered that not only was I pretty rusty, my recall of how to hit basic strokes was off — even though I’d had plenty of practice and repetition in the past.
So how is program evaluation like tennis? It’s the issue of “drift.” With my tennis game, I was hitting my forehand shot how I thought I remembered it was supposed to be done. However, I quickly discovered with a new instructor that I had a lot to re-learn! In evaluation, we design protocols and procedures for collecting data, we train our data collectors, and off they go. However, without regularly checking in with data collectors, doing refresher trainings, and conducting data quality assurance, it’s all too easy for people to drift in how closely they follow data collection procedures. Moreover, if a project has multiple data collectors and each drifts with respect to protocols in different ways, there is no longer a consistent protocol being followed. This can lead to systematic differences in how data are collected by different people.
How to minimize data collection drift? Open and frequent communication, monitoring, and periodic training for data collectors. All are important parts of the evaluation process to maximize data quality. It’s important to ensure that everyone involved has a consistent understanding of how to implement data collection, how to address unexpected situations as they come up, and to make adjustments to make sure our measurement tools are working as intended – just like the tennis player who wants to make sure she hits the ball where she wants it to go instead of into the net.
PS – I recently read a blog asking how statistics are like knitting (and of course I can’t find the link to share here, sorry). The upshot of that post was that you get better with practice. The same can be said about program evaluation – and tennis.
Evaluation Internship or Part-Time Position
Coding of Family Strengthening Themes in Qualitative Case Studies
The Measurement Group LLC has an immediate opening for a part-time temporary Evaluation Assistant/Intern position. The Evaluation Assistant/Intern will primarily assist with coding qualitative case studies from an evaluation of the Westside Partnerships for Families program, a child abuse prevention program that uses a mental health home visiting framework. The Evaluation Assistant/Intern will:
The Evaluation Assistant/Intern may also be assigned additional evaluation support tasks on other projects as needed. This is a short-term position. $15/hour. Work must be performed at the offices of The Measurement Group in Culver City, CA during business hours (9am – 5pm, M-F).
Requirements: Undergraduate or graduate psychology or related social science major, plus experience/knowledge of child welfare, child development, and/or related social services. B.A. preferred, although an undergraduate with relevant experience will be considered.
This project is part of an evaluation of the Westside Partnerships for Family program, which was recently featured in an article in Children and Youth Services Review. The qualitative coding is for a new article that examines in more depth how this mental health home visiting model promotes protective factors for child abuse prevention and family strengthening.
Contact Dr. Lisa Melchior
My co-authors Katherine Reuter, Ph.D. and Amber Brink and I recently published an article about a mental health home visiting service model for child abuse prevention. The journal – Children and Youth Services Review – allowed us to share our findings not only by publishing the full article, but also in a brief 5-minute audioslide presentation. Click here to view the presentation and learn more about the model and results that support its effectiveness in improving family functioning among families with young children who are at risk for child maltreatment.
It’s always exciting when we have the opportunity to publish findings from our program evaluations. We are pleased to announce that we have a new publication coming soon! An article resulting from our work with the Westside Partnerships for Families program has been accepted for publication in Children and Youth Services Review. The paper, An Intensive Mental Health Home Visiting Model for Two At-Risk Early Childhood Populations, is authored by Katherine Reuter, PhD, former Program Director at the Providence Saint John’s Child and Family Development Center (CFDC) in Santa Monica, California, and Lisa Melchior, PhD and Amber Brink, BA of The Measurement Group.
Partnerships for Families is a child abuse prevention program that was initially funded by First 5 LA in 2006. Part of a county-wide initiative, Providence Saint John’s CFDC implemented this program in the Westside Service Planning Area of Los Angeles County. In the PFF Mental Health Model, mental health professionals work with families in a home visiting setting. The purpose of the program is to reduce risk and build protective factors in families with young children at-risk for child maltreatment. The paper describes the PFF Mental Health Model and documents improvements in family functioning among participants. It also presents data showing how caregivers reduced their risk on a number of personal characteristics — particularly those characteristics amenable to change through mental health intervention. Results were examined in two groups referred to the program: families with young children (age 0-5 years) who were referred by the Department of Child and Family Services, and pregnant women who were referred by community service providers due to risk factors for child maltreatment (such as depression, substance use, and/or domestic violence). Positive outcomes were observed at both the individual caregiver and family level in both of these groups.
We are pleased to help disseminate the outcomes of this innovative program and contribute to the evidence base for its effectiveness. While the focus of this article is primarily quantitative, we are currently working on another that will present a qualitative analysis of selected case studies to illustrate ways in which program participants demonstrate aspects of family strengthening.
5757 Uplander Way, Suite 200
Culver City, California 90230
310.216.1800
