Program Evaluation and Data Culture in Resident Services
By Julia Doty, Director of Resident Services, Northwest Housing Alternatives
Chari Smith, President, Evaluation into Action
January 2019
Introduction
Northwest Housing Alternatives’ Resident Services team wanted to engage in a formal program evaluation to improve our data tracking, help tell our story, and increase our grant funding. We had difficulty communicating the impact of our work, and found that people sometimes didn’t understand what we were doing and why we were doing it. The data we were collecting weren’t effective.
In 2016, we contracted with Evaluation into Action (EIA) to create an evaluation system for our program. This 6-month process resulted in an increased ability to effectively communicate our impact, to accurately describe our services, and to explain why we offer these services. In fact, the data helped us secure two of our largest grants to date. The success of this process is demonstrated through our ongoing ability to gather data that are relevant to our work and to use them to continually improve our program. We have data collection tools to gather data on a regular basis. The data inform our work as well as clearly demonstrate our impact to key stakeholders.
Concurrent to this process, we had also begun implementing a new web-based database to track and monitor our activities. The evaluation process helped to clarify what data to track and to create a user-friendly database for our staff.
This case study outlines key steps in our process in three areas:
- Planning
- Implementation
- Results
Planning
“The ways in which your nonprofit organization feels and thinks about evaluation will dictate their attitudes and behaviors toward it. Is your organization doing evaluation because of a required mandate from funders or because of a desire to learn from the insights data provides?” [1]
Evaluation into Action uses a participatory approach to ensure all stakeholders are included in the process and, therefore, have ownership. Some staff members were initially hesitant about this process, with the concerns that it would be too time consuming and the data would not be useful. Others had prior negative experiences that shaped these attitudes. These concerns were valid and understandable. The honest comments that staff members shared in an initial evaluation opinion survey helped with understanding their attitude toward program evaluation which, in turn, helped inform the path to shift these beliefs.
As described in Building a Culture of Evaluation, there are three steps to shift to a culture of evaluation: validate, educate, and collaborate. Evaluation into Action validated these current attitudes toward evaluation in a staff meeting. The opportunity to educate staff on the Evaluation into Action approach, which is focused on what is realistic as well as meaningful to evaluate, helped to shift the mindset from one of “we have to do this,” to “we want to understand how data can help us effectively reach our goal.” Finally, a commitment to collaborate on defining key evaluation elements, including the program goal, activities, and outcomes, shifted staff toward a culture of evaluation.
The initial discussions with staff, a review of written materials, and staff opinion survey results informed the agenda for an evaluation planning session. This interactive session focused on aligning the following three questions:
Goals: What is our dream for the program?
Activities: What do we do to achieve this goal(s)?
Outcomes: What measurable changes are expected as a result of these activities?
[1] “Building a Culture of Evaluation” white paper (2017) by Chari Smith and Elaine Charpentier Philippi.
The alignment of these three responses provided the cornerstone of the evaluation plan. The result was an impact model that visually summarizes the goal, activities, and outcomes for this program.
Resident Services Program Impact Model
Program Goal: To connect residents to services that promote housing stability.
6 Resident Service Coordinators (RSCs)
Liaison between residents and fulfilling resident needs
Assumption: Fulfilling resident needs will increase the likelihood that they will remain housed.
Available to 2,400 residents across 32 properties
Services are delivered by RSCs directly or coordinated with a service partner to fulfill resident needs. Residents are seniors, families, and/or individuals with mental and/or physical disabilities.
Primary Direct Services – 6 RSCs:
- Assist with making phone calls
- Notice intervention/lease compliance
- Eviction prevention
- Conflict mediation with other tenants and/or property managers
- IDA program
- Coordinate services
- Social events
Primary Coordinated Services – 70 resident service partners
Onsite programs: adult education, health and wellness, financial education, mentoring, & youth engagement
Assistance: legal, employment, rent, utilities, childcare resources, tax, emergency food boxes, and health/wellness
Expected outcomes to occur as a result of services delivered
- Create a community within the property
- Improve housing stability
- Improve conflict resolution skills
- Decrease the eviction rate
- Increase understanding of notice prevention and lease compliance
- Increase knowledge of resources and opportunities for growth
Measurement Methods
- Track services delivered and eviction rates
- Housing stability ratings on a regular basis
- Surveys: property managers and residents
Implementation
The measurement methods standardized how all resident service coordinators collect data. Evaluation into Action created the data collection tools, as well as defined the process of when, by whom, and how data should be collected. This gave us a clear path to implementing the program evaluation plan.
1 – Impact Measurement Tool
2 – Data Tracking
3 – Training Staff
Our primary program goal is to promote housing stability; therefore, we needed to clearly communicate how we were accomplishing that objective. Evaluation into Action conducted a literature review on how housing stability is defined and measured nationally within Resident Services. Results showed that neither a standard definition nor a clear measurement method to truly measure housing stability exists. Evaluation into Action created a housing stability assessment customized to our program. This tool has become an integral part of our program operations, guiding us to understand what activities will best meet the needs of each resident.
The new database was designed to track the intake process, service forms, event attendance, move in/move out, and other relevant monitoring needs on an ongoing basis. The Housing Stability Assessment was also designed to be administered through the database, allowing us to cross-reference the rating with other factors, such as service requests, move-out data, and demographic factors. Resident Services Coordinators (RSCs) enter data on a daily basis, allowing for us to see an aggregate picture at any time on any variable or to hone in on a specific property or resident to guide program activities.
Our choice of database took into consideration these factors: ease of use, cost, and ability to easily pull information and cross-reference data points. The most important consideration was malleability; we needed a database that could be designed exactly as we envisioned it and continue to be adapted as our program shifted and changed. We contracted with Exponent Partners to utilize their Exponent Case Management (ECM) application.
The flexibility of this program has allowed us to glean a tremendous amount of insight into our impact. For example, we were able to determine that the average stability level of NHA households requesting services is 2.55 on a scale of 1 (low risk of losing housing) to 5 (high risk of losing housing). However, we can isolate housing stability data at specific properties to identify whether the community is more or less stable, and adjust intervention strategies and program planning accordingly.
Evaluation into Action administered property manager and service partner surveys online. Resident surveys were administered by the Resident Services Coordinators at each property. Evaluation into Action provided training and survey administration materials to ensure the resident survey was administered consistently across all properties.
Results
In addition to focusing a program’s data collection and outcomes, the evaluation process can also inform a program’s culture by using a lens of assessment that applies to a range of daily practices. Northwest Housing Alternatives’ Resident Services program experienced a culture shift as a result of program evaluation that was evident both internally and externally.
Measuring Housing Stability
From no assessment of impact to Housing Stability Assessment
Assess stability based on requests for services, program participation, lease violation notices, etc. Resident Services Coordinator directly performs assessment based on these factors after first service request, then every three service requests after that or every six months. This allows us to establish baseline, change over time, and end point for those residents who only need one intervention. Assessing housing stability over time allows us to offer the appropriate intervention to each household.
Example: Percentage of residents who maintain and improve in housing stability over time
Program Improvement
From reactive interventions to proactive interventions
Based on a household’s Housing Stability Assessment, we offer specific services and resources that help the household stabilize, avoid eviction, and/or access opportunities for growth. Whereas our prior intervention strategy was to have a resident household approach us with a need, we now are able to distill available resources to focus on just those that are most applicable and helpful to that specific household.
Example: Low risk of housing instability = IDA, asset building, etc. High risk of housing instability = service coordination, lease compliance education, basic needs, etc.
Demonstrating Impact
From outputs to outcomes
Program evaluation expanded our data tracking practices tremendously, while allowing us to cut out extraneous data points that were not contributing to our reporting. Data points that do not convey the change in a household as a result of our intervention are no longer tracked. Outcomes come from self-reporting (surveys), Housing Stability Assessments, and tracking of daily RSC activities.
Example: Number of service requests fulfilled per year vs. percentage of residents who had a service request fulfilled and improved their housing stability rating as a result
Communications
From annual reporting as a response to individual queries to proactively reporting throughout the year
By refining our data tracking and focusing on targeted outcomes, we are able to measure household progress over time and examine outcomes both individually and program-wide. This allows for ongoing reporting that helps key stakeholders to better understand the program.
Example: Grant reporting vs. Annual Survey Report produced for general public consumption
What’s Next?
NHA’s Resident Services team revisits our outcomes annually to ensure they are still relevant and adjusts our data collection questions and methods accordingly.
Based on outcomes data, we can make changes to program implementation that help our residents achieve the highest level of success in our housing.
About the Authors
About NHA
Northwest Housing Alternatives is a private nonprofit organization which builds and develops affordable housing for seniors, families, and people with disabilities. NHA’s mission is to create opportunity through housing, and their portfolio includes more than 1,800 units in 16 Oregon counties.
Visit NWhousing.org for more information. Stay up to date with the latest affordable housing news at www.facebook.com/NWhousing.
About Julia Doty
Julia Doty is the Director of Resident Services for Northwest Housing Alternatives. Prior to that, she spent five years providing Resident Services to NHA’s senior properties. She is one of the developers of the first trainings in Oregon for new Resident Services Coordinators. She has served as facilitator for Oregon’s Resident Services Peer Learning Group and the Resident Services Statewide Advocacy Group.
About Evaluation Into Action
Evaluation into Action, founded by Chari Smith, empowers nonprofits to transform program evaluation into an action-focused, highly cost-effective practice that can increase your organization’s capacity to accomplish its mission. Located in Portland, Oregon, we provide hands-on program evaluation services and also build internal skills with training, coaching, and technical assistance. Our pragmatic approach demystifies the program evaluation process, delivering a clear, actionable plan to achieve your goals.
Visit www.evaluationintoaction.com for more information, free resources, and our blog.
Stay Connected with Chari Smith
Twitter: @evalenthusiast
Linked In: Chari Smith
Sign up @ www.evaluationintoaction.com for program evaluation news and resources.
Interested in program evaluation services? Contact hello@evaluationintoaction.com. We offer a range of services to ensure your program evaluation needs are met.
Look for Chari’s new book on Nonprofit Program Evaluation (working title) to be released by Bold and Bright Media in 2020. To stay up to date on pre-orders, events, resources, insights, and news, sign up for the Evaluation into Action newsletter at www.evaluationintoaction.com
For more information, please contact:
Chari Smith – hello@evaluationintoaction.com – 503-246-1412
Julia Doty – doty@nwhousing.org – (503) 654-1007 x105
