CORONAVIRUS (COVID-19) RESOURCE CENTER Read More

Program Summary Tool: An Evaluation Framework for Public Health Program Success

State: GA Type: Model Practice Year: 2014

1) Located in the outskirts of Atlanta, Cobb and Douglas Public Health (CDPH), with our partners, promotes and protects the health and safety of over 830,000 residents of Cobb and Douglas counties. Cobb County has the 4th largest county population in Georgia, recorded at 688,078 in the 2010 Census. The Douglas County population, which is the 16th largest among counties in Georgia, increased by 43.6% between 2000 and 2010. According to the 2010 Census, 132,403 citizens reside in Douglas County, Georgia. 2) While the population in Cobb and Douglas counties continues to grow and diversify, public health funding continues to decline. Like many health departments across the nation, CDPH has been forced to do more with less. Given scarce resources, demonstrating effective use of funds is essential and increasingly required stakeholders and funders. As a result, CDPH has focus efforts on capturing not only agency, but program performance through a comprehensive program evaluation and performance measurement tool: The Program Summary Tool. This Tool, developed in partnership with the Centers for Disease Control and Prevention, provides a concise snapshot of program performance over the past fiscal year. In addition to performance measurement, the tool includes a logic model, stories in action/stories from the field component, performance scorecard, and recently, a quality improvement requirement. 3) The goal of this project was to provide a simple, fill-in-the-blank template that would capture program performance and quality improvement plans systematically across 30 diverse public health programs. This tool also satisfies objectives related to measure program performance for Public Health Accreditation (see Standard 9.2), educating staff, Board members, and the public about public health programs and services, and finally, capturing performance data readily available for use in grant applications and programmatic reports. 4) During the summer of 2009, staff systematically reviewed of all key components of program performance management. This included not only examining the program's historical performance, but it also included alignment of the program to the Ten Essential Services of Public Health, sorting of partnerships by partner type (Strategic, Sustaining, or Affiliate), and rating scales to evaluate the magnitude of the public health problem addressed by the program. Over a period of 2-3 months, staff selected criteria for the Program Summary Tool questions. In response to staff feedback from the first pilot round, the two evaluations were combined into one annual evaluation tool. Instructional Tool trainings are conducted approximately 4 months before the Tool submission deadline in November. 5-8) The Program Summary (Evaluation) Tool has grown in popularity- the State of Georgia, Department of Public Health is interested in sharing with Georgia's 18 health districts. Administered annually since April 2010, the Tool provides a snapshot of program performance in relation to achieving desired goals, which has been beneficial to raising new funds through grants and the Cobb and Douglas Foundations. 5-8 continued)This is accomplished through four main sections: Program Description Program Logic Model Program Performance A Quality Improvement Plan The Program Summary is a tool to help CDPH evaluate, improve, and garner support for their programs. Completion of the Program Summary Tool has increased program awareness and facilitated discussion among CDPH Program Managers, Clinical Staff, Center Directors, and Leadership Team Members. Externally, the tool serves as a resource for communicating our needs to potential funders for grant proposals, providing information to our Boards of Health, and services provided to the broader community. Comparable evaluation across programs has enabled the CDPH Leadership Team and District Health Director to make informed decisions to maximize use of limited resources. To maximize accessibility, completed Tools are also posted to the Intranet and shared drive to enable staff to learn more about all of CDPH's programs. Once completed, the goal of maximizing use of the Tool throughout the year is accomplished by scheduling individual program meetings with the District Health Director and program staff to review results. These meetings result in after actions or follow-up items for staff based on the District Health Director's recommendations and further quality improvement initiatives.
1) Cobb and Douglas Public Health (CDPH), with our partners, promotes and protects the health and safety of over 830,000 residents of Cobb and Douglas counties. Cobb County has the 4th largest county population in Georgia, recorded at 688,078 in the 2010 Census. With a relatively diverse population, 56.3 percent of Cobb County residents identify as white and non-white minorities represent 43.7 percent of the total population. The Douglas County population, which is the 16th largest among counties in Georgia, increased by 43.6% between 2000 and 2010. According to the 2010 Census, 132,403 reside in Douglas County, Georgia. African-Americans made up approximately 40.1% of the Douglas County’s population in 2011 and Hispanics 8.6% of the county population. 2) Statement of the problem/issue: Public health agencies are now forced to do more with fewer resources. For resources obtained, health agencies must demonstrate effective use of funds through increased performance measurement and evaluation. 3) The target population for this practice includes public health programs (the Tool is adaptable to fit a broad range of program types and could be used for performance evaluation assessment at a state or national level). Each of CDPH's 30 programs have used the Tool annually since 2010 (100% of target reached). Interest in the Tool has exceeded expectations and is being considered for use among state public health programs. 4-5) In the past, program managers struggled to meet grantee reporting requirements and quantify program performance. Some program managers claimed that our district was the "best in the state", but had little to no evidence to substantiate the claim. Very few program performance requirements were in place and fewer were reported on a regular basis in order to provide trend/tracking analysis. Often when performance data were needed, program managers placed a request to our already overburdened IT department. Now, program managers complete the Tool once a year and are readily able to demonstrate how effective the program has been in the past year. The Tool has further demonstrated success in the ability to help program managers and staff articulate the purpose and diversity of CDPH's public health programs. Annual data collection allows for program performance trending, which helps to assess performance improvement opportunities and QI projects. Sharing completed Tools has also fostered communication between staff (de-siloing) and improved morale (as staff work together through increased internal referrals and recognition). CDPH staff can read about the mission, number of people service, types of services provided, etc., from the Tools posted to the CDPH intranet and shared drives. 6-7) The Program Summary Tool was developed from the CDC's 6 Step Evaluation Framework and through consultation with the CDC's performance evaluation consultant, Dennis Barhart. Even so, application of the CDC's 6 Step Framework into a template applicable and meaningful to a wide range of diverse public health programs, is new to the field.  
Motor Vehicle Injuries
1) Goal(s) and objectives of practice: The Program Summary Tool serves primarily as a quality improvement and outcome evaluation tool for all CDPH public health programs. While this Tool may in part examine steps taken to improve operations, its purpose is to provide a snapshot of program performance in relation to achieving desired goals. This is accomplished through four main sections: (1) Program Description, (2) Program Logic Model, (3) Program Performance (consists of the program's full scorecard and complete performance metrics). The Program Description portion includes five main sections which include: A) Need/issue addressed; B) Target population served; C) Stage of the Program; D) Resources Available to Conduct the Program; E) An “Other” category is provided to assess the Program’s top 3 challenges/barriers, strengths, opportunities for improvement, and accomplishments. The Program Logic Model is a visual depiction of your program’s current activities and intended goals (outcomes). Finally, the Program Performance section is intended as a starting point to gear program staff for the development of Balanced Scorecard Program metrics. Objective 1: Establish an annual evaluation framework to quantify performance across CDPH's 30 programs and services, and allow for comparability (to benchmark performance across programs) and trending (through data collection over time). For example, the agency's strategic objective of, "Providing high quality services to our customers" was included as a program performance measure within the Tool. Agency targets were established for customer surveys. On a scale of 1-5, agency customer services was rated at 4.7 in May 2011 following a 27% response rate to survey requests. Objective 2: Identify and prioritize quality improvement efforts agency wide through re-occurring program performance issues. For example, space constraints at the agency were identified with improvement needs noted by several programs and a significant need within clinical areas in order to preserve patient privacy in accordance with HIPAA. These concerns were validated through several audits and the agency elected to bring in an architect in 2010. Part of the facilities redesign process, has required expanded use of patient and waiting room, increased signage, etc. Data compiled from the Program Summary Tools, including audit findings noting the need for increased clinic space, low scores on customer satisfaction surveys, and repeated reference among program managers reflecting "space" as a top opportunity for improvement, led to an agency commitment to find funding to support remodeling. Despite a challenging economic climate, agency staff undertook efforts to pursue inclusion in the county's special purpose local option sales tax (SPLOST). The agency was eventually approved to receive funds from the SPLOTS in an amount exceeding $6 million. In addition, the District Health Director met with the City Planner to discuss public health facility needs, which led to acquisition of the former Cobb County Juvenile Court Building. Objective 3: Promote health and prevent injury and disease to achieve healthy communities. Through cascading the Balanced Scorecards to over 30 agency programs, activity and outcome targets have been established and monitored to ensure delivery of grant and funding requirements and align all programs and staff to the overall objectives of the agency. Objective 4: Promote, develop and evaluate community partnerships. Over 400 partners participated in the Cobb2020 Initiative Inaugural Event on October 11, 2012 and pledged their support (with actual ways in which they would assist) of building a healthier Cobb County. Of these, over 40 partners meet monthly in each county as part of the MAPP Steering Committees. Many of these partners will be mobilized as part of the MAPP Action Phase to implement and address health issues according to our Community Health Improvement Plan (CHIP). 2)What did you do to achieve the goals and objectives; Steps taken to implement the program: During the summer of 2009, staff systematically reviewed of all key components of program performance management. This included not only examining the program's historical performance, but it also included alignment of the program to the Ten Essential Services of Public Health, sorting of partnerships by partner type (Strategic, Sustaining, or Affiliate), and rating scales to evaluate the magnitude of the public health problem addressed by the program. Over a period of 2-3 months, staff selected criteria for the Program Summary Tool questions. Additional components requested, including the program's logic model and scorecards, were added at a later date to provide an even more comprehensive assessment of the program's fiscal year performance. Initially, quantitative questions from the Tool were compiled into a separate Macro Level Evaluation while qualitative questions were assembled into a Micro Level Evaluation. In response to staff feedback from the first pilot round, the two evaluations were combined into one annual evaluation tool. Instructional Tool trainings are conducted from September up until the Tool submission deadline in November. Once completed, the goal of maximizing use of the Tool throughout the year is accomplished by scheduling individual program meetings with the District Health Director and program staff to review results. These meetings result in after actions or follow-up items for staff based on the District Health Director's recommendations and may lead to quality improvement initiatives. To maximize accessibility, completed Tools are also posted to the Intranet and shared drive to enable staff to learn more about all of CDPH's programs. 3)CDPH requires that all programs (30 total) use the Tool for performance monitoring. Completed Tools are submitted to the agency's performance and quality improvement officer. Program managers review program performance results (top accomplishments, resources needs, areas for improvement) for an informed discussion annually with the District Health Director.4)What was the timeframe for the practice: Tool development and training generally occur 4 months prior to the submission deadline (November). Tools are submitted from all CDPH programs up until the last business day in November. Tools are reviewed for accuracy and completeness through the end of December, by the CDPH Quality Council. The Tools are finalized for review by the District Health Director by January. From February through August, the Programs update scorecard performance and enact quality improvement initiatives relevant to achieving their prior Program Summary Tools. 5) Stakeholder involvement: The primary stakeholders in this practice are the CDPH Leadership Team and Quality Council. These two committees are responsible for assuring continued program performance and improvement. The OPSM Director oversees and monitors the process. Other stakeholders include the Director of Communications and Grant Manager who specifically benefit from the completed Tools as they are able to pull information for their use throughout the year. This practice involves examining the current operations of the health department and its ability to articulate performance for stakeholders including the public, Boards of Health, funders, partners, and internal staff. The local health department also involved Tom Chapel, evaluator and strategist of the CDC to conduct program evaluation and logic model workshops. Local educational institutional staff, including those from Rollins School of Public Health examined and provided recommendations on the question set. The primary stakeholders responsible for the Tool, the CDPH Leadership Team and Quality Council, provide input to guide all phases of the project. This is an internal practice which enables the health department to accomplish its mission of providing high quality services to the residents of Cobb and Douglas counties, through an internal ongoing evaluation of all programs and operations. Therefore, the primary stakeholders of the Tool involve the CDPH Leadership Team and Quality Council. These committees work very closely throughout all phases of the Tool's completion to ensure program staff thoroughly understand the purpose, provide valuable input and engage in discussion on areas for improvement, and enact quality improvement projects when necessary. In order to foster this collaboration, CDPH Leadership Team members and the Quality Council solicits input from program managers and staff throughout the year and particularly during the development phase of the Tool. Often times, programs find that they are able to better communicate their needs to CDPH Leadership Team members through the guided set of questions and are willing to share more once their voices have been heard. 6) Start up and in-kind costs: Direct costs include supplies for printing of the Program Summary Tools and training materials (less than $200). In kind costs are incurred through Tool development and review by the Quality Council, staff time for training and completion of the Tool. While in kind costs associated with completion of the Tool are no doubt substantial, the time saved in searching for necessary information for grant proposals, reports, or for marketing/educational proposals are far greater. Overall, time and effort dedicated to the Tools has improved CDPH’s infrastructure.
1) What did you find out? Each of our intended objectives were achieved as follows: Objective 1: Establish an annual evaluation framework to quantify performance across CDPH's 30 programs and services, and allow for comparability (to benchmark performance across programs) and trending (through data collection over time). This objective was completed through finalizing the Tool in May 2009 and ongoing Tool trainings. Objective 2: Identify and prioritize quality improvement efforts agency wide through re-occurring program performance issues. For example, space constraints at the agency were identified with improvement needs noted by several programs and a significant need within clinical areas in order to preserve patient privacy in accordance with HIPAA. These concerns were validated through several audits and the agency elected to bring in an architect in 2010. Part of the facilities redesign process, has required expanded use of patient and waiting room, increased signage, etc. Data compiled from the Program Summary Tools, including audit findings noting the need for increased clinic space, low scores on customer satisfaction surveys, and repeated reference among program managers reflecting "space" as a top opportunity for improvement, led to an agency commitment to find funding to support remodeling. Despite a challenging economic climate, agency staff undertook efforts to pursue inclusion in the county's special purpose local option sales tax (SPLOST). The agency was eventually approved to receive funds from the SPLOTS in an amount exceeding $6 million. In addition, the District Health Director met with the City Planner to discuss public health facility needs, which led to acquisition of the former Cobb County Juvenile Court Building. Provide high quality services to our customers. Agency targets were established for customer surveys. On a scale of 1-5, agency customer services was rated at 4.7 in May 2011 following a 27% response rate to survey requests. Space constraints at the agency were identified with improvement needs noted by an architect in 2010. Despite a challenging economic climate, agency staff undertook efforts to pursue inclusion in the county's special purpose local option sales tax (SPLOST). The agency was eventually approved to receive funds from the SPLOTS in an amount exceeding $6 million. Objective 3: Promote health and prevent injury and disease to achieve healthy communities. Through cascading the Program Summary Tool to over 30 agency programs, activity and outcome targets have been established and monitored to ensure delivery of grant and funding requirements and align all programs and staff to the overall objectives of the agency. In addition to achieving intended objectives, the Program Summary Tools have been beneficial in a number of additional ways: 1. The Tools have fostered awareness (internally, externally, and among board members) about public health programs and services. 2. The Tools have indirectly improved morale among staff and helped programs de-silo through increased communication among program staff and across departments. 3. As a result of the annual Tool completion, program managers are better evaluators and able to articulate the top accomplishments (as well as needs) of their programs over the past year. 4. The Program Summary Tools have helped to align agency quality improvement priorities and initiatives when examining performance (espcially areas for improvement) sections across programs. 2) Practice Evaluation: The Program Summary Tool has demonstrated effectiveness through the ability to pull information for use in grant proposals and program prioritization/advocacy in light of impending budget cuts. Most notably, the Grants Manager was able to pull information from the Program Summary Tools for the CDC, Community Transformation Grant proposals, which resulted in $499,000 in funding over the next five years. Information from the Program Summary Tools has also been helpful to advocate for programs during cutbacks. When our Adolescent Health and Youth Development Programs were being cut, we were able to educate our Boards of Health with readily available information on the number of clients impacted, and really for a few thousand dollars to continue providing counseling services to sustain the program by another five months. While these provide examples of the Program Summary Tool's success overall, the Tool also captures several important performance measures (noted as Objectives 1-3, above) which collected by each program and fold into the agency's overall performance measurement system through the Balanced Scorecard. 2a) and 2b), Primary and secondary data collection: The Program Summary Tools consist of primary program data provided by program managers and department heads to summarize program performance through a series of standardized questions and fill-in-the-blank style forms, over the past fiscal year. Data are collected and reported to the agency's performance improvement office by program managers. Although performance data most performance data is derived from primary sources (e.g., customer satisfaction survey, budget, and productivity data), a few questions require secondary data collection. These questions relate to researching evidence based and best practices applicable to the program, citing local laws/legislation concerning public health authorities, and data collection through Georgia's OASIS system to capture relevant health statistics).
1) Lessons learned in relation to practice? Initially, as a result of a new District Health Director and emphasis on a business model for performance management, the process to implement the Program Summary Tool was met by staff with skepticism and reluctance. In time, because the Tools have been used in ways helpful to programs (i.e., fundraising, marketing, communication to leadership including the Boards of Health), most staff see the value and spend a great deal of time ensuring their Tools are highly detailed and provide an accurate description of their program's performance.2) Lessons learned in relation to partner collaboration (if applicable): Given the diverse nature of public health programs and services, many partnering organizations and Board of Health members have commented that the Tools are helpful to foster awareness and communicate the value of public health. The tools also provide valuable performance data for stakeholders to be able to substantiate the hard work of public health professionals and need for increased public health resources and funding. From a fundraising perspective, the tools have been helpful to demonstrate to current and potential funders, a solid infrastructure for ongoing public health program performance management. The Program Summary Tools have also been instrumental to the Cobb Health Futures Foundation and Douglas Health Futures Foundation, which act as a separate 501C3 under the health departments Development Office to support public health programs. The Program Summary Tools are helpful to these Foundations by providing an 1) overview of each program, 2) program performance (and historical repository of prior performance through prior completed tools), 3) rubric and/or prioritization matrix for public health program fundraising. The health department has learned that continued funding and support, whether through the Foundations, partners, or external fundraisers is contingent upon results. The Program Summary Tool provides a concise and consistent (across programs) platform in an easy-to-read and complete (fill-in-the-blank format) template to demonstrate public health program performance and results. 3) Is this practice better than what has been done before: At the current time, CDPH is unaware of a similar evaluation tool for local public health programs, although in general, the principles outlined in the CDC's Six Step Evaluation Framework have been incorporated into the Tool. 4) Did you do a cost/benefit analysis? If so, describe: Initially, development of the Tool required substantial time from the Director of Organizational Performance and Strategy Management, as well as an internal Program Evaluation Manager. Dennis Barnhart, a strategy and performance management consultant for the CDC, as well as a number of for-profit and not-for-profit organizations in Canada, the US and Europe, collaborated to develop the Tool and provide several staff trainings. During the first year, staff estimated that they dedicated approximately 1-2 hours each to program evaluation and quality improvement (Tool related) trainings. They estimated approximately four hours were required to complete the Tool. Staff also presented their completed Tools annually with the District Health Director who provides guidance on future actions and recommendations to program managers for the coming year. The amount of time required in subsequent years has diminished substantially as staff have become accustomed to updating and revisions components on an annual basis. 5) Sustainability – is there sufficient stakeholder commitment to sustain the practice? Cobb and Douglas Public Health is committed to excellence through continual performance evaluation and quality improvement. The Program Summary Tool has demonstrated success in numerous ways as a source of readily available information for marketing, grant proposals, education regarding the type and scope of public health programs, alignment of our public health programs to the Ten Essential Services of Public Health and PHAB documentation requirements, etc. As a result of four years of demonstrated success and a historical repository of completed tools shared with all staff on our Intranet, the Program Summary Tool has become incorporated in agency operations. CDPH recommends use of the Tool for performance monitoring among other local health departments. The possibilities for collecting data through this adaptable Tool nationwide is endless and could be used to demonstrate the need for public health programs on a national scale.5a) Describe sustainability plans: Since the Tool has demonstrated efficiency and time-savings among staff, plans remain to continue its administration and use on an annual basis. Consensus is staff would prefer to complete the Tool once and house data in a shared location, if it will be helpful and readily available for use in grant proposals, legislative factsheets, communication to Board members, and marketing.
I am a previous Model Practices applicant|Conference|Model Practices brochure|E-Mail from NACCHO|NACCHO website|NACCHO exhibit booth|NACCHO Exchange|Public Health Dispatch|NACCHO Connect