Abstract
A systematic and scientifically rigorous program evaluation is essential in building best practices and integrating evidence-based practices in behavioral health care. However, program evaluation has many challenges that often shun away many health workers from practicing it. Funderburk and Shepardson (2017) therefore carried out a research to identify the common challenges facing the conduction of program evaluation and to identify ways of improving the quality of program evaluation. The study involved program evaluation of training program for intervention from brief alcohol while the second study involved program, evaluation monitoring service of depression medication. From this study, the researchers found out that there was a need for strategic planning using the conceptual framework and relevant study methods, measurement, and consistency to get a high-quality program evaluation. Therefore, the researchers concluded that by applying the recommendation found in the study, the quality of program evaluation would significantly improve, leading to better generalizable data that can be used to improve healthcare delivery.
Summary
According to the researchers, program evaluation is essential in delivering high-quality healthcare. Additionally, according to them, program evaluation enables healthcare administrators to determine whether the services they are providing meet the set objectives. Program evaluation also enables each program to analyze its outcomes, such as the effectiveness of its implementations, the effect of the changes it makes to its staff, assess their patient’s satisfaction, and whether there are changes in the behavior of patients and improvement of health care outcomes. Moreover, program evaluation is essential in health care as it enables the collection of local programs’ information for decision-making to further improve scientific knowledge. However, the researchers argue that many physicians and healthcare managers do not engage in program evaluation despite its importance due to the lack of training, failure to recognize its value, and lack of time and recourse to carry out more studies on program evaluation. The researchers also add that the pitfall associated with conducting program evaluation in the normal hospital setting makes the health workers shun away from a program evaluation.
Therefore, the research was carried out the study to encourage health care providers working within the context of integrated behavioral health care settings to practice generalizable program evaluation using well scientific studies. The researchers determined ways program evaluation can be used in integrated behavioral health care to achieve this objective. They also researched the significance of program evaluation and the challenges that physicians often encounter during pragma evaluation in the common hospital setting. This research involved two case studies. One was program evaluation done on a training program for brief alcohol intervention. The second study involved program evaluation on a monitoring service for the depressive medication (Funderburk, J. S., & Shepardson, 2017). From their research, the researcher identified the need for strategic planning to achieve high-quality program evaluation by ensuring \there is appropriate study design, measurements, and consistency.
Evaluation
The major objective of the study was to drive health care practitioners and management within the field on integrated behavioral health care to practice a generalizable program evaluation process having in-depth scientific rigor. To achieve this objective, the researchers first reviewed how program evaluation can be applied in integrated behavioral health care interventions and why it is essential to carry out high intensive program evaluation (Funderburk, & Shepardson, 2017). The researchers discussed the common challenges that face the process of conducting program evaluation in integrated behavioral health care. These included strategic planning challenges, choosing the ideal study design, consideration of measurement, and consistency. This research involved two case studies. One of them was a description of program evaluation which was carried out for a brief alcohol intervention training program. The objective of this case study was to examine the feasibility and the reaction of the behavioral health program to receiving training virtually. The other objective was to measure integrated behavioral health programs reported individual level of understanding of the components of behavioral alcohol interventions and the impacts made on the clinical practices after going through the training. The other case study described program evaluation carried out on monitoring service for depression medications. The objective of this study was to evaluate the impact of depression monitoring services through the use of the RE-AIM framework to help determine whether funds should be continued. The study procedure involved care managers repeating the assessment after the monitoring period to identify pre and post-intervention data.
Theoretical Perspective
To achieve the study’s objective, the theory of logical model was applied. The authors state that strategic planning is essential in program evaluation and needs individuals to state the program’s objective beforehand, identify the conceptual framework, and identify the targeted audience and the key question that should be answered. The program in this context was an action meant to the care clinic to improve the health outcomes of the patients (Funderburk, J. S., & Shepardson, 2017). For example, in this study, the program evaluation focused on the new depression medication instead of the whole telephone-based Management service. The study used the theory of the logical model.
This theory connects the program to the goals by systematically visualizing the proposed relationships among its resources, activities results, and impacts. Having an understanding o, the conceptual framework helped improve the program evaluation by identifying the program. These factors may affect the program’s outcome and how it is expected to achieve its outcomes. The objective of the program evaluation was determined by the objective of the program and conceptual framework, the current status in the system, and the feasibility of the data collected from within the organization. The program’s objectives, in this case, varied depending on the place to program exists in its developmental history in that in an early developmental phase, the objective will mainly consist of improving the quality or evaluation for adherence of the staff. If it has existed for a long, the focus will be on short-term and long-term health outcomes. The function and audience determined these options. Therefore, the examination and involvement of the audience are essential in developing the program evaluation. In this research, one of the theories that were ruled out was using the RE-AIM framework to inform the understanding of the program and program evaluation. This was focused on the new programed depression medication monitoring to determine the depression symptoms of the patients, compliance, and the side effects. This was not successful as the hospital’s internal factors made it difficult to implement the program, thus preventing the program from taking place.
The logical model guided researchers in carrying out their evaluation as it guided in strategic planning. The first step was to identify the resources, which included the presence of in-person training on behavioral alcoholic intervention, the physician familiar with the training needs of behavioral alcoholic interventions, access to online resources, and cooperation of leaders (Funderburk & Shepardson, 2017). The next step was identifying the activities, which included developing and implementing the training using Webinar, developing surveys to measure outcomes, and recruiting behavioral health practitioners. The third step was identifying the outputs, which included the number of behavioral health providers that attended a webinar, those who attended the optional teleconference, the number of behavioral health interventions the physician carried out before and after the webinar training, and the quality of behavioral health intervention that each physician reports after the training. The fourth step was the identification of the outcomes such as the increased quality of behavioral health interventions, increase in confidence and comfort when conducting behavioral health interventions, and increase in the number of patients that received behavioral health interventions. Finally, this model in identifies the impact, such as reduction in dangerous use of alcohol among patients, increase in the health and satisfaction of life, and reduction in utilization of health services. This model was the best one to implement as it leads to achieving set objectives (Murphy et al.,2018). Additionally, it is easier to implement. The researchers, therefore, chose the best type of conceptual framework.
Research Methods
The type of research design employed in this evaluation was quasi-experimental pre and post designs. This study involves a non-group design that is not equivalent and contains groups signed to conditions based on the nature of those who received the training program and those who did not receive the training program and comparing the results obtained from the two-time points. In the first case study, which involves describing program evaluation carried out for a training program if brief alcohol intervention, the stud involve pre and post assessment wherein the pre-assessment, the assessment involved measuring the percentage of the patients seen in the previous week where the clinician had discussed the reduction on alcohol use elements of the behavioral alcohol intervention and how of the behavioral health practitioners talked to the patients about the use of alcohol. The post-assessment involve measuring the confidence in the delivery of elements of behavioral alcohol interventions, determining the number of times the physicians conducted behavioral alcohol interventions with a patient in the general office and the number of times when the physician conducted behavioral alcohol interventions when not particularly referred for alcohol problems (Funderburk & Shepardson, 2017). Moreover, open-ended questions to find qualitative feedback about the Webinar components that behavioral health practitioners liked or did not like.
The second study involved using the RE-AIM framework to find out whether the funds should be counted. The program evaluation involved examines the number of patients showing clinical improvements in the depressive symptoms (Funderburk & Shepardson, 2017). In this, the program led to a percentage of patients similar to 32 percent of the patients, which the other research had found to produce in 6 months without intervention. The impact of the confidence e will be lower, but if it were higher than the 32 percent benchmark, this would show an improvement above the normal time of improvement.
The use of prep and post designs in program evaluation is very essential as it is shown to enhance potential conclusions. The assessment of outcomes only once after the program is implemented reduces the chances of evaluating the within-person changes across time (Owen, 2020). The integrated environment is challenging when collecting data before the patients experience the program. For instance, it is challenging to get baseline data when carrying out program evaluation a specific program such as cognitive behavioral therapy for a condition such as insomnia being delivered within the model of PCBH of service delivery is prioritized and come immediately after the primary visit(Funderburk & Shepardson, 2017). In the same way, getting a follow-up data after receiving the interventions is often challenging since the modal number of appointments with behavioral health providers happens immediately live after the primary care visit and is only one. Therefore, this disrupts prevent capturing of data after initiating the program and brings ambiguity due to a variety of reasons why patients may not return. Therefore, thorough consideration must be carried out to overcome these barriers and get multiple assessments.
Conclusion
The authors found various challenges associated with carrying out program evaluation from the research. One of the challenges identified is the lack of strategic planning. To overcome this challenge, the researchers recommended that experts should examine and define which program is to be evaluated and its components and understand the conceptual framework for the program. Another challenge the authors identify as study consideration. They recommended the use of pre and post-study methods.
Additionally, they should include the comparative data, carefully monitor these comparative data, consider alternatives, use quasi-experimental designs, and in case the use of a comparison group is not possible. They can compare the observed changes and those that can be seen within the field (Funderburk & Shepardson, 2017). Additionally, the researchers found another challenge of program evaluation: measurement consideration. To curb this challenge, it is important to determine the type of information to collect and how it should be collected, consider the strengths and limitations of each method of data collection, and consider supplementing it with self-report data. Furthermore, maintaining consistency was another challenge identified. To deal with this challenge, the researchers recommended training to ensure the program is being delivered and intended and orient the clinical staff to the measures.
Additionally, their researcher should ensure there is detailed training to ensure that training is delivered consistently. These recommendations can be achieved by program evaluation being carried out as a team. Therefore program evaluation requires integrated behavioral health providers working as a team.
Evaluation Concepts
Outcome evaluation- outcome evaluation refers to measuring the result of the outcome to determine whether the intended outcomes have been achieved. It was applied in this research by comparing the patient’s condition before and after participation in the training.
Program evaluation- program evaluation refers to the systematic collection of data and outcomes of programs to determine the programs’ effectiveness and aid in future decision-making about the program. In this research, program evaluation was carried out on training programs for brief alcohol intervention.
Experimental design refers to how the participant in the study was allocated to different groups in s research. In this research, participants were grouped in matching pairs of their alcohol use.
A quasi-experimental design is a type of research method that involves establishing a cause-and-effect relationship between the dependent and independent variables. In this research, quasi-experimental research was the recommended design for carrying out a program evaluation.
Qualitative methodology- a qualitative research method involves collecting and analyzing data whereby data is obtained through open-ended communication.
References
Funderburk, J. S., & Shepardson, R. L. (2017). Real-world program evaluation of integrated behavioral health care: Improving scientific rigor. Families, Systems, & Health, 35(2), 114.
Murphy, T. M., Chang, C. Y., & Dispenza, F. (2018). Qualitative clinical mental health program evaluation: Models and implications for counseling practitioners and educators. Journal of Mental Health Counseling, 40(1), 1-13.
Owen, J. M. (2020). Program evaluation: Forms and approaches. Routledge.
write