Electrifying the Adolescent Pregnancy Prevention Program Evaluation

The TPPI team achieved the following outcomes: 1. Increase satisfaction with the survey administration process by 20% by July 31, 2013. The baseline value was 4 (somewhat satisfied), and the final value was 5 (extremely satisfied), which equaled a 20% increase in satisfaction. 2. Reduce data submission errors on manual surveys by 50% by September 30, 2013. The baseline error rate was 80%, and the error rate on September 30, 2013, was 45%, which equaled a 43.8% reduction in errors. 3. Reduce the number of process steps by 50% by May 31, 2018. Measurement is still in process. 4. Decrease the turnaround time to 6 months or fewer for 90% of first draft evaluation reports by December 31, 2016. Measurement is still in process. 5. Increase agency satisfaction with the turnaround time of evaluation reports by 23.5% by January 31, 2017. Measurement is still in process. 6. Increase pilot site use of the final evaluation report by 33% by December 31, 2017. Measurement is still in process.

• Electronic data collection: NC DPH - Process steps were decreased from 15 to 9, or 40% improvement. - Survey storage in a secure cabinet was eliminated. - Motion waste (to and from copier and printer) was eliminated. • Electronic data collection: WHB - Process steps were decreased from 21 to 4, or 89.9% improvement. - Purchasing and vendor involvement and time were eliminated, thus eliminating waiting time. - Many handoffs and waiting wastes were eliminated. - The survey submission process and form was eliminated, which reduced submission errors that caused rework. • The number of process steps for agencies and WHB was reduced from 36 to 13 (electronic data collection), or 63.8% improvement. • The electronic survey reduced the overall process time from 24 months to 18 months, or 25% improvement, enabling agencies to apply report results to improve their programs. • The APPP orientation and manual provide standard work to clarify program process and requirements. • The pilot changed participant consent forms to cover a 3-year period versus a 1-year period. • Agency satisfaction with the survey administration process increased from 4 (somewhat satisfied) to 5 (extremely satisfied) per the satisfaction survey. • WHB staff members verbally reported increased satisfaction with the data processing steps (decreased data errors, eliminated purchase requests, aggregate report changes, etc.). • The TPPI team anticipates that a larger pool of agencies will apply for funding because of improved technology within APPP. • The electronic tools (laptops, tablets, smartphones) used for survey administration can also be used in teaching program material and for other administrative tasks like populating the TPPI database, which will improve program implementation. • The TPPI team anticipates improved program implementation due to receiving evaluation reports early enough to make program changes (e.g., improved fidelity, teaching methods, staff interaction with teens), which may increase adolescent sexual health knowledge, attitude, and behavior. • The state government's reputation is improved (e.g., saving time, "getting with the times" with electronics). • Not having to throw out surveys with errors improves overall survey results and the strength of the findings and eliminates waste. Agencies that had missing data in the past would now have an evaluation report. • Storing surveys at a warehouse for WHB was eliminated. • The need for involvement of the purchasing office for WHB was eliminated. • The survey submission form led to reduced errors and wastes (time). • The quick reference guides led to reduced errors and wastes (time). • Less rework by agencies cuts down on rework by WHB staff. • Less postage is needed to return agency surveys with errors for rework. • Agencies do not need to do rework if they are not making errors because of improved TA, orientation, the APPP manual, the survey submission form, and so forth. • Staff now scan copies instead of producing paper copies. • There are no repeated postage costs for rework. • Communication and teamwork have improved. • Staff are making more efficient use of meeting time.

In this process, TPPI team members learned that they had to be open to where the QI tools led. Having a diverse team with a clearly defined aim and scope dedicated to using the QI tools effectively aided in this process. When the team first started out with this initiative, they thought they had a definite answer to the problem: for all agencies to submit surveys electronically. However, by working with team members representing different views along the program spectrum and not taking shortcuts around using the QI tools, the team found that their ideas were only superficial solutions. The team discovered missing components that played a more pivotal role in successfully evaluating the programs. First, they realized that there was no standard work available for agencies, even for the seemingly smallest task, to promote accuracy and consistency. The team also found that, counter to their assumption, not everyone was ready and able to submit surveys electronically. Most shockingly, without asking the 5-Whys, the team would have never known the actual protocols and guidance around the evaluation reporting timeline (there were none) that seemed to be the driving force behind when final reporting was completed.

As part of the CPHQ QI 101 Program, the TPPI team involved a sponsor and the process owner (a manager of the process that was addressed) in creating a comprehensive sustainability plan to organize efforts. The TPPI team used CPHQ's sustainability checklist to create the plan. The plan/checklist is attached in the QI Tools section.