Process Evaluation in a Randomized Community Trial of a Youth-Aimed, Substance-Use Prevention Media Campaign

Published in the inaugural issue of the journal Cases in Public Health Communication & Marketing, this article explores the importance of process evaluation as a key to the effectiveness of any public health campaign, especially during longitudinal and multi-site field studies. In order to help meet the need for more information on process evaluation, which public health and communication researchers have increasingly articulated (according to the article), the authors here describe the undertaking - and share the results - of a community-based process evaluation of a substance-use prevention media campaign carried out in the United States called "Be Under Your Own Influence". They use this example to illustrate the development of a particular process evaluation instrument and approach, offering guidelines for other researchers who are implementing similar types of research on public health communication campaigns.

The paper begins with background information on the campaign itself. Funded by the National Institute on Drug Abuse (NIDA), this 2-year media intervention was implemented in schools, with some supporting community activities, and was designed to reduce the uptake of marijuana, alcohol, and tobacco among middle-school students. As detailed here, activities developed for the public health campaign were initiated in response to research indicating that the early initiation of substance use remains widespread among American adolescents. Designed to reach middle-school youth, the project involved exploring - and contributing to - a field of research that has found that success of media-based campaigns designed to prevent substance abuse has been mixed.

Specifically, "Be Under Your Own Influence" tested a media intervention that was conducted in various schools over a period of 2 school years. The tangible products of the school-based component of the intervention included: 2 sets of 4 posters, as well as materials such as book covers, banners, tray liners, T-shirts, water bottles, rulers, lanyards, and stickers. The authors (who designed, implemented, and evaluated the campaign), explain that "Schools served as the main distribution channel for these materials because of their potential to reach target audience members (youth) where they spend most of their time - at school. Maximizing this distribution channel was central..." That said, organisers also offered community mobilisation training to youth group organisation leaders, law enforcement officials, and health coalition members. A full-day community workshop, held at the beginning of the first school year, was intended to help participants assess their community's readiness to take action on the issue of youth substance abuse prevention. Whenever possible, the workshop was held in conjunction with local prevention coalition meetings, and participants discussed prevention strategies that would best use their resources. Each community contact received a "media kit," which contained press releases about the campaign and general articles about youth substance abuse prevention that could be used in local print media, as well as radio and television public service announcements (PSAs) that could be used in local broadcast media.

This article briefly reports on the methodology behind, and impact of, this research-based intervention. Eight media treatment and 8 control communities throughout the U.S. were randomly assigned to a condition. Within both media treatment and media control communities, one school received a research-based prevention curriculum and one school did not. Four waves of longitudinal data were collected over 2 years in each school. Youth in intervention communities (N = 4,216) showed fewer users at final post-test for marijuana [odds ratio (OR) = 0.50, P = 0.019], alcohol (OR = 0.40, P = 0.009), and cigarettes (OR = 0.49, P = 0.039). Growth trajectory results were significant for marijuana (P = 0.040), marginal for alcohol (P = 0.051), and non-significant for cigarettes (P = 0.114). In short, evaluators found that the campaign was effective in reducing uptake of marijuana and alcohol by up to 40%.

The focus of the article, however, is on the methodology of the process evaluation itself. The authors explain that, because research staff members were not on site to ensure distribution of campaign materials, process evaluation and partnerships with field contacts in schools (e.g., drug-free programme coordinators) and communities were key strategies for monitoring how these materials were being used. To help faciliate the effective involvement of treatment communities in the process evaluation exercise, participating schools and community contacts were provided with in-person training (in conjunction with the community workshop) and a handbook, which specified guidelines for implementation, expected uses of materials, and other ideas to support substance use prevention. However, since organisers anticipated variation in how schools and communities chose to implement materials, they did not expect rigid adherence to a set of instructions.

The following sections of the report outline the development and administration of the process evaluation instrument used as part of this research project. (A process evaluation was conducted every 6 months over the 2-year evaluation period). The authors examine in detail their approach, which involved first presenting contacts with questions about community efforts, followed by questions about media efforts, and ending with questions assessing the overall progress of the project. A table provides examples of questions asked as part of the community-efforts component; for example, "Of all the communication "tools" we've sent you (media manual, promotional items, public service announcements, press kits with articles/columns, etc.), which do you think has been most useful? Least useful? Why?" and "How will your participation in this project affect future prevention efforts in the school/community?"

Lessons learned from both the campaign (as part of the process evaluation) and from the actual undertaking of the process evaluation are shared in detail in the ensuing sections of the paper. To cite just a few examples of insights from participants in the campaign, "One community contact shared with research staff the idea of distributing wallet cards to 8th graders, bearing the "Be Under Your Own Influence" logo, as part of the 8th grade graduation ceremony. Other community contacts requested information aimed at parents, which explained the signs of substance use. This information could be distributed through local media, parent newsletters, or other channels in order to facilitate parent-child discussions. This prompted staff members to gather research-based materials, and to provide them to all of the treatment communities, so that they could be used to supplement existing materials." The following excerpted material summarises the lessons learned from the analysis of the process evaluation procedure itself:

• "Introduce the concept of process evaluation early. Some field contacts may not be familiar with the term "process evaluation," and may view it as an unnecessary burden. Therefore, it may help to provide potential participants with the rationale for conducting the process evaluation, the expected time commitment, and a tentative schedule, at the beginning of the project...
• If feasible, designate one person who is familiar with project components and who ideally has had some personal contact with communities to conduct the process evaluation...
• Become familiar with the calendar at each specific field site to guide the selection of dates for sending evaluations and conducting follow-ups, as well as to see if there are upcoming events that can dovetail with program goals. Calendars and list of upcoming events are often posted on organizations' websites.
• To encourage prompt response, ensure that process evaluation "packages" are easy to access and process...[drawing on such components as] a personalized cover letter, a real signature on the cover letter, information on multiple ways of responding, survey questions that are easy to interpret, and delivery via first-class mail. If contacts do not return their surveys within a reasonable amount of time, a reminder can be sent via e-mail (with the survey attached), or by phone.
• Even if contacts return surveys on time, plan on discussing the survey with them. Conversations can serve multiple goals, chief among them gathering information and building relationships...
• Ask contacts to prepare for evaluations by keeping an ongoing record of intervention activities....In this study, participants were asked to keep a file with notes on media material and promotional item usage, as well as any newspaper clippings of media coverage. We also suggest giving contacts the process evaluation form in advance...
• Share ideas gleaned from the process evaluation with key staff, as well as other treatment communities, as appropriate....Although some might argue that sharing ideas confounds treatment effects, we argue that it helped equalize conditions among communities, thus allowing for better evaluation of intervention effects."

In conclusion, the authors stress that "process evaluation enhanced the implementation of the 'Be Under Your Own Influence' campaign, which was later demonstrated to be successful in reducing uptake of substances among middle-school youth. The findings reported here point to challenges that may arise in school and community environments, as well as opportunities for resource sharing and relationship building."