Case Study 1: Evaluating an Awareness Campaign around the Use of Designated Drivers (continued)

Phase 2: Data Collection

The methodology needed for evaluation requires measurement of output, outcome, and impact. The data need to be collected at three different points:

1.                   Before the campaign

2.                   During the campaign

3.                   After the campaign

This allows measurements and data to be compared so as to determine impact.

Before the Campaign

Establishing a baseline

Data should be collected to establish existing levels of awareness and behaviors against which to measure any changes that may result from the campaign.

Some possible questions include:

·         What is the general level of awareness around drinking and driving?

·         What is the level of awareness among respondents around designated driver schemes?

·         How many respondents have used or are likely to use designated drivers?

·         How many respondents have themselves been designated drivers or are likely to be designated drivers?

There are various ways to gather these data:

·         Surveys conducted in premises where campaign was carried out

·         One-on-one interviews with target audience, also on premise

·         Focus groups drawn from target audience

Information is also needed on the demographics of the respondents:

·         What age groups are represented among the respondents?

·         What is the gender composition?

·         What are the respondents’ drinking patterns? (e.g., the duration and frequency of drinking, the quantity of alcohol consumed per occasion, the type of drinks normally consumed, regular settings and reasons for drinking, etc.)

·         What is known about individuals who are likely to drink and drive?

·         What is known about those who are familiar with or likely to use designated driver schemes?

Addressing confounders

Additional information is required to rule out other influences, including:

·         Other concurrent prevention initiatives and campaigns around designated driver schemes

·         New legislation or regulations around drinking and driving

·         Changes in enforcement (e.g., increased police vigilance, road blocks, and fines)

This information is needed before the campaign is launched. It allows any adjustments to be made to the study design so that such external factors do not interfere with the results.

Impact on objective measures (e.g., road traffic incidents) can be assessed through desk research, as can the analysis of official law enforcement statistics on crashes, fatalities, and breath-testing at road blocks.

Any potential confounders need to be acknowledged in the analysis of the evaluation results and their likely effects discussed.

During the Campaign

Measuring output

The materials used and the number of respondents targeted should be carefully tracked while the campaign is in progress:

·         How many brochures or other promotional materials were printed or otherwise produced?

·         How many were given to the target audience?

·         How many separate venues were involved in promotion?

·         How many times was information distributed?

After the Campaign

Once the campaign is finished, a second round of information-gathering will help with assessment of outcomes (and impact). These data can be collected through:

·         Surveys conducted in premises where the campaign was carried out

·         One-on-one interviews with target audience, also on premise

·         Focus groups drawn from target audience

Ideally, the same respondents used to set the baseline should also be used for data collection after the campaign.

It is important to also have a control group against which to measure any changes in the target group.  Both groups should have a similar demographic composition. The difference is that that the control group will not have been exposed to the campaign.

Measuring outcomes

·         Were the materials read by their intended audience? Was there interest in them?

·         Are target audience members aware of the campaign and the information contained in the materials?

Measuring impact

The most desirable results relate to the impact that the campaign may have had on attitudes and behaviors of the respondents. In this case, the impact also relates to any changes that may have occurred in road traffic crashes as a result of the campaign. The same measurements that are taken from the target group of the campaign should also be taken from the control group. For example:

·         Did the campaign change awareness about designated driver schemes among respondents? Did awareness increase/decrease/stay the same?

·         What was the level of awareness among the control group?

·         Were changes different among different groups of respondents (e.g., differences by age, gender)?

·         Did the campaign change intent to rely on designated drivers (i.e., would respondents be more or less likely to use a designated driver as a result of having been exposed to the campaign)? How does this compare with the control group?

·         Did the campaign result in a change in behavior? Among whom? Are members of target audience more likely to use designated drivers as a result? Have they changed their use of designated drivers?

·         Did the campaign have an impact on road traffic incidents (crashes and fatalities)? By how much?

(This sort of measurement is only realistic if the sample size used is large enough to effect a demonstrable change in objective measures like road traffic crashes.)

If possible, a follow-up data point should also be included. This should be collected some time after the campaign is concluded to allow assessment of whether any changes due to the campaign have persisted.  Follow-up is particularly important in determining any effect on the behavior of respondents.

Reliability and consistency

To increase the likelihood that data collected are reliable and consistent across different time points, it is important to pay attention to the following:

·         All data should be collected in the same way, using the same instrument (e.g., questionnaire), at each time point.

·         Where one-on-one interviews or focus groups are used, the same individuals should be included, to the extent possible, in each data collection round.

·         Where this is not possible, or where broader surveys are used, efforts should be made to maintain consistency among respondents: i.e., the same target groups should be used, and their demographic compositions should be kept as consistent as possible.

NEXT – Case Study 1: Reporting and Dissemination

PREVIOUS SECTION – Case Study 1: Planning and Inception