Case Study 2: Evaluating a School-based Alcohol Education Program (continued)
Phase 2: Data Collection
(A full guide to all steps and sample survey questions are provided in following resource: European Forum for Responsible Drinking, EFRD. [in press]. Evaluation guidelines for education interventions to promote responsible drinking amongst young people in seven European countries. Available: www.efrd.org)
The methodology needed for evaluation requires measurement of output, outcome, and impact. The data need to be collected at three different points:
1. Before the program is launched
2. During the program period
3. After the program is completed
This allows measurements and data to be compared so as to determine impact.
Before the Program
Given the nature of the study and that it involves young people, parents’ consent is required. This should be addressed as a first step before the program is put in place.
Also, all materials should be shared with teachers, and final adjustments made based on their feedback.
Establishing a baseline
Data should be collected to establish existing levels of awareness and behaviors against which to measure any changes.
A full description of questions and sample questionnaires are included in the EFRD Evaluation Guidelines for Education Interventions to Promote Responsible Drinking amongst Young People in Seven European Countries (www.efrd.org). However, some possible questions include the following:
· What is the level of awareness around drinking and alcohol, generally?
· What is the level of awareness among respondents around harms and benefits of alcohol consumption?
· How many respondents have consumed alcohol? At what age? How often do they drink?
· What are their general attitudes toward drinking?
Further qualitative data about attitudes and awareness can be collected through focus groups. Sample questions suitable for focus groups are outlined in the EFRD Guidelines.
Additional information can be gathered by conducting focus groups with parents or guardians to assess their views on their children’s attitudes and behaviors toward alcohol.
Additional information is required to rule out other influences, for example:
· Assess whether any other interventions or information are in place to which respondents may be exposed during the program period.
· Validate the questionnaires to be used by administering them to a subgroup of respondents. This will determine whether any adjustments are needed to the research design or the questions themselves.
This information is needed before the campaign is launched. It allows any adjustments to be made to the study design so that such external factors do not interfere with the results. Any potential confounders need to be acknowledged in the analysis of the evaluation results.
During the Program
The materials used and the number of program recipients targeted should be carefully tracked while the program is in progress:
· How many young people were involved in the educational program?
· What were their ages? What was the gender composition of the program recipients?
· What type of materials did they use (e.g., print, web-based, video)?
After the Program
Once the program is over, a second round of information-gathering will help with assessment of outcomes and impact. These data can be collected through:
· Surveys conducted in schools where the program was carried out
· One-on-one interviews with participating students, their parents, and teachers
· Focus groups drawn from among the young people, their parents, or teachers
It is important to also have a second target group to measure. These should be young people who have not been exposed to the program. They are the control group and should be asked the same questions and given the same surveys as the experimental group. It is important that the demographics of the control group closely match those of program participants.
· Were the materials used and understood? Was there interest in them?
· Were students aware of the topics covered, and did they understand them?
After the program has ended, questionnaires are to be administered to both the control and experimental groups. This will determine whether there has been any measurable impact on awareness and/or behavior.
· Survey questions that address drinking levels, patterns, attitudes, and awareness should be administered to both groups. (Sample surveys are included in the EFRD Guidelines.)
· Qualitative one-on-one interviews should be conducted with a sub-group of respondents.
· Additional focus groups with both parents and young people are also helpful.
If possible, a follow-up data point should also be included. This should be collected some time after the program is concluded to allow assessment of whether any changes resulting from the program have persisted. Follow-up is particularly important in determining any effect on the behavior of respondents.
The questions above should be repeated, and with the same respondent groups. This will allow insight into the following:
· Have changes been sustained over time? For how long? How much of the change is sustained?
(This will require follow-up measurements at different time points, with the same questions asked at each.)
Reliability and consistency
To increase the likelihood that the collected data are reliable and consistent across different time points, it is important to pay attention to the following:
· All data should be collected in the same way, using the same instrument (e.g., standard questionnaires) at each time point.
· Where one-on-one interviews or focus groups are used, the same individuals should be included, to the extent possible, in each data collection round.
· Where this is not possible, or where broader surveys are used, efforts should be made to maintain consistency among respondents: i.e., the same target groups should be used, and their demographic compositions should be kept as consistent as possible.
NEXT – Case Study 2: Reporting and Dissemination
PREVIOUS SECTION – Case Study 2: Planning and Inception