The success of the data collection phase often depends on the quality of the planning stage. How long data collection will take depends on the type of evidence required and the methods chosen for evaluation. The following questions will guide work at this stage:

·         Where is the information to be found?

·         Who is best positioned to get it and when?

·         In what form should the information be stored, analyzed, and presented?

Two different types of data can be collected for any assessment:

·         Qualitative data are descriptive and cannot be measured in absolute terms. They can be obtained from respondents’ verbal answers to interview questions, focus group discussions, or written commentaries and responses to open-ended questions. 

·         Quantitative data are measurable and definable and can be converted into numbers and statistics. They are useful in showing absolute differences in what is being measured, such as percent changed. These data are derived from ratings, rankings, or “yes” and “no” answers to questionnaires.

Both types of data are useful.  Where possible, it is often helpful to supplement objective, quantitative measures with more subjective and descriptive qualitative data.

How data are collected influences whether they will be qualitative or quantitative.

·         Several approaches can be taken to gather data, ranging from simple to complex, and all can be used in evaluation.

·         Each approach has its own set of strengths and weaknesses. 

·         The choice of the most appropriate evaluation approach will depend on an assessment of what is available and feasible on a case-by-case basis. 

When data are collected, it is useful to also have additional information that is not directly related to the intervention being evaluated.  For example:

·         It is important to know the sample size, the number of people who were reached, and their characteristics (e.g., age, gender, socioeconomic status, etc.)

·         In measuring change attributed to a particular intervention, it may be necessary to establish the baseline before the program is carried out (e.g., it is helpful to have the “before program” statistics, where available, on a particular topic).

The following are some of the most commonly used approaches to collecting data: surveys, observations, interviews, and focus groups.

3.2.1  Surveys

Surveys rely on data collection through questionnaires. These may be written (distributed in hard copy or electronically) or administered orally (e.g., by telephone).

Usually, in order to evaluate the impact of an intervention or a program, the survey is administered twice: once before and once after the intervention.

·         Administering the survey before the intervention establishes baseline responses. 

·         A second round of surveys among the same group after the intervention will show whether there is a change in knowledge, behavior, or whatever other outcome being measured. 

·         In some cases, the survey may be applied a third time to assess longer-term impact. This is particularly useful when measuring behavior changes. It also helps to determine whether any short-term changes are sustained over time.

Another approach is to survey those who receive an intervention and those who do not, and to compare the results. This provides a “control” group for measuring the effect of a particular intervention.

Although surveys are used extensively in evaluation and, when constructed properly, can be very useful, the response rate to a survey may pose a challenge.

·         Not all those who receive a survey will wish to respond to it. This may hinder the reliability and validity of the information that is extracted. It is therefore important to include a large sample to control for dropouts. 

·         Knowing the characteristics of the sample (e.g., age, gender, education level) will also help determine if the dropouts have some other significance. For example, conducting a household telephone survey during weekday mornings or afternoons will miss all those who are at work. This is a useful piece of information to have when interpreting results.

3.2.2  Observations

Observation of individuals who have been exposed to an intervention (e.g., observing serving practices at retail establishments after a server training program or observing group dynamics during the delivery of a program) can help measure any changes in behavior or outcomes.

·         As in surveys, a “before” and “after” assessment is needed to compare and measure effects.

Observation can help with determining whether a program is being delivered and implemented as planned and enable the evaluator to understand the situation and context. 

However, observation is time-consuming and expensive, and care must be taken with interpreting its results, usually requiring an expert.

3.2.3  Interviews

This approach allows a one-on-one relationship between the evaluator and the respondents, and is particularly useful where personal contact is important.

·         Interviews are especially helpful when the topic is complex and requires additional explanation or when there are language (e.g., high illiteracy rate) or cultural barriers.

·         Interviews also allow for immediate follow-up on interesting issues that may come up during discussion.

Given the nature of interviews, the sample size is likely to be relatively small, and the information obtained will depend heavily on the skill of the interviewer.

One useful form of interview involves so-called key informants who can speak on behalf of a larger population sample.

·         Key informants are experts in a particular field who can provide a broad view, representing the state of knowledge or different cultural views. 

·         These can be academics with expertise in a particular field or members of a community with knowledge of and ability to speak for the community as a whole.

Information can be obtained from key informants informally or in more formal ways through interviews, hearings, or surveys. Key informants are frequently used to inform the work of bodies like the World Health Organization (WHO).

3.2.4   Focus groups

Focus groups combine elements of both observation and interviewing:

·         A focus group is an interview with a gathering of 8 to 12 people, but uses group interaction to generate data and insights that would be unlikely to emerge in individual interviews.

·         Originally used as a market research tool to learn the appeal of various products, the focus group method has been adopted in other fields as a way to gather data on a given topic.

Focus groups participants must be selected with care so that they are representative of the wider sample of individuals who were intended to be reached by the program.

Focus groups are often used in the pilot phase of a study to provide an idea of likely outcomes, followed up with a larger-scale approach to evaluation using, for example, a survey.

NEXT  – Section 4: Data Analysis and Interpretation

PREVIOUS SECTION – Section 3.2: Planning an Evaluation