Print Page: Word; PDF

Tips

Module 5: Data You Need

• A Question of Rigor
• Avoiding Biases in Your Comparisons
• Constructing a BaseLine
• How Big a Sample Do You Need?
• How Can You Help People Remember?
• Kinds of Samples
• Pros and Cons of Using Different Comparisons
• Selecting a Comparison Group
• Selecting Data Sources
• What About a Control Group?
• When Is the Endline the End?

A Question of Rigor

When deciding your comparisons and data sources, balance rigor against costs and reality. Whenever possible, choose an option that tends towards more rigor within similar costs.

Download: Word

Whenever you are using less rigorous approaches, increase the rigor by:

• Collecting data from more than one source (triangulation).
• Combining qualitative and quantitative data collection methods (triangulation) (more about this in Module 6).
• Checking to avoid biases in all of your comparisons.

Avoiding Biases in Your Comparisons

Baseline vs. endline comparisons:

• Establish trust during the baseline assessment to avoid underreporting of socially criticized behaviors and attitudes, or over–reporting of popular or high status behaviors or attitudes.
  • When collecting data for your baseline you might have to explore topics that are difficult to talk about. Interviewees who have not yet entered the program may under-report sensitive or controversial behaviors or attitudes or boast about exploits.
  • Through participation in the program – where trust is likely to develop- you may find higher levels of behaviors that people were too shy to report about at the time of the baseline or lower levels of bragging about high status attitudes or behaviors. Either way, this will distort your comparisons.
  • Ask the respondents how comfortable they feel talking about sensitive issues, and ask on the post-test assessment if they felt more or less comfortable talking about such issues than before.
  • At mid-and/or endline, ask whether or not the participants perceived changes and whether they think those changes are due to the program. Even if you have a solid baseline, such a question will complement those data and may lead to important insights.
• Plan and use your monitoring data to measure developmental changes or other life-altering events or interventions that may occur between the baseline and endline assessments.
  • Young people may have passed important developmental milestones during the time of the program, such as becoming sexually active. This doesn’t mean that if your program was seeking to reduce unsafe sexual activity it had no effect. Their behavior may still be far safer than it would have otherwise been, or in comparison with similar youth who were not exposed to your program (a control group).
  • External events may have taken place that affect your findings, such as political upheavals or scandals, natural disasters, educational TV or radio programs that were aired, etc.
  • If you are using public records, staff may have received new incentives or training to fill them out better, or a new manager or evaluation system may have made them more fearful of registering certain information.
• Review the composition of the actual groups you are comparing.
  • Drop outs during the program, or an influx of late-comers (who arrived after the pre-test was applied) might make the groups during the pre-test and post-test different on some critical characteristics, such as motivation, education levels, family support, etc. By continually reviewing your monitoring data you can know if this is happening and hopefully avoid biases in your comparisons.
  • Decide if you are only looking at matched pairs of surveys- where you have devised a way to identify people in the pre and post test while still keeping their anonymity- so that you can know the program’s impact by individual.
• Plan ways to undertake person by person comparisons at pre and post-test moments. Even if this is a small subgroup for which you can match respondents, you can gain important information about change on the individual level, so if the groups actually differ you can still know whether or not the program made a difference
• If anonymity is an issue, make sure you include some unique identifier in your data collection instrument so you can match respondents without revealing the people’s identity. Unique identifier: A small group of questions the respondent will remember and answer the same each time he or she is interviewed without revealing his or her identity, such as first name of sister or brother closest in age, favorite color and number of brothers and sisters.
• Tips: Confidentiality and Anonymity (Module 6: Collecting Data)

• Find a group of people, events or documents that has similar characteristics to those exposed to the program. Tips: Selecting a Comparison Group
• If you can’t find a comparison for your entire population, find one for a subgroup of your population that is most important to you and your stakeholders and/or audience. Example: If your entire population includes youth from 10-24 years of age, but you are most interested in the 10-15 year-olds, you could look for a comparison group of 10-15 year-olds and then compare that sample group to the 10-15 year-olds in your original population.
• Use general population data instead of a comparison group (for example, compare the youth in your program with the youth in the entire town) if you have district/town/state level data that are appropriate. But make sure that the young people in your program do not differ on other important characteristics from the general population or that they make up such a sizeable proportion of the general population data that you would be comparing them to themselves.
• Ideally, use the comparison group assessment as a baseline for the next cycle of your program. That way, the people you interview will eventually be exposed to the program and you won’t feel you are investing resources without providing the kind of services or programs you want to provide.
• The comparison groups should show a similar level of interest in the relevant issues as your program group. Be aware though, that they may differ in terms of openness about the topics that are dealt with in the assessment, introducing the same bias sometimes found with baseline vs. endline comparisons.
• Regardless of how you selected the comparison group, always be sure to spell out how the comparison group is different from the program group in ways that might impact your evaluation findings.

Internal comparison groups:

• Find out if there were subgroups that were more or less exposed to the influence of the program. Examples:
  • People who attended all of the sessions versus those who only attended a few of the sessions
  • People who recall seeing or reading the messages your program disseminated versus those who did not
• Divide your program participants into two or three groups. Those who
  • received enough exposure to expect them to exhibit the desired change
  • those whose exposure was so minimal that you do not expect the program to be able to influence them and
  • possibly a group in between these extremes

Make sure that these groups don’t differ on issues as important as motivation, economic resources, etc.

Example: In an HIV/AIDS education program for miners, some of the miners who started in the program had been transferred out for several months to another region. Even though they returned in time for the post-test, they had not been exposed to the entire intervention. If they had been transferred according to some procedure or trait that is unrelated to program results they might serve as an excellent comparison group. However, if you find out that only the better educated miners or the most confrontational ones had been transferred out, you would have to only use the data you have on the miners who stayed for the entire program. If your comparison group was composed of miners in a different mine, you would also want to try to match them with your subgroup on education and disciplinary record so the two groups would be as similar as possible.

Thus your comparisons could be:
Miners who participated in your program versus miners who were transferred out randomly (and didn’t participate in your program)
-or-
Miners who participated in your program versus miners in another mine (with no program) who are similar to the miners who participated in your program (e.g. on educational level and disciplinary record)

• focus on more concrete events, and ask them to provide examples.
• ask the respondent to provide reasons or explanations for his or her perceptions.
• capture data from diverse sources and compare them to corroborate the findings, e.g. reports from hospital staff and hospital records.
• make sure you talk to people who have varying degrees of identification with the program.

Can You Rely on the Records?
In many cases, pre-existing records that are on file will have been filled out in less than optimal conditions. Interview the people who fill those records out to find out how reliable they are before relying on ones that might be highly misleading. If the endline assessment will rely on those same records, make sure there has not been some influence – such as greater interest in the topic, or a new policy requiring staff to make the time to fill the forms out – that might change the accuracy with which the records are filled out during the period of the evaluation between pre- and post-test times.

Choosing a sample: What Characteristics Should You Pay Attention To?
When choosing a sample or comparison group, pay attention to characteristics that are:

• directly or indirectly related to the program enrollment, participation and objectives and
• to how much the person, event or document would be influenced by the program.

Don’t select a sample or comparison group on characteristics that are irrelevant to the program objectives.

Example: If a program is educating legislators about women’s sexual and reproductive rights to encourage broadening laws and procedures,

• Relevant characteristics might include the legislators’ sex; attitude towards women’s empowerment; religious, tribal or party affiliation; and membership in key legislative committees.
• Less relevant characteristics for choosing a sample or comparison group might include their number of children, education, years in the legislature, and marital status.
• Irrelevant characteristics might include home ownership and number of status possessions.

Constructing a Baseline

If you are starting your evaluation of a new program:

Constructing a baseline is perhaps the most important thing you can do to ensure you have useful evaluation findings. Collect at least some baseline data on the questions most important to your evaluation.

When to collect baseline data:

1. prior to the beginning of the program.
   • If you had a needs assessment that included data collection on the indicators, use those data as your baseline.
   • Apply the data collection instrument when people sign up for the program
   • Review documents before the program starts
2. at entry into the program
   • in cases in which people register or start to participate at a defined moment, you can apply a pre-test instrument with the same questions that you will use for midline and/or endline assessments.
   • In cases in which the program participants are not a “captive audience”, apply the pre-test instrument to a sample of people, events, or documents which will be exposed to the program.
3. early in the program, before effects could be expected

How to keep costs low:

1. use needs assessment data if
   1. they fit the indicators you have decided are needed for the evaluation
   2. were collected from the same data source you will be using for your midline and/or endline
   3. were collected in the same way you will be collecting the midline or endline data
2. prioritize the questions that are most important
   1. focus on the objectives and activities that are most important to the program team, stakeholders and donors
   2. focus on the most important subgroups in the possible population, such as single mothers, youngest boys, most conservative politicians or newspapers, most stigmatized people - those most important to the results of your program.

A common challenge is evaluating a program for which no adequate baseline data exists.

If a baseline exists:
Although not very common, you may find some initial measures have been obtained. You can use those data if:

• the existing baseline includes information on the indicators you have already selected as necessary to assess changes on the immediate and intermediate objectives.
• the existing data can be re-analyzed or re-grouped to give you the information you need to measure change on the program’s immediate and/or intermediate objectives.
• the baseline was applied to people who are either those who are participating in the program or very similar to those in the program.
• the existing data were collected in ways that are similar to the ways you will be collecting the post-test data.

* It is better not to have a baseline than to derive the wrong conclusions from a poorly constructed one.

If no baseline exists:
This is the most common situation

• Use a comparison group. Tips: Selecting a Comparison Group
• If no comparison group is available or appropriate, construct a retrospective baseline comparison by
  • Creating retrospective versions of the midline or endline questions, e.g., before the program, or during such and such a year, what was the situation like on X, Y or Z. Tips: How Can You Help People Remember?
  • Use qualitative perceptions of change by asking your respondents how they perceive the differences from before the program to the current moment.
  • Use different ways of finding out what things were like before the program, for example people’s perceptions plus town records. This is another example of how triangulation can help.
  • Always include a question about whether or not the program participants perceived changes and whether they think those changes are due to the program. Even if you have a solid baseline, such a question will complement those data and may lead to important insights.

How Big a Sample Do You Need?

• It may come down to time and money.
  • Decide how much time, money and people you have available to collect the data.
  • Decide how much time and money is needed for each type of data collection (once you design your Data Collection Instruments in Module 6) and then work backwards.
  • Balance the kind of data collection instrument with the number of data sources you can include. For example, you may be able to apply a brief survey to everyone, but if you want to do in-depth interviews you may have to choose a small sample of respondents.
  • Keep in mind the type of evaluation question you want to answer. Sometimes more in-depth data collection on a smaller sample will be better than more superficial data collection on a larger sample.
• If you are using qualitative data collection instruments (see Module 6), plan to include at least 3, but preferably 5, people for each of the categories you used to select the sample. Example: If your sample needs to include men who have sex only with men, men who have sex only with women, and men who have sex with both, then try to form a sample with 3 to 5 men in each of these categories.

Back | Back to Top

How Can You Help People Remember?

There is always some risk when asking people to remember how things were before the program. They may have forgotten or other events may have made them remember things differently. To help them remember better, try anchoring them in the time you are interested in. For example, in the introduction to the interview or section, ask them to remember where they were living, who they lived with, what they were doing at that time. Once they situate themselves in the time period you can ask them the specific questions needed for the baseline.

Kinds of Samples

Pros and Cons of Using Different Comparisons

Given limited resources, it is not realistic to think you will be able to make all the comparisons possible. But you should be aware of the pros and cons of the comparisons you choose. The decision comes down to balancing resources and rigor.

Download: Word

Selecting a Comparison Group

Make sure the comparison group is as similar to the program group as possible. Thus you might include:

• People who fulfill the same selection criteria for the program but arrived once the program was full or are eligible for the program in some future time.

People selected from the general public who are similar to the program group on a set of essential characteristics. You can select a comparison group in the same way you might select a sample. Tips: Kinds of Samples.

Selecting Data Sources

For each potential data source ask yourselves the following questions:

What source or sources will provide the most complete, accurate and relevant information?

• Will the source be free of important biases and answer questions honestly?
• Will the respondent be able and willing to provide complete and honest written answers?
• If the source is a document, such as meeting minutes, clinical files, etc., is the record likely to be filled in accurately and systematically?
• Can direct observation provide information you could not obtain otherwise, and be done in such a way as to avoid unnecessary changes in people’s behavior?

Can you easily gain access to this data source?

• How difficult will it be to find the people you need to talk with?
• How difficult will it be to get people to talk with you or fill out your survey?
• Are the documents available and accessible to you and your team or to certain members of your team?
• Can you, or particular members of your team, attend the events you need to observe without altering the things you want to observe?

Does data collection pose any potential risks to the data source?

• Will the fact you are trying to survey/interview certain people cause those people any harm or discomfort, e.g., by identifying them as being members of a certain stigmatized category. How could you protect them from such harm or discomfort?
• Will it be possible to protect anonymity, confidentiality and privacy of the people involved?
• Will you be asking people to recount experiences that may be particularly upsetting for them to discuss?
  • Anonymity – condition in which the person or item can never be identified by anyone by name
  • Confidentiality – condition in which the person or item can never be identified by people other than the data collector, including other members of the evaluation and program teams.
  • Privacy – condition in which the person who is providing the data is not overheard by others or otherwise made uncomfortable about providing honest responses.

What About a Control Group?

A control group is a comparison group that was selected from the outset and not exposed to the program. They are ideal because they give you the pre-post-program comparison plus the contrast between at least two different groups at two different times. However, they have other disadvantages that make them very uncommon in reality for low resource programs.

• There may be no good control group available.
• Some organizations feel that it is unethical to contact the people but not provide them with an intervention.
• Concern about exposing the control group to the pre-program questions which, if they deal with sensitive issues, may awaken conflicts, open topics that could cause discomfort, etc. without providing the respondents with some form of professional counseling afterwards. Any interview, especially about sensitive topics, can have an unknown impact on the person interviewed so provide some kind of way to get help if this occurs.
• Concern about the control group differentially dropping out during the program period since they are not receiving anything or benefiting in any way. Since there will always be people who drop out of both groups anyway, the evaluator needs to make sure more or different kinds of people didn’t drop out of the control group making it not very useful after all.
• Some programs solve these dilemmas by providing a basic package to the control group, and the basic package plus some new intervention to the program group. This could increase the costs and make using and maintaining a control group too expensive unless the agency has multiple programs to offer.
• Contamination is also a problem. Sometimes the effect of the program can reach the comparison group, especially if both groups are from the same community. This only adds to the problems involving control groups that make them uncommon among low resource programs.

When Is the Endline the End?

How long after the program ends do you want to measure the endline?

• If you wait several months or years:
  • You are more likely to capture changes that persist and may have even grown stronger with time.
  • It may be more difficult and costly to find the people or records you need to assess.
  • People may not remember at all, or memory may be distorted.
  • Other factors may have augmented or reduced the results you are trying to assess.
• If you measure immediately at the close of the program
  • You will be able to assess immediate changes (objectives) The changes may be short-lived and you will not be able to demonstrate more lasting results.
  • It may be easier and less expensive to assess.

The ideal scenario is to do both, but with limited resources try to do the best comparisons you can, at the time you can afford to do them.

Back | Back to Top