Tips
Module 4: Your Indicators
Deciding How Far to Go
The level of resources you have for evaluation will determine what levels of evaluation indicators you will use:
Evaluation Indicators are directly linked to your objectives at all three levels: immediate, intermediate and final results. They provide data about:
- changes the program was expected to contribute to and
- unexpected changes that have occurred.
Keep in mind the distinction among the different levels of evaluation indicators:
- Indicators of immediate objectives reflect the changes that happen as a direct consequence of carrying out the program activities, such as: the number of adequately trained staff or people who are aware of the program or certain information through outreach efforts; the number of people who learned certain information in the program’s course; or the number of people who report positive reactions to the program’s campaign messages. (Some evaluation schemes call these process Indicators).
- Indicators of intermediate objectives reflect the changes that take place one or more step removed from your program. These are measures of what happened as a result of the immediate objectives being achieved. For example, if staff are trained by the program adequately, they will change their attitudes and behavior with clients (one step removed) and clients will feel more satisfied with the services offered (two steps removed); if a TV program focused on involving more men in “mothering” their children is seen and well-received, men might report a greater desire to engage in child care (one step removed) and actually participate more in child care after seeing the program (two steps removed)
- Indicators of final results or goals reflect the long-term changes to which your program will contribute, such as the HIV/AIDS infection rate; percent of girls who are not genitally cut, or percent of boys and girls who suffer sexual abuse. Remember, that even though you identified what the final results or goals your program is working towards, most community-based or small-scale SRHR programs will not need to develop indicators at this level. Such programs usually do not have sufficient resources, expertise or timeframe needed to affect and measure change at the goal/final results level which are quite complex and are the product of many kinds of interventions over time.
Elements Your Indicator Should Contain
A.
Provide clear and precise information on exactly what will be measured, avoiding the use of any jargon or terms that might not be widely understood. For example, if condom use will be measured, you will need to include information about when condom use will be assessed: will it be measured at last sex, or first sex, or the last three times sexual intercourse occurred, or over the past month? You will even need to make sure you are specific about what is meant by “sex”. Does it only refer to vaginal intercourse, or also include anal and oral intercourse. And what is meant by “condom use”? Does it mean every time there is sexual contact in one intimate session or only once during a session? And is the usage correct or does any kind of usage count?
B.
Specify the exact population from which the sample will be drawn, including the numerator and denominator for percentages, proportions and other rates. In addition, the indicator should identify the specific population among whom condom use will be assessed: will it be measured among women who have been heterosexually active in the past month, among heterosexually-active young people in a given school, or among truck drivers participating in your training program?
C.
Identify the type of calculation to be conducted, such as a count, percentage, rate, existence, perception, quote, paraphrase, typology, etc. Imagine you are interested in increasing condom use in your community. There are many things you could measure: the percent of people who report they always use condoms; frequency of certain kind of perceptions by men of women who carry or use condoms; the number of free condom distribution sites in the community; what “correct” condom use means for different groups of people; the meanings evoked in advertisements, etc. You’ll want to specify exactly what calculation you’ll do or how you will present the data you collected from interviews, observations, and document review.
D.
Indicate if data should be separated into categories, such as by sex, gender, age, type of respondent, location or site, etc. You will often want to know whether or not the program is equally effective with different groups, whether one group is participating differently than another, or whether different groups of participants have different perceptions/attitudes about the program. The indicator can tell you how you will look at the data by specifying categories such as age groups, sexual activity status, etc. For example, if you want to make sure that the young people who are being recruited to be peer educators in your program include youth aged 10-14 as well as older youth, or sexually active and non-sexually active young people, you might include an indicator such as: number of peer educators participating in the training program, by age and/or by sexual activity. In this case, you know that you will need to develop data collection forms that include information about each individual’s age and sexual activity. Specifying this in the indicator reminds everyone that data collection forms need to include space to record this information. For more on specifying the population, see the section below on numerators and denominators.
E.
Do not specify the direction of change being sought. The measurement will tell you whether something increased or decreased. In other words, your indicator will not say increased condom use at last sex since you will need to measure condom use to determine if it has increased, decreased, or stayed the same. Thus, your measure will be condom use at last sex.
How Do Numerical and Non-Numerical Indicators Relate to Qualitative and Quantitative Data?
- Qualitative data can be transformed into numerical indicators by counting the number of people who report certain kinds of perceptions, opinions, and meanings.
- When it is more important to maintain the actual words of the people, use non-numerical indicators such as direct quotes or paraphrased text.
- Quantitative data will always be reported as numerical indicators.
For a graphic description  Tips: Description of the relationships between the kind of data and the kind of indicators you will use
Indicators That Measure Components of Complex Social Concepts
Key Rights-Based Social Justice Considerations for Indicators
Monitoring Indicators are directly linked to your activities.
They provide data about:
- what and how many activities were conducted
- who participated or received the program’s messages
- how well those activities were carried out
- what your program is doing or did
- if the program did the activities as planned or better or worse
They do NOT tell you what effect or result the program had.
Since they should be useful for strengthening your work, if you can’t measure all your activities, choose to measure:
- new or innovative activities
- activities that have not been tested before
- activities you are not sure are being carried out correctly
- activities you want to highlight or share with others
- activities that are central to the program
What about the Goals? Or, How can you know if changes are happening at the level of your goal?
In most cases you will not be expected to collect indicators of the broad program goals (also called final results) on your own. However, other organizations such as universities, ministries of health, the United Nations, or other large-scale agencies periodically collect and publish data at the local or regional level. Far more complex and highly funded evaluations are needed to demonstrate the achievement of final results or goals.
Why Are Percents so Useful?
Percents are by far the most commonly used proportion. They translate numbers onto a scale that is familiar and standardized, and thus help us to understand the magnitude of “How much?” or “How many?”
For example, if 476 women and 341 men agreed with the statement, ‘If a woman is disrespectful to her husband, it is important that he punish her to remind her who is charge’, what would it mean? Is this a popular view, or does it characterize only a small group? Is the view held more by women than men? Even though more women than men agreed with this statement, it might be a misleading conclusion depending upon how many women and how many men participated in the survey. The interpretation changes if 1000 women (47.6%) and 1000 men (34.1%) participated in the survey, or if 1562 women (30.5%) and 547 men (62.3%) participated in the survey.
Why Is It Important to Assess Agreement on Possible Indicators?
This process of developing indicators separately and then agreeing on them actually helps you see which indicators are more valid and can be measured more reliably, i.e., people agree on them to a greater degree. Make sure you have included people in the discussion who belong to high priority groups, such as women; ethnic, racial, sexual minorities; and other vulnerable or marginalized people of relevance to the program. And make sure their voices are heard during the discussions.
|
Word
PDF
