Step 2: Developing Locally Relevant Indicators

You have developed immediate and intermediate objectives and now you need to think about how you will know if you have achieved them. Indicators are what you actually collect information on to see if the impact you have been aiming for has occurred. It is very difficult to know if your program actually CAUSED the change you see. But we use indicators to infer that the intervention (program) was a part of what caused the change we measure with the indicator. You will develop at least one indicator for each of your objectives.

Task 1: What do you see? Find out what your objective looks like in your program’s setting.

Now it’s time to select, adapt or create your evaluation indicators. The more specific, descriptive, and tied to your program your objectives are the easier it will be to develop and come to some agreement with your team about what an appropriate indicator is. SRHR programs that incorporate a rights-based social justice perspective address aspects of complex social concepts, such as gender equity and reproductive rights. These concepts can be difficult to "see" and measure. Therefore, you need think about what the components of these complex concepts are, and what they look like in your program’s setting. For indicators to do their job of telling you how well your program works, they need to be valid (it measures what it is supposed to), reliable (the meaning is stable across people, place and time), and feasible (collecting the data for the indicator will be possible and affordable). You will need at least one indicator for each of your objectives.

Use Pre-existing Indicators When Possible

Regardless of whether or not you have already developed some indicators you can benefit from looking at the indicators that others have developed. There is no need to re-invent the wheel. And it is often preferable not to do so. Indicators that have been developed by others will often have been tested for reliability and validity. Ask colleagues agencies similar to yours, and at universities, if they have already developed locally meaningful indicators. However, you need to be sure that they are really relevant to your program’s context or they will not provide a fair measure of your program’s impact. You will probably need to adapt the indicators you have found to make sure they work with your target population and in your program’s context.

Please see Indicators and Instruments: Selected References for existing indicators which have been used in sexual and reproductive health and rights programs around the world. In addition, the references in this module have links to some very useful pre-existing indicators.

Tips: Examples: Indicators for Complex Social Concepts

Tips: MEASURE Evaluation Publications, including Compendium of Indicators:
http://www.cpc.unc.edu/measure/publications?searchterm=ms-02-06

Example: Women’s Empowerment

Women’s empowerment is a key aspect to programs that use a rights-based social justice perspective on their work. One aspect of women’s empowerment is decision making in the home.

Here is a breakdown of how you might adapt and make more specific the indicator “decision making in the home.”

Download to view or print: Word, PDF

It is useful to consider indicators that have been developed for, and agreed upon by, a broad international audience. For example, this list of five dimensions of women’s decision making in the home has been found to be an internationally important and useful set of indicators. This list is considered an index of indicators, together they are a good measure of women’s empowerment (new citation- Bertrand, Jane T. and Gabriela Escudero. 2002. Compendium of Indicators for Evaluating Reproductive Health Programs. Volume One: Overview, Indicators that Crosscut Programmatic Areas. MEASURE Evaluation Manual Series, No. 6)

• Determining own health care
• Making large household purchases
• Making daily household purchases
• Visiting family or relatives
• Deciding what to prepare for daily meals

Examples: Tips: Indicators That Measure Components of Complex Social Concepts

Task 2: What would success look like in your setting?

Now that you have thought about what your objective looks like in your program’s setting, you need to decide what kind of change would be positive. In other words, if your program were successful and had the impact that you had intended (meets the objective that you had aimed for) what would that look like? How could an existing indicator be modified to reflect success in your setting?

For example, if women’s empowerment is an aspect of one of your objectives, and your team has decided that women’s perceptions of their role in household decision making is an indicator of empowerment, then your team needs to decide what success would be with regards to household decision making in the context of participants’ lives. If you had reached your objective would that be indicated by a particular type of decision- like the ability to decide when to go to the doctor? Is that a realistic expectation in your program setting? Or is that a change that could only occur after a much more intense program experience or length of time?

---

Group Exercise: What would success look like?

Whether or not you do the below exercise as a group (ideally with program staff and key stakeholders who are knowledgeable of the community and the program), the exercise should help guide your development of your locally relevant indicators.

Consider involving:

• Partner agencies that work in similar populations.
• Beneficiaries and members of the priority population, especially marginalized and vulnerable groups. If you involve community members who are not aware of these concepts, you may first have to teach them about the concept itself in order to get their perceptions of how it looks in their lives. Even though this may take a bit more time, it is often well worth the effort.
• Stakeholders and other knowledgeable colleagues.
• Local researchers, particularly anthropologists, sociologists, or public health specialists.

Materials:
-Worksheet A: Finding Out What Success Looks Like (half of worksheet also used in mod 3)
-Your Causal pathway

How to proceed:

• Refer to your program objectives, the complex social concepts that are part of your objectives, and the indicators that you or others have developed (if you have some). If you filled out the objectives section of Worksheet A (columns 1, 2, 3) then you will want to refer to the relevant components that you listed in column 3.
• For each component of your objective, answer the following questions:
  1. How would you know when you see success? In other words, what would it look like in the community if the program’s objective were met?
  2. How would you know when you haven’t achieved your objective? In other words, what would it look like if the objective were not met?
  3. List these suggestions in Worksheet A, column 4. Circle the ones that are most important meaning you would not consider the program successful without changes in those specific items.
  4. What would the range of behaviors, observations or events be in the program’s setting?
  5. How much agreement is there among the participants on different indicators they have suggested and on what high vs. low levels would look like in the program’s setting? See:Tips: Why Is It Important to Assess Agreement on Possible Indicators?
  6. List those indicators, with their high and low levels, on which the participants agreed. If you have held separate meetings with different kinds of stakeholders and colleagues, compare the results of those meetings and look for indicators on which there was agreement across the groups. Make sure the voices of more vulnerable and marginalized are heard and taken into account. List the “winners” in Column 5.Tips: Why Is It Important to Assess Agreement on Possible Indicators?
  7. Make the appropriate adjustments if you are using pre-existing indicators.

This worksheet can help you pinpoint what success would look like in your program setting.

Worksheet A: Indicators (Module 4)
Finding out What Success Looks Like

---

Task 3: Your Evaluation Needs and Your Indicators: Numerical, non-numerical indicators, or both.

You have thought about what success looks like in your program context- in other words what the indicators are. Now you need to think about how your indicators address the purpose of your evaluation. You should think about your evaluation questions and the audiences for your evaluation, so that you can develop and choose indicators that best address your evaluation needs. You will need to think about the kind of information, or data, you will need to collect for each indicator. Some measures (or indicators) are expressed as numbers (numerical) and others as words (non-numerical). Both numerical and non-numerical indicators have their roles and purposes in program evaluation. Making the choice between numerical and non-numerical indicators is shaped by your evaluation questions and audiences.

Review your main evaluation questions, if they are more focused on the scope, degree and quantity of change, you should use numerical indicators. If they focus on program process (why and how) and what things mean to the population, then use non-numerical indicators. The audience for your evaluation- who will be interested in the results- should also be considered. Some audiences may only be interested in how many participants learned something (numerical), while others may be more interested in questions that have to do with how a program works, or the words and stories of participants (non-numerical). It is likely to be the case that your evaluation questions will include both types of questions, and you will want to use both kinds of indicators.

The table below explains the kinds of questions each type of indicator can address and gives some examples of numerical and non-numerical indicators as they relate to sexual and reproductive health and rights program work. (How can we have this table without taking up so much room-don’t like the way it looks)

Tips: Types of Indicators: Numerical and Non-Numerical

Tips: How Do Numerical and Non-Numerical Indicators Relate to Qualitative and Quantitative Data?

Developing your numerical indicators:
Deciding What Numerators and Denominators to Use

To calculate percentages, proportions, and rates for numerical indicators and to know the size of the data set on which your non-numerical indicators are based, you have to determine:

• the actual number of who/that exhibit a particular trait (such as the number of women who have one or more sexual partners who talked with their sexual partner(s) about condoms in the last month or the number of healthcare facilities in the region that offer HIV testing). This is the numerator.
• the total number of possible events or phenomenon (such as the total number of women participating in your program who have one or more sexual partners or the total number of healthcare facilities in the region). This is the denominator.

The denominator (or total group you are selecting from) you choose should:

• include only units (e.g. people, clinics, households) that could be impacted by your program
• be relevant to the event or phenomenon you are measuring
• be meaningful in the program’s local context
• be relevant to your programming decisions

* basically the numerator and the denominator are two groups of people, events, or documents that you compare.

When you put the numerator over the denominator, you create a fraction that enables you to determine percentages, proportions and other rates. Tips: Why are percents so useful?
With small data sets, you will report how many people, events, or documents showed a certain finding (the numerator) "out of" the total you had studied (the denominator). For example, 7 TV series observed out of 10 showed women being beaten, threatened or exploited by their sexual partner.

Example: If you simply count the number of women who talked with their sexual partner(s) about condoms in the last month, and find that the number is 37, it is difficult to know if 37 is a significant achievement. However, if you know that the total number of women participating in your program is 50, then you know that 74% percent of the women participating in your program talked about condoms with their sexual partner(s) in the last month (in other words, 37 out of 50 women discussed condoms, or 37/50 = .74; (.74x100 = 74%).
If the total number of women participating in your program is 100, then only 37% or about one-third discussed condoms in the last month. The numerator remains the same (37) but the denominator (50 or 100 in these cases) provides information on the scope of the phenomenon.
Different denominators can have dramatic effects on the results.

Make sure:
" both the numerator and denominator are clear and specific
" you use a relevant denominator to give meaning to the numerator
" use denominators that reflect the reality in the local context

" to only include the number of people who actually answered the question in the denominator, if you are reporting a percent from an interview or survey in which all respondents were asked the same questions

Worksheet: Selecting the Most Relevant Denominator- Program Examples

Worksheet: Selecting the Most Relevant Denominator

Developing non-numerical indicators:

Although you will not be calculating percentages, you will need to report how many cases (people, events, or documents) showed a certain finding out of the total number that had been studied. This can give a sense of how common or typical the findings are and their implications for the program.

• You also need to decide in what form the indicator will be presented. They are:
  
  • Direct quotes or paraphrase or summaries
  • Themes identified from reading transcripts
  • Typologies of cases
  • Coding of pre-determined categories generated by: previous experience, the literature, and quantitative/numerical findings.

Task 4: Factors that Can Influence Participant Outcomes

It is important to keep in mind the factors that influence participant outcomes. Often these factors are demographic variables such as: gender, age, ethnic group/race, and socio-economic status. These factors may also have to do with factors that are relevant to program participation and impact in some other way.

For example, the experience of the participant before they got to your program.

It is critical for you to think about which factors (or variables) you think are likely to shape how your program impacts participants. That information might be important to include in our indicators.

For example, if you are interested in your program’s impact on women’s empowerment, which in your program setting can be indicated in part by the ability to go to the health care clinic unaccompanied, then it is important to consider the demographic variables as well as the locally relevant factors that would impact this, such as woman’s age, her marital status, and possibly her ethnic group/race. (Measuring Program Outcomes: A Practical Approach. 1996. United Way of America.) (this is a new citation- add to reference list at end of this module, and at end of toolkit if there is master list)

Types of information that you should consider:

1. demographic information
2. participant’s level on issue the program targets (e.g. experience with violence, or previous sexual experience)
3. any other factors that you think are important to how the program impacts the participant

Task 5: Write your Indicators

Write your indicators, once you have a clear idea of:

1. the complex concept each objective addresses
2. what it looks like in the program setting and what success for that indicator would look like
3. what your main evaluation questions and audiences are and the type of indicator that would be best (numerical and/or non- numerical) to use
4. other factors that can influence the program’s impact on participants

Worksheet: Selecting the Most Relevant Indicator