STEPS Case Examples
Case 2: STI, HIV and Pregnancy Prevention with Youth
Contents:
Based in a family planning program in the Latin America and the Caribbean region, this program addresses STI and HIV prevention and provides basic family planning services. The evaluation team will be working on a new program designed to implement youth-friendly services and gender equity promotion activities.
Background
In a semi-urban Latin American community, a family planning clinic serving rural communities throughout the surrounding hills, is struggling with a new problem. Since its formation they have provided contraception and basic reproductive health education, mainly to married women aged 25 to 40. But recently, more and more young girls, unmarried and as young as 15 or 16 years of age, were coming in with sexually transmitted infections (STIs) and unplanned pregnancies.
During a staff meeting, Blanca, the clinic’s head nurse, presents the data she has from a quick check of clinic records that confirm this impression. Not surprisingly, other staff members have observed the rise in the number of adolescents being seen with such problems. They are concerned because of the dangerous consequences that can result from some STIs if they are left untreated and the burden of unplanned pregnancies. While they agree that they would like to do something, they don’t feel they understand what is going on or how to address the problem. After treating such a patient, the nurses give a standard warning about the importance of avoiding STIs or unwanted pregnancy, but don’t feel comfortable addressing sexuality or gender issues or even recommending condoms. In some cases, some girls come back to be treated for the same thing. Clearly, the warnings are not working!
Dr. Hernandez, the clinic director, is quite receptive to a group of younger staff members who propose trying to understand the problem better and mounting some kind of effort to prevent girls from getting infected or pregnant. Since she firmly believes in using data to support the clinic’s work, she proposes a needs assessment, to understand the problem and plan their program according to the findings, and then carefully observe how well things proceed. She expects that they should be able to find some support for such work and would have a better chance if they have good data. They know they are working in a potentially hostile environment and will need to have solid evidence there is a need for the program they institute and that it is actually working.
Getting Started
To get them started Dr. Hernandez, asks Blanca to head up a small evaluation Team with some staff who were interested in the new program including the staff psychologist, Irene, and the social work coordinator, Jose. Dr. Hernandez will meet with the Team regularly so she can support and be part of the effort.
Since the clinic works closely with its citizen advisory board, Dr. Hernandez suggests that they meet with members of that board, as key stakeholders, to get their input and keep them informed of things as they go along. She knows they will be very important in rallying support for any new projects the staff may propose. In addition the Team decides that they need to involve young people from the community, who are the stakeholders most directly affected, so they invite Mirabel, who was a patient at the clinic, and her boyfriend Javier.
Now that they have the green light, the Team meets and realizes they need to be very clear about what the evaluation questions really are, or else they could get lost with all the questions they have. Since the evaluation’s primary audience is the agency staff, they decide to meet with a group formed by a member from each department, to find out what the staff want to know. The brainstorm session is very productive, and they end up realizing that there are three main questions they want to answer through the needs assessment: Why are more teens coming to the clinic with SRH problems?, What factors place them at risk of having such problems?, and What can the clinic do to prevent such problems? It’s a big order, but they are excited about answering these questions and then figuring out what kind of program they can propose. And once they are clearer about how to intervene, they know they will have other evaluation questions to answer.
Identifying the Problem
Jose, who just returned from a course on M&E, shows them a list of rights based social justice programming principles which he says will help them think about the areas they should explore to better understand the SRHR problem they have chosen to address. Irene is very excited by the discussion of the rights-based social justice considerations. She has begun reading about youth development programs and sees how significant these principles are to the kinds of programs most relevant in their context, especially the sections on rights, sexuality, and gender. She is also aware they should explore influences at the broader societal and community levels, as well as looking at influences within the family and the individual.
To undertake the needs assessment, they decide to create short surveys that key informants could fill out individually and also hold group discussions with three key populations: young people between 14 and 20 years of age, parents of young people, and service providers. They wanted to include other important groups, such as government officials, but did not have the resources or time, and decided that the first three groups were the most important. For each of these groups they focused on issues related to young people’s rights and responsibilities, sexuality and the meaning of sexual relations, how gender roles are defined among the young people and how they influence risks of SRH problems, and what institutional constraints there were for prevention programs in terms of laws, policies, attitudes of service providers and policy makers as well as opinion leaders, etc.
Two months later, at a meeting with their stakeholders to explore how everyone understood the factors contributing to STIs and unwanted pregnancy among teens, they started off presenting the findings of the needs assessment. The main findings showed:
- Many young people were indeed having sexual relations and taking major risks of pregnancies and STIs including HIV by not using any kind of protection or using it only sporadically.
- Macho values predominated and made it very hard for young boys and girls to talk openly about sex, and even harder to plan how to prevent STIs and pregnancies, or even if and when they were going to have sexual relations. Both boys and girls admitted that they thought about sex but were ashamed to talk about it openly, especially to someone of the opposite sex or someone older than themselves.
- There were major problems in terms of the attitudes of service providers who wanted to provide education, but only blushed when asked about sex, admitted they needed education themselves before they could educate anyone else, Also some of the men had fairly macho ideas themselves.
- Parents reacted almost violently and opposed other people talking about “those topics” to their children who they were afraid would be led astray by too much information about sex.
- One very interesting finding from the needs assessment was that the girls and boys who said they were active in sports and other hobbies and activities, were the ones who were not having sexual relations or had started at an older age.
Theory of Change and Causal Pathway
The discussion was very lively, and they did a few exercises to further explore the underlying causes of STIs and unplanned pregnancy among young people at each of the four levels. It was also very revealing when they asked questions based on each of the rights-based social justice dimensions. This was a real eye-opener! They created a theory of change for how to impact the problem (see the Theory of Change column that has been added to the log frame in the case study).
With all the information they had collected, they needed to put things in order so they could decide what program to implement and how to monitor and evaluate it. Jose really helped them go through this process using a worksheet that enabled them to move towards constructing a Causal Pathway that would help them work with program staff to plan the new program. They had some difficulty separating out what belonged in what level, but realized that it didn’t matter much. It was important to think beyond the walls of their clinic to broader factors that influence the kids’ lives. Here is how they filled out the worksheet.
 Constructing a Causal Pathway with Different Levels of Influence
Download to view and print:  Word;  PDF
Wow! This exercise taught them that some important factors were clearly out of reach of their clinic programs, but it was good to be aware of their importance. They put check marks next to the items they thought they could be responsible for working on and towards.
Maybe later they could join forces with other groups to work towards broader legal and other societal changes and school programs.
But it was clear to them that whatever program they decided to implement, they had to find a way to address how young people related to one another and how to help them form more equitable gender roles and values. Dr. Hernandez was very impressed with the needs assessment findings and asked the Team to present them to the community advisory board members, who were also quite excited by them. She gave the green light for the Team to work with program directors to design a new approach to solve these problems. And she wanted to make sure that they planned a good monitoring and evaluation system to accompany the new program plan from the beginning.
One of the members of the community advisory board donated a vacant storage house adjacent to the clinic for them to use for the new program. They wanted to create a safe space for young people to come in for free sexuality education classes in which they could ask honest questions without embarrassment. They knew they would need to provide education and support to the staff first, since they had some of the same prejudices and misinformation as the young people. They weren’t sure how to address the issue of parents so they decided to leave that for later.
Since there was no additional funding for classes, clinic staff were asked to volunteer a few hours every week to staff the center, and in exchange, they received individual counseling and education about their own problems and needs in the area of sexuality and family relations. They would also earn one additional vacation day for every five sessions they worked at the new Youth Center.
Once the M&E Team was clear about the activities that would be implemented and which they should evaluate, they filled in the Causal Pathway on the log frame that Jose showed them. As the discussions progressed they went back to this form repeatedly and adjusted it as things became clearer.
Objectives
Now that the program was really going to start, the Team needed to develop a clear M&E Plan. They knew that they would need good solid findings to present to potential private donors and local politicians who were far more conservative than the community advisory board. Their stakeholders now included a more formal youth advisory board that Blanca had insisted on when she was named director of the new Youth Center. Since they were planning their M&E plan from the outset, their evaluation questions would assess if they had achieved their objectives and if there were things that could make the program even better.
Together with the program directors, they carefully defined the program’s goal and objectives in the causal pathway/logical framework. When trying to define the objectives, the M&E Team realized that the concept of gender equitable relationships was quite complex. They decided to work with Mirabel and Javier and other members of the youth advisory board and a few other stakeholders to find out which components were most salient and how they would look in their local context. They used a dual worksheet to decide on which measurable objectives to work towards, and what indicators to use to measure those objectives.
First they reviewed some of the literature to find out how gender equity was defined in other studies. Then they discussed which of these were the most important in the context of their own program. The young people made it clear that it was most important how couples decided whether or not they would spend their free time together, if they even talked about contraception and sex, if the boy had any respect for the girl as an individual who had a right to her own development, and how much violence was acceptable in the relationship.
Activities
Two somewhat separate sets of activities were being planned by the broader program staff: one that would work closely with sister agencies to focus directly on promoting gender equity and others that would take into account gender but would focus more directly on providing a safe space for young people to learn about sexuality and access services and support. The Team hoped they could evaluate both of these strategies, but agreed that they would only try to measure immediate results – at least for now. But they felt that by thinking through the kind of intermediate results they hoped their program would lead to, they would have a stronger case for arguing that they needed more funding, time and staff to continue the program (if it looked promising), and for evaluating some of the intermediate results in the future.
Indicators
When deciding on their indicators, the Team decided that they had both “how many” and “why” questions to answer. For example, they would need to show how many young people came to the program, that the numbers were increasing as word spread about the program, how many services were being provided and how many different kinds of problems were being addressed. But they realized that many of their questions would require qualitative data collection to better understand the changes the program might be producing, and why they were taking place or not. Some of these might be coded for numerical indicators, but they felt that their audiences would be well served by hearing from the participants in their own words.
Worksheet: Measuring Complex Social Concepts | Finding out What Success Looks Like
Download to view and print: Word; PDF
Data Needed
Since they had planned the M&E system from the start of the program, they were able to design a baseline and then compare results once the program had been in operation for at least 6 months and then again at the one-year mark. They thought this was pushing things a bit, but Dr. Hernandez urged them to be ready to present results within one year so she could show them off at the annual health sector convention, and they certainly wanted to see how things were looking mid-way through so they could make adjustments if need be.
The data sources to use were fairly straight forward. They would need to talk with the service providers about changes in their attitudes, confidence level and knowledge learned. And they would need to collect data from the young boys and girls when they arrived at the program, after specific events, and once they had participated for several months. It would be important to interview those youth who participated regularly as well as those whose involvement was less optimal to find out why they didn’t participate more fully.
Collecting Data
The Team decided to use a combination of short surveys applied before and after the young people and clinic staff had participated, observations of the actual program education, and some qualitative interviews with a small sample of participants to find out what they thought of the program and what could be done better. They pilot tested the instruments and made the needed adjustments. They were glad they had taken time to do this since they had forgotten that some of the young people actually had more than one partner, and so they needed to add questions about all partners considered important.
When designing their data collection plan, Blanca was concerned about biasing the findings if the Team members themselves or the service providers were the ones to collect all the data. The youth might give the kinds of answers they thought were being looked for, and the service providers themselves might even bias the results as they would want to be seen as doing a good job at the Youth Center. Jose was able to extend the agreement with his social work students, and Blanca worked to free up some time of three young nurses from other departments to be able to help out periodically with data collection.
Analyzing Data
One month into the data collection they looked at how well things were going. They realized that more young people were coming in on Mondays and Tuesdays than they had expected and very few educators were available. The kids were complaining that they couldn’t get any private time to talk to an adult those days. The Team was able to convince some of the educators to change their schedule and be available at times when more young people were showing up.
Data analysis went smoothly. They were glad they had done a ‘dry run” before starting since they had caught some important errors and corrected them ahead of time. For example, they really wanted to compare how well the clinic staff did in providing the classes if they themselves had sexual experience or not; but they had originally forgotten to include a question about sexual experience among staff who were not married as was the case for many of the nurses and educators. They had fallen into traditional gender stereotypes themselves!
Given that staff had revealed their perception of their own and the clinic’s shortcomings, and even admitted to some of their own sexual problems, it was important that the Team handle the reporting of the findings in a sensitive manner. One way to handle this, they decided, was to only analyze and report the staff data by group, so that no one person would be identified.
Now that they had the findings analyzed, they were surprised by a major inconsistency and didn’t know how to interpret the findings. They certainly had increased the number of young people in the classes, improved gender attitudes and the frequency with which people reported positive gender equitable images. They also seemed to have improved gender equity in couple relations. But boys were reporting more episodes of violence in their relations with females (girlfriends, sisters and even their mothers) after the educational and consciousness-raising program, although they clearly had far more positive attitudes and intentions about being non-violent. What was happening?
To find out they presented their findings to the youth advisory board. Through the intense discussions that ensued, they found out that in the intake interviews, boys had not listed a number of kinds of violence since they were not even aware they were examples of violence. Now, after the program, the boys who “had seen the light” felt guilty about those same kinds of behaviors which they now catalogued as inappropriate. By refining the analysis, the Team was able to show that reports of physical violence had decreased markedly while reports of verbal violence had risen in comparison with the base-line. Ironically, this was a kind of success since the boys were now defining more behaviors as violence as a reflection of the increased awareness and good intentions.
Using Findings
Now that they really had a thorough understanding of the meaning of their findings, they presented them internally to staff and stakeholders. The community advisory board was thrilled. They particularly were impressed by the evidence of changes in the intentions and attitudes of the boys since some of them had originally been skeptical that any changes could take place (thinking “boys will be boys”), and certainly didn’t think such changes could be measured. Since the Team presented the qualitative data showing that girls were requesting access to sports as a way of having fun and getting out of the house, the board decided to seek ways to provide such services. One of the members thought he could get access to a nearby football field for the Youth Center; and one of the nurses said she would love to help coach the girls in football (soccer).
Next they presented highlights of their findings to an open community meeting that the community advisory board had organized. There was a lot of debate, and even some very conservative parents felt uncomfortable about promoting open talk about sex. Fortunately, Mr. Yañez, one of the board members who the Team had purposely invited to be there, was there to defend the work of the Center. Finally the parents admitted that they actually felt a bit left out, and the meeting ended with the surprising suggestion that the clinic staff actually start sex education classes for the parents. They would need to address gender roles in such classes since the girls in the program had confided in the discussion groups, that their parents were giving them grief for acting more assertively. Also to get the new female sports program accepted, the parents would have to be convinced that sports are appropriate and beneficial for girls.
The Team and Dr. Hernandez were very pleased with the M&E process so far. It had helped them rally support from the community and even the MoH youth supervisor who came to the open meeting. Now that is was time to plan the next cycle of the clinic’s work, they would certainly rely on the findings to guide them. Dr. Hernandez particularly wanted to find ways to break through the resistance that some staff members had shown to opening positive discussion of alternative sexual orientations and gender identities, and to working with parents who they feared would never understand their focus on empowering young people to demand services and actively change their environment. Next year, since they would continue working with some of the same kids, they would try to measure some of the intermediate changes they hoped their program was instigating, and if they got the grant they had been invited to apply for from a local community development fund, maybe they would be able to do just that.
Logical Framework with Examples:
Download to print: Word; PDF
|
