STEPS Case Example
Case 3: Sex Education, HIV Prevention with Marginalized Youth
Contents:
In a south Asian country, a community-based coalition of civil society organizations, the local schools, and government Ministries are working to mobilize resources for HIV prevention programs and more comprehensive sex education. Even though the program is on-going, they want to start monitoring and evaluating it to identify to what extent they are adequately involving marginalized populations and addressing issues such as stigma.
Background
Geeta smiled at the representative of the Ministry of Health’s suggestion that they were not doing enough to address HIV prevention and sexuality education among community youth. “We are living in an environment where these issues simply cannot be ignored,” he noted. She could not agree more. In her three years leading a community-based coalition of civil society organizations, Geeta had never felt as strongly about an issue as she did this one. “I am so happy to hear you say that,” Geeta noted, “our organizations would be very interested in pursuing such a plan.” She asked him if there were specific groups he would advise that they target, and he simply said, “All of them! My real concern, however, is that we are leaving out key segments of the population because we are not adequately addressing minority groups or dealing with issues such as stigma.”
Geeta was thrilled to go back to the members of the coalition and share the news. She gathered them together at the next coalition meeting the following week and explained that they had a mandate to increase and improve their programming in HIV/AIDS prevention and sexuality education, and that they had the backing of the MOH. The coalition members applauded; this was something everyone could get excited about!
Although they had been working for the previous three years to hone programming for youth, their HIV/AIDS prevention program was relatively new, and they were finding themselves facing more resistance from some community groups and parents than they had anticipated. To receive the endorsement of the Ministry of Health was an important voice of support, and energized the entire coalition.
Program Description
The coalition had started five years previously, when two agencies concerned with children’s rights had joined together. A year later, they had been joined by a labor union and a group working with homeless youth. The leader of the group working with homeless youth had forged a link with a group arguing for child labor laws to be enacted in Parliament and, as a result, several government ministries came on board the following year. By their fourth year the coalition, had grown to a dozen members, and recruited Geeta as their director. Geeta had been the principal at the largest school in the community, and brought several schools on board when she joined. The work was exhilarating, but Geeta had recently been growing concerned that they were simply not doing enough to address HIV/AIDS prevention, and that significant numbers of youth were not being reached at all with sexuality education or HIV testing.
To discuss this issue, she had gathered together the education director from a local youth club, the director of the clinic that served most of the in-school youth, and the leader of the Coalition for Homeless Youth. Importantly, a representative from the Ministry of Education attended this preliminary discussion, as well. These members needed no convincing that additional work with youth on comprehensive sexuality education was needed. “We’re seeing way too many girls, especially, falling through the cracks,” Rakesh, the leader of the Coalition for Homeless Youth, noted. “They are clearly in relationships where they are not on equal footing; they are ending up infected and, in some cases, also pregnant. These are conditions that are very challenging if you have a roof over your head, but some of these kids don’t even have that.” Saroj from the Ministry of Education agreed, noting that the statistics from the various schools confirmed increasing levels of STIs, including HIV, and pregnancy among in-school youth. “Even though these kids are in a very different economic situation from the youth you are working with, the story is the same for them.”
As a result of this discussion, 6 months ago the coalition had launched a comprehensive sexuality education program through the schools, at the railroad stations where many homeless and street youth lived, and in several factories with clinics that had high numbers of young employees. They were in such a rush to get the program going, however, that they had not taken the time to craft an M&E plan. When Geeta reflected on her recent conversation with the representative of the Ministry of Health, she realized that now was the time to try to create an M &E plan, and that they could do so in a way that illuminated whether they were adequately involving marginalized populations and overcoming the stigma such youth are subject to.
Getting Started
Geeta organized a coalition meeting for the following week, and invited a colleague from the local university to do a presentation on principles used in monitoring and evaluation, as well as in rights-based social justice programming. Since the coalition members were from such different backgrounds, she hoped this orientation would prove useful. The speaker was excellent and a lively discussion ensued.
Geeta quickly realized that she would need to select an M & E team that was small enough to work together efficiently. Geeta invited Rakesh from the Coalition for Homeless Youth, Mira from the All Schools Coalition, and Indu from the Transgender Action Group to join the M&E Team. She had been concerned that Indu might consider the coalition too mainstream to join, but was happy to learn that Indu readily accepted the invitation to join the Coalition, and to become a member of the M&E team. Geeta hoped that Indu could not only help the group to understand the needs of transgender youth better, but also help them to include transgender youth in the M and E process so that their voices and concerns would ultimately shape any new program interventions.
Indu suggested that they invite Nanda an anthropology graduate student who knew the communities of youth living on the streets, and the special challenges of those with minority sexual orientations and gender identities. Nanda had developed trust among this community, and could talk with them about what kind of outreach services they would feel most comfortable with. Nanda agreed that she could come to meetings when her schedule permitted, and especially to help the group identify gaps in their outreach strategies, and even with some qualitative data collection if needed.
At their first meeting, they realized that they needed to figure out who the key stakeholders were in addition to their own coalition members and the MoH, and brainstormed using the “stakeholder selection tool” to help them decide. Having examined a variety of options, they chose to focus their attention on some individuals who were well-placed to help disseminate their findings through their own institutions (such as the school board, parliament, youth clinic, etc) as well as a few who were “riskier” choices in that they may not believe in this work from the outset. The group agreed that these would be important individuals who might challenge their thinking but also ultimately become their supporters, such as a very outspoken but conservative parent. They also included youth, themselves. (Please see bolded names below for those the Team chose as stakeholders.)
 Stakeholder Selection Worksheet
Download to view or print: Word, PDF
The Team next worked on coming to agreement on their evaluation questions. Since there was so much interest at the MoH in reaching marginalized kids especially homeless and street youth; and lesbian, gay, bisexual and transgender (LGBT) youth, and the Team wholeheartedly shared this interest, they agreed that the question of whether or not they were reaching this sector would be one of the primary purposes of the evaluation. But they realized that first they must ascertain whether the sexuality education they were doing was appropriate for such groups and comprehensive in its approach to HIV prevention, and then they had to learn if they were really reaching the groups who are most marginalized.
They decided their evaluation questions were:
1. Was the sexuality education appropriate for harder-to-reach, marginalized populations they had hoped to reach?
2. How well received was the sexuality education they were implementing, especially by those priority groups?
3. What changes did the sexuality education produce, and were those changes greater for different kinds of young people?
Theory of Change and Causal Pathway
Since several members of the M&E Team had also participated previously in the planning of the coalition’s Comprehensive Sexuality Education (CSE) program, they thought they could skip over thinking through their understanding of the problem. But Geeta pointed out that they had never developed a Theory of Change or Causal Pathway and that they needed to review their understanding of the factors that influence their SRHR problem in a way that reflected the rights-based social justice programming and incorporated the experience of the other Team members and their stakeholder’s advisory committee. Geeta feared they may have left some major gaps out when they had planned and launched their CSE program 6 months ago. What could they do now about that? This fear made her nervous about the stakeholder meeting they were planning but excited about learning more about how to make their program even more effective.
They ended up having to hold two different stakeholder meetings to accommodate everyone’s schedule. But in each one they went through the exercise “Understanding the SRHR Problem.” They explored the factors that influence the sexual health of vulnerable youth and realized they had never actually involved such youth in their planning, nor had they focused enough on rights issues or even on cultural and psychological dimensions. Nanda helped them reflect. For a kid living on the street in a big city who is thinking about what he or she is going to eat and where to sleep, what does the future mean? Does he or she think about it? Does a condom campaign based on not acquiring a certain infection in the future make any sense? Does he or she care about the future? If the outreach is sex-segregated and a boy identifies as female, but looks male, which session does he attend?
But one of the most surprising outcomes from these meetings was a realization that negative attitudes and stigma against vulnerable youth, but especially the LGBT groups, were still very widespread within the coalition. Geeta and the M&E Team realized they would need to collect some information on this to better understand their findings and help create a solid foundation for refining the program in the future.
When the program began they had not constructed a causal pathway so they found the process of creating one now quite helpful. When doing their “logic check,” about whether their immediate results would actually bring about their intermediate results they realized from their lively stakeholder meetings that a number of changes had been left out from the original planning of the program. Just because young people had heard the educational messages and even learned new skills, they would still need to be to be motivated to protect themselves (and feel that life is worth living) and not be exposed to violent, coercive or transactional sex in their often very dangerous settings, and to find less dangerous ways to support themselves and survive. And then there was the question of whether or not services stigmatized them, and even the concern about whether or not they were engaging in intravenous drug use.
The Team realized that the program they were evaluating had not contemplated all of these aspects. For now they would sketch out the Causal Pathway of the program as it was implemented, put in brackets the main intermediate objectives that had not been contemplated initially (so they wouldn’t forget them but not be committed to achieving them), and collect some additional information relevant to them during the evaluation.
Goal
In the discussion about the program’s Goal everyone agreed their long-term goal was to prevent HIV and other STI infections among the most vulnerable young people in the region which the Coalition covered, the entire western side of the city. But there was considerable discussion about what ages were included in the term “young people”. They finally accepted Rakesh’s point that many homeless kids as young as 12 may already be engaging in sex, so they decided the evaluation had to include them in the goal statement as well as kids up to 20 years of age. Although the evaluation was not going to measure the goal (since that would require resources far beyond those of the Coalition), they were glad they had clarified what population they were working towards serving.
Objectives
When they came to defining the objectives they realized they had proposed to “increase the % of vulnerable youth with knowledge to protect themselves from HIV”. But when learning about SMART objectives, they realized they would have to break this down into the components of what that knowledge should be about. Checking into some of the literature that Nanda brought to the discussion, they decided to focus the objective on: knowledge of the causes of HIV, how to use a condom, where to get condoms, and whether or not they felt confident that they could obtain condoms and use them.
They also realized that the bracketed objectives included some pretty complex concepts –such as stigma and empowerment - that would mean very different things in different places and for different populations. If they were going to measure them, they would need to break them down into their components and find out what they “looked like” among the vulnerable youth populations they were serving. For now, Nanda suggested that during the evaluation, they collect qualitative data on the meaning of these concepts to inform the other findings and later, if they wanted to, they could use that information to develop more structured questions.
Activities
The Team’s discussion around program activities was not as simple as Geeta had imagined it would be. Fortunately they found a document that described each of the three main parts of the CSE program. However, the discussion became intense around the fact that their evaluation questions were much more focused on vulnerable youth, who were not as often in school as out of school. Since their resources were so limited, some Team members thought they should only evaluate the program activities that took place in the railroad station and in the factories. Mira certainly didn’t agree since she represented the All Schools Coalition; she argued that there were plenty of vulnerable young people in the schools who might be or have even out been out on the streets, and she expected that some in-school youth had LGBT orientation even if they didn’t make this public. So the Team agreed they would evaluate all three CSE activities, even if they would have to focus on fewer units in each category to complete the evaluation. Mira even offered to provide the Team with 2 or 3 students who worked with her as interns to help with data collection.
Indicators
As the Team began to talk through indicators they reviewed the worksheet on developing good indicators. They knew that the program kept good records of numbers of participants and even did some pre and post-test questionnaires, though these had never been reviewed nor analyzed thus far. So they felt they would be able to capture fairly good numerical indicators of numbers of participants reached, sessions held, and even what people had learned and their intended behavior. But the Team realized they had lots of why questions they wanted to answer and felt they should invest time and energy in exploring uncharted territory! Of course having Nanda on their Team helped give them more confidence that they could focus on some key non-numerical indicators such as:
- Why some of the vulnerable youth did not use condoms consistently
- How they conceived of HIV and of the feasibility of protecting themselves from it
- How they were or were not able to use the knowledge and skills they had learned in the courses
- How they viewed the courses’ relevance to their lives
Nanda showed them how to record the number of respondents reporting certain findings, and also how they could explore themes, and even look for different typologies, among the young people to identify kinds of vulnerable youth for whom the program could be expected to be more or less effective.
Data Needed
When the Team came up against the need to decide what comparisons to make in order to show change, they were sorry indeed that they had not established a baseline from the beginning of the program. Nanda had some interesting findings from one of the areas they worked in that could give a fairly good point of comparison. But they decided to ask the young people themselves what changes they perceived in their own behavior. The factories that had clinics could certainly give them information about numbers of youth served before the program which might give a good idea of increased interest in using condoms. Mira thought it would be a good idea to compare the schools in which the program had been working with schools in the southern part of town, which were quite similar but had not been exposed to the coalition’s program. She thought that some of the school directors there would be very interested in applying some of the evaluation instruments which would give them good ideas for their own programs while helping the coalition see if their CSE had made a difference.
And the entire Team thought that all the data they would collect could serve as a sort of mid-way mark to help them compare with similar measures later on; sort of as a baseline for their future work!
In most cases, the Team thought it was pretty straightforward who they should interview to obtain the data they needed and what records they would review. But Nanda urged them to also try to do some direct observations so they could check if the content of the actual sex education provided in the schools was of high quality. She was aware that they really needed to make sure that some of the negative attitudes some teachers had did not come through when they were giving the classes.
The Team decided that they would try to have more than one data source for the indicators that were particularly important and especially sensitive. In the case of the school program, they decided to obtain brief exit surveys with students leaving the CSE classes, some direct observations in sex ed classes, and the teachers’ self-reports about how their own participation in the training and in providing the CSE made them feel and had impacted their own lives.
Collecting Data
Since the Team clearly needed to collect quantitative and qualitative data from diverse data sources they would need to design different sets of data collection instruments. Nanda agreed to take charge of designing observation checklists and interview scripts and train a group of undergraduate students to collect the data on the observations and in-depth interviews. Nanda also created some simple pre-and post-education session forms to try to detect if there were solid gains in knowledge, realizing that this information might be especially helpful to Geeta when she goes to talk with the Ministries of Education and Health about securing more resources for an outreach program.
Rakesh pointed out that who and when they interviewed homeless and street youth would make a big difference so he wanted to make sure none of the students had the kind of negative attitudes the kids would be very sensitive to. His point was certainly a very important one as they learned when they pilot tested the data collection plan. When they reviewed the student’s field notes and met with them, they realized there was no sense trying to interview homeless youth who live at the railroad stations in the morning, since this was often a time when young people who have access to drugs in that environment get high. Later, when they sober and clean up and are hungry, and interviewers could bring them some food, they were fully capable of answering questions. Thus they had to modify their data collection plan (since the students were mainly available in the mornings) and recruited other interviewers and observers from the local management institute who could collect data in the afternoons and evenings. They also got permission to use an abandoned railroad car as a place to conduct interviews where the young people felt safe, so that there was no risk of others overhearing what a young person is saying. By being able to guarantee privacy and safety, they hoped the findings would be more accurate.
Analyzing Data
Nanda worked with the students to analyze the qualitative data as they come in. She was surprised to find pervasive reports of negative attitudes and stigma in the data collected from the young people, both in the school program group and the comparison school. When she brought this to Geeta’s attention and they discussed it with the broader Team, they decided it would be important to explore more closely exactly what the teachers’ attitudes are to vulnerable young people, and why the CSE training had not made a difference in the way the program teachers impart sex education. They decided to add a few more questions to explore this with the teachers they will be interviewing in the next few weeks.
They reviewed the data about number of young people who were using the services and noticed very low levels of repeat use. They discussed this with some of the peer educators and learned that these young people often use different names in order not to be “tracked” by authorities, especially if they are in a specific area for a week or more before moving on. So, although the data collector may log in someone as a “new client” if his or her name does not appear in the data base already, in fact, it may be the same person who used the services the day before, but just called herself a different name. Having realized this, the Team added a line to the survey question that asked clients specifically whether they had used the services before (and, if so, how many times), rather than asking them to share their first name, which they realized they did not need.
Nanda helped them review the qualitative data carefully to derive typologies from the narratives. They found that the homeless youth who they found at the railway station fell into three types with regards to how they felt about the program, and what impact the program had on their perceived and self-reported ability to get and negotiate condom use. The three groups were:
- The program was most effective in all aspects with young people who were totally alone, from another town, and/or not well connected to the community of homeless youth, and were lacking almost totally in social support. The program’s outreach staff became their first consistent social ties and support. However, these youth were also the hardest to get into the program.
- In terms of getting youth to use condoms, the program was most successful with those who reported having ways to earn money and get food other than sex and/or had a close friend who died of AIDS. Those who had been very close to someone who died of AIDS no longer felt that they were invincible. Also, for those who fell into both of these categories, the impact was especially strong because they tended to have more of an ability to think of the future.
- The group with whom the program did not seem to fit was those young people who were active drug users. They showed very sporadic attendance and although they were interested they were harder to get to the program. Clearly changes were needed to meet their needs and to develop an effective system of referrals to drug treatment programs that serve the same region.
When the Team and the data collectors sat down to interpret their findings they noticed something strange about teachers attitudes and comfort level with teaching the CSE content. Scores were generally fairly even, except in one school where there was a preponderance of negative reactions. The teachers there seemed to feel very uncomfortable teaching the information, and the students reported that the classes were boring. To find out what was going on, they decided to talk with some of the teachers from that school. Almost all of the teachers there were single and fairly young, but when they found themselves in front of the youngsters, they just didn’t feel comfortable admitting that they knew about such topics! This was a surprise that would need to be considered by the program staff in the future.
Using Findings
As the M& E Team worked with the data collectors and Nanda to organize their findings to present to coalition members and stakeholders, Geeta began to worry about the one very conservative parent among the group who might present strong objections to the data on sexual practices and attitudes and their focus on LGBT youth. To prepare for this possibility, the team decided to present the findings to the Coalition as a sort of “practice run”, and Geeta even asked one of her colleagues to play the “devil’s advocate,” to see if they could craft answers that might satisfy those with different perspectives. This actually turned into quite an amusing role play, with the person designated to object finally being won over when confronted with data about the numbers of youth dying at a nearby facility, information shared by an exasperated coalition member who said her clinic had cared for “far more young adults with HIV than she can count” in the last year. “Surely you realize that we are talking about an epidemic here,” she noted. “This is not about sitting around talking about IF our young people are having sexual relations or using shared needles. It’s about trying not to lose an entire generation.”
When they presented the findings to external audiences, given the intense emotional quality of some of their data, the Team thought it would be more impressive if they linked the numbers with the voices of the young people involved in the program. So whenever they presented figures about how many young people they had reached, they also presented testimonies about how important the CSE experience had been, such as the ones shown here.
“After talking with the counselors, I began to see that there was a choice. Before I thought that there was none. Now, I always demand that my partner uses a condom, or I run away.”
Girl, 15, living at the railroad station
“I am not happy to be an addict, but I am happy to know that the condoms I use now may keep me alive one more day.”
Boy, 17, living on the street
“Before I talked to one of the educators, I just thought that there was no hope…My mom is already dead, two of my friends are almost dead, I can’t find my littlest brother…and I just thought, ‘who cares about us anyway?’ Now I feel like someone does care, and I care enough to use condoms because I just might be around tomorrow and, who knows, maybe I will even end up married.”
Boy, 16, living on the street
“I always thought that this disease [HIV/AIDS] didn’t have anything to do with me, that it just had to do with poor, dirty people….but now I know that’s not true. Now, I have to educate my friends.”
Girl, 14, in-school
Next Geeta worked with the Team and additional members of the coalition, including the stakeholders, to plan the next program cycle, this time from the beginning with M&E. They realized that they had a number of unanswered questions, but the conversation with the Ministries of Health and Education went well, and they now had some additional funds in-pocket to do more and better outreach. They also felt like they had the beginning of the information they would need to seek funds from other donors, so Geeta recruited a young university student to help draft a few proposals.
From youth representatives of marginalized groups, they have learned some important lessons about the language in some of their education sessions, language that Geeta had not noticed was heterosexist, and potentially alienating to youth who defined their orientations differently. As a result, she formed a committee to review their training materials for appropriate language, and make sure that it was inclusive. She asked Indu for help, as well, to make sure that the training materials included appropriate language specific to LGBT youth. As part of this conversation, Geeta brought up the idea of Indu doing some values clarification on LGBT issues with the Coalition members, and Indu was enthusiastic about the idea, so they built it into the work plan for next year.
Although she was sure that there were many more changes to make, given all that they had learned in the past 12 months, Geeta was most excited to have gotten the Coalition on board with the idea of monitoring and evaluating their work regularly. Now that the members saw it as an important tool, they were optimistic not only about the programmatic work they could do, but also about using the data gathered to do more of the kind of work they wanted to do in the future with support from other funders.
Logical Framework with Examples:
Download to view and print: Word; PDF
|
Word
PDF
