Print Case: Word; PDF

STEPS Case Examples
Case 1: Obstetric Care

Contents:

• Background
• Getting Started
• Needs Assessment
• Theory of Change
• Objectives
• Activities
• Indicators
• Data Needed
• Collecting Data
• Analyzing Data
• Using Findings
• Logical Framework with Examples

The program is based in a small Government hospital in eastern Africa which focuses on providing emergency obstetric care. The evaluation work will help the on-going program find out why more women with complicated pregnancies do not come to the clinic early enough so that their condition can be treated. The needs assessment informs a quality of care intervention that is accompanied by a continued M&E process.

Background

In a dilapidated building Grace is trying to figure out how to make some important changes in a public sector hospital in East Africa. She has recently become the obstetric clinic director within the hospital. This facility serves an urban setting, as well as a large rural area that includes several remote villages. It is the only place in the region where women can go to receive obstetric care. As a government-funded hospital, operating in a region where most clients cannot afford to pay any fees, resources are scarce.

Grace also senses that there are a host of other problems: staff morale seems low; the infrastructure is in need of work, and there are challenges to providing privacy for clients since the examining rooms don’t have doors. The waiting area is sectioned off by a curtain, but sound travels through it easily. Staff members sometimes do not know who is responsible for what tasks within the clinic.

During a review of the obstetric program, Grace works with Mrs. Kamau from the Ministry of Health (MoH), and they notice that, according to client intake forms, there are very few women coming to the facility to deliver. Most of the women coming to the hospital do so because of complications arising when they try to deliver at home; the majority arrive when it is too late to save mothers or their babies. This raises a red flag for Mrs. Kamau and Grace since the primary reason for designating an obstetric wing is to have enough space for deliveries, whether in birthing rooms, or in the operating room designated for obstetric cases.

Getting Started

Following her conversation with Mrs. Kamau, Grace asked Alison (a local Peace Corps Volunteer who recently arrived and has some monitoring and evaluation experience) to pull together an M & E team. Grace promised to be very actively involved, but can’t head up the team, since she will be away for two months at a training early in the process. Alison quickly pulled together some key individuals for the team: Eunice, the clinic’s head nurse; Japheth, the clinic manager; and Claire, the social worker coordinator. Mpume, the hospital’s community liaison director, agreed to come to meetings whenever he could, but was too busy to be a full-fledged member of the team.

Next, the Team met and discussed who the people most interested in them succeeding (i.e., their main stakeholders) were. They were the women served by the clinic and their families, as well as those who were not coming to the facility to deliver, and the traditional birth attendants who they hoped would refer complicated cases to them. Mpume pointed out that the leaders of the communities the facility serves are also key stakeholders who should become involved in the evaluation. He offered to contact a small group of women, birth attendants and community leaders and invite them to come into town for occasional meetings. The Team was concerned that they did not have the resources to pay these people’s expenses to come to the meetings, and even more importantly, Grace insisted that if they invite them to participate, the hospital would have to commit to following up on their suggestions which would probably focus on broader community needs and outreach. She knew she could not commit to this until they had a better sense of where the problems lay and had gained support from higher up authorities to make the needed changes. She hoped that the initial evaluation report would create such interest.

In their next meeting, the Team focused on who the audience is for the evaluation, and what questions they would want answered. They agreed that the MoH officials were a primary audience, as well as hospital staff outside the emergency obstetric clinic. Another audience was public and private hospitals in other neighboring provinces and regions. The Team decided they would also present their findings to a group of traditional birth attendants and community leaders, hoping that they would become involved in helping the hospital learn from the findings at a later time. Mpume agreed this was better than not involving them at all, but felt their input should be considered from the outset in some way or other.

For now, their evaluation questions were:

1. Why do women who are in labor choose not to come to the facility or come so late?
2. What low or no-cost modifications could they make in the way they deliver services that might attract women to come to the clinic for delivery when they are in the first stage of labor?
3. Once they had made those modifications, would they attract more women earlier on in labor?

Needs Assessment

In order to understand the situation more fully, the Team decided to do a quick needs assessment. They created a short series of questions the discharge social worker could ask women and family members following their experiences at the hospital. But since they were worried that some people would not tell them face-to-face the problems they may have experienced, they established a drop box located at the exit of the facility. People were asked to fill out an anonymous form on which they could describe how they had been treated, list any problems they had had, and tell them whether or not they would recommend other pregnant women to give birth at the hospital. Mpume saw an opportunity to bring in the “voices” of community leaders in one of the most remote villages where he did outreach to ask them why women used or did not use the hospital for giving birth. He would also ask about the reasons given for not using the hospital.

Alison reminded them that their purpose was to try to uncover some critical information that will help them make some no cost or low cost changes in service delivery. To help frame the discussion, she also shared the rights-based social justice programming principles with staff, who were interested when they saw some obvious linkages with quality of care and clients’ rights.

The interviews with clients and family members, especially of women who had died, and confidential forms turned into the drop box were fairly shocking. Some women commented that they found the experience of giving birth at the hospital “insulting”, the space dirty, the staff haughty, and the lack of privacy horrible. One woman, who said that she would not consider or recommend giving birth at the facility, noted that the only two people in her family who had ever gone to the facility had died there. Another comment from the confidential drop box noted that none of the staff seemed to know where any of the health care providers were, so the wait to receive care was lengthy. “Had I known my daughter would have had to spend 13 hours waiting for the doctor to show up, I would not have bothered to come.”

These impressions were confirmed by the input Mpume had collected from one of the most remote villages. It seemed that word of mouth from clients who had been treated at the hospital was only making matters worse. Even the community elders told him, very respectfully, that if a woman tries to go to the hospital it ends up being a waste of time and money, since she is sometimes not even received until the next day, and goes there to die far away from her family.

During the first discussion of these findings, Alison and Grace decided that they needed to give staff time to process the information, voice their concerns, and be mad, if that would facilitate making some changes. Eunice, the head nurse, was reluctant to accept that quality of care issues might be at the center of the problem, “But what do these women expect from us when they have been laboring already for two days before they show up and they come in so dirty?” Patience, a ward nurse pointed out that cultural beliefs and gender factors might also be contributing, noting that it was the husbands who usually made the decision when to seek care during a difficult delivery, and they probably don’t know the warning signs. “OK,” said Japheth, the clinic manager. “Let’s assume, for a moment, that all of those issues were solved. What is a client’s experience like when she actually gets here?”

Together with hospital staff, the Team brainstormed all the kinds of problems they thought existed: clients are not routinely greeted, especially if they arrive in the middle of the night when the night watch guard and the professional staff tend to be asleep or away, and long waiting times. (The doctor or nurse may be attending to someone else, or simply have gone home, so that someone has to run to find the health care provider able to handle such cases), and lack of privacy in communicating with staff. Although use of partographs is part of the hospital’s mandate and forms are available tacked to the wall, no one seems to bother to use them. This makes for an especially difficult transition when there is a staff shift change, and no one is sure exactly how long a client has been in a certain stage of labor. Often the cart used for emergency Cesarean section is not adequately stocked, so some amount of scrounging has to be done to find the appropriate equipment.

Following this discussion, the team met with a small group of their stakeholders where they heard much of the same information they had learned through the needs assessment: staff were considered unfriendly and arrogant; the facility was considered dirty and not offering any privacy; there was a perception that people went there to die. The stakeholders, however, were also full of ideas and basically supportive, so the team left the meeting feeling energized to begin their planning process.

One thing that kept bothering Alison was that, despite all these good discussions, Eunice’s comments continued to blame women for showing up late. She sensed that it might be important to do an exercise called “Is Your Agency Walking the Talk?” This exercise draws on the rights-based social justice programming principles, and helps staff to look inward. One question asks, “Are the rights of staff and marginalized populations respected within the agency?” Alison asked Grace to do this exercise with staff and use it to have a conversation with staff about respect for each other and clients, regardless of whether the clients have completely different belief systems or values. Going through this process together helped everyone to realize that continuing to think poorly of their clients would prevent them from truly changing the situation. Maybe the most important thing they could do was help to change staff attitudes and willingness to provide care in a respectful way!

Theory of Change and Causal Pathway

The M & E Team and program staff met once a week for the next 3 weeks to brainstorm simple interventions to improve the quality of care. First they thought about the problem they wanted to impact and created a theory of change for how to impact it (see the Theory of Change column that has been added to the log frame in the case study). They used a worksheet on constructing a causal pathway to help them sort through a list of questions to help them get to where they wanted to go. They listed all the changes that would be needed if they were going to be successful in preventing maternal mortality by repeatedly asking themselves, “and what other change would be needed?”

Constructing a Causal Pathway

Download to view and print: Word; PDF

It was very tempting for everyone to look beyond the hospital walls and consider issues such as who decided when a woman should seek care, what kind of transportation options could be garnered quickly when the decision was made, what kind of resources were needed to make this happen quickly, who felt that a woman who could not push out a baby on her own was really not a woman, etc. However they had to keep reminding themselves that their current task was to consider what happened within the clinic walls and so they focused on clinic-related changes when they constructed their causal pathway and filled in column 1 of the logical framework. “Maybe next year, with some good evaluation under our belt, we can think bigger,” noted Japheth.

Alison knew that one way to keep the evaluation focused on the obstetric clinic’s programs, was to agree on the goal, objectives and activities. The Team started by agreeing that they were most interested in reducing maternal mortality in the southern district, even though they would not be able to measure that goal by themselves. It helped them realize that the final result they would be contributing to would be reducing maternal mortality among women from 15 to 40 years of age in the southern district.

Objectives

In discussions they decided that the changes they could be responsible for achieving, program objectives were specific components of quality of care. that were measurable so Alison and the team went back to their causal pathway exercise and put check marks on three changes (immediate objectives) they could commit to working on:

1. increase the staff’s technical capacity in resolving obstetric emergencies including their communications skills
2. improve cleanliness and privacy of the setting in which women are cared for
3. improve the efficiency and fluidity with which care is provided

They were now aware that their new evaluation questions were:

1. Can their interventions actually improve the quality of care?
2. Will women and their families perceive the hospital more favorably due to these modifications?
3. If changes are made and perceived as positive, will more women come to deliver in the hospital and arrive in early stages of labor?

Mpume thought there were many cultural, geographic and economic barriers they should be addressing, but reluctantly agreed with the other Team members that first they should fix things up in the obstetric clinic and then reach out more to the community level in view of their limited resources. So their prioritized intermediate objectives focused on improving how women and their families perceive the quality of care provided and increasing the number of women who arrive earlier in labor (a long shot indeed, but one they were hoping they could show some improvements in!)

Activities

They discussed several activities that they would do to meet each objective. For the first objective, they discussed with the program staff a variety of training possibilities, and ended up agreeing to use a training curriculum created by a local NGO that supported hospitals on technical (obstetric) issues, and that included a good section on communication and counseling skills and that focused on clients’ rights.

For the second objective, they needed to make sure that the activities could be done at no or low cost. They agreed to use some old doors that had been in storage behind the hospital to replace the curtains, after they had been cleaned up and painted. And they enlisted the cleaning staff to set up a more rigorous system for checking cleanliness of public areas on a regular basis, in addition to the routine checks on surgical areas.

For the third objective, they decided that they could make a chart to show which staff had which responsibilities within the clinic, and where staff could be found at any time. They also worked with Japheth to get commodities inventoried, identify what was missing, order them, and then organize them in the cart which would remain in the operating room.

Program staff allowed three months to plan and start implementing all of the changes.

Indicators

Next, the Team turned to translating their objectives into well-constructed indicators. Alison suggested that it might be easiest to start by discussing monitoring indicators, which would help them to measure the activities they had planned. First they came up with:

• Staff training provided
• Office manager checklist of essential equipment on cart
• Doors hanging where curtains were
• Chart placed above receptionist’s desk that notes individual professional staff ‘s responsibilities and where they are stationed throughout the day

As they filled out the log frame they realized they needed to be more precise. They would need to know whether or not staff attended the training and also what kinds of communication problems showed up during the sessions. They also wanted to check if the equipment checklist was actually used and kept up-to-date, and whether or not staff registered where they were on a regular basis. With the help of the Worksheet: Developing M&E Indicators they were able to be more specific about the kind of calculation they thought they needed, and what they should actually be measuring for the indicators they would use. (You can see the indicators they decided on in their log frame.)

Then the Team came up with immediate indicators; the changes you would expect at the facility due directly to the interventions. While doing this, they reviewed the rights-based social justice principles and realized that some of their quality of care questions were also related to client rights, and they decided to make sure their confidential drop box questionnaire explored to what extent clients felt their rights were respected.

In view of the pressure to impress the MoH the Team agreed to focus on only those intermediate results indicators that they thought they had some chance of achieving, in a year or so. They reasoned that if quality of care actually improved, the word might spread, and people might improve their impressions of the hospital and more women would be willing to come in earlier during labor. If they didn’t achieve these results, at least they would have good data to explain why this was not happening.

Data Needed

Now they needed to focus on what comparisons they could use to answer their evaluation questions and from which data sources they would collect their data since the needs assessment had given them a good idea of what the women thought quality of care should look like, they focused on evaluating whether the planned interventions would make a difference in how people perceived the quality of care, how well they are attended and whether those changes made a difference in how well cases were resolved. They only had three months while the interventions were being planned and launched to develop all their instruments to gather some base-line data. They planned using the needs assessment data and the drop box questionnaires, from before the intervention, as part of that base-line for client perceptions of quality. And they would assess the providers’ attitudes, knowledge and communication skills before and after the training.

Thus their base-line would consist of:

• Needs assessment data on women’s perceptions of quality of care and community leaders’ perceptions of why women do or do not go to the clinic to deliver
• Clients’ responses in the drop box and all other measures during the month when the intervention was being planned
• Knowledge, attitudes, and communication skills of providers as they entered into the re-training course

Then they would compare base-line measures with periodic measures over one year, with major analyses done at 6 months and then again at 12 months after training and all the interventions had begun. And they would collect post-training questionnaires right after the training sessions.

The different sources for the data they wanted to collect were:

• Doctors, nurses and social workers who participated in the re-training and those who had not
• Clients who came to the clinic with pregnancy complications
• Community leaders and traditional birth attendants (TBAs) in a remote village in the southern district
• Clinic and maintenance records
• Client flow analysis
• Observations of client provider interactions
• Chart reviews

Now they were excited about the client flow analysis tool the MoH had sent down, even though they had not seen its utility before. This would allow them to have a more objective measure of wait times to compare with staff and clients’ perceptions of how well the clinic was functioning, especially in terms of fluidity and efficiency of care which the clients had complained about.

Collecting Data

For each indicator and data source they knew there were different ways of collecting data but they had to keep things relatively simple, wherever possible building on ways data were already being collected.

Clinic records would give them numerical indicators of the intermediate objectives. But they felt they still wanted to know if women in the remote villages had more favorable impressions of the hospital and why or why not. Mpume was only too glad to repeat the interviews he did for the needs assessment in a different but similar village in the southern district. He said he would have to program it in around 8 or 9 months from the beginning of the intervention according to his schedule of outreach visits. Though not ideal, they thought this would be very useful.

The pre-and post-training questionnaire was fairly easy to develop since Claire had a lot of experience both in training and assessing how well training had gone. She realized the staff would prefer to have the formats filled in anonymously so they didn’t feel evaluated, but she insisted on adding a code that would let her match up the pre- and post-test scores of each person. She had learned the hard way that if she didn’t do this, she could get group results that were impossible to interpret if different people took the pre-test from those who took the post-test, for example when some people showed up late and didn’t turn in their pre or post-test form.

In addition to the questionnaire in the drop box, to which they added a few items on the basis of what they learned from the needs assessment and their interest in rights, they wanted to find other ways of measuring the changes. Alison suggested that she could train the receptionist to be an observer, and they could come up with a checklist of behaviors and styles of communication she could observe in the waiting room and in the hallway where doctors, nurses and social workers received and interacted with the clients. Japheth pointed out that the receptionist was in a good place to also observe and record the audibility of conversations from the consultation rooms. She was also well liked by everyone, and staff agreed she would be a good person to make the observations, especially since she was not going to be identifying staff by name anyway.

When it came time to plan who, when, where and how the data would be collected, they realized they needed to divide up the tasks evenly so no-one would be overburdened and so the data collection would flow smoothly. The receptionist was very enthusiastic and fortunately didn’t know what intervention was planned nor when it would take place. But Alison found out that she planned to do the observations on the days when there was less work to be done, meaning when fewer clients had come in. That would never do since even the grumpiest staff would be more courteous on those days! The team also realized that there would be too much data if observations were done everyday, so they set up a random selection of four days a month and made sure to cover all shifts on those days.

One of the clinic’s junior nurses, Feddis, offered to help do the clinic records analyses, and she also volunteered to help tabulate the quantitative data on the confidential drop box forms. But it was decided that Claire needed to be the person to look at the drop box questionnaires since they could contain some politically sensitive information, if for example a client had written the name of a staff person who she was complaining about!

Analyzing Data

They decided to keep close tabs on the data as they came in so they wouldn’t find too many unpleasant surprises at the end if data weren’t collected the right way. Alison did some “spot checks” with Feddis, and they took a sampling of responses from the drop box, as a start. In the sample they found two issues that warranted further attention. First, the majority of women seemed to be giving a numerical value on the scale that indicated that they felt their care had not been as good as they expected, but then did not offer specific details that the team could use to make appropriate interventions. Alison added two follow-on questions, one asking for specific information on what the problem had been, and the other asking for suggestions for improvements.

In response to the question about whether women felt their rights had been respected, they found that no one had bothered to answer the question. This had been a question they had added without pilot testing it, and they realized it was probably not being understood. Alison did a few quick informal interviews with women in the waiting room and found that they really didn’t understand what “client rights” meant. They decided they did not have time to explore how to better ask this question. For now they would take it out and work more carefully later to find out how women understood their rights and whether they felt their rights were being respected.

Having made these and other needed changes, the M&E Team met regularly and made sure the data were cleaned and entered into their tabulation system. They were fortunate to have help from two social work students who were doing their practical studies in another hospital ward, but had their afternoons free. Some of the client record forms had to be brought back to the sign out nurse who was especially good at deciphering some of the doctors’ handwriting, in order to make sure that outcomes were tabulated correctly.

Now that they were analyzing the data they were surprised to see the range of negative attitudes that the staff had shown at the outset of the intervention. They were very proud to see that most of the staff had become more positive, and now only a few of the staff who had worked there the longest seemed to hold on to their negativity. Their analyses revealed that the staff who had shown positive change from before to after the training course, showed very good levels of compliance with other new procedures that had been instituted.

Another analysis they were very proud of was looking at the changes in perceived noise level and respect for privacy. They were able to show that comparing the situation before the doors were put on the consultation rooms and 6 months later, the observed level of overheard conversations had gone down. And over the same time period, the exit interviews showed a decrease in the frequency with which clients complained about lack of privacy during medical exams.

The Team worked with the social work students and Grace on the interpretation. There were some findings that they expected, and there were other issues they found surprising. There was a small increase in the number of women coming into the obstetric clinic in early stages of labor to give birth within the facility; more consistent use of the partograph, and wait times that had decreased dramatically, but were still far short of the target.

Exit interviews and drop box surveys showed that women and their families had noticed improvements in the way they were being treated. Some women reported they had been counseled respectfully by the providers on duty, and one woman noted that because she felt comfortable talking with the nurse, she asked her for help discussing family planning choices after giving birth, and was surprised to learn that there were family planning services available at the hospital.

When the data were discussed among the Team, Feddis suggested they look at differences by shift, since she felt that it was a very different place in the day than in the night. When the Team separated the data gathered for the morning shift and the night shift, they saw marked differences. Clients reported a higher quality of care in the morning and afternoon, than at night. The Team discussed whether this might have been because the staff trainings had taken place in the morning, and not all of the providers who normally worked at night had been able to attend the trainings. They decided to use their monitoring data to analyze this, and were glad they had registered who had taken the re-training course and who had not. The findings showed that only one of the nurses on the night shift had participated in the training. Staff in the night shift also had a much higher turn-over rate, and several of them were new to the hospital and would not have had a chance to be re-trained. Of course there were other differences that would need to be explored, but here was a major problem they would need to resolve as soon as possible even if it meant repeating the training course at night for the evening shift and for new staff on a regular basis.

Using Findings

The Team felt they had important findings but knew that some of them would be difficult to present. The first thing was to share them internally with staff and stakeholders. Grace advised them to write a short report that chronicled problems and issues, but was careful not to lay blame, and then to circulate it among all staff before holding a meeting. That way, people would not be too surprised. And she suggested focusing the meeting on what changes would be needed and how to correct the problems found. A discussion on problem-solving, rather than blaming, should keep the conversation headed in a more positive direction. Importantly, there was good news to share, as well.

Before presenting to the MOH, Grace spoke with a close colleague at the MOH to discuss issues that she thought might be of most importance to Mrs. Kamau. From this list, she drafted a one page memo that outlined key preliminary findings and attached the full report. Grace also sent the report to the MOH, and then arranged a meeting with Mrs. Kamau in several weeks’ time. There were clearly some good data to show; in addition to that previously noted, preparations for essential equipment, such as the C section cart (ready for use at all times), changes in the log book and record forms, etc. She was confident that in a few more months they would see even more positive results.

The Team was dismayed the improvements observed in the clinic had not permeated out into the remote communities. Almost the same kind of opinions had been found when Mpume interviewed village leaders. Mpume pointed out that making changes at those levels would require other interventions and take more time. They realized they would need to find a way to “spread the word” about the improvements and find a way to work more directly with the traditional birth attendants and community leaders.

When Claire was invited to a meeting of sister organizations, some CSOs and others government programs who were organizing around Safe Motherhood, her presentation of the evaluation findings was very well received. In the discussion she realized that other agencies had observed many of the same problems they had observed with transportation, cultural beliefs, and fear and negative attitudes about the quality of the services being offered. She was glad they were well on the path to improving the quality of the care they provided, but realized there was a lot more to do so that the women who most needed their services would feel comfortable and able to use them.

Many of these issues seemed like they were going to necessitate much more work, and would probably need to be built into a series of meetings with the MOH, in order to get their support to move forward (ideally, with additional resources in hand). These issues included very important ones that, although the Team had decided they were outside the scope of this evaluation, kept appearing in notes on the confidential client forms and through the group discussion. For example, next to the question on the drop box form about how certain a women was about where she would deliver, some clients had written comments such as “depends on what my husband says” or “depends if there is transport available when I need to go to the clinic” etc.

But what they thought was most exciting was to use their data to help plan new ways of working within their limited resources. Fortunately Grace was very amenable to reviewing the findings regularly and working with the staff to decide what additional changes they could make without any new funds. The Team met with the program directors, and together they found ways of being more participatory and helping staff become more sensitive to the needs of the women, especially after a values clarification exercise. They set up an internal ongoing quality of care committee that met once a month in order to brainstorm other no cost or low cost changes that they might be able to make, as well as begin to change the perception that quality of care was something that only the M&E Team cared about. Also, they held meetings specially for the late shift, and they participated gladly so they were not seen as the ones bringing down the quality of care..

Mpume helped them launch a small-scale, regional public relations campaign (via word of mouth and free radio time) to communicate educational messages to women and their families about the importance of understanding danger signs in pregnancy and labor, and where to go during the first sign of those complications. Because they had used some rights-based language in their radio broadcast, as well as in the presentation to their external audience, a donor with a human rights and social justice mandate contacted them to discuss a possible collaboration.

Several months following the 12 month evaluation, the clinic has seen continued, though small increases in the numbers of women coming to deliver at the facility and reporting a higher quality of care. The staff feel a sense of empowerment and greater job satisfaction since they are receiving mostly positive feedback on the confidential drop box forms, and their clients are delivering healthy babies. They hope that in the long-term, the ongoing M&E will reflect a decrease in the maternal mortality rate in the clinic. Their example has also now been copied by other clinics and hospitals; and so they hope that together there will be reduced maternal mortality throughout the region.

Logical Framework with Examples:

Download to print: Word; PDF

Back | Back to Top