Tips
Module 6: Collecting Data
Adapting Instruments to Fit the Program’s Setting
If you have found an instrument, such as a scale or checklist, that fits the indicator you have chosen but was developed in a different culture or socio-economic setting, what should you do? You need to find a balance between making sure the instrument fits the program’s local context and the fact that making changes in the instrument may alter its ability to function well.
- Select instruments that fit very well with the indicator you are trying to measure
- Choose instruments that are relatively simple and that did not involve complicated analyses to develop them
- Apply the instrument with a group of stakeholders and ask them to explain what each item means to them and how they answered it
- Adjust the items that were not understood or seemed out of place in the local context; but try to substitute items that refer to the same issue or component but in locally relevant terminology
- Re-apply the modified scale to a group of people similar to the sample you will use in your evaluation. Once they have answered the scale, ask them questions about the meaning of their answers in ways that confirm if their answers make sense or not. For example, if you applied a scale on couple communication, and the person scored very high on equitable communication, ask them how they feel their communication is with their partner
- Check to see that there are not any items that yield confusing results: either everyone answered them the same way or they answered in ways that were very different from the other questions. Ask people why they answered that way. Once you understand what is wrong with that question, make the needed modifications. Reapply the scale to see if the responses are more in line with expectations
- When reporting the results of the scale, describe the process you used to modify the original scale. You will not be able to claim the same levels of reliability and validity as were reported for the original scale, but you will know that the instrument was probably useful for your evaluation
Confidentiality and Anonymity
Since personal information around sexuality, personal relationships, behaviors and health is very private, it is important that everyone providing information trust that you will not share their intimate information with anyone. In small towns where everyone knows everyone, people may be especially afraid that the details of their personal lives will become known. Honoring confidentiality and anonymity may be needed to ensure accurate information and to protect the privacy of the participants.
A few things you can do to ensure confidentiality and anonymity include:
- Guarantee that individual responses of the participants will not be shared with anyone but the evaluation team, unless the information shows respondent’s imminent intent to harm themselves or others. Explain that their responses will be reported in summary form along with the responses of other respondents, in such a way that no one can be identified.
- Collect data anonymously, and don’t ask people’s names. If you need to be able to match surveys to respondents, use secret codes or numerical or other unique identification in the place of names. You can also ask the person to choose a name for her or himself. You should never say that the survey will be anonymous and then secretly record identity.
- If you must use people’s names, make sure to safeguard any materials that contain personal information very carefully. Lock those materials in a drawer or room and control access to them.
- Be careful with materials even when throwing them away. If you toss personal materials into the garbage can, they can be seen by people in your office, and they can be blown by the wind into the wrong hands. It is best to shred or burn them. Similarly, electronic data should be scrambled.
- Clearly state to the staff and data collectors that you hold the participants’ confidentiality to be of paramount importance. Let everyone you work with know how highly their confidence is prized.
- In a setting where everyone knows each other, anonymity and confidentiality can be guarded by only reporting findings in the aggregate- that is only reporting group findings, not those for individuals.
A note on Anonymity
Whenever possible, data should be collected so each participant can remain anonymous. However, if you are measuring changes in participants over time (e.g., baseline compared to end-line), you may need to match the responses of a specific individual’s end-line score. In order to perform some kinds of statistical analyses, you may also need to be able to identify each case at different times. Therefore, if you need to protect anonymity, you will need to include some form of unique identifier each time. This can be a code the person learns to provide each time or the answer to a few questions that do not reveal the person’s identity but are unique to him or her, so you can match the data. For example, mother’s birthplace and favorite color.
Please see the sample informed consent form provided in this module for an example of questions you can use to identify someone in order to match a pre and post test, while still maintaining anonymity.
NOTE: If you hope to use photos or quotes in your reports or presentations, it will be essential to get permission from the people involved. Sometimes it is not appropriate to give their real names but rather use a name they have selected and only general identifiers, such as age, marital status and education, depending on what is relevant to your report.
Data Collection Methods and Instruments
Download: Word
Generic Format for Informed Consent
Download: Word
How Many People Will You Need to Collect the Data?
- This will depend on the length of the data collection, how many people/events/documents will be assessed, and how difficult the data collection process is
- Decide how many surveys/interviews/observation sessions each person can manage accurately in the real world based on the pilot test of the instrument and data collection plan
- If you don’t have the resources for enough interviewers to apply the instrument well, cut back on the number of interviews, and/or the size and complexity of the instrument.
Important: It is always better to collect fewer data well than more data badly.
How to Get at the Right Questions?
Start out by talking to people in their own settings and learning their own language. Ask a trusted insider if certain questions make sense and what do they mean. Once you have the questions you think you will use, apply them with a small group of people who you can ask for feedback. Ask them if the questions made sense, what they thought in order to answer the questions and how did the questions make them feel. Writing Good Questions.
How to Safeguard People’s Rights?
Special procedures need to be put in place to ensure the confidentiality, anonymity, privacy etc. that was promised to the respondents.
- set up ways to identify the data source without revealing the person’s identity (usually referred to as codes)
- make sure the data collectors take very seriously the need not to “gossip” or make light of information they have heard during interviews
- make sure that procedures are established to offer help to people who reveal some critical problem or risk through the interview, without violating their confidentiality or any of their rights
- if a person can not respond because someone is within earshot, try to set up an appointment elsewhere to continue the conversation
Is There Enough Time to Make a Difference?
Previously in planning your indicators, we urged you to consider indicators of change that could realistically take place due to the program. If you have had to use a midline instead of a baseline and if you will be doing your end-line very soon after the program is over, consider what kinds of change can happen in the available time period. If your midline and end-line are just three months apart, and you are looking for changes in things like gender equity, make sure your questions reflect those specific precursors or components that could change in that time period. Or consider collecting data at end of program, and then do a follow-up at a later time.
Mentioning the Unmentionable
It is often necessary to have special introductions to very sensitive or highly stigmatized topics. Without exaggerating the evidence, introduce the topic and mention to the respondent that some people engage in this behavior although they may find it difficult to talk about it. Example: Now we are going to talk about masturbation. Though some people find it difficult to talk about, many perfectly normal people do it. Have you ever masturbated? Or please tell me about your experiences masturbating.
Method: Survey
Download: Word
Method: Focus Group Discussion
Download: Word
Method: In-depth Interview
Download: Word
Method: Document and Record Reviews Including Media Analysis
Download: Word
Method: Observation (participatory, non-participatory)
Download: Word
Method: Community-Based Participatory Exercises
Download: Word
Some Things to Keep in Mind When Developing or Choosing Questions and Categories
Make sure the language is understandable to the people answering the questions:
- Do they understand the terms you are using in the same way you are intending?
- Local language needs to be considered. Wording, what is left out, double meanings, and slang (especially around sexual practices) can vary by neighborhood, and practices may emerge that you didn’t think to ask about.
Make sure the concepts are understood in the way you intend them to be understood:
- Ensure the concepts have the same meaning to the respondent as you would expect. Examples: You might ask teachers about young girls’ self-esteem. If you are getting many different kinds of responses, you may need to work on a shared understanding or a definition of key words or phrases being explored. If you don’t know what the local meanings are, it is better to first use qualitative methods to find out before you create close-ended categories that don’t make sense in your local setting.
Or you might be asking “Have you had sexual relations”- what does this mean? In some places, young people don’t count oral or anal sex as sexual relations, even though you do. Young people have a lot of exploration before “intercourse”, so how can you make sure you are capturing them? Since sexual behaviors are taboo topics and people often assume they know what sex means and who people have sex with, it is often a good idea to explore the reality through qualitative methods first so you can then design a quantitative tool that includes real and locally relevant categories.
Make sure the instrument is applicable to the specific characteristics of the population/setting
where it will be administered.
- Data collectors should not make unnecessary assumptions about the respondents.
Example: If you are learning about sexual relations among your focus population, will the tool be applied only to heterosexual women? Will some women be in or have had relationships with other women? Will they all be in a relationship with only one man at the time? Do some have multiple partners in the same time period? Are some married couples not having sexual relations?
- Keep gender (and age) issues in mind. Are the questions appropriate for girls vs. boys, women vs. men? If you ask the same questions to different subpopulations, you may fail to capture information, or you may be capturing different information from each group. Tip: When Should You Watch Your Language?
- Are the situations realistic enough that people will be able to respond? Example: If you wanted to assess the quality of couple relations around decision making on contraception, you might want to ask if a woman’s partner would “get violent” if asked to use a condom. In some cultures this would not be appropriate, since being asked to use a condom is not really identified as a cause for violence. You would get almost all “No” responses. Therefore, this question would not tell you much about the power in those couples’ relationships.
- Does the social or policy context make the question inappropriate although it would be appropriate in another setting?
Example: Questions about decision making around the number of children to have are appropriate in many settings where couples decide this freely. In China, however, where there has been a one-child policy for many years in urban settings, it would not be appropriate since the decision has been constrained by the social policy and context. It might be more appropriate in rural settings or in other countries.
Writing Good Questions
Questions should be precise:
- Use simple language that will be understood by everyone, especially groups that have lower education levels, use different dialects or languages
- Keep the number of words to less than 20 and the number of commas to less than 4
- Avoid negatives, especially double negatives
- Don’t be vague. Ask yourselves what does an answer to the question mean. If you get different responses even within your own M&E Team, you have certainly not been specific enough!
- Make sure there are not multiple interpretations about what you are asking or what the answers mean
- Present the questions neutrally. Avoid questions that have a socially correct answer and don’t lead the respondent into answering a certain way
- Only ask one thing per question. Avoid questions that include “and’ or “or” since you won’t know which the person is responding to
- Avoid overlapping options if the respondent is asked to choose just one answer
Questions should not make assumptions about the respondents:
- Include a “not applicable” option just in case the person really can’t answer the question
- Ask filter questions that clarify assumptions and identify relevant respondents for certain questions, such as “Do you have children”, “Are they alive?” before asking questions about child behavior; “Have you had sexual relations?” “Has it included vaginal intercourse?” before asking about pregnancy prevention
Questions should be presented in a balanced way:
- Present both sides of an attitude in the question itself
- Provide a balanced set of options as answers, both negative and positive
- Ask about an entire range of behaviors, including those that are taboo or stigmatized to soften the effect of talking about them
- Make sure the interviewer does not bias responses through gestures or tone of voice.
Questions should follow logically in a conversational manner:
- Group questions by topics
- Introduce each new topics with a short explanation of what you will now be talking about
- The questions should not be stiffly read but flow like a conversation)
Questions should not contain the answer
Questions should not make the respondent uncomfortable or feel guilty
You may decide to ask about what other people do if it is not possible to get people to talk about themselves doing something illegal or highly stigmatized. Example: In an evaluation of a program for men in prison, focused on HIV prevention, it may be too difficult for men to report having sex with other men and particularly reporting violence related to such encounters. You might have to ask them to report on whether or not they have heard of sexual encounters between male in-mates and/or with guards and whether or not violence was involved.
Suggestions for Training Data Collectors in Observation Skills
Observation can yield very rich data because you are literally observing life as it is lived. It gives you the opportunity to see how things unfold in real life and in the richness of a person’s socio-cultural context.
To train observers:
- start by having them read a good example of an observational study
- find an excerpt of good “field notes” to share with the people who will be doing observations
- take your data collectors to a setting- such a public place, someone’s home, the waiting room of a clinic –
- ask them to observe for about 30-60 minutes
- have them take notes on: what people say and don’t say
- their tone of voice, their body movements and gestures
- how people do and don’t interact with each other
- responses to events or other people
- the norms that appear to guide behavior in this setting
- the focus of the behavior and interaction, e.g. a birthday, trying to get an appointment, bargaining in the market
- differences between subgroups such as men and women, young and old, people of different skin colors or ethic or tribal back grounds
- When they leave the setting- tell them to sit somewhere quiet and write down for as long as it takes to record everything they had observed, including their impressions
- Review their notes together with an experienced ethnographer and provide in-depth feedback.
- Once you have decided what the main categories of information are that you want to collect data on you can provide your observers with forms that have each category on them, that way their observations and notes can be focused on what is of immediate interest
To Record or Not to Record?
It is always very tempting to tape record interviews or certain events but this has pros and cons.
- pros
- it’s the best way to know you have captured the details of complex events or interviews
- the data can be checked by others for reliability
- diverse sets of coding and analysis can be done and re-done
- most people feel comfortable and act naturally despite the recording
- cons
- may negatively affect the freedom with which people behave or share information
- need to take precautions to preserve confidentiality and anonymity, including storage and erasing tapes after data analysis
- can be very expensive to review in terms of time and money
- creates an additional physical piece of information that must be protected under your confidentiality plan
Using Both Types of Data
It is often a good idea to use both types in the same evaluation since you will be able to answer “what?” and “how much?” questions through quantitative approaches and also “why?” and “how?” through qualitative approaches.
Often your qualitative data will
- allow you to develop more locally-relevant quantitative methods
- help explain your quantitative findings
Often your quantitative data will
- lend legitimacy to your qualitative data
- show the magnitude of the findings you described qualitatively
What Does Informed Consent Really Mean?
Both parts of this term need to be taken seriously:
- Informed – means the people have to have received and understood the nature of the study and their participation, what the information will be used for and how their privacy and/or confidentiality will be protected. It is not enough to present a written or verbal description; you need to make sure the person understood the explanation. This is particularly challenging in cases of respondents who are illiterate, speak a different dialect or language, have very low educational levels or are timid or reserved about asking questions.
- Consent - the respondent must really have the possibility of saying no without being pressured, coerced or made to feel guilty. There may also be considerations of legal permission, for example if the respondent is a minor or, in some settings, a woman who is not allowed to grant such permission autonomously. While this kind of situation violates anonymity one can still guarantee privacy and confidentiality of the actual data collected.
What If This Information Is Very Sensitive?
It is always essential to respect the person’s comfort level. Never pressure someone to answer any question they seem reluctant to answer. Remember that people can refuse to answer any question and can end the interview/survey at any time.
- A good introduction and consent process will help if you carefully explain why the information is needed and how the information will be used. Try to enlist the person’s support. You should also explain how anonymity and/or confidentiality will be protected. If the person still refuses, respect her or his right to do so
- If an unexpectedly high number of respondents want to skip a particular question or decline to participate in the data collection at all, then re-think your method, the interviewing technique; and maybe even the comparisons you need to make. Ask them directly in a non- threatening way why they are refusing to participate to gain a better understanding of why your plan is not working
- If responses to sensitive questions are essential to the utility of your data, include those questions early on in the interview so that if the person refuses to provide the information, the interview can be ended before collecting all the other information.
- Interviewers should ask themselves, “Am I helping them talk about this topic?” Try different ways to make people comfortable talking about difficult and sensitive issues
What Is the Difference between Open-Ended & Closed-Ended Evaluation Questions
- Open-ended questions are queries that allow respondents to answer in whatever way they wish, without determined response categories. For example, “How do you feel about the health care you received at the clinic over the past year?”
- Closed-ended questions have a fixed set of possible responses from which respondents can choose one or several responses. For example, “Do you feel that the health care you received at the clinic over the past year was: a) excellent; b) good; c) fair; or d) poor?” or, “How did you travel to the clinic today? a) bus; b) taxi c) own car; d) borrowed car or driven by someone else; e) walked f) bicycle g) combination of more than one h) other”
What Kind of Data Should You Use?
Download: Word
What Kind of Information Should Your Data Collection Instruments Contain?
Recording descriptive information about the data source and data collection event:
You should include information about the site/location, date, time, length of data collection, and data collector’s name. When collecting data directly from people, include questions to gather information about respondents’ social and demographic information such as age, race/ethnicity/religion, occupation, and sex. Socio-demographic information should be collected no matter what type of data collection method you are using: focus groups, surveys, etc. Socio-demographic questions are important because they allow you to:
- Understand who you are collecting data on
- Divide your respondents according to important criteria relevant to the questions you are trying to answer
- Identify factors that could alter your findings
Make sure your pre- and post-test and comparison groups are comparable Tip: What If This Information Is Very Sensitive?
Focusing on your indicators:
The body of your data collection instrument will be specific questions, topics to be addressed or places for observations that provide information related to your indicators. Each section of the instrument should have clear instructions about how the section should be introduced and/or applied.
When developing the indicator parts of the instrument:
- Make sure that each indicator has been turned into one or more questions, topics, or categories of observations
- Make sure that each question or category for coding corresponds to one or more indicators. For complex indicators you may need to use several questions to get at a certain issue. You may even want to create a scale or index. To learn more about this, Tips: When One is Not Enough
- Do not include questions or categories of information that do not correspond to your indicators, except for information that allows you to identify the data source or respondent and that may be needed to ensure the logical flow of your questions, or identify which set of questions are applicable to a certain person, event or document. An exception would be when you have previously decided to collect a small amount of data about some important gaps in the program logic. But keep the number of such instances low, and think ahead about how you will use this information
- Don’t include questions or coding categories that that are so vague that they will be difficult to interpret. Examples:
- For example: “When my partner and I are together, I’m pretty quiet” should be “…I am more quiet than usual” so that it is relative to the person’s normal level of quietness. But if your indicator is focusing on the quality of the couple relationship, does such an answer mean there is enough trust to be quiet (perhaps a sign of a good relationship) or is the woman inhibited by her husband’s presence (perhaps a sign of a poor relationship)? It would be better to ask: When my partner and I are together, I feel comfortable enough to be more quiet than usual? Or …I feel less able to speak and become more quiet than usual”.
- Make sure each question, and answer options for each question, are framed in a way that matches the kind of data you need. Provide specific categories for answers if you want quantitative data. Leave the categories open and room to record the entire answer if you want qualitative data
For more information,  Tips: Writing Good Questions
Observations and field notes:
- Leave some space for the data collector to include notes on observations (referred to as field notes) about unanticipated circumstances, and non-verbal or spontaneous data, such as silences, discomfort, interruptions; informal cues that could alter the meaning of the data collected; and even subjective impressions about the adequacy of the data. Having data collectors take field notes is a real opportunity and can significantly add to the data. (Tips: Suggestions for Training Data Collectors in Observation Skills)
What Skills Need to Be Built?
If you have a choice, select people who:
- can be objective in listening and recording, as well as in interactions with respondents
- are comfortable with the kind of people they will be collecting data from
- are motivated to find out how well the program is working not who just want to show how good it is. This means the data collectors should be people who can separate themselves from negative feedback they may hear about the program, but know that such information will help the program in the longer run.
- Have the person being trained write his or her field notes. Share and discuss them. Make sure they are relevant, complete, and objective.
Skills building training should include:
- Observation and listening techniques, using role playing among the trainees
- How to avoid leading questions:
- Give lots of examples of leading questions
- Develop a careful script for each interviewer with specific probes when more information is needed
- How not to bias responses or reinforce behaviors:
- Be careful about gestures
- How to be neutral but supportive
- What not to do and what they can do
- Techniques for legitimizing discussion of taboo or highly sensitive subjects (for example, men may need help having some topics legitimized so that they can talk about their feelings.)
- Include an introduction to each section or sub-section that includes sensitive issues. Explain what topic will be discussed, that it may be difficult, but that it is more common than believed (or some accurate statement that legitimizes the topic), and why the information is important and how it will be used.
- Do not be judgmental
- If using closed responses, you may want to put the most stigmatized ones first so they seem more acceptable. In this way the respondent does not have to check off in their mind what they are not before coming to what they are.
- Sensitivity to atypical answers, and how to handle them
- Desensitize interviewers to atypical possibilities and teach them not to make any assumptions
- Make sure interviewers know they can report back at their regular meetings if certain questions make them uncomfortable and they will get help in handling these situations
- Capturing information on unsafe behaviors
- Be clear about legal guidelines that operate in your setting. If the interviewer is legally mandated to report certain information, they need to tell the respondent before the interview starts
- Even if no mandate exists, a clear protocol should be established by the M&E Team so that interviewers know what to do in the case they are told about unsafe or illegal behaviors
- The protocol should include some systematic way of offering help or referral to the respondent
- Cultural norms of respondents (where sub-cultures have different norms)
- Avoid situations in which the interviewer can not obtain the information due to some cultural bias, such as married women not feeling comfortable talking about sexual relations to a younger unmarried woman
- Make sure interviewers feel comfortable reporting back on such cultural dilemmas they may encounter
The training should:
- Involve lots of time for role plays and practice
- If you can, tape interviews and have each person assess how well or poorly they did. Or observe the person carry out an interview and provide specific feedback on how it went. Point out and correct:
- missed opportunities to ask questions or explore issues
- interruptions by the interviewer
- judgmental comments
- encouragement of certain answers
- tensions and stresses introduced by the interviewer
- places where the interview got side–tracked and turned into a conversation without focus on the questions
- whether or not the interviewer lost control of the interview and the respondent went off in another direction into irrelevant topics
- Focus on helping data collectors learn to always seek ways to help people provide the most accurate answers possible
- Identify people whose attitudes and skills just don’t make people comfortable. In these cases, try to find other tasks for these people and do not use them as data collectors
What to Do If You Have Never Trained Data Collectors Before?
If you are unsure of how to train others in data collection methods, consider finding someone from outside your organization who knows more about data collection and could train your data collectors, such as a professor at a nearby university, or a colleague at another organization (governmental or non-governmental). Psychologists or anthropologists are often skilled in providing this kind of training, and might be able to give you advice or help you learn how to do the training yourself.
What to Do When No One Is Home?
Not only in cases of home visits, but for any data collection, you may not find people or documents when and where you expected to. Make sure follow-up rules are clearly established and systematically followed by data collectors, and that missing data and attempts to find the data are recorded in a pre-established format.
- follow-up – decide on clear rules on what data collectors need to do to follow-up to obtain the needed data or to substitute for missing data sources in order to keep biases from creeping in. Decide:
- how many times the data collectors will re-visit a site to obtain the data
- how many phone calls will be made to set up an appointment
- what should the data collector do when a document is missing or a person refuses to provide information
- refusal rates- differences in how many people refused to participate in the data collection in different sub-groups or comparison groups can influence the meaning of your findings. Keep track of
- the total number of participants within each sub-group
- the number of refusals in each group
- absentee issues – some groups may be more difficult to locate and show up less frequently to the venue in which data collection is taking place. Drop-out rates in 2nd or 3rd time assessments could result from factors internal or external to the program. Keep track of
- the times you went to the data collection site and did not find the person you needed to contact
- how many people of certain groups or documents of certain kinds were missing e.g. in classes or in files. Example: If the base-line started out with equal numbers of boys and girls but boys did not show up at mid-line or end-line, the findings will be greatly altered and may reflect changes needed in the program or an external event that kept the boys away, such as a new football club in town.
When Is Different Better?
People often assume that people just like the respondents will be the most trusted and therefore illicit most honest answers. But this is often not the case. Sometimes people feel more comfortable confiding in someone they see as being different from them and who will not judge them as people from their own community might. People from outside a culture or social group can sometimes even see things more clearly. What is most important is the person’s ability to not be judgmental, to put the respondent at ease, and to listen openly and caringly.
When One is Not Enough
To make your data collection more rigorous you should collect more than one piece of information on each concept, especially on ones you are less certain about. These may be
- several questions about a single component of a complex social concept
- a list of different instances that correspond to one question, e.g., different spheres of domestic life in which a woman does or does not have the power to make autonomous decisions
Be careful about reporting these as one indicator. There are pros and cons to doing this:
Pros:
- if the answers actually belong together, then you can be more confident that you are actually measuring the concept you want to measure
- you will be able to present fewer findings
Cons:
- the answers may not really be reflecting the same concept
- some answers may be far more important than others and by joining them you may lose this information
- one or more of the items may counteract the others and hide the real results
To avoid the pitfalls:
- Always look at the separate pieces of data first. Are they all telling you similar things? When one is high or low are the others high or low? You should do this during your pilot test of the instrument, as well as during your actual analysis phase.
- Find out what your stakeholders think about the meaning of the different questions or observations. Do they think they all refer to the same concept? But you may want to go back to your stakeholders once you have the actual data collection instruments to make sure your stakeholders agree that the questions you think are reflecting a certain concept actually do.
Report separate pieces of data separately, unless there is a reputable precedent or clearly valid rationale for combining them.
When Should You Watch Your Language?
If you are exploring the same concept with different subgroups, the words you use may be understood differently by different groups, for example by teens and their parents. It’s more important that your respondents understand the question than using the exact same wording for everyone. Or you can include both sets of wording for all respondents, e.g. ‘By concept X, I mean (insert definition), sometimes referred to as (insert slang term)’.
If you adjust the wording for different groups
- Think ahead to how you will need to analyze the findings, probably as comparisons rather than putting the responses all together
- Make it clear in your final report how the questions were asked, and
- Explore the meaning of any differences you found
When Staff Become Data Collectors
When program staff become data collectors, they will need help to keep their different roles separate
- To avoid allowing their program expectations to influence the data that they collect
- To keep from intervening to help people during data collection
- And to make sure the respondents trust that they can answer honestly and will not lose services or access due to their answers or behavior
Follow these suggestions:
- discuss the ethical issues from the outset of the training, and make sure everyone understands the need to keep their roles separate
- make sure staff realize the evaluation will not be used as an employment assessment, and make sure it is not used that way
- emphasize the importance of the evaluation for learning new ways of improving the program and becoming even more effective in their programmatic roles
- teach them standard responses when respondents ask for advice or help, such as “we can discuss that later.” So they won’t do it their own way or interrupt the interview; but make sure they are empathetic and supportive and provide some helpful response or referral once the interview is over
- make guidelines very clear about their role as data collectors
- tell them what they can and can not do in terms of offering help or advice to the respondents
- make sure they can provide help at the end of the interview or observation session
- make sure they register exactly what advice or information was provided
- offer a pamphlet or fixed set of information at the end of the data collection
- do everything you can to avoid staff being responsible for data collection with people who they are responsible for in the program
- assign staff to data collection with people from a different part of the program or program site if possible
- assign staff to collect data from people they don’t personally know
- don’t tell staff if the person they are interviewing or observing is in the program or is part of a comparison group
- include clear explanations for all respondents who are program participants
- explain how data will be used
- explain that they are not expected to answer any certain way
- make sure they feel comfortable providing accurate information free of concern about hurting feelings or losing access to services
- use self-administered questionnaires (if adequate for capturing the kind of data you need)
- make sure respondents can deposit the questionnaires without concern that staff will know who they are or what they answered
- include reference to staff as data collectors in the “limitations” section of your final report
- explain the precautions you took to reduce the impact of this practice
- also emphasize the benefits you may have observed
- be honest if you have continuing concerns
When to only get Verbal Consent
Avoid asking for a signature (written consent) if you think it would cause the person to choose not to participate, put the person in a risky position by signing his or her name, or make the person so uncomfortable that it might compromise the honesty of their answers. If you are working with a population that may be illiterate, you should have a verbal consent process- you explain the nature of their participation and rights, as well as get their consent to participate verbally.
Why Turn Qualitative Data into Numerical Indicators?
Because, it can be very effective. You may need to collect qualitative data if you do not know enough about how some phenomenon looks in the program’s local setting. But once you know the range of behaviors they describe and the kinds of meanings they express, you may want to report on how many of the people you observed or interviewed did or said each thing of interest. By transforming qualitative, non-numerical data into numerical data, you can quantify and chart the numbers (via graphs, histograms, etc.) and better identify trends.
|
Word
PDF
