Step 2: Ensuring Adequate Analysis of the Data
Even if you planned things well, errors can creep into the analyses that might limit the value of your evaluation. To prevent this, make sure you carefully:
- Supervise data collection and correct any errors you may find
- Process the data as they come in
- Carry out simple analyses that answer your evaluation questions
Task 1: Managing Data Collection
=> Meet regularly with the people who are collecting the data:
- Go over the data collection procedures that you have developed and make sure everyone is following them
- Ask the data collectors to describe how collection is going, and listen for things that may impact the data you collect but that aren’t identified as problems
- Listen to the data collectors about the challenges they are encountering and any insights or suggestions they have developed
- Do role plays to test out ways to overcome problems they have encountered
- Encourage the data collectors, and let them know when they are doing a good job
- Identify differences between data collectors that may impact the quality and/or consistency of the data, and make needed adjustments
=> Review and clean up the data as they come in:
=> Provide additional or reinforcement training if needed:
- If there are persistent problems that come up
- As soon as you detect the problem. Don’t wait until the problem gets worse
=> Make adjustments that are urgent.
- It is better to make changes than to wait and discover the findings are useless
- Think through the implications of such changes for your analyses and findings
- Record when and what changes were made
- Report such changes when you present the findings
To help you through this process, use the Checklist for Supervising Data Collection
Worksheet: Checklist for Supervising Data Collection
Task 2: Process your data carefully
Rationale
To avoid introducing errors and new biases into your analyses, make sure you handle the data very carefully, from the time they are collected to the time they are actually analyzed. Qualitative and quantitative data will need to be handled differently.
An important part of processing your data is creating and applying codes. A code is a rule for converting a piece of information, for example the data you collect, into another form. For example, all women will be coded as “1,” or all women who attended all of the workshop sessions will be coded as “intensive participants.” You will decide on codes and all of the team members should apply the codes in the same way.
For Qualitative data:
=> Make sure data collectors edit and complete notes immediately in the field or as close to the time of data collection as possible. This means they will need a place they can write rich notes on a timely basis.
=> Have tapes transcribed to provide a written version of what was said in the interview.
=> Make sure data collectors take notes on non-verbal aspects of the interview, such as:
- gestures
- silences
- any sense of discomfort
- facial expressions
- clothing worn, the setting, etc.
Important: Even when an interview has been entirely transcribed there may be times where what was said does not make sense or its full meaning cannot be appreciated without the other non-verbal data.
=> Start to develop your approach to data analysis in order to make sure that you are collecting the data you need. For more info on the basic approaches to qualitative data analysis 
Tips: Steps for Qualitative Data Analysis
For Quantitative data:
=> Review data collection forms in the field
- make sure they are complete and legible
- do this when the data collector could still return to the data source to complete or correct anything that is needed (without violating confidentiality/anonymity agreements)
- => Make sure missing data are coded separately and consistently so that you don’t mistake a missing response for an answer. Code missing data as “9” or “99”
- Code not applicable data as “8” or “88”
Important: Despite your best efforts, there will always be some missing data.
Important: Be careful not to confuse these numbers as “real data”. Separate them out before making any calculations. And always report the percentage of missing data for each indicator.
=> Make sure the rules for coding data are very clear
- check that each data collector knows them well
- If coding questions arise during data collection, make sure all decisions are recorded, dated, and known to the data collectors.Tips: How to Keep Your Codes Straight?
=> Create a spreadsheet to tabulate the data by hand (unless using a computer program)
- Create a spreadsheet possibly using a program such as Excel or Lotus
- Make sure each cell has a number so that you can see if something is missing
Tips: Suggestions for Data Tabulation
Task 3: Use descriptive statistics to summarize group responses and make the comparisons that answer your evaluation questions
You will now use simple analytical techniques to
- describe the major groups for which you have collected data
- make the basic comparisons you planned
- explore differences within the groups you have studied
=> Describe the findings for each group.
- Tally how many people had each type of response (e.g. used or didn’t use a condom at last sex) in each meaningful categories (e.g. men before and after program, women exposed to the program and those who were not) by hand or using a function of the spreadsheet program you are using. You will do this for both qualitative and quantitative data.
- You can use an empty data collection sheet to register how many of each kind of answer you received
- Even at this point, check for things that don’t make sense and might be coding errors. Check if the tallies are consistent with other answers. Example: If there were 21 women who said they had been victims of violence; make sure there are also 21 women who provided answers to the questions about violence they had experienced. If the number is different, you may need to go back to the interviews to see if something was coded incorrectly or you may need to code some cases as missing data.
- Group the results according to topics that are logically related.
- Look carefully at the data collected from different data sources on one indicator to see in what ways they do and do not bring you to the same conclusion
- Look at the answers to different items aimed at the same indicator to see if they show you the same thing
- Present the findings in simple tables or graphs showing counts, frequencies, averages, medians, or percentages of responses by topic. Tips: When Not to Use a Percentage?
=> Compare the findings between groups to answer your evaluation questions.
With your quantitative data you will most likely be using basic descriptive statistics to show differences between the groups you are comparing to answer your evaluation questions. With your qualitative data you also may use descriptive statistics by converting your data into numbers. For example you can count how many people had a particular kind of outcome in the program. However, especially if you have qualitative data, you should compare findings between groups using qualitative analysis techniques
=> Use intra-group data to make additional comparisons that shed more light on the basic comparisons
Are there sub-groups within each of your basic comparison groups that responded differently to your program? To find this out use your
- monitoring data
- socio-demographic information on the respondents
- if you have findings for both immediate and intermediate results, use your immediate results to better understand the intermediate results
Important: Observe if results are better when people were more exposed to the program. Important: Explore if the program was more or less effective with different kinds of people.
| Kind of information | Examples of intra-group comparisons |
| Monitoring |
- Low exposure vs higher exposure to the program
- Attendance or exposure to certain parts of the program rather than others
|
| Socio-demographic |
- People with certain SRHR histories vs those without, such as prior abortions, complications in pregnancy, STIs, etc
- Groups who differ by economic levels, ethnic groups, degree of marginality or vulnerability
|
| Factors identified in your Theory of Change (Module 2) and in your understanding of the Complex Social Concepts involved (Module 3) |
- People who express more negative vs positive beliefs, attitudes, or higher/lower levels of knowledge
- High peer pressure to conform to traditional gender roles vs support for equitable non traditional roles
- Communities with higher vs lower levels of tolerance for violence; more or less restrictive social norms, etc.
|
| Immediate results |
- Greater or lesser change in attitudes or knowledge at end of program
- People who assessed the program positively vs those who assessed it negatively
|
Cross-tabulations will be a good method for presenting such data that compare two different data items to see how they are related, such as how men and women with high vs. low educational levels differ in terms of their knowledge score at the end of a workshop. Tips: Descriptive Statistics for Analysis of Numerical Indicators
=> Discuss the analysis with data collectors, program staff and other close stakeholders.
Present your preliminary findings before you have made your final charts or slides. Use simple formats to present and discuss them, first with the people most closely involved with the data collection. Write relevant descriptive statistics right on a blank data collection form next to corresponding items. This format is familiar to data collectors, clearly links the findings with the data, and is in a format that has already been created.
At this point in time the purpose of the discussion is to:
- Identify findings that do not coincide with the observations of those closest to the data. Such discrepancies may be pointing to errors in coding or analysis, or they may be interesting and valuable findings in and of themselves
- To make sense out of comparisons among different data sources for the same indicator. Any discrepancies among such data need to be carefully examined to know what is happening before you can conclude that one kind of data source is better than another. Such discrepancies may actually lead to greater insights into the instruments, the program or the people you are reaching
- Find out if there are other analyses you need to do to really understand the data completely. Do the data collectors or staff ask questions that you can answer by breaking down the data even further? Example: The data collectors ask if there are differences in the findings among three different clinic sites since they have heard that the director in one site has very negative attitudes towards the program. You decide to compare the findings by clinic site to see if such a leadership issue has had an impact on the program’s effectiveness.
Important: Time spent trying to understand what the data are telling you is time well spent.
=> Carry out further analyses that those discussions suggest are needed.
To help you through this process, fill in the Checklist of Good Analysis Practice
Worksheet: Checklist for Good Analysis Practice
Beyond descriptive statistics?
If you have the resources to conduct a more complex statistical analysis, such as inferential statistics (which would require planning for this from the outset in terms of sample selection and size and survey construction) you should consult with your statistical expert.
Tips: When Can You Use Inferential Statistics? And
Tips: How Can You Know If a Difference or Change is Significant or Meaningful?
