Step 1: Deciding What Data, Which Instruments?

Rationale

Since there is more than one way to collect data on any indicator, you will need to make a decision about which ways will work best for your evaluation based on experience, effectiveness and cost (time and money).

The two important decisions to make for each indicator:

Qualitative or quantitative data collection?
Data collection method?

Important: Even if resources are scarce, there is always a way of assessing any indicator.

Task 1: Decide what kind of data you want to collect for specific indicators

This will depend on:

• The kind of questions you want the data to answer
• Your familiarity with possible categories of response

Quantitative data answer who, what, where, and how much. They

• Use fixed categories linked to definitions and numbers
• Have questions and response categories that are pre-determined by the developers of the data collection tool
• Are more structured and standardized (this means the same exact procedure is used with each person) than qualitative methods
• Present the challenge of organizing the numerical data in a meaningful way
• Apply to larger numbers of data sources more economically than qualitative data

Qualitative data answer why and how, and usually involve talking to or observing people or reading or viewing texts or images. They

• Use words instead of numbers
• Focus on meanings, patterns of relationships, and perceptions
• Do not require knowledge of pre-determined response categories
• Present the challenge of organizing the thoughts and beliefs of participants into themes or typologies
• Usually involve fewer data sources (e.g. people or documents) than quantitative methods
• Can be used to help formulate more effective (closed ended) survey questions by providing an understanding of the language, meaning, and experiences of respondents

It is often a good idea to collect both types of data. Tips: Using Both Types of Data

An example of different questions relating to different types of data:

If you are trying to show whether or not there were changes in the way young men view what masculinity means after participating in a series of sensitization workshops, you might want to know:

How many young men changed their attitudes towards violence and their “need” to have multiple sexual partners? Was these large and predictable changes? For this you would collect Quantitative Data.

Or

How young men envision the new form of masculinity they have acquired and what it means for their relations with women and their vision of themselves. You would probably best collect Qualitative Data by capturing the words of the young men themselves.

For each indicator and data source make sure you consider:

• What will answer your questions the best way
• Which approach fits your resources
• Which makes most sense in your local context
• If and how you might use both kinds of data to your advantage

Important: Do not base your decision on which approach to use just because that is what you have done before or know best. Base it on which approach or combination of approaches will provide you with the kind of information you need to answer the questions you need to answer.

Tips: What Kind of Data You Should Use for more information on the pros and cons of each kind of data.

In Module 4 you learned that indicators can be numerical and non-numerical. Is the difference between quantitative data and qualitative data confusing? Tips: Description of the relationships between the kind of data and the kind of indicators you will use (Module 4).

Remember that qualitative data can be converted into numerical indicators. Tips: Why Turn Qualitative Data into Numerical Indicators?

This is done by counting categories found in the original data. An example:

If you are unsure why – in your local setting - it takes pregnant women with complications a long time to arrive at a hospital, you would first want to collect qualitative data by talking with them and with their family members. If the Health Clinic director wants to know how often certain reasons, e.g. those related to discrimination by health personnel, are cited, you could create categories based on the ones mentioned in each interview. By counting the number of women who had evidence of discrimination in their history, you would have a numerical indicator (such as a percent or rate) of how often certain problems had been reported.

To choose what kind of data you need to collect, see Worksheet: Choosing the Kind of Data that will Answer Your Questions

Worksheet: Choosing the Kind of Data that will Answer Your Questions

Task 2: Decide which Data Collection Method you will use

Whether using numerical or non-numerical indicators, you will have to select among a wide variety of Data Collection methods. Each method has its pros and cons and is more useful under some circumstances and less in others.

Choose methods that will:

• Be most respectful of the people from whom you are collecting data
• Be relevant and appropriate in the local setting
• Help you obtain the most accurate and complete data possible
• Not introduce unnecessary biases

Key issues you need to consider:

• Sample size requirements or limitations
• Appropriateness of methods in light of relevant characteristics of certain respondents e.g., their literacy level, their willingness to speak in a group
• Your knowledge of the potential answer categories of unexplored topics
• Need for anonymity, confidentiality or personal relationship to elicit sensitive information. Tips: Confidentiality and Anonymity
• Your resources -- money, time and staff expertise

Below are some of the most relevant methods: Click on each one to see its characteristics and pros and cons.

Tips: Method: Survey

Tips: Method: Focus Group Discussion

Tips: Method: In-depth Interview

Tips: Method: Document and Record Reviews Including Media Analysis

Tips: Method: Observation (participatory, non-participatory)

Tips: Method: Community-based Participatory Exercises

For each method there are a number of kinds of instruments you might use. Tips: Data Collection Methods and Their Instruments

Important: Your evaluation will be more rigorous if you use more than one data collection method for key indicators. The use of several data collection methods (and/or data sources) to gather data on the same topic (called triangulation) can help you validate data and may lead to richer understanding of the topics being explored.

Logical Framework: Fill in Column 3 of the Logical Framework ONLINE

(Website registration required)

Option 1: Download Word version of the log frame in order to work on your own computer. We highly recommend you download it to your PC before working in it. The Word version is best for modifying the form to suit your needs, for example, adding rows and columns.

Option 2: Download PDF version of the log frame in order to print. PDF files are best for viewing and printing.

Option 3: Fill out the form Online. To enable you to view the entire form at a glance, we have restricted the number of characters to be entered into each text box. Therefore, we only recommend using the online form if you do not need to enter a lot of information. In order to use the online form, you will first need to create an account, and then log in

Once you have decided what methods you will use (e.g. focus group, direct observation, short interview, survey), you will need to design the formats on which you will register the data.

You can:

• Use instruments others have used in similar settings. Tips: Constructing Indicators and Instruments: Selected References (Module 4)
• Adapt pre-existing instruments to fit your setting. Tips: Adapting Instruments to Fit the Program's Setting
• Design your own instruments. Tips: Writing Good Questions, How to Get at the Right Questions?

=> Include sections for 3 kinds of information in your instruments

• Recording descriptive information about your data source and data collection event
• Collecting data on your indicators
• Registering field notes or observations

For more information on each of these general categories of information. Tips: What Kind of Information Should Your Data Collection Instruments Contain? => Check that the data collection instruments are understandable, appropriate, and culturally relevant.

Meet with your stakeholders so they can review pre-existing instruments, ones you have adapted or want to adapt, and ones you are developing.

They can:

• Provide qualitative input about the categories that are most relevant locally;
• Suggest important questions, and help you think through how best to ask them; and
• Help you test out your instruments BEFORE you start data collection. Ask them some of your most difficult questions and find out if they understand them, or provide the type of answers you expect to get.
• Help you detect problems ahead of time and make needed changes before you collect data. Apply the instruments with them and ask them to tell you about how they arrived at their answers, and what those answers mean.

To ensure your instruments fit the local setting. Tips: Some Things to Keep in Mind When Developing or Choosing Questions and Categories.

=> Once you have designed your instrument.

• Make sure the data collection instrument includes data items (questions, observations, checklists, etc) that are specific for each of the indicators you have identified, based on the objectives for which your agency or program is responsible (i.e., the immediate and possibly intermediate results)
• Make sure the instrument is designed to collect a small amount of data about the gaps you have identified in your Causal Pathway
• Reconsider whether or not the instrument can be used for baseline, midline (if relevant), end-line and/or comparison group. Remember that the same questions will be asked at different times or to different groups to measure change
• Make sure the instrument is designed to collect information that can be used to measure changes that can realistically take place between its first application and the next application. Tips: Is There Enough Time to Make a Difference?