Using Surveys Effectively: What are Impact Surveys?



Designing a survey methodology

Who will train and supervise the
data collectors?
What local resources
exist, how much will these cost and what
continued support they can ensure, in
the form of supervision and further
training?

How will you ensure respondents
answer truthfully?
Interviewees may
be motivated to respond in particular
ways - for example, those who have
recently joined may want to stress how
poor they are to justify their
membership. To get the most truthful
answers make sure your data collectors
are well-trained. Also make sure they
explain the purpose of the survey to the
respondents.

How will you motivate your data
collectors?
You will also need to think
about how much your staff are prepared
to get involved in the survey exercise,
and how to motivate them to collect the
best quality data possible. Case study 3
gives some suggestions.

How much pilot testing is
necessary?
There is no blueprint
answer to this, but your testing should
leave you feeling confident about your
questionnaire, about the skills of your
data collection team, and about your
ability to make use of all the data
collected. It is also essential that pilot
testing includes some initial analysis of
data, to assess where there might be
problems.

7 Select samples

Decide on your sample size
Conducting a study of all your clients is
usually very costly and time-consuming.
The solution is to select a sample - a
small number of clients who are
randomly selected from all clients or
from a certain group. Samples have to
be of sufficient
size to produce
statistically valid data, and yet not so big
that they become unmanageable and
costly. The appropriate sample size for
your survey will also depend on the
issues you want to investigate and how
much time you can spend analysing the
data.

The groups can be, for example, non-
clients, early stage clients or mature
clients. You should plan to have not less
than 30 respondents in each group
(called a sub-sample). If you want to
also make other comparisons, for
example, between male and female
clients in each of these groups then you
will need to double the sample size (i.e.
from 30 to 60). If you want to compare
the difference between e.g. rural and
urban respondents as well, you will need
to double this number again, bringing it
to 120. As you can see from this
example, sample sizes increase quickly
depending on the number of dimensions
of analysis involved. This again
emphasises how important it is to be
clear about your survey’s objectives from
the start, as it is not possible to
introduce these dimensions afterwards if
you have not planned for them.

Decide how you will sample
You need a sampling strategy. This
means sorting individuals into categories
and then randomly selecting them for
the survey. You will also need a strategy
for selecting alternative respondents,
because it is always the case that you
will not find some of the people or that
they do not want to be interviewed. You
must give clear guidance to interviewers
on how to deal with these situations and
find an alternative respondent. Another
problem you might have to face is
deciding how to include returning or
“resting” clients in different sample
groups (see
Imp-Act Practice Note 3 on
Client Exit).

8 Data processing
and analysis

Always check your data after you
have collected it

Once you have collected your data you
need to enter it into the database
system, which will allow statistical
analysis. It will also need “cleaning”
(checking for inconsistencies). It is a
good idea to train the people entering
data to look for strange entries, such as
a household with 148 members, so they
can go back to the original data to
check. The tasks associated with data
inputting can be tedious and time-
consuming, and it is therefore important
to plan ahead who will be doing them,
and how these people will be trained and
motivated if this is in-house.

Plan how you will analyse the
statistical data

After the data has been entered into a
database, you can begin statistical
analysis with appropriate computer
software. This will involve carrying out
statistical testing exercises that require
considerable skills. You need to plan for
this stage, so that it does not delay your
results. Decide in advance who will do
the analysis and how it will be done.
Statistical analysis software often
requires on-going training and support
that you will need to build into your
budget and plans.

Interpret your data

The key to good data analysis is to
examine the results of the survey and
look for different possible interpretations
of the findings. You need to consider the
strengths and weaknesses of the survey
design and how it was conducted in
practice, when drawing out conclusions.
The findings have to be compared
against your hypotheses to see how far
the evidence supports the expectations
the MFI has about impact. For example,
if your hypothesis was that participation
in the programme leads to increased
self-esteem for women clients, you will
need to consider what the data tells you
and think about whether the changes
observed are clear and strong - for
example, can you say that the majority
of women clients have greater self-
esteem than before. Can you attribute
this to your programme’s activities? In
drawing your conclusions you also need
to consider whether the changes have
been brought about by external factors
not related to your programme.

PAGE SIX I M P-ACT PRACTICE N OTES NUMBER FOUR 2005



More intriguing information

1. The name is absent
2. The name is absent
3. The Social Context as a Determinant of Teacher Motivational Strategies in Physical Education
4. A Rare Case Of Fallopian Tube Cancer
5. The name is absent
6. The name is absent
7. POWER LAW SIGNATURE IN INDONESIAN LEGISLATIVE ELECTION 1999-2004
8. Om Økonomi, matematik og videnskabelighed - et bud på provokation
9. Short report "About a rare cause of primary hyperparathyroidism"
10. Literary criticism as such can perhaps be called the art of rereading.
11. The name is absent
12. The name is absent
13. FOREIGN AGRICULTURAL SERVICE PROGRAMS AND FOREIGN RELATIONS
14. Bird’s Eye View to Indonesian Mass Conflict Revisiting the Fact of Self-Organized Criticality
15. Reform of the EU Sugar Regime: Impacts on Sugar Production in Ireland
16. The name is absent
17. Mean Variance Optimization of Non-Linear Systems and Worst-case Analysis
18. WP 48 - Population ageing in the Netherlands: Demographic and financial arguments for a balanced approach
19. The name is absent
20. The name is absent