Designing a survey methodology
Who will train and supervise the
data collectors? What local resources
exist, how much will these cost and what
continued support they can ensure, in
the form of supervision and further
training?
How will you ensure respondents
answer truthfully? Interviewees may
be motivated to respond in particular
ways - for example, those who have
recently joined may want to stress how
poor they are to justify their
membership. To get the most truthful
answers make sure your data collectors
are well-trained. Also make sure they
explain the purpose of the survey to the
respondents.
How will you motivate your data
collectors? You will also need to think
about how much your staff are prepared
to get involved in the survey exercise,
and how to motivate them to collect the
best quality data possible. Case study 3
gives some suggestions.
How much pilot testing is
necessary? There is no blueprint
answer to this, but your testing should
leave you feeling confident about your
questionnaire, about the skills of your
data collection team, and about your
ability to make use of all the data
collected. It is also essential that pilot
testing includes some initial analysis of
data, to assess where there might be
problems.
7 Select samples
Decide on your sample size
Conducting a study of all your clients is
usually very costly and time-consuming.
The solution is to select a sample - a
small number of clients who are
randomly selected from all clients or
from a certain group. Samples have to
be of sufficient size to produce
statistically valid data, and yet not so big
that they become unmanageable and
costly. The appropriate sample size for
your survey will also depend on the
issues you want to investigate and how
much time you can spend analysing the
data.
The groups can be, for example, non-
clients, early stage clients or mature
clients. You should plan to have not less
than 30 respondents in each group
(called a sub-sample). If you want to
also make other comparisons, for
example, between male and female
clients in each of these groups then you
will need to double the sample size (i.e.
from 30 to 60). If you want to compare
the difference between e.g. rural and
urban respondents as well, you will need
to double this number again, bringing it
to 120. As you can see from this
example, sample sizes increase quickly
depending on the number of dimensions
of analysis involved. This again
emphasises how important it is to be
clear about your survey’s objectives from
the start, as it is not possible to
introduce these dimensions afterwards if
you have not planned for them.
Decide how you will sample
You need a sampling strategy. This
means sorting individuals into categories
and then randomly selecting them for
the survey. You will also need a strategy
for selecting alternative respondents,
because it is always the case that you
will not find some of the people or that
they do not want to be interviewed. You
must give clear guidance to interviewers
on how to deal with these situations and
find an alternative respondent. Another
problem you might have to face is
deciding how to include returning or
“resting” clients in different sample
groups (see Imp-Act Practice Note 3 on
Client Exit).
8 Data processing
and analysis
Always check your data after you
have collected it
Once you have collected your data you
need to enter it into the database
system, which will allow statistical
analysis. It will also need “cleaning”
(checking for inconsistencies). It is a
good idea to train the people entering
data to look for strange entries, such as
a household with 148 members, so they
can go back to the original data to
check. The tasks associated with data
inputting can be tedious and time-
consuming, and it is therefore important
to plan ahead who will be doing them,
and how these people will be trained and
motivated if this is in-house.
Plan how you will analyse the
statistical data
After the data has been entered into a
database, you can begin statistical
analysis with appropriate computer
software. This will involve carrying out
statistical testing exercises that require
considerable skills. You need to plan for
this stage, so that it does not delay your
results. Decide in advance who will do
the analysis and how it will be done.
Statistical analysis software often
requires on-going training and support
that you will need to build into your
budget and plans.
Interpret your data
The key to good data analysis is to
examine the results of the survey and
look for different possible interpretations
of the findings. You need to consider the
strengths and weaknesses of the survey
design and how it was conducted in
practice, when drawing out conclusions.
The findings have to be compared
against your hypotheses to see how far
the evidence supports the expectations
the MFI has about impact. For example,
if your hypothesis was that participation
in the programme leads to increased
self-esteem for women clients, you will
need to consider what the data tells you
and think about whether the changes
observed are clear and strong - for
example, can you say that the majority
of women clients have greater self-
esteem than before. Can you attribute
this to your programme’s activities? In
drawing your conclusions you also need
to consider whether the changes have
been brought about by external factors
not related to your programme.
PAGE SIX • I M P-ACT PRACTICE N OTES • NUMBER FOUR • 2005