BMC Medical Research Methodology 2008, 8:45
http://www.biomedcentral.com/1471-2288/8/45
Quality assessment
Assessing the quality of qualitative research has attracted
much debate and there is little consensus regarding how
quality should be assessed, who should assess quality,
and, indeed, whether quality can or should be assessed in
relation to 'qualitative' research at all [43,22,44,45]. We
take the view that the quality of qualitative research
should be assessed to avoid drawing unreliable conclu-
sions. However, since there is little empirical evidence on
which to base decisions for excluding studies based on
quality assessment, we took the approach in this review to
use 'sensitivity analyses' (described below) to assess the
possible impact of study quality on the review's findings.
In our example review we assessed our studies according
to 12 criteria, which were derived from existing sets of cri-
teria proposed for assessing the quality of qualitative
research [46-49], principles of good practice for conduct-
ing social research with children [50], and whether studies
employed appropriate methods for addressing our review
questions. The 12 criteria covered three main quality
issues. Five related to the quality of the reporting of a
study's aims, context, rationale, methods and findings
(e.g. was there an adequate description of the sample used
and the methods for how the sample was selected and
recruited?). A further four criteria related to the sufficiency
of the strategies employed to establish the reliability and
validity of data collection tools and methods of analysis,
and hence the validity of the findings. The final three cri-
teria related to the assessment of the appropriateness of the
study methods for ensuring that findings about the barri-
ers to, and facilitators of, healthy eating were rooted in
children's own perspectives (e.g. were data collection
methods appropriate for helping children to express their
views?).
Extracting data from studies
One issue which is difficult to deal with when synthesis-
ing 'qualitative' studies is 'what counts as data' or 'find-
ings'? This problem is easily addressed when a statistical
meta-analysis is being conducted: the numeric results of
RCTs - for example, the mean difference in outcome
between the intervention and control - are taken from
published reports and are entered into the software pack-
age being used to calculate the pooled effect size [3,51].
Deciding what to abstract from the published report of a
'qualitative' study is much more difficult. Campbell et al.
[11] extracted what they called the 'key concepts' from the
qualitative studies they found about patients' experiences
of diabetes and diabetes care. However, finding the key
concepts in 'qualitative' research is not always straightfor-
ward either. As Sandelowski and Barroso [52] discovered,
identifying the findings in qualitative research can be
complicated by varied reporting styles or the misrepresen-
tation of data as findings (as for example when data are
used to 'let participants speak for themselves'). Sand-
elowski and Barroso [53] have argued that the findings of
qualitative (and, indeed, all empirical) research are dis-
tinct from the data upon which they are based, the meth-
ods used to derive them, externally sourced data, and
researchers' conclusions and implications.
In our example review, while it was relatively easy to iden-
tify 'data' in the studies - usually in the form of quotations
from the children themselves - it was often difficult to
identify key concepts or succinct summaries of findings,
especially for studies that had undertaken relatively sim-
ple analyses and had not gone much further than describ-
ing and summarising what the children had said. To
resolve this problem we took study findings to be all of
the text labelled as 'results' or 'findings' in study reports -
though we also found 'findings' in the abstracts which
were not always reported in the same way in the text.
Study reports ranged in size from a few pages to full final
project reports. We entered all the results of the studies
verbatim into QSR's NVivo software for qualitative data
analysis. Where we had the documents in electronic form
this process was straightforward even for large amounts of
text. When electronic versions were not available, the
results sections were either re-typed or scanned in using a
flat-bed or pen scanner. (We have since adapted our own
reviewing system, 'EPPI-Reviewer' [54], to handle this
type of synthesis and the screenshots below show this
software.)
Detailed methods for thematic synthesis
The synthesis took the form of three stages which over-
lapped to some degree: the free line-by-line coding of the
findings of primary studies; the organisation of these 'free
codes' into related areas to construct 'descriptive' themes;
and the development of 'analytical' themes.
Stages one and two: coding text and developing descriptive themes
In our children and healthy eating review, we originally
planned to extract and synthesise study findings according
to our review questions regarding the barriers to, and facil-
itators of, healthy eating amongst children. It soon
became apparent, however, that few study findings
addressed these questions directly and it appeared that we
were in danger of ending up with an empty synthesis. We
were also concerned about imposing the a priori frame-
work implied by our review questions onto study findings
without allowing for the possibility that a different or
modified framework may be a better fit. We therefore tem-
porarily put our review questions to one side and started
from the study findings themselves to conduct an the-
matic analysis.
Page 4 of 10
(page number not for citation purposes)