Chapter 2 Methods 5
2.5 Data extraction
The studies remaining after application of the
criteria were data-extracted using a set of
questions developed specifically for this SREA (see
Appendix 2.2). All data-extraction was carried out
on EPPI-Reviewer.
2.6 Quality assurance process
All report authors were involved in screening and
coding. Prior to screening, the inclusion criteria
were discussed and modified by all the team
members. A screening exercise was then completed
where the whole team applied the criteria to ten
different papers that could possibly have been
included in the review. The results were compared
and discussed and final modifications to the
inclusion criteria made. During the course of the
screening the Principal Investigator (MN) double-
screened a random sample of 10 papers coded
by each team member. One disagreement was
identified for five out of the seven coders and none
for the other two. The majority of the differences
were in fact errors (usually the fact the publication
date was prior to the cut-off point had been
overlooked).
Prior to the coding, a similar exercise was
undertaken whereby all members of the team
coded one paper included in the review.
Differences in interpretation were discussed and
the coding framework modified. Papers were
re-screened for inclusion during the process of
coding, usually by a different member of the team.
The Principal Investigator reviewed screening and
coding decisions on one third of the potentially
included papers at this stage.
The data produced from the coding were checked
again prior to analysis. Any coding errors were
identified and corrected by checking against the
study report.
2.7 In-depth review
2.7.1 Moving from broad
characterisation (mapping) to in-depth
review
Although they addressed the broad review
question, the studies in the map were quite
diverse, addressing a number of distinct sub-
questions. It was therefore necessary to identify
and prioritise a specific question for the in-depth
review. A refined in-depth review question with
an additional inclusion/exclusion criterion was
developed after the review team consulted with
advisory group members on the results of the
mapping analysis.
The initial selection criteria included a need to
focus on families where multiple problems spanned
multiple generations. However it was recognised
that in many of the studies in the map the second
generation in question were young children and
thus the extent to which the ‘problems’ could
be considered truly multi-generational was
questionable. It was therefore decided that the
in-depth review would focus on studies where
there were clearly two distinct generations of the
household with multiple problems.
The in-depth review question was
How effective are interventions that aim
to improve the delivery of services to
multi-generational HCHhHU through
integration/co-ordination mechanisms at
producing improved outcomes (broadly defined)?
Studies were excluded from in-depth analysis if:
• The ‘target group’ for the service provision in
the study did NOT explicitly include families in
which ‘problem’ or ‘poor outcomes’ span two
or more generations of secondary school age or
above (N.B. studies that referred to the younger
of the two generations as youth, juvenile,
adolescent, teenager were included)
2.7.2 Detailed description of studies in
the in-depth review
Studies identified as meeting the additional set of
inclusion criteria were coded using the in-depth
analysis coding framework. The in-depth analysis
coding framework was an expanded version of the
coding framework used at the mapping stage. An
additional set of questions were added to extract
more details on characteristics of HCHHHU and
service delivery, sampling designs, methods used in
studies, and study results.
2.7.3 Assessing quality of studies and
weight of evidence for the review
question
Studies that were included in the in-depth review
were assessed for quality and relevance. Three
components of Weight of Evidence (WoE) were
used to help in making explicit the process of
apportioning different weights to the findings and
conclusions of different studies. A study’s WoE was
based on the following:
i. The soundness of studies (internal
methodological coherence), based upon the
study only (WoE A)
ii. The appropriateness of research design and
analysis for addressing question, or sub questions
of this SREA using the Scientific Methods
(Maryland) scale (WoE B)
The University of Maryland’s Department of
Criminology and Criminal Justice developed the
Maryland scale, a five-point scale to rank the