6 Interventions for HCHHHU: technical report
overall internal validity of studies from 1 (weakest)
to 5 (strongest). This scale aims to answer
‘effectiveness’ questions. The higher the score,
the better a study generally controls for four main
threats to internal validity: causal relation, history,
selection bias, and chance factor. The following is
the five-point scale employed in this SREA:
Level 1 Single group single point (post-test only or
correlational study)
Level 2 Single group pre- and post-test OR
non-equivalent control group (with no
adjustment in analysis)
Level 3 Cluster randomised trial with only one
cluster in each arm OR non-random
cluster OR non-equivalent control group
pre- and post-test design where outcome
= change in pre/post-test score (with no
other adjustment in analysis)
Level 4 Non-randomised controlled trial where
groups are demonstrated to be equivalent
on important variables (includes studies
where post-hoc analyses are used to
create equivalent groups - e.g. path
analysis or structural equations modelling)
Level 5 Randomised controlled trial with cluster
or individual allocation of multiple
individuals/clusters into groups
The Maryland scale scores were translated into
‘High’, ‘Medium’ and ‘Low’ scores for Weight of
Evidence B using the framework given in Appendix
2.3.
iii. The relevance of the study topic focus (from
the sample, measures, scenario, or other
indicator of the focus of the study) to the SREA
question (WoE C)
iv. An overall weight taking in account A, B, C
(WoE D) using weight average of WoE A, B, C,
except that WoE D cannot be higher than WoE B
2.7.4 Synthesis of evidence on outcomes
Three steps were undertaken to synthesise the
evidence on outcomes.
Step one
The first stage of the analysis was to compare
the different studies to assess whether they
were sufficiently homogenous in terms of their
contexts (i.e. family and service characteristics)
to allow synthesis across studies. This was done
prior to analysing the results from the individual
studies. It is argued that the characteristics of
the household units and the types of services that
are co-ordinated/integrated, although varied, are
sufficiently similar for a synthesis of the impact of
co-ordination/integration of services efforts for
these household units to be carried out across all
the studies in the review.
Step Two
The second stage consisted of the synthesis of
results from the individual studies. The variety of
types of outputs and outcomes measured precluded
the calculations of outcomes in standard formats
and thus meta-analysis. A narrative numerical
approach to synthesis was therefore used.
Synthesis across study outcomes took place within
categories identified in the stage one analysis. Alls
the outcomes for each study were included in their
relevant category and coded as either positive or
negative.
A positive finding indicates that the study result
favours the intervention or control group i.e.
the co-ordination/integration effort produced a
better result. A negative coding means simply the
opposite. Whether or not studies were statistically
significant (where relevant) was noted but not used
as part of the judgment. A coding of a positive
or negative result could only be undertaken for
outcomes where some indication of a change in
that outcome has been measured. For studies that
used a single group post-test or cross-sectional
design (i.e. the outcomes were only measured on
completion of the project without any possibility of
comparative data), this was not possible. In some
of the studies the review team was able to create
comparison data using the intervention group as
its own control by examining the reports for any
baseline data that may be used for this purpose.
Otherwise the outcome was excluded from the
synthesis.
The synthesis within each category took into
account each study’s Weight of Evidence score.
This was used as a means of exploring patterns of
results within each category.
2 Sherman LW, Gottfredson DC, MacKenzie DL, Eck J, Reuter P, Bushway SD (1998) Preventing crime: what works, what doesn’t, what’s
promising. National Institute of Justice Research in Brief. Washington: US Department of Justice.