The Role of Evidence in Establishing Trust in Repositories

The Role of Evidence in Establishing Trust in Repositories

be sent to and returned by the target repository to provide auditors with a profile of the
institution's technical architecture, organisational structure, and financial position. It will give
auditors information concerning such areas as security, performance, and management
control. Supporting documentation will be requested and reviewed in advance of the on-site
audit. These materials will, for instance, give auditors material to support decisions about
where and how to probe during a visit, and in some cases evidence to ascertain where
processes, procedures, and practices are adequate. It will enable the audit team to establish an
'Audit Plan'. This will facilitate the identification of areas where observation of practice,
interviews, checking of documentation and testing (e.g., disaster recovery tests, evaluation of
stratified random samples of digital objects at different points in their lifecycle) should be
used. The scope and nature of these data collection instruments and the types of
documentation requested will be refined as the DCC and others working in this area, such as
RLG/NARA and nestor, develop a richer understanding of the information requirements
necessary to assess repositories as an outcome of pilot audits.

Each of the three DCC pilot audits will produce three types of output, each meeting a
particular need:

• First will be a 'confidential report' for the participating repository itself, detailing the
results of the evaluation, offering suggestions for future developments that might
improve their effectiveness, processes, and documentation procedures, giving guidance
as to how the repository could use the audit tools to manage regular internal audits, and
indicating how the repository could better prepare for future externally run audits.

• Second, in consultation with the audited repository the DCC audit team will make
publicly available a report appraising the audit approaches it employed and indicating
the kinds of improvements that should be made to the process.

• Revised criteria and descriptions of the audit process will be delivered to RLG/NARA
and nestor as the third anticipated outcome of each pilot audit.

4. What Is the Evidence Base?

Significant intellectual effort has been committed to the identification of the necessary
technological, organisational, and financial characteristics repositories must have if they are
to be granted a kite-mark of trustworthiness. This is perhaps realised most notably within the
nestor and RLG-NARA audit checklists [6, 8]. The issue of the categories of evidence
necessary to facilitate audits and enable certification needs to be given adequate
consideration: any tool that omits to describe the evidence that will contribute to the audit
process is incomplete. If an audit checklist has aspirations of practical applicability, its
criteria must detail not only the expected and required standards, but also the means by
which their attainment can be demonstrated and assessed. Similarly, if such tools are to
promote self-assessment of a rigorous and reliable kind, and to be likely to provide a good
predictor for the outcome of an independent external audit, then they must be comprehensive,
either independently or in combination with one or more linked resources. With no indication
of its acceptable evidence base a checklist for structuring and guiding the audit and
certification of a repository has mainly theoretical value. It lacks practical applicability and
does not support unbiased measurement. It becomes too open to interpretation, and a risk
arises that it will be extrapolated to endorse even those repositories with recognisable
shortcomings. Current work does not, so far, focus adequately on the evidence base; a further
development stage is necessary to conceive a document that is practically useful within an
audit. Efforts must probe for evidence of concrete processes, structures, and functionality.

In reviewing the audit tools that are being developed [6, 8] we have identified and reported a
gap in the documentation requirements necessary to provide an evidence base for measuring
repository compliance with the expectations for best practices as outlined in the checklists.

<<   <   1   2   3   4   5   6   7   >   >>