The Role of Evidence in Establishing Trust in Repositories

The Role of Evidence in Establishing Trust in Repositories

planning or the documentary review, the template can be adjusted on an audit-by-audit basis
to allow the auditors to probe for the necessary information.

8. Conclusion and Next Steps

Evidence will play a crucial role in the process of repository certification. Without an agreed
base of evidence against which to validate the checklist criteria, audits are likely to lack
consistency and will depend too much on judgement(s) that may prove difficult to replicate,
substantiate, or validate. Unless, therefore, a checklist is associated with a defined evidence
base, its usefulness is diminished. Here we have considered the kinds of evidence that might
provide auditors with necessary information to assess the levels of risk associated with a
particular repository and to determine whether it should be certified as worthy of trust. In
order to conceive an 'objective' and usable resource it is vital that any checklist offers
repositories and auditors the means to understand the criteria necessary to achieve a 'worthy
of trust' status in measurable (although not necessarily quantifiable) terms, and offers clear
insights into how they might determine whether their own institution meets the criteria. This
approach will not only facilitate the audit process, but will also assist institutions engaged in
establishing new repositories in defining the processes and types of documentation they
should put in place and maintain if they are to ensure that their organisation is 'working
smart' and that it is 'audit ready'. Even existing repositories may benefit from guidance on the
kinds of documentation that auditors are likely to seek when assessing levels of trust.
Although the community is a year or more away from spinning out audit and certification
procedures, it is not too early to consider the kinds of documentation that a repository should
be keeping.

While we have suggested kinds of evidence that might underpin the use of repository
checklists, metrics for measuring compliance or the level²⁵ at which a repository meets a
particular criterion require more research. As a corollary we might consider Chapin and
Akridge's reflection that '[t]raditional security metrics are haphazard at best; at worst they
give a false impression of security that leads to inefficient or unsafe implementation of
security measures' [2]. This is a scenario that the repository community should wish to avoid.
And, if we are to avoid it, we need to establish a secure evidence base and agree metrics for
evaluating it.

There is a downside to considering the repository audit and certification process from the
point of view of evidence - it makes it readily apparent how much effort will be involved in
the audit process and how high the cost is likely to be in a way that checklists alone do not.
On the other hand, by considering the evidence and underlying processes, at an early stage,
repositories will be able to contain costs through adopting best practices.

Finally as a community we need to consider how other audit and certification tools might be
integrated with our emerging checklists or tailored to meet our needs. As Hans Hofman, of
the Dutch National Archives, has observed on many occasions, any new methods need to be
placed in the larger audit context that includes such approaches as the COSO framework for
audit,²⁶ COBIT framework (Control Objectives for Information and Related
Technologies),²⁷ ITIL (IT Infrastructure Library) service management,²⁸ ISO 9000 family of
quality management and assurance standards,²⁹ and ISO 17799 for information security.³⁰
The digital repository building and management community, such as libraries and archives,
does not appear so far to have paid sufficient attention to these other strands of activity and
tools. They do have much to offer us and, we believe, we have much to learn from them.
This work should be undertaken alongside further refinement of evidence requirements and
development of metrics for assessing and measuring checklist compliance.

9. Contributors

<<   <   6   7   8   9   10   11   12   >   >>