зо
Joachim Funke
deviations become increasingly higher the farther away they are from the
target state. A good discussion of the frequently used RMS criterion can be
found in Poulton (1973) and Bosser (1983).
There is, however, an aspect which reflects an ugly property of this kind
of system performance evaluation: Assume that an operator has knowledge
about the system. Reaching the goal is of little or no difficulty for him and
the resulting RMS will be low. But what can we say about the operator with
little or no knowledge? The resulting distance to the goal state as measured
by the RMS criterion varies as a function of his (random) interventions.
Depending on the weight matrices of the system, this would result in a large
variety of measured distances. Therefore, different values of the RMS, in this
case, do not reflect different degrees of quality of system performance. The
argument here is one of different reliabilities of the RMS criterion for
different states of an operator’s knowledge, being best in the case of correct
knowledge (RMS indicating reliable values near zero) and worst in the case
of purely random intervention (RMS indicating an enormous range of values
due to decreasing reliability).
One potential solution for this problem is a logarithmic transformation of
the RMS. This transformation leads to an evaluation of distances which is
more efficient: Larger distances are no longer weighted more heavily. Rather,
they are considered less important by this measure. It does not matter if
someone missed the goal by 10,000 or 100,000 points. This difference has
the same importance as the difference between a deviation of 1 and 10. The
transformation, thus, reduces the error variance that increases as a function
of the operator’s distance to the goal state. In the experimental section the
variable “QSC” refers to this kind of dependent variable (“Quality of System
Control”—a low QSC reflecting a good score because of low discrepancies
between goal values and the values subjects reached on the endogenous
variables through their control behavior).
Measuring the structural knowledge an operator has acquired about a
system requires also some kind of distance or similarity measurement. In this
case the distance exists between the structural relations hypothesized by
subjects and those implemented in the system. For this purpose, the operator
marks on a sheet (or in some versions directly on the screen) the assumed
causal relationships at certain points in time. The problems with this kind of
measurement are: (1) Response tendency: Subjects differ in their degree with
which they indicate relations in cases they are not quite sure about. Therefore,
one has to count both hits (i.e., Conespondence between assumed and existing
relation) and false alarms (like in conventional recognition tests). (2) Differ-
ent quality of knowledge: On the lowest level it is only assumed that a
relationship between two variables exists (relational knowledge). On the next
higher level the sign of the relation is known (sign knowledge). In the optimal
case the numerical weights are known (numerical knowledge). (3) Function-
ality instead of correctness: False models can be useful for system control,
at least within a restricted area of values. Similarity measures are blind to this