Dealing with Dynamic Systems:... 39
the last criterion, this definition is also applicable to the systems SINUS and
altOl.
What are the main goals for future research? Three tasks will be outlined
briefly: (1) a differentiation between factors influencing complex problem
solving that come from individual, situational and system attributes; (2)
reliability and validity research on complex problem solving scenarios; (3)
adequate measurement of the actual “mental model” and of the potential
heuristics that complex problem solvers use, also over time.
Concerning the first task, separation of person, situation, and system
influences on performance measures, the approach taken by Streufert et al.
(1988) seems to point into an interesting direction: Instead of using “free”
simulations in which decisions can change a system’s state quite drastically,
they use a “quasi-experimentai simulation technology,” in which the system
reacts in part independently of subjects’ interventions such that each subject
receives comparable information and events. Despite this fact, subjects still
believe that they have a direct or delayed impact on the system. This tech-
nique should be explored further in order to standardize the conditions under
which subjects’ performance quality is measured independent of system
attributes. Also, one has to make clear how this kind of pseudo-feedback
affects knowledge acquisition or even prevents it. In the case of small-scale
systems like SINUS the technique may not be indicated, but in large-scale
systems like LOHHAUSEN the real feedback appears not to be of central
importance. In this latter case, strange constellations may seem plausible
because even in case of correct feedback not all the information can be
processed by the problem solver. In the former case of a deterministic
small-scale system, pseudo-feedback would quickly cause irritation.
Even if subjects did not detect different degrees of side effects or Eigendy-
namik—an argument used recently by Strohschneider (1991) in a critique of
the present experimental approach—the effects of this manipulation on the
dependent measures cannot be denied. Also, Brehmer and Allard (1991) point
to the critical role of system characteristics. In their studies, introducing
different degrees of feedback delay leads to a detrimental problem solving
behavior (see also Sterman, 1989).
Concerning task 2, reliability and validity aspects, there is a good deal of
work to be done, and it is not quite clear how to proceed: Up to now,
researchers mainly pointed to the face validity of their tasks and dependent
measures. Jager (1986, p. 274) called this “uncovered checks” that have to
be cashed in subsequent research. It is simply not enough to show that there
are no correlations between dynamic tasks and standard intelligence tests
because many reasons can account for such results. Rather, one has to show
positive connections to other psychometric instruments as well as to external
criteria. Reliability studies have to demonstrate that the indices used do not
show “random walk” characteristics. Much progress would also be possible
if at least results from empirical studies were subjected to replication. By
now, such replications are widely missing. If no direct assessment of relia-