the process does not converge. If T is too small, a premature convergence to a local
minimum might occur.
Initial Values
The algorithm has three parameters: the initial temperature T, the number of iterations,
N, over which the correlations are averaged for annealing, and the step size δ. The
temperature T and the N-iterations cycles seem to be of secondary importance for the
final performance of the algorithm. The initial temperature T may be set to a large value
of about 1,000. This allows the algorithm to get an estimate of the average correlation
in the first N iterations and reset it to an appropriate value according to Equation (31). N
may be chosen between 10 and 100. In contrast to T and N, δ is a critical parameter
that has to be selected with care. There is no way to a priori identify δ .
The Termination Criterion
An important issue associated with network training is the termination criterion. The
main goal of training is to minimise the learning error while ensuring good model
generalisation. It has been observed that forceful training may not produce network
models with adequate generalisation ability, although the learning error achieved is
small. The most common remedy for this problem is to monitor model performance
during training to assure that further training improves generalisation as well as reduces
learning error. For this purpose an additional set of validation data, independent from
the training data is used.
In a typical training phase, it is normal for the validation error to decrease. This trend
may not be permanent, however. At some point the validation error usually reverses or
its improvement is extremely slow. Then the training process should be stopped. In our
implementation of the Alopex procedure network training is stopped when к - 40,000
consecutive iterations are unsuccessful.
к has been chosen so large at the expense of the greater training time, to ensure more
reliable estimates. Of course, setting the number of unsuccessful iterations to 40,000 (or
more) does not guarantee that there would be any successful steps ahead if training
19