problem at hand. Given the mountainous error surface that is characteristic for product
unit networks, a local search algorithm such as backpropagation of gradient descent
errors is ineffective and usually converges to local minima. In contrast, global search
algorithms such as, for example, the Alopex procedure have heuristic strategies to help
escape from local minima.
The success of global search procedures in finding a global minimum of a given
function such as Q (xu 1, yu 1, w) over w eW hinges on the balance between an
exploration process, a guidance process and a convergence inducing process (Hassoun
1995). The exploration process gives the search a mechanism for sampling a
sufficiently diverse set of parameters w in W. The Alopex6 procedure performs an
exploration process that is stochastic in nature. The guidance process is an implicit
process that evaluates the relative quality of search points [i.e., two consecutive search
points] and uses correlation guidance to move towards regions of higher-quality
solutions in the parameter space. Finally the convergence-inducing process ensures the
convergence of the search to find a fixed solution w* . The convergence-inducing
process is realised effectively by a parameter T, called temperature, that is gradually
decreased over time. The dynamic interaction among these three processes is
responsible for giving the Alopex search process its global optimising character.
4.2 The Alopex Procedure
Consider a training data set (xu 1, yu 1 ) with u1 -1,..., U1. We assume that the data was
generated by some true underlying function g(x). Our objective is to learn the
parameter w = (wk | k = 1,...,3H ) of the approximating function ΩSL ( x, w) whose form
is dependent upon the choice of H.
Alopex is a correlation-based method for solving the parameter estimation problem.
The error function Q is minimised by means of weight changes that are calculated for
the n-th step (n > 2) of the iteration process in batch mode as follows7:
wk (n) = wk (n -1) + ^ sgn (* - Pk (n))
(28)
17