It is possible, using the distributions given above, to define
the information transmitted from the Y to the Y process using
the Shannon source uncertainty of the strings:
^. .. .∙^. .. .^. . ^.
I (Y, Y) ≡ H (Y ) - H (Y | Y = H (Y ) + H (Y) - H (Y, Y),
(10)
where H (..., ...) is the joint and H (...|...) the conditional
uncertainty (Cover and Thomas, 1991; Ash, 1990).
If there is no uncertainty in Y given the retranslation Y,
then no information is lost, and the systems are in perfect
synchrony.
In general, of course, this will not be true.
The rate distortion function R(D) for a source Y with a
distortion measure d(y, y) is defined as
R(D)= min I (Y,Y).
P(y,y);P(y,y) p(y)p(y∣y)d(y,y)≤D
(11)
(13)
X is the message, Y the channel, and the probability dis-
tribution P (X) is chosen so as to maximize the rate of infor-
mation transmission along a Y.
Finally, recall the analogous definition of the rate distor-
tion function above, again an extremum over a probability
distribution.
Recall, again, equations (4-8), i.e., that the free energy of a
physical system at a normalized inverse temperature-analog
K = 1∕κT is defined as F (K ) = - log[Z (K )]/K where Z(K )
the partition function defined by the system Hamiltonian.
More precisely, if the possible energy states of the system are
a set Ei, i = 1, 2, ... then, at normalized inverse temperature
K , the probability of a state Ei is determined by the relation
P[Ei] = exp[-EiK]/ Pj exp[-EjK].
The partition function is simply the normalizing factor.
Applying this formalism, it is possible to extend the rate
distortion model by describing a probability distribution for
D across an ensemble of possible rate distortion functions in
terms of available free metabolic energy, K = 1∕κMτ.
The key is to take the R(D) as representing energy as a
function of the average distortion. Assume a fixed K , so that
the probability density function of an average distortion D,
given a fixed K, is then
The minimization is over all conditional distributions p(y∣y)
for which the joint distribution p(y,y) = p(y)p(y∣y) satisfies
the average distortion constraint (i.e., average distortion ≤
D).
The Rate Distortion Theorem states that R(D) is the min-
imum necessary rate of information transmission which en-
sures communication does not exceed average distortion D.
Thus R(D) defines a minimum necessary channel capacity.
Cover and Thomas (1991) or Dembo and Zeitouni (1998) pro-
vide details. The rate distortion function has been explicitly
calculated for a number of simple systems.
Recall, now, the relation between information source un-
certainty and channel capacity (e.g., Ash, 1990):
P[D, K] = exp[ R(D)K]-----.
Dmax exp[-R(D)K]dD
Dmin
(14)
Thus lowering K in this model rapidly raises the possibility
of low distortion communication between linked systems.
We define the rate distortion partition function as just the
normalizing factor in this equation:
(12)
H[X] ≤ C,
(15)
Dmax
ZR [K] ≡
Dmin
exp[-R(D)K]dD,
where H is the uncertainty of the source X and C the
channel capacity, defined according to the relation (Ash, 1990)
again taking K = 1∕κMτ.
We now define a new free energy-analog, the rate distortion
free-energy, as
C≡ max I(X|Y).
P(X)
Fr[K] ≡-1oZ.. K]],
K