AN ANALYTICAL METHOD TO CALCULATE THE ERGODIC AND DIFFERENCE MATRICES OF THE DISCOUNTED MARKOV DECISION PROCESSES

5 Conclusions and comments

Identical results would be obtained for ν∞ (β) if we use directly formula (10)
(passing over difficulties connected with inverse matrix (I - βP)) . Using formu-
la (19) we can separate two components of total reward; a constant component
connected with 1/ (1 - β) factor and ergodic matrix and variable component
which represents this part of ν∞ (β) which arises under the influence of unste-
ady transient process. Effect of this process is especially visible during the initial
phase of Markov Decision Process. Value of this part of component of quantity
ν_∞ (β) rises along with decrease of discounted factor β and increase of disturban-
ces which are generated by matrix P . Two presented examples show it.

We can create, relying on ab ove observations, a performance index of tested Di-
scounted Markov Decision Processes. Let ν_∞¹ (β) mean component which stand
in front of (1-β), and ν∞ (β), k = 2, 3,..., N mean components which stand in
front of 1_-01_κβ. Then performance index of mentioned Markov Chain can have
the following form:

Pκ=2,3,... ^ν∞,i (^β)
^ν∞,i (^β)

From definition of coefficient J (β) for given β results that when absolute value
of this coefficient is more close zero, then better properties have tested Markov
Process. For presented two examples, values of coefficients amount to:

We observe that given stochastic matrix P always generates the same transient
process for n = 0, 1, 2 (it means Pⁿ). It results from calculation that effect of the
process depends on value of β. Hence optimisation of Discounted Markov Deci-
sion Process can rely on selection of adequately large factor β for given quality
coefficient. But usually β is given and depends on different economic-technical
conditions. Then optimisation MDP can rely on selection of adequately matrix

13

<<   10   11   12   13   14   15   16   >>

More intriguing information