AN ANALITICAL METHOD TO CALCULATE THE ERGODIC
AND DIFFERENCE MATRICES OF THE DISCOUTED MARKOV
DECISION PROCESSES
Jan Kaiuski
The Silesian Technical University of Gliwice
Departament of Organization and Management
Chair of Computer Science and Econometrics
ul.Roosevelt,26 - 28, 41 - 800 Zabrze, Poland
Fax: (48 32) 237 21 77
Tel: (48 32) 2777 354
E-mail: [email protected]
Summary: In the work the analitical method to calculate the ergodic
and difference matrices of finite state discouted Marcov decision processes
is presented. On the basis well - known literature the result for overall disco-
unted value, this one in interpretation of the calculated matrices is shown.
The obtained results gives a possibility to distinguish the constant and va-
riable parts of the overall discounted value.
The presented analitical method is illustrated by two simple examples. New
performance index to discounted optimal Markov control problem is propo-
sed.
1 Introduction
Marcov Decision Processes (MDP) are also called Controlled Marcov Processes
or Marcov Processes wiht reward since 1960, when Howard [25] introduced them,
ones became extremely attractive research tool and they found wide application
in different technical and research disciplines. From the beginning it can be ob-
served that the method is more and more improved [6, 9, 10, 11, 12, 16, 19].
The improvement concerns connections of Markov Processes with different me-
thods of mathematical programming [3, 10, 11, 12, 17, 20, 22, 24, 25, 29, 30, 34,
35, 38, 42], their optimization and searching of computational methods of diffe-
rent matrices which are connected with stochastic Marcov matrix of transitions.
Simultaneously with development of MDP (some scientists think that even ear-
lier) developed discipline which is connected with game theory, namely - stocha-
stic games [4, 5, 6, 15, 17, 21, 23, 28, 31, 33, 36, 37, 38, 40]. Quite not long
ago on the basis of two topics: stochastic games and Markov Decision Processes
arose new discipline connected with competition in Marcov Decision Processes
(so called Competitive Marcov Decision Processes) [18].
Especially practical importance for development of mentioned methods has the-
ory of irreducible Marcov Decision Chains with finite set of states and given