Steady State Markov Chains

Steady state Markov chains is the idea that as the time period heads towards infinity then a two state Markov chain’ state vector will stabilise.

$T_{s}=lim_{n \rightarrow \infty}[T]^{n}$

If we keep multiplying the initial state vector like we showed in the last video, the initial state input becomes less important and the conditional probability input more dominant in the final answer.

Transition matrix decides the likelihood over a large time period what the probability of the system is.

Equation of Steady State

Luckily for us there is an easier way to find the steady state than putting the conditional probability to the power of infinity. Once we reach the steady state we will find that the next time period will be equal to the previous one that is:

$|X_{n+1}|=|X_{n}|$

Our standard Markov chain probability calculation is:

$|X_{n+1}|=|T||X_{n}|$

To get the elements of our transition matrix we will set the probabilities to:

$Pr(X_{n+1}=1|X_{n}=0)=a$
$Pr(X_{n+1}=0|X_{n}=1)=b$
$Pr(X_{n+1}=0|X_{n}=0)=1-a$
$Pr(X_{n+1}=1|X_{n}=1)=1-b$

Therefore if we substitute these values into the transition matrix, and the current state in for the the future state we get, a new matrix that has two equations and two unknowns therefore we can expand the matrix, and solve for the steady state probabilities:

$\left|\begin{array}{cc}Pr(X_{n}=0)\\Pr(X_{n}=1)\end{array}\right|= \begin{bmatrix}{1-a} &{a} \\ {b} &{1-b}\end{bmatrix} \left|\begin{array}{cc}Pr(X_{n}=0)\\Pr(X_{n}=1)\end{array}\right|$

$Pr(X_{n}=0)=(1-a)Pr(X_{n}=0)+bPr(X_{n}=1)$

$Pr(X_{n}=1)=aPr(X_{n}=0)+(1-b)bPr(X_{n}=1)$

Now we also know that since it is a two state system:

$ $Pr(X_{n}=1)=1-Pr(X_{n}=0)$

Now we can sub this into equation one:

$Pr(X_{n}=0)=(1-a)Pr(X_{n}=0)+b[1-Pr(X_{n}=0)]$

$Pr(X_{n}=0)[1-(1-a)+b]=b$

$Pr(X_{n}=0)=\frac{b}{a+b}$

Substitute this into equation 3:

$Pr(X_{n}=1)=1-\frac{b}{a+b}=\frac{a}{a+b}$

Exam Practice question

A Multiple Choice Question from Exam 2 in 2013

As soon as we are asked about the long term probability then we know that we need to find the steady state of a markov chain.

Step 1: Set each option to a state

$Pr(X_{n}=0)=drives$

$Pr(X_{n}=1)=bus$

Step 2: Find the probabilities of a and b

$Pr(X_{n+1}=1|X_{n}=0)=a=1-\frac{3}{5}=\frac{2}{5}$
$Pr(X_{n+1}=0|X_{n}=1)=b=1-\frac{7}{10}=\frac{3}{10}$

Step 3: Use the equation for steady state.

$Pr(X_{n}=1)=\frac{a}{a+b}=\frac{\frac{2}{5}}{\frac{2}{5}+\frac{3}{10}}\newline Pr(X_{n}=1)=\frac{\frac{2}{5}}{\frac{7}{10}}=\frac{4}{7}$

So the multiple choice answer is C

It is critical to state which is your state one and state zero at the start of the question to limit any confusion in the question.

Feedback

Want to suggest an edit? Have some questions? General comments? Let us know how we can make this resource more useful to you.

While we endeavour to provide you with great study material - we’re not qualified teachers, as such Engage cannot guarantee the validity of the information here.

The Victorian Curriculum and Assessment Authority (VCAA) does not endorse this website and makes no warranties regarding the correctness or accuracy of its content. VCE is a registered trademark of the VCAA. Current and past VCE exams and related content can be accessed directly at www.vcaa.vic.edu.au

Contents