Steady State Markov Chains

Steady State Markov Chains

Steady state Markov chains is the idea that as the time period heads towards infinity then a two state Markov chain’ state vector will stabilise.

T_{s}=lim_{n \rightarrow \infty}[T]^{n}


If we keep multiplying the initial state vector like we showed in the last video, the initial state input becomes less important and the conditional probability input more dominant in the final answer.

Transition matrix decides the likelihood over a large time period what the probability of the system is.

Equation of Steady State

Luckily for us there is an easier way to find the steady state than putting the conditional probability to the power of infinity. Once we reach the steady state we will find that the next time period will be equal to the previous one that is:


Our standard  Markov chain probability calculation is:


To get the elements of our transition matrix we will set the probabilities to:

  • Pr(X_{n+1}=1|X_{n}=0)=a

  • Pr(X_{n+1}=0|X_{n}=1)=b

  • Pr(X_{n+1}=0|X_{n}=0)=1-a

  • Pr(X_{n+1}=1|X_{n}=1)=1-b

Therefore if we substitute these values into the transition matrix, and the current state in for the the future state we get, a new matrix that has two equations and two unknowns therefore we can expand the matrix, and solve for the steady state probabilities:

 \left|\begin{array}{cc}Pr(X_{n}=0)\\Pr(X_{n}=1)\end{array}\right|= \begin{bmatrix}{1-a} &{a} \\ {b} &{1-b}\end{bmatrix} \left|\begin{array}{cc}Pr(X_{n}=0)\\Pr(X_{n}=1)\end{array}\right|



Now we also know that since it is a two state system:


Now we can sub this into equation one:




Substitute this into equation 3:


Exam Practice question

A Multiple Choice Question from Exam 2 in 2013

Markov steady state

As soon as we are asked about the long term probability then we know that we need to find the steady state of a markov chain.

Step 1: Set each option to a state



Step 2: Find the probabilities of a and b

  • Pr(X_{n+1}=1|X_{n}=0)=a=1-\frac{3}{5}=\frac{2}{5}

  • Pr(X_{n+1}=0|X_{n}=1)=b=1-\frac{7}{10}=\frac{3}{10}

Step 3: Use the equation for steady state.

Pr(X_{n}=1)=\frac{a}{a+b}=\frac{\frac{2}{5}}{\frac{2}{5}+\frac{3}{10}}\newline Pr(X_{n}=1)=\frac{\frac{2}{5}}{\frac{7}{10}}=\frac{4}{7}

So the multiple choice answer is C

It is critical to state which is your state one and state zero at the start of the question to limit any confusion in the question.