The Azimuth Project
Markov processes in non-equilibrium thermodynamics



Much of what is discussed below is either referred to as non-equilibrium thermodynamics or stochastic thermodynamics.

Markov processes and the master equation

A continuous-time, finite state Markov process is a pair (V,H) (V,H) where V V is a finite set of states and H: V VH : \mathbb{R}^V \to \mathbb{R}^V is an infinitesimal stochastic,

(1)H ij0ijand iH ij=0, H_{ij} \geq 0 \; \: i \neq j \quad \text{and} \quad \; \sum_i H_{ij} = 0,

Hamiltonian generating the time evolution of a population distribution p(t) Vp(t) \in \mathbb{R}^V via the master equation

(2)dpdt=Hp. \frac{dp}{dt} = Hp .

The infinitesimal stochastic condition requires that the off-diagonal components of HH are non-negative and that the columns sum to zero, meaning that H ii= jiH jiH_{ii} = -\sum_{j \neq i} H_{ji}, or that the diagonal elements are equal to minus the sum of all the other elements in that column. The entry H ijH_{ij} is interpreted as the transition rate from state jj to state ii. Thus the diagonal entries are equal to minus the sum of the outgoing rates from a certain state.

An equilibrium distribution q Vq \in \mathbb{R}^V is a population distribution that does not change with time,

(3)dqdt=Hq=0. \frac{dq}{dt} = Hq = 0.


In terms of the indices, the master equation reads:

(4)dp idt= jH ijp j. \frac{dp_i}{dt} = \sum_{j} H_{ij}p_j.

We can use the infinitesimal stochastic property of HH to rewrite this as:

(5)dp idt= jH ijp jH jip i. \frac{dp_i}{dt} = \sum_{j} H_{ij}p_j - H_{ji}p_i.

If we define the current from jj to ii as J ij=H ijp jH jip iJ_{ij} = H_{ij}p_j - H_{ji}p_i, we can write the master equation in the simple form:

(6)dp idt= jJ ij. \frac{dp_i}{dt} = \sum_j J_{ij}.

Note that this definition of current requires an implicit choice that a positive current J ijJ_{ij} corresponds to a net flow population into the i thi^{\text{th}} state.

Detailed Balance

We say an equilibrium distribution qq satisfies detailed balance if

(7)H ijq j=H jiq i. H_{ij}q_j = H_{ji}q_i.

Equilibrium distributions always satisfy dq idt= jJ ij=0\frac{dq_i}{dt} = \sum_j J_{ij} = 0 for all ii, but for an equilibrium satisfying detailed balance each term in this sum vanishes identically.

Non-equilibrium steady states

For an equilibrium not satisfying detailed balance there can be non-zero individual currents flowing J ij0J_{ij} \neq 0 in such a way that jJ ij=0\sum_j J_{ij} = 0 is still satisfied. Such an equilibrium is typically referred to as a non-equilibrium steady state.

Representation of Markov processes using labelled graphs

One can represent a Markov process as a directed graph (V,E,s,t) (V,E,s,t) , where VV is a finite set of states, EE is a finite set of edges, and s,t:EVs,t : E \to V are source and target maps respectively, along with a map r:E[0,)r : E \to [0,\infty) giving the labels or the transition rates associated to each edge, i.e. for an edge eEe \in E, r(e)=H t(e)s(e)r(e) = H_{ t(e) s(e) } .

The cycle basis

Given a labelled, directed graph G=(V,E,s,t,r)G=(V,E,s,t,r) one can forget the directedness of the edges and consider the undirected graph (V,E)(V,E). A spanning tree is then a connected subgraph containing all the vertices VV of GG, but with no cycles. A connected undirected graph with no cycles is called a tree. Given a spanning tree T=(V,E T)T = (V,E_T ) the leftover edges EE TE-E_T form the set of chords of the graph. Adding any chord to the spanning tree produces a cycle. The set of cycles obtained when adding each of the chords individually to a given spanning tree provides one choice of a cycle basis for the underlying graph GG.

For Markov processes one cannot discard the information regarding the direction of the edges. Similarly we will need a directed cycle basis. To achieve this we choose an arbitrary orientation on all of the cycles in our basis, such as clockwise. It is easy to see that for a graph GG with nn-vertices |V|=n|V|=n a spanning tree T=(V,E T)T=(V,E_T) on those n-vertices will have n1n-1 edges, |E T|=n1|E_T| = n-1. Therefore we have ν=|E||E T|=|E|n+1\nu = |E|-|E_T| = |E| - n + 1 chords and hence ν\nu elements in our cycle basis. Therefore let us denote the cycle basis by {C 1,C 2,....C ν}\{C_1,C_2,....C_{\nu} \}.

Given an arbitrary directed cycle CC on the graph GG we can write C= α=1 ν(C,C α)C αC = \sum_{\alpha=1}^{\nu} (C,C_{\alpha})C_{\alpha}. Here we have introduced an inner product on cycle space (C,C α)=S α(C)S α(C α)1αν(C,C_{\alpha}) = S_{\alpha}(C)S_{\alpha}(C_{\alpha}) \quad 1 \leq \alpha \leq \nu , where

(8)S α(C)={+1 if α||C 1 if α||C 0 if αC. S_{\alpha}(C) = \left\{ \begin{array}{ccc} +1 & \text{if} & \alpha || C \\ -1 & \text{if} & -\alpha || C \\ 0 & \text{if} & \alpha \notin C. \end{array} \right.

Stochastic Thermodynamics


Entropy, relative entropy, and entropy production