Much of what is discussed below is either referred to as non-equilibrium thermodynamics or stochastic thermodynamics.
A continuous-time, finite state Markov process is a pair $(V,H)$ where $V$ is a finite set of states and $H : \mathbb{R}^V \to \mathbb{R}^V$ is an infinitesimal stochastic,
Hamiltonian generating the time evolution of a population distribution $p(t) \in \mathbb{R}^V$ via the master equation
The infinitesimal stochastic condition requires that the off-diagonal components of $H$ are non-negative and that the columns sum to zero, meaning that $H_{ii} = -\sum_{j \neq i} H_{ji}$, or that the diagonal elements are equal to minus the sum of all the other elements in that column. The entry $H_{ij}$ is interpreted as the transition rate from state $j$ to state $i$. Thus the diagonal entries are equal to minus the sum of the outgoing rates from a certain state.
An equilibrium distribution $q \in \mathbb{R}^V$ is a population distribution that does not change with time,
In terms of the indices, the master equation reads:
We can use the infinitesimal stochastic property of $H$ to rewrite this as:
If we define the current from $j$ to $i$ as $J_{ij} = H_{ij}p_j - H_{ji}p_i$, we can write the master equation in the simple form:
Note that this definition of current requires an implicit choice that a positive current $J_{ij}$ corresponds to a net flow population into the $i^{\text{th}}$ state.
We say an equilibrium distribution $q$ satisfies detailed balance if
Equilibrium distributions always satisfy $\frac{dq_i}{dt} = \sum_j J_{ij} = 0$ for all $i$, but for an equilibrium satisfying detailed balance each term in this sum vanishes identically.
For an equilibrium not satisfying detailed balance there can be non-zero individual currents flowing $J_{ij} \neq 0$ in such a way that $\sum_j J_{ij} = 0$ is still satisfied. Such an equilibrium is typically referred to as a non-equilibrium steady state.
One can represent a Markov process as a directed graph $(V,E,s,t)$, where $V$ is a finite set of states, $E$ is a finite set of edges, and $s,t : E \to V$ are source and target maps respectively, along with a map $r : E \to [0,\infty)$ giving the labels or the transition rates associated to each edge, i.e. for an edge $e \in E$, $r(e) = H_{ t(e) s(e) }$.
Given a labelled, directed graph $G=(V,E,s,t,r)$ one can forget the directedness of the edges and consider the undirected graph $(V,E)$. A spanning tree is then a connected subgraph containing all the vertices $V$ of $G$, but with no cycles. A connected undirected graph with no cycles is called a tree. Given a spanning tree $T = (V,E_T )$ the leftover edges $E-E_T$ form the set of chords of the graph. Adding any chord to the spanning tree produces a cycle. The set of cycles obtained when adding each of the chords individually to a given spanning tree provides one choice of a cycle basis for the underlying graph $G$.
For Markov processes one cannot discard the information regarding the direction of the edges. Similarly we will need a directed cycle basis. To achieve this we choose an arbitrary orientation on all of the cycles in our basis, such as clockwise. It is easy to see that for a graph $G$ with $n$-vertices $|V|=n$ a spanning tree $T=(V,E_T)$ on those n-vertices will have $n-1$ edges, $|E_T| = n-1$. Therefore we have $\nu = |E|-|E_T| = |E| - n + 1$ chords and hence $\nu$ elements in our cycle basis. Therefore let us denote the cycle basis by $\{C_1,C_2,....C_{\nu} \}$.
Given an arbitrary directed cycle $C$ on the graph $G$ we can write $C = \sum_{\alpha=1}^{\nu} (C,C_{\alpha})C_{\alpha}$. Here we have introduced an inner product on cycle space $(C,C_{\alpha}) = S_{\alpha}(C)S_{\alpha}(C_{\alpha}) \quad 1 \leq \alpha \leq \nu$, where
B. Altaner, Foundations of Stochastic Thermodynamics Altaner’s PhD thesis. Contains lots of information.
B. Altaner, S. Grosskinsky, S. Herminghaus, L. Katthan, M. Timme, and J. Vollmer, Network representations of nonequilibrium steady states: Cycle decompositions, symmetries, and dominant paths. Very interesting paper. Describes mapping from state space to cycle space such that even Markov processes not satisfying detailed balance map into processes defined on the cycle space satisfying detailed balance in that space.
B. Mélykúti , J. P. Hespanha , M. Khammash, Equilibrium distributions of simple biochemical reaction systems for time-scale separation in stochastic reaction networks. Describes a number of relevant issues such as separation of time scales, deletion of fast states, as well as gluing together two Markov processes at one vertex.
Xue-Juan Zhang, Hong Qian, and Min Qian, Stochastic theory of nonequilibrium steady states and its applications. Part I. First in a two-part review covering many aspects of nonequilibrium steady states.
Hao Ge, Hong Qian, and Min Qian, Stochastic theory of nonequilibrium steady states and its applications. Part II: Applications to chemical biophysics.
Udo Seifert, Stochastic thermodynamics, fluctuation theorems, and molecular machines. This review of stochastic thermodynamics might be interesting. “Various integral and detailed fluctuation theorems, which are derived here in a unifying approach from one master theorem, constrain the probability distributions for work, heat and entropy production depending on the nature of the system and the choice of non-equilibrium conditions. For non-equilibrium steady states, particularly strong results hold like a generalized fluctuation-dissipation theorem involving entropy production. Ramifications and applications of these concepts include optimal driving between specified states in finite time, the role of measurement-based feedback processes and the relation between dissipation and irreversibility. Efficiency and, in particular, efficiency at maximum power, can be discussed systematically beyond the linear response regime for two classes of molecular machines, isothermal ones like molecular motors, and heat engines like thermoelectric devices, using a common framework based on a cycle decomposition of entropy production.”
J. Schnakenberg, Network theory of microscopic and macroscopic behavior of master equation systems. Seminal reference for the network theory of Markov processes.
Melissa Vellela and Hong Qian, Stochastic dynamics and non-equilibrium thermodynamics of a bistable chemical reaction system: the Schl $\ddot{\text{o}}$gl model revisited.
C. Van Den Broeck, Stochastic thermodynamics: A brief introduction.
George Oster,Alan Perelson, Aharon Katchalsky, Network Thermodynamics. Discusses state variables, constitutive relations, Kirchhoff’s laws in the context of chemical reactions and reversible/irreversible thermodynamics.
Matteo Polettini and Massimiliano Esposito, Irreversible thermodynamics of open chemical networks I: Emergent cycles and broken conservation laws.
J. L. Wyatt, Network representation of reaction-diffusion systems far from equilibrium. Develops the network theory of chemical reaction systems from first principles.
S. Sieniutycz and J. S. Shiner, Variational and Extremum Properties of Homogeneous Chemical Kinetics. I. Lagrangian- and Hamiltonian-Like Formulations
S. Sieniutycz and J. S. Shiner, Variational and Extremum Properties of Homogeneous Chemical Kinetics. II. Minimum Dissipation Approaches