# Automatic differentiation

## Idea

Given a function defined by a section of computer code, automatic differentiation produces a piece of computer code which, except for certain extreme cases, computes its derivative function. This page accumulates some notes of it.

## Details

### Forward rules

Extended number system based rules for forward automatic differentiation (here a prime denotes derivative with respect to some basic variable – let’s call this $x$ – and $u$ denotes some general expression):

(1)$variable \quad x \quad gives \quad \langle x, 1 \rangle$
(2)$variable \quad u \ne x \quad gives \quad \langle u, 0 \rangle$
(3)$constant \quad c \quad gives \quad \langle c, 0 \rangle$
(4)$\langle u,u'\rangle +\langle v,v'\rangle = \langle u+v, u'+v' \rangle$
(5)$\langle u,u'\rangle -\langle v,v'\rangle = \langle u-v, u'-v' \rangle$
(6)$\langle u,u'\rangle *\langle v,v'\rangle = \langle u v, u' v+u v' \rangle$
(7)$\langle u,u'\rangle /\langle v,v'\rangle = \left\langle \frac{u}{v}, \frac{u' v-u v'}{v^2} \right\rangle \quad ( v\ne 0)$
(8)$\sin\langle u,u'\rangle = \langle \sin(u) , u' \cos(u) \rangle$
(9)$\cos\langle u,u'\rangle = \langle \cos(u) , -u' \sin(u) \rangle$
(10)$\exp\langle u,u'\rangle = \langle \exp u , u' \exp u \rangle$
(11)$\log\langle u,u'\rangle = \langle \log(u) , u'/u \rangle \quad (u \gt 0)$
(12)$\langle u,u'\rangle^k = \langle u^k , k u^{k-1} u' \rangle \quad (u \ne 0)$
(13)$\left| \langle u,u'\rangle \right| = \langle \left| u \right| , u' \sign u \rangle \quad (u \ne 0)$

For a two argument function

(14)$g(\langle u,u' \rangle , \langle v,v' \rangle ) = \langle g(u,v) , g^{(1)}(u,v) u' + g^{(2)}(u,v) v' \rangle$

where $g^{(1)}$ and $g^{(2)}$ are the derivatives of $g$ with respect to its first and second arguments, respectively.

Will chrome allow me to post this? No, the browser is deleting the user’s content: cool feature!