The Azimuth Project
Automatic differentiation (Rev #3)

Automatic differentiation

Idea

Given a function defined by a section of computer code, automatic differentiation produces a piece of computer code which, except for certain extreme cases, computes its derivative function. This page accumulates some notes of it.

Details

Forward rules

Extended number system based rules for forward automatic differentiation (here a prime denotes derivative with respect to some basic variable – let’s call this xx – and uu denotes some general expression):

(1)variablexgivesx,1 variable \quad x \quad gives \quad \langle x, 1 \rangle
(2)variableuxgivesu,0 variable \quad u \ne x \quad gives \quad \langle u, 0 \rangle
(3)constantcgivesc,0 constant \quad c \quad gives \quad \langle c, 0 \rangle
(4)u,u+v,v=u+v,u+v \langle u,u'\rangle +\langle v,v'\rangle = \langle u+v, u'+v' \rangle
(5)u,uv,v=uv,uv \langle u,u'\rangle -\langle v,v'\rangle = \langle u-v, u'-v' \rangle
(6)u,u*v,v=uv,uv+uv \langle u,u'\rangle *\langle v,v'\rangle = \langle u v, u' v+u v' \rangle
(7)u,u/v,v=uv,uvuvv 2(v0) \langle u,u'\rangle /\langle v,v'\rangle = \left\langle \frac{u}{v}, \frac{u' v-u v'}{v^2} \right\rangle \quad ( v\ne 0)
(8)sinu,u=sin(u),ucos(u) \sin\langle u,u'\rangle = \langle \sin(u) , u' \cos(u) \rangle
(9)cosu,u=cos(u),usin(u) \cos\langle u,u'\rangle = \langle \cos(u) , -u' \sin(u) \rangle
(10)expu,u=expu,uexpu \exp\langle u,u'\rangle = \langle \exp u , u' \exp u \rangle
(11)logu,u=log(u),u/u(u>0) \log\langle u,u'\rangle = \langle \log(u) , u'/u \rangle \quad (u \gt 0)
(12)u,u k=u k,ku k1u(u0) \langle u,u'\rangle^k = \langle u^k , k u^{k-1} u' \rangle \quad (u \ne 0)
(13)|u,u|=|u|,usignu(u0) \left| \langle u,u'\rangle \right| = \langle \left| u \right| , u' \sign u \rangle \quad (u \ne 0)

For a two argument function

(14)g(u,u,v,v)=g(u,v),g (1)(u,v)u+g (2)(u,v)v g(\langle u,u' \rangle , \langle v,v' \rangle ) = \langle g(u,v) , g^{(1)}(u,v) u' + g^{(2)}(u,v) v' \rangle

where g (1)g^{(1)} and g (2)g^{(2)} are the derivatives of gg with respect to its first and second arguments, respectively.

Will chrome allow me to post this? No, the browser is deleting the user’s content: cool feature!

References