The Azimuth Project
Automatic differentiation (Rev #1)

Automatic differentiation


Given a function defined by a section of computer code, automatic differentiation produces a piece of computer code which, except for certain extreme cases, computes its derivative function. This page accumulates some notes of it.


Forward rules

Extended number system based rules for forward automatic differentiation:

(1)u,u+v,v=u+v,u+v \langle u,u'\rangle +\langle v,v'\rangle = \langle u+v, u'+v' \rangle
(2)u,uv,v=uv,uv \langle u,u'\rangle -\langle v,v'\rangle = \langle u-v, u'-v' \rangle
(3)u,u*v,v=uv,uv+uv \langle u,u'\rangle *\langle v,v'\rangle = \langle u v, u' v+u v' \rangle
(4)u,u/v,v=uv,uvuvv 2(v0) \langle u,u'\rangle /\langle v,v'\rangle = \left\langle \frac{u}{v}, \frac{u' v-u v'}{v^2} \right\rangle \quad ( v\ne 0)
(5)sinu,u=sin(u),ucos(u) \sin\langle u,u'\rangle = \langle \sin(u) , u' \cos(u) \rangle
(6)cosu,u=cos(u),usin(u) \cos\langle u,u'\rangle = \langle \cos(u) , -u' \sin(u) \rangle
(7)expu,u=expu,uexpu \exp\langle u,u'\rangle = \langle \exp u , u' \exp u \rangle
(8)logu,u=log(u),u/u(u>0) \log\langle u,u'\rangle = \langle \log(u) , u'/u \rangle \quad (u \gt 0)
(9)u,u k=u k,ku k1u(u0) \langle u,u'\rangle^k = \langle u^k , k u^{k-1} u' \rangle \quad (u \ne 0)
(10)|u,u|=|u|,usignu(u0) \left| \langle u,u'\rangle \right| = \langle \left| u \right| , u' \sign u \rangle \quad (u \ne 0)