The Azimuth Project
Blog - Exploring regression on the El Niño data (Rev #4, changes)

Showing changes from revision #3 to #4: Added | ~~Removed~~ | ~~Chan~~ged

Regression using an $l_{1/2}$ prior

We can write the regression cost function—adding an $l_{1/2}$ prior—using explicit summations as

cost {cost}_{2} = \sum_{e = 1}^{E} {(\sum_{i = 1}^{P} M_{i}^{(e)} x_{i} - y^{(e)})}^{2} + λ \sum_{i = 1}^{P} \sqrt{| x_{i} |} | x_{i} |^{2}

~~cost~~ cost_2 = \sum_{e=1}^{E} \left( \sum_{i=1}^P M^{(e)}_{i} x_{i} ~~\right)^2~~ - y^{(e)}\right)^2 + \lambda \sum_{i=1}^P ~~\sqrt{|x_i|}~~ |x_i|^2

cost_1 = \sum_{e=1}^{E} \left( \sum_{i=1}^P M^{(e)}_{i} x_{i} - y^{(e)}\right)^2 + \lambda \sum_{i=1}^P |x_i|

cost_{1/2} = \sum_{e=1}^{E} \left( \sum_{i=1}^P M^{(e)}_{i} x_{i} - y^{(e)}\right)^2 + \lambda \sum_{i=1}^P \sqrt{|x_i|}

Regression regularization paths

If we restrict to one variable from the numerous vectors and denote this variable by $x$ , we get

\frac{\partial cost}{\partial x} = A x + B + \frac{\lambda sgn(x)}{2\sqrt{|x|}} = 0

where $A$ and $B$ don’t depend on $x$ . If we denote $\lambda sgn(x)/2$ by $C$ , we can multiply through by $y =\sqrt{|x|}$ to find the minimum (along this co-ordinate) is

\pm A y^3 + B y + C = 0

where the $\pm$ depends whether $x$ is positive/negative and all subject to needing to ensure the solutions in $y$ are also consistent with the original equation. Since this is a cubic equation we have a simple closed form for the solutions to this equation and hence can efficiently solve the original equation.

Revision from September 12, 2014 02:59:22 by davidtweed

The Azimuth Project Blog - Exploring regression on the El Ni&ntilde;o data (Rev #4, changes)

Regression using an l 1/2l_{1/2} prior

The Azimuth Project
Blog - Exploring regression on the El Niño data (Rev #4, changes)

Regression using an $l_{1/2}$ prior