Introduction to the Hessian Matrix

In mathematics, Hessian is a matrix of all partial derivatives of a function that is mostly used in linear algebra and to calculate local extreme points. It is used when a function is twice differentiable and continuous on a point. Let us learn more about the Hessian matrix and Hessian determinant. Before this, let's understand the Hessian first.

Understanding of the Hessian Matrix

The Hessian Matrix is a square matrix of the second derivatives of a function. It is also known as simply Hessian of a scalar-valued function. It was developed by the German mathematician Ludwig Otto Hesse in the 19th century and named after him. The order of the Hessian matrix can be 2 or 3 depending on the variables in a function. 

Hessian is used in machine learning and data science algorithms to optimize a desired function. We can define a Hessian matrix as;

“If a function is twice differentiable, and all of its partial derivatives exist, then we can represent these derivatives as a squared matrix which upon expanding determines local extreme points of the function.”

Hessian Matrix Formula

The Hessian matrix is used to find the curvature. For a function with n variables, the Hessian matrix is represented as;

If $n=2$ the Hessian matrix of 2nd order will be written as,

Where, the function f depends on two variables x and y. 

Discriminant of Hessian Matrix

The determinant of the Hessian matrix is known as the determinant of the Hessian matrix. For a function f(x, y), the discriminant of Hessian matrix is represented as;

The order of the Hessian matrix and discriminant will always be a squared matrix and equal to the number of the variables involved in the function.

Second Derivative Test and Hessian Matrix

The use of the Hessian matrix with the second derivative test allows us to calculate local extreme points of a function. There are three conditions used to determine local minima, local maxima and saddle point. If a function f(x, y) is twice differentiable twice, then for a point (a, b),

• If $f_{xx}(a,b)>0$ then the function $f(x, y)$ has local minimum and the discriminant will be $D(a,b)>0$.
• If $f_{xx}(a,b)<0$ then the function $f(x, y)$ has local maximum and the discriminant will be $D(a,b)>0$.
• The function $f$ has a saddle point if $D(a,b)<0$.
• The second derivative test fails when $D(a,b)=0$.

How to calculate the Hessian matrix?

The Hessian is calculated by finding all second-order partial derivatives of a function and finding the discriminant of these derivatives. Here are a few simple steps that will guide you to compute Hessian. 

1. Determine the order of the Hessian matrix by counting the variables involved in the function. For example, if a function has two variables, x and y, then Hessian's order will also be 2.
2. Calculate second-order partial derivatives with respect to x and y.
3. Substitute the values of derivatives in a matrix form. 
4. Find the determinant of the matrix obtained in step 3. Use the expansion method to find the determinant. 
5. Simplify to get a solution.

Let’s understand how to calculate the Hessian of a function in the following example.

Hessian Matrix Example

Compute the Hessian of the following function, 

$f(x,y)=x^3-2xy-y^6$ at the point $(1, 2)$

To calculate Hessian, we need to calculate a second order partial derivative of the given function. Let's calculate the partial derivative with respect to x first.

$f_x(x,y)=\frac{\partial}{\partial x}(x^3-2xy-y^6)$

We get, 

$f_x(x,y)=3x^2-2y$

Since we are required to calculate partial derivative of fx(x,y) with respect to y also. So, 

$f_{xy}(x,y)=-2$

Again differentiating with respect to x, 

$f_{xx}(x,y)=6x$

Now calculate the partial derivative with respect to y, we get, 

$f_y(x,y)=\frac{\partial}{\partial y}(x^3-2xy-y^6)$

$f_y(x,y)=0-2x-6y^5$

Derivative of fy(x,y) with respect to x is, 

$f_{yx}(x,y)=-2$

Now the second partial derivative with respect to y is,

$f_{yy}(x,y)=-30y^4$

Since the Hessian matrix formula is, 

Substituting the values of derivatives, 

At point (1, 2),

Comparison between Hessian Matrix and Jacobian

The comparison between the Hessian and Jacobian can be easily analysed using the following difference table.

Conclusion

In calculus, many fundamental applications of derivatives are used to solve complex problems. The Hessian matrix is an advanced application that allows us to find important information about a function. The Hessian matrix is also known as functional determinants. It has many applications in calculus and other fields of science, such as engineering. Because in engineering, it helps in the mathematical modeling of a problem.