Maximum, Minimum, or Saddle Point?
It is quite common that one wants the maximum or minimum of something. If your company's annual budget is in relationship with the number of your products, your maintenance frequency, the maintained quality, and so on, you would like to know how to organize these factors to get a minimum budget. Similarly, you would like to know how to plan your sales to maximize your business value.
If there's a functional relationship between your inputs (product amount, maintenance...) and the budget, you can leave the computation for the smartest organization plan to a computer. All you have to do is enter your function, give an initial guess for the inputs, and the computer should be able to spit out the point of minimum after some programming.
The programming itself is not the topic we are discussing today, but it should be useful to understand a bit of the basic idea behind it, which is looking for an extreme point.
Single-variable case
We know from single-variate calculus that if we want to find a maximum or a minimum of a function, we look for a point where the derivative is 0 (called a stable point). The derivative is the slope of the function's tangent line at a point.
Alter the variable a to see the slope of tangent line at point (a, f(a)).
At maxima or minima, the slope of the tangent line is 0. Here we are only interested in local extrema. The global ones are hard to find, because the function goes on forever. So we often just consider the stable points in a given domain of the variable
Multivariable case (two variables)
The multivariable case is similar. Suppose you have two inputs and one output, like the function shown below, where is represented as the height above each input point on the x-y plane. It's a curved surface, so you'd want the gradient of the tangent plane at some point to be 0, which means a horizontal tangent plane.
First, let's get warmed up with some basic multivariable calculus (if you know what's the gradient of a tangent plane, you could skip to "Now here comes the problem").
In multivariable calculus, when you want to calculate the derivative of a function with respect to one variable, you pretend that other variables are constants. If you have a point on the curved surface below, say , taking the derivative of the function with respect to x at point can be interpreted like this: Slice the curved surface with a plane (pretend that y is a constant) to get an intersection curve, and the derivative with respect to x is the slope of the intersection curve's tangent line at point . And when you do some calculations, that's just .
Alter point A(a, b) to see the derivative with respect to x at that point. You can move the graph with your finger or with your mouse to observe it from different angles.
Likewise, you may imagine the derivative of the function with respect to y. Slice the curved surface with the plane to get an intersection curve, and our derivative is just the slope of the curve's tangent line at that point, and which is just .
Alter point A(a, b) to see the derivative with respect to y at that point.
The tangent plane at point has to do with these two tangent lines, or more precisely, is the plane defined by these two tangent lines. So if both derivatives with respect to and are zero, then this tangent plane should be horizontally flat. In this case, the point that satisfies this requirement is , which we see as a minimal point.
Now here comes the problem. Assume that we've found a point where the derivatives of the function with respect to both variables are 0 (often written as and ), namely found a stable point, how do we know if it's a maximum, a minimum, or even a saddle point (neither maximum nor minimum)? Of course, you could plot out the graph of the function, and observe the graph around the point you got.
Mimimum point
Maximum point
Saddle point
But sometimes the function is too complicated to be plotted out, too demanding and time-consuming for your computer. Or it's even impossible to do so.
If we can't see the graph, then it's as if we are standing on some kind of mountain with our eyes blind. The gradient is zero, so you're comfortable standing on your feet, not somehow tending to slide down. You want to know what the landscape is like around you. You stick out your walking stick in all directions to reach the places around you and feel that all the places around seem to be below your feet. Tada! You've found yourself on a mountaintop. Similarly, if all the points are above you, you are at a minimal point; if there are both points above you and below, you're standing on a saddle point.
When handling a function, however, it's impossible to compute all the points around your stable point, because there are infinitely many points at some certain radius around. The problem for us is to find out a way to identify a stable point as a maximum, minimum, or saddle point, without having to analyze all the points around.