OPTIMIZATION

The surface to the right is actually the graph of

$z = f(x, y) = \sin(x)\sin(y), -\pi < x, y < \pi$,

a Google map 'street view' of part of Yosemite, perhaps! It has mountains, Local Maxima, and it has valleys, Local Minima, both of which occured for curves. But it also has a pass through the mountains at which the terrain slopes up in one direction and down in another direction just like a saddle. So we also have to bring in the notion of Saddlepoints, corresponding to a pass through the mountains. But, just as in the one variable case, everything centers on an algebraic and graphical understanding of local extrema and critical points.

Definition:

A function $z=f(x, y)$ is said to have a Local Extremum at $(a, b)$ if

Local Maximum: $f(x, y) \le f(a, b)$ for all $(x, y)$ near $(a, b)$,

Local Minimum: $f(x, y) \ge f(a, b)$ for all $(x, y)$ near $(a, b)$.

In one variable locating local extrema usually meant finding where $f'(x) = 0$. For a function $f(x, y)$ let's see how $\nabla f$ gets involved: suppose $\nabla f(a, b) \ne 0$. Then the vector $\nabla f(a, b)$ will point in a direction of positive slope on the graph of $f$, and so we can find a larger value of $f(x, y)$ than $f(a, b)$ near $(a, b)$ by moving in that direction. But then $(a, b)$ cannot be a local maximum of $f$. How would you modify this argument for a local minimum (remember $-\nabla f$ points in the opposite direction to $\nabla f$)?

Definition:

A point $(a, b)$ is said to be a Critical Point of $f(x, y)$ when $\nabla f(a, b) = 0$ or at least one of $f_x(a, b), f_y(a, b)$ does not exist.

Two important observations:

1. if $(a, b)$ is a local extremum, then $(a, b)$ is a critical point.

2. if $f_x(a, b), f_y(a, b)$ both exist, the tangent plane is horizontal at the point $\,\,\,\,\,\,((a, b), f(a, b))$ on the graph of $f$ when $(a, b)$ is a critical point.

Critical points, along with the location and classification of local extrema, can thus be studied algebraically and graphically.

Example: to the right is the earlier graph of

$z = f(x, y) = \sin(x) \sin(y) , -\pi < x, y < \pi,$

now with axes. By the Product Rule,

$f_x(x, y) = \cos(x)\sin(y), f_y(x, y) = \sin(x) \cos(y).$

Thus $f_x, f_y$ are always defined for $-\pi < x, y < \pi$, so the only critical points occur at solutions of the equations $f_x = 0, f_y = 0$. These we find to be $(0,\, 0)$ and

$\displaystyle{\Big( \frac{\pi}{2}, \, \frac{\pi}{2} \Big), \Big( -\frac{\pi}{2}, \, \frac{\pi}{2} \Big), \Big(-\frac{\pi}{2}, \, -\frac{\pi}{2} \Big), \Big( \frac{\pi}{2},\, -\frac{\pi}{2} \Big).}$

The tangent plane is supposed to be horizontal at all these points. Is that clear from the graph?

The graph of $f(x, y) = \sin(x)\sin(y)$ also helps classify these critical points:

$f$ has a local maximum at $\displaystyle{\Big(\frac{\pi}{2},\,\frac{\pi}{2}\Big), \,\,\,\, \Big(-\frac{\pi}{2},\, -\frac{\pi}{2}\Big)}$,

$f$ has a local minimum at $\displaystyle{\Big(-\frac{\pi}{2},\, \frac{\pi}{2}\Big), \,\,\,\, \Big(\frac{\pi}{2}, -\frac{\pi}{2}\Big)}$.

Near $(0,\, 0)$, however, $f$ looks like a 'saddle', and we formalize this in general with

Definition:

A critical point $(a, b)$ of a function $f$ is said to be a Saddle Point if,
no matter how close $(x, y)$ is to $(a, b)$ there are always points at which $f(x, y) < f(a, b)$ and other points at which $f(x, y) > f(a, b)$.

So a critical point $(a, b)$ is either a local maximum, a local minimum, or a saddle point. But how can we determine which? In the case of $f(x, y) = \sin(x) \sin(y)$, knowing the graph was enough. There are other ways.

Using gradients: suppose we have a plot of the gradient
vectors of $f$ as shown to the right for

$f(x, y) = \sin(x)\sin(y)$

with the arrows indicating the direction of $(\nabla f)(x, y)$ and the length of the arrow indicating its length $|(\nabla f)(x, y)|$. This vector points in the direction of maximum slope of the graph of $f$ at $(x, y, f(x, y))$ and has length equal to the value of that maximum slope. So, as $(x, y)$ approaches a local maximum point $(a, b)$, the arrows all point inward to $(a, b)$ and get sucessively smaller. Why inward? This happens when $P = (a, b)$ is the local maximum in the first quadrant. At the local minimum $Q$ in the second quadrant, however, the arrows all point outward. Why outward?

But at the saddle point $R$, the arrows get successively smaller as $(x, y)$ approaches the origin, yet some arrows always point inward and some outward, because the graph slopes both up and down near $(0, 0)$. So what can you say about $S$ and $T$?

Contour map: suppose now we have the contour map for

$f(x, y) = \sin(x)\sin(y)$,

as shown to the right with the contours $f(x, y) = k$ shaded so that the darker the color the smaller $k$ is. Some contours are labelled with a specific value of $k$. At a local minimum like $Q$, the contours encircle $Q$ and the color darkens as the contours shrink to $Q$. Why shrink? And at a local maximum like $S$, the contours again encircle and shrink to $S$, except that now the colors lighten because the height increases as we approach a local maximum. At a saddle point, like $R$, however, the contours won't encircle $R$ because in some directions from $R$ the slope is increasing, while in others it is decreasing. Notice that at $R$ there are two straight contour lines through $R$ having the same height so the tangent plane at $R$ is certainly horizontal!

These make good visuals to explain what's going on, but for most functions the second order partial derivatives turn out to be more useful - just as it was for functions of one variable. Suppose $f(x, y)$ is a function with continuous second order partial derivatives and let

$A = f_{xx}(a, b),\qquad B = f_{xy}(a, b), \qquad C = f_{yy}(a, b)$

be the values of these second order derivatives at a point $(a, b)$. To determine whether $(a, b)$ is a local maximum, local minimum or a saddle point, we need to check the sign of the Discriminant

$D = D(a, b) = AC - B^2.$

Second Derivative Test: at a critical point $(a, b)$, a function $f$ has a

Local Maximum: if $D > 0$ and $f_{xx}(a, b) < 0$,

Local Minimum: if $D > 0$ and $f_{xx}(a, b) > 0$,

Saddle Point: if $D < 0$.

So what about $D = 0$? Well, here the test fails. It tells us nothing!!

The example of $f(x, y) = \sin(x) \sin(y)$ illustrates this well. For then

$f_{xx}(x, y) = -\sin(x)\sin(y), \qquad f_{xy}(x, y) = \cos(x)\cos(y), \qquad f_{yy}(x, y) = -\sin(x)\sin(y)$,

confirming what was already clear from all the graphs

at $\displaystyle{\Big(\frac{\pi}{2},\,\frac{\pi}{2}\Big)}}$:

$A = C = -1, \,B = 0, \, D = 1 > 0,$

$f$ has a local maximum,

and at $\displaystyle{\Big( -\frac{\pi}{2},\,\frac{\pi}{2}\Big)}$:

$A = C = 1, \, B = 0, \, D = 1 > 0$,

$f$ has a local minumum,

while at $(0,\,0)$:

$A = C = 0, \, B = 1, \, D = -1 < 0$,

$f$ has a saddle point.