If is a smooth function on a manifold , and is a critical point of , recall that the Hessian is the quadratic form on (in local co-ordinates, the coefficients of the Hessian are the second partial derivatives of at ). Since is symmetric, it has a well-defined index, which is the dimension of the subspace of maximal dimension on which is negative definite. The Hessian does not depend on a choice of metric. One way to see this is to give an alternate definition where and are any two vector fields with given values and in . To see that this does not depend on the choice of , observe
because of the hypothesis that vanishes at . This calculation shows that the formula is symmetric in and . Furthermore, since only depends on the value of at , the symmetry shows that the result only depends on and as claimed. A critical point is nondegenerate if is nondegenerate as a quadratic form.
In Morse theory, one uses a nondegenerate smooth function (i.e. one with isolated nondegenerate critical points), also called a Morse function, to understand the topology of : the manifold has a (smooth) handle decomposition with one -handle for each critical point of of index . In particular, nontrivial homology of forces any such function to have critical points (and one can estimate their number of each index from the homology of ). Morse in fact applied his construction not to finite dimensional manifolds, but to the infinite dimensional manifold of smooth loops in some finite dimensional manifold, with arc length as a “Morse” function. Critical “points” of this function are closed geodesics. Any closed manifold has a nontrivial homotopy group in some dimension; this gives rise to nontrivial homology in the loop space. Consequently one obtains the theorem of Lyusternik and Fet:
Theorem: Let be a closed Riemannian manifold. Then admits at least one closed geodesic.
In higher dimensions, one can study the space of smooth maps from a fixed manifold to a Riemannian manifold equipped with various functionals (which might depend on extra data, such as a metric or conformal structure on ). One context with many known applications is when is a Riemannian -manifold, is a surface, and one studies the area function on the space of smooth maps from to (usually in a fixed homotopy class). Critical points of the area function are called minimal surfaces; the name is in some ways misleading: they are not necessarily even local minima of the area function. That depends on the index of the Hessian of the area function at such a point.
Let be a (compactly supported) -parameter family of surfaces in a Riemannian -manifold , for which is smoothly immersed. For small the surfaces are transverse to the exponentiated normal bundle of ; hence locally we can assume that takes the form where are local co-ordinates on , and is contained in the normal geodesic to through the point ; we call such a family of surfaces a normal variation of surfaces. For such a variation, one has the following:
Theorem (first variation formula): Let be a normal variation of surfaces, so that where is the unit normal vector field to . Then there is a formula:
where is the mean curvature vector field along .
Proof: let denote the image under of the vector fields . Choose co-ordinates so that are conformal parameters on ; this means that and at .
The infinitesimal area form on is which we abbreviate by , and write
Since are the pushforward of coordinate vector fields, they commute; hence , so and therefore
and similarly for . At we have , and so the calculation reduces to
Now, , and so the conclusion follows. qed.
As a corollary, one deduces that a surface is a critical point for area under all smooth compactly supported variations if and only if the mean curvature vanishes identically; such a surface is called minimal.
The second variation formula follows by a similar (though more involved) calculation. The statement is:
Theorem (second variation formula): Let be a normal variation of surfaces, so that . Suppose is minimal. Then there is a formula:
where is the Jacobi operator (also called the stability operator), given by the formula
where is the second fundamental form, and is the metric Laplacian on .
This formula is frankly a bit fiddly to derive (one derivation, with only a few typos, can be found in my Foliations book; a better derivation can be found in the book of Colding-Minicozzi) but it is easy to deduce some significant consequences directly from this formula. The metric Laplacian on a compact surface is negative self-adjoint (being of the form for some operator ), and is obtained from it by adding a th order perturbation, the scalar field . Consequently the biggest eigenspace for is -dimensional, and the eigenvector of largest eigenvalue cannot change sign. Moreover, the spectrum of is discrete (counted with multiplicity), and therefore the index of (thought of as the “Hessian” of the area functional at the critical point ) is finite.
A surface is said to be stable if the index vanishes. Integrating by parts, one obtains the so-called stability inequality for a stable minimal surface :
for any reasonable compactly supported function . If is closed, we can take . Consequently if the Ricci curvature of is positive, admits no stable minimal surfaces at all. In fact, in the case of a surface in a -manifold, the expression is equal to where is the intrinsic curvature of , and is the scalar curvature on . If has positive genus, the integral of is non-negative, by Gauss-Bonnet. Consequently, one obtains the following theorem of Schoen-Yau:
Corollary (Schoen-Yau): Let be a Riemannian -manifold with positive scalar curvature. Then admits no immersed stable minimal surfaces at all.
On the other hand, one knows that every -injective map to a -manifold is homotopic to a stable minimal surface. Consequently one deduces that when is a -manifold with positive scalar curvature, then does not contain a surface subgroup. In fact, the hypothesis that be -injective is excessive: if is merely incompressible, meaning that no essential simple loop in has a null-homotopic image in , then the map is homotopic to a stable minimal surface. The simple loop conjecture says that a map from a -sided surface to a -manifold is incompressible in this sense if and only if it is -injective; but this conjecture is not yet known.
Update 8/26: It is probably worth making a few more remarks about the stability operator.
The first remark is that the three terms , and in have natural geometric interpretations, which give a “heuristic” justification for the second variation formula, which if nothing else, gives a handy way to remember the terms. We describe the meaning of these terms, one by one.
- Suppose , i.e. consider a variation by flowing points at unit speed in the direction of the normals. In directions in which the surface curves “up”, the normal flow is focussing; in directions in which it curves “down”, the normal flow is expanding. The net first order effect is given by , the mean curvature in the direction of the flow. For a minimal surface, , and only the second order effect remains, which is (remember that is the second fundamental form, which measures the infinitesimal deviation of from flatness in ; the mean curvature is the trace of , which is first order. The norm is second order).
- There is also an effect coming from the ambient geometry of . The second order rate at which a parallel family of normals along a geodesic diverge is where is the curvature operator. Taking the average over all geodesics tangent to at a point gives the Ricci curvature in the direction of , i.e. . This is the infinitesimal expansion of area of a geodesic plane under the normal flow, and has second order. The interactions between these terms have higher order, so the net contribution when is .
- Finally, there is the contribution coming from itself. Imagine that is a flat plane in Euclidean space, and let be the graph of . The infinitesimal area element on is . If has compact support, then differentiating twice by , and integrating by parts, one sees that the (leading) second order term is . When is not totally geodesic, and the ambient manifold is not Euclidean space, there is an interaction which has higher order; the leading terms add, and one is left with .
The second remark to make is that if the support of a variation is sufficiently small, then necessarily will be large compared to , and therefore will be positive definite. In other words all variations of a (fixed) minimal surface with sufficiently small support are area increasing — i.e. a minimal surface is locally area minimizing (this is local in the surface itself, not in the “space of all surfaces”). This is a generalization of the important fact that a geodesic in a Riemannian manifold is locally length minimizing (though typically not globally length minimizing).
One final remark is that when is big enough at some point , and when the injectivity radius of at is big enough (depending on bounds on in some neighborhood of ), one can find a variation with support concentrated near that violates the stability inequality. Contrapositively, as observed by Schoen, knowing that a minimal surface in a -manifold is stable gives one a priori control on the size of , depending only on the Ricci curvature of , and the injectivity radius of the surface at the point. Since stability is preserved under passing to covers (for -sided surfaces, by the fact that the largest eigenvalue of can’t change sign!) one only needs a lower bound on the distance from to . In particular, if is a closed stable minimal surface, there is an a priori pointwise bound on . This fact has many important topological applications in -manifold topology. On the other hand, when has boundary, the curvature can be arbitrarily large. The following example is due to Thurston (also see here for a discussion):
Example (Thurston): Let be an ideal simplex in with ideal simplex parameter imaginary and very large. The four vertices of come in two pairs which are very close together (as seen from the center of gravity of the simplex); let be an ideal quadrilateral whose edges join a point in one pair to a point in the other. The simplex is bisected by a “square” of arbitrarily small area; together with four “cusps” (again, of arbitrarily small area) one makes a (topological) disk spanning with area as small as desired. Isotoping this disk rel. boundary to a least area (and therefore stable) representative can only decrease the area further. By the Gauss-Bonnet formula, the curvature of such a disk must get arbitrarily large (and negative) at some point in the interior.