If the function is f, and your path is p(t), and f(p(t)) is constant, then the derivative of f(p(t)) = 0. By the chain rule, ∇f(p).p′ = 0, hence you are always traveling perpendicular to the gradient. In the mountain example, you are walking perpendicular to the direction we call "up hill".
Let your mountain be a perfect hemisphere, with equation z = sqrt(1-x2-y2). Verify that ∇f always points towards the origin, the top of the sphere. Any circle centered at the origin is a level curve, or level set, since it is everywhere perpendicular to the gradient.
One can also derive a curve that follows the gradient, like a ball rolling downhill. Given a starting point, and nonzero gradients all around (not a flat table top), a unique "downhill" curve exists, though we don't have the machinery to prove this yet. These curves are sometimes called integral curves. In our example, each integral curve is a straight line through the origin, as the ball rolls down the sphere and away from the top.
Moving to a higher dimension, the sphere is a level surface in 3 space. If f = x2+y2+z2, then setting f to the constant 1 produces the sphere. The gradient has to be perpendicular to the level surface. A vector that points directly out from the surface of the sphere, at the point x.y.z, is parallel to 2x.2y.2z, the gradient at that point. The tangent plane can be found by using the vector components as coefficients. If the perpendicular vector is a.b.c, the tangent plane is ax+by+cz = t, where t is just enough to shift the plane so that it contains the point of tangency.