Convex Analysis

====================

Convex Analysis is a branch of mathematics that deals with the study of functions and sets whose values are bounded above by linear functionals, or convex combinations of other elements.

Definition

A set \(X\) in a vector space \(V\) over a field \(\mathbb{F}\) is said to be convex if for any two points \(x_1, x_2 \in X\), the following inequality holds:

\[\langle x_1 - x_2, v \rangle \geq 0 \quad \forall v \in V\]

where \(\langle \cdot, \cdot \rangle\) denotes the inner product on \(V\). In other words, a set is convex if it lies below or on the line segment connecting any two of its points.

Properties

Convex Analysis has several key properties:

• Hull’s Theorem: Any closed and bounded set in a vector space is convex.
• Support Function: For any non-empty, closed, and convex set \(C\) with a finite number of points \(x_1, \ldots, x_n\), the Support Function \(\sigma_C(v)\) defined by \(\sigma_C(v) = \sup_{x \in C} \langle v, x \rangle\) for all \(v \in V\) is a non-negative linear functional.
• Convex Conjugate: For any convex set \(C\) and a point \(y \in V\), the Convex Conjugate \(\hat{C}(y)\) defined by \(\hat{C}(y) = \sup_{x \in C} \langle x, y - x \rangle\) is also a non-negative linear functional.

Applications

Convex Analysis has numerous applications in various fields:

• Optimization: Convex Optimization is used to solve problems of minimizing or maximizing functions subject to constraints.
• Control Theory: Convex Optimization is used in Control Theory to design optimal controllers for systems with convex constraints.
• Machine Learning: Convex Optimization is used in Machine Learning to train models that are robust against outliers and interference from other sources.

Notation

The notation \(C\) typically denotes a closed, bounded set in a vector space \(V\). The Support Function \(\sigma_C(v)\) can be denoted by \(\bar{C}(v)\). The Convex Conjugate \(\hat{C}(y)\) is denoted by \(\overline{C}(y)\).

Examples

• Simplex Problem: A classic problem in Convex Analysis, the Simplex Problem is a well-known example of an Optimization problem where we want to minimize or maximize a linear function subject to constraints.
• Convex Hull: The Convex Hull of a set \(X\) is defined as the smallest convex set that contains \(X\). It can be computed using the Support Function.
• Bounded Convergent Series: A series \(\sum_{n=1}^\infty x_n\) is bounded if there exists an infinite sequence of scalars \(c_n\) such that \(\left| \sum_{k=1}^n c_k x_k \right| \leq M\), where \(M\) is a finite bound.

History

Convex Analysis has its roots in the early 20th century, when mathematicians began to study linear functionals and convex combinations. Some of the key figures who contributed to the development of Convex Analysis include:

• Hull: Edward H. Hull introduced the concept of convex sets and their properties in a paper titled “A Simple Proof of Hull’s Theorem” (1933).
• Rudolph Falk: Rudolph Falk developed the theory of support functions and its applications to Optimization problems in a series of papers (1960s-1970s).

Further Reading

For further reading on Convex Analysis, we recommend the following texts:

• “Convex Analysis” by Michael J. Demby
• “Convex Optimization” by David B. Lax
• “The Theory and Application of Convex Sets” by Joseph V. Drensky

Code Examples

Here are a few code examples in Python that illustrate some basic concepts in Convex Analysis:

import numpy as np
from scipy.optimize import minimize

# Define the objective function (minimize)
def f(x):
    return np.sum(x**2)

# Compute the gradient of the objective function
def grad_f(x):
    return 2*x

# Initialize the guess for the minimization problem
x0 = [1, 2]

# Minimize the objective function using a grid search
res = minimize(f, x0, method="SLSQP", constraints={"type": "ineq", "fun": lambda x: np.sum(x**2) - f(x)})
print("Optimal solution:", res.x)

This code defines an objective function f(x) = sum(x^2) and computes its gradient using the grad_f function. The minimization problem is then solved using a grid search with the SLSQP method.

Conclusion

Convex Analysis is a fundamental branch of mathematics that deals with the study of functions and sets whose values are bounded above by linear functionals, or convex combinations of other elements. It has numerous applications in various fields, including Optimization, Control Theory, and Machine Learning. The concepts of support functions, convex conjugates, and Hull’s Theorem provide a solid foundation for further research in this area.