7. Numerical Optimization¶

A mathematical optimization problem consists of maximizing or minimizing a real valued function under a set of constraints. We shall assume $V$ to denote a finite dimensional real vector space. Typical examples of $V$ are $R^{n}$ and $S^{n}$ .

Formally, we express a mathematical optimization problem as:

\begin{array}{r} \begin{aligned} minimize & f_{0} (x) \\ subject to & f_{i} (x) \leq b_{i}, & i = 1, \dots, m . \end{aligned} \end{array}

$x \in V$ is the optimization variable of the problem.
$f_{0} : V \to R$ is the objective function.
The functions $f_{i} : V \to R, i = 1, \dots, m$ are the (inequality) constraint functions.
The (real scalar) constants $b_{1}, \dots, b_{m}$ are the limits for the inequality constraints.
A vector $x \in V$ is called feasible if it belongs to the domains of $f_{0}, f_{1}, \dots, f_{m}$ and satisfies all the constraints.
A vector $x^{*}$ is called optimal if is feasible and has the smallest objective value; i.e. for any feasible $z$ , we have $f_{0} (z) \geq f_{0} (x^{*})$ .
An optimal vector is also called a solution to the optimization problem.
An optimization problem is called infeasible if there is no feasible vector. i.e. there is no vector $x \in V$ which satisfies the inequality constraints.
An infeasible problem doesn’t have a solution.
A feasible problem may not have a solution if the objective function is unbounded below. i.e. for every feasible $x$ , there exists another feasible $z$ such that $f_{0} (z) < f_{0} (x)$ .
If a feasible problem is not unbounded below, then it may have one or more solutions.

Topics in Signal Processing

Numerical Optimization

7. Numerical Optimization¶