5.2. Differentiation in Banach Spaces¶

We introduce the concept of differentiation in Banach spaces. Recall that Banach spaces are normed linear spaces that are complete.

5.2.1. Gateaux Differential¶

Definition 5.14 (Directional derivative)

Let $X$ and $Y$ be Banach spaces. Let $f : X \to Y$ be a function with $S = dom f$ . The directional derivative of $f$ at $x \in int S$ in the direction $h \in X$ where $h \neq 0$ , denoted by $f^{'} (x; h)$ is given by

f^{'} (x; h) ≜ lim_{t \to 0^{+}} \frac{f (x + t h) - f (x)}{t}

whenever the limit exists. This is also known as the Gateaux differential. By convention, $f^{'} (x; 0_{X}) = 0_{Y}$ . This is consistent with the definition above.

There is no single directional derivative at a point $x$ .
The directional derivative depends on the direction $h$ .
In one dimension, there are two directional derivatives at each $x$ .
In two or more dimensions, there are infinitely many directional derivatives.
The directional derivative is a one dimensional calculation along the direction $h$ .
It is usually easy to compute the directional derivative even when the space $X$ is infinite dimensional.

Definition 5.15 (Gateaux differentiability)

Let $X$ and $Y$ be Banach spaces. Let $f : X \to Y$ be a function with $S = dom f$ . Let $U \subseteq S$ be an open set. We say that $f$ is Gateaux differentiable at $x \in U$ if the Gateaux differential $f^{'} (x; h)$ exists for every direction $h \in X$ .

Accordingly, we can define a bounded operator $T_{x} : X \to Y$ given by

T_{x} (h) ≜ lim_{t \to 0^{+}} \frac{f (x + t h) - f (x)}{t} \forall h \in X .

The operator $T$ is called the Gateaux derivative of $f$ at $x$ .

Example 5.17 (Gateaux differential of exponential function)

Let $f (x) = e^{x}$ . Then,

\begin{aligned} f^{'} (x; h) & = lim_{t \to 0^{+}} \frac{e^{x + t h} - e^{x}}{t} \\ = e^{x} lim_{t \to 0^{+}} \frac{e^{t h} - 1}{t} \\ = e^{x} lim_{t \to 0^{+}} \frac{t h}{t} = h e^{x} . \end{aligned}

We note that the Gateaux derivative depends linearly on $h$ .

Theorem 5.7 (Gateaux differential nonnegative homogeneity)

The Gateaux differential of a function $f : X \to Y$ is nonnegative homogeneous in the sense that

f^{'} (x; α h) = α f^{'} (x; h)

for every $α \in R_{+}$ and every $h \in X$ .

However, the Gateaux differential may not be additive. Thus, the Gateaux differential may fail to be linear.

Example 5.18 (Gateaux differential of absolute value function)

Let $f (x) = | x |$ . Then, the Gateaux differentials are given by

\begin{array}{r} f^{'} (x; h) = {\begin{cases} h \frac{x}{| x |} & x \neq 0; \\ | h | & x = 0. \end{cases} \end{array}

We note that the Gateaux differential of $f$ exists everywhere. However the Gateaux differential depends on $h$ in a nonlinear way at $x = 0$ . At $x \neq 0$ , the Gateaux differential depends linearly on $h$ .

Example 5.19 (Gateaux differential of square function)

Let $f (x) = x^{2}$ . Then, the Gateaux differential is given by

\begin{aligned} f^{'} (x; h) & = lim_{t \to 0^{+}} \frac{f (x + t h) - f (x)}{t} \\ = lim_{t \to 0^{+}} \frac{x^{2} + t^{2} h^{2} + 2 x t h - x^{2}}{t} \\ = 2 x h . \end{aligned}

We note that the Gateaux differential is linear w.r.t. $h$ .

Example 5.20 (Gateaux differential of linear functional)

Let $f (x) = a^{T} x$ where $a \in R^{n}$ is a given fixed vector.

f^{'} (x; h) = lim_{t \to 0^{+}} \frac{a^{T} x + t a^{T} h - a^{T} x}{t} = a^{T} h .

We note that the Gateaux differential is linear w.r.t. $h$ .

Example 5.21 (Gateaux differential of simple quadratic)

Let $f (x) = x^{T} A x$ where $A \in S^{n}$ is a given symmetric matrix.

\begin{aligned} f^{'} (x; h) & = lim_{t \to 0^{+}} \frac{(x + t h)^{T} A (x + t h) - x^{T} A x}{t} \\ = lim_{t \to 0^{+}} \frac{t^{2} h^{T} A h + 2 t h^{T} A x}{t} \\ = 2 h^{T} A x = 2 x^{T} A h . \end{aligned}

We note that the Gateaux differential is linear w.r.t. $h$ .

In particular, if $f (x) = x^{T} x$ , then $f^{'} (x; h) = 2 h^{T} x = 2 x^{T} h$ .

Theorem 5.8 (Gateaux differential of a constant function)

The Gateaux differential of a constant function is zero.

Theorem 5.9 (Gateaux differential sum rule)

Gateaux differential distributes over sum.

Let $f, g : X \to Y$ both have Gateaux derivatives at $x$ in the direction $h$ . Then,

(f + g)^{'} (x; h) = f^{'} (x; h) + g^{'} (x; h) .

Also,

(f - g)^{'} (x; h) = f^{'} (x; h) - g^{'} (x; h) .

Theorem 5.10 (Gateaux differential product rule)

Let $f, g : X \to Y$ both be Gateaux differentiable at $x \in int dom f \cap dom g$ . Let $h$ be their (pointwise) product function given by

h (x) = f (x) g (x)

with $dom h = dom f \cap dom g$ . Then,

h^{'} (x; h) == (f g)^{'} (x; h) = f^{'} (x; h) g (x) + g^{'} (x; h) f (x) .

Theorem 5.11 (Gateaux differential chain rule)

Let $f : X \to Y$ and $g : Y \to Z$ be functions. Let $h : X \to Z$ be the composition of $f$ and $g$ given by $h = g \circ f$ . Let $U \subseteq dom h$ be an open set. Let $x \in U$ . Assume that $f$ is Gateaux differentiable at $x$ and $g$ is Gateaux differentiable at $f (x)$ . Then,

h^{'} (x; h) = g^{'} (f (x); f^{'} (x; h)) \forall h \in X .

We recall the little- $o$ notation. We say that a quantity $q$ is $o (t)$ if

lim_{t \to 0^{+}} \frac{q}{t} = 0.

For vector valued functions, a quantity $q$ is $o (t)$ if

lim_{t \to 0^{+}} \frac{‖ q ‖}{t} = 0.

or

lim_{t \to 0^{+}} \frac{q}{t} = 0 .

Proof. If $f$ is Gateaux differentiable at $x$ , then

f^{'} (x; h) = lim_{t \to 0^{+}} \frac{f (x + t h) - f (x)}{t} \forall h \in X .

In terms of little-o notation,

f (x + t h) = f (x) + t f^{'} (x; h) + o (t) .

Similarly, if $g$ is Gateaux differentiable at $y$ , then

g (y + s u) = g (y) + s g^{'} (y; u) + o (s) .

Now,

\begin{aligned} h^{'} (x; h) = (g \circ f)^{'} (x; h) & = lim_{t \to 0^{+}} \frac{g (f (x + t h)) - g (f (x))}{t} \\ = lim_{t \to 0^{+}} \frac{g (f (x) + t f^{'} (x; h) + o (t)) - g (f (x))}{t} \\ = lim_{t \to 0^{+}} \frac{g (f (x) + t (f^{'} (x; h) + t^{- 1} o (t))) - g (f (x))}{t} \\ = lim_{t \to 0^{+}} \frac{g (f (x)) + t g^{'} (f (x); f^{'} (x; h) + t^{- 1} o (t)) + o (t) - g (f (x))}{t} \\ = lim_{t \to 0^{+}} \frac{t g^{'} (f (x); f^{'} (x; h) + t^{- 1} o (t)) + o (t)}{t} \\ = lim_{t \to 0^{+}} [g^{'} (f (x); f^{'} (x; h)) + t^{- 1} o (t)) + t^{- 1} o (t)] \\ = g^{'} (f (x); f^{'} (x; h)) . \end{aligned}

Example 5.22 (Chain rule for square of inner product)

Consider the function $h (x) = (x^{T} x)^{2}$ .

Define $g (t) = t^{2}$
Define $f (x) = x^{T} x$ .
Then $h = g \circ f$ .
We have $f^{'} (x; h) = 2 h^{T} x$ .
We have $g^{'} (y; u) = 2 y u$ .
Thus,

$\begin{aligned} g^{'} (f (x); f^{'} (x; h)) & = 2 f (x) f^{'} (x; h) \\ = 2 (x^{T} x) (2 h^{T} x) \\ = 4 (h^{T} x) (x^{T} x) . \end{aligned}$

We can compute the same thing using the product rule.

We note that $h (x) = f (x) f (x)$ .
Applying the product rule:

$\begin{aligned} h^{'} (x; h) & = f^{'} (x; h) f (x) + f^{'} (x; h) f (x) \\ = 2 f^{'} (x; h) f (x) \\ = 2 (2 h^{T} x) (x^{T} x) \\ = 4 (h^{T} x) (x^{T} x) . \end{aligned}$

5.2.2. Fréchet Derivative¶

Definition 5.16 (Fréchet differentiability)

Let $X$ and $Y$ be Banach spaces. Let $f : X \to Y$ be a function with $S = dom f$ . Let $U \subseteq S$ be an open set. We say that $f$ is Fréchet differentiable at $x \in U$ if there is a bounded and linear operator $T_{x} : X \to Y$ given by

T_{x} (h) = lim_{t \to 0^{+}} \frac{f (x + t h) - f (x)}{t} \forall h \in X .

The operator $T_{x}$ is called the Fréchet derivative of $f$ at $x$ .

We note that $T_{x}$ depends on $x$ .

Remark 5.1 (Fréchet differentiability alternate forms)

By definition, if $f$ is Fréchet differentiable at $x$ , then it is Gateaux differentiable at $x$ . Since $T_{x}$ is linear, we can write it as

T_{x} (h) = A h

emphasizing the fact that the essential part of $T_{x}$ doesn’t depend on $h$ . $A$ may still depend on $x$ .

Using the little- $o$ notation, we can write

f (x + t h) = f (x) + t T_{x} (h) + o (t) = f (x) + t A h + o (t) .

If we set $t h = y$ , then $t \to 0$ if and only if $y \to 0$ . In particular, $‖ y ‖_{X} = t ‖ h ‖_{X} = o (t)$ . Now,

\begin{aligned} f (x + y) = f (x) + A y + o (t) \\ ⟺ f (x + y) - f (x) - A y = o (t) = o (‖ y ‖_{X}) \\ ⟺ lim_{‖ y ‖_{X} \to 0} \frac{‖ f (x + y) - f (x) - A y ‖_{Y}}{‖ y ‖_{X}} = 0 \\ ⟺ lim_{y \to 0} \frac{‖ f (x + y) - f (x) - A y ‖_{Y}}{‖ y ‖_{X}} = 0 \\ ⟺ lim_{y \to 0} \frac{‖ f (x + y) - f (x) - T_{x} (y) ‖_{Y}}{‖ y ‖_{X}} = 0. \end{aligned}

Therefore $f : X \to Y$ is Fréchet differentiable at $x \in U$ if and only if

lim_{y \to 0} \frac{f (x + y) - f (x) - T_{x} (y)}{‖ y ‖_{X}} = 0

for every $y \in X$ .

It is worthwhile to compare this definition to the definition of differentiability of $f : R^{n} \to R^{m}$ in Definition 5.1. If we put $z = x + y$ , we can rewrite the condition as

lim_{z \to x} \frac{‖ f (z) - f (x) - T_{x} (z - x) ‖_{Y}}{‖ z - x ‖_{X}} = 0.

Thus, $T_{x}$ plays the same role as the Jacobian matrix $D f (x)$ in (5.1).

Theorem 5.12 (Existence of Fréchet derivative)

The Fréchet derivative of a function $f$ exists at a point $x = a$ if and only if all Gateaux differentials of $f$ at $x$ are continuous functions of $x$ at $x = a$ .

Theorem 5.13 (Uniqueness of Fréchet derivative)

If the Fréchet derivative of a function $f$ exists at a point $x = a$ then it is unique.

5.2.3. Gradient¶

Definition 5.17 (Gradient)

Let $V$ be a Hilbert space. Let $f : V \to R$ is a real valued function. Let $S = dom f$ and $U \subseteq S$ be an open set. Assume that $f$ is Fréchet differentiable at $x \in U$ . Then, the Fréchet derivative $T_{x} : V \to R$ is a bounded linear functional.

The gradient of a real valued function is denoted by $\nabla f (x)$ and $\nabla f (x) \in V^{*}$ satisfying

⟨ h, \nabla f (x) ⟩ = T_{x} (h) .

Topics in Signal Processing

Differentiation in Banach Spaces

Contents

5.2. Differentiation in Banach Spaces¶

5.2.1. Gateaux Differential¶

5.2.2. Fréchet Derivative¶

5.2.3. Gradient¶