Kernels and Hilbert Space

Reference：
‘Introduction to Hilbert Spaces with Application.’
‘Introduction to RKHS, and some simple kernel algorithms.’

Since Kernel trick is one of the core methods in SVM and SVGD also involves expertise related to RKHS. I looked up several books on Kernel method, trying to get a systematic understanding of Kernel and Hilbert space. This blog can also be regarded as a summary and summary of the book ‘Introduction to Hilbert Spaces with Application ’.

Introduction

Kernel

Definition: Let \(\mathcal{X}\) be a non-empty set. A function \(k: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}\) is called a kernel if there exists an \(\mathbb{R}\)-Hilbert space and a map \(\phi: \mathcal{X} \rightarrow \mathcal{H}\) such that \(\forall x, x^{\prime} \in \mathcal{X}\) \[ k\left(x, x^{\prime}\right):=\left\langle\phi(x), \phi\left(x^{\prime}\right)\right\rangle_{\mathcal{H}} \]

Normed Vector Spaces

First, the space defined in mathematics can be divided from simple to complex as:

• Vector Space
  a nonempty set \(E\) with two operations: addition and multiplication by scalars.
  e.g. \(\mathbb{R}^{N}\) \(\mathbb{C}^{N}\)

• Normed Space
  norm is an abstract generalization of the length of a vector:
  function \(x \mapsto\|x\|\) from a vector space \(E\) into \(\mathbb{R}\)

• Banach Space: complete normed space
  A normed space is complete if and only if every absolutely convergent series converges. (The contents of Cauchy sequence and Cauchy series are put in the appendix)
  Actually, Banach space introduces the concept of Limits

• Inner Product Spaces
  The space that defines the inner product.

• Hilbert Spaces: A complete inner product space

Hilbert Spaces

Appendix

Cauchy sequence and Cauchy series

Definition of Cauchy sequence. A sequence \(\left\{f_{n}\right\}_{n=1}^{\infty}\) of elements in a normed space \(\mathcal{H}\) is said to be a Cauchy sequence if for every \(\epsilon>0\), there exists \(N=N(\varepsilon) \in \mathbb{N}\), such that for all \(n, m \geq N,\left\|f_{n}-f_{m}\right\|_{\mathcal{H}}<\epsilon\)

Inner product

Definition of Inner product. Let \(\mathcal{H}\) be a vector space over \(\mathbb{R}\). A function \(\langle\cdot, \cdot\rangle_{\mathcal{H}}: \mathcal{H} \times \mathcal{H} \rightarrow \mathbb{R}\) is said to be an inner product on \(\mathcal{H}\) if:

• \(\left\langle\alpha_{1} f_{1}+\alpha_{2} f_{2}, g\right\rangle_{\mathcal{H}}=\alpha_{1}\left\langle f_{1}, g\right\rangle_{\mathcal{H}}+\alpha_{2}\left\langle f_{2}, g\right\rangle_{\mathcal{H}}\)

• \(\langle f, g\rangle_{\mathcal{H}}=\langle g, f\rangle_{\mathcal{H}}{ }^{1}\)

• \(\langle f, f\rangle_{\mathcal{H}} \geq 0\) and \(\langle f, f\rangle_{\mathcal{H}}=0\) if and only if \(f=0\).

the inner product between matrices \(A \in \mathbb{R}^{m \times n}\) and \(B \in\) \(\mathbb{R}^{m \times n}\) is \[ \langle A, B\rangle=\operatorname{trace}\left(A^{\top} B\right) \]