Linear algebra
Weiquan Luo
2022-07-18
Introduction¶
Linear algebra usually work on three subjects: Scalar, Vector, and Matrix. They occupy on different Mathmetical space. So the algebra of the three subjects has different properties.
| subjects | Mathmetical space | Common definition in linear algebra |
|---|---|---|
| Scalar (0d-array) |
field $\mathbb{F}$ | usually real numbers $\mathbb{R}$ (contains $\mathbb{Q}$, $\mathbb{Z}$, $\mathbb{N}$), complex number $\mathbb{C}$ |
| Vector (1d-array) |
vector space over a field, $\mathbb{V}$ | usually real vector space $\mathbb{R}^n$ or complex vector space $\mathbb{C}^n$ |
| Matrix (2d-array) |
vector space of functions | linear (Bijective) mapping $f: V \to W$ |
| Tensor (nd-array) |
N/A | N/A |
Hint:
- Avoid using
np.dot, because its reactions vary by dimentions of the input array. Instead, using specifialized functions such as:np.inner()for inner product 1darraynp.matmul()for matrix multiplication of 2darray and ndarray
Scalar: Field¶
In mathematics, a field is a set on which addition, subtraction, multiplication, and division are defined and behave as the corresponding operations on rational and real numbers do. A field is thus a fundamental algebraic structure which is widely used in algebra, number theory, and many other areas of mathematics. (Cite: wiki))
In real number field $\mathbb{R}$:
| Algebra | Mathematical expression | python operation |
|---|---|---|
| Addition | $a + b$ | a+b |
| :: commutative property | $a+b$ $=b+a$ |
|
| :: associative property | $(a+b)+c$ $=a+(b+c)$ |
|
| Subtraction | $a - b$ | a/b |
| :: Ordered property | $a \geq b$ $a + c \geq b + c$ |
a>b, a>=b a>b, a<=ba==b a!=b |
| Multiplication | $a \times b$ | a*b |
| :: commutative property | $a \times b$ $=b \times a$ |
|
| :: associative property | $(a \times b) \times c$ $=a \times (b \times c)$ |
|
| :: distributive property | $c \times (a+b)$ $=c \times a + c \times b$ |
|
| :: exponentiation | $a^b$ | a**b |
| :: root (even root only for Non-nagative) |
$ \sqrt[b]{a}$ | a**(1/b) |
| Division | $\frac{a}{b}$ | a/b |
| :: Modulus (reminder) | - | a%b |
| :: Floor division (rounded division) | - | a//b |
Vector: Vector spaces¶
A point in a coordinate system can be represented by a vector:
Usually, the vector space is over the real number field $\mathbb{R}$, then it forms a Euclidean space. Euclidean space is a normed, completed vector space.
- The length of a vector can be measured with L1-norm
- the angles between vector can be measured with L2-norm.
- By combining length and angle, similarity between any two vectors can be measured with inner product.
- The other feature property can be derived, such as normal vector by cross product.
| Algebra | Mathematical expression | Numpy operation |
|---|---|---|
| Scalar | ||
| ::Element-wise addition | $\vec{a} + \vec{b}$ | a+b np.add(a,b) |
| ::Element-wise subtraction | $\vec{a} - \vec{b}$ | a/b np.subtract(a,b) |
| ::Element-wise multiplication (Hamandard product) |
$\langle \vec{a}, \vec{b} \rangle_{ham}$ | a*b np.multiply(a,b) |
| ::Element-wise division | $\langle \vec{a}, \vec{b}\rangle_{div}$ | a/b np.divide(a,b) |
| Vector | ||
| ::Frobenius norm/L2-norm (length) |
$\lvert \vec{a} \rvert$ | np.linalg.norm(a)) |
| ::Inner product (similarity) |
$\vec{a} \cdot \vec{b}$ | np.inner(a,b) np.dot(a,b) (not recommanded) |
| ::Cross product (normal vector) |
$\vec{a} \times \vec{b}$ | np.cross(a,b) |
Matrix: Vector spaces of functions¶
A linear equations system can be rewrote into matrix form:
The coefficient matrix is also called linear mapping, denote as $f: V \to W$, which also defines the change of basis. This coefficient matrix is within the Vector spaces of functions, whose properties are different from that of Vector spaces over a real number field.
- It is no longer necessarily a normed vector space, so lengths or angles might not be defined.
- As the most important property of linear mapping, the determinate is defined to describe the changing ratio of area/volume due to the linear mapping.
- The eigenvectors of the transformation are also defined to describe the vector that staying on thier original direction.
- Then, the eigenvalues further describe the factor of streching/squashed of the lenght of these Eigenvectors.
| Algebra | Mathematical expression | Numpy operation |
|---|---|---|
| Scalar (Element-wise) | ||
| ::Element-wise addition | $A + B$ | A+B np.add(A,B) |
| ::Element-wise subtraction | $A - B$ | A/B np.subtract(A,B) |
| ::Element-wise multiplication (Hamandard product) |
$\langle A, B \rangle_{ham}$ | A*B np.multiply(A,B) |
| ::Element-wise division | $\langle A, B\rangle_{div}$ | A/B np.divide(A,B) |
| Vector | ||
| ::Frobenius inner product (similarity) |
$\langle A,B\rangle_F$ | (A*B).sum() np.multiply(A,B).sum() |
| ::Row-wise inner product | np.inner(A, B)) |
|
| Matrix | ||
| Matrix multiplication product (transformation, mapping) |
$AB$ $A \cdot B$ |
np.matmul(A, B) np.dot(A,B) (not recommanded) |
| ::Determinant | $\lvert A \rvert$ | np.linalg.det(A)) |
| ::Transpose | $(A_{ij})^{T} = A_{ji}$ | A.T A.transpose() |
| ::No commutative exception A is Invertible/Nonsingular |
$AB \neq BA$ | |
| ::associative | $(AB)C=A(BC)$ | |
| ::Distributive | $C(A+B)$ $=CA +CA$ |
|
| ::Uniqueness of inverse | $B = A^{-1}$ $(A^{-1})^{-1} = A$ |
Special matrices¶
| Special matrices | example | Some unique property |
|---|---|---|
| identity matrix |  |
|
| diagonal matrix |  |
|
| upper/lower-triangular matrix |  |
|
| symmetric matrix |  |
$A = A^T$ |
| zero matrix |  |
|
| positive-definite matrix | all eigenvalue are positive | |
| orthogonal matrix | $A^{T}= A^{-1}$ $AA^{T} = A^{T}A = I$ $det(A)=\pm1$ |
|
| invertable/Nonsingular matrix | $AB=BA=I$ | |
| determinate of trianglal matrix is produce of diagnoal | $det(U) = \prod_{i=1}^{n} u_{ii}$ |
Tensor:¶
Tensor is not tipically appear in the context of linear algebra. Instead, it is the typical form of storing 2d/3d/4d raster data such as images.
- Because it is no longer within any vector space of field or function, so it does not have any proerty relative to normed (length, angle, similarity) or mapping (eigenvector, eigenvalue).
- It represents the actual value at each location in a Cartesian coordinate system $\mathbb{R^3}$ or euclidean space $\mathbb{R^n}$. In most of the time, the most interesting things are the spatial information such as distribution, gradient and the isosurface of these value.
Einstein summation convention¶
| Description | Numpy function | Call signature in np.einsum |
|---|---|---|
| Return A | A | (‘i’, A) |
| Summation | np.sum(A) | (‘i->’, A) |
| Element-wise multiplication | A * B | (‘i,i->i’, A, B) |
| Inner product | np.inner(A, B) | (‘i,i’, A, B) |
| Outer(cross) product | np.outer(A, B) | (‘i,j->ij’, A, B) |
| Description | Numpy function | Call signature in np.einsum |
|---|---|---|
| Return A | A | (‘ij’, A) |
| Transpose | A.T | (‘ji’, A) |
| Diagonal | np.diag(A) | (‘ii->i’, A) |
| Summation over diagonal | np.trace(A) | (‘ii’, A) |
| Summation | np.sum(A) | (‘ij->’, A) |
| Summation over col | np.sum(A, axis=0) | (‘ij->j’, A) |
| Summation over row | np.sum(A, axis=1) | (‘ij->i’, A) |
| Element-wise multiplication of A and B | A * B | (‘ij,ij->ij’, A, B) |
| Element-wise multiplication of A and B.T | A * B.T | (‘ij,ji->ij’, A, B) |
| matrix multiplication | np.matmul(A, B) | (‘ij,jk’, A, B) |
| matrix-vector multiplication, finite or infinite dimensional field, $\mathbb{K}$ |
np.inner(A,B) | (‘ij,jk’, A, B) |
| matrix-matrix multiplication, $\mathbb{R}^n$ | np.dot(A,A.T) | (‘ij,jk’, A, B) |
| Inner product | np.inner(A, B) | (‘ij,kj->ik’, A, B) |
| A的每一行乘以B | A[:, None] * B | (‘ij,kj->ijk’, A, B) |
| A的每个值乘以B | A[:, :, None, None] * B | (‘ij,kl->ijkl’, A, B) |
In [1]:
!python3 -m jupyter nbconvert jupyter_LinearAlgebra.ipynb --to html
[NbConvertApp] Converting notebook jupyter_LinearAlgebra.ipynb to html [NbConvertApp] Writing 595405 bytes to jupyter_LinearAlgebra.html