I am having trouble with HTML format of blog post. Here is draft PDF PDF.

*I recommend http://www.pyrunner.com/weblog/2016/05/26/compressed-sensing-python/ . This is much better written.*

Can you solve $latex Ax=b$ for $latex x$ when number of rows are less than number of columns in (A) i.e there are more variables than equations? Such a system is called underdetermined system. For examples, give me 3 numbers whose average is 3. Or solve (x/3+y/3+z/3=3). You are right! There are many solutions to this problem!

If one adds more containts then one can hope to get a unique solution. Give me a solution which has minimum length (minimize (L_2)-norm)? Answer is 3,3,3 (hopefully). Give me a solution which is most sparse (least number of non-zero entries)? Answer is 9,0,0 (or 0,0,9).

Many signals are sparse (in some domain). Images are sparse in Fourier/Wavelets domain; neural acitivity is sparse in time-domain, activity of programming is sparse in spatial-domain. Can we solve under-determined (Ax=b) for sparse (x). Formally,

where (L_0) is the 0-norm i.e. number of non-zero entries.

This problem is **hard** to solve. (L_0) is not a convex function. In this case, one uses *convex relaxation* trick and replace the non-convex function with convex one.

It is now well-understood that under certain assumptions on (A), we can recover (x) by solving this tractable convex-optimization problem rather than the original NP hard problem. Greedy strategy based solvers also exists which works even faster (though, according to Terry Tao blog, they do not guarantee the same performance as interior-point method based).

If (x_N) (size (N)) is (S-sparse) then an under-determined system (A_{kN}x_{N} =b_k) — where (b) is vector of (k) observations — can be solved *exactly* given that (A) (**mesurement matrix**) has following properties:

- where \(y = A x\) for any sparse signal \(x\) i.e application of measurements matrix does not change the length of sparse signal \(x\) ‘very much’. See Restricted Isometric Property in sec.Â 3.1 .

Lets there is a sparse signal (x) of length (N). Its sparse. We take a dot-product of x with a random vector (A_i) of size N i.e. (b_i = A_i * x). We make (k) such measurements. We can put all k (A_i) into a matrix (A) and represent the process by a linear system:

Note that each (A_i) has the dimension (1 \times N). We can rewrite eq.Â 1 as following:

where dimension of (A) are (k \times N) and (k << N). This is an under-determined system with the condition that (x) is sparse. Can we recover (x) (of size N) from b (of size k (<<N))?

Figure fig.Â 1 shows a real system. Compressed sensing says that (x) can be recovered by solving the following linear program.

# A demonstration using Python

Data for fig.Â 2 is generated by script `./compressed_sensing.py`

and for fig.Â 3, data is generated by script `./magic_reconstruction.py`

. Both of these scripts also generate similar figures.

**Dependencies** In addition to `scipy`

, `numpy`

, and `matplotlib`

, we also need pyCSalgo. It is available via pip: `pip install pyCSalgo --user`

.

Code is available at github

## Algorithm

**input** : sparse signal x of size N
- Generate a random matrix (
**measurement matrix**) \(A\) of size \(k\times N\).
- Make \(k\) measurements of \(x\) using \(A\). That is \(b = A x\). Note that each measurement of \(x\) (i.e. entry of \(b\)) is some linear combination of values of \(x\) given by \(A_i x\) where \(A_i\) is ith row.
- Now find \(x\) by solving \(\min_x \left\Vert x \right\Vert_1 \text{given}\; Ax=b\).

For step 4, we are using function `l1eq_pd`

from Python library `pyCSalgo`

. This is reimplementation of `l1magic`

routine written in Matlab by Candes. When (k >> 2S) where (S) is the sparsity of (x), we can recover (x) quite well. In practice, (k \ge 4S) almost always gives exact result.

There are other algorithms available which works faster; they are based on **greedy** strategy. We are not discussing them here.

## Sparse in time domain

## Sparse in Fourier domain

# Mathematical definitions

## Restricted isometric property (RIP)

Restricted isometric property (RIP) of a matrix (A) is defined as the following.

Given (\delta_s \in (0,1)), for any sub-matrix (A_s) of (A), and for any sparse vector (y), if following holds

then matrix (A) is said to satisfy (s-)restricted isometric property with restricted isometric constant (\delta_s). *Note that a matrix A with such a property does not change the length of signal (x) ‘very much’*. This enables us to sense two different sparse signal (x_1) and (x_2) such that (A x_1) and (A x_2) are almost likely to be different.

# References

To learn about compressed sensing, I recommend following articles

There are many many other great articles and papers on this subject. There are some dedicated webpages also.