Peak detection in signal using Python

Usually people use scipy.signal to detect peak in signal.

scipy.signal.find_peaks_cwt( vec )

which returns list of index where vec has maximas. However this does not work all
the time, especially when vector is noisy. I could not get it working very well to compute the modality of distributions.

This is the most sophisticated library (AFAIK):

Its can be installed via pip:

$ pip install analytic_wfm

To use,

>>> import analytic_wfm
>>> vec # this is the 1d array in which I want to detect peaks.
>>> maxPeaks, minPeaks = analytic_wfm.peakdetect( vec, lookahead = 10 )

This returns maximum peaks and minimum peaks. There are various other functions
described in readme.txt file.

Here is an example where I used this library to compute modality of a distribution:



Compress/decompress a file using Huffman codes

I wrote a small python script which can be used to compress/decompress a text file. On large files (more than 5MB), one can reduce the size of the file by a factor of ~ 0.35 which is not far from the best compression ratio. Since it is in pure Python, it is slow. This script requires module huffman and bitarray .

pip install huffman bitarray --user

The script accepts two options -c or compression and -d for decompression. Following is help message produced by the script.

./compress -h
usage: compress [-h] [--compress] [--decompress] file

Compress/Decompress a file using Huffman codes.

positional arguments:
 file File to compress/decompress.

optional arguments:
 -h, --help show this help message and exit
 --compress, -c Compress file

This script is intended for Information Theory students who are looking for a python script to play with.

We read the input file and compute the frequency of each character (Counter from standard python library does the job). We pass this frequency information to huffman module and ask it to give a code-book (Huffman code). Next, we replace each character with code and save it to a binary file. We also add the code-book to the file as prefix. Adding code-book is necessary if we want to decompress it. The Huffman code gives the shorted binary code to the symbol which occurs most frequently, and longest to the one which occur most rarely. I ran the script (with time command) on a large file (76MB).

time ./compress ~/Downloads/camk12_pp54_voxel2_diff--PP1--1e-13-_suOFF.dat -c
Reading /home1/dilawars/Downloads/camk12_pp54_voxel2_diff--PP1--1e-13-_suOFF.dat (76 MB).. done.
Generating codebook.. done.
Compressing files .. done.
Average codeword length : 2.924465
| Optimal average code length: 2.869759
Compressed files is written to /home1/dilawars/Downloads/camk12_pp54_voxel2_diff--PP1--1e-13-_suOFF.dat.dx
| Original file size : 80169086
| Compressed file size : 29307774
| Compression ratio : 2.735421

real 0m40.917s
user 0m40.822s
sys 0m0.096s

gzip takes less than a second to compress it (0.75 sec) and this script took more than 40 secs. With pypy, it took more time because bitarray is already a c-module.

The point of this script is demonstration, and not the efficiency. gzip reduced the file to size 2878272 bytes. Ours (29307774 bytes) is not bad. Note that we are storing our codebook in plain text and not in binary format.

We also print the Optimal average code length which is information theoretic optical average codeword length; no one can do better than this. And Huffman codes are quite close to this number.

On small file, you won’t see much reduction.

To decompress, we replace the codeword with the symbol and write to the output file.

File can be found on github.

Performance of “sorting dictionary by values” in python2, python3 and pypy

The script is hosted here . It is based on the work of

My script has been changed to accommodate python3 (iteritems is gone and replaced by items — not sure whether it is a fair replacement). For method names and how they are implemented, please refer to script or the blog post.

Following chart shows the comparison. PyPy does not boost up the performance for simple reason that dictionary sorted is not large enough. I’ve put it here just for making a point and PyPy can slow thing down on small size computation.

The fastest method is sbv6 which is based on PEP-0265 is the fastest. Python3 always performing better than python2.




Writing Maxima expression to text file in TeX format (for LaTeX)

You want to write an Maxima expression to a file which can be read by other application e.g. LaTeX.

Lets say the expression is sys which contains variable RM. You first want to replace RM by R_m .  Be sure to load mactex-utilities if you have matrix. Without loading this module, the tex command generates TeX output, not LaTeX.

load( "mactex-utilities" )$ 
sys : RM * a / b * log( 10 )$
texput( RM, "R_m")$
sysTex : tex( sys, false)$
with_stdout( "outout.txt", display( sysTex ) )$

Other methods such as stringout, save and write put extra non-TeX characters in file.

I get the following in file outout.txt after executing the above.

{{\log 10\,R_m\,a}\over{b}}

A csv reader based on Haskell-cassava library : performance

I implemented my own csv reader using cassava library. The reader from missingh library was taking too long (~ 17 seconds) for a file with 43200 lines. I compared the result with python-numpy and python-pandas csv reader. Below is rough comparison.

cassava (ignore #) 3.3 sec
cassava (no support for ignoring #) 2.7 sec
numpy loadtxt > 10 sec
pandas read_csv 1.5 sec

As obvious, pandas does really well at reading csv file. I was hoping that my csv reader would do better but it didn’t. But it still beats the parsec based reader hands down.

The code is here