Derived Result from Butter Filter and FFT doesn't change over time
<p>I am very new in Signal Processing, have met a situation that I am not sure if it is correct or not.  Please correct me then I will update more details.</p>

<p>My data is <a href="https://app.box.com/s/8ijccq76z65itv9jluid3nadfzp160qr" rel="nofollow">here</a> </p>

<p>I acquired an accelerometer signal taken from my cellphone (Samsung Galaxy Note 2, sampling rate $\approx 99 Hz$). I would like to analyze frequency from $0.3 Hz$ to $5.0 Hz$</p>

<p>My procedure is following steps:</p>

<ol>
<li>combination: let say a sensor yields 3 channels $x$, $y$, $z$. The combination is to produce a new channel $v = \sqrt{(x * x + y * y + z * z)}$</li>
<li>Perform a median filter: to make signal smoothly</li>
<li>Butter-worth filter: my cutoff is from $0.3 Hz$ to $5.0 Hz$</li>
<li><p>FFT</p>

<p>Below image is my demonstrate with segment of 120 time-points with 4 steps: (can explore more at my <a href="https://www.youtube.com/watch?v=v3gsYjdC9dY" rel="nofollow">video</a>)
<a href="http://i.stack.imgur.com/m1C5J.png" rel="nofollow"><img src="http://i.stack.imgur.com/m1C5J.png" alt="a segment of 120 time-points"></a></p></li>
</ol>

<p>I observed result of step 3 and 4 do not change while signal varies over time</p>

<p>My question is if there is anything I can make sure this result is correct or not? Thanks in advance</p>

<p>Below is my code was used for applying filters</p>

<pre><code>from __future__ import division
import numpy as np
from numpy.fft import rfft, rfftfreq
from numpy import absolute
import matplotlib.pyplot as plt
from scipy.signal import medfilt, hilbert
import pandas as pd

chunk = 120
LOW_CUT = 0.3
HIGH_CUT = 5.0
FS = 99
freqs = rfftfreq(chunk, 1 / FS)
_accel = pd.read_csv('data.csv')
for k, g in _accel.groupby(np.arange(len(_accel)) // chunk):
    _v = g['v'].values
    _v = medfilt(_v, 7)
    _v = butter_bandpass_filter(_v, LOW_CUT, HIGH_CUT, FS, order=4)
    v = 1 / chunk * absolute(rfft(_v))
    plt.stem(freqs, v)
</code></pre>

<p><strong>update 1</strong> another link to download data <a href="https://1drv.ms/u/s!At6qHz_a5mXhgp1KcAYpvsiJeTXsmg" rel="nofollow">https://1drv.ms/u/s!At6qHz_a5mXhgp1KcAYpvsiJeTXsmg</a></p>

<p><strong>update 2</strong> updated sampling rate in code <code>FS = 99</code></p>

<p><strong>update 3</strong> increased chunk size to 512, <a href="https://youtu.be/CALuLN1vYtc" rel="nofollow">plotted</a> data again. Made a video of result <a href="https://youtu.be/2jc21kS5VnA" rel="nofollow">without bandpass</a> </p>

<p>The problem is that every time a new chunk of data is processed in your loop, the filtering is initialized with a default state (which correspond to the state of the filter if all previous samples were zeros). As a result, the filter barely has time to settle after the initial transient (caused by the step from those "previous" zeros to the actual data sample values), then does the same thing again for the next chunk of data. </p>

<p>One way to fix this is to filter the entire data set in one shot before processing blocks of data with the FFT:</p>

<pre><code>_v = _accel['v'].values
_v = medfilt(_v, 7)
_v = butter_bandpass_filter1(_v, LOW_CUT, HIGH_CUT, FS, order=4)
for k in np.arange(1,len(_accel)//chunk):
     v = _v[chunk*k:chunk*(k+1)]
     v = 1 / chunk * absolute(rfft(v))
     plt.stem(freqs, v)
</code></pre>

<p>Alternatively you could also keep track of the filter state (<code>zi</code> below):</p>

<pre><code>def butter_bandpass_filter(data, lowcut, highcut, fs, zi, order=5):
    b, a = butter_bandpass(lowcut, highcut, fs, order=order)
    if (zi == None):
        zi = lfilter_zi(b,a)
    y,zf = lfilter(b, a, data, zi=zi)
    return y,zf

zi = None
for k, g in _accel.groupby(np.arange(len(_accel)) // chunk):
    _v = g['v'].values
    _v = medfilt(_v, 7)
    _v,zi = butter_bandpass_filter(_v, LOW_CUT, HIGH_CUT, FS, zi, order=4)
    v = 1 / chunk * absolute(rfft(_v))
    plt.stem(freqs, v)
</code></pre>

python signals fft sensor