How to read binary files in Python using NumPy?
<p>I know how to read binary files in Python using NumPy's <code>np.fromfile()</code> function. The issue I'm faced with is that when I do so, the array has exceedingly large numbers of the order of 10^100 or so, with random <code>nan</code> and <code>inf</code> values. </p>

<p>I need to apply machine learning algorithms to this dataset and I cannot work with this data. I cannot normalise the dataset because of the <code>nan</code> values.</p>

<p>I've tried <code>np.nan_to_num()</code> but that doesn't seem to work. After doing so, my min and max values range from 3e-38 and 3e+38 respectively, so I could not normalize it.</p>

<p>Is there any way to scale this data down? If not, how should I deal with this?</p>

<p>Thank you.</p>

<p><strong>EDIT:</strong></p>

<p>Some context. I'm working on a malware classification problem. My dataset consists of live malware binaries. They are files of the type .exe, .apk etc. My idea is store these binaries as a numpy array, convert to a grayscale image and then perform pattern analysis on it. </p>

<p><strong>EDIT 2</strong></p>

<ul>
<li><p>Refer this answer:  <a href="http://stackoverflow.com/a/11548224/6633975">http://stackoverflow.com/a/11548224/6633975</a></p>

<p>It states: <code>NaN</code> can't be stored in an integer array. This is a known
limitation of pandas at the moment; I have been waiting for progress
to be made with NA values in NumPy (similar to NAs in R), but it will
be at least 6 months to a year before NumPy gets these features, it
seems:</p>

<p>source:
<a href="http://pandas.pydata.org/pandas-docs/stable/gotchas.html#support-for-integer-na" rel="nofollow">http://pandas.pydata.org/pandas-docs/stable/gotchas.html#support-for-integer-na</a></p></li>
</ul>

<p><a href="http://stackoverflow.com/questions/12708807/numpy-integer-nan">Numpy integer nan</a><br>
Accepted answer states:<code>NaN</code> can't be stored in an integer array. A <code>nan</code> is a special value for float arrays <strong>only</strong>.
       There are talks about introducing a special bit that would allow    non-float arrays to store what in practice would correspond to a    <code>nan</code>, but so far (2012/10), it's only talks.
       In the meantime, you may want to consider the    <a href="http://docs.scipy.org/doc/numpy/reference/maskedarray.html" rel="nofollow"><code>numpy.ma</code></a>    package: instead of picking an invalid integer like -99999, you could    use the special <code>numpy.ma.masked</code> value to represent an invalid    value.</p>

<pre><code>a = np.ma.array([1,2,3,4,5], dtype=int)
a[1] = np.ma.masked
masked_array(data = [1 -- 3 4 5],
             mask = [False  True False False False],
       fill_value = 999999)
</code></pre>

<p><strong>EDIT 1</strong></p>

<p>To read binary file:</p>

<ol>
<li><p>Read the binary file content like this:</p>

<pre><code>with open(fileName, mode='rb') as file: # b is important -&gt; binary
    fileContent = file.read()
</code></pre>

<p>After that you can "unpack" binary data using <a href="http://docs.python.org/library/struct.html#struct.unpack" rel="nofollow">struct.unpack</a></p></li>
<li><p>If you are using <code>np.fromfile()</code> function:</p>

<p><a href="http://docs.scipy.org/doc/numpy/reference/generated/numpy.fromfile.html" rel="nofollow"><code>numpy.fromfile</code></a>,
which can read data from both text and binary files. You would first
construct a data type, which represents your file format, using
<a href="http://docs.scipy.org/doc/numpy/reference/generated/numpy.dtype.html" rel="nofollow"><code>numpy.dtype</code></a>,
and then read this type from file using <code>numpy.fromfile</code>.</p></li>
</ol>

python numpy machine-learning data-mining