How to read binary files in Python using NumPy?
<p>I know how to read binary files in Python using NumPy's <code>np.fromfile()</code> function. The issue I'm faced with is that when I do so, the array has exceedingly large numbers of the order of 10^100 or so, with random <code>nan</code> and <code>inf</code> values. </p>

<p>I need to apply machine learning algorithms to this dataset and I cannot work with this data. I cannot normalise the dataset because of the <code>nan</code> values.</p>

<p>I've tried <code>np.nan_to_num()</code> but that doesn't seem to work. After doing so, my min and max values range from 3e-38 and 3e+38 respectively, so I could not normalize it.</p>

<p>Is there any way to scale this data down? If not, how should I deal with this?</p>

<p>Thank you.</p>

<p><strong>EDIT:</strong></p>

<p>Some context. I'm working on a malware classification problem. My dataset consists of live malware binaries. They are files of the type .exe, .apk etc. My idea is store these binaries as a numpy array, convert to a grayscale image and then perform pattern analysis on it. </p>

<p>If you want to make an image out of a binary file, you need to read it in as integer, not float.  Currently, the most common format for images is unsigned 8-bit integers.</p>

<p>As an example, let's make an image out of the first 10,000 bytes of /bin/bash:</p>

<pre><code>&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; import cv2
&gt;&gt;&gt; xbash = np.fromfile('/bin/bash', dtype='uint8')
&gt;&gt;&gt; xbash.shape
(1086744,)
&gt;&gt;&gt; cv2.imwrite('bash1.png', xbash[:10000].reshape(100,100))
</code></pre>

<p>In the above, we used the OpenCV library to write the integers to a PNG file.  Any of several other imaging libraries could have been used.</p>

<p>This what the first 10,000 bytes of <code>bash</code> "looks" like:</p>

<p><a href="http://i.stack.imgur.com/IFGu8.png" rel="nofollow"><img src="http://i.stack.imgur.com/IFGu8.png" alt="enter image description here"></a></p>

python numpy machine-learning data-mining