Using more memory than available
<p>I have written a program that expands a database of prime numbers. This program is written in python and runs on windows 10 (x64) with 8GB RAM.</p>

<p>The program stores all primes it has found in a <code>list</code> of <code>integers</code> for further calculations and uses approximately <code>6-7GB</code> of RAM while running. During some runs however, this figure has dropped to below <code>100MB</code>. The memory usage then stays low for the duration of the run, though increasing as expected as more numbers are added to the prime array. Note that not all runs result in a memory drop.</p>

<p><em>Memory usage measured with task manager</em></p>

<p>These, seemingly random, drops has led me the following theories:</p>

<ol>
<li>There's a bug in my code, making it drop critical data and messing up the results (most likely but not supported by the results)</li>
<li>Python just happens to optimize my code extremely well after a while.</li>
<li>Python or Windows is compensating for my over-usage of the RAM by cleaning out portions of my prime-number array that aren't used that much. (eventually resulting in incorrect calculations)</li>
<li>Python or Windows is compensating for my over-usage of the RAM by allocating disk space instead of ram.</li>
</ol>

<h1>Questions</h1>

<ol>
<li>What could be the reason(s) for this memory drop?</li>
<li>How does python handle programs that use more than available RAM?</li>
<li>How does Windows handle programs that use more than available RAM?</li>
</ol>

<p>Have you heard of paging? Windows dumps some ram (that hasn't been used in a while) to your hard drive to keep your computer from running out or ram and ultimately crashing.</p>

<p>Only Windows deals with memory management. Although, if you use Windows 10, it will also compress your memory, somewhat like a zip file.</p>

python windows memory memory-management