If number comparison too slow
<p>I have a buffer and I need to make sure I don't exceed certain size. If I do, I want to append the buffer to a file and empty it.</p>

<p>My code:</p>

<pre><code>import sys

MAX_BUFFER_SIZE = 4 * (1024 ** 3)

class MyBuffer(object):
    b = ""

    def append(self, s):
        if sys.getsizeof(self.b) &gt; MAX_BUFFER_SIZE:
            #...print to file... empty buffer
            self.b = ""
        else:
            self.b += s


buffer = MyBuffer()
for s in some_text:
    buffer.append(s)
</code></pre>

<p>However, this comparison (<code>sys.getsizeof(self.buffer) &gt; MAX_BUFFER_SIZE</code>) is way too slow (ie. without comparison, the whole execution takes less than 1 second, with comparison it takes like 5 minutes).</p>

<p>At the moment I can fit the whole <code>some_string</code> to memory and so buffer actually never gets bigger than <code>MAX_BUFFER_SIZE</code>, but I must make sure my code work for huge files (several TBs in size) too.</p>

<p>Edit:</p>

<p>This code runs in under 1 second:</p>

<pre><code>import sys

buffer = ""
for s in some_text:
    buffer += s

#print out to file
</code></pre>

<p>The problem is that the buffer might become too big.</p>

<p>Similarly, this code also runs under 1 second:</p>

<pre><code>import sys

MAX_BUFFER_SIZE = 4 * (1024 ** 3)

class MyBuffer(object):
    b = ""

    def append(self, s):
        print sys.getsizeof(self.b)


buffer = MyBuffer()
for s in some_text:
    buffer.append(s)
</code></pre>

<p><strong>EDIT 2:</strong></p>

<p>Sorry, the slow part is actually appending to the buffer, not the comparison itself as I thought... When I was testing the code, I commented out the whole <code>if / else</code> statement instead of just the first part.</p>

<p>Hence, is there an efficient way to keep a buffer?</p>

<p>Undeleting and editing my answer based on edits to the question.</p>

<p>It's incorrect to assume that the comparision is slow. In fact the comparision is fast. Really, really fast. </p>

<p>Why don't you avoid re inventing the wheel by using buffered IO?</p>

<blockquote>
  <p>The optional buffering argument specifies the fileâs desired buffer size: 0 means unbuffered, 1 means line buffered, any other positive value means use a buffer of (approximately) that size (in bytes). A negative buffering means to use the system default, which is usually line buffered for tty devices and fully buffered for other files. If omitted, the system default is used. [2]</p>
</blockquote>

<p><a href="https://docs.python.org/2/library/functions.html#open" rel="nofollow">https://docs.python.org/2/library/functions.html#open</a></p>

python