Reading and closing large amount of files in new class result in OSError: Too many open files
<p>I have large amount of files and I need to traverse them and search for some strings, when string is found, the file is copied into new folder, otherwise its closed.</p>

<p>Here is example code:</p>

<pre><code>import os
import stringsfilter


def apply_filter(path, filter_dict):
    dirlist = os.listdir(path)
    for directory in dirlist:
        pwd = path + '/' + directory
        filelist = os.listdir(pwd)
        for filename in filelist:
            if filename.split('.')[-1] == "stats":
                sfilter = stringsfilter.StringsFilter(pwd, filename, filter_dict["strings"])
                sfilter.find_strings_and_move()
</code></pre>

<p>and here is stringsfilter.py:</p>

<pre><code>import main
import codecs
import os
import shutil


class StringsFilter:

    strings = None

    def __init__(self, filepath, filename, strings):
        self.filepath = filepath
        self.filename = filename
        self.strings = strings
        self.logger = main.get_module_logger("StringsFilter")
        self.file_desc = codecs.open(self.filepath + '/' + self.filename, 'r', encoding="utf-8-sig")
        self.logger.debug("[-] Strings: " + str(self.strings))
        self.logger.debug("[-] Instantiating class Strings Filter, filename: %s " % self.filename)

    def find_strings_and_move(self):
        for line in self.file_desc.readlines():
            for string in self.strings:
                if string in line:
                    self.move_to_folder()
                    return
        self.close()

    def move_to_folder(self):
        name = self.filename.split('.')[0]
        os.mkdir(self.filepath + '/' + name)
        shutil.copyfile(self.filepath + '/' + self.filename,
                        self.filepath + '/' + name + '/' + self.filename)
        self.close()

    def close(self):
        if self.file_desc:
            self.logger.debug("[-] Closing file %s" % self.filename)
            self.file_desc.close()
</code></pre>

<p>main.py:</p>

<pre><code>import logging

def get_module_logger(name):
    # create logger
    logger = logging.getLogger(name)

    # set logging level to log everything
    logger.setLevel(logging.DEBUG)

    # create file handler which logs everything
    fh = logging.FileHandler('files.log')
    fh.setLevel(logging.DEBUG)

    # create console handler
    ch = logging.StreamHandler()
    ch.setLevel(logging.INFO)

    # create formatter and add it to the handlersi
    formatter = logging.Formatter('[%(asctime)s] [%(name)-17s] [%(levelname)-5s] - %(message)s')
    fh.setFormatter(formatter)
    ch.setFormatter(formatter)

    # add the handlers to the logger
    logger.addHandler(fh)
    logger.addHandler(ch)
    return logger
</code></pre>

<p>in log I can see following:</p>

<pre><code>[2016-10-13 10:07:07,002] [StringsFilter    ] [DEBUG] - [-] Strings: ['DEVICE_PROBLEM']
[2016-10-13 10:07:07,002] [StringsFilter    ] [DEBUG] - [-] Instantiating class Strings Filter, filename: file1.stats 
[2016-10-13 10:07:07,003] [StringsFilter    ] [DEBUG] - [-] Closing file file1.stats
[2016-10-13 10:07:07,003] [StringsFilter    ] [DEBUG] - [-] Strings: ['DEVICE_PROBLEM']
[2016-10-13 10:07:07,003] [StringsFilter    ] [DEBUG] - [-] Strings: ['DEVICE_PROBLEM']
[2016-10-13 10:07:07,004] [StringsFilter    ] [DEBUG] - [-] Instantiating class Strings Filter, filename: file2.stats 
[2016-10-13 10:07:07,004] [StringsFilter    ] [DEBUG] - [-] Instantiating class Strings Filter, filename: file2.stats 
[2016-10-13 10:07:07,004] [StringsFilter    ] [DEBUG] - [-] Closing file file2.stats
[2016-10-13 10:07:07,004] [StringsFilter    ] [DEBUG] - [-] Closing file file2.stats
[2016-10-13 10:07:07,005] [StringsFilter    ] [DEBUG] - [-] Strings: ['DEVICE_PROBLEM']
[2016-10-13 10:07:07,005] [StringsFilter    ] [DEBUG] - [-] Strings: ['DEVICE_PROBLEM']
[2016-10-13 10:07:07,005] [StringsFilter    ] [DEBUG] - [-] Strings: ['DEVICE_PROBLEM']
[2016-10-13 10:07:07,005] [StringsFilter    ] [DEBUG] - [-] Instantiating class Strings Filter, filename: file3.stats 
[2016-10-13 10:07:07,005] [StringsFilter    ] [DEBUG] - [-] Instantiating class Strings Filter, filename: file3.stats 
[2016-10-13 10:07:07,005] [StringsFilter    ] [DEBUG] - [-] Instantiating class Strings Filter, filename: file3.stats 
[2016-10-13 10:07:07,006] [StringsFilter    ] [DEBUG] - [-] Closing file file3.stats
[2016-10-13 10:07:07,006] [StringsFilter    ] [DEBUG] - [-] Closing file file3.stats
[2016-10-13 10:07:07,006] [StringsFilter    ] [DEBUG] - [-] Closing file file3.stats
</code></pre>

<p>And it goes on, it seems like with every iteration, each statement from <strong>init</strong> is done once more, until there are too many files open and program ends with </p>

<pre><code>OSError: [Errno 24] Too many files open
</code></pre>

<p>I can't understand, why statements from <strong>init</strong> are called multiple times each time the instance is created.</p>

<p>Reason why you have same thing logged multiple times:
Every time <code>main.get_module_logger("StringsFilter")</code> is called, you call <code>logger.addHandler(...)</code> on <strong>the same logger</strong> returned from <code>logging.getLogger(name)</code>, so you get multiple handlers in one logger. Better make module-level logger</p>

<pre><code>import ...
LOG = main.get_module_logger("StringsFilter")
class StringsFilter:...
</code></pre>

<p>Regarding open files, I don't see the reason, but consider using <code>with open(filename) as f:</code> syntax in <code>find_strings_and_move()</code>. </p>

<pre><code>LOG = main.get_module_logger("StringsFilter")
class StringsFilter:

    strings = None

    def __init__(self, filepath, filename, strings):
        self.filepath = filepath
        self.filename = filename
        self.strings = strings
        LOG.debug("[-] Strings: " + str(self.strings))
        LOG.debug("[-] Instantiating class Strings Filter, filename: %s " % self.filename)

    def find_strings_and_move(self):
        with open(self.filepath + '/' + self.filename, 'r') as file_desc:
            lines = file_desc.readlines()
        for line in lines:
            for string in self.strings:
                if string in line:
                    self.move_to_folder()
                    return

    def move_to_folder(self):
        name = self.filename.split('.')[0]
        os.mkdir(self.filepath + '/' + name)
        shutil.copyfile(self.filepath + '/' + self.filename,
                        self.filepath + '/' + name + '/' + self.filename)
</code></pre>

<p>This way you make sure the file is closed 1) before move 2) always</p>

python python-3.4