P4Python: use multiple threads that request perforce information at the same time
<p>I've been working on a "crawler" of sorts that goes through our repository, and lists directories and files as it goes. For every directory it enounters, it creates a thread that does the same for that directory and so on, recursively. Effectively this creates a very short-lived thread for every directory encountered in the repos. ( it doesn't take very long to request information on just one path, there are just tens of thousands of them )</p>

<p>The logic looks as follows:</p>

<pre><code>import threading
import perforce as Perforce #custom perforce class
from pathlib import Path

p4 = Perforce()
p4.connect()

class Dir():
    def __init__(self, path):
        self.dirs = []
        self.files = []
        self.path = path

        self.crawlers = []

    def build_crawler(self):
        worker = Crawler(self)
        # append to class variable to keep it from being deleted
        self.crawlers.append(worker)
        worker.start()

class Crawler(threading.Thread):
    def __init__(self, dir):
        threading.Thread.__init__(self)
        self.dir = dir

    def run(self):
        depotdirs = p4.getdepotdirs(self.dir.path)
        depotfiles = p4.getdepotfiles(self.dir.path)

        for p in depotdirs:
            if Path(p).is_dir():
                _d = Dir(self.dir, p)
                self.dir.dirs.append(_d)

        for p in depotfiles:
            if Path(p).is_file():
                f = File(p) # File is like Dir, but with less stuff, just a path.
                self.dir.files.append(f)

        for dir in self.dir.dirs:
            dir.build_crawler()
            for worker in d.crawlers:
                worker.join()
</code></pre>

<p>Obviously this is not complete code, but it represents what I'm doing.</p>

<p>My question really is whether I can create an instance of this Perforce class in the <code>__init__</code> method of the Crawler class, so that requests can be done separately. Right now, I have to call <code>join()</code> on the created threads so that they wait for completion, to avoid concurrent perforce calls.</p>

<p>I've tried it out, but it seems like there is a limit to how many connections you can create: I don't have a solid number, but somewhere along the line Perforce just started straight up refusing connections, which I presume is due to the number of concurrent requests.</p>

<p>Really what I'm asking I suppose is two-fold: is there a better way of creating a data model representing a repos with tens of thousands of files than the one I'm using, and is what I'm trying to do possible, and if so, how.</p>

<p>Any help would be greatly appreciated :)</p>

<p>I found out how to do this (it's infuriatingly simple, as with all simple solutions to overly complicated problems):</p>

<p>To build a data model that contains <code>Dir</code> and <code>File</code> classes representing a whole depot with thousands of files, just call <code>p4.run("files", "-e", path + "\\...")</code>.
This will return a list of every file in <code>path</code>, recursively. From there all you need to do is iterate over every returned path and construct your data model from there.</p>

<p>Hope this helps someone at some point.</p>

python multithreading perforce p4python