Matching entities with Google AppEngine MapReduce
<p>I have the following problem: I am using Google Datastore to store two kinds, <strong>Companies</strong> and <strong>Applicants</strong>, both of them come from different data sources so there is no way to link them directly (no common ID).</p>

<p>What I am trying  to achieve is to compare the <strong>name</strong> of the entities of these two kinds (after doing some normalization) and save these as a new entity of type <strong>Match</strong>.</p>

<p>For that I created a MapReduce job on appengine that go through all the applicants (~ 1M entities) and for each entity queries for Companies with the same name (~ 10M entities in total), but the process is painfully slow, I am getting the following throughput:</p>

<p>mapper-calls: 691411 (9.27/sec avg.)<br/>
mapper-walltime-ms: 1724026200 (23108.41/sec avg.)</p>

<p>The wall time seems for me a bit too high but I am not sure of what it means, I am running 32 shards and the code is the following:</p>

<pre><code>def match_map(applicant):        
    if(applicant.applicant_name_normalised != ""):
        # Check against the companies
        cps = Company.query(Company.name_normalised == applicant.applicant_name_normalised).fetch(projection=[Company.dissolved])
        if(len(cps) &gt; 0):
            is_potential = True
        else:
            return

        m = Match(id=applicant.key.id())
        idList = []

        for c in cps:
            idList.append(c.key)
            if(c.dissolved != True):
                is_potential = False

        m.companies = idList
        m.applicant = applicant.key
        m.is_potential = is_potential
        if(is_potential):
            yield op.db.Put(m)
    idList[:] = []
</code></pre>

<p>How could I implement this so it runs faster? I am almost to the point of leaving google datastore and doing this on another database or using scripts, I know queries can be expensive but 10 entities/s is just way lower than what I was expecting.</p>

<p>Thanks</p>

<p>I can think of minor tips that could optimize what you do such as:</p>

<p>Instead of fetching all the results, iterate and bail fast if c.dissolved != True (e.g. starts with is_potential = True, set it to True in iterator or return if c.dissolved != True)</p>

<p>However, understanding the data model and what you are trying to achieve may help to come up with a more optimal solution. In general Cloud Datastore is not
designed for cross Kinds joins and maybe Google Cloud SQL would better fit this problem.</p>

<p>Also, data growth expectation would help. As of now loading (10 + 1) millions record (or projected data to save space) into memory and doing an in-memory join
would be simple and feasible but clearly does not scale.</p>

<p>Regarding scale, your current approach may also have an issue if the number of company keys per match would be to large to fit in one entity.</p>

python google-app-engine mapreduce google-cloud-datastore