Double loop takes time
<p>I have a script which takes a lot of time and can't finish so far after 2 days...
I parsed 1 file into 2 dictionaries as the following:</p>

<pre><code>gfftree = {'chr1':[(gene_id, gstart, gend),...], 'chr2':[(gene_id, gstart, gend),...],...}
TElocation = {'chr1':[(TE_id, TEstart, TEend),...], 'chr2':[(TE_id, TEstart, TEend),...],...}
</code></pre>

<p>.</p>

<p>--The aim is to find TE_id whose TEstart or TEend or both are located between gene_id' gstart and gend in each chr(key).</p>

<h1>The above should be changed to "find TE_id whose range(TEstart, TEend) overlaps with any gene_id's range(gstart,gend)"</h1>

<p>Here is my code:</p>

<pre><code>TE_in_TSS = []
for TErange in TElocation[chromosome]:
    TE_id, TEstart, TEend = TErange
    for item in gfftree[chromosome]:
        gene, gstart, gend = item       
        if len(list(set(range(int(gstart),int(gend)+1)) &amp; set(range(int(TEstart),int(TEend)+1)))) &gt; 0:
            TE_in_TSS.append((gene, TE_id, TEstart, TEend))
        else:
            pass
</code></pre>

<p>So far I'm sure this loop is fine with small data, but when it comes to bigger one like 800,000 TE_id and 4,000 gene_id, it takes time...and I don't know if it could finish...</p>

<p>If the aim of the process is purely to find the gene IDs falling inside a specific start range and <em>you're not too worried about how you achieve this but are simply looking for the fastest solution</em>, then you may want to consider dropping the concept of a loop altogether and looking at a pre-existing solution mechanism.</p>

<p>Assuming your data is in CSV format, the following would suit your requirements, returning a dataframe containing the IDs, gene names and associated chromasones, grouped by chromasone.</p>

<p><strong>File: genometest.py</strong></p>

<pre><code>import pandas as pd
columns = ['id', 'chromasone', 'start', 'end', 'gene_name']

te_locations = pd.read_csv('Sequences/te.bed', delimiter='\t', header=None, names=columns)
gene_locations = pd.read_csv('Sequences/gene.bed', delimiter='\t', header=None, names=columns)

dataframe = pd.merge(te_locations, gene_locations, on=['gene_name', 'chromasone'], how='outer', suffixes=('_te', '_ge'))
dataset = dataframe.query('start_te &gt;= start_ge &amp; start_te &lt;= end_ge')[['peak_id_te', 'gene_name', 'chromasone']]
dataset.groupby('chromasone')
</code></pre>

<p><strong>Input sizes</strong></p>

<ul>
<li>TE_Locations dataset size = 337848</li>
<li>Gene_Locations dataset size = 50307</li>
</ul>

<p><strong>Output size</strong></p>

<ul>
<li>dataset size = 7085</li>
</ul>

<p><strong>Performance</strong></p>

<pre><code>$ python3 -m timeit 'import genometest'
10 loops, best of 3: 0.391 usec per loop
</code></pre>

python for-loop