Is the index_db method from the SeqIO object in Biopython slow?
<p>I've got this:</p>

<pre><code>files = glob.glob(str(dir_path) + "*.fa")
index = SeqIO.index_db(index_filename, files, "fasta")

seq = index[accession] # Slow
index.close()

return seq
</code></pre>

<p>and i'm working on big files (gene sequences) but for some reasons, it takes about 4 secondes to get the sequence I'm looking for. I'm wondering if the index_db method is suppose to be that slow? Am I using the right method?</p>

<p>Thanks.</p>

<p>The first time the database is created it can take some time. The next times, if you don't delete the <code>index_filename</code> created, it should go faster.</p>

<p>Lets say you have your 25 files each with some genes. This method creates a SQLite DB that helps locating the sequences among the files, like "Get me the gene XXX" and the SQLite/index_db <em>knows</em> that the gene is in the file <code>12.fasta</code> and its exact location inside the file. So Biopython opens the file and scans quickly to the gene position.</p>

<p>Without that index_db you have to load every Records into memory, which is fast but some files might not fit in the RAM.</p>

<hr>

<p>If you want speed to fetch regions you can use <a href="http://pysam.readthedocs.io/en/latest/api.html?highlight=fasta#fasta-files" rel="nofollow">FastaFile from pysam</a> and <a href="http://www.htslib.org/doc/samtools.html" rel="nofollow">samtools</a>. Like this:</p>

<ol>
<li><p>You have to index all the fasta files with <code>faidx</code>:</p>

<pre><code>$ samtools faidx big_fasta.fas
</code></pre></li>
<li><p>From your code write something like this:</p>

<pre><code>from pysam import FastaFile

rec = FastaFile("big_fasta.fas")  # big_fasta.fas.fai must exist.
seq = rec.fetch(reference=gene_name, start=1000, end= 1200)

print(s)
</code></pre></li>
</ol>

<p>In my computer this times 2 orders of magnitude faster than Biopython for the same operation, but you only get the pure sequence of bases.</p>

python biopython