Separating HTML file with keyword for scraping
<p>I am programming in Python with Scrapy and have a huge <code>html</code> file with a structure similar to the one demonstrated below:</p>

<pre><code>&lt;span&gt;keyword&lt;/span&gt;
&lt;title&gt;Title 1&lt;/title&gt;
&lt;span&gt;Date 1&lt;/span&gt;
&lt;div&gt;Content 1&lt;/div&gt;

&lt;span&gt;keyword&lt;/span&gt;
&lt;title&gt;Title 2&lt;/title&gt;
&lt;span&gt;Date 2&lt;/span&gt;
&lt;div&gt;Content 2&lt;/div&gt;

...

&lt;span&gt;keyword&lt;/span&gt;
&lt;title&gt;Title N&lt;/title&gt;
&lt;span&gt;Date N&lt;/span&gt;
&lt;div&gt;Content N&lt;/div&gt;
</code></pre>

<p>My goal is to get all the <code>title</code>, <code>date</code>, and contents inside <code>div</code> for each section, but the sections themselves are not located in separated <code>div</code> or other elements, just one after another until the N-th section.</p>

<p>While I can try to get all the <code>title[1:N]</code>, <code>date[1:N]</code>, and <code>div[1:N]</code> as a list with <code>len() = N</code>, doing so prevent debugging as if <code>N</code> goes to 10,000 and <code>len(title)==len(date)==len(div) -&gt; False</code>, it will be hard to find where thing goes wrong (e.g. some titles are put in <code>&lt;strong&gt;</code> instead of <code>&lt;title&gt;</code>).</p>

<p>One item I noticed is to the <strong>keyword</strong> located between each section. With the help of that keyword, is it possible to separate the entire <code>html</code> into N parts, and hopefully get <code>item[i] = ["Title_i", "Date_i", "DIV_i"]</code> for each section through iteration?</p>

<p>This way missing data will be represented as <code>item[1]=["", Date_i, Div_i ]</code> and will be much easier to locate. </p>

<h2>Update</h2>

<p>Carl, the thing is that regex and even mere text processing (split, etc.) are not effective to work with big text samples. </p>

<p><strong>xPath</strong> is the technique that fits best in your case. </p>

<p>Though a target html might be malformed, still the modern DOM libraries are good to turn html into DOM-nodes structure, the latter is well fitting for parsing.
See the posts:</p>

<ul>
<li>Scrape ... using <a href="http://scraping.pro/python-lxml-scrape-online-dictionary/" rel="nofollow">Python and lxml</a> library</li>
<li><a href="http://scraping.pro/extracting-sequential-html-element/" rel="nofollow">Extracting sequential HTML elements with XPath</a> </li>
</ul>

<h2>Old answer</h2>

<p>Carl, you may try to split html file content into concise parts by keywords. </p>

<ol>
<li>You should be able to know a full set/dictionary of all possible keywords.</li>
<li>Some keywords might be duplicated inside of any <code>Content</code> part... so you'd better split not with pure keyword values, nor with <code>&lt;span&gt;keyword&lt;/span&gt;</code> expression but with the most unique <code>&lt;span&gt;keyword&lt;/span&gt;\s*&lt;title&gt;</code> and <code>&lt;span&gt;keyword&lt;/span&gt;&lt;strong&gt;</code> expressions. Thus you split parts correctly with a big probability.</li>
</ol>

python web-scraping scrapy