Extract nested tags with other text data as string in scrapy
<p>I am trying to extract data.
Here is the specific part of the html-</p>

<pre><code>     &lt;div class="readable"&gt;

        &lt;span id="freeTextContainer2123443890291117716"&gt;I write because I need to. &lt;br&gt;I review because I want to. 
    &lt;br&gt;I pay taxes because I have to. 
    &lt;br&gt;&lt;br&gt;If you want to follow me, my username is @colleenhoover pretty much everywhere except my email, which is colleenhooverbooks@gmail.com
    &lt;br&gt;&lt;br&gt;Founder of 
&lt;a target="_blank" href="http://www.thebookwormbox.com" rel="nofollow"&gt;www.thebookwormbox.com&lt;/a&gt; 
&lt;br&gt;&lt;br&gt;&lt;/span&gt;

    &lt;/div&gt;
</code></pre>

<p>I want output like this-</p>

<pre><code>    I write because I need to.
    I review because I want to.
    I pay taxes because I have to.

    If you want to follow me, my username is @colleenhoover pretty much everywhere except my email, which is colleenhooverbooks@gmail.com 
Founder of www.thebookwormbox.com 
</code></pre>

<p>I am trying this-</p>

<pre><code>aboutauthor=response.xpath('//div[@id="aboutAuthor"]/div[@class="bigBoxBody"]/div[@class="bigBoxContent containerWithHeaderContent"]/div[@class="readable"]/span[1]/text()').extract() if len(response.xpath('//div[@id="aboutAuthor"]/div[@class="bigBoxBody"]/div[@class="bigBoxContent containerWithHeaderContent"]/div[@class="readable"]/span')) == 1 else  response.xpath('//div[@id="aboutAuthor"]/div[@class="bigBoxBody"]/div[@class="bigBoxContent containerWithHeaderContent"]/div[@class="readable"]/span[2]/text()').extract()
    print aboutauthor
</code></pre>

<p>And get the output-</p>

<pre><code>[u'I write because I need to. ', u'I review because I want to. ', u'I pay taxes
because I have to. ', u'If you want to follow me, my username is @colleenhoover
pretty much everywhere except my email, which is colleenhooverbooks@gmail.com',
u'Founder of ', u' ']
</code></pre>

<p>What i do so that i get <code>www.thebookwormbox.com</code> with the output ?</p>

<p>As per my comment, you can use xpath with <code>//text()</code> to get all the children's text content.</p>

python scrapy