Select text from either a DIV or an underlying container, if it exists
<p>I'm having a div table where each row has two cells/columns.
The second cell/column sometimes has a clear text (<code>&lt;div class="something"&gt;Text&lt;/div&gt;</code>) while sometimes it's hidden within an "a" tag inside: <code>&lt;div class="something"&gt;&lt;a href="url"&gt;Text&lt;/a&gt;&lt;/div&gt;</code>.</p>

<p>Now, I have no problem in getting everything but the linked text. I can also get the linked text separately, but I don't know how to get everything at once, so I get three columns of data: 
1. first column text, 
2. second column text no matter if it is linked or not, 
3. link, if it exist</p>

<p>The code that extracts everything not linked and works is:</p>

<pre><code>times = scrapy.Selector(response).xpath('//div[contains(concat(" ", normalize-space(@class), " "), " time ")]/text()').extract()
titles = scrapy.Selector(response).xpath('//div[contains(concat(" ", normalize-space(@class), " "), " name ")]/text()').extract()
for time, title in zip(times, titles):
    print time.strip(), title.strip()
</code></pre>

<p>I can get the linked items only with</p>

<pre><code>ltitles = scrapy.Selector(response).xpath('//div[contains(concat(" ", normalize-space(@class), " "), " name ")]/a/text()').extract()
for ltitle in ltitles:
    print ltitle.strip()
</code></pre>

<p>But don't know how to combine the "query" to get everything together.</p>

<p>Here's a sample HTML:</p>

<pre><code>&lt;div class="programRow rowOdd"&gt;
  &lt;div class="time ColorVesti"&gt;
                        22:55
  &lt;/div&gt;
  &lt;div class="name"&gt;

                        Dnevnik

  &lt;/div&gt;  
&lt;/div&gt;

&lt;div class="programRow rowEven"&gt;
  &lt;div class="time ColorOstalo"&gt;
                        23:15
  &lt;/div&gt;

  &lt;div class="name"&gt;
    &lt;a class="recnik" href="/page/tv/sr/story/20/rts-1/2434373/kulturni-dnevnik.html" rel="/ajax/storyToolTip.jsp?id=2434373"&gt;Kulturni dnevnik&lt;/a&gt;
  &lt;/div&gt;

&lt;/div&gt;
</code></pre>

<p>Sample output (one I cannot get):</p>

<pre><code>22:55, Dnevnik, []
23:15, Kulturni dnevnik, /page/tv/sr/story/20/rts-1/2434373/kulturni-dnevnik.html
</code></pre>

<p>I either get the first two columns (without the linked text) or just the linked text with the code samples above.</p>

<p>If I understand you correctly then you should probably just iterate through program nodes and create item for every cycle. Also there's xpath shortcut <code>//text()</code> which captures all text under the node and it's childrem</p>

<p>Try something like:</p>

<pre><code>programs = response.xpath("//div[contains(@class,'programRow')]")
for program in programs:
    item = dict()
    item['name'] = program.xpath(".//div[contains(@class,'name')]//text()").extract_first()
    item['link'] = program.xpath(".//div[contains(@class,'name')]/a/@href").extract_first()
    item['title'] = program.xpath(".//div[contains(@class,'title')]//text()").extract_first()
    return item
</code></pre>

python scrapy