Nesting item data in Scrapy
<p>I'm fairly new to Python and Scrapy and have issues wrapping my head around how to create nested JSON with the help of Scrapy.</p>

<p>Selecting the elements I want from HTML has not been a problem with the help of XPath Helper and some Googling. I am however not quite sure how Iâm supposed to get the JSON structure that I want. </p>

<p>The JSON structure I desire would look like:</p>

<pre><code>{"menu": {
    "Monday": {
        "alt1": "Item 1",
        "alt2": "Item 2",
        "alt3": "Item 3"
    },
    "Tuesday": {
        "alt1": "Item 1",
        "alt2": "Item 2",
        "alt3": "Item 3"
    }
}}
</code></pre>

<p>The HTML looks like:</p>

<pre><code>&lt;ul&gt;
    &lt;li class="title"&gt;&lt;h2&gt;Monday&lt;/h2&gt;&lt;/li&gt;
    &lt;li&gt;Item 1&lt;/li&gt;
    &lt;li&gt;Item 2&lt;/li&gt;
    &lt;li&gt;Item 3&lt;/li&gt;
&lt;/ul&gt;
&lt;ul&gt;
    &lt;li class="title"&gt;&lt;h2&gt;Tuesday&lt;/h2&gt;&lt;/li&gt;
    &lt;li&gt;Item 1&lt;/li&gt;
    &lt;li&gt;Item 2&lt;/li&gt;
    &lt;li&gt;Item 3&lt;/li&gt;
&lt;/ul&gt;
</code></pre>

<p>I did find <a href="http://stackoverflow.com/a/25096896/6856987">http://stackoverflow.com/a/25096896/6856987</a>, I was however not able to adapt this to fit my needs. I would greatly appreciate a nudge in the right direction on how I would accomplish this.</p>

<p>Edit: With the nudge provided by Padraic I managed to get one step closer to what I want to accomplish. I've come up with the following, which is a slight improvement over my previous situation. The JSON is still not quite where I want it.</p>

<p>Scrapy spider:</p>

<pre><code>import scrapy
from dmoz.items import DmozItem

class DmozSpider(scrapy.Spider):
    name = "dmoz"
    start_urls = ['http://urlto.com']

    def parse(self, response):
        uls = response.xpath('//ul[position() &gt;= 1 and position() &lt; 6]')
        item = DmozItem()
        item['menu'] = {}
        item['menu'] = {"restaurant": "name"}
        for ul in uls:
                item['menu']['restaurant']['dayOfWeek'] = ul.xpath("li/h2/text()").extract()
                item['menu']['restaurant']['menuItem'] = ul.xpath("li/text()").extract()
                yield item
</code></pre>

<p>Resulting JSON: </p>

<pre><code>[  
    {  
        "menu":{  
            "dayOfWeek":[  
                "Monday"
            ],
            "menuItem":[  
                "Item 1",
                "Item 2",
                "Item 3"
            ]
        }
    },
    {  
        "menu":{  
            "dayOfWeek":[  
                "Tuesday"
            ],
            "menuItem":[  
                "Item 1",
                "Item 2",
                "Item 3"
            ]
        }
    }
]
</code></pre>

<p>It sure feels like I'm doing a thousand and a one things wrong with this, hopefully someone more clever than me can point me the right way.</p>

<p>You just need to find all the uls and then extract the lis to group them, an example using lxml below:</p>

<pre><code>from lxml import html

h = """&lt;ul&gt;
    &lt;li class="title"&gt;&lt;h2&gt;Monday&lt;/h2&gt;&lt;/li&gt;
    &lt;li&gt;Item 1&lt;/li&gt;
    &lt;li&gt;Item 2&lt;/li&gt;
    &lt;li&gt;Item 3&lt;/li&gt;
&lt;/ul&gt;
&lt;ul&gt;
    &lt;li class="title"&gt;&lt;h2&gt;Tuesday&lt;/h2&gt;&lt;/li&gt;
    &lt;li&gt;Item 1&lt;/li&gt;
    &lt;li&gt;Item 2&lt;/li&gt;
    &lt;li&gt;Item 3&lt;/li&gt;
&lt;/ul&gt;"""

tree = html.fromstring(h)

uls = tree.xpath("//ul")

data = {}
# iterate over all uls
for ul in uls:
    # extract the ul's li's
    lis = ul.xpath("li")
    # use the h2 text as the key and all the text from the remaining as values
    # with enumerate to add the alt logic
    data[lis[0].xpath("h2")[0].text] =  {"alt{}".format(i): node.text for i, node in enumerate(lis[1:], 1)}

print(data)
</code></pre>

<p>Which would give you:</p>

<pre><code>{'Monday': {'alt1': 'Item 1', 'alt2': 'Item 2', 'alt3': 'Item 3'},
 'Tuesday': {'alt1': 'Item 1', 'alt2': 'Item 2', 'alt3': 'Item 3'}}
</code></pre>

<p>If you wanted to put it into a single comporehension:</p>

<pre><code>data = {lis[0].xpath("h2")[0].text:
               {"alt{}".format(i): node.text for i, node in enumerate(lis[1:], 1)}
                    for lis in (ul.xpath("li") for ul in tree.xpath("//ul"))}
</code></pre>

<p>Working with your edited code in your question and following the same required output:</p>

<pre><code>def parse(self, response):
    uls = response.xpath('//ul[position() &gt;= 1 and position() &lt; 6]')
    item = DmozItem()
    # just create an empty dict
    item['menu'] = {}
    for ul in uls:
        # for each ul, add a key value pair {day: {alti: each li_text skipping the first}}
        item['menu'][ul.xpath("li/h2/text()").extract_first()]\
            = {"alt{}".format(i): node.text for i, node in enumerate(ul.xpath("li[postition() &gt; 1]/text()").extract(), 1)}
    # yield outside the loop 
    yield item
</code></pre>

<p>That will give you data in one dict like:</p>

<pre><code>In [15]: d = {"menu":{'Monday': {'alt1': 'Item 1', 'alt2': 'Item 2', 'alt3': 'Item 3'},
                  'Tuesday': {'alt1': 'Item 1', 'alt2': 'Item 2', 'alt3': 'Item 3'}}}

In [16]: d["menu"]["Tuesday"]
Out[16]: {'alt1': 'Item 1', 'alt2': 'Item 2', 'alt3': 'Item 3'}

In [17]: d["menu"]["Monday"]
Out[17]: {'alt1': 'Item 1', 'alt2': 'Item 2', 'alt3': 'Item 3'}

In [18]: d["menu"]["Monday"]["alt1"]
Out[18]: 'Item 1'
</code></pre>

<p>That matches your original question expected output more than your new but I see no advantage to what you are doing in the new logic adding  <code>"dayOfWeek"</code> etc.. </p>

python json scrapy