Crawling Links from JSON file
<p>So, I am new to the world of web crawlers and I'm having a little difficulty crawling a simple JSON file and retrieving links from it. I am using scrapy framework to try and accomplish this. </p>

<p>My JSON example file: </p>

<pre><code>{

"pages": [

{

  "address":"http://foo.bar.com/p1",

  "links": ["http://foo.bar.com/p2",

   "http://foo.bar.com/p3", "http://foo.bar.com/p4"]

 },

 {

  "address":"http://foo.bar.com/p2",

  "links": ["http://foo.bar.com/p2",

   "http://foo.bar.com/p4"]

  },

 {

   "address":"http://foo.bar.com/p4",

    "links": ["http://foo.bar.com/p5",

    "http://foo.bar.com/p1", "http://foo.bar.com/p6"]

   },

   {

      "address":"http://foo.bar.com/p5",

       "links": []

     },

     {

       "address":"http://foo.bar.com/p6",

       "links": ["http://foo.bar.com/p7",

        "http://foo.bar.com/p4", "http://foo.bar.com/p5"]

      }

    ]

  }
</code></pre>

<p>My items.py file</p>

<pre><code>import scrapy
from scrapy.item import Item, Field


class FoobarItem(Item):
     # define the fields for your item here like:
    title = Field()
    link = Field()
</code></pre>

<p>My spider file</p>

<pre><code>from scrapy.spider import Spider
from scrapy.selector import Selector
from foobar.items import FoobarItem

class MySpider(Spider):
    name = "foo"
    allowed_domains = ["localhost"]
    start_urls = ["http://localhost/testdata.json"]


   def parse(self, response):
       yield response.url
</code></pre>

<p>Eventually I would like to crawl the file and return the links in the object without duplicates but right now I am even struggling to crawl the json. I thought the code above would crawl through the json object and return the links but my output file is empty. Not sure what I'm doing wrong but any help would be appreciated</p>

<p>So first thing is you need to have a way to parse the json file, <code>json</code> lib should do nicely. Then the next bit would be to run your crawler with the url.</p>

<pre><code>import json

with open("myExample.json", 'r') as infile:
     contents = json.load(infile)

#contents is now a dictionary of your json but it's a json array/list
#continuing on you would then iterate through each dictionary 
#and fetch the pieces you need.

    links_list = []
    for item in contents:
         for key, val in item.items():
               if 'http' in key:
                    links_list.append(key)
               if 'http' in value:
                  if isinstance(value, list):
                       for link in value:
                              links_list.append(link)
                  else:
                       links_list.append(value)
    #get rid of dupes
    links_list = list(set(links_list))
#do rest of your crawling with list of links
</code></pre>

python scrapy web-crawler