Scrapy Post Data
<p>Im moving from python requests to scrapy, I'd like to make a post request that clicks a button at the bottom of an instagram hashtag page.</p>

<p>The cURL is this</p>

<pre><code>curl "https://www.instagram.com/query/" -H "cookie: mid=VwBJIwAEAAGiVNY3epWm9pRgD9Ge; fbm_124024574287414=base_domain=.instagram.com; ig_pr=1; ig_vw=956; s_network=; fbsr_124024574287414=5HQEzU7XMqOLO4KeQMmSvyBcKsH2svemV1-nWIE4_iM.eyJhbGdvcml0aG0iOiJITUFDLVNIQTI1NiIsImNvZGUiOiJBUUQ0TnNLMjVCZmdvUFN4TjdfODNQaW81Z3U4MTNaZmZWVlNCcEdJNUdRWlczdmdfNGVXNXJyck5Sc3NXRFlSWjZiZEpWMU95V3hNUUcwSE9qMHItYlRiYk40VXpNZG5aLUJ5Zzk0VWZNSW1RZTd4R1JzTS1yaXRabmc0Z3FYNkpwbnF4b0VXajRPNEVGSDVoTXBCUFNHUGNHN0RHQ01uSjFLeXh1dllOc2cyaFpnSDFheVI0RUhMbE1nZGM4emVrNm9DXzdLa2s1TUoyYzhyYmEwWXo1VkI1bVVmS3NvLS11dXVxdjJlRmxFUHpYczVNQ3E1bW5BRk5IeWxxMG9veENQcXcwWUVLSnpsNnZSUzFReGUzQWpsQzFPU0cySU1QM0wwMGhUcnRraFF4ZEFhZElVMUtNNUw5VTRab2dlbjltdUFadkJjV0U3UUMxeTdibDRyTzhwWCIsImlzc3VlZF9hdCI6MTQ3NDkzODQ3MywidXNlcl9pZCI6IjEzNzc3ODgzNjkifQ; csrftoken=th33gPnvrsNS74reomY69ETfojX2avQ7" -H "origin: https://www.instagram.com" -H "accept-encoding: gzip, deflate, br" -H "accept-language: en-US,en;q=0.8" -H "user-agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36" -H "x-requested-with: XMLHttpRequest" -H "x-csrftoken: th33gPnvrsNS74reomY69ETfojX2avQ7" -H "x-instagram-ajax: 1" -H "content-type: application/x-www-form-urlencoded" -H "accept: */*" -H "referer: https://www.instagram.com/explore/tags/love/" -H "authority: www.instagram.com" --data "q=ig_hashtag(love)+"%"7B+media.after(J0HV-nGYwAAAF0HV-nGXAAAAFjgA"%"2C+10)+"%"7B"%"0A++count"%"2C"%"0A++nodes+"%"7B"%"0A++++caption"%"2C"%"0A++++code"%"2C"%"0A++++comments+"%"7B"%"0A++++++count"%"0A++++"%"7D"%"2C"%"0A++++comments_disabled"%"2C"%"0A++++date"%"2C"%"0A++++dimensions+"%"7B"%"0A++++++height"%"2C"%"0A++++++width"%"0A++++"%"7D"%"2C"%"0A++++display_src"%"2C"%"0A++++id"%"2C"%"0A++++is_video"%"2C"%"0A++++likes+"%"7B"%"0A++++++count"%"0A++++"%"7D"%"2C"%"0A++++owner+"%"7B"%"0A++++++id"%"0A++++"%"7D"%"2C"%"0A++++thumbnail_src"%"2C"%"0A++++video_views"%"0A++"%"7D"%"2C"%"0A++page_info"%"0A"%"7D"%"0A+"%"7D&amp;ref=tags"%"3A"%"3Ashow" --compressed
</code></pre>

<p>So for the form data I have tried two things:</p>

<pre><code>body = response.xpath("//body")
html = str(body.extract())
end_cursor = re.search(r"\"end\_cursor\"\: \"(.+?)\"", html).group(1)

data = "q=ig_hashtag({})+%7B+media.after({}+10)+%7B%0A++count%2C%0A++nodes+%7B%0A++++caption%2C%0A++++code%2C%0A++++comments+%7B%0A++++++count%0A++++%7D%2C%0A++++comments_disabled%2C%0A++++date%2C%0A++++dimensions+%7B%0A++++++height%2C%0A++++++width%0A++++%7D%2C%0A++++display_src%2C%0A++++id%2C%0A++++is_video%2C%0A++++likes+%7B%0A++++++count%0A++++%7D%2C%0A++++owner+%7B%0A++++++id%0A++++%7D%2C%0A++++thumbnail_src%2C%0A++++video_views%0A++%7D%2C%0A++page_info%0A%7D%0A+%7D&amp;ref=tags%3A%3Ashow".format(tag, end_cursor)
url = 'https://www.instagram.com/query/'

yield Request(url, body=data, method="POST", callback=self.parseHashtag)
</code></pre>

<p>and this</p>

<pre><code>data = {"q" :"ig_hashtag({})+%7B+media.after({}+10)+%7B%0A++count%2C%0A++nodes+%7B%0A++++caption%2C%0A++++code%2C%0A++++comments+%7B%0A++++++count%0A++++%7D%2C%0A++++comments_disabled%2C%0A++++date%2C%0A++++dimensions+%7B%0A++++++height%2C%0A++++++width%0A++++%7D%2C%0A++++display_src%2C%0A++++id%2C%0A++++is_video%2C%0A++++likes+%7B%0A++++++count%0A++++%7D%2C%0A++++owner+%7B%0A++++++id%0A++++%7D%2C%0A++++thumbnail_src%2C%0A++++video_views%0A++%7D%2C%0A++page_info%0A%7D%0A+%7D&amp;ref=tags%3A%3Ashow".format(tag, end_cursor)}
yield FormRequest(url, formdata=data, callback=self.parseHashtag)
</code></pre>

<p>I am getting a 403 error so I am obviously sending the data incorrectly, am I formatting the data incorrectly or calling the post incorrectly? Those are my two thoughts but I'm quite unsure. Any help would be very appreciated, thank you.</p>

<p>The url is this - <a href="https://www.instagram.com/explore/tags/love/" rel="nofollow">https://www.instagram.com/explore/tags/love/</a></p>

<p>This is my git, <a href="https://github.com/Fuledbyramen/instagram_crawler/blob/master/instagram/spiders/instagram_spider.py" rel="nofollow">https://github.com/Fuledbyramen/instagram_crawler/blob/master/instagram/spiders/instagram_spider.py</a></p>

<p>You seem to be missing correct headers or any headers for that matter.</p>

<p>You should provide every header that you see in the network inspector, aside from cookies that scrapy managed and populates by itself.</p>

<p>You can easily extract the headers from the curl string network inspect gives you by:</p>

<pre><code>foo = '''-H "accept-encoding: gzip, deflate, br" -H "accept-language: en-US,en;q=0.8" -H "user-agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36" -H "x-requested-with: XMLHttpRequest" -H "x-csrftoken: th33gPnvrsNS74reomY69ETfojX2avQ7" -H "x-instagram-ajax: 1" -H "content-type: application/x-www-form-urlencoded" -H "accept: */*" -H "referer: https://www.instagram.com/explore/tags/love/" -H "authority: www.instagram.com"'''
headers = [s.strip(' "').split(': ') for s in foo.split('-H')]
headers = [h for h in headers if any(h)]
headers = {k: v for k,v in headers}
</code></pre>

<p>And you'll get:</p>

<pre><code> {'accept': '*/*',
 'accept-encoding': 'gzip, deflate, br',
 'accept-language': 'en-US,en;q=0.8',
 'authority': 'www.instagram.com',
 'content-type': 'application/x-www-form-urlencoded',
 'referer': 'https://www.instagram.com/explore/tags/love/',
 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36',
 'x-csrftoken': 'th33gPnvrsNS74reomY69ETfojX2avQ7',
 'x-instagram-ajax': '1',
 'x-requested-with': 'XMLHttpRequest'}
</code></pre>

<p>Some of these are totally not necessary, like referer is mostly used for analitics, accept-language, accept and accept-encoding can most likely be ignored. User-agent is managed by scrapy too.  </p>

<p>So what you have left is <code>x-crsftoken</code> which might do nothing, but usually those are hidden somewhere in the html source; <code>x-instagram-ajax</code> seems like a static header to indicate ajax request; <code>x-requested-with</code> shows request type and is mainly there to prevent man in the middle attacks, you should have it as it is to indicate that the request type to avoid being blocked.</p>

<p>Edit:
I've tried the website and you can actualyl just do a GET request with body as url parameters. Just right click request in network inspect and click <code>copy location with parameters</code> this will automatically convert the dict-like data from the body in url parameters.</p>

<p>i.e. <a href="https://www.instagram.com/query/?q=ig_hashtag(scrapy)%20%7B%20media.after(J0HV-vvswAAAF0HV-Qp7AAAAFiYA%2C%2016)%20%7B%0A%20%20count%2C%0A%20%20nodes%20%7B%0A%20%20%20%20caption%2C%0A%20%20%20%20code%2C%0A%20%20%20%20comments%20%7B%0A%20%20%20%20%20%20count%0A%20%20%20%20%7D%2C%0A%20%20%20%20comments_disabled%2C%0A%20%20%20%20date%2C%0A%20%20%20%20dimensions%20%7B%0A%20%20%20%20%20%20height%2C%0A%20%20%20%20%20%20width%0A%20%20%20%20%7D%2C%0A%20%20%20%20display_src%2C%0A%20%20%20%20id%2C%0A%20%20%20%20is_video%2C%0A%20%20%20%20likes%20%7B%0A%20%20%20%20%20%20count%0A%20%20%20%20%7D%2C%0A%20%20%20%20owner%20%7B%0A%20%20%20%20%20%20id%0A%20%20%20%20%7D%2C%0A%20%20%20%20thumbnail_src%2C%0A%20%20%20%20video_views%0A%20%20%7D%2C%0A%20%20page_info%0A%7D%0A%20%7D&amp;ref=tags%3A%3Ashow" rel="nofollow">https://www.instagram.com/query/?q=ig_hashtag(scrapy)%20%7B%20media.after(J0HV-vvswAAAF0HV-Qp7AAAAFiYA%2C%2016)%20%7B%0A%20%20count%2C%0A%20%20nodes%20%7B%0A%20%20%20%20caption%2C%0A%20%20%20%20code%2C%0A%20%20%20%20comments%20%7B%0A%20%20%20%20%20%20count%0A%20%20%20%20%7D%2C%0A%20%20%20%20comments_disabled%2C%0A%20%20%20%20date%2C%0A%20%20%20%20dimensions%20%7B%0A%20%20%20%20%20%20height%2C%0A%20%20%20%20%20%20width%0A%20%20%20%20%7D%2C%0A%20%20%20%20display_src%2C%0A%20%20%20%20id%2C%0A%20%20%20%20is_video%2C%0A%20%20%20%20likes%20%7B%0A%20%20%20%20%20%20count%0A%20%20%20%20%7D%2C%0A%20%20%20%20owner%20%7B%0A%20%20%20%20%20%20id%0A%20%20%20%20%7D%2C%0A%20%20%20%20thumbnail_src%2C%0A%20%20%20%20video_views%0A%20%20%7D%2C%0A%20%20page_info%0A%7D%0A%20%7D&amp;ref=tags%3A%3Ashow</a></p>

python curl scrapy