How to disable SSL verification on Python Scrapy?
<p>I am writing data scraping scripts for past 3 years in PHP.</p>

<p>This is simple PHP script </p>

<pre><code>$url = 'https://appext20.dos.ny.gov/corp_public/CORPSEARCH.SELECT_ENTITY';
$fields = array(
    'p_entity_name' =&gt; urlencode('AAA'),
    'p_name_type' =&gt; urlencode('A'),
    'p_search_type' =&gt; urlencode('BEGINS')
);
//url-ify the data for the POST
foreach ($fields as $key =&gt; $value) {
    $fields_string .= $key . '=' . $value . '&amp;';
}
$fields_string = rtrim($fields_string, '&amp;');
//open connection
$ch = curl_init();
//set the url, number of POST vars, POST data
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_POST, count($fields));
curl_setopt($ch, CURLOPT_POSTFIELDS, $fields_string);
//execute post
$result = curl_exec($ch);
print curl_error($ch) . '&lt;br&gt;';
print curl_getinfo($ch, CURLINFO_HTTP_CODE) . '&lt;br&gt;';
print $result;
</code></pre>

<p>It works fine only if <code>CURLOPT_SSL_VERIFYPEER</code> is <code>false</code>. It returns empty response if we enable <code>CURLOPT_SSL_VERIFYPEER</code> or if use <code>http</code> instead of <code>https</code>.</p>

<p>But, I have to do this same project in Python Scrapy, here is same code in Scrapy.</p>

<pre><code>from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import Selector
from scrapy.http.request import Request
import urllib
from appext20.items import Appext20Item

class Appext20Spider(CrawlSpider):
    name = "appext20"
    allowed_domains = ["appext20.dos.ny.gov"]
    DOWNLOAD_HANDLERS = {
    'https': 'my.custom.downloader.handler.https.HttpsDownloaderIgnoreCNError',}
    def start_requests(self):
        payload = {"p_entity_name": 'AMEB', "p_name_type": 'A', 'p_search_type':'BEGINS'}
        url = 'https://appext20.dos.ny.gov/corp_public/CORPSEARCH.SELECT_ENTITY'
        yield Request(url, self.parse_data, method="POST", body=urllib.urlencode(payload))

    def parse_data(self, response):
        print('here is repos')
        print response
</code></pre>

<p>It returns empty response. it needs to be disabled SSL Verification.</p>

<p>Please pardon my lack of knowledge in Python Scrapy, I have searched a lot about it but didn't find any solution.</p>

<p>I would recommend having a look at this page: <a href="http://doc.scrapy.org/en/1.0/topics/settings.html" rel="nofollow">http://doc.scrapy.org/en/1.0/topics/settings.html</a> it would appear that you can alter the way that the module behaves and change settings on various handlers.</p>

<p>I also believe this is a duplicate question from: <a href="http://stackoverflow.com/questions/32950694/disable-ssl-certificate-verification-in-scrapy">Disable SSL certificate verification in Scrapy</a></p>

<p>HTHs</p>

<p>Thanks,</p>

<p>//P</p>

python ssl scrapy