How to submit a form to get URL of next webpage using mechanize?
<p>I am trying to scrape some data, decided to employ mechanize in conjunction with beautifulsoup. I have to enter the field that I want to search in a form on this webpage, then click the search button to get to the next relevant page whose URL I want to get to scrape data off.</p>

<p>The developer mode shows the following code for the form-</p>

<pre><code>&lt;form name="topsearch" id="topsearch" method="get" onsubmit="javascript:return search_post();" action=""&gt;
        &lt;input type="hidden" name="search_data" id="search_data" value=""&gt;
            &lt;input type="hidden" name="cid" id="cid" value=""&gt;
            &lt;input type="hidden" name="mbsearch_str" id="mbsearch_str"&gt;
            &lt;input type="hidden" name="topsearch_type" id="topsearch_type" value="1"&gt;
            &lt;input name="search_str" id="search_str" autocomplete="off" onkeyup="getAutosuggesion();" type="text" value="Search Quotes, News, NAVs..." onblur="if(this.value=='')this.value='Search Quotes, News, NAVs...';" onfocus="if(this.value=='Search Quotes, News, NAVs...')this.value='';if(this.value=='Search Quotes, News, NAVs...')this.value='';" class="txtsrchbox"&gt;     
            &lt;div id="autosugg_mc" class="sugbx"&gt;&lt;/div&gt;
            &lt;div class="PR srch_qote"&gt;
                &lt;div class="srchdrp" id="srchR"&gt;Quotes&lt;/div&gt;
                &lt;div id="srch" class="qubx"&gt;
                    &lt;ul class="qlist"&gt;
                        &lt;li&gt;&lt;a onclick="tab_topser('1');getAutosuggesion();" id="tab1" href="javascript:void(0)" class=""&gt;Quotes&lt;/a&gt;&lt;/li&gt;
                        &lt;li&gt;&lt;a onclick="tab_topser('2');getAutosuggesion();" id="tab2" href="javascript:void(0)" class=""&gt;NAVs&lt;/a&gt;&lt;/li&gt;
                        &lt;li&gt;&lt;a onclick="tab_topser('5');" id="tab5" href="javascript:void(0)" class=""&gt;Commodities&lt;/a&gt;&lt;/li&gt;
                        &lt;li&gt;&lt;a onclick="tab_topser('9');" id="tab9" href="javascript:void(0)" class="active"&gt;Futures&lt;/a&gt;&lt;/li&gt;
                        &lt;li&gt;&lt;a onclick="tab_topser('3');getAutosuggesion();" id="tab3" href="javascript:void(0)" class=""&gt;News&lt;/a&gt;&lt;/li&gt;
                        &lt;li&gt;&lt;a onclick="tab_topser('4');" id="tab4" href="javascript:void(0)" class=""&gt;Messages&lt;/a&gt;&lt;/li&gt;                                    
                        &lt;li&gt;&lt;a onclick="tab_topser('6');getAutosuggesion();" id="tab6" href="javascript:void(0)" class=""&gt;Notices&lt;/a&gt;&lt;/li&gt;
                        &lt;li&gt;&lt;a onclick="tab_topser('7');" id="tab7" href="javascript:void(0)" class=""&gt;Videos&lt;/a&gt;&lt;/li&gt;
                        &lt;li&gt;&lt;a class="" onclick="tab_topser('8');" id="tab8" href="javascript:void(0)"&gt;All&lt;/a&gt;&lt;/li&gt;
                    &lt;/ul&gt;
                &lt;/div&gt;                  
            &lt;/div&gt;
            &lt;a href="javascript:;" onclick="$('#topsearch').submit()" style="float:left;" class="btn_search"&gt;&lt;/a&gt;    
            &lt;div class="CL"&gt;&lt;/div&gt;
            &lt;/form&gt;
</code></pre>

<p>I fill up the form with my relevant search item using-</p>

<pre><code>import pandas as  pd
import urllib2
import BeautifulSoup as bs
import mechanize

baseURL = "someBaseURL"
br = mechanize.Browser()
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)


#Open the Website
r = br.open(baseURL)

#Selecting the first form of the page
br.select_form(nr=0)
print br.geturl()

br.form['search_str'] = "Some Search"
br.submit()

print br.geturl()
</code></pre>

<p>After submitting the form, the url does NOT change to the url it goes to if I search the same string on the website manually. </p>

<p>I am getting the url after submitting as -</p>

<pre><code>'baseURL?search_data=&amp;cid=&amp;mbsearch_str=&amp;topsearch_type=1&amp;search_str=Kiri+Industries'
</code></pre>

<p>whereas if I submit manually I get to the next page with the URL -</p>

<pre><code>'baseURL/stockpricequote/dyes-pigments/kiriindustries/KDC01'
</code></pre>

<p>This is the URL I need to be able to scrape the data.</p>

<p>Is the submit button using javascript that cannot be called using mechanize, if that is the issue how can I make it to work?</p>

<p>Any help is appreciated, thank you.</p>

<p>It seems , at least from my similar problem that mechanize does not handle Javascript at all. Try using selenium it handles javascript well. I am building my script on this I'll update if it solves my issue.</p>

javascript python mechanize